Find out who has been stealing from your fridge

Good day everyone! 

No, you are not on the wrong page, it's just that today's post is written in English.

This is Kai from the Product Internationalization team of Progate, and you are looking at the post for the 17th day of the Progate Advent Calendar.

Today, I will be doing a short write-up on what you can do with small form factor computers with the computer vision library, OpenCV, and what better ways to do it than to show an actual (maybe useless) application of it?

Quick Jumps

Topic

Ever had something missing in your fridge at home or at the office? Tried asking around but no one really wants to own up? What if you had video evidence? Too troublesome to scan through the whole video footage?

What if you could capture and save the faces of suspects who looked into the fridge?

Disclaimer: Please ensure your countries' laws (or your office) allows you to capture images of other people's face!

Foreword

As this is a short write-up on a simple usage of computer vision on a small form factor computer (in this case, a Raspberry Pi -or RPi-), I will not be going into details on how the computer vision libraries and classifiers work, but instead, focus on creating this usage.

Concept

Here is a concept image on how we can capture and save the faces of people who look into the fridge.

Concept on how to capture faces of people who look into the fridge

Device

To realize the concept, we need to combine together some parts to form a device.

Here is an image of what I rigged.

RPi + Camera + Power bank and some tape

Set up in fridge

How it should work

1. A RPi with a camera module and power source is placed in the fridge.
2. A script is executed on the RPi to use the camera module and detect faces.
3. When the fridge's door is opened, the script on the RPi will attempt to detect a face from video frames captured by the camera.
4. Once a face is successfully detected, the script on the RPi crops and stores the image of the person and saves it on the RPi locally.
5. The saved face image can be accessed on the RPi GUI using a separate terminal.

You may be thinking, isn't this too obvious?

That's the whole point, you want the people who have been taking your things to know you mean business. Their faces will already be captured before they can react!

How the script works

Flowchart

Flowchart

So basically, this is an endless loop of reading video frames, processing and saving images of faces.

Script explanation

Some snippets of important parts of the script:

face_detector = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

The above line of code loads the prebuilt XML face classifier model.

---

image_frame = video_capture.read()
gray = cv2.cvtColor(image_frame, cv2.COLOR_BGR2GRAY)
faces = face_detector.detectMultiScale(gray, 1.3, 5)

The above snippet basically does the facial detection! Simple and straightforward, right?

The reason why the image frame needs to be converted to grayscale is that most of the functions provided by OpenCV require a grayscale image.

The second and third parameters in the method, detectMultiScale() determines the speed and accuracy of the detection. The default values for most tutorials are 1.3 and 5, so you can play with it.

---

for (x,y,w,h) in faces:       

Each detected face in a frame comes with 4 values, x as the initial horizontal position, y as the initial vertical position, w as the width, and h as the height. The origin coordinates (0, 0) begins on the top left!

To get a 4-sided frame, we need to get 4 coordinates using the 4 values. By utilizing w and h, we can determine the 4 coordinates with x and y.

A for-loop is used here in the case there is more than 1 face detected.

---

allowance = 20
crop = image_frame[y-allowance:y+h+allowance,x-allowance:x+w+allowance]
cv2.imwrite("suspects/suspect " + str(count) + ".jpg", crop)

The snippet above crops and saves the detected face!
The allowance is added so that in the case of a badly detected face, it is not over-cropped.

---

cv2.imshow('frame', image_frame)

This is not important to what we want to achieve here, but it creates a video frame for debugging purposes!

With the device placed in a suitable location (inside or outside the fridge), whenever a face is detected by the device, it will automatically be saved as an image and stored in the RPi. You can then access the set save directory, and then find out who has been peeping into the fridge!

Sample test

Saved faces

Here we can my captured face when I opened the fridge! (Due to COVID, I couldn't get someone else to test this out with me :( )

Some suggestions to improve the device

As this is a simple rig, the application is definitely not perfect.
There are many ways we can work towards improving the device.
I will not be going into detail as it will make this write-up too long, but here's the gist of it!

I am my own suspect

As this only uses a simple face detector, the device will save your own face data if you look at the camera too.
A workaround is to train a model of your own facial data, and when the device detects a face, it checks with the trained model of your facial data to see if it is you. If it is, the device can then choose to not save that captured face.

Destroyed evidence

As everything is stored locally in the RPi, it can be easy for the suspect to destroy the data. A workaround is to set up a simple messaging application on the existing script to send an email, or a message to your phone whenever a face is detected!

Afterword

With this, we have set up a simple device that can detect faces found during video capture. Hopefully, this will you catch and deter people who have been stealing from your fridge!

This is the end of my write-up! Please look forward to the write-up by Satoshi-san from the Engineer Team tomorrow!

The boring stuff

You find the scripts used and some references on how to recreate this on the Github repo here!