Determining how many people in a video are wearing masks.
The basic problem can be broken down into the following steps.
- Get a video that includes people
- Break down the video into individual frames
- Detect if there is a person in the frame
- Determine if the person exists in other frames
- Detect if the person is wearing a mask
There are a number of YouTube channels that produce videos of a person walking around an area. Luckily for us, these videos include other people walking around as well. One such example of this is the channel Rambalac which features a man walking around and exploring different parts of Japan. Using the "Download Video and Extract Frames" file we can use pytube, a package for downloading YouTube videos, along with code from this handy article (with some slight modifications) to extract each frame in a video. Depending on the video's frame rate, this can be quite a bit of data so you may want to only take the first few minutes worth of frames when starting out. Now that steps one and two are completed, we can move on to something a bit more challenging.
There are many types object detection solutions readily available. I've decided to go with something that is relatively easy to use out of the box which is a pretrained model from TensorFlow Hub. Using the tensorflow-hub package, importing and using the model takes only a few lines of code. The "Detect objects" file looks through each of the frames of the video and finds every instance of an object in the scene. We then take the confidence and class of the object to save only pictures of where the class is Person and the confidence is 40% (this can be tweaked depending on what you're trying to accomplish).
Arguably the trickiest part: we now need to determine if a person in one frame is the same as a person in subsequent frames. We can take advantage of a method called deep sort (you can read the paper here if you're interested). The basic idea is to look at a detected person and predict their trajectory using something called a Kalman Filter (a handy tutorial about these can be found here). This is then combined with a model that extracts the features of each detected person. Deep Sort compares the predicted location and features of a person to all of the locations and features of people in the next frame(s) and uses this to determine if people are the same between frames. The file "ModifiedDeepSort" is responsible for this step. The original code was written for a previous version of TensorFlow and was fixed and explained in this nanonets blog post and related GitHub repo. I've made some modifications to use the detections generated by TensorFlow rather then generate them using YOLO (though if you're looking to use YOLO, this repo is the way to go).
We can examine the before and after of our detection step and person comparison step. You can see a box is drawn around each person who has been detected along with a unique number that identifies them.
Before:
After:
The final part of the puzzle is to determine if a person is wearing or not wearing a mask. For this we need to first build a data set. The data can generally be one of three types: Mask, No Mask, and Unsure.
| Mask | No Mask | Unsure |
|---|---|---|
 |
 |
 |
There are other edge cases such as when the picture is too blurry to determine what is going on or you get some bad data and the image isn't even of a human but in general Mask, No Mask, and Unsure are the big three. Let's focus on the binary task determining if a person is wearing a mask or not. We first need to separate the pictures into groups so they can be accurately labeled. We can then build a convolutional neural network and train it on each of the images to predict if they are wearing masks or not. This is handled in the file "Mask Model Building". Once the model has been trained we can evaluate its accuracy.
We can see that the model is about 80% accurate which is fine for a first pass.
As alluded to above, 80% accuracy is decent at first but a lack of precision will make detecting masks for a series of frames difficult. For example, if we track a person through 100 frames of a video and the model sees that they are wearing a mask in 20 of the frames, does this mean that the model is 100% accurate and we only saw the person's back for the first 80 frames before they revealed their masked face? Does it mean that the person was not wearing a mask but the model detected it in 20 of the frames? We simply don't know. The endless number of combinations of scenarios can really only be dealt with by using a higher precision model or creating some rules and guessing. The real challenge now begins which is gathering data, then gathering some more data, and finally... gathering even more data to retrain and refine the model.