Using OpenCV to add face and eye detection to the app

Since I have used OpenCV before, I will start with that even though it looks like other options could be better. Even if it is not trivial to integrate with a Swift iOS app, it is not that difficult either. OpenCV is C++ and talking to C++ from Swift can’t be done directly but can be done from Objective C so that is used as a bridge.

OpenCV has a number of pretrained HAAR cascades models suitable for face and eye detection that can be used directly. After testing a lot of different things, I settled on using the haarcascade_frontalface_alt2 model for detecting faces and then within the detected area, using individual detectors for each eye (haarcascade_lefteye_2splits and haarcascade_righteye_2splits). These detectors seem to be faster but tend to fail more often than the more general models so if detection fails, I use a fallback model (haarcascade_eye_tree_eyeglasses, not that infants wear eye glasses, it is just practical for testing on myself).

As can be seen, face and eye detection is pretty good but a slightly different area is found every frame so there is a lot of jitter. Also, the eye regions found seem unnecessarily large vertically, although I’ve noticed that some pupil localization algorithms use the eyebrows, so maybe not.

The most difficult part here is setting good cascade parameters. First, the detections can be slow (and that goes especially for the eye detection) so if we can limit the size range, especially downwards, of the detected features it should help a lot. However, it is not obvious from what distance the app would be used. Perhaps the maximum distance (minimum size) will be limited by the pupil detection accuracy?

Second, there are some parameters that essentially give better accuracy but less detections. They also seems correlated to detection speed to me so I’m not sure if it’s worth it, especially since it does not seem to solve the issue of the detection being unstable in terms of area.