OpenCV/JavaCV face recognition - Very similar confidence values

Question

I will explain what I am trying to do, as it seems to be relevant in order to understand my question.

I am currently trying to do face recognition of people that step in front of a camera, based on known pictures in the database.

These known pictures are being collected from an identifying Smart Card (which contains only a single frontal face picture) or a frontal face profile picture from a social network. From what I've read so far, it seems that for a good face recognition, a good amount of training images is required (50+). As such, since my collected images are very few to create a reliable training set, I instead tried using my live camera frame captures (currently using 150) as the training set, and the identified pictures collected previously as the test set. I'm not sure if what I'm trying with this is correct, so please let me know if I'm screwing up.

So, the problem is that after I have let's say, 5 identified pictures that I got from Smart Cards, I tried to do face recognition using as a training set, the 150 frames which the camera captured of my face. When trying to recognize, the confidence values for each of the 5 test faces is EXTREMELY similar, making the whole program useless, because I cannot accurately recognize anyone. Often, using different camera captures as training I get higher confidence values from pictures of random people than the picture of myself.

I would appreciate any help you can give me, because I'm at a loss here.

Thank you.

Note: I'm using the JavaCV wrapper for OpenCV to make my program, and the haarcascades that come included in the package. Eigenfaces being the algorithm used.

Solution 1

Face Recognition

Intro

I want to add this. libfacerec has been included into the official OpenCV 2.4.2, see:

• http://opencv.org/opencv-v2-4-2-released.html

That means if you are using OpenCV 2.4.2, then you have the new cv::FaceRecognizer in the contrib module. I know a Python wrapper has been added lately (thanks for that!), probably Java is also wrapped at time of writing this.

cv::FaceRecognizer comes with an extensive documentation, that'll show you how to do face recognition with lots of full source code examples:

• http://docs.opencv.org/trunk/modules/contrib/doc/facerec/index.html

If you want to know how the available face recognition algorithms (Eigenfaces, Fisherfaces, Local Binary Patterns Histograms) work, then especially read the Guide To Face Recognition with OpenCV. In there I explain how the algorithms work and mention their shortcomings:

• http://docs.opencv.org/trunk/modules/contrib/doc/facerec/facerec_tutorial.html

Face Recognition with few images

Now to your original problem of recognizing faces, when your training dataset is small. I'll write you a thorough answer, so it probably helps people coming here from Google.

Actually Eigenfaces and Fisherfaces should not be used, when you only have very few samples per person in your data set. You need data for these models to work, I can't stress that enough. The more the better. These methods are based on estimating the variance in your data, so give them some data to estimate your model from! A while ago I ran a small test on the AT&T Facedatabase (with the facerec framework), which shows the performance of these methods with a varying number of images per person:

I am not writing a publication here, nor will I back these figures with a detailed mathematical analysis. It has been done before, so I recommend everyone doubting these figures to look into (2), in order to see a very detailed analysis of the PCA (Eigenfaces) and LDA (Fisherfaces) for small training data sets.

So what I suggest is using Local Binary Patterns Histograms (3) for Face Recognition in the small sample scenario. These are also included in the OpenCV FaceRecognizer and have been proven to perform very well on small training data sets. If you combine this with a TanTriggs Preprocessing (4), you should have a really robust Face Recognition model. The TanTriggs Preprocessing is a 8-liner (or so) in Python, see https://github.com/bytefish/facerec/blob/master/py/facerec/preprocessing.py#L41 for the implementation. That should be easy to adapt to Java (or I can implement it with OpenCV, if people request it).

Literature

• (1) Belhumeur, P. N., Hespanha, J., and Kriegman, D. Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 7 (1997), 711–720.
• (2) Martinez, A and Kak, A. PCA versus LDA IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No.2, pp. 228-233, 2001.
• (3) Ahonen, T., Hadid, A., and Pietikainen, M. Face Recognition with Local Binary Patterns. Computer Vision - ECCV 2004 (2004), 469–481.
• (4) Tan, X., and Triggs, B. Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Transactions on Image Processing 19 (2010), 1635–650.

Solution 2

What you want to know is how you can perform a face recognition with only one training image. This is possible but also depends on the number of different persons you want to classify.

50+ training images are certainly not needed. For a basic face recognition you need around 50 faces to calculate your face space (eigenfaces). Perhaps you mixed it up with that. See that you have lots of variations in this faces (skin colour, glasses, form ...) You can take these faces from any face database you like. http://www.face-rec.org/ lists several databases and explains different algorithms.

After you calculated your face space you train with as many faces you have. In your case you have only one. Depending on how many different subjects you want to classify this could already work.

If you get too many false classifications I would take a look at hybrid methods. Hybrid methods combine a template matching algorithm (eigenfaces, fisherfaces) with a feature based one. In this case you take the output of your first algorithm and match the eyes, nose, eyebrows, chin shape etc. with your test face.

In short:

1. extract faces from each image with haarcascades
2. calculate your face space
3. train for each face
4. ask for a face classification
5. take the most likely classifications and check for face features

In case you haven't found it, OpenCV has also a face recognition library: https://github.com/bytefish/libfacerec

EDIT: I wouldn't use more than 10-15 components (eigenfaces).