Computer Vision - filtering convex hulls and convexity defects with OpenCV

Question

I have the problem with processing digital signals. I am trying to detect fingertips, similar to the solution that is presented here: Hand and finger detection using JavaCV.

However, I am not using JavaCV but OpenCV for android which is slightly different. I have managed to do all the steps presented in the tutorial, but filtering of convex hulls and convexity defects. This is how my image looks like:

Here is an image in another resolution:

As you can clearly see, There is to many yellow points (convex hulls) and also to many red points (convexity deffects). Sometimes between 2 yellow points there is no red point, which is quite strange (how are convex hulls calculated?)

What I need is to create simillar filtering function like in the link provided before, but using data structures of OpenCV.

Convex Hulls are type of MatOfInt ... Convexity defects are type of MatOfInt4 ...

I created also some additional data structures, because stupid OpenCV uses different types of data containing same data, in different methods...

convexHullMatOfInt = new MatOfInt();
convexHullPointArrayList = new ArrayList<Point>();
convexHullMatOfPoint = new MatOfPoint();
convexHullMatOfPointArrayList = new ArrayList<MatOfPoint>();

Here is what I did so far but it is not working good. The problem is probably with converting data in a wrong way:

Creating convex hulls and convexity defects:

public void calculateConvexHulls()
{
    convexHullMatOfInt = new MatOfInt();
    convexHullPointArrayList = new ArrayList<Point>();
    convexHullMatOfPoint = new MatOfPoint();
    convexHullMatOfPointArrayList = new ArrayList<MatOfPoint>();

    try {
        //Calculate convex hulls
        if(aproximatedContours.size() > 0)
        {
            Imgproc.convexHull( aproximatedContours.get(0), convexHullMatOfInt, false);

            for(int j=0; j < convexHullMatOfInt.toList().size(); j++)
                convexHullPointArrayList.add(aproximatedContours.get(0).toList().get(convexHullMatOfInt.toList().get(j)));
            convexHullMatOfPoint.fromList(convexHullPointArrayList);
            convexHullMatOfPointArrayList.add(convexHullMatOfPoint);    
        }
    } catch (Exception e) {
        // TODO Auto-generated catch block
        Log.e("Calculate convex hulls failed.", "Details below");
        e.printStackTrace();
    }
}

public void calculateConvexityDefects()
{
    mConvexityDefectsMatOfInt4 = new MatOfInt4();

    try {
        Imgproc.convexityDefects(aproximatedContours.get(0), convexHullMatOfInt, mConvexityDefectsMatOfInt4);

        if(!mConvexityDefectsMatOfInt4.empty())
        {
            mConvexityDefectsIntArrayList = new int[mConvexityDefectsMatOfInt4.toArray().length];
            mConvexityDefectsIntArrayList = mConvexityDefectsMatOfInt4.toArray();
        }
    } catch (Exception e) {
        Log.e("Calculate convex hulls failed.", "Details below");
        e.printStackTrace();
    }
}

Filtering:

public void filterCalculatedPoints()
    {
        ArrayList<Point> tipPts = new ArrayList<Point>();
        ArrayList<Point> foldPts = new ArrayList<Point>();
        ArrayList<Integer> depths = new ArrayList<Integer>();

        fingerTips = new ArrayList<Point>();

        for (int i = 0; i < mConvexityDefectsIntArrayList.length/4; i++)
        {
            tipPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i]));
            tipPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i+1]));
            foldPts.add(contours.get(0).toList().get(mConvexityDefectsIntArrayList[4*i+2]));
            depths.add(mConvexityDefectsIntArrayList[4*i+3]);
        }

        int numPoints = foldPts.size();
        for (int i=0; i < numPoints; i++) {
            if ((depths.get(i).intValue()) < MIN_FINGER_DEPTH)
                continue;

            // look at fold points on either side of a tip
            int pdx = (i == 0) ? (numPoints-1) : (i - 1);
            int sdx = (i == numPoints-1) ? 0 : (i + 1);

            int angle = angleBetween(tipPts.get(i), foldPts.get(pdx), foldPts.get(sdx));
            if (angle >= MAX_FINGER_ANGLE)   // angle between finger and folds too wide
                continue; 

            // this point is probably a fingertip, so add to list
            fingerTips.add(tipPts.get(i));
        }
    }

Results (white points - fingertips after filtering):

Could you help me to write proper function for filtering?

UPDATE 14.08.2013

I use standard openCV function for contour approximation. I have to change approximation value with resolution change, and hand-to-camera distance, which is quite hard to do. If the resolution is smaller, then finger consist of less pixel, thus approximation value should be lover. Same with the distance. Keeping it high will result in completely losing the finger. So I think approximation is not good approach to resolving the problem, however small value could be useful to speed up calculations:

Imgproc.approxPolyDP(frame, frame, 2 , true); 

If I use high values, then the result is like on the image below, which would be good only if distance and resolution wouldn't change. Also, I am quite surprised that default methods for hulls points and defects points doesn't have useful arguments to pass (min angle, distance etc)...

Image below presents the effect that I would like to achieve always, independently from resolution or hand-to-camera distance. Also I don't want to see any yellow points when I close my palm...

To sum everything up, I would like to know:

• how to filter the points
• how can I make resolution and distance independent approximation which will always work
• if someone knows or have some materials (graphical representation, explanation) about those data structures used in OpenCV, I would be happy to read it. (Mat, MatOfInt, MatOfPoint, MatOfPoint2, MatOfPoint4 etc.)

Solution

The convex hull at low res can be used to identify the position of the hand as a whole, it is not useful for fingers but does provide a region of interest and appropriate scale.

The higher resolution analysis should then be applied to your approximated contour, it is easy to skip any points that do not pass the "length and angle" criteria from the last two, though you may wish to "average in" instead of "skip entirely".

Your code example is a single pass of calculating convexity defects and then removing them .. that is a logic error .. you need to remove points as you go .. (a) it is faster and simpler to do everything in one-pass (b) it avoids removing points at a first pass and having to add them back later because any removal changes previous calcs.

This basic technique is very simple and so works for a basic open palm. It doesn't intrinsically understand a hand or a gesture though, so tuning the scale, angle and length parameters is only ever going to get you "so far".

References to Techniques: filter length and angle "Convexity defect" Simen Andresen blog http://simena86.github.io/blog/2013/08/12/hand-tracking-and-recognition-with-opencv/

Kinect SDK based C# Library with added finger direction detection http://candescentnui.codeplex.com/ http://blog.candescent.ch/2011/11/improving-finger-detection.html

"Self-growing and organized neural gas" (SGONG) Prof Nikos Papamarkos http://www.papamarkos.gr/uploaded-files/Hand%20gesture%20recognition%20using%20a%20neural%20network%20shape%20fitting%20technique.pdf

Commercial product David Holz & Michael Buckwald founders of "Leap Motion" http://www.engadget.com/2013/03/11/leap-motion-michael-buckwald-interview/

Comments

[Carlos Diego]

i'm really in need of help with this question, can you help me, please? stackoverflow.com/questions/61216402/…

[Marek]

I do approximation (updated post), but even then results are not acceptable (sometimes I still get more than one point on the fingertips. And also approximation size should be resolution and hand-to-camera distance dependent which is quite difficult to achieve.

[xxx]

The first github page is down. Here is the repositroy: github.com/simenandresen/handDetectionCV