Improving gesture recognition: discarding invalid gestures

Several problems arised with our method of gesture recognition based on convexity defects with respect to the convex hull. We were able to discard some invalid convexity points based on the distance from the deepest point in the defect to the convex hull.

However, this approach still had a problem, which is shown in the next picture.

Poor segmentation can yield false positives, as is shown in the picture.

We did not find an easy way to tackle it, so in the end it was decided that this gesture would not be recognised. Instead, we decided that the only gesture with one valid convexity defect which would be accepted would be the thumb-up gesture, rotation-invariant.


As can be observed, this gesture has a particular width to height ratio, so gestures which don't meet a certain ratio threshold can be discarded. In particular, 1.6 was selected, being the result of dividing width/height or the inverse, whichever is >1.

Another criteria we used to validate gestures was to establish a minimum and maximum number of sides that the gesture's polygonal approximation can have. With the polygonal approximation we have chosen, the start and finish points of the convexity defect, the fingertips, can be either spikes or flat.

We chose the minimum number of sides to be 6, which we found happened with a closed fist. Then, for every convexity defect we can count 4 sides. Invalid defects change what would have been one side to two or more. Removing already counted sides, we get the following formula:

Min_sides = 6
Max_sides = min_sides + (4 x valid_defects) - (valid_defects-1) - invalid_defects

Even though not perfect, with these two simple methods we are able to discard many invalid gestures due to poor segmentation and thus have a more robust recognition.