GSoC 09: Finger detection improvements DI/DSI and orientation detection (hand detection? DSI/DI)
Posted: 20 March 2009 09:11 PM   [ Ignore ]
Rank
Joined  2008-08-17
Total Posts:  9
New Member

Name: Ariel Molina
Email: ariel_zarzamora.com.mx
Location/Timezone: GMT-6
Age: 26
Education/Qualifications: PhD student
Academic and Industry Background: I studied physics and math, made a Masters in computer science (thesis in 3D monocular object tracking), currently i am studying a PhD in computer science, my work is on markerless 3D human hand posture tracking.
Open source development experience: I created a monocular 3D object tracker which i opensourced via the local server at my institute. I am very experienced in collaborative tools and versioning systems (GIT and SVN)
Activity level within the NUI Group Community: Visited on a regular basis, reading a lot of posts.
Time working multi-touch, natural user interfaces, HCI and related fields: More than 3 years.
The reason for picking specific project:
I chose NUI because i am working on vision based human-machine interfaces so it is natural for me to be interested in vision based stuff. The problems faced by NUI group in finger tracking are in my direct area of research (object tracking). I am sure good contributions can be made in refining object tracking and even create a basis for hand detection.

Project Proposal:
Finger Detection Improvements (hand detection?)
---------------------------------------------------------------------
Finger detection in DI and DSI is a greater challenge than in FTIR, because of the reduced contrast and additional artifacts (hand palm, arms). DSI and DI are also preferred for creating nice looking tables (MS Surface does DI) because there is no need of a compliant surface. I propose to improve finger detection using Meanshift for detecting “peaks”, please read [1] for a great Meanshift explanation. This would work great for DSI, and DI where the pams of the hands detection are a side effect and currently this can be an issue because just thresholding is not always the ideal way to do things and value boosting introduces noise. This kind of finger detection can be used in conjunction with current contour detection methods to greatly improve accuracy and to separate fingers (gaussian peaks), non-fingers and hand palms (mid level regions non-peaks). Fingers can be seen as peaks in a 3D surface like the ones displayed here (in spanish, Figura A) [3]. This could be a basis for palm detection.

Finger Orientation
--------------------------
There are two approaches to explore, a rather simplistic one is contour analysis over the already detected blobs, other idea is to apply Continuously Adaptive Mean Shift [2]. Camshift works on probability maps represented as grayscale images, and luckily the finger detection can be seen as a probability map because the finger is more likely to be where more pixels are, and where whiter pixels are. Thus the finger tracking can be seen as a probability map and the Continuously Adaptive Mean Shift created by Gary Bradsky can be applied. As a Bonus by using Continuously Adaptive Mean Shift we get an estimated window of where the fingers are, and then we can restrict detection areas to those areas and by boosting probabilities in such areas we could theoretically get improved finger tracking. Read [4] and scroll down for an animated explanation (girl painting) of how Camshift works and how it gets orientation.

I would like to propose myself to work on any of these areas, the first one is the one i think has more potential, as it can be the start point for true hand detection, it also it the one that requires more work. And additionally i can do the ROI selection stuff as this is not really a difficult task.

My background: I studied physics and math, masters in computer science (thesis in 3D monocular object tracking), and currently i am in a PhD in computer science my work is on markerless 3D human hand posture tracking.

I can elaborate on the methodology of the proposed ideas at your request.

[1] http://www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
[2] ftp://download.intel.com/technology/itj/q21998/pdf/camshift.pdf
[3] http://www.cimat.mx/~alram/VC/MSAA.htm
[4] http://www.robot.jussieu.fr/?op=view_profil&id=32&lang=fr

Profile
 
 
Posted: 21 March 2009 03:32 PM   [ Ignore ]   [ # 1 ]
Rank
Joined  2008-06-15
Total Posts:  30
New Member

Ohhhhh, this is a very interesting post!

The ideas is great but what about latency? And do you plan to use opencv or what else?

i’m really really really interested!

Pole

Profile
 
 
Posted: 21 March 2009 04:00 PM   [ Ignore ]   [ # 2 ]
Rank
Joined  2008-08-17
Total Posts:  9
New Member

OpenCV has an implementation of Meanshift intended for color histogram backprojections (probability maps), Using that implementation adapted to finger detection and some ROI trickery it is theoretically possible to find the peaks.  is

Camshift is being used for hi-speed colored object tracking, i used it myself for some tests. I believe we can get more than 30FPS but i can’t tell precisely how fast it would be.

On the other hand, when using Meanshift it might be unnecesary/redundant have many image processing filters. In fact a couple of filters are required now, because the need of solid and disjoint finger contours, meanshift can work on non-solid and non-disjoint clusters. So the overload of using Meanshift might be compensated by reducing image filtering, anyway testing would be need. The non disjoint cluster peak finding ability of Meanshift is the reason we could archive palm hand identification, the hand palm is just a quasi-homogeneous mid-valued region, sometimes surrounded by at least 2 peaks.

Finally, if it works but we discover is rather slow, the code and ideas generated from this project can be ported to CUDA or OpenCL because the finger peak finding algorithm would be parallelizable. I believe it will not be necessary, but it’s an option.

Profile
 
 
Posted: 22 March 2009 07:06 AM   [ Ignore ]   [ # 3 ]
Avatar
RankRank
Joined  2009-01-06
Total Posts:  213
Member

This is really som interresting stuff you got here. I don’t know as much as i’d like to know about this. But still, I think this would ge a great project.

 Signature 

---- Interactive Multi Touch Systems ----
-------- The future of computing! --------
------- Black Hole Productions -------

Profile
 
 
Posted: 02 April 2009 11:32 PM   [ Ignore ]   [ # 4 ]
Rank
Joined  2008-08-17
Total Posts:  9
New Member

Updated proposal.

File Attachments
proposal.pdf  (File Size: 524KB - Downloads: 786)
Profile
 
 
Posted: 03 April 2009 08:44 AM   [ Ignore ]   [ # 5 ]
Rank
Joined  2008-11-04
Total Posts:  8
New Member

just making some lobby(or something like that) grin , as this could be a beneficial or even a parallel project to mine. Good luck!

Profile
 
 
Posted: 03 April 2009 12:18 PM   [ Ignore ]   [ # 6 ]
Rank
Joined  2008-08-17
Total Posts:  9
New Member

Yes, i read your proposal. Even when my project focus is for finger detection enhancing, it would certainly help in your project’s hand recognition.

Profile