I was trying to grab the source for CCV, but nuicode seems to be down.  I’ve made an openFrameworks add-on called “Magnetic” that uses OpenCV to do tracking of blobs, outlines, peaks, and skeletons.  You basically extend ofxMagneticApp instead of ofBaseApp and just call “getBlobs()” whenever you want info (has some nice features like integrated fullscreen settings and calibration).  As much as we love CCV here, we’re finding we need something lightweight integrated directly into the applications we build.

But I’m not writing about Magnetic, I’m writing to ask what is the best approach to remap the camera image to the image you analyze for blobs?  CCV seems to work well with both linear and lens distorted images (so long as there are more points), whereas mine sucks with a wide angle lens.  Does CCV use any undistorting techniques, or are y’all just using cvFindHomography?  I’m doing the latter, and just wanted to figure out which route to pursue.

My options are either:

A) Continue to use cvFindHomography() and allows users to add more points (a la CCV).
B) Stick with just four corners and then use arrow keys to adjust radial distortion values.

I’m thinking A would be the right route, as B could fail if the image is lopsided.  It’s important that works for any type of screen.  In fact, half of Magnetic’s whole purpose is to track limbs, etc, not touch points, so dedicated radial distortion correction is not an option.

Ideas?

A few early Magnetic videos, if interested:

More:
http://www.youtube.com/watch?v=W8kR7jbbtpY
http://www.youtube.com/watch?v=9NUa3k6cWY4

Wow Lets’ amazing results! Really awesome that it can shift context and track body vs hands. As for the the distortion correction question there are a couple routes you could take:

One would be doing barrel distortion correction which could use a checker board pattern to calibrate (http://nuigroup.com/forums/viewthread/1162/)
As for CCV method we use a typical input mapping technique using floats, so the more points you add the more distortion can be corrected with no major performance impact (thx to cerupcat)
Also if your using a PS3 Eye camera you can use the CL-Eye API to do distortion correction at the driver level which is best performance wise.

One thing to note is if you dont have “direct input” using the first solution (using a checker board pattern may be good solution) as it does not require user to “touch” the input points when calibrating.

Sorry about NUI Code being down the servers been getting a lot more traffic lately soon we’ll try upgrading to a dedicated server just almost 2x the costs. (It should be up right now heh)

Really great stuff Let’s Go Outside!

I just wanted to mention that if you use openFrameworks you have direct access to CCV as an addon (may need to update a couple things from API changes in .061). So you can get touchevents directly from CCV by including ofxNCoreVision. You can see this by looking at the testApp.cpp that comes with CCV which uses the ofxNCore addon.

CCV has chosen not to do any undistortion, but then as you noted requires more points for greater lens distortion. However, there’s many examples of undistortion on the oF forum and if you decide to undistort then you only really need 4 point calibration. You may want to take a look at ofxTouch addon which has an undistortion method + 4 points calibation. In general, if you’re using it for your own internal code, it may be better to just undistort, but for distribution I find that people have a hard time getting undistortion working well themsevles as it’s slightly more involved.

I would probably avoid homography if you’re going to use the CCV technique cause it’s not really necessary. I would use 4 point homography if you choose to undistort first though. Check out the ofxTouch and also the laserTag example on oF which both use the undistortion technique (can’t remember if it works for barrel distortion though).

Hi,
as far as I understand the CCV code it creates at startup a map in the dimension of your screen.
Every point in this map has some displacement value.
This value comes from an interpolation of three points from the calibration grid.(The screen is triangulated between the calibration points to get small triangles for interpolation).
The position of the points that where tracked as blobs are remapped with the displacement value on this pixel position.
So with this approach you have to calibrate your screen with a grid of points, but there is no fancy image processing.(But its work for blob tracking).
The opencv approach with a chessboard pattern (i guess you mean this) to get instrinsics and distortionCoeffs has the advantage to really undistort your image for better pattern recognition, but it’s a performance killer if you undistort every frame.
If you look at my augmented CCV Code from : http://nuicode.svn.com/svn/tbeta/branches/tbeta/CCV-fid/ i already implemented the OpenCV undistortion for better fiducial tracking. The code is most from the oreilly ‘Learning openCV’ book chapter 11.
(hope this explanation is understandable ) .
Your project is really amazing. Hope that some of your work is open source

Good explanation sloopidoopi;I think typically you undistort once and then save that undistortion into a map so that you don’t have to do it every frame and then you really don’t have any performance impacts.

Seth (cerupcat) - 11 March 2010 07:33 PM

Really great stuff Let’s Go Outside!

I just wanted to mention that if you use openFrameworks you have direct access to CCV as an addon (may need to update a couple things from API changes in .061). So you can get touchevents directly from CCV by including ofxNCoreVision. You can see this by looking at the testApp.cpp that comes with CCV which uses the ofxNCore addon.

CCV has chosen not to do any undistortion, but then as you noted requires more points for greater lens distortion. However, there’s many examples of undistortion on the oF forum and if you decide to undistort then you only really need 4 point calibration. You may want to take a look at ofxTouch addon which has an undistortion method + 4 points calibation. In general, if you’re using it for your own internal code, it may be better to just undistort, but for distribution I find that people have a hard time getting undistortion working well themsevles as it’s slightly more involved.

I would probably avoid homography if you’re going to use the CCV technique cause it’s not really necessary. I would use 4 point homography if you choose to undistort first though. Check out the ofxTouch and also the laserTag example on oF which both use the undistortion technique (can’t remember if it works for barrel distortion though).

Thanks!  Yeah, I realize I could have integrated CCV in directly, but I wanted to start from scratch for learning purposes.

Thanks for the explanation.  I wish I could get into the SVN to take a look.  Are you guys using like cvWarpPerspectiveQMatrix or cvWarpPerspective?  I think those aren’t too far off cvFindHomography, but I could be wrong.  Will check it out when nuicode is back online.

The more I think about it, the more I imagine show-stopper issues with undistortion.  I think I’ll take the CCV-like approach using numerous points.  And will definitely install the CL-Eye driver and see how that works for me.

Thanks,
Steve

I did look at the cvWarpPerspective when originally doing tbeta (now CCV), but any type of warping became unnecessary when doing the type of calibration CCV uses since adding more points essentially gives more accuracy without the performance hit of warp. CCV actually uses the same calibration method as touchlib with a couple performance additions so you can also look at the calibration classes from touchlib if you’d like (http://touchlib.googlecode.com). When I tried unwarping even slightly it was just too slow to be useful and adding more points seemed to be a good fix. It’s not ideal to have to calibrate 100 points, but at the same time you shouldn’t need to calibrate every time you use the hardware so i guess it’s not too bad. :D The checkerboard undistortion works really well but depending on the system (or size) it can be hard to do. From what I understand, undistortion is a pretty common thing in CV so I wouldn’t worry too much about show-stopper issues.

Love what you’re doing Steve; will any of this be available? Whatever you end up doing, let us know what route you take.

P.S- you working with fashionbuddha?

I see, so it looks you’re creating a triangle mesh that covers the camera’s area, which you then map to the screen?  Will have to try that.  I’ve added the user-defined many points thing, but am still using cvFindHomography.  I’m only running it on load and after calibration changes (not every frame), so the performance seems fine at the moment (still remapping every frame, though).

Yep, I’m at Fashionbuddha.  I think I helped out on one of your projects once.

1am+.  Bed time.  Will be hitting it hard tomorrow and should have more to show.  It’s not quite shareable yet, but working on it.

Thanks,
Steve

Yeah, I’m going to have to go with this technique (triangle mesh).  I finished out my warping method, but it just isn’t quite nailing the radial distortion.  Fortunately, I’ve already got the UI and calibration point storage going on, so its really just a matter of swapping out and moving a function call.

Thanks for pointing me to this!

I’ve got arrow keys working right now to allow you fine tune each point if you’re off a bit.  I made it so calibration points only work if there is a single blob present.  This means you can hold two or more fingers down in a spot and then use arrow keys to position the blobs perfectly to your fingers.

As is, though, things are going well with it.  Here it is tracking my limbs at 50 - 60fps, camera at 640x480, screen at 1680 x 1050.

might inspire some… when i saw this i thought of:

Ha!  I’m almost caught up with 1988 technology. 

Thanks so much guys for pointing me to this technique!  I’ve got it implemented now and radial distortion will no longer be an issue.  Actually wrote it a bit differently, but more-than-the gist is still there.  Big thanks.

I did want to mention that it’s actually a tad slower than the cvFindHomography technique for me.  But that’s because I’m having to look up the remapped coordinates of each point in a blob’s outline.  But it’s still great—I’m getting 60fps @ 640 x 480 capture size on my laptop where I was getting 90fps before.  It’s still plenty fast and the accuracy is worth it.  And I’m sure this technique is faster with higher resolution cameras.

Here’s a screenshot of the new calibration mode (attached).  I’m rendering the mesh so you can better align it with what you’re calibrating.  The idea is to run through calibration once by touching each point, then use the arrow keys to fine tune.

Ending out taking advantage of the nice weather and went camping all weekend, but will be shooting some video today.

Thanks,
Steve

Strike that… I was doing something really boneheaded in there.  I’m now getting 100+fps on my laptop capturing at 640 x 480 with *spot on* calibration.  This is really damn sweet!  Thanks again!

Here’s a screenshot of the new calibration.  In the past, we weren’t able to dial in the corners on our table without going into the XML because of the IR light bleeding over the edge.  Now, we can calibrate as usual, and then iterate through the points and use arrow keys to clean up the edges.  Rendering the mesh makes it super easy to line up.