Hey everyone,

I think somewhere in one of the other threads i mentioned that I was working on some Cocoa multi-touch stuff.  I built a big prototype codebase, and it all worked out fairly well, enough that I started to move toward something a bit more ‘real’.  So I am in the process of refactoring the prototype code, adding comments, etc..

Anyhow, so far the new code that I have up on my site just detects blobs, and displays them. Not too exciting, but it is the basis for my tracking code (that I am working on cleaning up from the prototype)

Also, it still needs some optimization, but i thought i would post it up there for anyone who was interested. you can find the code here:

http://benbritten.com/blog/2008/05/29/70/

feel free to do with it what you will, although i would love to know if you are using it for something, just out of curiosity. 

I will be posting more code as the project comes along, but since I am a freelancer, sometimes jobs come up quick and i get busy, so i thought i would post it even though it is not really a multi-touch ‘solution’ so much as a start to one.

Cheers!
-panda

That’s awesome will check it out. So many people start wanting to share/ be opensource and end up not delivering.

ding

Just curious why did you create your own capture object and did not use the QTKit capture and delegate?

BTW A tip of the hat to you my friend for that blob finding algo and all the research.  You the MAN! Plus great commenting.

ding

Great work Ben,

We will definitely use it in Cocoa OpenTouch project that we are working on with Bridger and Mantas (GSoC Student).
I will drop you an email to discuss more details.
Keep up the good work!

Best,
Pawel Solyga

ding, QTKit Capture has been introduced in Leopard if I remember correctly.
And there are still some users of Tiger around, so probably that’s the reason of not using QTKit in Ben’s code, but I might be work.

Best,
Pawel Solyga

I believe tiger 10.4.10 + quicktime 7 has QTKit. It sure makes life a whole lot easier with this kind of stuff.

EDIT:

The QuickTime Kit is a Objective-C framework (QTKit.framework) with a rich API for manipulating time-based media. Introduced in Mac OS X v10.4…

BTW awesome work on opentouch Powel!

ding

ding, yep you are right.

I myself use Leopard, but I also found this blog post about QTKit Capture

http://macdaddyworld.com/2007/07/13/quicktime-72-brings-leopard-functionality/

Thx for kind words We are all doing a great job by being part of NUI Group community. Thank you for that.

I will talk with Ben, maybe we can add QTKit Capture support to his code.

Best,
Pawel Solyga

Well keep posting here and I will be happy to help with that.  I want to see this cocoa thing succeed.

ding

will do will do

Best,
Pawel Solyga

Hey I am finishing putting together a fiducial finder class in cocoa/c if you think it will be useful for opentouch/anything I can post it. I have been veering away from an app/listener model to a more frameworks model lately.  I think that creating an executable .framework with objects that over-simplify image modification/blob tracking/etc with an xcode app template is what I am liking. I think having 2 apps open ( a blobtracking app sending tuio messages to a reciprocating app) can be a bit cumbersome.  Just throwing out some discussion materials. What do you all think?

ding

Yep for Mac OS X development, frameworks is a way to go. Of course any help is welcome and your fiducials support would be a great add-on. Are you using any external libs or you wrote everything from scratch ?
We are thinking about using NSResponder or some evens queue and don’t use TUIO in future if you decide to use framework directly.

Best,
Pawel Solyga

I am using the libfidtrack files from reactivision. They are written in c so integration is no problem. Particularly the files below.

efault_trees.h
fidtrackX.h
fidtrackX.c
floatpoint.h
segment.c
segment.h
treeidmap.c
treeidmap.h
all.trees

You basically input a 0/255 image and it returns an SNarray with fiducials objects.  Right now I have it working with an opencv IplImage so I will have to rework it for straight cocoa.  I am using QTkit and grabbing the cvimageref and converting it to an IplImage but I can just skip the conversion step and pass the cvimageref bitmapData straight to it. The bitmapData has to be a single channel with only 0’s & 255’s for the tracker to work properly so thats why I have been using opencv to process the image.  I still havent gotten as far as figuring out how to process the image natively to get the results.  I have a bunch of projects going on at once plus work and my family so progress is slow.

ding

Hey all!

I am using the older sequence grabber stuff, the new qtkit capture is great, but it doesnt allow you to get to the camera setting very easily (if at all).  (at least not that I could tell for the few days I was playing with it). With my setup, (unibrain fire-i) the auto-brightness and auto-exposure wreak havok on the threshold settings and the background subtraction so i bit the bullet and went with the not nearly as friendly sequence grabber stuff just so i could turn off the auto-stuff on the camera. 

If i am wrong and the qtKit allows access to the raw camera settings then i am all over it.  it is definitely so much nicer to use than the SG stuff. 

I did, however design the code in such a way so that converting it over to grabbing images from a qtkit delegate would be really easy (because I suspect that soon the qtkit will allow for more granular control of the hardware).

cheers!
-panda

ps: i mentioned on my blog, but i will mention it here as well.  after posting the code up there i sat down to see if i could make it run faster, and after a few hours managed to more than triple the performance (went from average 15ms to blob detect a single frame to around 4.5ms.  and another doubling of performance when you do a release build, so my release builds (ie compiler optimised) now will blob detect a single frame in about 2.5ms on my macbook pro. (havent tried the g4 12” yet) in any case, once i strip out the debugging code and add some comments i will post it up on my blog.

OK I see. Well from what I know it “auto adjust to capture the best settings for your camera.” I knew that was too good to be true.  So the unibrain has funky setting that need to be manually adjusted? I am thinking of getting this camera myself for the same purpose. I wonder if you can bypass all that “auto” stuff with qtkit? Did you get the unibrain API.  I read somewhere on the site about it but I could not tell if it was still available.  Anyway I have never liked the way you have to code for the SG but I guess that might be the best way to go, for now.

ding

Yeah, almost all the firewire cameras do the auto-exposure thing.  I highly recommend the fire-i, it is a great little cam.

Basically what happens is that when the cam is looking at just the screen with no blobs on it, it all has a certain average luminosity, and the auto exposure and auto brightness adjust for that (this is your background image) now, as soon as you put some blobs in the image, the average luminosity of the entire image goes up (because they are brighter than everything else) of course, if the luminisoty went up by the same amount every time, this wouldnt be an issue, but the more blobs, the more the auto-exposure change. 

Now, dont get me wrong, this isnt a big change, it is a subtle one. But it is enough to make the whole system just that much less sensitive. If your entire foreground image is changing in relation to the background image, then in order to successfully find the blobs, you will have to have a higher threshold, which means that your optics (ie how much IR is shining on your surface minus however much IR is coming in from the ambient light) will have to be higher. Of course, it is a bit of a spiral, the better your blobs stand out from the background, the more the camera will auto-adjsut the exposure.

now, as i mentioned it is still a subtle change.  Even with the auto-settings on, my blob detector still works fairly well, but there are more false-detections (ie detecting your palm as it hovers over the surface etc.. ) since I am doing a diffused illumination, this is kinda a big deal.  If you were doing FTIR, you might not even notice the auto-settings. 

The other thing to do, as you mentioned, is to use the firebrain API to adjust that stuff, thus enabling you to be able to use whichever API you wanted to capture.  I avoided this route simply to try and keep the code a bit more portable.  (and i was too lazy to try and find the fire-i api and get it working

All that said, i am very keen on the qtKit capture stuff.  and it is a fairly easy bit of code to replace the SC stuff in my code with the qtKit stuff. Once I get all the basic bits in place (ie a working configurator for all the little bits and an event generator) then I will start adding more robustness to the code. things like the ability to use CIImages, maybe convert the blob detector so it can act as a CIImage filter, the ability to use the qtCapture if you want, all that sort of stuff.

I should have also mentioned that Pawel and I ahd a great chat this morning (well, my morning his middle of the night) and he is going to help me get a code repository started for this stuff.  And I think it would be great if Ding could maybe work on a qtKit replacement for the current SG camera object.  That would be awesome.

cheers!
-ben