1 of 4
1
BBTouch, now with TUIO support
Posted: 22 July 2008 12:30 AM   [ Ignore ]
Sr. Member
Avatar
RankRankRank
Total Posts:  75
Joined  2008-03-23

Hey All,

I know a few people here were waiting for TUIO support for BBTouch, so I am posting here just to let you guys know that I have just committed all the new TUIO enabled code to the google repository.  (since you probably check out this website more often than my blog, ha!)

I had it running just fine with the Processing TUIODemo sketch from the rectivision website.  It should be compatible with any app that is able to receive TUIO commands (via OSC/UDP).. BBTouch should be all reacTIVision 1.4 compliant (presuming i read the spec right, and read the code right as well grin

note: BBTouch doesn’t do fiducials (yet) so all it generates are the 2DCursor profile events. 

as always the code is here:
http://code.google.com/p/bbosc/

and some notes (and some pics of BBTouch and Processing playing nice) on my blog here:

http://benbritten.com/blog/2008/07/22/tuio-support-is-here/

Let me know how you guys go (either here or on the blog) and if you have any trouble of find any problems.

Cheers!
-panda

 Signature 

http://benbritten.com/blog/category/multitouch/

Profile
 
 
Posted: 22 July 2008 01:53 AM   [ Ignore ]   [ # 1 ]
Elite
RankRankRankRank
Total Posts:  190
Joined  2008-02-01

Rock on.  I’m not at work today, I’m playing…

I tried to comment on your blog, but Wordpress doesn’t like my password which I know is correct and the reset link they send is always invalid!  I love computers. wink

I’m well up for using this, are you aware of any demo code which says ‘marker seen, run this function. etc?  One thing I love about TouchLib is it’s so easy to code in AS3, which until I properly learn another language is what I’m stuck with.

But yeah, thanks again.  I’m off to play.

 Signature 

fingerpuk.tumblr.com

Profile
 
 
Posted: 22 July 2008 02:25 AM   [ Ignore ]   [ # 2 ]
Elite
RankRankRankRank
Total Posts:  119
Joined  2008-06-04

A true Kung Fu Panda!

 Signature 

}---)))@

Profile
 
 
Posted: 22 July 2008 02:36 AM   [ Ignore ]   [ # 3 ]
Sr. Member
Avatar
RankRankRank
Total Posts:  75
Joined  2008-03-23

hey Larky,

I went to remove your user from my blog (so you could re-apply if you wanted) but I couldnt find any user: ‘larky’ or anything like that, so maybe just try registering again grin (i wish i didnt have to require registration for my lame-o-blog, but if i dont then the spam bots get it bad… they get it bad enough even as it is)

Also, If you are using AS3 and touchlib, i presume that you are using Flosc and your AS3 stuff is converting the TUIO events? if that is the case then BBTouch should work as a backend (ie event generator) just as well.. (well, it should work in any case grin I am not an AS3 guy, and I am not fully up to speed on the AS3/touchlib stuff. but if there is something i can add on my end that will help let me know and i will add it to the list grin

Anyhow, have fun with the code and let me know if you have questions!

Cheers!
-panda

 Signature 

http://benbritten.com/blog/category/multitouch/

Profile
 
 
Posted: 23 July 2008 01:48 AM   [ Ignore ]   [ # 4 ]
New Member
Rank
Total Posts:  1
Joined  2008-05-23

Awesome job, you effectively replaced all of the touchlib stuff I compiled with a nice, tiny, stable app with a GUI and all. It even uses less processing power. I was lazy and downloaded the binary rather than compiling it, seconds to set up rather than hours.

Profile
 
 
Posted: 23 July 2008 03:24 AM   [ Ignore ]   [ # 5 ]
Elite
RankRankRankRank
Total Posts:  190
Joined  2008-02-01

Just one question, which I already know the answer to but I can hope and pray! wink Will I have to re-code my AS3 stuff which is currently coded for TouchLib?  For example, instead of TouchEvent you’d use?…

 Signature 

fingerpuk.tumblr.com

Profile
 
 
Posted: 23 July 2008 03:50 AM   [ Ignore ]   [ # 6 ]
Sr. Member
Avatar
RankRankRank
Total Posts:  75
Joined  2008-03-23

Larky,

that is a good question.  I am not a flash guy, but BBTouch generates TUIO events just like touchlib (i think) so theoretically, if the AS3 stuff is looking for TUIO events then you shouldn’t have to recode anything.  i wonder if someone more knowledgeable about the whole TUIO -> Flash thing can chime in. 

I am pretty sure you will still need flosc and whatever it is that converts the OSC to flash XML. but i am fairly confident that it should work.  It worked fine with the Processing TUIO sketch, so i hope it works plays well with the other TUIO decoders out there.

Let me know what you find grin

-panda

 Signature 

http://benbritten.com/blog/category/multitouch/

Profile
 
 
Posted: 23 July 2008 10:47 AM   [ Ignore ]   [ # 7 ]
Elite
RankRankRankRank
Total Posts:  190
Joined  2008-02-01

Just to let you know, BBTouch works perfectly as a substitute for TouchLib if you want to run on OSX, no need to change any code at all.

So, after that very dry sentence…

DUDE I FRICKIN WANT YOUR BABIES.  This is the best thing to happen in the last 4 weeks, it’s so fast!  120 fps is not an issue, it’s so so so quick and easy and I am now drooling from my panties (I wear manly boxers of course, but you get the meaning).

I honestly am so happy.  I need a wee wee.

A.

:D

EDIT:  Two thing.  First it crashes if the threshold goes near or under 10 and would it be possible to have a 90 degree rotate, so I can mount my camera in odd places to get it to see all it needs to see! smile

First I dig up my own Potatoes, now BBTouch saves the day. smile

 Signature 

fingerpuk.tumblr.com

Profile
 
 
Posted: 24 July 2008 06:30 PM   [ Ignore ]   [ # 8 ]
Sr. Member
Avatar
RankRankRank
Total Posts:  75
Joined  2008-03-23

Hey Larky,

I actually wrote quite a long reply to this yesterday, but apparently i totally forgot to submit it.. oops… must’ve seen something shiny..

Anyhooo…

first off: 120FPS!! Sweet Zombie Jeebus! that is awesome.  What are you running it on i wonder? (I am generally presuming that is the output of your touch-enabled application, unless you have some super-firewire-cam that can output 120 fps?)

and second: yes it should be fairly straightforward to add the ability to rotate the coordinates arbitrarily.  (I actually have all the code to do it in my old prototype, but it hasnt made it into the ‘real’ version just yet) the current plan is to get CIImages working as a back for the data path (instead of NSIMage, which is what it is at the moment) and once that is all working, then we can use all the super-shmancy filters to do cool things like affine transform the input image into a nice rectangular output image. (like if you wanted to mount your camera at an odd angle, in the corner of your box, you could still transform the video into a ‘flat’ image for the blob detector)

anyhow, that is all in the works grin

Cheers!
-panda

 Signature 

http://benbritten.com/blog/category/multitouch/

Profile
 
 
Posted: 25 July 2008 11:11 AM   [ Ignore ]   [ # 9 ]
Sr. Member
RankRankRank
Total Posts:  59
Joined  2007-07-09

panda,

Dude you rock! I know I promised fiducial support for this but I have been crazy bussy to do any coding lately.  I will be going on a week break with my family next week but I’m going to get on that as soon as I get back.  I wrote a simple fiducial finder for openFrameworks not too long ago that can be found her:
http://code.google.com/p/ofxding/downloads/list
This is what I’m going to base it of off. The only problem with this is that you have to send it image data that is either 0 and 255 only and I haven’t got around to figure out how to implement this. In openFrameworks I just use the opencv image class for thresholding the image. If you have any ideas let me know.

One more thing of note.  From what I have been able to surmise from the time I have been trying to figure out blob tracking in os x the actual blob gathering/contour finding is always very speedy no mater what method you use. The part where the cpu take a very heavy hit is in the actual blob tracking itself.  Say I have blob:1, blob:2, blob:3 and blob:4. Say I remove blob 2 and 3. Now the software must figure out that blob 1 and 4 remain and label them as such. Usually what happens is the software just says “Oh! there is two blobs left so they must be blob 1 and 2.”
If I am being confusing here is a good explanation by erik2003:

implementation of the blob tracker

1- Find the blobs in the screen. This is done by counting the number of connected white pixels in the screen. Pixels that touch more than 3 (from top of my head) pixels around them that are also white are part of the same blob. Then it does some checks on the shape too, to see if it is roughly circular. From this part, the blobtracker knows coordinates of blobs on the screen.

2- Now the blobtracker needs to identify which finger corresponds to which blob. This is done by comparing the position of each of the blobs with the blobs in the previous frame:
If there is a blob close enough in the previous frame, the blob in the current frame gets the ID of the blob in the previous frame: it’s the same finger that moved.
If there is no blob close enough: it is a new finger, so make a new unused ID for it.
There are also blob IDs present in the previous frame that get no new blob assigned to them: these fingers are no longer touching the screen.

3- Now we know for each blob either if it is new, if it moved, and we also know the fingers that stopped touching the screen. For each of these options send an event for finger-down, finger-up or finger-moved.

4- OSC then listens for these events and bundles them into a frame and sends them out as TUIO messages to Flash or other applications.

Basically the number of fingers is so small (for the computer) that even for 100 fingers the naive algorithm of comparing each finger to all options from the previous frame is good enough. The native algorithm in touchlib uses a double-recursive function which is more complicated and also performance heavy.

The above naive approach does have the drawback that the data that is collected is a bit rough. For instance, when a finger ‘skips’ over the screen or the user moves really fast, the blob may sometimes get lost esp. with lower framerate cameras. If the blob is lost only for 1 camera frame the finger gets a new ID, and what appears as 1 movement to the user is several movements to the application.

I am referring to number 2 above.

Here is a simple explanation on how Touchlib does it from AlexP.

The way it finds the optimal blob match is by minimizing the sum of all the distances between the blobs of the current frame from the previous frame (global minima).
In order to do that we iterate through all of the permutations of the blobs, building the list and then in the next step find the item in this list with the minimum sum of distances. You can imediately see the problem with this approach when the blob count gets larger. Besides the fact that the algorithm is recursive in nature, it also has a problem in allocating a large amount of data to hold the blob permutations used in compare step. This data size is a factoriel of the blob count.
So, for example for 5 blobs, the algorithm needs to iterate through 120 permutations. For 10 blobs, the algorithm needs to iterate through 3628800 permutations. As you can see, this number grows very fast. Although they have placed some safeguards such as maximum blob distance (in order to cut some of the permutations), at some point this approach becomes not practical. This is even more true having the fact that we have to do all of this in real-time.

Anyways I havent checked out your code in a while but I am curious if you have implemented a blob tracker yet?

ding

Profile
 
 
Posted: 25 July 2008 01:24 PM   [ Ignore ]   [ # 10 ]
Administrator
Avatar
RankRankRankRankRank
Total Posts:  938
Joined  2007-04-08

Great job panda =)

Ding, please PM when you get back. I’m not sure you’re aware, but we’re making (almost finished) with a new cross-platform tracker using openframeworks (custom tracker).I want to add in fiducial support (I was going to use the ofxfiducialfinder), and I’m curious why you don’t want to use the reactivision one? If you think yours is better, let’s talk some about adding it.

 Signature 

My Multitouch Blog
My Multitouch Software Direct Downloads
My Youtube

Profile
 
 
Posted: 25 July 2008 04:09 PM   [ Ignore ]   [ # 11 ]
Sr. Member
RankRankRank
Total Posts:  59
Joined  2007-07-09

hey cerupcat,

Sorry to highjack just answering.  I was the one that wrote ofxfiducialfinder in the openFrameworks forums.  I basically looked at the reactivision code and used their libfidtrack and all.trees files to write ofxfiducial tracker so they are pretty much the same root.  I will PM you when I get back to find out more about the OF stuff.

[edit]I wrote it in like an hour or two so I still need to tweek it a bit.

ding

Profile
 
 
Posted: 26 July 2008 09:21 AM   [ Ignore ]   [ # 12 ]
Sr. Member
Avatar
RankRankRank
Total Posts:  75
Joined  2008-03-23

hey ding,

Yes, BBTouch has tracking.  And it works basically like the simpler version of what you described. (ie number 2) (ie simple distance check from the last frame) (it detects, tracks and generates TUIO style cursor events)

I had given it some thought (the idea that the tracking could be ‘better’ but to perfectly honest I havent had much of an issue with it yet.)

The way i had outlined (in my head) to attack the problem was more of an motion capture/iterative system kind of approach.  (my background is motion control of big physical systems, so i tend to approach everything from that direction grin

if we presume that the thing we are tracking has a certain top speed (which is fairly arbitrary, but really, at some point, no matter the interface, you can ‘overspeed’ it. try this with a ballpoint pen sometime, if you go too fast the pen will skip, or tear the page, or the little ball just wont spin and no ink will come out, the pen-paper interface simply is not designed to go that fast grin similarly, for multi-touch you can make some assumptions about user behavior, and make some decisions about realistic speeds that still provide the system with use-able information

Similarly for the acceleration, even tho you can move fairly quickly if you want, you can’t move your hand with infinite acceleration.

I dont know if i am explaining this very well, but i will persevere anyhow grin SO basically you come up with a reasonable set of parameters for ‘normal’ and even ‘fairly abnormal’ behavior and using those numbers you can interpolate your blob wherabouts much easier. 

But what we are really talking about here is the skipping problem.  (frame drop or finger-bobble type) naturally we want the user to be unaware of these kinds of things which is why we have complicated tracking grin

In my idea: to overcome the skipping probem; we use the above real world speeds and accelerations that we have figured out for the surface to guess where each blob will be on any give frame based on the previous movement, and then use that interpolated position to ID them

So instead of the blobs being a certain distance between the current point and their last known point, you instead interpolate a movement radius based on the prior speed and direction. this will give you basically a ‘cone of movement’ (in 3d space, in 2d it is more like a pie-shape) and if the detected blob falls into this space then you can ID it as a continuation of the previous blob. 

this approach is not a thorough as the touchlib approach (ie you can still confuse it if you try hard enough) but the tradeoff is that you only need to do a few small physics calculations for any moving blob instead of a double recursion.  (or it might not work at all, i haven’t implemented it yet grin

Anyhow, I am just thinking out loud here really.  BBTouch tracks, but doesn’t do anything like the above (not yet in any case, it has a top speed factor built in, and if the blob exceeds that then it ID’s it as a new blob, the cam I am using to test easily outputs 30fps so i havent ever had a real issue with skipping or going too fast (well, i can easily go too fast for it, but then I am off the surface, so it doesnt really matter anyhow grin

I hope that made some tiny amount of sense to someone grin Sometimes I am terrible at explaining things grin

BTW: cerupcat, saw your post on the front page re: cool new tracker, looks great!

Cheers!
-panda

 Signature 

http://benbritten.com/blog/category/multitouch/

Profile
 
 
Posted: 27 July 2008 04:21 PM   [ Ignore ]   [ # 13 ]
Elite
RankRankRankRank
Total Posts:  119
Joined  2008-06-04

Panda it makes perfect sense, why over program something when simplicity is the key!  Unless you want to do some high speed impressionistic brush strokes or really whack that pong paddle around!  Perhaps some averages need to be captured to get a feel for how fast most users want to operate?

 Signature 

}---)))@

Profile
 
 
Posted: 18 August 2008 10:15 AM   [ Ignore ]   [ # 14 ]
Sr. Member
RankRankRank
Total Posts:  59
Joined  2007-07-09

Panda I noticed your new blog post. I have a question though.  Is the mesh going to be the ROI.  From the pictures you have up it looks like if that is the case then the ROI is going to be centered and crooked but I am guessing that you have something up your sleeve.

Profile
 
 
Posted: 18 August 2008 04:45 PM   [ Ignore ]   [ # 15 ]
Sr. Member
Avatar
RankRankRank
Total Posts:  75
Joined  2008-03-23

Hey Ding,

currently the mesh defines the ROI, but the ROI can be larger than the mesh.  for instance in the wacky angle example i was testing that i put up on the blog, the ROI would be the smallest ‘normally’ oriented rectangle that contains all the vertices.  So the ROI would be a bit bigger than the mesh itself, and the detector will be scanning through some non-mesh pixels.  (so it is still culling much of the image, but it is not super-optimal)

that said, i do have something up my sleeve grin I am building a newer/better/more accurate blob detector that uses ‘hints’ to subtly alter the performance.  for instance, the roi will be mesh-dependant (instead of just a rect) and the detector will build an roi map so that it only scans through mesh pixels.  other ‘hints’ it will use are a variable threshold based on background luminance.  in other words, in darker spots the threshold would be lower than in brighter spots.  This is mostly aimed toward those of us with DI setups where it is really hard to get nice even lighting.  add to that a dynamic background image so that if you have variable lighting conditions (ie you are near a window or something and the incoming IR is changing slowly throughout the day, then you shouldnt need to keep manually updating the BG image. 

Of course the goal of the new hinted detector is to not lose any performance either, currently the blob detector can scan a full 640x480 image in about 1/500 of a second on my old macbookpro.  my goal is to keep it under 1/250 with the newer version (on my hardware) so that people like larky can still get 120fps (that still amazes me grin

of course, the new hinted detector is lower on the list than the calibration stuff, lots of implementation details to work out. so it wont be done for a bit yet grin

anyhow, i am breaking my own rule of not doing technical posts early in the morning, so hopefully this all makes sense grin

cheers!
-panda

 Signature 

http://benbritten.com/blog/category/multitouch/

Profile
 
 
   
1 of 4
1