1 of 6
1
TUIO multiplexer suite
Posted: 31 January 2010 05:09 PM   [ Ignore ]
Avatar
RankRank
Joined  2008-10-23
Total Posts:  148
Member

Hi everybody,

I’m happy to announce the release of some enhancement to the CCV Tracker. I call it ”TUIO multiplexer suite
It consists of a enhanced CCV Version (based on Version 1.2) and a TUIO multiplexer.
A short feature List:
CCV:
Fiducial Tracking
TUIO 1.1 message sending (to distinguish the messages from different trackers).
UDP-calibration data sending, to inform the TUIO multiplexer about a calibration operation.
UDP-Alive trigger function, to get an “alive” message from a CCV instance to monitor if it’s still running.
GUI Interface elements position are now set in the conf.xml file.
FFMV implementation modified (for PointGrey cameras) to connect multiple FireFly cameras to multiple instances of CCV on one computer.
You now set a “DEVICE_MODEL"- tag , so you could connect multiple cameras from different vendors at the same time.
Some minor modification for problems that arise with multi camera setups and tiny bugs from CCV.(no 0,0 position sending if the finger is outside the calibration rectangle, finger acceleration data is now not undefined)

To say it clear: This is not a multi camera enhancement for one CCV instance like stitching-solutions with the PS3 driver, but a solution to run multiple instances of CCV concurrently!

TUIO Multiplexer:
Multiplexing of the TUIO data from different TUIO sources who support the TUIO 1.1 specification (At this time only the 2DCur and 2Dobj )
Sending of TUIO 2Dcur and 2Dobj messages.
Visual representation of your multi-camera setup to make a calibration possible.
Memory consumption monitoring with automatic shutdown/restart (there is some ATI driver problem on some cards that causes a memory leak with the OpenGL glDrawPixels call ! Perhaps some OpenGL guru will fix it with better FBO-code wink )

First i wanna talk about the CCV upgrade:

Fiducial support with the ofxFiducial addon from Alain Ramos based on the fidTrack.lib from Ross Bencina
There where some consideration i had to made as i realized in my first naive implementation that a concurrent Blob and Fiducial tracking does not work in one image.
I decided to create a second filter-chain, so there are two independent images for the image processing.
This is far better then a single Image solution but it’s not the “Silver Bullet” solution. You still have to life with some drawbacks.
The fiducial tracking works like a charme, as fingers are not recognized as fiducials. On the other site a fiducial could be recognized as a finger, so it depend on your filter-settings to get a satisfied solution.
Well if someone gets wet eyes on this he has something to consider about the fiducial-markers from reacTIVision . Martin Kaltenbrunner has mentioned there is some licence restriction on the original fiducial-set from reacTIVision . So your are not allowed to use them outside of the Reactivision tracker in commercial projects, without inquire for a licence.
There is another way to circumvent this with the FidGen-tool from Toxi, so i did not include a tree-file for a marker-set.
You have to create your own or get anyone elsewhere. It should placed in the “data” folder and you could set the right name in the config.xml.
If you wanna run CCV without Fiducial tracking there is a flag in the Config.xml to start in “normal” mode for better performance.

So for a normal CCV user, all you’ve got more from this CCV-version is fiducial support, but the real power of the system comes in conjunction with the TUIO-multiplexer.
With this combination your are able to make a scalable multi camera/screen setup on basis of CCV. Theoretically it should scale infinitely, as you could daisy-chain multiple multiplexers.
The key for this scalability is my different approach for multi camera support, then the solutions who trying to bring more cameras into one CCV instance, so at one point your CPU will have not enough power. I know for the most of the setups the two camera solution with the PS3 driver will work, but i had to face with a project where i had to track a round surface with a diameter of 2,4m (~ 100 Inch) with 8 Cameras. And as the PS3 driver is constrained to two Cameras, i conceived this solution. So i have to say this is not a cheap solution based on the PS3 camera, as I’m aware the PS3 driver did not give you a different DeviceID if you trying to run multiple PS3 cameras on one Computer.
If you look for large multi camera solution you will stumble across the incredible capabilities of the Firefly MV camera from Pointgrey. They have a long experience with large setups and you get a incredible driver for their products. CCV has native support for the Firefly but the current implementation prevent you to start muliple cameras at the same time on one computer without crashing.
Well it was a quick fix, so now CCV doesn’t scan all plugged cameras but only the one you specify with the deviceID in the config-xml.
For the best performance and data throughput i use the Firewire version. I connected 4 cameras(640x480) at two Firewire cards on a QuadCore machine without any problems.Two machines with this setup where multiplexed on a third machine.

TUIO Multiplexer:

So the next big question is, how do i could calibrate such an setup? CCV has it’s own calibration routine wich works excellent and why reinvent the wheel?
For this problem there is a connection between CCV and the TUIO multiplexer, so you define your whole camera setup in the conf.xml file from the multiplexer with all the translations and rotations of the different cameras
and get a visual representation from that.
You also get feedback where your fingers are mapped with green dots for fingers and a tiny quadrat with the ID for fiducials.
This is your visual reference of the setup and now you could align the cameras to observe the right area.
If you step into the calibration mode in CCV the multiplexer get notified and get the amount of x/y calibration points you set in CCV.( Your BoundingBox settings in the calibration-file from CCV must have 0,0,1,1!)
Now you get a visual representation of the calibration points and the current selected point is highlighted and you see where you have to put your finger down. CCV Will see it and do the calibration for you. After calibration the green points position should match your finger position for that screen. To handle this you should have at least two monitors if you work on one machine, as CCV goes in full-screen mode and you aren’t able to see the multiplexer screen.
The whole system could be distributed on different computers so your sourcename@IPAddress combination should match between the values in CCV and the multiplexer. Look at the conf.xml files and everything should be self explanatory.
(Is anybody following?)
Every instance of CCV who wants to communicate with the multiplexer is represented as a “source"- tag in the conf-file. If a source@address combination from a CCV instance doesn’t match with the values in the source-tag, your data will be dropped by the Multiplexer.
The multiplexer has two modes: “circle” and “square”. Play around and you will see the difference for your screen-layout. For a better usability you could move the window with the arrow-keys.(+shift => stepsize 10) ,"h" and you hide the window.
Internally the multipexer receives all the TUIO data (it’s a normal TUIO-client). All fingers who are matching a source@ipaddress entry in the config-file where taken and from their position data i draw some circles into a Texture. Then i apply the same blob tracking from CCV. The fiducial administration is done by the TUIO-client code from Martin.

So i don’t know if anybody could use this tool or my approach missed the point and there are better solutions I’m not aware. I think this solution makes only real sense on very large setups, or if you using other cameras then the PS3.
Feel free to drop me a line.

At this time i will only give a binary version for testing. So give me feedback if you find some bugs or have ideas for improvements.
When i’m finished with some code clean up and some tiny fixes i will post the sources.

A big gratitude(beside the NUI Group) goes to Martin Kaltenbrunner for the incredible work on the TUIO platform and his reference implementations!
Here a video to get a picture of the capabilities:

http://www.youtube.com/watch?v=11G90pG5wbU

(My responsibility was just the development of the multi touch middleware and camera setup, so I’m not quite happy with the lack of good interaction design . rolleyes )

cheers

File Attachments
CCV_Fiducial.zip  (File Size: 900KB - Downloads: 1003)
TUIO Multiplexer.zip  (File Size: 1628KB - Downloads: 1528)
Profile
 
 
Posted: 31 January 2010 06:11 PM   [ Ignore ]   [ # 1 ]
Avatar
RankRankRankRank
Joined  2008-04-30
Total Posts:  612
Elite

Great work sloopidoopi!
I know many people have been waiting for CCV fiducial support. All the other enhancements seem useful, too. Maybe some of this can be merged with the CCV codebase. Way to go.

 Signature 

MT4j - Multi-Touch for Java!

Profile
 
 
Posted: 31 January 2010 06:14 PM   [ Ignore ]   [ # 2 ]
RankRankRankRank
Joined  2009-08-21
Total Posts:  555
Elite

Ahh man I saw the video . What an awesome implementation. Its worth the 4 months. I had done nothing compared to it. I am sure it will be used by people. I hope you release the source code soon so that we can also go in and see what happens in there !!! grin . Good work mate.

 Signature 

~~Amit~~

Blog - Twitter - Facebook

Profile
 
 
Posted: 31 January 2010 10:14 PM   [ Ignore ]   [ # 3 ]
Rank
Joined  2009-09-28
Total Posts:  94
New Member

While I am excited about the fiducial support, the multiplexing will allow us non-commercial builders to explore larger and larger installations. Very very exciting!  Thank you for all of your hard work.

Can someone post links to the fiducial treefiles that Sloopi references above?

 Signature 

The Dynamic Media Institute at Massachusetts College of Art and Design

Profile
 
 
Posted: 01 February 2010 02:33 AM   [ Ignore ]   [ # 4 ]
Avatar
RankRankRankRankRankRank
Joined  2007-04-08
Total Posts:  2539
Dedicated

From first glimpse, you’ve done a fantastic job! :D

Hopefully we can start to get a real ‘team’ together to make consistent development like this.

I’ll let you know when I get a chance to test! Keep up the great work :D Amit, I know you’ve been working hard on fiducial support - now you don’t have to work quite as hard smile

 Signature 

MTmini, MTbiggie, & Audiotouch creator & Community Core Vision Co-founder

Follow on:
My Blog | Facebook | Twitter | Youtube

Profile
 
 
Posted: 01 February 2010 02:40 AM   [ Ignore ]   [ # 5 ]
Avatar
RankRank
Joined  2009-11-05
Total Posts:  173
Member

I really like the idea of running multiple instances of CCV for whatever you want! Simple and clever! Thanks for sharing all this stuff with us sloopi smile
You got three-points shot out of 7.25m!

 Signature 

coming soon - My MT proposal

First taste of my mt surface table: http://www.youtube.com/watch?v=lZaFvm1nbps and http://www.youtube.com/watch?v=jy3fPCdqX6g (one video split to two)

Profile
 
 
Posted: 01 February 2010 07:37 AM   [ Ignore ]   [ # 6 ]
Avatar
RankRankRankRank
Joined  2007-09-18
Total Posts:  881
Moderator

WOWowow , that’s wild, SloopiDoopi !!
Ground breaking implementations !
What a hard work, Many Thanks for sharing it with us !
Your approach is def. a great support

This video is also very impressive, this round tabletop is huge ! Can you us a bit more info about it ? It looks like DSI setup, right ?
It seems that it suffers some latencies issues ...is it due to the computer you use or the parsing latency between computers ?

A few questions :

FFMV implementation modified (for PointGrey cameras) to connect multiple FireFly cameras to multiple instances of CCV on one computer.

Does it allow now to use FFMV with resolution>640*480 ?

licence restriction on the original fiducial-set from reacTIVision . So your are not allowed to use them outside of the Reactivision tracker in commercial projects, without inquire for a licence.

Does it means that the restriction is only for the fiducial set & not for the Reactivision tracker ?

I decided to create a second filter-chain, so there are two independent images for the image processing.This is far better then a single Image solution but it’s not the “Silver Bullet” solution.

In order to get less latency, is it possible (or relevant) with your TUIO multiplexer, to track finger with one CCV on a machine & track Fiducial with another CCV on another machine & merge both results ? 

I was expecting such features for quite a long time, so i ‘ll test that as soon as i get enough hardware items…
Thxs again for shaking the house!!

 Signature 

How many touches can you simultaneously perform ? 
Coming soon : EveryWall MT / Multi LaserPointers / MT SMS Wall
le WIKI en Français

Profile
 
 
Posted: 01 February 2010 07:44 AM   [ Ignore ]   [ # 7 ]
Avatar
RankRank
Joined  2009-03-10
Total Posts:  178
Member

Wow thanks for sharing this SloopiDoopi!

 Signature 

Multitouch Music

Profile
 
 
Posted: 01 February 2010 09:03 AM   [ Ignore ]   [ # 8 ]
Avatar
RankRankRank
Joined  2009-06-09
Total Posts:  268
Sr. Member

Nice.

Actually I also had the idea of using a second camera with its own image processing pipeline just for fiducial tracking. I might add that, on OSX/Linux, there does not seem to be a restriction as to the amount of PS3Eyes you can use.
I would like to test your changes on OSX, so it would be great if you could make the source code available.

IMHO
If your changes work well, I think they should be merged back into CCV. Perhaps starting with a new repository or something. I don’t quite see activity in the future of the windows-only PS3Eye-centric CCV 1.3 stuff and I don’t think it’s in anyone’s interest that people work behind closed doors for months before publishing a patch that then gets lost. There should be a more open, cross-platform collaboration model.
/IMHO

 Signature 

My Blog
PyMT—Create Multitouch Software easily!
Movid—New cross-platform tracker, work in progress!

Profile
 
 
Posted: 01 February 2010 09:10 AM   [ Ignore ]   [ # 9 ]
Avatar
RankRank
Joined  2008-10-23
Total Posts:  148
Member
jimihertz - 01 February 2010 07:37 AM

It looks like DSI setup, right ?

The bar is DSI with endlighten.
The round table is DI from the innerside.

This video is also very impressive, this round tabletop is huge ! Can you us a bit more info about it ?

The Problem with the table was, that the hardware already exists, and we had to deal with a lot of things we could not change. The main problem wasn’t the IR-light or the camera setup. I found a smart solution for the cameras, but the main problem was (and still is) the projection with four beamer. But this was luckily not my business.
The Hardware Setup was:
2 camera Computers (QuadCore) each with four cameras.
1 computer with the multiplexer and the globe frontend(this is a 8 core beast).
2 Matrox dualHead boxes to connect four acer beamer to a ATI HD5850 Card.

I will post some fotos…

It seems that it suffers some latencies issues ...is it due to the computer you use or the parsing latency between computers ?

I think that the latency issue isn’t a problem with the distribution on several machines. I suppose the main bottleneck are the beamers. But it’s up to the community to test,test,test and give response with different setups!
( If you mean latency in the street-map app on the bar, this is not a latency issue on the tracker side. The frontend software needs time to calculate a new route on the map!)

Does it allow now to use FFMV with resolution>640*480 ?

Well you have to test it by yourself, as i’m not aware of it and at this time i don’t in possession of some FireFly. They all gone away from me.  downer 
But there is no difference to the old FFMV implementation.

Does it means that the restriction is only for the fiducial set & not for the Reactivision tracker ?

I’m not quite shure about the licence model from reacTIvision. So you better look on Martins site. He is the master of good documentation.

In order to get less latency, is it possible (or relevant) with your TUIO multiplexer, to track finger with one CCV on a machine & track Fiducial with another CCV on another machine & merge both results ? 

Thats one possible solution to test . (I didn’t consider if there is a difference).

So you now have a toy to play with endless possible combinations. It’s only a question of your Budget for the Hardware.

I would be happy to get response from others if they get cheaper setups that do the work.

Profile
 
 
Posted: 01 February 2010 09:37 AM   [ Ignore ]   [ # 10 ]
Avatar
RankRank
Joined  2008-10-23
Total Posts:  148
Member
dennda - 01 February 2010 09:03 AM

Nice.

Actually I also had the idea of using a second camera with its own image processing pipeline just for fiducial tracking. I might add that, on OSX/Linux, there does not seem to be a restriction as to the amount of PS3Eyes you can use.
I would like to test your changes on OSX, so it would be great if you could make the source code available.

IMHO
If your changes work well, I think they should be merged back into CCV. Perhaps starting with a new repository or something. I don’t quite see activity in the future of the windows-only PS3Eye-centric CCV 1.3 stuff and I don’t think it’s in anyone’s interest that people work behind closed doors for months before publishing a patch that then gets lost. There should be a more open, cross-platform collaboration model.
/IMHO

Hi dennda,

it would be great if somebody will do the crossplatform development.
As i’m a little tired with this project i’m not in the mood to test it on other platforms and do the SVN integration stuff. So the community could do some work.

I know it’s not the best way to develop behind closed doors, but i don’t wanted to live up to others expectations, how and when this or that feature will be ready. I had no plan at the start, how to realize the problems i had to deal with, so it’s more like an organic development with some trial and errors. At some point i realized that my enhancements are much bigger, as i thought they would be and more like an platform for CCV and other TUIO trackers.

So things often come different as you intended.

Profile
 
 
Posted: 01 February 2010 09:44 AM   [ Ignore ]   [ # 11 ]
Avatar
RankRankRank
Joined  2009-06-09
Total Posts:  268
Sr. Member

I think you got me wrong. I wasn’t criticizing you. Lately I’ve just seen several independent patches come out for the same problems in CCV and I think the problem is that the current development model may not be open enough. That should be fixed so as not to duplicate people’s effort.

 Signature 

My Blog
PyMT—Create Multitouch Software easily!
Movid—New cross-platform tracker, work in progress!

Profile
 
 
Posted: 01 February 2010 12:36 PM   [ Ignore ]   [ # 12 ]
Avatar
RankRankRankRank
Joined  2007-09-18
Total Posts:  881
Moderator

Hi Sloopi Doopi,

i got your CCV fid. running , i copied an amoeba “default” tree & renamed it to “ fiducials.tree” ,& put it in CV data folder . BUt i have no tag detection ...
Is a previous calibration required ?
Did i miss smtg ?

CHeers,

By the way, what projection surface did you use on the DSi setup ?

Image Attachments
test_0.jpg
 Signature 

How many touches can you simultaneously perform ? 
Coming soon : EveryWall MT / Multi LaserPointers / MT SMS Wall
le WIKI en Français

Profile
 
 
Posted: 01 February 2010 01:28 PM   [ Ignore ]   [ # 13 ]
Avatar
RankRank
Joined  2008-10-23
Total Posts:  148
Member
jimihertz - 01 February 2010 12:36 PM

Hi Sloopi Doopi,

i got your CCV fid. running , i copied an amoeba “default” tree & renamed it to “ fiducials.tree” ,& put it in CV data folder . BUt i have no tag detection ...
Is a previous calibration required ?
Did i miss smtg ?

CHeers,

By the way, what projection surface did you use on the DSi setup ?

As i see in my conf.xml that i posted, your file should be named fiducials.trees not .tree. does this fix it? have you looked at

<TREE_FILE>data/fiducials.trees</TREE_FILE>

And the camera resolution should be 640x480.
And you have to calibrate your system!!! I changed the way,CCV tracks objects outside the calibration area. Sorry.

The projektion surface was the rearprojektion Plexiglas 99561 RP 3mm.
I’v made tests with the 5mm 7D006RP and fiducial tracking,but it was to blurry as the designer wanted a tiny marker . so we made a tradeoff . (The quality of the projection from one site was better, but too blury, when i flipped the sides the projection quality was equal to the 99561 , so it’s cheaper with almost the same results. Fiducial tracking makes the whole choice of the right projection surface a little bit complicated. It depends of the size of your markers, or wich camera objective you choose....

Profile
 
 
Posted: 01 February 2010 02:32 PM   [ Ignore ]   [ # 14 ]
Avatar
RankRank
Joined  2009-03-10
Total Posts:  178
Member

Hey just tested ccv with fiducials,
I noticed that if I recapture the background subtraction image it only uses this image for blob tracking,
the background subtraction image used for fiducial tracking remains the one captured when the application is launched.
Is there a way around this?

 Signature 

Multitouch Music

Profile
 
 
Posted: 01 February 2010 02:45 PM   [ Ignore ]   [ # 15 ]
Avatar
RankRank
Joined  2008-10-23
Total Posts:  148
Member
grmcom - 01 February 2010 02:32 PM

Hey just tested ccv with fiducials,
I noticed that if I recapture the background subtraction image it only uses this image for blob tracking,
the background subtraction image used for fiducial tracking remains the one captured when the application is launched.
Is there a way around this?

Thanks for finding the bug! I’ll fix it! (if you hit ‘b’ it should work right now)

Profile
 
 
   
1 of 6
1