1 of 5
1
present and future of TUIO
Posted: 26 January 2009 07:45 AM   [ Ignore ]
Avatar
RankRankRank
Joined  2008-06-16
Total Posts:  330
Sr. Member

Hello,
there was a discussion in the reacTIVision forum about the present and future of the TUIO protocol,
so I thought I should share my thoughts with you on that topic in a dedicated thread here as well.

I am aware that there are some shortcomings of the current protocol specification, which I am planning to address with a TUIO2 protocol.
I am planning to publish the draft soon, so we all can discuss and shape the new specification to fit our needs and ideas.

Here I give you some examples to make a bit clearer of what I am thinking about:

* make session IDs a 64bit value (although the current 32bit integer is not a real problem)
* adding cursor IDs and pressure (and eventually even a rotation value) to the cursor profile
* adding a string descriptor to the object profile in order to a allow encoding of semacode or barcode contents
* making the protocol namespace more OSC style by including the “commands” to a new namespace
e.g. /tuio2/2Dobj/set session_id ... ... ...
* adding timing information to the fseq message
* a more detailed specification of the bundle format/contents

I have seen that some TUIO implementations added extra values such as width and height to the end of the messages.
For the description of the shape such as the width and height of the blobs I want to go even much further and introduce a new “shape” profile, that should encode an optimized description of the blob with descriptors such as: bounding box, oriented bounding box and even a reduced contour description.

So basically I think, since there are so many radical new changes in this approach it justifies the introduction of a TUIO2 protocol. 
But maybe we should also discuss a less radical and background compatible TUIO 1.1 protocol, but in this case the background compatibility should be taken seriously in order not to break existing implementations.

best,
Martin.K

 Signature 

TUIO community site: http://www.tuio.org
reacTIVision framework: http://reactivision.sf.net
TUIO & reacTIVision CVS: http://sf.net/projects/reactivision
reacTIVision forum: http://sf.net/apps/phpbb/reactivision

Profile
 
 
Posted: 26 January 2009 09:32 AM   [ Ignore ]   [ # 1 ]
Avatar
RankRankRank
Joined  2008-06-16
Total Posts:  330
Sr. Member

Hello again,
I’d like to add as well some observations and explanations to the current TUIO protocol specification and some implementations.

* the session ID should be an incremental unique number only used once for any appearing object or cursor.
ideally the session ID should be unique for all profiles. I have seem some implementations are repeating the cursor session ID as if it would be a cursor or finger ID, this is not correct. At the moment my client implementations are deriving a cursor ID though, and as I said in my last post I am eventually planning to add a second cursor ID to a future version of the protocol, which then could be used to identify fingers, hands or even users. A diamond touch would allow user identification for example.

* many users point out the redundancy of the included speed and acceleration values. I am aware that most people event don’t want to use these values, but they are there for a good reason: If individual frames are lost, the correct speed and acceleration can’t be calculated on the client side, even if timing information is included (which isn’t at present). Again, a future version should probably also include timing information, but I rather would not drop the speed and acceleration values for the set message. The only values that I could imagine to be trimmed would be the separate speed values for each axis.

* it is not a good idea to expand or even change the protocol within the existing namespace, which as we can see with some implementations will break the clients and sacrifice the compatibility and interoperability of the TUIO protocol, which I think should be maintained by any means.

* if some of you need additional information about cursors or objects (such as width and height), then a custom profile can be defined. My current TUIO client examples are not ready to deal with custom profiles yet though. But I would suggest that it would be probably better to team up to complete the existing TUIO client codebase, rather than creating a variety of incompatible forks. Adding a proper custom profile decoder should not be a big deal.

* shape description could be defined as a separate standard profile as well, where using the same session ID one could encode some more information about the actual blob.
In my last post I outlined this idea for a new TUIO2 protocol, but this could be even included in the current TUIO specification without breaking backward compatibility.

best,
Martin.K

 Signature 

TUIO community site: http://www.tuio.org
reacTIVision framework: http://reactivision.sf.net
TUIO & reacTIVision CVS: http://sf.net/projects/reactivision
reacTIVision forum: http://sf.net/apps/phpbb/reactivision

Profile
 
 
Posted: 26 January 2009 11:26 AM   [ Ignore ]   [ # 2 ]
RankRankRankRank
Joined  2007-07-14
Total Posts:  819
Elite

Could you please explain why OSC? smile

 Signature 

HID Multitouch driver for Windows 7 http://multitouchvista.codeplex.com/

Profile
 
 
Posted: 26 January 2009 11:40 AM   [ Ignore ]   [ # 3 ]
Avatar
RankRankRank
Joined  2008-06-16
Total Posts:  330
Sr. Member
Daniel D - 26 January 2009 11:26 AM

Could you please explain why OSC? smile

Of course I can,
we developed TUIO for the reacTable, which is as you might know a musical instrument.
Therefore we chose OSC in order to support this emerging standard for musical instrument control.
Another advantage is that TUIO can be decoded from within any platform that supports OSC.

The only disadvantage related to this choice I see at the moment is the way the Flash clients have to deal with the protocol,
but this would have been the case for a “generic” implementation as well, since the first choice would have been UDP packets anyway,
simply because of speed reasons.

BTW, OSC itself does not define the UDP transport at all, it can also be implemented via a TCP, serial, etc. connection ...

best,
Martin.K

 Signature 

TUIO community site: http://www.tuio.org
reacTIVision framework: http://reactivision.sf.net
TUIO & reacTIVision CVS: http://sf.net/projects/reactivision
reacTIVision forum: http://sf.net/apps/phpbb/reactivision

Profile
 
 
Posted: 26 January 2009 02:32 PM   [ Ignore ]   [ # 4 ]
Avatar
RankRankRankRankRankRank
Joined  2007-04-08
Total Posts:  2539
Dedicated

Thanks for writing back and starting this discussion here Martin. I’m the one who wrote on the reactivation forum.

Many of the things you wish to implement are things we’d also like to add and have ready in some of our trackers already. We’re just waiting for the proper way to send them (angle, blob contour - vector list?, etc).

I think a custom profile implementation is definitely what we need so various tracking software don’t have to match each other. There will inevitably be things in reactivation that we don’t want to send and things not in reactivation that we do want to send. Having a standard way of parsing these custom messages are really ideal. Until then though, we’ve had to customize and hard code the TUIO Client to work with trackers people use on NUI Group. As you saw from my post on your forum, I just finished recompiling the Max, QC, and PD clients and another poster in that same thread recompiled the java one (so we’re not alone) to work with height/width. While I understand deviating and changing the standard set isn’t ideal, until we can get a good custom profile parsing, I’m not sure what else to do.

-As a side note, for flash users, some of our trackers can send TCP directly and avoid any issues with sending TUIO to flash through the FLOSC gateway (no FLOSC needed).

 Signature 

MTmini, MTbiggie, & Audiotouch creator & Community Core Vision Co-founder

Follow on:
My Blog | Facebook | Twitter | Youtube

Profile
 
 
Posted: 26 January 2009 04:33 PM   [ Ignore ]   [ # 5 ]
Avatar
RankRankRank
Joined  2008-06-16
Total Posts:  330
Sr. Member

Hello Seth,
yes I thought it might be better to bring the discussion over here as well.
I’d like to invite all the other developers of TUIO sources and clients to join in, if they are around. I am currently writing up my dissertation which is widely based on the TUIO interaction framework and would be therefore very interested in the opinions, experiences and motivations of developers who chose the TUIO protocol and TUIO client API for their projects.

As I mentioned in my previous post, over the time we also observed some shortcomings of the protocol, and although we tried to foresee some alternative uses, it was obviously impossible to meet the needs of all possible future scenarios. Since with the work on my dissertation I am currently reflecting on the current and future design and since I can see a certain need and movement in the community, I think we should see how we can solve the obvious need for an update and extension of the protocol together.

But I do not think that the current process that I am observing of simply adding parameters to a well defined protocol, and therefore breaking many existing applications is the correct way to go, neither the approach of just forking the full codebase of the TUIO framework. I understand that adding a few numbers to a parameters list seems easier for the moment, but this really isn’t good style.
The best intermediate alternative to solve your needs is the proper and also quite simple definition of a custom profile in TUIO. Rather than forking the whole TUIO client codebase, it would be therefore more productive to team up and implement the correct custom profile parser.

Regarding the future development of the protocol, I did not yet see or do not have access to changes in the trackers you mention, but it would be interesting to have a look at it or hear more about your ideas or needs you have encountered with the development of your multi-touch trackers. But what you mention is rather close to the shape descriptors we are planning: bounding boxes, contour points, skeleton etc.
In any case I will post the draft soon, and I am looking forward to the discussion ...

best,
Martin.K

 Signature 

TUIO community site: http://www.tuio.org
reacTIVision framework: http://reactivision.sf.net
TUIO & reacTIVision CVS: http://sf.net/projects/reactivision
reacTIVision forum: http://sf.net/apps/phpbb/reactivision

Profile
 
 
Posted: 26 January 2009 05:33 PM   [ Ignore ]   [ # 6 ]
RankRankRankRank
Joined  2007-07-14
Total Posts:  819
Elite

My opinion, is that base protocol should be something that developers already know and can use tool they have. Any programming language can process xml. I’v never heard of OSC. And no language can process it.

My features requests: raw images, minor axis, major axis, pixel area, bounding box, orientation (rotation angle).

 Signature 

HID Multitouch driver for Windows 7 http://multitouchvista.codeplex.com/

Profile
 
 
Posted: 26 January 2009 05:47 PM   [ Ignore ]   [ # 7 ]
Avatar
RankRankRankRankRankRank
Joined  2007-04-08
Total Posts:  2539
Dedicated

The overhead of xml I think is more than using UDP (OSC). Would have to benchmark some more to get a better feel. As Martin said, TUIO does not necessarily have to send with OSC, it’s just a standard protocol for formatting the messages (is that correct Martin?). So in essence we’re really after the same thing Daniel.

I’m on my way out so I can write more in depth a little later, but I just wanted to say Martin, we in no way want to tear apart the integrity or framework of TUIO. Since we currently have 50,000+ downloads of touchlib and tbeta we needed a quick ‘temporary’ fix even though it’s not ideal; I’m sure you would appreciate us not bugging you every time we update our parameters and we can’t expect you to update something you don’t use yourself and this is why we made this temporary quick fix.

I think a collaboration would be great. If you’re able to give an example of how to implement a custom profile for parsing, that would be a nice start. I heard about TUIO 2.0 about 5 months back, but thought it was a rumor due to not hearing anymore on the subject. I think a whole new approach as you mention may be in order.

Hope to continue this conversation and also get more people involved that are willing to help in a more direct approach (coding); I don’t consider myself the best person to be coding for this. I’ll also PM the developers of all the trackers to see if they want to chime in on this thread.

 Signature 

MTmini, MTbiggie, & Audiotouch creator & Community Core Vision Co-founder

Follow on:
My Blog | Facebook | Twitter | Youtube

Profile
 
 
Posted: 26 January 2009 08:32 PM   [ Ignore ]   [ # 8 ]
Avatar
Rank
Joined  2009-01-12
Total Posts:  16
New Member

About maintaining backward compatibibly? Has Multi-touch applications or software developed to the stage or in enough quantity that we need worry about backward compatibility. In no time at all those older basic applications would be replaced and the newer more refined ones on TUIO2 created.

I dont know alot about TUIO beyond its use in multitouch so I dont know what else its used for, but if it were to brake alot of MT apps atm, I dont recon it’d be such a bad thing as those programmers would probibly update their apps to take advantage of TUIO2 extra’s almost immediatly. You said yourself TUIO was created for another purpose so really everything around now would be grafted unto its origional design and made to work with it, instead of it working for the them. But TUIO2 would be designed with its other uses in mind and more ammendable to future improvements.

The time it would take to make a 1.1 version to maintain compatability with older software could probibly be better spent developing TUIO2 perhaps? This OSC, before NUI forums I’d never heard of it either. XML would be handy though.

Profile
 
 
Posted: 28 January 2009 05:12 AM   [ Ignore ]   [ # 9 ]
Avatar
RankRankRank
Joined  2008-06-16
Total Posts:  330
Sr. Member
Seth (cerupcat) -

The overhead of xml I think is more than using UDP (OSC). Would have to benchmark some more to get a better feel. As Martin said, TUIO does not necessarily have to send with OSC, it’s just a standard protocol for formatting the messages.

Yes, TUIO is actually very much bound to OSC, but what I said is that OSC is not necessarily bound to UDP.

Seth (cerupcat) -

I just wanted to say Martin, we in no way want to tear apart the integrity or framework of TUIO. Since we currently have 50,000+ downloads of touchlib and tbeta we needed a quick ‘temporary’ fix even though it’s not ideal; I’m sure you would appreciate us not bugging you every time we update our parameters and we can’t expect you to update something you won’t use yourself and this is why we made this temporary quick fix.

Well, what seems to be an unimportant “temporary fix”, actually is an incompatible fork and we really should avoid that considering the large user base the tangible or multi-touch community already has. As you mention touchlib and tbeta have 50.000 downloads and I just had to look it up, reacTIVision has reached around 60.000 as well, and the TUIO client base reached around 80.000 downloads to complete the statistics. Looking at these numbers, which probably mean that we have a shared user base of a few thousand people it is more important than ever to consider and maintain interoperability.
If you are missing a feature in HTML you also just won’t go and fork Mozilla to support that feature ...

Seth (cerupcat) -

I think a collaboration would be great. If you’re able to give an example of how to implement a custom profile for parsing, that would be a nice start. I heard about TUIO 2.0 about 5 months back, but thought it was a rumor due to not hearing anymore on the subject. I think a whole new approach as you mention may be in order.

Well, the TUIO update was on my todo list for quite a while as so many things, but since I was travelling a lot with the reacTable lately I hardly found any time for coding or other conceptual work. Now as I mentioned I took some time off in order to finish my dissertation, so this is the perfect time for me to touch these issues. Hopefully I will be able to publish the draft by the end of next week. I already outlined the mayor topics here in the first posts, so maybe you could add your comments here already.

In the meanwhile you can also have a look at the custom profile definition in TUIO, you will be able to find a more detailed description in the actual protocol specification (I placed the link in the message footer). Basically you just need to create a new custom profile, where the name actually encodes it’s contents. I personally never needed to use it myself, so I am not 100% sure if that is really the universal concept as we initially designed it, but basically it allows you to create a mixed profile of known and unknown TUIO parameters. So in your case you could create a profile similar to the 2Dcur, leaving out the speed and acceleration and adding two floats for width and height.
Looking at it now, I think one design flaw is that it only allows a common float parameter, but in your case this should be sufficient.

/tuio/_[formatString]

The letters of the format string are defined in the protocol specification as well.
s = session ID (int)
x = x coordinate (float 0 .. 1)
y = y coordinate (float 0 .. 1)
P = free float parameter

in your case for example:
/tuio/_sxyPP set s x y w h

w & h should probable also be in the range of 0 .. 1

That way, any TUIO client that is capable of parsing the custom profile (which it should of course) will be able to extract at least the known parameters from that custom profile.

best,
Martin.K

 Signature 

TUIO community site: http://www.tuio.org
reacTIVision framework: http://reactivision.sf.net
TUIO & reacTIVision CVS: http://sf.net/projects/reactivision
reacTIVision forum: http://sf.net/apps/phpbb/reactivision

Profile
 
 
Posted: 28 January 2009 05:21 AM   [ Ignore ]   [ # 10 ]
Avatar
RankRankRank
Joined  2008-06-16
Total Posts:  330
Sr. Member
MorbidChimp -

I dont know alot about TUIO beyond its use in multitouch so I dont know what else its used for, but if it were to brake alot of MT apps atm, I dont recon it’d be such a bad thing as those programmers would probably update their apps to take advantage of TUIO2 extra’s almost immediately. You said yourself TUIO was created for another purpose so really everything around now would be grafted unto its original design and made to work with it, instead of it working for the them. But TUIO2 would be designed with its other uses in mind and more amendable to future improvements.

The time it would take to make a 1.1 version to maintain compatibility with older software could probably be better spent developing TUIO2 perhaps? This OSC, before NUI forums I’d never heard of it either. XML would be handy though.

Yes, I have been thinking a lot about backward compatibility, it would be actually the more elegant way, since then older clients still would continue to work while newer ones could take advantage of the new features.

But for many reasons, I am also looking into a completely new TUIO2 protocol, since as you said we can update the software rather easily. the TUIO sources could implement a legacy mode and the TUIO clients actually have their client API which would not be touched too much by the changes in the protocol.

Regarding the question of OSC I still strongly support the initial decision to have TUIO based on this standard, and since it has proved to be very easy to implement in any environment so far, I don’t see any reason to change that. So I am not really open to discuss that part of the protocol layer I am afraid.

best,
Martin.K

 Signature 

TUIO community site: http://www.tuio.org
reacTIVision framework: http://reactivision.sf.net
TUIO & reacTIVision CVS: http://sf.net/projects/reactivision
reacTIVision forum: http://sf.net/apps/phpbb/reactivision

Profile
 
 
Posted: 28 January 2009 06:28 AM   [ Ignore ]   [ # 11 ]
Avatar
RankRank
Joined  2008-06-26
Total Posts:  243
Member
Martin Kaltenbrunner - 28 January 2009 05:21 AM

Regarding the question of OSC I still strongly support the initial decision to have TUIO based on this standard, and since it has proved to be very easy to implement in any environment so far, I don’t see any reason to change that. So I am not really open to discuss that part of the protocol layer I am afraid.

best,
Martin.K

Martin, i have to second that. Beside the “pure” MultiTouch community there are a LOT of people out there using OSC (with or without TUIO) with different techniques and platforms. I would like to mention here only the VJ - scene… but also the whole bunch of things happening with WiiMotes. I think that TUIO or later a TUIO2 protocol as a standard is a very good thing. We at xtuio.com are also commited to this and are supporting the current approach as a standard way to communicate with different client programs. I had a short discussion with Ben Britten about this topic - he will join this discussion sometimes later this week or next week (toght deadlines here). But far as i can say, if we are evolving BBTouch we would like to do that in that way - if possible maintaning backward compatibility…

Cheers,

 Signature 

Sandor Rozsa
--
http://www.xtuio.com - home of uniTUIO: bringing MultiTouch in the 3’rd dimension
http://www.cd-cologne.de - my company homepage

Profile
 
 
Posted: 28 January 2009 06:43 AM   [ Ignore ]   [ # 12 ]
RankRankRankRank
Joined  2007-07-14
Total Posts:  819
Elite
Martin Kaltenbrunner - 28 January 2009 05:21 AM

Regarding the question of OSC I still strongly support the initial decision to have TUIO based on this standard, and since it has proved to be very easy to implement in any environment so far, I don’t see any reason to change that. So I am not really open to discuss that part of the protocol layer I am afraid.

How would you transfer images using OSC/TUIO?

 Signature 

HID Multitouch driver for Windows 7 http://multitouchvista.codeplex.com/

Profile
 
 
Posted: 28 January 2009 06:56 AM   [ Ignore ]   [ # 13 ]
RankRankRankRank
Joined  2007-07-14
Total Posts:  819
Elite
sandor - 28 January 2009 06:28 AM

Martin, i have to second that. Beside the “pure” MultiTouch community there are a LOT of people out there using OSC (with or without TUIO) with different techniques and platforms. I would like to mention here only the VJ - scene… but also the whole bunch of things happening with WiiMotes.

Programs are still written by developers and not VJ rasberry I bet less than 1% of developers have ever heard of OSC.  OSC was designed for musical instruments as a replacement for MIDI. You can run a musical software on MultiTouch hardware, but still MultiTouch is a general hardware, like a mouse. Or did you saw a mouse sending OSC messages to windows?
We discussing here a low level protocol for transferring touch events from device to framework, not a protocol between device and custom program.

P.S.: WiiMote don’t uses OSC/TUIO. There is a program that converts WiiMote input in TUIO, but WiiMote is not a “TUIO” hardware.

 Signature 

HID Multitouch driver for Windows 7 http://multitouchvista.codeplex.com/

Profile
 
 
Posted: 28 January 2009 07:35 AM   [ Ignore ]   [ # 14 ]
Avatar
RankRank
Joined  2008-06-26
Total Posts:  243
Member
Daniel D - 28 January 2009 06:56 AM

sandor - 28 January 2009 06:28 AM
Martin, i have to second that. Beside the “pure” MultiTouch community there are a LOT of people out there using OSC (with or without TUIO) with different techniques and platforms. I would like to mention here only the VJ - scene… but also the whole bunch of things happening with WiiMotes.

Programs are still written by developers and not VJ rasberry I bet less than 1% of developers have ever heard of OSC.  OSC was designed for musical instruments as a replacement for MIDI. You can run a musical software on MultiTouch hardware, but still MultiTouch is a general hardware, like a mouse. Or did you saw a mouse sending OSC messages to windows?
We discussing here a low level protocol for transferring touch events from device to framework, not a protocol between device and custom program.

P.S.: WiiMote don’t uses OSC/TUIO. There is a program that converts WiiMote input in TUIO, but WiiMote is not a “TUIO” hardware.

LOL! Daniel, slow down the horses grin first, i am not a developer so sorry if i make some mistakes in my statements or i use the wrong terminology… We “could” discuss what MultiTouch is - a system, a hardware or whatever. But i think that it would be better to do that in another thread and keep the topic of the thread “present and future of TUIO” intact grin Sorry again if i have said something that does not makes sense…

Cheers,

P.S.: “Or did you saw a mouse sending OSC messages to windows?” - Nope. Sorry. But i’m on a Mac grin wink

 Signature 

Sandor Rozsa
--
http://www.xtuio.com - home of uniTUIO: bringing MultiTouch in the 3’rd dimension
http://www.cd-cologne.de - my company homepage

Profile
 
 
Posted: 28 January 2009 11:42 AM   [ Ignore ]   [ # 15 ]
Avatar
RankRankRank
Joined  2008-06-16
Total Posts:  330
Sr. Member
Daniel D - 28 January 2009 06:43 AM

How would you transfer images using OSC/TUIO?

Well, you simple would use a blob data structure for that.

But answering you earlier suggestion of what you would want to see in an extended TUIO implementation:
I think the whole purpose of TUIO is to transmit an ABSTRACTION of the scenery on the interactive surface.
Therefore I do not really see the point of transmitting the original image data to the client.
If you really need to work on or with the original image you probably should choose a computer vision library.

Regarding the proposed additional shape description I clearly imagined the transmission of shape DESCRIPTORS rather than the image itself.
You already mentioned some of those descriptors in an earlier post.
At the moment I imagine the following data, maybe all represented with a dedicated message:

* basic shape data: id, center, angle, area
* oriented bounding box: described with four points (8 floating point numbers from 0..1)
the OBB describes the object enclosure in much more detail and additionally one can derive the width and height, as well as the “normal” bounding box and its dimensions on the client side
* contour: a list of contour points, that describes a simplified contour of a region
this means that using iterative methods the outline is reduced until a certain maximum error
the result is a lossy but in most cases probably accurate description of the region contour
* skeleton: a (more complex) list of skeleton points
since the skeleton can have branches this is not a simple list of points but needs to encode the joints as well
* span list: if you need access to the actual region pixels, they can be easily reconstructed from a list of span lines
this would be a kind of “run length encoding” compressed blob description format which would also conveniently exclude the eventual neighbouring regions within a certain area.
I am not even sure if I would include this in an actual implementation, but I think it is important to include it into the protocol

please let me know if you are missing any important shape descriptor for you here ...
saludos,
Martin.K

 Signature 

TUIO community site: http://www.tuio.org
reacTIVision framework: http://reactivision.sf.net
TUIO & reacTIVision CVS: http://sf.net/projects/reactivision
reacTIVision forum: http://sf.net/apps/phpbb/reactivision

Profile
 
 
   
1 of 5
1