Daniel D - 28 January 2009 06:56 AM

less than 1% of developers have ever heard of OSC.  OSC was designed for musical instruments as a replacement for MIDI. You can run a musical software on MultiTouch hardware, but still MultiTouch is a general hardware, like a mouse. Or did you saw a mouse sending OSC messages to windows?
We discussing here a low level protocol for transferring touch events from device to framework, not a protocol between device and custom program.

Hey Daniel, as I said I am not really willing to get into an OSC discussion, but I see that for some reason you apparently don’t like OSC.
First I think, that for now yourself and probably the other 99% of the developers from your empirical study here are already aware of its existence since they worked with TUIO, so we can tick that knowledge problem off.

Apart from that I think we also agree that we would like to see a network socket based protocol, so we can work across platforms, programming languages and different machines. I think the nice iPhone examples we have seen recently and the big variety of supported programming environments are making my point quite clear here.
So I think the “windows mouse” argument is also obsolete with that. Apart from the distributed architecture I have been trying to define a very clean TUIO client API, that I try to maintain consistent across different programming languages.

And finally again talking about the origin and qualities of OSC itself, although it was apparently made for musical instrument control, it is now more and more used for device control or component communication in “multi-media” installations in general. And this simply because most of those environments already come with some OSC implementations and the OSC message and bundle format is just a very simple, compact, elegant and straightforward way of encoding and transmitting control data. And this is what we are basically doing with TUIO as well.

BTW, I don’t know the nicknames here in the forum, so it would be nice if you could briefly tell me about you, your work and your interest in contributing to TUIO in general.

cheers,
Martin.K

I think an example of sending an image, as Daniel suggests, would be something similar to Microsoft’s raw application where it shows the users a filtered image of what’s going on with the surface. Also the shapes of these raw images are then used for various things like painting and whatnot. So there can be times when an image is important, although the contour and maybe some filtered pixel data is probably more important. I think rather than discussing if something is important, it’s more important to acknowledge if it’s doable with the current protocol and I think that’s where Daniel is speaking (I could be wrong?). No matter the outcode of TUIO, it’s clear that certain people are going to need custom profiles. For an example, a university studying pressure, might want to send a significant more amount of data regarding pressure information than say tbeta or reactivision does.

With that said, I think shape descriptions are probably one of the most missing things (contours specifically) and a shape profile with all that you mention would be a great addition. Although does this mean that all the things listed would then be mandatory for basic TUIO? For example, would all trackers then have to go and make skeleton’s and span lists? I think the list sounds good. One extra thing, which I’m not sure what to call it is ‘real center.’ For example, the center of a blob is not always the center of a finger and possibly it would be good to send both.

Martin, I just wanted to quickly respond to this also as I think there may be a misunderstanding:

Well, what seems to be an unimportant “temporary fix”, actually is an incompatible fork and we really should avoid that considering the large user base the tangible or multi-touch community already has. As you mention touchlib and tbeta have 50.000 downloads and I just had to look it up, reacTIVision has reached around 60.000 as well, and the TUIO client base reached around 80.000 downloads to complete the statistics. Looking at these numbers, which probably mean that we have a shared user base of a few thousand people it is more important than ever to consider and maintain interoperability.
If you are missing a feature in HTML you also just won’t go and fork Mozilla to support that feature ...

I’m not sure why you think this is an unimportant fix? Clearly, if many people weren’t writing in asking for this compatibility we wouldn’t create a quick fix. You said yourself that people using reactivsion have also needed such changes and some have went ahead and hacked the code to add such functionality. The fact that no one has created a compatible client for touchlib in more than 8 months (the time touchlib has been sending height/width) shows that it was time for someone to bite the bullet and come up with something that’ll ease people’s pain at least temporarily. Surely you know we can’t wait on one person (we know you’re busy) to fulfill the needs of a whole community and this is why we made a temporary fix while also sparking discussion with you and now the community so we can solve this on terms we can all agree too. Just as you are busy, so are we and therefore the straightest line to fixing the issue was taken (which we can agree is unfortunate). If you feel we are ‘forking’ you and reactivision in some way, please let us know since that is not the intention. In the meantime, since we did update clients to work with touchlib/tbeta, we’ve now fixed the issue of breaking the client from working with reactivision (so it’ll work with both for now).

BTW, Martin - I’ve used reactivision in the past, before I began doing multitouch work. It’s great and you can see some of my work using reactivision here: http://www.youtube.com/watch?v=Fqi5l-LwA5w and http://www.youtube.com/watch?v=jxznZU_XQIA&feature=channel_page

Martin Kaltenbrunner - 28 January 2009 12:15 PM

BTW, I don’t know the nicknames here in the forum, so it would be nice if you could briefly tell me about you, your work and your interest in contributing to TUIO in general.

I’m a developer of MultiTouchVista. It is a multitouch framework comparable to Surface SDK. It provides information not only about touches, but also raw image from camera/-s. Client side of Framework has a core part - usable by any application (winforms, xna, console), and a WPF part usable by WPF applications.
Touch and raw image information is delivered by “input providers”. Already exist “multiple mice”, “touchlib”, “tuio” and “vision system” input providers. Server part of MTV uses input provider to send data to applications. Multiple applications can receive data at same time. Server part can be run as a console application (mostly for debugging), and as windows service. Data is sent using WCF (web services like) throug named pipes. Without any changes in protocol data can be sent over any other protocol, from tcp/udp, http web services till exotic things like MSMQ or E-mail
Because vision system input provider (only it has raw image support) is part of a comercial project it would be great if TUIO or any other open source input provider could provide raw images.

Seth (cerupcat) - 28 January 2009 03:00 PM

You said yourself that people using reactivsion have also needed such changes and some have went ahead and hacked the code to add such functionality. The fact that no one has created a compatible client for touchlib in more than 8 months (the time touchlib has been sending height/width) shows that it was time for someone to bite the bullet and come up with something that’ll ease people’s pain at least temporarily. Surely you know we can’t wait on one person (we know you’re busy) to fulfill the needs of a whole community and this is why we made a temporary fix while also sparking discussion with you and now the community so we can solve this on terms we can all agree too. Just as you are busy, so are we and therefore the straightest line to fixing the issue was taken (which we can agree is unfortunate). If you feel we are ‘forking’ you and reactivision in some way, please let us know since that is not the intention. In the meantime, since we did update clients to work with touchlib/tbeta, we’ve now fixed the issue of breaking the client from working with reactivision (so it’ll work with both for now).

Hey Seth, thanks for your comments, I’d like to apologize if I caused any misunderstanding, I am yet not very experienced in forum communication. I’d like to emphasize that I am really happy that now after a long (and maybe too long) period of experience with the usage of TUIO we are now able to kick off a hopefully constructive discussion of how to tweak and improve a technology that apparently up to a certain extent has proved useful for many applications. So let’s now concentrate on the several open questions, I think in this thread there are already quite a few concrete technical statements where the discussion should lead. I am happy to find some competent partners here in this forum, who also already gathered some valuable experience in areas which I might not have considered myself so far. I will dedicate the next month to the review and improvement of the TUIO protocol, source and client implementations so any constructive contribution will be really appreciated.

good night,
Martin.K

Seth (cerupcat) - 28 January 2009 03:00 PM

With that said, I think shape descriptions are probably one of the most missing things (contours specifically) and a shape profile with all that you mention would be a great addition. Although does this mean that all the things listed would then be mandatory for basic TUIO? For example, would all trackers then have to go and make skeleton’s and span lists? I think the list sounds good. One extra thing, which I’m not sure what to call it is ‘real center.’ For example, the center of a blob is not always the center of a finger and possibly it would be good to send both.

Yes, I also think that the general purpose shape descriptors will hopefully solve most of those additional requirements, and therefore I am convinced they should be able to replace the need for knowing about the raw image itself.
Regarding the centre, there are probably many ways of calculating that, but I would leave it to the implementation to determine what its best centre should be.
Apart from that, the other data such as oriented bounding box,skeleton and contour should also leave some additional options to the client to determine an alternative centre point if necessary.

And I don’t think that the full list of those descriptors should be mandatory at all.
Just as like now some TUIO sources only send cursor messages, without the need of sending object messages.
The client just should be able to deal with less detail of shape descriptors if the source does not fully implement all of them.

cheers,
Martin.K

Yeah I fully agree. Good points Martin.

It’s good that we’re able to have this discussion and we’re really glad to be discussing this with you directly since there’s no better person to discuss this with. We’re here to help you, so hopefully when we work out some more specifics, we’ll be able to help you implement some of this so you don’t have to do it all yourself.

night for now,
-Seth

Daniel D - 28 January 2009 03:11 PM

I’m a developer of MultiTouchVista. It is a multitouch framework comparable to Surface SDK. It provides information not only about touches, but also raw image from camera/-s. Client side of Framework has a core part - usable by any application (winforms, xna, console), and a WPF part usable by WPF applications. Touch and raw image information is delivered by “input providers”. Already exist “multiple mice”, “touchlib”, “tuio” and “vision system” input providers. Server part of MTV uses input provider to send data to applications. Multiple applications can receive data at same time. Server part can be run as a console application (mostly for debugging), and as windows service. Data is sent using WCF (web services like) throug named pipes. Without any changes in protocol data can be sent over any other protocol, from tcp/udp, http web services till exotic things like MSMQ or E-mail .

Hello Daniel, this sounds quite exciting, if I understand that correctly this is something similar to what MPX is doing for X11/Unix?
I would like to play with the multiple mice support for example, so I will try your framework to get a better idea about it.

Daniel D - 28 January 2009 03:11 PM

Because vision system input provider (only it has raw image support) is part of a comercial project it would be great if TUIO or any other open source input provider could provide raw images

OK, but what would you like to do with the raw image data if you already have all the necessary meta data available about it?
And what is that commercial project or vision system input provider you are mentioning?

ciao,
Martin.K

I don’t know what MPX is.
See this video for raw image usage examples: http://on10.net/blogs/nic/Live-Labs-Shadow-Box/
I have to ask my chef if I’m allowed to speak about commercial project.

Daniel D - 29 January 2009 03:04 AM

See this video for raw image usage examples: http://on10.net/blogs/nic/Live-Labs-Shadow-Box/

Hello Daniel, the application shown in this video looks quite nice to me, but I’d say that it shouldn’t be the purpose of TUIO to transmit the full image data for later processing on the client side. As I stated before TUIO transmits an abstraction to the clients, simply to take away the actual burden of image processing, so the developers can focus on the input gestures and HCI design of their applications. Needless to say that the transmission of the raw image at full frame rate should be considered video streaming, and I think there are many other libraries around that do that job quite well. So, I think if you’d want an input source for raw video for your framework you should probably look in that area.
best,
Martin.K

I already have a way/protocol to transmit video from input provider to framework and from framework to application.
What I needed, is an input provider that will support sending raw images to the framework. If TUIO will not support it, I will have to look for another input providers.

Hi NUI Group, Hi Martin,

I’ve talked with a few of you already, but wanted to join in on this conversation.  I am well aware of the need for a protocol that extends beyond the current version of TUIO.  There are a lot of ways we can go, so it’s great to discuss it all openly and design it together as we move forward.

Over the past few months I’ve been developing a very low-cost, easy-to-setup tag tracking system called Trackmate.  I’ve spent a lot of time combing through the protocol layer (TUIO in particular) trying to look at how it could scale to a large variety of spatial input devices (interfaces in physical space that senses objects, fingers, materials, shapes, motion, etc.). 

I think there are many great design decisions in TUIO, but it is very Reactable-centric (understandably) and as a community we need to politely fork.  I realize that sounds a bit harsh, but the impact of such a protocol is vast and will greatly determine which new interface ideas are enabled or constrained as this community (and many other interface research groups) move forward.  Forking should not be seen as disrespectful; it is simply the decision to take an open source project in a new direction.  The new direction I propose is an open, community-centric, evolving protocol that works across all (as many as practically possible) spatial input devices—it’s more than changing a few profile names or adding a few message parameters; it’s a change of mindset. 

Why is the protocol layer so important?  Most of you probably have thought about this to some degree, but I think it is worth saying.  The protocol layer sits between spatial input devices (such as Reactable, SenseTable, g-speak, various NUI Group tables, Trackmate, Nabaztag, MS Surface, etc.) and user-level applications.  If every spatial input device has its own profile, then in essence, each system is vertically integrated with its own special applications, developer-base, and community.  This is detrimental with so many new interfaces emerging since it spreads us too thin.  Developers are writing specialized software (some of it quite amazing!) for each input device and risk their code being obsolete when a new interface is adopted.  Ideally, the protocol layer needs to have a common ground for all spatial input devices (such as the most basic id, position, and rotation), which can then be easily extended (for pressure, temperature, velocity, shape, motion, gestures, etc.) without giving up compatibility (backwards, but more importantly, forwards).

This new initiative to extend the ideas of TUIO, but with an open, flexible, scalable, community-centric view is being developed at LusidOSC.  The second version is already underway and everyone is encouraged to join in its development (see the “LusidOSC v1.1 document” on the website).

As for the comments so far in this thread, I’d like to express my opinion about some of the design decisions we face.

* UDP/TCP/OSC: What’s it all mean?  UDP is fast, but can lose packets.  TCP is reliable but often functions in bursts since it depends on acknowledgment of data received.  For interface applications, UDP seems like the best choice since: 1. the process switching time would greatly affect the transmission rate locally over TCP since data uses flow control (potentially adding between 5-100mSec additional delay, depending on how your OS is setup); and 2. UDP pretty much never loses data when run locally or within the same local network (as long as things aren’t congested).  OSC is a layer on top of either (although, UDP is the one almost always used because of the aforementioned reasons).  OSC has a ton of implementations in various programming languages so it makes for pretty straight-forward development (that’s great!).

* Large data transport, like sending entire images, sounds, videos, etc.?  OSC would definitely choke on this, but it seems beyond the scope of the interface layer.  Perhaps a secondary layer that has different design constraints would be a good decision for handling this. (i.e., you could serve and stream the data, similar to streaming video, and your app could request/use it when desired).  I personally think that trying to pack too much data into the interface protocol layer would be breaking a layer of abstraction (the data and the control information should generally be in separate pipes), but that’s a big question and definitely needs some more discussion. 

* Ad-hoc protocol extending? It is inevitable that next month, or next year, technology will change and there will be new data that people want to add to the protocol.  This brings up two important points: 1. when a system is first being developed, how difficult will it be for the developers to extend the protocol in an ad-hoc way for their needs?; and 2. when the technology is adopted, how can things stay nicely backwards compatible so that every application doesn’t need to be modified to stay up with the latest trends?  I think LusidOSC v1.1 is starting to have some nice ideas with respect to this.

Let’s keep the dialog open and feel free to join the effort at LusidOSC.

Adam

Martin Kaltenbrunner - 29 January 2009 07:53 AM

Hello Daniel, the application shown in this video looks quite nice to me, but I’d say that it shouldn’t be the purpose of TUIO to transmit the full image data for later processing on the client side. As I stated before TUIO transmits an abstraction to the clients, simply to take away the actual burden of image processing, so the developers can focus on the input gestures and HCI design of their applications. Needless to say that the transmission of the raw image at full frame rate should be considered video streaming, and I think there are many other libraries around that do that job quite well

There are benefits to pumping raw image data to apps in the same stream as the processed/abstract data.  There are a number of scenarios where an application just wants to analyze the image corresponding to an abstract contact - they don’t want/need to analyze the entire raw image.  For example, the SurfaceWare demo (http://blogs.msdn.com/surface/archive/2008/10/22/surfaceware.aspx) looks for contacts with specific tag values (what you guys call fiducials) and then passes those contacts into the GetRawImage API to retrieve a copy of the raw image cropped to the contact.  This is only possible because we tie the data stream for raw images to that of processed contacts… if they were coming from seperate non-synchronized streams, you’d end up cropping the images incorrectly.  The degree of incorrectness will be based on the relative speed of the two data streams, the framework’s event queue technique(s), and how fast the contacts are moving.

This synchronization is non-trivial to implement (we still haven’t done it for the WPF event queue) and probably even harder over OSC, so I can understand why it would be left out of TUIO, but it *is* important for people creating mixed-mode applications that work with both raw images and abstract contacts.

-Robert
Program Manager, Surface SDK

rlevy - 22 February 2009 03:03 PM

This synchronization is non-trivial to implement (we still haven’t done it for the WPF event queue) and probably even harder over OSC, so I can understand why it would be left out of TUIO, but it *is* important for people creating mixed-mode applications that work with both raw images and abstract contacts.

Hello Robert,
I perfectly understand that there will be always a good reason or need to have access to the raw image data, especially for AR type applications.
But as I outlined above I understand TUIO as an abstract description of what is happening on an interactive surface. According to this concept, and of course the technical and bandwidth limitations of the UDP based communication protocol, I’d rather omit the transmission of the full frame or even partial raw image data.

On the other hand I am currently restructuring the TUIO client API, which many people here are probably already using on the client side. I can perfectly imagine to use this client API also directly bound to the tracker component (without the TUIO network layer in between) as a simple C++ or even Java library with JNI interface. In this case I would consider to add full frame or partial raw image data to the TuioListener callback API or directly as an attribute of the TUIO objects.

BTW, I still didn’t have the chance to get my hands on a MS Surface, so maybe we should do some reacTable <-> Surface interchange one day
But in the meanwhile I’ll read a bit into your Blog, which looks quite exciting!
best,
Martin.K

Martin, what do you mean “interchange”? Know-how or communication (send data from one to other)?
If second then I find Surface communication is pretty simple and extensible. I even made Surface to run on my custom hardware using MultitouchVista Input providers of MTV makes Surface hardware independent

Well, I meant that it would be nice to get together some day in order to have a chat and eventually try each others table and/or applications.
cheers , Martin.K