I’m rather new to this, so this may have been covered…

I understand that hands can be recognised, but is it possible with tools?

By this I mean, in a multitouch paint program, if you use a paint brush on the screen, would the program not only detect the location of the brush, but also be able to tell the shape of the bristle. [concievably as a height map and the program just bases actions on that] - The user has to be able to use brushes that the program is not programmed for - there are countless different types of brush…

From the shape (which might change as the brush moves), the computer would “blob” the paint on. The microsoft surface demo seems to only use the brush for location and not take into account the shape changing.

Is this possible?
Redgeneral

That’s a good question. I’m sure someone else could answer this better, but I doubt that a brush’s exact bristles would be detectable. First, this would require that the cameras be very high resolution and high FPS so that you can track as the bristles change across the screen.

What may be possible, is to create an application that uses different size brushes. If a small brush is used, you’d have a smaller blob. Then you could use a preprogrammed bristle brush in the application that matches the general size. When a bigger brush is used, it creates a bigger blob, and the applications brush can automatically change to a bigger brush. So you’re not getting the exact brush that you’re using, but something of similar size and shape (if programmed as so).

Hey Red General I can’t say I have personally researched this much, but try;

http://nuigroup.com/forums/viewthread/41/

And regarding M$ Surface, their (patented) vision techniques allow them to see physical objects on the table. This is allowed because it is a Diffused Illumination lightning method compared to FTIR.

I would love to see the following with an accurate brush physics… detecting bristles.. pressure… etc

yeah its pretty cool, DI is more usable in combination with this opposed to FTIR

Don’t let the word “patented” scare you

DI does this out of the box with blob shape recognition. Very cool, and a great way to bridge the gap between physical interaction and computer interaction without ugly gesture recognition garbage.
For pressure, you can use stuff like recognizing the distance of a blob from the surface (via its brightness), or assume that the thing contacting the surface is flexible, thus that the amount of it touching the surface increases with pressure.

MS Surface does see shapes, by the way. This may change with different applications, however, since recognizing the shape of a blob could be quite expensive. Paint programs would want to always be watching blob shape when a blob is over the drawing, but other programs may have less need to, so would perhaps only ask for the blob shape to be determined once for each blob, or maybe wouldn’t ask at all.

An optimized multitouch library would figure out blobs using very little of the data received so that a more advanced operation like finding the actual shape of a blob (instead of just tracking movement from key points) would not be needlessly executed, and better controlled by the application using the library. (I’m not sure if this is done at the moment?).

I don’t know how you all are doing detection, but this is not a problem for us.

We have a separate process for blob detection that does three things:

(1) Detects changes in the input ( “Hey!  There’s a blob over there")
(2) Captures the position of the blob
(3) Captures an IMAGE of the blob (this is the important bit)

We then send this off to an (optional) pattern matcher - the pattern matcher has an extremely short list of objects it compares the blob image against - so we can say “hey, that’s a finger-shaped/sized blob” or “look, a rectangle” - stuff like this.  This COULD be used in a fancy paint program.  We don’t have this doing much now, but plan on using it for things like “dominos”.

We take the processed touch event (coordinates and optional pattern) and pass it off to a touch processor - this links previous and the current frames together.  Each blob is given a touch ID that links it back to previous touches (so we know that finger A is the same finger A as three seconds ago and things like that).

Okay - here’s the cool piece -

The touch event is then passed along to the application - the original IMAGE (the blob) is still in the event and the application can take that blob, colorize it (or not) and paint it directly to the screen.  So, if you push hard on the screen, squishing your finger out to make it big, the paint program can take that blob and draw it on the screen, making a BIG smear.  Or, if you press lightly, a little smear.  Or, if you you a pencil (gasp - not on my table) - it will make a point.  Or, if you use a paintbrush, it can make a bent paint-brush-y shape.

It’s up to the application to use the matched pattern or not - if we don’t match a pattern, we still send the touch event through.  But having the blob in the event through the entire process is a God send in terms of application processing.  The key is to NOT just get points sent to your application, get the blob too.

FWIW - I only really know our system - other libraries may do this too.  Your mileage may, of course, vary.