Hello everyone,

I am looking for people interested in bringing some intelligence in the ccv module…

For the moment the tracking as I understood from the source is based upon a shader pipe.
I am interested in bringing some intelligent recognition and learning into the software by implementing some simple heuristics.

For instance, gestures constraints are determinated by the speed of the gesture as well as the kinetic properties of the human body.

Kinetic properties of the human body

A finger movement going from up to down with a certain speed will most likely always go into a certain direction,
due to the kinetic energy of this movement it is rather unlikely that the movement will pop up somewhere else on the screen in a certain laps of time.
So you can say “if the movement has this acceleration, and is going in this direction, predict a blob in Cur(x,y) + DistPos(blob(n-1))/2 where Cur(x,y) is the current position and DistPos(blob(n-1)) is the distance achieved between the last two camera frames.

If we generalize this and couple it to a database of heuristic constraints on gestures, for example a gesture with this identity will have this behavior we can actually enhance the recognition of speedy gestures and interpolate this movement by sending predictive blobs into the network pipe, enhancing hereby the applications behavior.

Blob guessing, false blob elimination, adaptive calibration for DI

When making calibration very sensitive to blobs you make them as well very sensitive for other unpredicted events, especially when DI is used.
For instance when putting your finger you may as well get your other fingers even when not touching the acrylic illuminated an seen as unstable blobs in the way that most of the time they dissapear and appear making event based tracking difficult.

By normalizing, like for gesture recognition, the distance between the detected blobs, which will always be within a distance given by the proportions of the human body, we can come to the conclusion that it is actually a finger that we’re tracking, an inference database can actually store the finger patterns by counting the number of positive finger patterns, finger patterns that made it above a certain number of frames tresshold, this way we now that we can eliminate all blobs around in a certain neigbourhood around the blob.

Now imagine we have the sunshine coming in, the luminosity increases and shadows of the users are projected on the screen with a certain intensity.
Rear illumination is not usefull anymore, the light sensor that comes with some infared illuminators switches the light off, we know that the camera automaticly increases shutter speed to avoid saturation, if we monitor this and define a tresshold we can actually ( to start ) invert the CCV settings to track dark blobs.

By implementing these couple of simple heuristics I believe we could give the app a better behavior in semi-sunshine conditions with variations of ambient light conditions, increasing the domain of applications of image analysis multi touch based applications, enhancing stability, making it possible to create industry standard applications…

Bonne journée,
Pixelpower

Well lets get started hahahahaha.

Well go ahead and help instead of laughing…

Updated by Moderator

since i´m no programmer but a mechanical engineer, i cannot help you, but i consider your thoughts to be very usefull. hope you get some more support, not just the “go ahead and we are waiting for the source to be released” answer…

well if I hurt you I’m deeply sorry, but it was really nothing meant to…
I was just laughing because those are all great ideas but very difficult to code, and that’s all… Please don’t leave this forum, we need guys like you.
En espérant te revoir sur le forum....
MashineGun

[EDIT] and of course that wasn’t at all the type of respond that Vega describe ("waiting for the others to do the work").

Never mind,

Did not understood it in that way.

Any positive critics ?

There is actually one thing that could be not too difficult to code an that is switching from rear DI to front DI by triggering the track dark blobs…
To do this we need a tresshold setting on the average brightness of the image or on the shutter speed of the camera, or on combination of both…

Once the image goes above the tresshold we enable the dark blobs mode...( plus some other things or a second configuration file)
The sensivity of the IR illuminator’s sensor is synchronized with the tresshold so that it doesn’t illuminate anymore when the tresshold is reached

other way:
Calculate average brightness with illumination on and no ambient light + obtain shutter speed, this gives us a reference

if avg brightness goes up or stays stable and shutter speed increases
switch on alternative config
if teh combination falls back to the reference
pick the reference config

other way:
monitor brutal changes in the average brightness by calculating brightness acceleration and ignoring the camera’s recalibration

Any critics are welcome !

Hey pixelpower,

Really great ideas there!
I’d be interested in all of them, but especially in the false blob elimination part.
I have a DSI setup in a public area and as I observed people using the screen it can give them hard time when there are false blobs caused by hovering fingers.

We could also enhance the multiple blob treatment part of the applications, so that people will be able to drag pictures with 4 or 5 fingers or zoom and rotate with 8 or 10 fingers.
Since I have some experience in development we could start to think about how to do it and what to start with.
Greg

I’m very keen on those ideas, really.
And could it be possible to run twice ccv, one with white blobs setup, one with dark one and merge the two tuio streams ?

Good ideas, some may even be relatively simple to implement.

The biggest problem I can think of with using body kinetics for predictive analysis is the multi-user (or dynamic position single user) aspect of multitouch. Unless you can associate inputs to a particular user, certain things become unpredictable (if you don’t know the position/orientation of the user creating the input, it’s difficult to apply constraints that are based on that info). On the other hand, that same problem could be used as an advantage from the opposite perspective: by measuring characteristics of inputs (motion velocity, acceleration, position, direction, input patterns, etc.) it could be possible to determine information about the user (which user created the input, where that user is relative to the surface, etc.)

I’m not sure what your background is (I get the feeling you may have already looked into this), but you might try looking into Kalman filtering as an interesting starting point for ideas (has potential application for improving blob tracking/prediction).

Hi greg,

Thanks for the support.

I am in teh process of creating my business in multi-touch applications based on ccv to AS3 applications, so I need to feed my family ( and myself a little bit) so I am really motivated to make this thing more adapted to industrial use, beta users don’t necessarily know the mechanisms behind it,
so they are not able to adapt their touching, what they want is, to do what they think it will expect it to do.

I believe the first thing to do is to obtain interest from the developers of CCV, so they can provide us an read&write svn account, and collaborate with them to understand where how we/they could adapt the code.

In the meanwhile:

We can already work on an algorithm to eliminate false blobs or to take advantage of them if they appear to be in the hand region
We can for instance take the information that is transmitted using osc protocol and list it, define an algorithm that will detect those false blobs:
for instance: each blob has it’s own id, when an id is late ( for example the newest blob has a framecount of 100 and the late id has a framecount of 60 ) this means that this late id is stable in time

Now an idea,draft, of basic algorithm based on osc information ( which is different from the image analysis treatment) could be the following:

## false blob elimination, creating a hand object
frame 0: a blob is detected, treat it as a normal blob, dispatch a touchdown event
frame 1: check around (in the close neighborhood, defined by the human hands max/min distance) for simultaneous created blobs
place them in a competition mode and elect the most stable in time ( did not disappear ) to be the directing finger,
treat the unstable blobs with less care, do not delete them, but keep them referenced, to see if they appear in n+1 frames

## rotation
frame 2: we now can define a rotation event based on the rotation information transmitted by the hand object, similarly as we would do with a fiducial

##Deeper thoughts
If we take a deeper look at this we see that the hand object is actually a tree structure with as “topleaf” the most stable blob and directly below the n-1 less stable blob for the hand region.
This leads us to believe that we actually would be able to create a hand tree structure for each blob with a certain predefined depth based on his id and the delta in the id.
Once we have a tree structure we have computer science fundamentals and the ln(n)* log(n) search and sort functions that were invented for them.

If we take a look on the tree structure, what do we see after four frames( the frame number is in ()):

--------------------------- topleaf (n most stable blob in region)(3)
---------------------------
----------------------------------/---\ ----------------------------\
---------------------------------/-----\ ----------------------------\
-----------------------------(n-1(1))-(n-1(0))<--newest child---(n-1(2))<-- most stable child

We actually have a hash map structure ( which is even quicker for what we want to do):

stabilty hashmap hand region:
hash key: the stability, denoted by the most active frames the blob has
hash value: the blobs id or object

We can now add a nice heuristic:  “ As long as the brave topleaf president exists give the the less unstable children the biggest chance to survive “
The only thing we need right now is some kind of simple distance based region split algorithm transposed to the detected blobs

I will try to post an algorithm tomorrow…
Hope it helps

cheers !

Kalman filtering—seems to be very interesting—I believe it had some applications in pixel based missile detection…
thanks man

Actually yes be reading wikipedia kallman filtering seems to be the thing i wouldlike to perform but on image regions and not the whole image, this would imply some kind of threading for each hand or a single threaded transposition of the multithreading, working with data structures, this would enhance performance and stability as well.
thanks again

OK so what I understood is,

Based on the current and previous state of the hand structure we predict a next state of the hand structure and compare the difference between the predicted state and some supposed real state noise using an interpolation function, if the interpolation function is within a delta we keep the noise as a real state.
The more our predictions are correct the more our interpolation function is correct if not we go back to the previous or inital state of informations ?

Concretely:

##Create a hand object for each finger in the far region with the associated fingers in the near region

finger_neighbourhoud = 10 pixels
hand_neighbourhoud = 100 pixels

# we could use a probabilty or a proportion between appearings and dissaperaings as
# discriminator

MaxStabilityTolerance = 2

class handfinger{

# this should be based on min/max of hands
private int fingerdistance = 5;
private int handdistance = 100;

private blob _id;
private int _stab = 0
private int _unstab = 0
private handfinger children = [];

handfinger(blob id){
_id=id;
_stab = id.ID;
_unstab = 0;

dispatchEvent(new TouchEvent(TouchEvent.handdown),this.getRootHandFinger)
}

public find(blob id,discriminator d){
# this could be a BFS but we decided not to chain unstability levels
fingersinhand = []
if(d.equals(’finger_neighbourhoud’)
for each finger in children
if(finger.isClose())
return [finger];

if(d.equals(’hand_neighbourhoud’)
for each finger in children
if(finger.isClose())
fingersinhand.add(finger);
return fingersinhand;
return null;
}

public remove(handfinger){
children[handfinger.getID()] = null;
}

public add(handfinger){
children[handfinger.getID()] = handfinger;

}

public set(blob id){
if(parent!=null)
parent.children[_id.ID]=this;
_id = id;
_stab = id.ID;
_unstab++;
dispatchEvent(new TouchEvent(TouchEvent.this.getRootHandFinger))

}
public update(blob id){
_id = id;
_stab = id.ID;
if(_unstab>0)
_unstab--;
dispatchEvent(new TouchEvent(TouchEvent.this.getRootHandFinger))
}
## we can see here that we could implement a kallman filter
## for the moment we just make an approximation on the distance and the uncertainty
## to find the blob in some small distance, if we want to be more precise we could create and interpolation function
## based on the previous (position’s,velocity,angle,..) blob and the current blob and make our discriminator
## to be an error function between the interpolated position and the real position

public isClose(Point p,discriminator d){

if(d.equals(’finger_neighbourhood’)
if(euc_distance(p,new Point(_id.x,_id.y)) return true;

if(d.equals('hand_neighbourhood'))
if(euc_distance(p,new Point(_id.x,_id.y)) return true;
return false;

}

}

static function UpdateFinger(blob id):
#check if exists already
for finger in elected_fingers:
if (finger[id.ID])
#exist already we just update the id object this will reduce the unstability as well
finger[id.ID].update(id)
break
if (finger.children[id.ID])
#exist already we just update the id object this will reduce the unstability as well
finger.children[id.ID].update(id)
break

static function SearchUpdateFinger(blob id): 
#if it does not exist check if we can find it in the finger neighbourhood of the elected fingers
elected_finger = elected_fingers.find(id,'finger_neighbourhoud')
if(elected_finger)
#check if stability is still ok
if(elected_finger.getStability(id)<=MaxStabilityTolerance)
#replace the blob with the current one
elected_finger.set(id)
break
else
#do some maintainance on the hands
#since he exceeded his stability tolerance
#delete elected finger,
#we could eventually see if some children can be a new elected_finger
# delete hand and create a new hand
elected_fingers.remove(elected_finger) # will delete hand
dispatchEvent(new TouchEvent(TouchEvent.handup, elected_finger))
elected_fingers.add(new handfinger(id))


else
#check if we can find it in the hand neigbourhood
#here is where our program will go and could search into a tree
#thing is we don't want to maintain levels and levels of depth of unstability
fingers = elected_fingers.find(id,'hand_neighbourhoud')
if(fingers && fingers.length > 0)
possible_finger = fingers.find(id,’finger_neighbourhood’)
if(possible_finger.getStability(id)<=MaxStabilityTolerance )
possible_finger.set(id)
break
else
#do some maintainance on the fingers
#since he exceeded his stability tolerance
fingers.remove(possible_finger)
# add the new finger
If(fingers.length<3)
fingers.add(new handfinger(id))

The main program
ElectedFingers = []
while(){
newblobs = TUIO.returnBlobs()
for each(id in newblobs){
UpdateFinger(id);
SearchUpdateFinger(id);
}

Any Comments,implementations,corrections are welcome, I will try to implemnt this in an EventDispatcher class in AS3 or directly modify the TuiO class…
See you !