3 of 6
3
OS X Multitouch? 
Posted: 15 May 2008 02:16 PM   [ Ignore ]   [ # 31 ]
New Member
Rank
Total Posts:  59
Joined  2007-07-09

thanks for the heads up.

Profile
 
 
Posted: 16 May 2008 02:01 AM   [ Ignore ]   [ # 32 ]
New Member
Avatar
Rank
Total Posts:  64
Joined  2008-02-14

hi ding,
sorry for my late answer but i’ve had lots to do wink

i’ve studied your code examples and tried to test everything but i don’t know if i’ve done everything right.

NSBitmapImageRepbitmap [[[NSBitmapImageRep alloc]  initWithCIImageendOfChainImage] autorelease];

 
// Getting pixels data
// this returns a pointer to the pixel data
 
int *data [bitmap bitmapData];

that seems to work but i don’t know the bitmap structure. if i try to track blobs in data there will be 0 blobs recognized.

- (void)captureOutput:(QTCaptureOutput *)captureOutput didOutputVideoFrame:(CVImageBufferRef)videoFrame
      withSampleBuffer
:(QTSampleBuffer *)sampleBuffer fromConnection:(QTCaptureConnection *)connection;

i don’t know how to implement that method in my controller/model code. additionally, i haven’t been working with threads under os x, yet. thus you would have to tell me some more about how to use this delegate.

- (IplImage*) iplImageFromCIImage:(CIImage*) image
{

}

that definitely works. but it’s sloooooooooooooooooooooooooooow. and i can’t see the results in my QTCaptureView view (i haven’t used NSOpenGLView so far).

actually, i’ve come to the decision that a native cocoa multi-touch implementation would surely be nice but then everything needed to be written in cocoa. switching between cocoa and opncv is deadly. too much bit-shifting -> bad performance.

either someone comes up with a CoreImage filter that takes endOfChainImage and computes the blobs natively out of the CIImage (or whatever bitmap structure that is) or we should try to focus on integrating e.g. OpenCV and its corresponding filters into a Cocoa GUI. for the latter that would mean to capture the cam’s image directly e.g. with OpenCV’s cvQueryFrame (camera), show the captured IplImage inside of e.g. an NSOpenGLView, influence the image with OpenCV filters that are configured via Cocoa sliders and compute the blobs with OpenCV’s blob lib. then we would stay consistently inside of one framework and the performance shouldn’t be too bad.

anyway, we should use a/the code repository in order to work on the same code!!!

 Signature 

_________________________________________________
my personal blog: johannesluderschmidt.de
our project blog: digitalvolvelles.wordpress.com

Profile
 
 
Posted: 16 May 2008 09:28 AM   [ Ignore ]   [ # 33 ]
New Member
Rank
Total Posts:  59
Joined  2007-07-09

What do you mean by slow.  Have you timed the process using system time. What is the time of conversion in milliseconds?

By the way here are the docs for your questions.

http://developer.apple.com/documentation/QuickTime/Reference/QTCaptureVideoPreviewOutput_Class/Reference/Reference.html#//apple_ref/doc/uid/TP40004570-CH3-DontLinkElementID_1
http://developer.apple.com/documentation/QuickTime/Conceptual/QTKitCaptureProgrammingGuide/Introduction/chapter_1_section_1.html

Don’t use a QTCaptureView because it only shows you what you are capturing use an Custom NSView and link it i IB.
Are you using QTCaptureDecompressedVideoOutput?

Google using @synchronized in video capture and threading. All this stuff is kinda long to get into in here thats why I’m posting links.

Dumb ramblings below___________________________________
I think from what you said maybe you could stick with touchlib and build a cocoa gui around it. You would be better off because all of the hard work is already done for you. Or use the portVideoBlobDetection project and build an app that listens to the tuio messages it sends. Or even use portVideoBlobDetection as the base for your app.

I think the idea is to try and get a true objective-c/cocoa framework/app that works natively with little or no c++. Most if not all of opencv is straight C so that combines perfectly with objective-c.
I think your idea of a Core Image Filter for blob tracking is a good one.
______________________________________________________
Anyway enough of my ramblings.

If you use the delegate above instead of the current ciimage capture delegate you are using you can try converting the CVImageBufferRef to an IplImage using:

CVPixelBufferLockBaseAddress();
CVPixelBufferGetBaseAddress();
CVPixelBufferGetBytesPerRow();
CVPixelBufferUnlockBaseAddress();

but then you would have to stick with opencv filters.  I think CIImage filters are much cooler and easier to use. Also OpenCV’s cvQueryFrame (camera) is a bit slower than native QTKit capture.

BTW what code repository?

Profile
 
 
Posted: 16 May 2008 01:04 PM   [ Ignore ]   [ # 34 ]
New Member
Avatar
Rank
Total Posts:  29
Joined  2008-05-15

i want help
rasberry

 Signature 

DAEMON IS’NT A DIGIMON ¬¬

Profile
 
 
Posted: 16 May 2008 01:17 PM   [ Ignore ]   [ # 35 ]
New Member
Avatar
Rank
Total Posts:  64
Joined  2008-02-14

@ding

code repository means a central point where people put their code like cvs or svn (sourceforge or googlecode). thus every part of a project can access, review, change and submit code to the repository. we don’t necessarily need to develop 2 different versions but work on one…

yeah, if you want to stick with purely Cocoa and have a performant app a CoreImage filter that does the tracking would be nice… no conversions between different bitmap formats.

 Signature 

_________________________________________________
my personal blog: johannesluderschmidt.de
our project blog: digitalvolvelles.wordpress.com

Profile
 
 
Posted: 16 May 2008 02:26 PM   [ Ignore ]   [ # 36 ]
New Member
Rank
Total Posts:  59
Joined  2007-07-09

Ok, well.  Have you checked out openframeworks?

http://www.openframeworks.cc/faq

I think maybe a good place to start might be to work on an application imbedded framework that is simple like “of” but cocoa based. I love what the people from openframeworks are doing.  I am almost tempted to drop cocoa and just stick with “of” and c++ but there is a lot of cool features I would miss from cocoa/objective-c. check this out:

http://developer.apple.com/documentation/MacOSX/Conceptual/BPFrameworks/Tasks/CreatingFrameworks.html

What do you think?

Ding

Profile
 
 
Posted: 17 May 2008 10:12 AM   [ Ignore ]   [ # 37 ]
New Member
Avatar
Rank
Total Posts:  38
Joined  2008-01-31

Does anyone know if nuiman’s framework is actually the OS X operating system or a wrapper that looks like OS X running on OS X?

Never mind. I miss read the article.

Profile
 
 
Posted: 17 May 2008 11:36 AM   [ Ignore ]   [ # 38 ]
New Member
Avatar
Rank
Total Posts:  64
Joined  2008-02-14

you can’t actually see if nuiman’s multi-touch app is running on os x or not. anyway, it looks like it was created with openframeworks. i just downloaded http://www.openframeworks.cc/versions/preRelease_v0.05/of_preRelease_v0.05_xcode_FAT.zip and tried out their examples. especially the app in apps/addonsExamples/opencvExample is very interesting. i adjusted the app to my needs (use a camera, lower threshold) and it’s fast as hell. if someone wants to test my adjusted mac os x openframeworks blob tracking app (changed 2 lines of code in the example…): http://www.vildan.de/app/openFrameworks.app.zip

additionally there’s another app called oscSenderExample in the same folder. you just have to combine both examples and voila: you have a very fast, native mac multi-touch implementation!

come on nuiman: show us your app!

 Signature 

_________________________________________________
my personal blog: johannesluderschmidt.de
our project blog: digitalvolvelles.wordpress.com

Profile
 
 
Posted: 18 May 2008 09:12 PM   [ Ignore ]   [ # 39 ]
New Member
Rank
Total Posts:  59
Joined  2007-07-09

So lagerkoller,

Im guessing you are liking of.  Are you interested in a cocoa framework similar to of? I guess you can always use cocoa as a gui as long as you write in objective-c++.
Or just use glui- http://www.cs.unc.edu/~rademach/glui/
I think the of team is thinking of integrating glui in a future release.
If you are sticking with of I think it would be worthwhile writing an ofITouchListener addon.

Ding

Profile
 
 
Posted: 20 May 2008 09:29 PM   [ Ignore ]   [ # 40 ]
New Member
Rank
Total Posts:  11
Joined  2008-05-06

hi, ive been trying multitouch wiht openframework for mac. it does the blob detection and using the as3 classes from touchlib to build apps in flex. and it works great. the only thing im missing is e mouse driver for mac. does any knows how to build a virtual mouse on osx to catch multitouches and sendit to osx?

Profile
 
 
Posted: 21 May 2008 05:58 AM   [ Ignore ]   [ # 41 ]
New Member
Rank
Total Posts:  1
Joined  2008-05-21

sounds good lseba

is there anychance you could upload a zip of your framework and as3 classes with maybe a flex working example?
i dont have the coding knowledge as of yet to get it going myself, and am sick of using touchlib on my old pc when i have a new macbook

thanks

Profile
 
 
Posted: 21 May 2008 08:27 AM   [ Ignore ]   [ # 42 ]
New Member
Rank
Total Posts:  11
Joined  2008-05-06

ill get all together and rapidshare it

Profile
 
 
Posted: 22 May 2008 02:43 PM   [ Ignore ]   [ # 43 ]
New Member
Avatar
Rank
Total Posts:  38
Joined  2008-01-31

Does the developer for openframework participate in this community? I was reading over the code for it’s video capture class for the Mac, and I have a few questions about his approach.

Profile
 
 
Posted: 22 May 2008 08:53 PM   [ Ignore ]   [ # 44 ]
New Member
Rank
Total Posts:  4
Joined  2008-05-18

@lagerkoller
I tried your compiled modifications: very nice!!
Could you write exactly what code you modified?
+ I need this to work with the second video device as the first one is the laptop’s internal isight.... would this be easy to set?

Thanks

Ok, found solutions myself:
1. modification to use webcam: uncomment the line for webcam in testApp.h
2. call videoSettings and choose cam wink

I made a video of the CvExample: it indeed is good and fast!!

http://web.mita.keio.ac.jp/~mh070504/of-cvexample.mov

Profile
 
 
Posted: 23 May 2008 12:32 PM   [ Ignore ]   [ # 45 ]
New Member
Rank
Total Posts:  11
Joined  2008-05-06

still trying to send events to osx.
i ve some progress. ive managed to send events to system. move the cursos and press and release button. but when i exit the app i get this error code that worries me:
“openFrameworks(1283,0xa0695fa0) malloc: *** error for object 0xbffff0e0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug

The Debugger has exited with status 0.”

here is my code

#include “testApp.h”

void testApp::setup() 
ofSetFrameRate
60 ); 
cwidth 320
cheight 240
threshold 60
mouseon=false
bLearnBakground true

TUIOSocket.setupHOSTPORT ); // Set in Header

vidGrabber.initGrabbercwidthcheight ); 
colorImg.allocatecwidthcheight ); 
grayImg.allocatecwidthcheight ); 
bgImg.allocatecwidthcheight ); 

blobTracker.setListenerthis ); 
alivecounter=0
}

void testApp
::update() 
ofBackground
100100100 ); 
vidGrabber.grabFrame(); 

if( 
vidGrabber.isFrameNew() ) 
colorImg 
vidGrabber.getPixels(); 
colorImg.mirrorfalsetrue ); 
grayImg colorImg

if( 
bLearnBakground 
bgImg 
grayImg
bLearnBakground false
}

grayImg
.absDiffbgImg ); 
grayImg.blur11 ); 
grayImg.thresholdthreshold );

//findContures( img, minSize, maxSize, nMax, inner contours yes/no ) 
contourFinder.findContoursgrayImg10,2000010false ); 
blobTracker.trackBlobscontourFinder.blobs ); 


}

void testApp
::draw() 
ofSetColor
0xffffff ); 

colorImg.draw20,20 ); 
// grayImg.draw( 360,200 ); 

blobTracker.draw20,20 ); 
string strhelp="[space] to learn background\n[+]/[-] to adjust threshold: "
ofToString(threshold2)
+
" \n[m] to toggle system mouse events:"
+ofToString(mouseon); 
ofDrawBitmapStringstrhelp20,300 ); 

ofxOscMessage m2
m2.setAddress/tuio/2Dcur” ); 
m2.addStringArg“alive” ); 
for (
int i 0blobTracker.blobs.size() ; i++)
// ofCvTrackedBlob blob = blobTracker.getById( id ); 
ofxOscMessage m1
m1.setAddress"/tuio/2Dcur" ); 
m1.addStringArg"set" ); 
m1.addIntArgblobTracker.blobs.id ); //id (id can’t be == 0) 
m1.addFloatArgblobTracker.blobs.center.x/360 ); // x/camWidth 
m1.addFloatArgblobTracker.blobs.center.y/240 ); // y/camHeight 
m1.addFloatArgblobTracker.blobs.deltaLoc.); //X 
m1.addFloatArgblobTracker.blobs.deltaLoc.); //Y 
m1.addFloatArgblobTracker.blobs.area ); //m 
m1.addFloatArg(blobTracker.blobs.box.width ); // wd 
m1.addFloatArgblobTracker.blobs.box.height );// ht 
TUIOSocket.sendMessagem1 ); 

m2.addIntArgblobTracker.blobs.id ); //Get list of ALL active IDs 


TUIOSocket
.sendMessagem2 );//send them 
alivecounterblobTracker.blobs.size(); 
}

void testApp
::keyPressedint key 
if( key == ‘ ‘ 
bLearnBakground 
true
else if( key == -’ 
threshold 
MAX0threshold-); 
else if( key == +’ || key == =’ 
threshold 
MIN255threshold+); 
}else if( key == ‘m’ && !mouseon 
mouseon 
true
}else{ 
mouseon
=false


void testApp
::mouseMovedint xint y {} 
void testApp
::mouseDraggedint xint yint button {} 
void testApp
::mousePressedint xint yint button {} 
void testApp
::mouseReleased() {}

//-------------------------------------------------------------- 
void testApp::setmouse(float xfloat y,int event )
CGPoint pt

pt.=x
pt.y;

switch(
event)
case 1://move 

CGPostMouseEventpt11); 
break; 
case 
2://press 

CGPostMouseEventpt11); 
break; 
case 
3://release 

CGPostMouseEventpt11); 
break; 
}





void testApp
::blobOnint xint yint idint order 
if(mouseon && alivecounter==&& order==)
blobmouse
=id
setmouse(blobTracker.blobs[order].center.xblobTracker.blobs[order].center.y,); 
cout << "blobOn() - id:" << id << " order:" << order << endl;

}


void testApp
::blobMovedint xint yint idint order
// cout << "blobMoved() - id:" << id << " order:" << order << endl; 



void testApp
::blobOffint xint yint idint order 
// 
if( blobmouse==id)

setmouse
(blobTracker.blobs[order].center.xblobTracker.blobs[order].center.y,); 
cout << "blobOff() - id:" << id << " order:" << order << endl;

}

}

void testApp
::blobAlivevector a {

}

Profile
 
 
   
3 of 6
3