This is my morning now, ha!! Sure I can help, but you have to figure out if you want to stick with nsimage or go with ciimage which might be easier as far as applying filters to it.

EDIT: I have been thinking and researching and seeing as though QTKit has no options for controlling the camera settings maybe the SG is the optimal way to go for now.  I think most people doing these kind of projects in os x will be using the fire-i. I personally don’t have one yet, I have a lame ass usb 1.1 with maccam for my tests, so I have been using the QTKit and didn’t know the repercussions of it.

ding

Here is some helpful information:

1. I think QTKit is a great step forward, but it is not as stable as the SG framework. Yes it lacks the camera control function call, but I think you can achieve all the same control using function calls, no pretty dialog window. I also feel that there is a performance difference between SG and QTKit.

2. The best camera to use with the Mac is ironically the Xbox live camera. Once modified it is extremely sensitive to IR light. It also works natively with OS X and does not require maccam, which hinders performance. THE FIRE-I SUCKS!!!!! I was extremely disappointed with the results of that camera. I am under the impression that the crappier the camera the better it is for our project. The more expensive cameras are going to have a higher quality CCD. The lower quality CCDs will me more susceptible to IR light. I could be doing something wrong with the fire-i, but there is a HUGE difference between the images from the xbox and the fire-i camera. Plus the Xbox camera is cheaper ^_^.

3. I need to double check this one, but there are 3 main image classes in OS X: NSImage(cocoa), CImage(core image), and CGImage(carbon). Apple has this concept called “toll free bridge”. This concepts allows for certain carbon objects to be placed where documentation requires cocoa objects. I am 90% sure that NSImage and CGImage have this property. Also, do not quote me on this one either, I think CImages are not intended to be drawn. I think the entire purpose of the CImage is a data struct that can be manipulated by the CIFilters. Thus the NSImage and CGImage were created as a shuttle for displaying image data.

From what I understand NSImage uses small amounts of RAM but uses CPU for everything, CIImage uses GPU for calculations, saving the CPU power for something else but uses a lot of RAM. CIImages are intended to be drawn. This is what QTCaptureView uses to display the video. Now some people can argue about which type is faster to draw but thats another story.

Here is the reference.

Delegates of QTCaptureView can implement this method to modify the image that is to be drawn into a QTCaptureView.
- (CIImage *)view:(QTCaptureView *)view willDisplayImage :(CIImage *)image

view
A QTCaptureView object that identifies the view which is about to draw.
image
A CIImage object that represents the frame that will otherwise be drawn to the QTCaptureView.
Return Value
Delegates should return a CIImage object to be drawn by the capture view, or NIL if the capture view
should draw the original image.
Discussion
The image parameter is a CIImage representing the captured frame that is about to be drawn into a
QTCaptureView. The delegate can return another image that modifies the source image (by applying
a CIFilter, for example). The returned image will then be drawn into the capture view instead of
the source image. The delegate can also return nil or the original image to leave the drawn image
unmodified.

I personally like to use the QTCaptureDecompressedVideoOutput and use:

- (void) captureOutput: (QTCaptureOutput*) captureOutput
didOutputVideoFrame: (CVImageBufferRef) videoFrame
withSampleBuffer: (QTSampleBuffer*) sampleBuffer
fromConnection: (QTCaptureConnection*) connection

and just use the CVImageBufferRef.

ding

My guess is that behind the scenes, the CIImage gets converted to an NSImage before hitting the view or the views are based on OpenGLViews which I think would allow them to dump the pixel data and some other info directly into the view and have it display. I haven’t work with the QTKit stuff that much yet so I did not know about it using CIImages in that manner. I was going completely off of the descriptions of the objects:

CIImage:

The CIImage class represents an image. Core Image images are immutable. You use CIImage objects in conjunction with other Core Image classes, such as CIFilter, CIContext, CIVector, and CIColor, to take advantage of the built-in Core Image filters when processing images. You can create CIImage objects with data supplied from a variety of sources, including Quartz 2D images, Core Video image buffers (CVImageBufferRef), URL-based objects, and NSData objects.

Although a CIImage object has image data associated with it, it is not an image. You can think of a CIImage object as an image “recipe.” A CIImage object has all the information necessary to produce an image, but Core Image doesn’t actually render an image until it is told to do so. This “lazy evaluation” method allows Core Image to operate as efficiently as possible.

NSImage:

An NSImage object is a high-level class for manipulating image data. You use this class to load existing images or create new ones and composite them into a view or other image. This class works in conjunction with one or more image representation objects (subclasses of NSImageRep), which manage the actual image data.

It would make sense for the QTKit to use CIImages because it does not waste time or RAM space by rendering the image all of the time. It lets it live out in VRAM until it is need in the system. Now this has some give and take to it. You either take a hit by doing all of the calculations with RAM and the CPU or by doing everything on the GPU and having the bottleneck of transferring the data from VRAM to RAM.

This is why a GPU based blob detection would be very nice. It would take lest time to move the resultant blobs from VRAM to RAM than it would take to move an entire 640x480 or 320x240 image.

It also makes sense for QTKit to use CIImages because it takes CVPixelBuffer in one of constructors. This is a common by product when dealing with video processing, even with SG.

Makes sense.

Can someone with an intel mac post bench marks while running this code(CPU and RAM Usage)?

Just had to thank you for your works Panda… really a great addition to our ever growing project

@mac daddy I have a intel mac i can test with… I am going to try and compile the SVN code tommorrow and I’ll try to post my results.

NUI GROUP 4 LIFE

Just put more code up on the repository for anyone interested.

code is here:
http://code.google.com/p/opentouch/

brief notes are here:
http://benbritten.com/blog/2008/06/01/more-new-code/

cheers!
-panda

Hey All,

just had a chance to read over all the info on the various image formats, very interesting! so thanks to all for that info everyone.

MacDaddy, did you get the fire-I with the no-ir-coating lens?  i found it to be way plenty sensitive, even to the crappy amount of IR I am projecting onto my surface, and even with the crappy 35mm film-stock reject visible light filter i am using. I got the OEM board version, have been very happy with it.  Anyhow, just curious.

as far as image format goes: I do like the CIIimage just because they have all the image filters available if you need them for your set up.. and i think it would be fairly trivial to make the blob detector into a filter. (much like others have done in the other projects) Obviously the code i wrote is all NSImage based, but really there is very little NSImage specific stuff in there, it is mostly just dealing with the bitmap buffers.  Later this week i will look into doing a CIImage version (unless one of you beats me to it).

I also think it would be fairly easy to do a qtKit drop in replacement for the SG stuff in my code as well.  The only real interface requirement is that the output needs to be an 8bit NSImage (until any CIImage stuff is added) and since you probably have to convert from an ARGB chunky camera raw source to get an 8bit image, then stuffing it into an NSImage is really pretty easy.  (just as easy as stuffing it back into a CIImage)

I forsee in the future, the blob detector could be a nice robust object that allowed for say [myDetector analyzeNSImage:] as well as [myDetector analyzeCIImage:].. analyzebitmap, analyzeArrayOfIntegers...whatever. (and also act as a CIIfilter).  everything eventually boils down to a big list of unsigned chars anyhow, so it really matters very little what the source of those values are. 

In my current implementation, images go in and tracking info comes out, so the images are never meant to be drawn. I know most of the new core image stuff is optimized for being eventually drawn, so I am not sure how much of that optimization is lost on us.  anyhow, just thinking out loud here really.

Hey MacDaddy, as far as the qtKit goes, you say that you think you can achieve all the hardware control via function calls?  I so hope that is true, i couldn’t seem to find it tho.  for my purposes a way to simply lock in the current setting would be basically all I would ever need (ie let the smart algorithms figure out the best quality, then just lock that in so that they dont change with the slight changes as i add blobs to the image.) I will have another look at the docs, but if you have some more info i would love to know
And if not, I wonder if Pawel or anyone else who is going to be at WWDC this year can talk to the apple guys to see if there is any movement toward the hardware level control in the qtKit.

Also, i think i should mention that the code i added today is a bit rough around the edges.  (i think that the sampleEventView doesn’t actually do anything yet) I am going to be gone for a few days and just wanted to get the new stuff up in case you guys wanted to play with it or had more great ideas One thing you can do is see the blob tracking info in the log window, which is anti-climatic, but it is something

Cheers!
-panda

Here is my inventory from my order:

QTY Product Unit Price Total
--- --------------------------------------------- ------------ ------------
1 Fire-i Board Camera Color $105.00 $105.00
1 Power supply adapter 12V $19.95 $19.95
1 Fire-i camera tripod kit $12.00 $12.00
1 Plastic housing for Fire-i board camera $9.95 $9.95
1 4.3mm Standard Lens (no IR coating) $19.00 $19.00

They could have sent me the wrong lens maybe?

Hey MacDaddy,

Yeah, i guess it is possible they gave you the wrong lens.  what you list is basically exactly what i have.  I dont know if there is a good metric for how ir-sensitive the camera is so i could show you/tell you, but in a dark room (ie very little ambient IR) if i point the camera at a white wall about 10 feet away, and i point my apple remote at the same wall, i can clearly see the IR spotlight on the wall.  (even tho it is pretty diffuse at that distance.) This was without the visible light filter. I dont think i tried it with the visible light filter on. 

if you can come up with a scene that you think would work for comparison, i am happy to pull mine out and get a photo for you to compare to.  (tho not for a few days, since I am not back home until later this week)

Cheers!
-p

and with the draw kit for vector works ?

it maybe can help

Hey all, just wanted to let anyone who is interested know that I have put up a bunch of code changes and updates.  There is now a working configurator, and all the basic building blocks for custom multi-touch cocoa apps. I still need to make a full-screen configure thing so that you can easily make the blobs fit your surface but you can always hardcode that for now. (i have also started work on a CIImage drop in converter for the blob detector so you will be able to use the CIFilters on the raw images)

code is here:
http://code.google.com/p/opentouch

notes are here:
http://benbritten.com/blog/category/multitouch/

cheers!
-p

You guys have made me happy, all I need to do is start learning to dev on the Mac.  I have bought some books to help me start.

If I can help in any way let me know, I have pretty much every Mac released in the last 8 years at my disposal for testing.

I do not see any code to download at the link provided.