This thing is really interesting! Its all 3D vision based really interesting stuff. Seems like there is some decent intelligence and or marketing fluff from Microsoft … either way great vision hope it works and evolves… Ohh yea and looking forward to hacking it!
Compatible with any Xbox 360 system, the “Project Natal” sensor is the world’s first to combine an RGB camera, depth sensor, multi-array microphone and custom processor running proprietary software all in one device. Unlike 2-D cameras and controllers, “Project Natal” tracks your full body movement in 3-D, while responding to commands, directions and even a shift of emotion in your voice.
In addition, unlike other devices, the “Project Natal” sensor is not light-dependent. It can recognize you just by looking at your face, and it doesn’t just react to key words but understands what you’re saying. Call a play in a football game, and players will actually respond.
Yeah, thanks for the post! Now how do I catch up because I wanna hack it too!!! The possibilities are a mathematical equation I can’t pronounce. But unlimited is a good start.
I found a link regarding the tech behind the controller > TOF (Time Of Flight)
Lies!! “Product vision: actual features and functionality may vary”
Seriously, if they manage to make half of what is shown work decently this would be a major technology breakthrough.
I kinda hate, though, how they try to attract kids by showing cool stuff that is clearly impossible to do.
Thanks for the heads up, Christian.
Any info on what’s inside the device?
I have tried a motion controller for a driving game before. It had gesture support and stuff like Natal. The tracking was no where near as smooth as shown in the video but this was done by a smaller company.
This tech all works with the 3D vision cameras they are using that can resolve a depth matte based on what it sees. I think this system uses two of the cameras to deal with occlusion. The scene with the dress is total bullshit… unless every dress designer starts putting 3d models of their dresses into a Microsoft database. And the scene with the skateboard… if it takes a picture of the board and uses that then there is slight issue of the fingers in the way, this could easily be fixed with some good algorithms and two pictures taken with different finger placement but this video makes it look to damn simple. The TV in the video is clearly replaced and faked.
@Cris, that’s rather cool boss! Did you take it any further?
I did. I create a piece for the AdAge Dinner in NY last year to demo some of my companies work. It was very similar to the new xBox demo with Skateboard decks and the Netfilx Video player at the end. But i really never got a chance to do anything really exciting with it.
I’m excited to hack this and see what awesome things we can do with it.
since its not shipping, who knows how good it really works or when it will actually arrive. on the other hand, they have chosen to play their cards to generate a lot of press and pique our curiosity. if it can be done, its of great interest many in this group. lets find out how.
Ha ha ! Johnny lee (who did low cost touch table with wiimote ) is a team member of Project Nathal . Cool !
Anyway , I wonder that if Microsoft want to take cover all licenses on Time of Flight tech ? I know that Microsoft bought 3DV (one small company has Time of flight tech) http://en.wikipedia.org/wiki/ZCam . Big M$ also bought licensees from GestureTek on Time of flight tech , in this webpage http://www.gesturetek.com/3ddepth/introduction.php , look at demo , it is the same Project nathal , exactly , and scrool to the bottom of the webpage you can see EYETOY and XBOX360 logo . So , what can I say ? I think sony bought licensees for ps2 eyetoy , but that gesture tech version is the old one, just 2D gesture . In the case of project Nathal , It can sense 3d object , so this gesture version is the new version . And I think big M$ bought exclusive licensees for long term (maybe 10 years ..) , adn they also bought 3DV company . So, Micrsoft will be the winner in next generation console war . Oh man ! who know . Sony and Nintendo can’t purchase Time Of Flight tech due to exclusive licensees .
This is how 3d sensor chip capture the light and produce the pixel to render 3d model :
Most people understand that light takes a finite time to travel between two points—that photons of light from two different stars, for example, may have started their journeys years, or even millennia apart. Since light travels essentially at a constant speed, if you know the time, you can calculate the distance.
The light illuminating each individual pixel in an image sensor comes from a different feature in the scene being viewed. Canesta recognized that if you could determine the amount of time that light takes to reach each pixel, you then could calculate with certainty the exact distance to that feature. In other words, you could develop a three-dimensional “relief” map of the surfaces in the scene. In three dimensions, objects previously indistinguishable from the background, for example, metaphorically “pop” out. For a broad class of applications, this proves extremely helpful in reducing the mathematical and physical complexity that has plagued computer vision applications from the start.
In a recently-granted U.S. patent, Canesta describes several of its inventions for “timing” the travel time of light to a unique, new class of low-cost sensor chips.
Fundamentally, the chips work in a manner similar to radar, where the distance to remote objects is calculated by measuring the time it takes an electronic burst of radio waves to make the round trip from a transmitting antenna to a reflective object (like a metal airplane) and back. In the case of these chips, however, a burst of unobtrusive light is transmitted instead.
The chips, which are not fooled by ambient light, either then time the duration it takes the pulse to reflect back to each pixel, using high speed, on-chip timers in one method, or simply count the number of returning photons, an indirect measure of the distance, in another.
In either case, the result is an array of “distances” that provides a mathematically accurate, dynamic “relief” map of the surfaces being imaged. The image and distance information is then handed off to an on-chip processor running Canesta’s proprietary imaging software that further refines the 3-D representation before sending it off chip to the OEM application.
Lies!! “Product vision: actual features and functionality may vary”
Seriously, if they manage to make half of what is shown work decently this would be a major technology breakthrough.
I kinda hate, though, how they try to attract kids by showing cool stuff that is clearly impossible to do.
Thanks for the heads up, Christian.
Any info on what’s inside the device?
gsoto,
Frankly, almost all of what’s shown in that video already exists.
The karate and monster parts are totally doable and 3DVS has a boxing demo that’s pretty similar, http://www.3dvsystems.com/gallery/gallery.html
The driving part has already been demoed at E3 with Burnout
The football part is just the same as the karate/monster part but it’s tracking two players instead of one (and the tech right now is supposedly able to track 4 players so 2 should be no strech)
The skate scanning part has been demoed at E3 with the Milo project (you can draw to Milo and have it scanned by the webcam - of course that the blue highlight in the promotional video is total bs)
The dress part is there to highlight facial recognition and Xbox Live automatically logging in after it sees your face (and facial recognition will be greatly improved due to the the combination of a 3D + 2D camera as it will not only rely on a “photo” but will also analyze facial features, eg. nose size)
The 1 vs 100 part will highly depend of the ability of the system to properly track multiple players
The movie part, browse through menus and use gestures to control, is already doable and the tech has been demoed at E3
Afaik, what’s inside is merely a IR cam, IR light source, HD webcam and multiple microphones. Plus the hardware to make it all work like magic
tetsu81vn - 03 June 2009 03:44 AM
Anyway , I wonder that if Microsoft want to take cover all licenses on Time of Flight tech ? I know that Microsoft bought 3DV (one small company has Time of flight tech) http://en.wikipedia.org/wiki/ZCam . Big M$ also bought licensees from GestureTek on Time of flight tech , in this webpage http://www.gesturetek.com/3ddepth/introduction.php , look at demo , it is the same Project nathal , exactly , and scrool to the bottom of the webpage you can see EYETOY and XBOX360 logo . So , what can I say ? I think sony bought licensees for ps2 eyetoy , but that gesture tech version is the old one, just 2D gesture . In the case of project Nathal , It can sense 3d object , so this gesture version is the new version . And I think big M$ bought exclusive licensees for long term (maybe 10 years ..) , adn they also bought 3DV company . So, Micrsoft will be the winner in next generation console war . Oh man ! who know . Sony and Nintendo can’t purchase Time Of Flight tech due to exclusive licensees .
tetsu, your assessment is pure fud. We know for sure that GestureTek licensed their tech to more than MS (Sony itself is included with their EyeToy), so no need for irrational fear, ok
Very exicting possiblities for hacking though. Full body motion capture for a (lower) price without any kind of sensors? This could be very good for animators.
… either way great vision hope it works and evolves… Ohh yea and looking forward to hacking it!
