NUI Group - Open Source Interface

[GSoC 2011 Proposal] Continuous Musical Fingerboard for Kivy

2011-04-01T08:30:38-05:00

edit:
I’ve finished the proposal document, it’s here: http://dl.dropbox.com/u/16643844/gsoc2011.pdf

Abstract:

This document details the proposal of a project to the NUI Group (Natural User Interface Group) for the Google Summer of Code, 2011 edition. The project’s goal is to create a musical instrument using the multi-touch application development framework called Kivy, specifically an interface widget implementing a keyboard-like and continuous instrument. The implementation will be taking advantage of the flexibility and ease of use provided by a graphical and natural user interface environment for music production. The widget will output a message format which is used for controlling electronic music, such as MIDI (Musical Instrument Digital Interface) or OSC (Open Sound Control). A sound module will receive this data and produce the resulting sound. This sound module will be implemented and included out-of-the-box and developed in a technology called PureData. The motivation behind this project is the study of the applicabilities of a multi-touch interface for music production and the implementation of an innovative instrument with great potential, but not popular due to its high cost. The future work for this project will involve adding more features to the interface, improving the latency of the setup and adding additional effects/samples to the sound module. It is also a possibility to use the sound module for controlling for example graphics or video, to allow for a visual experience in addition to just the audio.

-------------------------------------------------------

Name: Rui Reis Costa Campos
Email:
Location/Timezone: Porto, Portugal/GMT
Age: 22
Education/Qualifications: Student of the Integrated Masters of Informatics and Computing Engineering in the University of Porto, Portugal ( 5th year)
Project Proposal:
Hello, I’m currently in my final year of my degree and I’m working on my master’s thesis. My master’s thesis has the title of “Framework for Role Playing Games in Multi-Touch Multi-User devices”. In the past year my interest has spiked towards the area of interaction mainly due to the opportunity of making this thesis. I have worked with PyMT for most of the development of my thesis and am now switching to Kivy.
Joined with my recent interest in HCI is my music hobby which gave me an idea for a side project. This idea is to develop a platform for controlling midi via touch but most importantly, to implement a system similar to the Haken Continuum (http://www.youtube.com/watch?v=Mrmp2EaVChI&feature=related). The project also includes implementing a series of algorithms to do things like pitch rounding or automatic pitch sliding between consecutive contact points, see http://www.youtube.com/watch?v=yCM-WBqDZ-Q for a few of the cool features that can be implemented.
Ever since I saw this instrument being played I had the impression that it would be the next generation keyboard, because the possibilities are tremendous. However, it has not seen widespread adoption mainly due to the fact that it’s quite expensive.
So my project proposal is to implement an Haken Continuumish system with Kivy and a set of midi controls all touch based. My implementation will have three main tasks:
Implementation of the Interface:
This represents the interaction component where I will implement the continuous keyboard and various controls for the keyboard. I already know how work with PyMT/Kivy so I expect this task to be the less time consuming. One week for an initial prototype so I can start working on the other modules and two weeks spread along the other tasks for improving the user experience and some additional controls that may arise from the rest of the development.
Touch information treatment and sound production:
This is the component which will be responsible for treating the touch information received and generate the sound. As TUIO is based on OSC maybe it would be possible to simply send the TUIO directly to synth software and implement the parser of the touch information in the synth directly, but I’m afraid that this might restrict the implementation to that specific software. The analysis of these tradeoffs along with the fact that this is the module where I have less know-how( I will learn about OSC and how the synth software works) ,I consider that this is the task which will be the most time consuming and thus predict an estimate of 6 weeks for completion.
Optimizations for sound production:
Here I will see how much latency the current system has and try to optimize it using code optimizations, setup analysis, etc. I know that there are some proposals for using the GPU and for using regions of interests, the latter is especially interesting for me as the main interaction will be done in a specific region of the display and there may be a considerable performance gain in this; I will address this in the end of the implementation time so there may be some more advancements then. I expect to spend 2/3 weeks addressing this issue.

I believe that this project will be beneficial for the community given that we can bring even more people from the music community to the NUI field of research. The main problem with this kind of system is the latency, I consider it the biggest challenge for this project to be realistically used by musicians, however, I hope that this project can arise more interest in this area and since it is a field which is constantly evolving, the advancements in the processing area will eventually make this project a real success.
Bibliography/Inspirations:
http://www.cerlsoundgroup.org/Continuum/
http://opensoundcontrol.org/sensor-gesture-based-electronic-musical-instruments
http://www.jazzmutant.com/lemur_gallery_videos.php
http://www.livelab.dk/tablet2midi.php

New in world of Kivy

2011-12-21T08:18:10-05:00

Hey everyone,

I am totally new in Kivy. I made some applications for Multi Touch using ActionScript 3.0. But, now I want to switch to Open Source. So, I want to learn Kivy. So any one could tell me any good tutorial or any suggestion for newbie in Kivy ??

I also found API, but that’s not enough. Please reply.

[GSoC 2011 Propoal] Kivy Arcade

2011-03-31T17:35:55-05:00

Name: Petr Viktorin

E-mail:

Location/Timezone: Europe: Finland, Czech Republic

Age: 24

Education/Qualifications:
- Bachelor’s degree in computer science from Brno University of Technology, Czech Republic
- Currently a second year of Master’s student at the Univerity of Eastern Finland

Academic and Industry Background: I have built a FTIR touch screen as part of my Master’s project, and I’m planning a thesis on multi-touch games. (I have built a proof-of-concept PyMT game for this, but the code is not what I’d call open-source quality: https://github.com/encukou/touchgames)

Open source development experience:
- Lastscrape GUI (http://encukou.github.com/lastscrape-gui/)
- Ongoing internationalization for veekun.com
- Translation and some patches for Tiled (http://www.mapeditor.org/)

Development Methodologies: Depends on project; for GSOC I’d go with a fixed plan-prototype-evaluate-refactor spiral.

Project Proposal: Kivy Arcade

Implement a collection of simple, fun games for a multi-user multi-touch table.

1. Create an arcade framework:
- Plugin system for games
- Game selection
- High score system
- Common UI for pausing/quitting a game
- Displaying rules/instructions

2. Create games:
- Maze: A two-player game where one player solves a maze while the other modifies the maze. (See demo video at http://www.youtube.com/watch?v=96pQ0xqHkJQ)
- Towers: A two-player “Tower Defense” style game: the user can build as many towers at once as they have fingers, and also direct enemies at the opponent’s weak spots if they find some unused digits.
- Air hockey: Multi-touch classic with a twist—the “mallet” is normally circular, but with an un-pinch gesture it can be stretched into an oval, then a line, and then broken into two pieces.

These are multiplayer games that do not need touches to be explicitly associated to players (in Maze, each player uses different gestures; the other two will have a split screen). This makes them ideal for generic touch tables.

The project is intended to be merged into Kivy as examples, which means two things:
- The code will illustrate best Kivy and Python practices.
- The games will emphasize gameplay over appearance.
- Games will be kept as simple as possible (but no simpler); if more time is available it’ll go into additional games rather than additional game features.

Ultimate goal: The coolness of Kivy Arcade should drive new enthusiasts into building their own table, and getting into Kivy-based development!

The reason for picking specific project: As said above, I plan to analyze the games for my Master’s thesis. Also, I am a fan of Python, especially its “batteries included” philosophy that should apply to Kivy as well.
Activity level within the NUI Group Community: Recently registered. Lurking on IRC as EnCuKou.

[GSoC 2011 Proposal] MT4j network serialization

2011-03-31T12:32:42-05:00

Name: Marcos Paulino Roriz Junior
Email:
Location/Timezone: Brasília (-3)
Website: http://marcosroriz.wordpress.com
Age: 23
Education/Qualifications:
- I just started my Computer Science Master (Federal University of Goiás / Brazil)
- Bachelor of Computer Science (Federal University of Goiás / Brazil)

Academic and Industry Background:
As an undergrad, I worked in the brazilian national project GingaCDN, which aimed to build the official brazilian Digital TV middleware. More specifically I worked on isolating applications on the same JVM using context classloaders and making a communication bridge between procedural (Java) and declarative languages (NCL). My master’s topic is not yet defined, but will be in the area of mobility models/algorithms (+synchronization) in ubiquitous middleware. In the masters I’m working, on the university lab, at a research project from Dell, to build a middleware for ubiquitous applications for interactive classrooms. We received a bunch of Dell Multi-Touch Laptops . I participated in GSoC 2010 for the GNU Classpath project and completed, but in the end I felt extremely unmotivated due to a feeling that what I was doing would not server for much stuff.

Open source development experience:
- spotSHOUT (An RMI Middleware for Sun SPOT).

Project Proposal:

“Network serialization -> Create the possibility to send and share MT4j components across networks with other MT4j applications and implement a showcase application (e.g. document sharing, image sharing etc).”

On the research project, I did some demos applications on top of MT4j in order to find the difficulties of developing a ubiquitous application. This was done in other to gather the middleware requirements; that is, we wanted to find out what problems that are common on developing this kind of application and what of this problems/services we can abstract to the developers.

And one of the applications was made in order to understand the concept of mobility of objects between applications, since this is my focus area. The application basically allowed geometry shapes (represented as MTRectangle, MTPolygon, etc) to be transferred from one computer to another. But there is a problem there, the MTComponents are not serializable. So we did a “workaround” around this issue.

I hacked on MT4j and looked on the code of the MTComponent hierarchy. The problem is that we cannot just make the MTComponent implement the java.io.Serializable interface because it’s dependency are not serializable. So what that means is that if we want to every single field of MTComponent and it’s subclass needs to have their fields serializabled. And since MTComponent have many non-serializable fields such as OpenGL buffers, which are tied natively to the Operating System, they break the “normal” serialization process.

What I did as a workaround to this situation was to make a selective serialization, that is, instead of trying to serialize the entire component I will serialize its metadata, such as GeometryInfo and Vertex. I created a superclass on top of MTComponent called SerializableComponent that implemented the java.io.Externalizable interface. This interface allows us to control the serialization process (on both read/write data), that is, we can now serialize just the fields that we want. And I created a annotation called @Save and putted on the fields that I wished to save. I did a recursively algorithm on the serialization mechanism to look for every field from the child class (ex: MTRectangle) until SerializableComponent that has the annotation and save it on the buffer. The same logic is used on the reading the variables from the buffer on the other side.

With this I *can* serialize the metadata of the MTComponents but however I cannot just use them 100% directly. Why? I think it’s because they are tied with Processing variables (Papplet, buffers, etc) which I haven’t found a way to serialize. So what I do is create a fake object (such as a MTRectangle) and insert the metadata of the object received to make the fake object receive the personification of the original object (like colors, vertex, position).

As a proposal for the project I will continue my journey in the inside of the Java serialization mechanism specially on Processing side (Best way to make Selective Serialization?). And make it possible to transfer and restore easily 100% any MTComponent. Make a really cool app that use this mechanism.

Feedback: Please do not hesitate to criticize the proposal, and the mechanism proposed. Thank you in advance.

[GSoc 2011 Proposal] Fusion - A multimodal interaction framework based on CCA/CCV for task-oriented and object-oriented interfaces.

2011-04-07T02:05:16-05:00

EDIT: The full proposal, with diagrams, is available at http://shalstvedt.xen.prgmr.com/soc/

A good illustration of this concept done by someone else: http://www.youtube.com/watch?v=8EoLFNQ5JRQ

[GSoC 2011 Proposal] Fusion - An application framework based on CCA/CCV for multimodal interaction.

Name: Scott Halstvedt
Email: ,
Location/Timezone: Missoula, MT (GMT-6)
Age: 20
Education/Qualifications: 2nd year Computer Science student at the University of Montana (with a focus on Artificial Intelligence). I’ve been taking CS at a college level since 2007, and learning for far longer than that.
Academic and Industry Background: Studied artificial intelligence and CS at the University of Montana and briefly at the University of the Pacific, CA, USA. Good breadth of theoretical computer science classes with compilers, op. sys., architecture, graph theory, discrete math, etc. I’ve been working on speech recognition and synthesis for a year or so, starting with an Arduino and speech IC and lately I’ve been exploring CMUsphinx.
Open source development experience: Limited. I’ve written a couple of small patches for a few open source projects (mostly Roundup, a Python issue tracker that I modified slightly to use in an organization). I’ve long been a lurker behind major projects like WebKit, Git, and Linux; I read the mailing lists a lot and I’ve spent the last couple weeks checking out code and producing minor patches to get up to speed. I’m also fairly adept at version control.
Development Methodologies: I learned two methods: prototyping and, more recently, UML/formal FSM and event-driven FSM methods. I like formality.

Finding information, reading academic papers, and solving hard problems are my strengths.

Proposal: Fusion - A multimodal interaction framework based on CCA/CCV for task-oriented and object-oriented interfaces.

Abstract / Problem Description:
A good deal of new (and old) research on Natural User Interfaces discusses multi-modality as a critical component of natural interaction. Interface modes are singular input types like ASR, QWERTY text, stylus, multitouch, etc. A multi-modal interface processes two or more of these inputs simultaneously, allowing the user to specify information in the most comfortable way (i.e., it’s much more efficient to specify a 2d point in Cartesian space by touching it than by guiding an indicator by voice… ‘left, a little more left, now right… actually, just (245, 678).’ On the other hand, flicking through a list of cities in a travel app is less efficient than just saying where you want to go, ‘Chicago’. A multi-modal interface combines partial semantics from several input modes and generates a meaningful result, so the user can simultaneously say ‘move the yellow ball here,’ and also touch a point on the screen, to far more richly and easily express a complicated instruction, and in one utterance/gesture.

Helpfully, many applications with the potential to utilize CCA/CCV and other language technologies have a similar interface paradigm; that is, many of them can be defined as members of a class of object/symbol and (the recent reign of object-oriented paradigms in industry and academia helps here too). Members of a class of objects, and the methods (including required additional information) that are used to manipulate their state.

This proposal aims to use code from CCA/CCV (different API of course, and upgraded ASR engine), multitouch frameworks like Kiva and PyMT, and a new system that generates (simple, initially) NL grammars from (simple) object graphs and methods, to implement a multimodal interaction framework called Fusion. Fusion incorporates functionality from recent versions of CMU Sphinx to enable an interface which reacts to simultaneous vocal and tactile input and correlates the input into a unified semantic; i.e., resolves ‘here’ to a Cartesian point on a multitouch surface, or ‘this object’ to a selected object.

Implementation Plan:
1. Move CCA from sphinx3 (in C, stable but missing many useful features to post-process lattices and generate data structures meaningful to the application from a grammar and recognized input) to sphinx4 (Java, more modern, more maintained). Implement an API for more complicated grammar support (currently CCA just reads in a list of statements and generates a grammar that only contains the sentences in this list).
2. Implement a proof-of-concept grammar generator in CCA; takes simple serializations of objects and their methods and outputs a JSGF grammar including ECMAScript extensions (in Sphinx4) for all the operations on those objects. This can be translated into a finite state machine grammar that can be used directly in Sphinx.
3. Implement a multi-modal interface agent; this coordinates modular modes of input (CCA, CCV, Kiva, whatever) and shares the information necessary between them to derive meaningful semantics from information split between multiple modes.

Deliverables:
- A new, refactored CCA version that supports the latest version of Sphinx, Sphinx4. There is a language change involved in this(C->Java), but CCA currently doesn’t contain a whole lot of code other than Sphinx initialization and now would be the time to modernize. Also, I read that Jimbo Zhang and others were debating whether to stick with oF for the CCx projects in the future, and since Jimbo has proposed to move away from oF in CCV this summer, I propose we do the same with CCA as well. The new CCA also contains a (simple, for now) JSGF grammar generator that reads serialized object data and operations that can be done on objects (Various serialization/specification formats are under exploration).
- A (simple, for now) multi-modal interface framework that resides in the application and provides arbitration between the discrete input modes, the grammar generator, and the actual object model, and also an interface for the application to receive multimodal events. Also provides bindings for other languages (ex. Python for PyMT and Kiva)
- A demo application, probably written in Kiva, that allows a user to manipulate (move, recolor, create, delete) colored balls multi-modally. This demo application is expanded upon in the longer, more complete proposal.

Timeline:
May 23 - Jun 15 2011, Port CCA to Sphinx4 and begin adding features related to post-processing the result into a meaningful data structure
Jun 15 - Jul 15 Multi-modal application framework API and grammar generator
Jul 15 - Aug 22 Refine API and Demo Application implementation

[GSoC 2011 Proposal] CCV - Multiple Camera Support

2011-04-05T10:14:16-05:00

I build a new English blog for this year’s GSoC:
http://blog.meiyou.org/ —04/30/2011 Update
========================================
Full Version of The Proposal:
http://meiyou.org/test2011/GSoC/
========================================
Name: Yishi Guo

Email:

Location/Timezone: Changchun, Jilin, China/ UTC+8

Website: http://meiyou.org/ (Simplified Chinese)

Age: 23

Education/Qualifications:

- I’m a junior student majoring in Network Engineering in Changchun University of Science and Technology.
- I got the Cisco Certified Network professional certificate last year.

Academic and Industry Background:

I have coded more than 42,034 lines since entering the college. I’m specialized in C/C++, PHP, OpenCV and other image processing programming.

Because of my specialization in OpenCV, I recently obtained an internship position in a interactive media company as an internship. I am currently studying multi-touch projects and developing multi-touch based products there, and we have just brought out a 88’’ LLP multi-touch table.

Open source development experience:

* Switchnet – a software switches IP and Physical address according to different network environment.

* For CCV:
- Changed the CCV calibration order from “Z” to “S”: #12187;
- Found the method of getting the count of camera device: #12174;
- Optimized the code of DrawCircleLoader function: #12105;

Project Proposal:

* Problem Description:

CCV does not currently support multi-cam fundamentally.[1] So when confronted with largesize screen, users have to turn to a PS3 multi-cam license or choose a wide-angel lens that will result in a fish-eye effect. Obviously neither of them would be a good choice for most of us.[2]

So we’re desperate for implementing the multi-cam support for CCV.

* Theory:
Camera Calibration and 3D Reconstruction – OpenCV

* Process:

1. Use new multi-cam user interface to let users configure the number of cameras.
2. Select cameras in the optional device list and put them in order.
3. Run the calibration process and follow the simple directions by touching the calibration points.
4. Get parameters (camera intrinsics matrix and distortion coefficients) from previous calibration steps.
5. Use these parameters to undistort and rectify the images captured by cameras. Then stitch and align the multi-cam’s images into one image.
6. Deliver the source image, which was generated from the stitching and alignment process to CCV.

* Profit for NUIgroup/Deliverables:

- Provide advanced multi-cam support for CCV, which does not have the complication of the multi-cam stitching process and the limitation of wide-angle lens’ fish-eye effect.
- Make it possible to support a large size screen with the multi-cam support.
- User-friendly - all the user has to do is choose the camera devices and follow the directions then let the code do all the boring jobs.
- It’s cross-platform in implementation because of its functionality by OpenCV and OpenGL libraries.
- It’s free!

References:
[1] http://nuigroup.com/forums/viewthread/11901/
[2] http://nuigroup.com/forums/viewthread/9623/

Feedback:

I’m glad to receive your feedback.

[GsoC 2011 Proposal] Interface as a potential child development tool for the grandparents

2011-04-07T15:50:09-05:00

Abstract: We stand amidst the global innovation and industrial growth, where health and life expectancy are rapidly increasing. The 14% rise in the elderly population is expected from 2011 to 2014. With the culture of home dwelling elderly population increasing at such a rate, the psychological problems and mental stamina are seen to be affected. The project aims at building a touch based NFC device for the elderly as well as children population. Their interconnection and linkage is our key to psychological prosperity in the world. The core problem areas for the children(generally of age group 6-15) lies in their isolation while parents are off to work, their inquisitiveness to learn more, lack of attention span, not being so social etc. The elderly population however has problems areas like sensory and vision problems, technology fearing, loneliness, low memory etc. The problem areas when solved together can help us build a new inter generational linkage which can solve each of their problems individually and help in their mutual growth. The age old storytelling in the newer context, narrations , building up of emotional strength at early years, feeling of companionship, elderly care and many others problem statements can be solved by mere bridging the two groups of the society: children and elderly; with an interactive device so that the largest chunk of psychological issues can be tackled. ADHD, old age depressions, dyslexia, low mental activity are some of the key health problems that can be solved here; all by creating an NFC based mobile device for these user groups with the elderly population acting as a support system.

Those who say too Emotional, less practical:
Technology has never been viewed as solution to any emotional problems, but HCI and NUIs have the potent and thats why we study human ergonomics, psychology and design and engineering for.

Technical blueprint:
Near Field Communication (NFC) enabled mobile phone can be used as a user interface element and multi touch-based user interface in the everyday life activities of elderly users. The touch-based user interface is easier to learn and adopt for the children as well as the elderly regardless of their physical or cognitive weaknesses.
The MT4J framework would be used to devise the user modules and interactivity.

Name: Nishant Lodha

Location: Bangalore(India) +5:30 HRS ( GMT )

Email:

Age: 22 years

Educational Qualifications: B.Tech(Computer Science), have done projects in user research, local grievances control system for the Govt of India, mathematical toolkits for complex computations.

Industry Experience : Worked as a “Design Researcher” for the “Vaccine Delivery Innovation Initiative” project for The Bill and Melinda Gates Foundation.

Open Source Experience: Have been new to MT4J and processing frameworks.

Project Proposal:
The reason for picking specific project:
My motivation for using HCI and multi touch for solving various social issues and its potent to be beneficial to the developer and the user community both. The problem highlighted here has always been researched by scientists and leading technologists but has never floated to the top with other issues like global warming, recession, energy conservation, corruption, and poverty and health concerns. The solution to this issue will not only create well nurtured intelligent beings from an early stage but also be a remedy to old age psychological disorders and help in their well being. So thats how I would like to do my share by being socially responsible and technologically creative.

I have been working for around a year on multi touch and HCI and can truly state that this concept remains untapped and can help solve a variety of solutions for the masses throughout the world.

Please send your feedbacks and suggestions.

Bibliography:
http://www.leonardo-energy.org/webfm_send/273
https://dspace.lboro.ac.uk/dspace-jspui/bitstream/2134/3583/1/OAP-PIM6.pdf
http://www.bth.se/fou/cuppsats.nsf/all/d5dc37149f464156c12576be0058ecaa/$file/ATTUWQ58.pdf

[GSoC 2011 Proposal]

2011-03-26T09:20:24-05:00

[GSoC 2011 Proposal] Improve CCV - Support tracking skeleton by kinect .

2011-03-28T00:54:41-05:00

Abstract:
My proposal is developing a cross-platform kinect module ,which can track skeleton, send skeleton message by tuio. kinect

can only used in Xbox 360, So it is not convenient for other platform developers.Many people develop kinect driver such as

openkinect,but there is only raw depth image and color image.Microsoft give a paper which shows recognition human pose base

on Randomized decision forests.So in my proposal , the first is train a classifier to detection human pose,besides,base on

color image ,detection human by adaboost human face detection to init body position. The body part can be tracking by optical

flow and mean shift tracking method.As classifier give a probable pose,optical flow and mean shift tracking will give exactly

position. After that , Give a ID of every body part, send body part motion information such as position ,direction ,speed by

T, then Application can use body motion information just like multi-touch message.

Name: Tan Guofu

Email:

Location/Timezone: Beijing,China,UTC +8

Website: http://tanguofu.wordpress.com/
Age: 25

Education/Qualifications:

I am a third-year graduate student of Graduate University of the Chinese Academy of Sciences.

Academic and Industry Background:
My research direction is multi-touch table,We build a multi-touch table Base On LLP, I develop a algorithm to resolve block

problem in LLP ,by extracted touch region from raw image , keep the connectivity among region match them with the finger

which is being tracking.

I also realize a new calibration which can be seen my last year proposal.(http://nuigroup.com/forums/viewthread/9197/)

we develop a UI widget and a gesture lib base on WPF , the gesture lib can estimate finger orientation and pressure by

fitting ellipse of finger touch region, estimate touch hand by touch finger cluster and recognition multi-finger zoom

,pinch,trains gesture and strokes gesture by $1 recognition algorithm.

Project Proposal:
My proposal is developing a cross-platform kinect module of CCV, which can track body skeleton, send skeleton message by

tuio.

Firstly,Add openkinect lib to CCV,Grab the depth image and color image ,show them with CCV.

Secondly,Develop a train program, label human pose manually, train a randomized decision forests classifier with point cloud

library(PCL).In my Opinion,We use head, neck, L/R shoulder, LU/RU/LW/RW arm, L/R elbow, L/R wrist, L/R hand, LU/RU/LW/RW

torso, LU/RU/LW/RW leg, L/R knee,L/R ankle, L/R foot as body part,each part given a fixed ID.

Thirdly, base on color image ,we can detection human by adaboost human face detection to init body postion. The body part can

be tracking by optical flow and mean shift tracking method.As classifier give a probable pose,optical flow and mean shift

tracking will give exactly position.

finally, send body part motion information such as position ,direction ,speed by T, then Application can use body motion

information just like multi-touch message. I also will develop a demo application base on body part motion information

Timeline
This time is approximate time. I will give more time if the work is hard.

April 25 - May 1 :
Finish first, add openkinect lib to CCV,Grab the depth image and color image ,show them with CCV.

May 1 - May 15 :
be familiar with randomized decision forests classifier and point cloud library(PCL) develop the training program.

May 16- June 1 :
Finish Second, label human pose and training randomized decision forests classifier.

June 2 - June 15 :
Study optical flow and mean shift tracking method.

June 15 - July 13 :
Finish Third, Add optical flow and mean shift tracking method to get the a precisely body part.

mid term evaluation

July 15 - August 1 :
Finish last step,send body part message by tuio and develop a demo.

August 2 - August 22:
optimization code ,getting the final documentation and test the module,fixed bugs.

Final Evaluation

Reference:
kinect: http://en.wiki.org/wiki/Kinect
openkinect: http://openkinect.org/wiki/Main_Page
nestk : http://nicolas.burrus.name/index.php/Research/KinectUseNestk
openni: http://www.openni.org/

Feedback:
Any commit is welcome,look for your suggestion,Thanks.

[GSOC 2011 Proposal]GPU Implementation

2011-03-27T13:49:44-05:00

Name: Remaldeep Singh
Email:technoremal@gmail.com
Location/Timezone: Chandigarh, India (GMT +5.30)
Age: 21
Education/Qualifications: Pursuing BE at Chandigarh college of Engeneering and Technology, Panjab University. Currently in 3rd year.

Project proposal:
I plan to develop GPU implementation of CCV this summer of code. This can be achieved using Nvidia CUDA programming language.

Since Nvidia CUDA is based on c/c++ it is cross platform capable of running on Windows,Mac and Linux. But it runs only on Nvidia graphic cards.
I choose CUDA over Opencl because it is much mature than opencl and moreover the Opencv has built in CUDA functions taking full advantage of
CUDA GPU architecture.If the need for porting it on ATI GPU’s arise than it can be done using “swan” which converts CUDA code to Opencl code.

Coding will include using CUDA libraries like CUBLAS(CUDA implentation of Basic linear algebra subprograms) and NPP(Nvidia Performance Primitives)
and Opencv GPU functions.

Feedback:
It would be really helpfull if you can post any queries regarding the proposal or post anything that i missed.It will help me in formulating my final proposal.

EDIT: I have submitted my proposal to the nuigroup community. Here is the link to the proposal in pdf format: http://www.box.net/shared/nv66hkb0g9