Project Proposal: A Cross-platform TUIO Gestural TouchPad using the iPhone

Project Objective: By far, there has been only the touchpad of Mac notebook supporting true multi-touch function. So why should these intuitive gestures be exclusive only to the Macbook Pro users? Due to the iPhone’s widespread use, I think it will be suitable to use the iPhone as an alternative for current pc users to experience multitouch capabilities on their own computers even if they’re not using a MacBook Pro.  I propose to implement a cross-platform TUIO-based multi-touch touchpad server using the iPhone as an alternative to current keyboard and mouse controls that works on the Linux, Windows and MacOS operating environment. The TUIO server receives the iPhone’s (Client) command via TUIO, and maps the commands into mouse or keyboard events on the local OS (Server) to control/interact with the PC. I also plan to allow users to pre-define 2 points，3 points or 4 points gestures as well as recognize single-point gestures, similar to current mouse gestures.

Features:
1. Implement a TUIO TouchPad server on existing PCs (notebook or desktop) to support multi-touch functionalities.
2. Support basic gesture recognition up to 4 point gestures although the iPhone is able to detect up to a maximum of 5 points as 5 points gestures would be unintuitive and not ergonomic given the small size of the iPhone’s touchscreen area.

Name: Zhang Zhen
Email:
Location/Timezone: China,Shanghai
Website: http://www.progenlabs.com
Age: 28
Education/Qualifications: Ph.D. student
Academic and Industry Background: Shanghai Jiao Tong University, Research Institute of Robotics
Open source development experience: I developed a CORBA based 3D simulator for dual-arm mobile service robot (SmartPal) using Java and Java3d (My paper: http://chinesesites.library.ingentaconnect.com/content/mcb/049/2009/00000036/00000001/art00005)( http://journals.cambridge.org/abstract_S026357470800489X).I am also a main programmer of the Jiao Long Robocup team, using C++ to control soccer robots in the Middle sized matches of RoboCup World Championship from 2006 to 2008.  All of which the codes are open-sourced on our lab’s website.
Development Methodologies: Open framework, Mouse driver in multiple platform

All reviews and opinions are welcome

Seems a bit similar to http://nuigroup.com/forums/viewthread/4857/ also might be good to work in collaboration with Memo http://nuigroup.com/forums/viewthread/4772/

Memo already developed an iPhone app, he’s trying to push it into Apple app store, although not yet succesfully.

The pdf file of the proposal has been added. All reviews and opinions are welcome.

The final proposal
1. Project Objective

By far, there has been only the touchpad of Mac notebook supporting true multi-touch function. So why should these intuitive gestures be exclusive only to the Macbook Pro users? Due to the iPhone’s widespread use, I think it will be suitable to use the iPhone as an alternative for current pc users to experience multitouch capabilities on their own computers even if they’re not using a MacBook Pro.  I propose to implement a cross-platform TUIO-based multi-touch touchpad server using the iPhone as an alternative to current keyboard and mouse controls that works on the Linux, Windows and MacOS operating environment. The TUIO server receives the iPhone’s (Client) command via TUIO, and maps the commands into mouse or keyboard events on the local OS (Server) to control/interact with the PC. I also plan to allow users to pre-define 2 points，3 points or 4 points gestures as well as recognize single-point gestures, similar to current mouse gestures.

2. Features

(1) Implement a TUIO TouchPad server on existing PCs (notebook or desktop) to support multi-touch functionalities.
(2) Support basic gesture recognition up to 4 point gestures although the iPhone is able to detect up to a maximum of 5 points as 5 points gestures would be unintuitive and not ergonomic given the small size of the iPhone’s touchscreen area.

3. Previous work

I’m a member of ProgenLabs. So far, we have developed a TUIOmouse driver (http://nuigroup.com/forums/viewthread/3447/ ) which works with both Touchlib and Tbeta. Here’s a video (http://nuigroup.com/forums/viewthread/3362/ ) to show the performance of the TUIOMouse driver.  In this video, you’ll see us using an ipod touch as the input device, which is running OSCemote to send out OSC messages. However, it has only been used in Windows OS. We need to write a TUIO mouse driver for both Linux and Mac OS.

4. Schedule

Step1 (20 April-23 May)
Discuss with mentors about using TUIO protocol for gesture recognition and the approach to implement a gesture Library. Also, it is time pick up development methods in Linux and Mac OSs.

Step2 (24 May -13 June)
Begin to write the mouse drivers for Linux and Mac OSs, and test their performances.

Step3 (14 June -10 July)
Begin to implement a gesture recognition library for cross-platform TUIO. It works by analyzing fingers’ or iphone movement while dragging, and comparing the motion against a regular expression to find a match. Some test clients will also be developed using the (iphone SDK) to test the gesture recognition library’s performance.

Step4 (11 June -4 Agust)
After mid-term evaluation this period is used for discussing and analyzing the results with the mentors and to establish the priorities for successful completion of the GSOC proposal.

Step5 (5 Agust -17 Agust)
Project conclusion. This release includes project packaging through Google Summer of Code and NUI Group sites. Also the code fully documentation is provided within a tutorial, if needed for adapting your setup to be able to use the program.

Step6 (18 Agust - )
Time to relax, enjoy the community feedback… If possible, plan to visit google China in Shanghai.

5. Author Information
Name: Zhang Zhen
Email: ,
Location/Timezone:  China,Shanghai/GMT+8
Website: http://www.progenlabs.com
Age: 28
Education/Qualifications: Ph.D. student

Academic and Industry Background: Shanghai Jiao Tong University, Research Institute of Robotics

Open source development experience: I developed a CORBA based 3D simulator for dual-arm mobile service robot (SmartPal) using Java and Java3d (My paper: http://chinesesites.library.ingentaconnect.com/content/mcb/049/2009/00000036/00000001/art00005 )( http://journals.cambridge.org/abstract_S026357470800489X ).
I am also a main programmer of the Jiao Long Robocup team (http://robolab.sjtu.edu.cn/robocup/ ), using C++ to control soccer robots in the Middle sized matches of RoboCup World Championship from 2006 to 2008.  All of which the codes are open-sourced on our lab’s website.

Development Methodologies: Open framework, Input driver (HID) on multiple platforms

What is your plan to share this application if you encounter resistance from Apple, like Memo has?  What will you do if Apple doesn’t see the need for this on the AppStore?  What argument will you make?

Ori Rawlings - 02 April 2009 05:22 PM

What is your plan to share this application if you encounter resistance from Apple, like Memo has?  What will you do if Apple doesn’t see the need for this on the AppStore?  What argument will you make?

We will communicate with Apple and show our applications, get the feedback from Apple and improve it… If we can develop some good applications, I think Apple will support us.

I think a simple alternative might be to simply use the iphone as a multitouch client that sends the inputs to the server on the computer.  If that is the case, using an already approved app like oscemote or mrmr will suffice.  Since the main function of recognizing the gestures should be done by the server.