Web Based Multi-touch Application Framework with Custom Gesture Definition Support
Posted: 01 April 2010 02:46 AM   [ Ignore ]
Joined  2009-05-25
Total Posts:  1
New Member


I would like to share my idea for Google Summer of Code for NUIGroup. In short, I am trying to develop an application framework that will:

a) work on different platform and devices,
b) provide the flexibility to build applications with most natural and meaningful gestures, and
c) portable: minimum or no installation

Now, “different devices” includes different types of devices like Microsoft Surface, Multi-touch Tablets & Desktops, Mobile Devices (i.e. iPhone, Android) and “different platform” refers to different types of operating systems like Windows 7, Mac OS Snow Leopard, iPhone OS, and the like.

Although most devices and platforms have their own technological distinctness, they all support a common platform – browser based web applications. Also, web applications are highly portable as they don’t need installation. And, most popular web browsers (i.e. FireFox, IE9) now provides support for touch and mobile device browsers (i.e. iPhone - Safari) also provide touch events.

Finally, lets look into possible applications:
1) Software Designing tool: Sketch enabled UML Designer (i.e. SmartUML),
2) Molecule Bond drawing: could be a useful tool for chemistry students (I saw an app for iPhone… lost the url)
3) Different drawing applications
4) Planning tools
5) GIS data presenting tools
6) Collaborative tools

If we study the requirements, the graphical & programming challenges can be meet by new HTML5 features. But, for user interactions, research studies show that often pre-defined gestures are not good enough and application developers need to create/define new gestures that is natural & meaningful to particular application context.

Here, we have two issues: First, existing gesture recognition toolkits are build in a way that only locally installed applications can use them. So, we can’t directly use them in web applications. Second, we don’t actually have any nice and easy way to define new gestures. To address this issue, I propose a flexible and extensible gesture definition language. I have been working on this for sometime now and you can see some examples here: http://bit.ly/9BuZPG. Its in early stage of development but I have some demo that I will upload shortly.

I initially started to develop the framework on Silverlight but the main problem of these web frameworks (i.e. Silverlight, Flash) is that it require additional plugin which is not supported in all platforms. On the other hand, a pure browser based (HTML5 & Javascript) framework could have much wider access.

Please let me know what do you think. A more detailed proposal is available here: http://bit.ly/anc39L