Hi everyone, I have started getting all my equipment together to build my screen, so far I have cut and sanded the perspex sheet, wired up some test LEDs and built a pine frame for the screen. Well thats where I am on the construction side of things, but now to the interesting stuff, Software!

I am going to be focusing on Windows Vista as my OS, this is because Vista uses a radically different method to render the desktop. Let me explain…

All the programs in Windows XP and below use GDI to draw what you see on the screen, Windows uses messages to ask each application to redraw areas that need to be redrawn and the application draws straight to the screen. Vista manages the drawing slightly differently by asking each application to draw to an off screen surface, then the desktop is composed by the DWM (desktop window manager) in directx, this allows for the glass effect and the other cool effects used in vista.

Basically what I’m going to try and to is create my own shell and DWM that will be a zooming user interface for windows vista.

Here is how its going to work…

The project will be split into 3 parts, Input, Shell Management and Output

The Shell Management will handle all of the Windows API stuff, it will keep track of all the open windows, their sizes and their shared resource handles. This will be the executable of the project that may or may not run as a service. This exe will load 2 dlls at runtime that will handle the input and the output. The idea behind splitting it this way is to allow 3rd parties to develop new input librarys for different types of projects, and allow directx programmers to develop new desktop renderers without needing to worry about any windows API or input stuff. The Shell manager will initially handle basic gesture recognition, but I may make this a separate plugin at a later date.

The input library will send input events to the Shell Manager, the first plugin I will write will be for multi touch screens, but I may make a multi mouse plugin too as it should be relatively easy. The multitouch plugin will handle the Web Cam image acquisition and processing and send events to the Shell Manager.

The Output library will do all of the rendering using Direct3D 10. The Shell manager will provide a collection of windows and trigger events for new and closed windows that the plugin will have to handle. The first Output plugin I will create will allow you to move windows applications around like flash examples on the home page.

I have created a managed shell for windows xp before so I’m confident I can create the Shell Manager without too many problems, however there are still quite a few programming elements here that I have never used before. I will keep this thread updated on my progress.

There are a few hurdles in this project, here they are…
DONE 1. the DWM is a private API used by Microsoft and is not supposed to be used by the public (DWMAXX!)
UNNECESSARY 2. I will be creating this project using only managed code, and currently there is no managed version of directx 10 or 9L
DONE 3. My current graphics card doesnt support DX10 (Bought one, however now its not necessary)
DONE 4. There are no Vista drivers for my Web cam yet (Phillips SPC900-NC) (Now available)
UNNECESSARY 5. I dont have any experience in directx programming (Not using DirectX at the moment so this can wait.)

Stages of Development

UNNECESSARY 1. Create a proof of concept application to draw a shared texture from the DWM in DirectX10 (DWMAXX!)
DONE 2. Create a basic plugin system
DONE 3. Add Shell handling to the plugin system
UNNECESSARY 4. Create a proof of concept direct draw image processing app to convert web cam input to coordinates (tBeta!)
DONE 5. Create the first basic Input Plugin
6. Create the first Basic Output Plugin
7. Release an alpha version for people to test and give feedback
8. Implement the ideas from the feedback
9. Create a SDK for others to create plugins

Well thats it so far, sorry for the lengthy post, this has been cooking for quite some time and I only just feel all the pieces coming together that will make this project work.

*EDIT*
Iv’e recently had more time to work on this and have made some decent progress and with the help of DWMAXX I should be able to get some videos of what Iv’e been working on uploaded soon
*/EDIT*

Thanks for reading
Pleh

i stopped reading when i read vista, no just kidding, u got a nice list of things you want to accomplish

so at the end of the project you will be able to write code so you can do the blobtracking by using the gpu?

i must say, this sounds really exciting and interesting.

yeah,i think you should be able to get the gpu to do the blobtracking with this way. as i understand it the gpu does all the rendering in vista - thats why the dx10 card is needed.

making the gesture recognition a plug-in would be a good idea, as you never know when a new gesture might be thought up. but overall, great ideas!

also, i’m hoping to install vista sometime within the next few weeks, so if you need some testing on a different system with different webcams done, just give a shout!

I dont know i will be able to do the blob tracking on the GPU, but if i can, i will. The main goal of this project is to be able to run any windows application in a multitouch environment. We will be limited to a single click per application, but i will make gestures send other commands like scrolling etc. and we will also be able to have multiple users using multiple applications simultaneously.

As soon as I get my new graphics card I will let you guys know how step 1 goes.

Fingers crossed!

*EDIT* Just found this http://brahma.ananthonline.net/static.php?page=static061128-020520 Looks interesting and should allow us to do the blob tracking on the GPU fairly easily.

Just a quick update, I have the first half of number 4 done, I can get the input of the camera into my app on a frame per frame basis to allow for blob tracking. Next I will work on some blob tracking algorithms.

As far as step 1 goes, I have my new gfx card but DirectX 10 seems quite confusing to me, I need to do a bit more research. If anyone with some directx experience wants to help, A simple example of a textured quad using managed direct3d 10 would be a great help.

You can find the MDX10 files here
http://www.codeplex.com/Wiki/View.aspx?ProjectName=MD3D10

I like the idea. I have my system running up on vista with a geforce 8800 gts 640 mb (direct X). I’ve been able to run all the standard directx 10 demos (cascades, adrianne, froggy, box of smoke) working w/ my multitouch setup.

I definitely have more experience with processing and opengl, but I am definitely interested in seeing just how far directx 10 can go. It looks so nice.

let me know if you need a beta tester.

richard

Hey Pleh, I like where you are going with this! I’m running on Vista as well, with a GeForce Go 7400 gfx card and core2.

A few things I’m thinking:

1) This would be amazing, and needs to be done. I’m glad you’re doing it, and I’ll think about ways I can help you. I mainly do C++/OpenGL/CG Shading language coding, but I do have a mild amount of experience with .NET. I’m mainly turned off by the fact that the CLR doesn’t have any decent implementations on linux or mac.... that I know of.... does it?

2) I see you’re looking to do your own image processing into input coordinates. Have you looked at all into touchlib? You should see how your planed algorithms overlap with what touchlib already has, and we should all maybe think about starting up a thread on the optimal algorithm for blob detection. It would be nice if the .NET coders and native coders traded secrets for optimality.

3) GPU acceleration = nice! I’ve used CG (Nvidias shading language) to make a GPU accelerated ray casting volume renderer, and I have to tell you, 60 fps true light simulations through volume grids look incredible. I haven’t tried out Brahma, since I’m mainly non-.NET. It would be nice to think about a way to generalize your work here to work outside windows vista. But.... since I have vista… lets first get the vista part working =)

4) You mention you’re using the product of reverse engineering to get the shared resource handles (I’m guessing this leads you to each window’s framebuffer?). I know microsoft has many bells and whistles and warnings associated with reverse engineering. You (or someone) might want to check out of this work will be violating some license somewhere. I think they intentionally make the process of finding this info boring, to keep the lawyers employed =(

5) You might not necessarily be restricted to single mouse inputs. We should look into whether or not vista supports multi-point input.

Sogar - 12 May 2007 10:55 PM

I’m mainly turned off by the fact that the CLR doesn’t have any decent implementations on linux or mac.... that I know of.... does it?

i think there is the mono-project . but i haven’t used it, so i can’t really comment.

All versions of windows support multi input, in the form of raw input messages. but applications have to register for these messages themselves so almost no applications use them.

I have very limited directx programming skills especially in directx 10, once i have all the basics in I will release the plugin source to allow more experienced directx programmers to make their own.

I havnt had much free time to work on this, work is crazy at the moment. I will keep you all updated and let you know when i have anything to show you guys.

http://openvidia.sourceforge.net/ I am not sure if this is helpful, but I did find it rather interesting

cmoore - 13 May 2007 02:38 PM

http://openvidia.sourceforge.net/ I am not sure if this is helpful, but I did find it rather interesting

looks like a GPU accelerated openCV. Could be the next step for touchlib. although i don’t know if it will run on non-nvidia cards…

There are a few hurdles in this project, here they are…
1. the DWM is a private API used by Microsoft and is not supposed to be used by the public
2. I will be creating this project using only managed code, and currently there is no managed version of directx 10 or 9L
3. My current graphics card doesnt support DX10
4. There are no Vista drivers for my Web cam yet (Phillips SPC900-NC)
just found them here...
5. I dont have any experience in directx programming

Thanks mctouch, now i can finally get to testing on vista

Finally some progress

I have just got a test app working that draws a window to a WPF Image. The significance of this means that I wont have to use directx 10 and also I can use WPF to do all the rendering and rotation.

Its not the sleekest solution, its kind of like cramming a square peg in a round hole. But it works, which is enough for now untill Microsoft create a managed wrapper for Direct3D 10 or create a public MILCore API.

To cut a long story short, doing it this way (Although more CUP intensive that my prefered method) means that all my “hurdles” have been overcome, Full steam ahead!

hi,just wondering how you are progressing?

Pleh - 11 June 2007 12:49 PM

Its not the sleekest solution, its kind of like cramming a square peg in a round hole. But it works, which is enough for now untill Microsoft create a managed wrapper for Direct3D 10 or create a public MILCore API.

WPF IS a wrapper for Direct3D. Actualy all you make with WPF is being processed by GPU. And you can also make 3D things in WPF.