1 of 2
1
Attempt at “flat” MTT project
Posted: 17 April 2009 03:24 PM   [ Ignore ]
Rank
Joined  2009-04-17
Total Posts:  43
New Member

EDIT : big update at the bottom !

Hi everyone !

After trying out an MTMini, and looking at the different technologies and variation of MTTs i decided to give it a go as well , and hope this will be my first contribution to NUi

EDIT : newer videos in the latest post

So what is it exactly ?
-Well, since i have very very little spare space in my flat, and just love tinkering, i looked at the different possible ways to create a flat (or “flatter") multi touch screens , specifically in these threads :
lLink as well as link

I whipped up a very small scale test last week end : you can see the result of some very preliminary testing in the video above:
-It will most likely be rear DI based.
-This how it works : IR diodes are each surrounded by 4 Photo-transistors (well, 4+2, since i didn’t have the space for a real 3*3 matrix) linked to the analog pins on an Arduino board.
The program was done in Processing ( first processing program for me , but so much fun !) (processing.org/)
-the program translates the ir intensity from the photo-transistors to a few different visual representations : “circles “(white) for each, based on intensity, plus numerical values, and some interpolation attempts (red circles). Will see where this goes… (resolution/tracking problems, as well as overall cost, for now my estimates are quite good)

-as i do not have the budget or space for a projector, i will be using some small disassembled lcd screens (16 ” and 19″): I have already successfully disassembled (that means the screen still works! yeah !) the 16″ Asus Screen (VW161), so i will post some disassembly step by step pictures on my site as soon as i can.

As pictures are worth a 1000 words ...
General layout of smallest size matrix:
irmatrixlayout-150x150.png

This is most likely what the wiring will be for my next bigger prototype (with more than just 4 phototransitors of course, it should scale well, and have the smallest possible amount of components and wiring yet still efficient, i hope)
irmatrix_med.png

Advantages:
-Obviously this would enable to add multi touch to a screen while keeping it very thin (around 5mm)
-I do not intend to use a very dense sensor matrix so construction cost and difficulty should be kept relatively low (around 60 dollars for all the sensors and electronics for a 16” screen, not including the arduino)
-If my calculations are correct the arduino should be able to handle the sensor data acquisition as well as some low level interpolation (needed since the matrix is not very dense) at very high fps
-the system can be made modular by linking multiple smaller sized matrices (this is not an immediate concern of mine though)
-the plan is to have a lot of pre computing done on the arduino itself, and then send a videofeed/ frame by frame pictures directly to touchlib/some other library , so it can use the already existing frameworks and programs ( i am looking around the documentation and haven’t found a way for this yet tbh)

Disadvantages:
-the ad-hock resolution is quite low but with a little bit of trickery it can be compensated (by interpolation etc)
-the bigger the screen , the more sensors and ir emitters you need
-the bigger the screen the bigger the computation cost (an arduino can only handle so much)

I hope you will excuse me for the long winded explanation, and i hope that this can be usefull, i am very open to any sort of criticism , feedback and suggestions, and will post back when i have some progress done.
I will also provide additional information on my site as soon as i can :Kaosat.net

Bbye !

Image Attachments
irmatrixlayout-150x150.pngirmatrix_med.png
 Signature 

Flat Mtt : in progress http://www.kaosat.net and http://nuigroup.com/forums/viewthread/5200/

Profile
 
 
Posted: 17 April 2009 05:27 PM   [ Ignore ]   [ # 1 ]
Avatar
RankRankRankRank
Joined  2008-02-12
Total Posts:  1120
Member

Very interesting approach… definitely one i’ll follow.
Nice work

 Signature 

Company Website - Knode Research and Development
My Youtube

Profile
 
 
Posted: 18 April 2009 12:56 PM   [ Ignore ]   [ # 2 ]
Avatar
Rank
Joined  2009-03-31
Total Posts:  71
New Member

gr8!....amazing idea!

 Signature 

Team Zugard
Jaipur, INDIA
WE HACK INTO HE@RTZ!!!
smile

Visit ZUGARD’s Blog!!!

Profile
 
 
Posted: 21 April 2009 06:02 AM   [ Ignore ]   [ # 3 ]
Rank
Joined  2009-04-17
Total Posts:  43
New Member

Thanks Taha and Zugard!
Taha : i checked out your site: you really have some very impressive projects!
And i can’t imagine how great it could have been to work on that 10m*1.5 wall smile Very nice work!

So , a tiny bit of progress report: as always, if you see anything wrong with the underlying methods/technology that i am using, do not hesitate!

Since, like an idiot, i managed to burn out the serial chip of my only arduino (lesson : don’t let metal screws lie around the table ), i expanded a bit more in the software area, and used the mouse to simulate a single finger/object, and the way it would change the ir received by the phototransistor matrices below it.

ir_interp_emulation003_screen-300x104.png

-first i did a quick mockup of the type of sensor grid i would be using, and used bilinear interpolation to generate “spots” (not blobs yet at this point) : you can find more info and the applet here http://www.kaosat.net/?p=329
-i then used the openCV library for processing to do some very basic blob recognition and tracking
more info here : http://www.kaosat.net/?p=425#more-425

-excuse the general “jumpiness” of the blobs, it mostly come from the mouse position to ir values approximation
-in the second link you will notice i also included an applet that uses a different approach at position determination : linear interpolation across two axes, which can help pinpoint rather precisely the position of an object with just the values from 3 sensors.

I hope to get my new Arduino soon so i can try this out “for real"…

Once again thanks in advance for your feedback and comments!

Image Attachments
ir_interp_emulation003_screen.png
 Signature 

Flat Mtt : in progress http://www.kaosat.net and http://nuigroup.com/forums/viewthread/5200/

Profile
 
 
Posted: 24 April 2009 05:46 AM   [ Ignore ]   [ # 4 ]
Rank
Joined  2009-04-17
Total Posts:  43
New Member

Another progress report:
got my arduino working again, received the big pack of phototransistors as well, so work is continuing:
-This week end i will post a video of the first tests of the small phototransistor strip with a basic form of tracking working (so great to have blobs recognized and tracked, even with not so great accuracy)

I am facing a few problems though so any help would really be appreciated:
Software:
-in the current tracking software , is interpolation of blob position between frames used? (i seen info on interpolation in openCV, but not sure it is actually used by anyone)
-i am running into trouble with the amazing sensitivity of the phototransistors : not only do i detect fingers, but the whole hand ! even objects 30 cm away are registered: the only way i found to alleviate this is to modify the thresholds so only the peak values are generating blobs: is this the common method ?

Hardware:
-having the ir emitter in the middle of a 2x2 matrix is very nice for basic testing purposes BUT it is impossible to have correct calibration as i have no diffuser above (essentially i ONLY have hotspots)
-i tried using what i thought was the diffuser from an old stripped down lcd screen, but almost all the ir gets reflected back so any cheap solution for experimentation would be more than welcome!
-what kind of ir projection method do you think would be the best for this kind of project? i initially thought of using simple rear di, but with the diffusion problem i wonder if other methods would not be better..if only enlighten wasn’t so expensive..

Thanks in advance!

 Signature 

Flat Mtt : in progress http://www.kaosat.net and http://nuigroup.com/forums/viewthread/5200/

Profile
 
 
Posted: 24 April 2009 06:22 AM   [ Ignore ]   [ # 5 ]
Rank
Joined  2009-04-24
Total Posts:  3
New Member

I view these forums a lot, but this post made me make an account, because I am looking into the same subject.

I see you are using LEDs as wel as photodiodes. Do you know that leds can act as photodiodes themself?.

This video demostrates it:

I can’t find my bookmarks that I use at home, since I am at work now.

You might want to look into this smile

Gr,
Pim

Profile
 
 
Posted: 24 April 2009 06:45 AM   [ Ignore ]   [ # 6 ]
Rank
Joined  2009-04-17
Total Posts:  43
New Member

Hi Opaque!

You said you are looking into the same subject : have you tried experimenting with it?

i actually seen that video (just last week in fact) , and that is also a method i will try to experiment withsmile
there are two possible drawbacks with that method, from what i gathered:
-i looked up phototransistor vs photodiodes, and the signal you get from a photodiode is considerably less strong, so you would most probably need additional amplifiers
-i am not sure the “sensing” capability of irdiodes uses as photodiodes would be enough and fast enough

i will give it a try this week and and compare the results
thanks for reminding me about itsmile

 Signature 

Flat Mtt : in progress http://www.kaosat.net and http://nuigroup.com/forums/viewthread/5200/

Profile
 
 
Posted: 24 April 2009 07:07 AM   [ Ignore ]   [ # 7 ]
Rank
Joined  2009-04-24
Total Posts:  3
New Member

Hey Ckaos,

What I remember right now, is that the led operates in pulses, alternating between sending and receiving very fast.
Also remember that you do indeed need amplification for light sensing, since a Diode is >| ofc. Wished that the guy who posted that video had given a bit more explanation smile

Atm, I am too busy with uni/work/other stuff that I am not working on my multitouch screen. I did however just learned how to operate and program an arduino at my study (Industrial Design) so I am going to use that in my next build.

Gr,
Opaque

Profile
 
 
Posted: 24 April 2009 07:44 AM   [ Ignore ]   [ # 8 ]
Rank
Joined  2009-04-17
Total Posts:  43
New Member

-following your post i looked around and found a few sites dealing with using leds as photodiodes : for ex http://itp.nyu.edu/physcomp/sensors/Reoprts/LEDAsPhotodiode so i will look into all this, no reason why it would not work for ir leds

-the “pulse” method sounds interesting, because my current test setup has some similarities:
as shown in the diagram in my first post i only have one data pin for a group of phototransistors, and each one of them gets powered in turn : so it goes like this :
-bit shifting on the multiplexer to select the correct pin to power
-set the correct pin to “high”
-read the ir intensity value
-bit shift again, and repeat

The arduino is a jewel ! i might no be a pro and more of a software guy, but if you need a bit o’ help, just asksmile
oh and lucky you for being in industrial design smile must be great!

 Signature 

Flat Mtt : in progress http://www.kaosat.net and http://nuigroup.com/forums/viewthread/5200/

Profile
 
 
Posted: 24 April 2009 07:50 AM   [ Ignore ]   [ # 9 ]
Rank
Joined  2009-04-24
Total Posts:  3
New Member

Industrial Design is ace, although I am more of a worker than a student! smile

Take a look at this:
http://www.merl.com/papers/docs/TR2003-35.pdf

It explains exactly how to wire/make a bi-directional LEDs. The end product they make is different, but the base idea is the same!

Gr,
Opaque

File Attachments
TR2003-35.pdf  (File Size: 531KB - Downloads: 313)
Profile
 
 
Posted: 24 April 2009 01:05 PM   [ Ignore ]   [ # 10 ]
Avatar
RankRankRankRank
Joined  2008-02-12
Total Posts:  1120
Member

BI Directional LEDs are pretty simple with Arduino, but from my experience they arent as accurate, they rely on non-haptic method input detection, and as such they wont perform and feel as good, and since they rely on the reflection of light on to the LED for detection they fall short when you have an LCD in between the input detection then is far worse… Ideally if you were to employ such a technique it would only work well 3~5mm LED displays.

@ ckaos i’ve been doing a lot of research for my thesis project that is similar to what you are trying to achieve i have a couple of schematics and ideas that may help you a lot and in turn the community if we can get it to work. send me a PM

 Signature 

Company Website - Knode Research and Development
My Youtube

Profile
 
 
Posted: 25 April 2009 02:18 PM   [ Ignore ]   [ # 11 ]
Rank
Joined  2009-04-17
Total Posts:  43
New Member

@Opaque : thanks a lot i have started to read through that document, whether i use it or not in the final project, it is full of interesting information
@Taha Thanks a lot for that very valuable information !
Do yo mean that bi directional LEDs would only work on basic LED matrices such as in the video posted by opaque ?
And your proposal would be very welcome, and i am more than willing to, in turn share any information that might help anyone accomplish something similar! Sent you a PM!

A side note regarding the photo transistors:
The good: -in the evening have plugged a few of the new ones i received (3mm) into the interpolation program i wrote a few days ago, and after tweaking a few settings i was able to get some rather sharp, distinct and multiple blobs , and this on a very small surface (2*4 photo transistor matrix) at good fps
The bad: -i am a bit bummed , but the 300 i just received (19 dollars, was very cheap) are very precise, unfortunately, having a peak sensitivity of 880 nm is really much worse than i thought : they get completely saturated by ambient (indirect) sun light very easily…

 Signature 

Flat Mtt : in progress http://www.kaosat.net and http://nuigroup.com/forums/viewthread/5200/

Profile
 
 
Posted: 21 May 2009 04:44 PM   [ Ignore ]   [ # 12 ]
Rank
Joined  2009-04-17
Total Posts:  43
New Member

Has been a while since i posted an update on this project but here goes, : lots of progress done :

I’ll start with two videos of the latest prototype (8*10 cm) in action.

More info:
-the whole system is about 4-5 cm thick (including screen , box, everything)
-the sensor matrix itself is about 1cm thick
- I managed to keep my idea of relatively few photodiodes (exactly 40 for this size) more info on this soon
-i compensated the relatively low resolution of the sensor grid with a two pass interpolation : an averaging pass and a 3*3 kernel Blur pass
-this data is then sent to openCV for blob detection
-I developed my own experimental tracker for processing , using openCV ( the tracked blobs are surrounded in green, on the right side of the screen)
-in the video , it runs at 120fps :

And even more info:
-the setup is/will be endligtthen based/dsi (it works) although i cheated in this video i just inverted the data from the “IR Shadow” of my hand ( i accidentally ripped off the wiring from the IR Leds earlier today :( but if anyone wants proof i have a video of the V1 prototype working with endlighten)
-the endlighten will be used both as IR Diffuser AND back-light, although that requires some more adjustment to work smoothly with this prototype, but i can confirm it works ( but i do not believe it would work with a modified camera/webcam, due to the extensive trickery i have to go through to make it work in this case)
-total costs , including fabrication of industrial quality pcbs should not exceed 400 euros including everything ( an ion mobo and processor as well)-the price for everything excluding the pc but including the 16” screen (bought new) is around 250

A lot of things still need tweeking and work, but it’s starting to take shape…
I will post more information as soon as things are more finalized…

Comments and criticism is more than welcome as usual !

 Signature 

Flat Mtt : in progress http://www.kaosat.net and http://nuigroup.com/forums/viewthread/5200/

Profile
 
 
Posted: 25 May 2009 03:54 PM   [ Ignore ]   [ # 13 ]
Rank
Joined  2008-07-16
Total Posts:  25
New Member

wow ckaos, great! you continued exactely my plan, which got a little bit stuck due work & university.

have you attached the ir leds to a shift register yet to go trough?  i think time multiplexing should be possible so that you have more accurate data.

altough im really thinking alot, if this whole idea is really the right way (it sure is interesting)
pros are:
+ touch free / distance sensor
+ height
cons:
- inaccuracy
- scalability
- speed (if you scale and multiplex)
- COSTS
- daylight

OT:
the other method i want to try is the one commercial IR touch sensors work like autouch.net.
they have MT with like 8 points (which is ok for most HCI stuff in the next 1-2years) and scales very well.

as said in the other thread im currenty working with IR photo diodes, (daylight filter, but sooo much noise). i hope to be able to contribute some stuff soon, but i just
lost all my code for the multiplexer and readout. :(

so keep up the great work ckaos.

edit: ckaos: why do you go via graphics and opencv.
shouldnt the whole point be to do the interpolation in simple matrix transformations which can then be ported to an uc?

Profile
 
 
Posted: 25 May 2009 04:28 PM   [ Ignore ]   [ # 14 ]
Rank
Joined  2009-04-17
Total Posts:  43
New Member

Hi Nemster!
Hey it’s great that you reply since your thread was amongst those that inspired me to start this smile
actually , i have a different approach as far as getting the data goes (i will post some information on this later), but i might switch back to multiplexers in the end, the precision is the same (tried both)and i also use only one data line for now.

for your cons and pros list i agree for some things but not all smile
-speed , for the size of screen i have can actually be a lot faster than with most cameras and could be faster even with a higher end microcontroler
-cost: well if you consider the price of higer end cameras some people use is close to the price i am aiming for
-daylight : well i have found a way to get rid of that problem at least partially although i need to run more tests for this (my method involves using “reference sensors” which subtract the ambient light/useless information from the useful data)
-about opencv and graphics : well while in the end some dedicated system that would not rely on this would be better, at this stage it still is the easiest for debugging and feedback, and the Arduino does’nt have the processing power to get the data AND do these kind of transformation at sufficient speeds.
I also wanted something that integrates easilly with the already existing frameworks. And believe me , even like this is very fastsmile ( i am using simple matrix transformations “behind the scenes” btwsmile

Ouch, yeah, just taken a look at the autouch.net site, and their products do seem quite better than anything i could ever do .. but well ..that is to be expected...if i had more fund, more time and more knowledge..ah well..
Thanks a lot for your comments Nemster smile i will soon have more to post (recieving final piece of endlighten and other parts of the hardware soon)
Good luck on your project then smile my photodiodes don’t even have daylight filter haha, so you could get better results than me for suresmile

 Signature 

Flat Mtt : in progress http://www.kaosat.net and http://nuigroup.com/forums/viewthread/5200/

Profile
 
 
Posted: 25 May 2009 04:38 PM   [ Ignore ]   [ # 15 ]
Rank
Joined  2008-07-16
Total Posts:  25
New Member

you probably have transistors not diodes

ok news:

http://e-collection.ethbib.ethz.ch/eserv/eth:41461/eth-41461-01.pdf

pros: wow!
cons: 2cm minimal distance between points!, FTIR

File Attachments
eth-41461-01.pdf  (File Size: 331KB - Downloads: 488)
Profile
 
 
   
1 of 2
1
 
‹‹ DI Table. Is this ok?      LLP Problem ››