1 of 2
1
Multitouch Musical Instruments - Surface Editor
Posted: 24 July 2009 12:13 PM   [ Ignore ]
Rank
Joined  2008-07-03
Total Posts:  9
New Member

The research group at Geneva’s Conservatory of Music presents the Surface Editor and Surface Tracker software applications for turning ordinary surfaces into multi-touch musical instruments.  The Surface Editor allows users to custom design interfaces by dragging and dropping components, such as buttons, sliders, keyboards, and many others, onto the interface. The surface can then be used to control digital audio workstations, virtual instruments, hardware synthesizers and samplers, VJ software, and other applications, via either the MIDI or OSC protocol.  The Surface Tracker is a multi-touch tracking application created for tracking movements on a surface using high speed infrared OptiTrack cameras.  These cameras have on-board image processing, which allows them to process 100 frames/sec while sparing the host computer’s CPU.  The applications are available at http://www.surface-editor.com.

Profile
 
 
Posted: 24 July 2009 02:22 PM   [ Ignore ]   [ # 1 ]
Avatar
RankRankRank
Joined  2008-06-16
Total Posts:  330
Sr. Member

Hey Greg, thanks for posting the video, documentation and application of your Surface Editor project.
What a pity that I missed NIME this year, I really like the simple paper based reference frame approach of your system,
It reminds me bit of Enrico’s d-touch sheets, I reckon you are collaborating within the future instruments project?
The performance of the employed OptiTrack system looks quite good as well, are you also providing actual code for the camera,
or do they come with the tracking functionality off the shelf?
best, Martin.K

 Signature 

TUIO community site: http://www.tuio.org
reacTIVision framework: http://reactivision.sf.net
TUIO & reacTIVision CVS: http://sf.net/projects/reactivision
reacTIVision forum: http://sf.net/apps/phpbb/reactivision

Profile
 
 
Posted: 27 July 2009 05:23 AM   [ Ignore ]   [ # 2 ]
Rank
Joined  2008-07-03
Total Posts:  9
New Member

Hi Martin. 

Yes, it would have been good to see you at NIME.  I would have liked to have had a chance to talk to you about how things are going with reacTable Systems.  I tried to get together with Gunter for lunch last time I was in Barcelona but he canceled at the last moment b/c something came up… The OptiTrack cameras are mainly used for tracking the motion of bodies in 3 dimensional space, and they have released applications for that.  But I had to write an application to use these cameras for multi-touch tracking.  The application is open source and GPL.  It’s linked to from the surface-editor.com site, and all the code is at surfacetracker.sourceforge.net.  It supports both the tracking of fingers as well as fiducials.  We’ve been working with Enrico on integrating his latest d-touch work into the application… The one drawback of using OptiTrack cameras at the moment is that they only provide a driver for Windows, but we’ve been trying to work out a licensing agreement for an OS-X driver that was developed by a 3rd party…

Best wishes,
Greg

Profile
 
 
Posted: 27 July 2009 04:29 PM   [ Ignore ]   [ # 3 ]
Avatar
RankRankRankRank
Joined  2007-09-18
Total Posts:  882
Moderator

hi Greg,
thxs for sharing this awesome project . It ‘s very inspiring in so many ways .
Answering specific needs (musical use )
Lowering the floor, (reducing the hardware needs) ,
Opening a wide range of possibilities ( providing tools for creation )

Here goes a few comments & questions .

1/ MT Musical Instruments have been a recurrent topic in this forum, some music apps have been released around some using Flash , Processing or Max ,
but your object approach (musical object editor ) is quite unique .
There is also this ongoing musical GSOC project developed under NUIGroup supervision .
Your insights will be definitly welcome .

2/the form factor :

a / As outlined by Martin ,

the simple paper based reference frame approach of your system,

is simply stunning .
It fits perfectly for a musical use, since the “zones” indicates the right place to interact, the audio gives the feedback to the user .

b/ It also gives a great example of new ways to interact with a multitouch surface (no display (lcd, monitors, or video projector)) .
Even more affordable since the vp is the most expensive part

c/ as described in your paper , your LLP method is using a mirror before reaching the line lens.
Can you post some pics of this setup ?  ( transforming ordinary surfaces into MT controllers)

3/ Surface Tracker software

do your tracker implement the built-in tracking from the Optitrack camera ? Is the blob tracking is still done inside the cam ?

4/Zone creator

Your “tool for creating polygons for drawing arbitrary “ interactive shape is also a great contribution

5 / Do you plan to share the audio apps showcased in the vid ? 

Do you plan to update the documentation ? to provide more examples ? or an example suite (as the Trackmate project did )?

How do you envision the future of this awesome project ?

 Signature 

How many touches can you simultaneously perform ? 
Coming soon : EveryWall MT / Multi LaserPointers / MT SMS Wall
le WIKI en Français

Profile
 
 
Posted: 28 July 2009 03:09 PM   [ Ignore ]   [ # 4 ]
Avatar
Rank
Joined  2007-07-11
Total Posts:  29
New Member

Hi Jimihertz, I’m a colleague of Greg at the Music Conservatory of Geneva and can answer some questions.

First of all, thanks for featuring our work on the blog, that’s really nice grin

2/c) We abandoned the mirror for tuning the alignment of lasers in favor of a simpler system, with laser and lens attached together on a platform adjusted with 3 screws. We’ll have a few of those modules ready after the Summer holiday.

3) The camera is outputting a thresholded image and the tracking is done in the PC. Still, this is reducing dramatically the bandwidth and CPU usage because only a B&W image is transmitted instead of a Greyscale one (my 4 years old laptop can’t even cope at 100 fps with Greyscale mode, while it’s no problem with pre-processing).

5) Greg used Max for the two demos using a paper interface and I used a bunch of free or cheap softwares and plugins for mine (3rd demo in MMI II vid). These are Plogue Bidule (it can be used free but I opted for the 80$ or so voluntary registration because it’s a great software and its developers deserve to be supported), Native Instruments Kore II player (free), and Moebius for live looping (free as well).

As for the future of the project, first we hope that more people will start to write plugins for the Surface-Editor. It can be extended with new and unexpected components that way, also going beyond the musical application. Multitouch musical instruments are just at their beginning and what we have done and also seen elsewhere so far is a rather conventional approach, creating virtual interfaces that mimic physical controllers like faders, buttons, etc. There is still a large field of exploration to invent new controllers that exploit the full potential of multitouch, leading to new musical gestures, and that’s what the plugins architecture of the Surface Editor should favor.

Second, we are moving further in the ‘no display’ direction, with the idea to build interfaces on the fly using visually tagged cards (instead of fixed paper interfaces like in the vid). We hope to have the first demos this coming fall…

Best

Alain

 Signature 

Multitouch Musical Instruments

http://www.future-instruments.net
http://www.surface-editor.net

Profile
 
 
Posted: 07 August 2009 01:32 PM   [ Ignore ]   [ # 5 ]
Rank
Joined  2008-04-28
Total Posts:  15
New Member

hi everybody

what a nice project greg, congratulations. i’m using LLP multitouch surface and I was wondering if it is possible to adapt the code to use my ps3eye couse it works to 125 fps and I think it would be great for the rest of this community.

sorry if is a silly question

wink

Profile
 
 
Posted: 18 October 2009 02:06 PM   [ Ignore ]   [ # 6 ]
Avatar
RankRankRankRank
Joined  2007-09-18
Total Posts:  882
Moderator

HI guys,

The surface-editor.com is down since several weeks now....

the Beta period ended grrr

What should i do with my on-going projects ?

Is there any release on the way ?

Image Attachments
surface_editor.jpg
 Signature 

How many touches can you simultaneously perform ? 
Coming soon : EveryWall MT / Multi LaserPointers / MT SMS Wall
le WIKI en Français

Profile
 
 
Posted: 28 October 2009 10:27 AM   [ Ignore ]   [ # 7 ]
Avatar
Rank
Joined  2007-07-11
Total Posts:  29
New Member

Hi Jimihertz,

Sorry for that. Surface-editor.com is now redirecting to: http://www.future-instruments.net/fr/editor.html . To get a new version, without expiration date, please send a mail to editor (at) future-instruments (dot) net.

Best

Alain

 Signature 

Multitouch Musical Instruments

http://www.future-instruments.net
http://www.surface-editor.net

Profile
 
 
Posted: 03 November 2009 09:14 AM   [ Ignore ]   [ # 8 ]
Avatar
Rank
Joined  2009-11-02
Total Posts:  11
New Member

Hello, Surface-Editor development team,

I downloaded S-E, but it seems not to be fully working, the program is loading, and i can edit layouts(i liked how it’s easy)), but in play mode it doesn’t react on the OSC input out of Nuigoup CCV 1.3. When using mouse to click on elements, programs like Live, Reaktor and Midi Monitor do not receive any midi data through MIDI Yoke driver. I tested it on 2 machines with WinXP SP2 and SP3, my Java version is JRE 6 Update 16.

Best regards,

llya

Profile
 
 
Posted: 03 November 2009 03:09 PM   [ Ignore ]   [ # 9 ]
Avatar
Rank
Joined  2007-07-11
Total Posts:  29
New Member

The TUIO server (CCV in this case) must be configured to send events on port 3333. Touch events are normaly processed also in Edit mode (unless this option disabled in Edit/Play mode settings) and mouse events only in Play mode. To send MIDI events, the MIDI driver must be selected under ‘MIDI Out Device A’ in the MIDI settings window. It is possible to select different MIDI drivers for device A, B, C, etc. Then, when configuring an action (eg. MIDI CC or MIDI note), events can be sent to the desired MIDI device by selecting the corresponding letter.

 Signature 

Multitouch Musical Instruments

http://www.future-instruments.net
http://www.surface-editor.net

Profile
 
 
Posted: 03 November 2009 08:37 PM   [ Ignore ]   [ # 10 ]
Rank
Joined  2008-04-28
Total Posts:  15
New Member

{KGJ}, it’s a known TUIO/UDP problem in ccv 1.3....., nothing to do until a new compilation (read this thread). in the meantime back to the ccv 1.2

wink

Profile
 
 
Posted: 04 November 2009 08:37 AM   [ Ignore ]   [ # 11 ]
Avatar
Rank
Joined  2009-11-02
Total Posts:  11
New Member
Easy Looping - 03 November 2009 03:09 PM

To send MIDI events, the MIDI driver must be selected under ‘MIDI Out Device A’ in the MIDI settings window. It is possible to select different MIDI drivers for device A, B, C, etc. Then, when configuring an action (eg. MIDI CC or MIDI note), events can be sent to the desired MIDI device by selecting the corresponding letter.

I did it! Here is prooflink:

Profile
 
 
Posted: 04 November 2009 10:05 AM   [ Ignore ]   [ # 12 ]
Avatar
Rank
Joined  2007-07-11
Total Posts:  29
New Member

Did you set an Action? Nothing is happening in a Zone or a Controller until at least one Action is added (Add button in the Activator Tree or right click on the Component name). If for some reason you don’t see the Appearance/Action window (right window, see enclosed pictures), click ‘Restore original panel disposition’ in the View menu, and then ‘Factory settings’. It might also be that you have to restart the Surface-Editor after configuring the MIDI driver for the first time. If it still do not work send me the interface (.mds file), I’ll have a look at it. Anyway, the Editor might have some bugs, but this basic functionning is working, believe me wink

Image Attachments
Presse-papiers-1.jpgPresse-papiers-2.jpg
 Signature 

Multitouch Musical Instruments

http://www.future-instruments.net
http://www.surface-editor.net

Profile
 
 
Posted: 04 November 2009 10:33 AM   [ Ignore ]   [ # 13 ]
Avatar
Rank
Joined  2009-11-02
Total Posts:  11
New Member

Did you set an Action?

smile I didn’t

BTW is it okay what where is only that much options:

ya7wBJvSQL.jpg
?

Profile
 
 
Posted: 04 November 2009 02:06 PM   [ Ignore ]   [ # 14 ]
Avatar
Rank
Joined  2007-07-11
Total Posts:  29
New Member

This is really strange. I have no idea why the MIDI and OSC actions are not appearing in the menu, I have never seen that before :-(

Do this happens also with other controllers and zones? What if you load one of the provided examples, which is already configured with Actions?

 Signature 

Multitouch Musical Instruments

http://www.future-instruments.net
http://www.surface-editor.net

Profile
 
 
Posted: 04 November 2009 02:34 PM   [ Ignore ]   [ # 15 ]
Avatar
Rank
Joined  2009-11-02
Total Posts:  11
New Member

Hmm, maybe you uploaded some wrong build? I’ll send you link from which i downloaded it into PM.

Yes, this happens to other controllers and zones. Examples loading empty.

Profile
 
 
   
1 of 2
1
 
‹‹ I Eat Beats      our 100" multitouch wall ››