Multi-touch interface design

[Bornish]

I've seen a couple of presentations / demos about this technology; here's a link for one of them: http://www.ted.com/index.php/talks/view/id/65.
I'd like to learn more about what is involved in making applications that support such devices. If anyone has (or can find) any kind of information about how it works and if there's a direct communitation between the applications and its drivers, please share them here, in this thread.
I wouldn't mind giving your two cents about what do you think about such interfaces being used widely in the near future, and how do you see us (software developers / architects) in this "transition". How easily do you think we can change the UI design / architecture of an application if we're planning to support such kind of user interaction? Do you think the manufacturers will provide these with some software package (like DirectInput) that can be easily integrated into existing applications? Would that sufice for adding the necessary support, or will we have to redesign entirely some of our products since most of the thread balancing will change, as well as any kind of input-process relationship? Unfortunatelly I can't provide a link to the other demo (is ~8.5 Mb in size) but the interesting thing observed in it was that, simultaneously, more than one person can use these multi-touch displays, handling more than one application at the same time. In such environment, I believe that concepts like "foreground window" are utterly removed.

Any info / opinion on the topic will be greatly appreciated.

// I had no idea which forum is best for this thread... hope it doesn't hurt being in the Gen Dev Topics

Thanks,

[Bornish]

Found a link to the other presentation (Perceptive Pixel) on YouTube:
http://www.youtube.com/watch?v=ysEVYwa-vHM&mode=related&search=
... and another one proving an amazing reaction speed and interactivity:
http://www.youtube.com/watch?v=Q77Cuau28I0&mode=related&search=
Jefferson Y. Han has a short description site about his Multi-Touch Interaction Research (http://cs.nyu.edu/~jhan/ftirtouch/index.html), where also talks about the refined sensing technique, called "frustrated total internal reflection" (http://cs.nyu.edu/~jhan/ftirsense/) (FTIR) and he only mentions that the stroke event information is sent to applications using lightweight OSC protocol over UDP.

Apple has introduced the technology in the iPhone: http://www.apple.com/iphone/technology/. One interesting issue about phones' displays with multi-touch support is that software must differentiate between finger touches and cheek ones, while talking on the phone. :)

Microsoft has a similar product (Microsoft Surface (http://www.microsoft.com/surface/)), probably launching at the end of this year, which they say will also provide wireless conectivity with other devices.

I've found a bit more resources about (multi-)touch panels / displays, one even augmented with computer vision (camera watching your hands movement on the display table), but all lack in focusing or detailing their work.

Am I the only one here interested in the subject? :confused:

Regards,

[PeejAvery]

Personally, I think that multi-touch depends greatly on what it going to be accomplished. Apple uses it for a phone that depends on multiple fingers for various reactions of a phone and some side apps. Microsoft is using it for multimedia based entertainment. I am a web developer and graphic designer. To design, I don't want to use my fingers, I want a mouse in my hand for precision.

Unfortunately, the products that make use of multi-touch are not for the average person's wallet.

[dglienna]

Don't forget about the MS TABLE thingie. It talks to cellphones and credit cards that get dropped on it.

[PeejAvery]

Don't forget about the MS TABLE thingie. It talks to cellphones and credit cards that get dropped on it.
That was already mentioned. It is called Microsoft Surface (http://www.microsoft.com/surface/).

[Zaccheus]

At work I write software which runs on touch-screen kiosks, so I'm following this with interest.

I think multi-touch interfaces will mainly be used in the kinds of environments where single-touch screens are currently used. That is to say, it will change a small portion of the current market quite radically.

I have no idea how the multi-touch input is handled, but I could imagine something like a session id being used, because 'mouse-down', 'mouse-move', 'mouse-up' which are done with the same finger need to be associated with each other.

Also I'd like to know how multi-touch input deals with 'bounce' where the user's finger briefly leaves the surface.

[Bornish]

Thanks guys for participating in this discussion. I wrote here because I'd like to hear developers and SW achitects oppinion about it. My intuition tells me that it may affect many of us, depending on some aspects: production costs and how existing applications will be changed to support such devices. In a way, it is like someone is inventing today a processor which isn't much faster, but supports "special" features, lets say some form of AI, which none of the existing applications are using. Would be probably used soon in some research applications, started from scratch, but if it "hits" the market depends on the its price and how many existing "code" can benefit from it.
@PeejAvery: I've noticed that in one of the demos, Jeff Han is actually using some "object instead of his fingers. As far as I understood, the multi-touch interface is presure sensitive and also has great resolution. I believe would be a much more accurate device to use for graphic designers, and definitely more intuitive. The same demos are showing web browsing. Imagine web pages having less limitations than today's ones, where their components are "movable" and the whole page is treated as a piece of paper, grabable, foldable. Have a look at the "BumpTop desktop" (http://www.ted.com/index.php/talks/view/id/131) presentation, and imagine how many environments can make use of it.
@Zaccheus: absolutely agree with the big changes in how events should be handled by such applications. That's why I'm surprised to see in these demos, existing applications already using it and even without loss of speed. I would assume that really powerful machines are used at these presentations, sure... but handling a few hundreds of live streaming video windows, with perfect multiple interaction from the user, is indeed an amazing achievement. While single touch panels can easily replace other pointing devices, and as PeejAvery mentioned, not many of them are so accurate, multi-touch can't be simply handled as single event with parameters (location, time,...). I recall when "gestures" were introduced to enhance human interaction with a machine, since simple button actions combined with movement were not enough. Now we get something extra: pressure sensitivity, area of interaction (location isn't a single point), speed / acceleration of movement (kind of gesture) and "unlimited" control points (remember in one demo, those shapes being moved by changing the location of some control points).
I can see most of the "industry" being changed by such enhancement, depending, at least, on the two aspects I've mentioned above. We can continuously struggle to build faster / better / smarter applications, but without enhanced UI, we will remain quite limited. I feel that not much was done on this aspect of computing... until this! More and more ideas are coming on the market, for example the nintendo controllers, or those game systems (forgot how are called) where your character moves as you do, by being "watched" by a camera. I think people are tired of grabbing a mouse, trying to move it on top of a small button, press it... and then look at the application for too long, while displaying some sort of "wait for processing". Too many applications are gracefully lacking at UI responsivity. Ok, I'll stop now this kind of "non-sense", but please try to see some kind of hope in such research.

PS: still looking for more details about how this works, how we can make use of it, and what is required from "our" side.

Thanks for reading and your feedback,

[PeejAvery]

Yeah, I first heard of Bumptop in the beginning of 2006. I thought it was neat.

I still disagree, fingers do not have needlepoints, unless your Freddy Kruger. The precision that can be found in a laser mouse of high DPI cannot be accomplished by the moving of a finger. Yes, it can be very sensitive, but overly so in many ways. It would be to effect of having to use a trackpad in Photoshop.

[PeejAvery]

Speaking of multitouch fun...build your own (http://www.instructables.com/id/S3S8UI1F3JESX18/)!

[Bornish]

I still disagree, fingers do not have needlepoints, unless your Freddy Kruger.LOL :D
So, take a needle in your hand and use it on the multi-touch display. In fact, take 3 needles and draw 3 parallel curves at the same time, if you need.
Well, I wouldn't consider myself an expert in accuracy, but I've build a stereo-viewing engine a few years ago, and one of it's requirements was sub-pixel accuracy. What I can tell you about it is that, no matter the pointing devices being used, the accuracy is obtained by choosing an appropriate displaying scale while having a very good displaying resolution, backed up by some techniques as antialiasing. That's why such systems that require extremely high precision must have ajustable input sensitivity (stereo-viewing requires different settings for XY movements than for Z movements) and a clutch functionality that can switch from 2 predefined sensitivity settings.
Thanks for your input (and the link to the "instructables").
Best wishes,