Microsoft Research
LucidTouch

Microsoft researchers are closely examining what they consider a staggering amount of wasted space on handheld devices. Where? On the flipside. Imagine a game or phone that lets you use your fingers for input on a gesture pad behind the screen. Even with your fingers hidden, you know what you're doing because they're silhouetted on the front screen—as if the device itself were transparent.

LucidTouch MAJOR INNOVATION Turns useless space on the back of handhelds into input surfaces. WHY IT MATTERS Multi-finger touch control could drastically alter interaction with phones, portable games, and other mobile devices. ESTIMATED ARRIVAL 2009

Patrick Baudisch hatched the idea for LucidTouch after finding his finger blocked the view of his touch screen. Luckily, he's in a position to do something about it; he's one of four researchers in the Adaptive Systems and Interaction research group at Microsoft Research (MSR). His focus for the last few years has been mobile device interaction, specifically how to overcome screen-size limitations. We always want devices smaller, but smaller touch screens are inherently less useful. Since 2006, Baudisch has worked closely with Mitsubishi Electric Research Labs (MERL) on LucidTouch—so closely that at least one MERL researcher has recently joined MSR.

Since a transparent finger is out of the question until Microsoft develops invisibility, Baudisch decided that the back of the device is the perfect location for a touch interface. Yet the user still needs to see his fingers to stay in control. Baudisch calls the digital silhouette that makes a LucidTouch appear see-through pseudo-transparency.

"By interacting with the back side, we get fingers out of the way. You touch the back side, you see the document on screen, you see a finger where your fingers are. It's like touching the front," he says. Controls on the back means a user's finger size is out of the equation. Even the fattest fingers can work without blocking the screen.

"Users instantly understand what it means," says Baudisch, adding that pseudo-transparency is so intuitive that there's no need to explain what's going on to a new user. A colored dot on the screen indicates the exact interaction point for each finger. It's like having eight mouse cursors under your control, all at once. The dot could be as small as a pixel, and would change color to indicate the finger is touching the pad on the back.

How does LucidTouch deliver this pseudo-transparency to the screen on a handheld? The prototypes use a cumbersome camera-boom attachment that points at the user's fingers, which are then rendered on screen. Ultimately, a set of capacitor sensors or optical sensors could do the trick. The secret is to get just enough data from the sensor to indicate locations of the fingertips.

Exactly what a user will do with all those fingers is still up in the air. The research team has tried LucidTouch with an on-screen keyboard worked by just the index fingers, plus a standard QWERTY interface with the layout rotated, so the A key is still under the left pinky. It remains to be seen what works best for users.

Baudisch's dream for LucidTouch goes beyond text-entry on mobile devices. Imagine using backside controls on a PlayStation Portable for a real-time strategy game. And with the addition of filters and smoothing technology, multi-finger controls could be useful for the disabled, especially those with motor disabilities such as Parkinson's disease. LucidTouch could even become the foundation for a keyboard/mouse interface to use with any device, from the tiniest phone to the most powerful PC.—Next: Graspables >