Custom Multi-touch Interfaces

We have constructed multi-touch display, based in principle on a design by Jefferson Han's team at NYU that uses frustrated total internal reflection of infrared light.

Music Interfaces

A short video clip of some MTD music interfaces in action (click to play)

Click to Play

Medical Image Navigation

In addition to the musical interfaces, a multi-touch application was developed to allow pathologists to directly interact with high resolution medical images. In a modern medical facility’s pathology department, it is essential to be able to easily view and manipulate large numbers of high-resolution medical slide images. Since these images increasingly are created and stored digitally, the computer is becoming the primary device for navigating and interacting with these slides. For decades human interaction with computers has been mostly limited to a keyboard and mouse, but recent developments in simultaneous multi-touch display (MTD) technology allow a user to interact with and manipulate objects directly on the screen in an intuitive manner. High-resolution medical images are too large to load entirely into memory, so only portions of an image are loaded at any given time determined by the area being viewed. In combining the MTD with high-resolution medical imaging technology, we have developed an interface that allows a pathologist to directly interact with an image through touch and simple gestures for panning and scaling, creating a highly intuitive and interactive image environment.

This medical image project was presented at Drexel University Research Day 2008 and won Best Undergraduate Poster Presentation in Computation, Modeling and Simulation.

Research Day Poster 2008

Medical Image Application (click to play)

Click to Play

Flash Library

Recent advances in multi-touch technology make a practical, full-scale multi-touch surface a possibility in the near future. However, most research in this technology is done by private companies or small groups of hobbyists. These unconnected research groups work separately, and oftentimes redundantly design and tweak all aspects of the technology. Some progress has been made to alleviate these inefficiencies through the development of open source, easily usable software to interface with a multi-touch surface. Our goal is to take advantage of existing open source tools to create a platform for the rapid prototyping and development of multi-touch applications.

To further simplify the application development, we expanded upon the popularly used open source TouchKit API. A simple program was written to allow for this application to communicate with other outside programs. This “wrapper” application allows us to send information packets from the tracker, over a computer’s network. The tracker application is now able to communicate with any other user application able to receive these network packets.

Flash is becoming a popular medium for rapid prototyping of cross-platform, visually appealing applications. Flash includes the ability to draw 3-D graphics and to generate dynamic sound, which makes Flash an accessible, yet robust platform. Flash is mostly used in web development; however, using Adobe AIR, a free cross-platform Flash runtime environment, the developer can create desktop Flash applications, as opposed to running them through a web browser.

To take advantage of multi-touch features in Flash, we developed a touch-interface library to receive touch network packets. This library recognizes touch types and gestures and allows a developer to easily incorporate them in their own Flash applications. Using this simple library, programs can respond to a user's gestures, such as rotating an on-screen 3-D cube by physically grabbing and twisting, or by generating a changing tone or chord based on the up and down movement of a user's touches. 3-D graphics, audio, and gesture recognition can be combined to create a rich, easily created, cross-platform, multi-touch experience.

A poster about this was presented was presented at Drexel University Research Day 2009 and once again won Best Undergraduate Poster Presentation in Computation, Modeling and Simulation. There is a link to it below.

Research Day Poster 2009

Updated Design

In the original design, the display was mounted upright so that the user could stand and interact with a virtual wall like display. This works well for demonstration purposes, however, for the average person using the MTD as a tool for long periods of time, the standing interaction may be cumbersome. Due to an increased time period of solo user interaction, such as in the Medical Image Viewer, a different design approach may be taken to allow a more comfortable use of the MTD.

The main difference in the two designs is the way the user more comfortably interact with the display. The new design is set up similar to a drafting table at which the user can sit at comfortably and interact with. A short throw projector sits at the base of the table and throws an image to a tiltable acrylic screening surface. IR Light is reflected through this surface, and when touched reflects the IR light to a camera. A computer calculates touch points and controls the users applications.