Whenever Stimulant embarks on a new installation project, we survey the landscape of hardware solutions to determine what we can use to create a robust and immersive experience. Our offices are full of touchscreens, sensors, video cards, and gaffers tape and we’re constantly putting those pieces together in different ways to see what we can accomplish.
Our team has been spending a lot of time lately with depth-sensing cameras, which have gone from being a futuristic dream to a mature product category since the release of the Microsoft Kinect in 2010. We’ve worked with all of these devices in an array of different projects. How does each device fare? Read on.
The Original: Microsoft Kinect
Version: Microsoft Kinect for Windows SDK 1.7
It didn’t take long for the Microsoft Kinect accessory for XBox 360 to be reverse-engineered by the open-source community and repurposed for use in a greater context instead of just games. Fortunately, Microsoft decided to foster that community rather than fight against it by launching Kinect for Windows, a slightly-tweaked version of the hardware combined with a fully-featured set of development tools. The Kinect is actually a package of sensors, in addition to the depth-sensing camera there’s an infrared camera, an RGB camera, and an array of directional microphones.
Microsoft Kinect Advantages:
- Skeletal tracking: Applications can track the positions of user’s joints (head, shoulders, hips, hands, etc.) in space. Two people can be tracked simultaneously, in either standing or sitting poses.
- Face tracking: Various attributes of the user’s face can be tracked, including the relative positions of lips and eyebrows, which can be interpreted as facial expressions.
- Multiple sensors: Multiple Kinect sensors can be used together in order to track more users simultaneously, or to get a wider view of a space.
- Raw data: The Kinect SDK provides access to the raw depth data from the sensor, as well as images from the IR and RGB cameras. This can be processed by the application developer as they see fit.
- Voice control: The microphone array in the sensor can be pointed toward the user to better capture speech. Developers can pre-define a set of commands constituting a grammar for an application. Users are informed when one of a set of commands has been spoken.
Microsoft Kinect Issues:
- Compared to other sensors, the device is fairly large making it more difficult to conceal in an installation.
- A dedicated power cord is required because the sensor uses more power than can be provided by a USB bus. The amount of data generated by the sensor also tends to saturate a USB controller, so it’s recommended that each sensor be on its own controller.
- The sensing resolution is not as advanced as other sensors on the market. For example, the Kinect cannot easily distinguish individual fingers on a hand, which means gestures tend to involve more gross movements than simple pointing.
- Up to six users can be recognized in the field of view of the sensor but only two users can be tracked in detail.
- While there are open-source toolkits for working with the Kinect on non-Windows platforms, most of the features listed above require the Microsoft SDK, which is only supported for desktop applications on Windows 7 and 8.
- Kiosks, installations, and digital signage projects where the user will be standing fairly far away from the display.
- Windows development environments.
The New Challenger: Intel Perceptual Computing SDK 2013
Version: Intel® Perceptual Computing SDK 2013
Intel’s Perceptual Computing SDK 2013 aims to provide many of the same features as Kinect in a smaller package and a less Microsoft-centered toolset. Intel recently opened up the Perceptual Computing Challenge to encourage development of innovative multimodal interfaces with this SDK. The hardware is designed for tracking of a single seated user at close range and includes RGB and depth cameras as well as dual microphones.
Intel Perceptual Computing SDK 2013 Advantages:
- Smaller and less expensive: The Intel camera (produced by Creative) is smaller than a Microsoft Kinect for Windows sensor, is powered over USB, and is designed to sit on top of most computer displays.
- Close-range tracking: It is specifically built for close range tracking, with a range of 0.5ft to 3.25ft (and a diagonal frame of view of 73 degrees).
- Hand posture/gesture recognition: The SDK allows recognition of hand postures like thumbs up, thumbs down and peace, and gestures like waving, swiping and circling with a hand. Other gestures like grab and release or pan and zoom can be implemented by examining the openness of the palm and fingertip positions.
- Facial analysis: The Intel SDK provides capabilities for face tracking, recognition and detection as well as age and gender determination. Expressions like smiling and winking can also be detected.
- Speech: Developers can leverage speech recognition by specifying a predefined list of commands, or multiple lists constituting a grammar. The SDK also has built-in support for speech synthesis powered by Nuance.
- Raw data: Developers have access to raw color and depth data from the sensor along with a confidence map to account for distance and light conditions.
- Framework support: The Intel SDK supports frameworks like Processing, Unity and OpenFrameworks and ships with basic examples to make setting up and writing simple apps with these frameworks easier.
- Getting some of the deeper features (like age and gender detection) to work is a bit tricky.
- Due to the close range of the tracking system, hand gestures must be designed such that a user’s hand doesn’t occlude their own view of the display.
- Desktop/laptop applications where the user will be seated in front of the PC.
- Close range applications where features, apart from hand tracking and recognition, are necessary without too much precision or accuracy.
The Small Wonder: Leap Motion
Version: Leap Motion SDK v 0.7.7
The Leap Motion controller is a tiny unit designed to be placed below and in front of a display, and uses an infrared technique to determine the position and orientation of fingers (or finger-like things, such as pencils) while they are positioned over it. Tracking is very accurate and fast, and the device can be calibrated to map fingertip positions to precise locations on screen. Unfortunately, the device’s range is quite limited at 8 cubic feet, but it seems like quite a bit less in our experience. There is also no access to depth data or a point cloud. If what you want is very accurate finger tracking, the Leap Motion is perfect, but it’s not very well-suited for much else.
Leap Motion Advantages:
- Gesture recognition: Finger tracking is fast and accurate.
- Even smaller and less expensive: The device is physically very small and is inexpensive.
- Developer friendly: The “Airspace” application store provides a way for developers to market and distribute Leap apps.
- Framework support: The Leap supports a number of frameworks, including .NET, Processing, Cinder, etc.
- Compatible: Works on both Mac OS and Windows.
Leap Motion Issues:
- Sensing range is fairly limited.
- Only fingers are tracked. There is no skeleton or face tracking.
- No access to the raw sensor data.
- Controlled kiosk environments with a pointing-based UI.
- Generally best for general audience desktop apps which can be distributed in the Airspace store.
We’ve discovered that while there are a number of interesting camera-based sensors available, each tends to specialize in a few features, which are different from the others. For standing skeletons, Kinect is the only way to go. For seated voice and gesture, the Intel device may be a better choice. For precise pointing, the Leap is accurate and fast. It’s also possible to combine multiple devices. What could we do with two Kinects combined with a Leap, for example?
At Stimulant we always have a dual goal of creating amazing interactions beyond the typical mouse and keyboard, but we also require our installations to be bulletproof enough to withstand sustained public use (and abuse). These robust and affordable systems open up a huge range of possibilities for our work, and hopefully yours as well. Share your experiences with depth-sensing devices with us on Twitter @stimulant.