The SLIPstream project aims to enable interactive applications driven by real-time processing of high-rate streaming data. Examples of such applications include unconstrained gesture recognition based on frame-rate spatio-temporal event detection and robot control and actuation based on real-time object recognition in video streams. Such interactive and actuation applications are computationally demanding, and require both high computational throughput and low-latency operation of the vision systems. A key component of SLIPstream that attempts to satisfy these requirements is Sprout, a runtime for parallel stream processing that parallelizes vision tasks across a cluster of compute nodes. Unlike traditional schemes for parallelizing computation, Sprout does not simply replicate processing stages to maximize throughput; rather, Sprout applies intelligent replication with careful refactoring of tasks so as to minimize latency. The SLIPstream project is investigating various techniques for runtime adaptation, refactoring applications, and constructing algorithms amenable to such techniques.
An important application focus of the SLIPstream project is creating novel natural user interfaces, such as multimodal gesture/speech interfaces where the user points to devices in the environment and then controls them using voice commands. Another area of interest is simultaneously processing the input from hundreds of video streams, such as those generated in a virtualized reality studio. Since many computer vision problems become easier when the sensor sampling density is increased (whether spatially or temporally), we seek to enable real-time 3D reconstruction for large-scale multi-user environments, where people can interact with each other and the space without props, awkward wearable tracking devices or motion capture markers.
The SLIPstream project collaborates extensively with the Everyday Sensing and Perception (ESP) project and is strongly aligned with efforts in Computational Perception and the Data Rich Theme at Intel Labs.