MITRE immersive spherical vision system

Immersive vision systems for teleoperation are a valuable tool for many applications including inspections of old buildings, pipelines and sewages, search and rescue, and military, i.e., detection and neutralization of roadside bombs. Immersive systems work by presenting a virtual representation of the world seen via a camera that is situated away from the operator (note: I use the term virtual representation of the world liberally here because what the operator sees is actually images of the real world captured by a remote camera; I call it virtual because a explained later these images arrive delayed which means that the real world may have change since the data was collected.)

The operator views slices of the virtual world using a head mounted display while a sensor detects his movements and updates the view accordingly. A large problem with such methods is the latency between the operator moving his head and the system updating his view; the latency often comes from the fact that the remote camera has a limited field of view meaning that every time the operator moves his head, a pan-tilt mechanical unit has to reposition the camera delaying the relay of the images and making it difficult to operate the remote system often causing lots of distraught for the operator.

MITRE scientists have worked out a solution to this latency problem replacing the limited field of view camera on the pan-tilt unit with a spherical vision camera which has no moving parts. The camera of choice is the Ladybug commercially sold by Point Grey Research in Canada. Stanford's Urban Challenge autonomous car also used the same camera for part of its perception system. This spherical vision system consists of 6 cameras which capture images simultaneously covering a large portion of the view sphere around it. Software stitches the images together into a single view in real-time. These spherical vision images are then available to the operator to view for any orientation of his head. The latency I mentioned earlier is thus eliminated by the fact that the camera need not be repositioned every time the operator moves his head. Moreover, more than one operators can be using the system looking in different directions.

The below promotional video shows the capabilities of the immersive spherical vision system including some of its potential applications. The true power of the system is clearly visible in the part of the video where a car driver is shown driving down a street while perceiving the world in real-time via a head mounted display.