Stereoscopic Display Research
at the University of Toronto

Paul Milgram*
Dept. of Industrial Engineering
University of Toronto

Insight: Newsletter of Visual Performance Technical Group of Human Factors and Ergonomics Society, Vol. 13(2), June 1991.


Research on the benefits of stereoscopic displays, as well as the developing of novel stereoscopic display concepts, have been underway at the Teleoperation and Control Laboratory of the Department of Industrial Engineering at the University of Toronto since 1987. At that time, two independent projects were started, one to develop a stereoscopic computer graphics (SG) display capability, and the other to develop a closed circuit stereoscopic video (SV) system. In this short paper results of progress in those two projects are reviewed, and a summary is given of our ongoing efforts to merge the two technologies (SV+SG), to create our own version of virtual reality.

Our stereo computer graphics (SG) research has centred on comparing the practical benefits afforded by binocular disparity cues with those offered by alternative monoscopic cues. In particular, we have concentrated on comparing the effectiveness of SG relative to the powerful object rotation depth cue, as well as examining the benefits using both cues in combination. Having been motivated initially by the potential for using stereoscopic angiograms to assist neurosurgeons in visualising the structure of blood vessels in the brain, an experimental task was developed in which subjects are required to trace a path through a network of intertwined three dimensional tree structures, a task which presupposes perception in depth of connectivity between branches (Sollenberger & Milgram, 1989). To date, a set of four experiments have been carried out, all of which not only have demonstrated the effectiveness of each cue in isolation but support also a "super-additive" model of how the visual system combines these two important sources of depth information (Sollenberger & Milgram, 1991).

Our stereoscopic video (SV) research originated as a contract with the Defence and Civil Institute of Environmental Medicine (which continues to support our work) to construct a closed circuit SV system and demonstrate the usefulness of the concept for assisting operators of mobile explosive ordnance disposal robots. Rather than add to the existing collection of experiments in which the superiority of SV displays for telemanipulation tasks has already effectively been demonstrated, typically for well-trained subjects, we decided to look instead at whether SV can affect the relative ease with which operators learn to operate such systems. Two experiments were performed to examine the role of mono vs stereo video systems and operator experience on performance of a set of simple remote driving and placing tasks. In general, it was found that the benefits of SV are unequivocal for more difficult tasks, but are very much dependent on earlier experience for simpler teleoperation tasks (Drascic & Milgram, 1991a).

As a follow-up to our original SV experiments, in which a fixed camera alignment was used, we are currently addressing questions concerning the usefulness of having a dynamically adjustable SV camera mount. Such a feature could theoretically provide an "optimal" camera configuration at all times, but might also conceivably present operators with a perceptually disorienting visual world. Our current experiments are examining the effects of dynamic camera alignment changes on judgement of depth (Krüger & Milgram, 1991).

Our initial effort to merge our SG with our SV capability culminated in the creation of a "virtual stereographic pointer", a tool for exploring and making measurements within a real three dimensional video world (Milgram et al, 1990). As stated earlier, we are using the term "virtual reality" to describe our SV+SG system, based on the observation that our computer generated virtual SG images (when properly drawn) appear to be really physically present within the video world being viewed, and to move about, in three dimensions, as would real physical objects in that world.

The development of the SG pointer was motivated by the need of operators of remotely controlled systems to be able to perceive not only relative locations in space of objects in the remote world, but also absolute {x,y,z} locations in 3D space. The former need is typically provided for by SV displays; the latter is not. Our SG pointer acts effectively as a "virtual tape measure", which is superimposed onto the real-world SV image seen by the stereo cameras. The virtual tape measure is used by aligning it with a starting point, "stretching" it across the video scene and measuring the 3D displacement of a selected end point. What we have developed, in other words, is a virtual instrument for making real measurements in a real world.

In response to the critical question concerning the accuracy with which operators will actually be able to align such virtual pointers (which of course can be allowed to take any form and any colour, at any orientation), a (2x2 factorial) experiment was performed in which alignment performance with a complete set of real and virtual pointers and targets was tested. The results of that experiment showed that, under the particular minimal depth cue laboratory conditions imposed, subjects were indeed able to align all combinations of virtual and real objects approximately equivalently (Drascic & Milgram, 1991b).

We are currently addressing the alignment accuracy problem, by adding a frame grabbing capability to the system, which will enhance the reliability of the system by creating a means of reinforcing the perceptual capabilities of the human and machine elements. That system will permit the operator to specify real-world points, which will then be measured and confirmed via on-screen (SV+SG) display by the computer. Ultimately, we intend to develop a capability for "virtual control", by which an operator will be able to indicate path trajectory waypoints through the real world and/or control a virtual 3D simulated manipulator, that will be followed subsequently by the real manipulator (Milgram & Drascic, 1991).

Going beyond the realisation of simple pointers and linear trajectories, the opportunity exists for realising a broad new class of previously unachievable "on-screen" displays, such as predictor displays, guidance displays, etc. Furthermore, one may carry out CAD-like analyses of the impact of humans and/or equipment on real workplaces, without necessarily having a world model of that workplace, by superimposing a computerised SG operator model, for example, at the actual SV work site. Such a capability presupposes, of course, the ability to superimpose fairly complex graphic (wireframe) models onto the SV images. At present, in the context of an advanced vision system for space telerobotics, we are developing a means of superimposing wireframe images of actual target objects being manipulated by a robotic arm and tracked by an independent tracking system, in order to enhance the images of those real-world objects and permit head-up display of information such as proximity warnings (Maclean et al, 1990).


REFERENCES

Drascic, D. & Milgram, P. (1991a): "Transfer effects in skill acquisition using monoscopic and stereoscopic video for teleoperation", forthcoming in Proc 35 HFS Annual Meeting.

Drascic, D. & Milgram, P. (1991b): "Positioning accuracy of a virtual stereoscopic pointer in a real stereoscopic video world", Proc. SPIE Vol. 1457, Stereoscopic Displays and Applications II.

Krüger, M. & Milgram, P. (1991): "Issues in the use of interactive stereoscopic displays for exploring hazardous environments", forthcoming in Proc. 11th Congress of IEA.

Maclean, S.G., Rioux, M., Blais, F., Grodski, J., Milgram, P., Pinkney, H.F.L., Aikenhead, B.A. (1990): "Vision system development in a space simulation laboratory", Proc. ISPRS (Int'l Soc. Photogrammetry & Remote Sensing), Commision 5: Close Range Photogrammetry & Machine Vision; Zürich.

Milgram, P. & Drascic, D. (1991); "Enhancement of 3-D video displays by means of superimposed stereo-graphics", forthcoming in Proc 35 HFS Annual Meeting.

Milgram, P., Drascic, D. & Grodski, J. (1990): “A virtual stereographic pointer for a real three dimensional video world”, in Human-Computer Interaction: INTERACT’90, D. Diaper, D. Gilmore, G. Cockton & B. Shackel (Ed's), Elsevier Science, 695-700.

Sollenberger, R.L. & Milgram, P. (1989): "Stereoscopic computer graphics for neurosurgery", in Designing and Using Human-Computer Interfaces and Knowledge Based Systems, G.Salvendy & M.J. Smith (ed's), Elsevier Science.

Sollenberger, R.L. & Milgram (1991), P.: "A comparative study of rotational and stereoscopic computer graphic depth cues", forthcoming in Proc 35 HFS Annual Meeting.