Perceptual Localization of Surface Normal

Ming Hou

Ergonomics in Teleoperation and Control (ETC) Laboratory

Department of Mechanical and Industrial Engineering

University of Toronto, ON Canada M5S 3G8


The objective of this study is to find an effective evaluation method for human perceptual localization of surface normal by investigating the influences on positioning accuracy of a 3D probing tool from both surface features and the probing tool itself. Although stereoscopic displays allow enhanced depth perception and object identification, surface contour of objects cannot be easily determined. The research results will not only increase understanding of human visual perception with stereoscopic display, but will also provide insight into human-machine interaction in teleoperation tasks using augmented reality.


Visual perception, surface normal, stereoscopic display, virtual pointer, augmented reality, human interaction


How can a neurosurgeon accurately measure an aneurysm in a 3D world so that a blood vessel can be filled with a clip that is the proper size? How can an operator of remotely controlled systems align with a round rock so that the rock can be drilled and broken accurately and effectively? In order to solve these kind of problems and make absolutely measurements of what the viewer sees in a stereoscopic scene which shows only the relative position of objects, Drascic and Milgram developed a technology called augmented reality through graphic overlays on stereo video--ARGOS. To measure distance in the real world stereo video (SV) image, a stereographic pointer (virtual pointer) could be manipulated along X, Y and Z axes consecutively at two features of interest in the SV image [1]. Through the experiments, they found that people could align virtual pointer essentially and accurately as well as they are able to align an equivalent real pointer. But when the pointer went behind a surface, a contradiction between the binocular disparity cue and the apparent monoscopic occlusion cue was created, resulting in a double image; that is, the eye no longer fuse the left and right images.



Another phenomenon in augmented reality is that observers are difficult to align with the normal direction of a curved surface with 3D displays. The reason is partially because the combination of real and virtual objects can sacrifice traditional depth cues such as consistent light and shadow, texture gradient and distant object occlusion.

Human observers can infer size, depth and orientation information using a variety of visual depth cues, and including properties of object surfaces. In natural vision, perceived surfaces can be redundantly specified by many different surface characteristics, and depth cues do not ordinarily conflict with each other. Consequently, the human visual system has the ability to infer the normal direction of a curved surface. However, the question of how we are to evaluate this ability remains? Is there any mental model for estimating the normal direction of a curved surface? For that matter, are there perhaps specialized surface normal detectors in the human visual system? In other wards, what kinds of depth cues and which aspects of surface features have influences on human perception of surface normal?

Having been motivated by these problems and the potential for using stereoscopic display to assist neurosurgeon and operator visualizing the structure of blood vessels and round rock surfaces in the brain, this research is trying to find an effective evaluation method for human perceptual localization of surface normal in augmented reality using a 6 degree-of-freedom (dof) virtual tool -- stereographic pointer.

Figure 1 Virtual Pointer Localization of Surface Normal


The normal direction of a surface must be relative to some specific small area or even a single point in this area and on this surface. In order to localize the initial position and draw a normal line, a surface must have an appropriate level of structural/surface complexity, at an appropriate scale to be detected by the human visual system. One possible hypothesis to consider about the effects of surface complexity is that the estimation precision varies as a fixed proportion of the smallest spatial scale on the surface to which human observers are perceptually sensitive [2].

To draw a line normal to a surface when see surface is partially viewed from different orientations, observers may use mental rotation to match the mental shapes with familiar models at comfortable viewpoint according to surface features. Obviously, the surface orientation relative to the observer is an important surface property. Phillips and Todd studied the abilities of observers to localize the position of individual surface points viewed from different orientations [2]. They found that if a surface moves relative to the observer, or vice versa, the depths and orientations of each local region would change, but the magnitudes of their principal curvatures would remain invariant. Their conclusion is that the curvature is the most perceptually relevant property for localizing surface position.

Another two aspects of surface features are shading and texture. Todd and Mingolla conducted three experiments to investigate the shading effects on cylindrical surfaces with and without texture [3]. Their results indicated that the shininess of a surface enhances the perception of curvature, but has no perceived direction of illumination; and that shading is generally less effective than texture for depicting surfaces in three dimensions. A further study investigated how patterns of optical texture provide information about three dimensional structure of objects in space [4]. One of their experimental results revealed that judged depth increases linearly with simulated depth, although the slope of this relation varies significantly among different types of texture patterns.



Considering the situation for drawing a normal line departing from a surface, observers will not only watch the local characteristics for localizing a starting point, but also simultaneously look at the global features around this specific area for adjusting the orientation of the normal line. Therefore, two groups of surface features should be investigated: local features and global features. The former group emphasizes textures and curvatures as well as depths, and the latter group will stress orientation, size, shading and luminance as well as binocular disparity. In addition, a six dof virtual pointer is essential for probing a surface normal with different orientations. The hypothesis is that the characteristics of this special tool can promote the convenience and efficiency for observers' task, such as the shape, the color contrast and textural difference between virtual pointer and target surface, etc.

In order to study the human perceptual localization of surface normal, subjects will be asked to manipulate a six dof virtual pointer to draw a normal line on a specific surface. The proposed investigating aspects are : first, to measure the precision with which subjects are able to draw a normal line relative to individual surface under a variety of conditions; second, to examine how this precision is influenced by the local surface features (such as curvatures, textures and depths) and the global surface features ( such as size, orientation, shading and luminance as well as binocular disparity); third, to examine the influence from the characteristics of virtual pointer, such as the shape, the color contrast and the textural difference between the virtual pointer and the surface, and the length and thickness, and fourth, to examine how localization errors covary with different conditions and their interactions.


This dissertation should provide the field of human computer interaction with an empirically validated evaluation method that people can draw a normal line for a curved surface. The most important significance of this contribution is to increase our understanding of human perception of surface normal and help interface designers by explaining how people conduct visual search in stereoscopic displays by providing theory needed to design graphical objects or images of real objects to help manipulating a probing tool in augmented reality. The practical application of this project is to use a virtual probing tool effectively guide the drill in a variety of mine faces in a remote mining application.



This study is supported in part by Canadian NSERC Fellowship and IRIS-SMART project. I greatly appreciate Paul Milgram's guidance in this research.



  1. David Drascic & Paul Milgram. Position Accuracy of A Virtual Stereographic Pointer in A Real Stereoscopic Video World. SPIE 1457 - Stereoscopic Displays and Applications II, 1991, 302-313.

2. Flip Phillips, James T. Todd, et al. Perceptual Localization of Surface Position. Journal of Experimental Psychology: Human Perception and Performance, 1997, Vol. 23, No.5, 1481-1492.

3. James T. Todd & Ennio Mingolla. Perception of Surface Curvature and Direction of Illumination From Patterns of Shading. Journal of Experimental Psychology: Human Perception and Performance, 1983, Vol. 9, No. 4, 583-595.

  1. James T. Todd & Robin A. Akerstrom. Perception of Three-Dimensional Form From Patterns of Optical Texture. Journal of Experimental Psychology: Human Perception and Performance, 1987, Vol. 13, No. 2, 242-255.