This work was partially supported by NIH grant TW-05753. I would like to thank Professor Myron Braunstein for his helpful comments.
1Visual motion is a powerful source of information about navigation through the environment, collision avoidance, segmentation and binding of objects parts, etc. It also provides information for the recovery of the objects’ three-dimensional shape from the two-dimensional optic flow (e.g. Wallach and O’Connell, 1953), a phenomenon called structure-from-motion (SFM). The existing theoretical studies and experimental evidence (e.g. Todd, 1999 for review) suggest that, in general, the optic flow does not contain sufficient information for the metric recovery of the object shape. Moreover, the recovery of SFM seems to be based predominately on the information specified in the first-order structure of the image displacements, allowing the extraction of mainly the affine-invariant image features of the stimulation. If the motion is confined to a plane, it is possible to obtain a unique rigid three-dimensional interpretation of the object motion (Hoffman & Flinchbaugh, 1982; Lappin, 1990).
2Most of the existing theoretical models on the recovery of structure from motion are feature based (e.g. Hildreth, Ando, Andersen & True, 1991), but the experimental evidence suggests that human observers can reliably discriminate changes in the object structure based on the information provided from the occluding contours, the shading pattern or the specular highlights (Norman, Todd & Orban, 2004) or to detect inconsistencies between the motion of a rotating object and the motion of the highlights (Hartung & Kersten, 2003). In these conditions there is no projective correspondence of the image features in the successive frames. Norman et al. (2004) suggested that the subjects exploit the existing relationship between the surface curvature and the magnitude and direction of motion of the specular highlights or of the local maxima in the diffuse shading pattern. They indicated that the ability of the human observers to use shading information may be based on more qualitative aspects of the 3D structure like ordinary, affine and topological relations. These studies suggest that the human visual system is competent about the relationship between the brightness variation and the motion of an object.
3In pilot studies, however, we observed that slanted planar surfaces, rotating around a vertical axis appeared non-rigid when textured with luminance gratings with gradual changes in their brightness pattern. These observations contradict the theoretical predictions that the three-dimensional structure of planar surfaces could be recovered uniquely. They do not support also the theoretical considerations about the recovery of the affine structure in SFM displays based on the motion information or on the relationship between the surface curvature and the displacements of brightness maxima.
4The aim of the present study was to obtain quantitative measures of the observed effect and to specify its dependence on the steepness of the luminance gradient, the direction of motion and the slant of the surfaces, as well as to test some of the possible explanations of the observed phenomenon.
5Participants:Six naïve observers from the staff of the Bulgarian Academy of Sciences (BAS) with normal or corrected to normal vision participated in the experiment.
6Stimuli: A set of five luminance gratings with periodically changing brightness pattern (“luminance profile”) were generated. Their horizontal waveform consisted of triangular impulses with different skew (Figure 1). Five values of skew were used: -1.0, -0.5, 0.0, 0.5, and 1.0. When the peak of the brightness variation was in the middle of the triangular impulse, its skew was 0.0; when it was at an endpoint of the impulse, the skew had magnitude of 1.0, while when it bisected the interval between the endpoint and the midpoint the skew had magnitude of 0.5. The sign of the skew was negative when the peak was to the left, and positive – when the peak was to the right. A control grating with the same position of the brightest and darkest stripes as the grating with skew 0.0, but with a step transition to gray was added as a control.
7The luminance gratings were used as texture over a plane receding in depth with the top of the surface slanted away from the observer. On every trial, the rotation of the plane around a vertical axis was simulated. The rotating surface was centered vertically on the screen and presented on a black background. The surface width exceeded the horizontal dimensions of the screen so that its edges were never visible during the motion. The simulated rotation of the plane started at 25º away from the frontal position to the left or to the right and ended at 25º from the frontal position at the opposite side (Figure 2). The horizontal edge of the plane closer to the observer was 14 cm in front of the simulated axis of rotation.
Figure 1. The luminance gratings used in the study (a) and a schematic representation of their luminance profile (b);
8The rotation cycle lasted 6.6 sec with a frame rate of 60 frames/sec and at its end, the planar surface disappeared and the background turned gray. The simulated angular velocity was .125º/frame.
9The stimuli were presented in 8-byte mode on the Dell P1130 21” monitor in a 1024 x 1280-resolution mode with gamma-correction to linearize the display intensities. The mean luminance of the screen was 30 cd/m2.
Figure 2.Schematic representation of the side view of the display in Experiment 1 (a) and a frontal view of the stimulus in the image plane at the final moment of rotation (b);
10Procedure: The subject sat at 80 cm from the monitor for stimulus presentation. The observation was monocular, through a viewing tube. At this distance, the visible portion of the texture pattern subtended 28 x 10 degrees of visual angle, a single period of the pattern being 3.5º of visual angle. A response monitor was positioned to the right of the observers at about 60 cm. The task of the observers was to adjust, using joystick, the orientation of a line presented on the response monitor, so that it corresponded to the perceived orientation either of the brightest, or of the darkest stripes on the surface at the end of the rotation cycle.
11The experimental factors were the type of the luminance gratings, the slant of the plane, the direction of rotation, and the task of the subject – to adjust the line orientation in correspondence to the perceived orientation of the brightest/darkest stripes. The two tasks were performed in separate blocks on different days in contra-balanced order over the subjects. Four different values of slant were used: 15º, 25º, 30º and 35º. Each experimental condition was repeated 7 times.
12The experimental session contained two blocks of 60 trials, but the data of the first four presentations were not taken into account. Each subject participated in 6 experimental sessions.
13The experiment started with a practice block of 10 presentations to familiarize the subject with the task and the stimuli.
14In orthographic projection of a slanted plane, rotating around a fixed axis, the projected velocity of its points depends on their distance to the rotation axis. For a plane slanted around a horizontal axis and rotating around a vertical axis the projected velocity is the same for all points in horizontal direction and varies in the vertical direction, depending on the plane slant. The larger the slant, the larger will be the difference in the distance to the rotation axis, and thus – in the projected velocities in vertical direction. As a consequence, when the rotation axis is behind the plane, the projected displacement of the points at the lower edge will be larger than that of the upper edge and the orientation of the stripes in the luminance gratings will deviate from vertical at the end of the rotation cycle. Their inclination, however, will be the same for the brightest and the darkest stripes due to the same projected velocity of the points in horizontal direction. If the apparent orientation of the brightest and the darkest is not the same, this would suggest that they have moved with different projected velocity and hence, the plane was deforming during its rotation. Thus, the adjusted orientation of the brightest and the darkest stripes can be used as measure whether the rotating plane appeared non-rigid.
15To test the effect of the experimental factors on the apparent orientation of the brightest and the darkest stripes the deviation of the adjusted line orientation from the vertical axis was calculated. In order to eliminate the differences in the adjustments due to the different direction of motion, the responses for the leftward rotation were multiplied by -1.0. This way, all the responses become positive. If the surface tilt was misperceived, the direction of motion would reverse, leading to negative deviations from the vertical. From all 4032 responses there were 6 negative deviations and they were excluded from the further analyses. In the sequel the term “adjusted orientation” will refer to the deviations of the adjustments from the vertical after the elimination of the effect of the motion direction.
16A mixed factor ANOVA with factors: skew of the luminance profile, planar slant, rotation direction and the subjects considered as a between-subjects factor was performed separately for the adjustments of the brightest and the darkest stripes. The Pilai-Barlett trace statistics (PB) (Keselman, 1998) was used as a robust multivariate test of the factors’ effect.
17The results show that the planar slant affected significantly the adjusted orientation of the darkest stripes (PB =0.06; F(3,23)=121.46; p<.05) – the more slanted the plane, the more the apparent inclination of the darkest stripes deviates from the vertical. There were significant individual differences at p=0.05 with respect to the effects of the motion direction (PB =0.41; F(5,25) = 3.43), and the planar slant (PB =0.12; F(15, 75) =2.73).
18The ANOVA results for the brightest stripes show significant main effects of the skew (PB =0.47; F(5,25)=5.61; p<.05) and of the planar slant (PB =0.94; F(3,27)=152.51; p<.05). There was significant interaction between the motion direction and the skew of the luminance profile (PB =0.49; F(5,25)=4.73; p<.05). Significant individual differences were observed with respect to the effects of the motion direction (PB = 0.51; F(5,29)=5.94, p<0.05) and the slant of the plane (PB =1.08; F(15,87)=3.29; p<0.05).
19Figure 3shows the mean adjusted orientations of the darkest and of the brightest stripes of the luminance gratings, while Figure 4 - the effect of the motion direction on the mean adjusted orientation of the stripes in the two tasks. It is clear, that when the luminance profile is more asymmetric, the brightest stripes look more tilted at the end of the rotation. When the direction of motion is opposite to the direction of the most extended gradient in the luminance profile, the brightest parts appear more tilted than when these two directions coincide. The motion direction has no significant effect on the performance for the control pattern.
20For all luminance gratings, except those with skew +/-0.5, the apparent inclination of the darkest stripes at the end of the rotation cycle deviates more from the vertical than the inclination of the lightest stripes.
Figure 3. Dependence of the adjusted orientation of the darkest and brightest parts of the gratings at the end of the rotation cycle on their luminance profile. The error bars represent 95% confidence intervals.
Figure 4. Mean adjusted orientation of the darkest and the brightest parts at the end of the rotation cycle for the different gratings and motion direction.The error bars represent 95% confidence intervals.
21If the brightest and the darkest stripes of the luminance gratings were perceived as pigment changes i.e. as painted over the plane, their relative orientation at the end of the rotation cycle should have appeared the same because the distance to the to the rotation axis varied only in vertical direction and was the same in the horizontal direction. As a consequence the projected velocity varied only along the stripes, but was independent of the brightness variation. Therefore, if the luminance gratings were perceived as texture and the brightest and the darkest stripes – as pigment changes over the surface, their apparent orientation should be the same at the final moment of the rotation. If the subjects were able to determine the projected velocity of the stripes, the motion information in the displays should have been enough to allow them to recover the affine properties of the surface - that the stripes are parallel and co-planar and to form an unique rigid three-dimensional representation of the plane motion (e.g. Hoffman and Flinchbaugh, 1982; Lappin, 1990).
22The gradual changes in the luminance profile of the gratings may be perceived as resulting from the orientation of the surface toward a stationary light source. The recovery of shape from shading is a relatively quick process - it takes less than 80 ms (e.g. Sun & Perona, 1996) while SFM requires recruiting information for least 200 ms (e.g. Eby, 1992; Atchley, Andersen & Wuestefeld, 1998). Shading information, however, depends not only on the surface geometry, but on the material properties of the object and the pattern of its illumination. If shape recovery from shading precedes the shape recovery from motion because of the longer time intervals needed for the later process, the illusory relative motion of the brightest and darkest stripes may be related to the constraints used by the visual system to recover the object’s shape from the changes in the brightness pattern, naturally occurring when an illuminated object moves with respect to fixed light source.
23If the brightness variation of the luminance grating was interpreted as shading, it would correspond to a surface, corrugated in horizontal direction since no brightness variation exist in the vertical direction. For a rigid corrugated surface rotating around a fixed axis all parts will preserve their relative distance to the rotation axis and hence, in the image plane the brightest and the darkest parts should remain parallel. A difference in the motion of the brightest and the darkest parts of the surface may occur because of their different perceptual significance. For example, Ostrovsky, Cavanagh & Sinha, (2005) have shown that the visual system is unable to detect the inconsistencies in the shadows over the entire visual scene, which suggest that the illuminated and shadowed parts may be treated in a different way and thus, may be differently affected by the motion.
24The aim of Experiment 2 was to obtain information about the perceived direction of the incident light and its relation to the skew of the patterns in static and dynamic conditions.
25Participants:Five subjects, from the staff of the BAS participated.
26Stimuli: The same luminance gratings as in Experiment 1 (without the control pattern) were used.
27Procedure:On each trial, a single stimulus was presented in the middle of the screen. Below it, a semicircle with diameter 3.5º of visual angle was drawn. A randomly oriented line with length equal to the radius of the semicircle was presented inside it. The task of the observers was to change the orientation of this line with the help of the computer mouse, so that it coincided with the perceived direction of the incident light.
28The stimuli were presented either static or in simulated rotation as in Experiment 1. The two conditions were performed on separate days. Each luminance grating was presented to each observer 10 times in random order. For the dynamic conditions the rotating plane was slanted by 30º and the motion direction varied. The static condition represents the plane at its fronto-parallel position.
29The Rayleigh’s test of uniformity of circular data was applied to test whether the adjusted directions could be regarded as samples from a uniform distribution. The results suggest that both for the static and for the dynamic conditions the null hypothesis of uniformity should be rejected for all subjects and luminance gratings at p=.05. The Watson-William’s test (with a correction suggested by Mardia, 1972) showed that the null-hypothesis of equal adjusted orientation for the two motion directions should be rejected at p=.05 for 15 out 25 comparisons (5 subjects x 5 gratings). This suggests that the illuminant was not perceived as stationary. Interestingly, Caudek, Domini & di Luca, (2002) showed that when a light source was moving opposite to a surface rotation in SFM displays, the perceived motion of the surface was changed, which suggests that the relative motion was ascribed to the surface and not to the light source (see also Mamassian, Knill & Kersten, 1998).
30The absolute deviation of the adjusted direction from straight ahead varied consistently with the skew of the luminance patterns for the static condition, but not for the moving patterns (Figure 5). The inter- and intra-individual variability of the estimates also differed: standard errors were in the range 1.03º to 7.00º for the static and in the range 10.95º to 27.91º - for the dynamic conditions.
31The results of the experiment suggest that in dynamic conditions the perceived direction of illumination, determined from the luminance gratings, is quite unreliable and changes with the direction of motion. These results contradict with the existing experimental data (Norman, Todd & Philips, 1995) showing that the local orientation of shaded objects rotating in depth is estimated even more accurately than in static conditions when the light source is stationary. If, in the present experimental conditions, the light source appeared moving, its apparent direction of motion should coincide with the motion direction to be consistent with the information in the motion displays. Caudek et al. (2001), however, have shown that when a surface and a light source move in the same direction, the illuminant direction is adjusted with the same accuracy for both dynamic and static conditions. Therefore, the shape-from-shading interpretation of the moving displays cannot explain the results of Experiment 1.
32Figure 5. Mean absolute deviation of the perceived direction of the light source from the straight-ahead direction as a function of the skew of the luminance profile in static (a) and dynamic conditions (b). The error bars represent the 95% confidence intervals.
33Blurred edges appear sharper when moving than when stationary (e.g. Bex , Edgar & Smith, 1995; Hammett & Bex 1996; Georgeson and Hammett 2002; Hammett, Georgeson, Bedingham & Barbieri-Hesse, 2003 ). This effect, called motion sharpening, depends on the speed of motion and on the amount of blur. If the luminance gratings used in Experiment 1 were perceived as patterns with blur edges, most of the results may be explained under the assumption that the perceived blur of the edges depended on the steepness of the luminance gradients. In the conditions of the experiment, the upper edge of the slanted planes was closer to the rotation axis than the lower one and thus, the upper half of the surface moved with lower linear velocity than the lower half. Therefore, motion sharpening would change the perceived width of the stripes over the plane making the lower ends narrower than the upper ones and effectively changing the orientation of their edges. This effect should be larger the larger the perceived blur of the edges. Because the luminance profiles of the gratings with skew not equal to zero are asymmetric, the narrowing of the stripes would not be the same and their left and right edges would be affected differently by the motion of the surface. If the perceived orientation of the stripes at the end of the rotation cycle was determined by the leading edge of the stripes, the performance would change with the change of the rotation direction as this will change which edge of the stripes is the leading one. As a consequence, motion sharpening will cause changes in the adjusted orientation of all stripes over the planes at the end of the rotation cycle and these changes will depend on the skew of the luminance profile, on the rotation direction and the speed of motion.
- 1 The suggestion that motion sharpening may explain the results from Experiment 1 was suggested by F (...)
34Experiment 3 tests whether the effects of the skew and the motion direction, observed in Experiment 1, could be explained by motion sharpening1.
35Participants:Four subjects, who have participated in Experiments 1 and 2, took part.
36Stimuli: The same luminance gratings as in Experiment 2 were used. The stimulus conditions differed from Experiment 1 by the following changes: only one value of slant was used (50º), the edge of the planar surface closest to the observer was either at 14 or at 28 cm away from the rotation axis and the visible part of the displays was restricted to a rectangular aperture of size 11.75 x 10 degrees of arc, so no more that 3.5 periods of the gratings were visible. The horizontal position of the gratings over the plane was randomly shifted on every presentation up to +/-1 degrees of arc in order to change the relative position of the stripes with respect to the vertical edges of the aperture. A stationary condition, representing the last frame from the motion sequence for the two directions of rotation was also included.
37 Procedure:A moving or a stationary stimulus was presented on every trial. After the end of the presentation, the screen turned gray and a horizontal white line with random length appeared 1.5 degrees of arc below the lower edge of the aperture. The line was centered at the middle of the screen in the horizontal direction. The subjects had two tasks – to adjust the perceived width of the upper/lower edge of the central brightest stripe in the aperture. For the moving planes the subjects were told that the adjustment refers to the final moment of the rotation. To perform the task the arrow keys of the keyboard were used. Each task was performed on a separate day in a contra-balanced order over the subjects.
38The experimental factors were: the task, the state of the surface – moving with different linear velocity (due to the different distance to the rotation axis) or stationary, the motion direction and the skew of the luminance grating. Each experimental condition was repeated 5 times in random order.
39The effects of the experimental factors on the adjustments were evaluated by a mixed model ANOVA. Only the effect of the skew was significant (Pillai-Bartlett’s trace statistics = 0.91; F(4,13)=32.01; p<0.05). No other main effect or interaction term were significant at p=0.05 (see Figure 6).
40These results contradict the predictions based on the motion sharpening as a factor in explaining the results of Experiment 1. Though the adjusted width of the brightest stripes of the gratings varied with the skew of their luminance profile, it was not less in the dynamic than in the stationary conditions. No difference was observed in the adjusted width of the stripes depending on the distance of the plane to the rotation axis, even though this would have changed their projected velocity. No difference was observed also between the adjusted width of the upper and lower edges of the moving gratings as should be expected due to motion sharpening.
Figure 6. a. Mean adjusted width of the brightest stripes as a function of the skew of the luminance gratings for the stationary and dynamic conditions. The distance from the plane’s frontal edge to the rotation axis of 14 and 28 cm are marked by v1 and v2
b. Mean difference between the adjusted width of the upper and the lower edge of the brightest stripes for the different gratings and motion conditions. The error bars represent 95% confidence intervals.
41The results of the present study show that a planar surface textured with luminance grating and rotating around a fixed axis appeared non-rigid since the brightest and the darkest stripes over the plane appeared non-parallel at the end of the rotation cycle. The apparent orientation of the darkest stripes was independent of the skew of the luminance gratings and varied systematically with it for the brightest stripes. The direction of motion and the directions of the luminance gradient of the brightest stripes in the gratings significantly interacted.
42 The observers’ performance indicates that the brightness variations in the luminance gratings were not perceived as a pigment change. The observed non-rigidity of the gratings during their motion could not be explained by the dominance of the shape-from-shading recovery with a stationary light source, or by motion sharpening. Even though the motion information in the display should have been sufficient not only to recover the affine properties of the surface, but also – to obtain a unique representation of its motion, the plane apparently deformed during its rotation. This suggests that the effect may be a related to the initial stages of the SFM processing and motion detection. For example, the luminance of the stripes may interact with the perceived motion in a manner similar to the Hess effect (e.g. Wilson & Antis, 1969; Whitney, 2002) where the darker of two physically aligned moving bars appeared to lag behind a lighter one, most probably due to the different processing time needed to perceive targets of different luminance. The relative difference in luminance with the background may also affect the performance (as shown for apparent motion by Antis, Smith & Mather, 2000). Interactions of this type may explain the differences in the perceived orientation of the brightest and the darkest stripes in a single luminance grating; however, they could not predict the effect of the luminance gradient steepness on the perceived orientation of the brightest stripes.
43The steepness of the luminance gradients in the gratings determined how far the edges are. If the brightness percept occurs in accordance with the “filling-in” models (e.g. Cohen & Grossberg, 1984; Grossberg & Todorovic, 1988), the spreading of neural activity will require different time depending on the edge separation. This assumption may account for the effect of the gradient steepness on the performance. The process of filling-in may interact with the motion direction since the position of the boundaries that control the spreading of the neural activity will be shifted during image displacement. In addition, if the on- and the off-channels have different filling-in domains (as suggested by Pesoa, Mingolla & Neumann, 1995) and different processing time (e.g. Del Viva, Gori, Burr, 2006) the darkest and brightness stripes will behave differently. The dynamics of the luminance and motion processes and their interactions requires further experimental studies.