This study was carried out with support to the first author by a Fulbright Research Fellowship. The research was supported in part by grant DC00271-13 to the second author from the National Institute on Deafness and Other Communication Disorders, NIH. We thank Nancy Matteer for her assistance in analyzing the data and all the subjects for their patient participation.
1Several studies have compared visual perception, tactile (haptic) perception, and visual-haptic perception of stimuli (e.g. Easton, Greene, & Srinivas, 1997; Millar, 1981; Abravanel 1971; Lobb, 1965; Rudel & Teuber, 1964). Often, performance in tasks involving unimodal visual perception exceeds performance in both unimodal haptic and cross-modal tasks. But not always: Norman, Norman, Clayton, Lianekhammy, and Zielke (2004), for example, found that unimodal haptic comparisons of natural three-dimensional shapes could be as good as visual-haptic and haptic-visual comparisons. Norman et al inferred that vision and touch have functionally overlapping, though not equivalent, representations of 3-D space.
2Often, studies of cross-modal perception use a sequential design, which places demands on memory, and demands on memory may matter more to unimodal haptic tasks and cross-modal haptic-visual tasks than to visual tasks (Woods, O’Modhrain, & Newell, 2004). The present study compared unimodal visual, unimodal haptic, and cross-modal visual-haptic discrimination of pairs of objects differing (or not differing) in their radii of curvature, when each pair of objects was presented simultaneously.
3 Studies of intramodal and cross-modal curvature perception in vision and touch have attributed differences in performance to several factors. For example, Kappers, Koenderink, and Oudenaarden (1997) found that participants overestimated haptic curvature relative to visual curvature and concluded that mutually inconsistent representations of surface curvature in the haptic and visual modalities may coexist within a single observer. Subsequent studies have shown that curvature discrimination is based on the slope differences at the far ends of the stimuli (Pont, Kappers, & Koenderink, 1997) and are unaffected by the tilts of the curved surface (Pont, Kappers, & Koenderink, 1998) during both static and dynamic touch (Pont, Kappers & Koenderink, 1999). But the influence of shape on curvature perception is still largely unexplained (Vogels, Kappers, & Koenderink, 1999) and the underlying mechanisms still await elucidation.
4 Underlying much of the research comparing unimodal visual and tactile perception to cross-modal visual-tactile perception is a long-standing theoretical issue: Do perceivers ‘naturally’ recognize common features of objects perceived through vision and through touch? Or, alternatively, do accurate cross-modal comparisons develop largely or wholly through experience? Molyneux’s famous question about the relation between touch and vision – Can a person born blind distinguish between a cube and a sphere after recovering sight in adulthood? – continues to occupy philosophers (Gallagher, 2004).
5 In this regard, research has shown that very young infants not only show cross-modal transfer of object properties such as texture and hardness (e.g., Meltzoff & Borton, 1979; Gibson, 1979), but also can recognize by sight objects previously presented to touch (Streri, 1987; Streri & Gentaz 2003). These results challenge the empiricist philosophy and modern connectionist models (McClelland & Rumelhart, 1986; Elman, 1996) that assume independent sensory modalities at birth. Presumably, the capacity found in older children and adults to make cross-modal as well as intramodal comparisons evolves from intrinsic capabilities in infants.
6 The present study extends findings of Ittyerah and Marks (2007), who examined the role of vision and haptic touch in memory for stimulus objects that varied in curvature when different tasks intervened between initial stimulus exposure and subsequent comparison. The results suggested that different intervening activities affected performance in different ways, depending on whether the initial stimulus was presented visually or haptically. This outcome suggests possible differences between vision and touch in processes of encoding or representing stimulus curvature. It was of interest, therefore, to compare visual, haptic, and visual-haptic discrimination of these same stimuli when the two stimuli within each pair were presented simultaneously.
7 Sixty men and women between the ages of 18 and 30 years, volunteered to participate, 20 in each of three conditions, as described below. Most were students at Yale University. Subjects reported no visual or tactile sensory deficits, and all had normal or corrected-to-normal vision. Although the experiments required participants to use the index fingers of one or both hands to assess curvature haptically, the handedness of the subjects was not tested because that lateralization does not affect general tactile ability (Ittyerah 1993, 2000). More directly to the point, Kappers et al (1997) and Kappers and Koenderink (1996) measured handedness and reported no differences between preferred and non-preferred hands in perceiving curvature.
8 There were two sets of six concave stimuli, constructed from blocks of maple wood measuring 5.08 x 10.16 x 3.81 cm. These dimensions were identical for all stimuli. Only the depth of concavity in each stimulus, and thus its radius of curvature, varied. Curvature is inversely related to radius of curvature: the greater the radius of curvature, the less curved (flatter) the stimulus. The radii of curvature of the stimuli from the flattest to the most curved were 34.1, 25.7, 20.6, 17.3, 15.0, and 13.2 cm, corresponding to maximal depths of indentation of 3.81, 5.08, 6.38, 7.62, 8.89, and 10.16 mm, respectively (roughly equal differences between successive stimuli). For convenience of exposition, we label the successively increasing curvatures of these six stimuli by the numerals 1-6.
9 Figure 1 depicts each of the six stimuli, which differ only in curvature. Stimulus 1 has a difference of 3.81 mm between its midpoint and its height at the ends. Stimulus 2 has a difference of 5.08 mm between its midpoint and the height at its ends and, therefore, has greater curvature than stimulus 1. The remaining stimuli vary similarly, such that stimulus 6 has the greatest curvature and stimulus 1 the least curvature.
Figure 1: Dimensions of the six stimuli used in Experiment 1.
10 The radii of curvature were selected for study to ensure imperfect (but not impossible) discrimination in both touch and vision. All stimulus surfaces were smooth to touch and did not differ in texture. In the analyses reported below, the stimulus pairs are defined in terms of step sizes, as follows: step size 0 = same pairs, 1&1, 2&2, 3&3, 4&4, 5&5 and 6&6; step size 1 = different pairs, 1&2, 2&3, 3&4, 4&5, and 5&6; step size 2 = different pairs, 1&3, 2&4, 3&5, and 4&6; step size 3 = 1&4, 2&5, and 3&6; step size 4 = 1&5 and 2&6; and step size 5 = 1&6.
11Each subject sat before a table that supported a box measuring 46 x 53 x 30.5 cm. Between the top surface of the table and the bottom of the box was a gap, located approximately waist-high, of about 12 cm into which the subject could reach (but could not see) in order to examine the stimuli haptically. For visual inspection, stimuli could also be placed on top of the box, at eye level, about 30 cm away. The stimuli were presented in a fronto-parallel plane with the long side of the stimulus facing the subject, so that the subject could inspect the curved plane.
12 Each subject was tested in one of three conditions: (1) both stimuli on a given trial presented haptically (T-T); (2) both stimuli presented visually (V-V), or (3) one stimulus presented haptically and the other visually (T-V). Separate groups of 20 subjects served in each condition. The subjects were told that they would be presented pairs of objects, identical in height and width, that might be the same or different in curvature, and that their task was to indicate whether the two stimuli in each pair were the same or different.
13 Given 6 different stimuli in the set, there were 6 possible pairs in which the stimuli were the same (Same Set) and 15 possible pairs in which the stimuli were different (Different Set). Each pair in the Same Set was presented 6 times in the course of a session and each pair in the Different Set was presented 3 times, making a total of 6 x 6 + 3 x 15 = 81 trials per session.
14 The subjects were instructed to explore the two stimuli with the forefingers of the two hands by simultaneously moving the fingers, in the same direction, from one end of the stimuli to the other in one continuous motion. A single continuous movement of the finger was considered to specifically focus on the curvature being felt, whereas free exploration of the stimulus with the entire hand might have distracted the participant by the irrelevant dimensions of the stimulus. There was no fixed time limit for a single continuous movement, although this was considered equivalent to a one second visual exposure of the same stimulus. The fingertip was positioned at the centre of the surface of the curvature and not at the edges, so that the finger could easily move from one end of the curved surface to the other. There were no restrictions in the movement of the elbow while the finger felt each surface. On each trial, one stimulus was placed on the left and the other on the right (separated by about 10 cm), the left-right positions for each possible pair counterbalanced across repetitions for each participant.
15 Pairs of stimuli were presented simultaneously at eye level. Again, on each trial, one stimulus was placed on the left and the other on the right (separated by about 10 cm), the left-right positions for each possible pair counterbalanced across repetitions for each participant. The visual stimuli were exposed for one second.
16 The visual stimulus was placed above the haptic stimulus, and the participant examined the two simultaneously. On half of the trials, both the visual and tactile stimuli extended leftward from the midpoints of the surfaces upon which they rested. The participant simultaneously felt the haptic stimulus with the left hand in a single continuous motion and saw the visual stimulus. On the other half of the trials, the stimuli extended to the right of the midpoints of the surfaces, and the participant felt the tactile stimulus with the right hand.
17 The results were analyzed in two ways, each of which has its strengths and its limitations. The first analysis compared separately the judgments given to the Same Set and the judgments given to the Different Set. The second analysis combined the judgments given to the Same Set and Different Set into unitary measures of d’ for each stimulus pair in the Different Set in order to characterize overall discrimination in each modality.
18 The averages and standard errors of the response proportions given to each pair of stimuli are depicted in Figure 2. Inspection of the figure reveals two general trends. (1) Performance tended to be best with visual presentation (V-V), worst with haptic presentation (T-T), and intermediate with cross-modal presentation (T-V), and (2) on those trials in which the stimuli were different, performance improved as the physical difference between the stimuli increased.
Figure 2: Proportion (and standard error) of ‘different’ judgments as a function of the size of the stimulus difference (in stimulus steps), for visual-visual, tactile-visual, and tactile-tactile comparisons of curvature.
19 The measure of stimulus difference used in the figure corresponds to the number of stimulus steps. Note that the stimulus steps are almost all equal in terms of radius of curvature; number of steps provides a convenient means for assessing overall trends. Values plotted at Step 0 give the average proportions obtained with the Same Set. Values at Step 1 give the average proportions obtained with those proximal stimulus pairs in the Different Set, e.g., radii of curvature of 34.1 and 25.7 cm. Values at Step 2 give average proportions for stimuli proximal + one, e.g., radii of curvature of 34.1 and 20.6 cm, and so forth.
20 In pooling different stimulus pairs by step size, we have ignored any small differences that might depend on absolute curvatures, although such differences likely exist. For example, Nefs, Kappers, and Koenderink (2001) measured Weber fractions for discrimination of spatial sinusoids, that is, for stimuli whose amplitude varied sinusoidally over the surface, and found sensitivity to be greater (threshold lower) the greater the spatial period (the smaller the spatial frequency). Discrimination may, therefore, be slightly better when the mean radius of curvature is larger rather than smaller, although the present data cannot decide whether this is so.
21 To determine whether the differences among the modality conditions are significant, the proportions of correct responses were arcsine transformed and subjected to a 3 (modalities) x 6 (step sizes) ANOVA, using Statistica 1997, version 5.5. The results indicate that the main effect of modality is significant (F (2, 57) = 24.23; MSE = 1.21; P < .001), showing that V-V judgments are the most accurate, followed by V-T and T-T (means: V-V = 1.59; V-T= 1.06; T-T = .96). The main effect of step size is also significant (F (5, 285) = 32.59; MSE = 1.22; P < .001), showing that accuracy with same stimulus pairs (step size 0) differs from accuracy with different stimuli (step sizes, ss, 1 to 5): (mean: ss0 = 1.06; ss1= .81; ss2 = 1.01; ss3 = 1.12; ss4 = 1.13; ss5 = 1.13). The modality x step size interaction is significant (F (10, 285)= 5.2; MSE= .2; P< .001). Newman-Keuls post hoc tests show that for V-V, step sizes 0 and 1 differ, and step size 1 differs from step sizes 2, 3, 4 and 5 (P< .05). For T-V, step size 1 differs from step sizes 2 and 3, and step sizes 2 and 3 differ from step sizes 4 and 5. For T-T, step sizes 1 to 5 differ from step size 6 (P< .05). In summary, the results indicate that curvature discrimination is the best with unimodal vision, followed by cross-modal perception, then unimodal haptic perception, with accuracy (not surprisingly) increasing as the difference in curvature increases.
22 The second approach combined judgments given on same trials with judgments given on the various different trials into values of d’. The value of d', from signal detection theory (Green and Swets, 1966), provides a measure of discriminability of each pair of different stimuli, taking into account both the ability to identify stimuli of different pairs as different and the propensity to (mis)identify the stimuli of same pairs as different. Measures of d' thus take into account variations in criterion for responding 'different': The more lax the criterion, the more likely a subject will respond 'different' to a different pair, but also the more likely to respond 'different' to a same pair.
23 To determine each value of d', we used a differencing model of same-difference judgment (see Macmillan & Creelman, 1991). This model assumes that a subject samples, independently, each stimulus of the pair on a given trial, calculates the difference, then compares this difference to an internal criterion, responding 'different' only if the difference exceeds criterion. The proportions of 'different' responses given to both same pairs and different pairs are then converted to d' (Macmillan and Creelman, 1991). The values of d’ are plotted in Figure 3.
Figure 3: Discriminability (d’) of curvature, derived from same/different judgments, as a function of the stimulus difference (in stimulus steps), for visual-visual, tactile-visual, and tactile-tactile comparisons.
24 In general, the modalities order themselves consistently, with visual performance (V-V) again best and haptic performance (T-T) poorest. Over the smallest stimulus differences (steps 1 to 3), where the measures of d’ are least susceptible to fluctuations in extreme probabilities and therefore most reliable, the values of d’ average 3.5, 2.2, and 1.0 in conditions V-V, T-V, and T-T, respectively. The absolute values of d’ reflect, of course, the particular values of curvatures used in the experiment. Nevertheless, the way these values vary across modalities suggests that, for any given pair of stimulus objects, unimodal visual sensitivity to small differences in curvature is superior to cross-modal sensitivity, which in turn is superior to unimodal haptic sensitivity. Perhaps haptic performance is affected by the way the fingers moved over the curved surfaces. It may be relatively easier for the left and right index fingers to move in opposite directions than for both to move in the same direction (Kappers & Koenderink, 1996). It is not clear, however, whether a preference for movement direction would affect sensitivity to differences in curvature.
25 The results indicate that when two object surfaces, either the same or different in curvature, are presented simultaneously for comparison, unimodal visual performance exceeds cross-modal performance, which in turn exceeds unimodal haptic performance. A different trend appeared, however, in a study of memory for curvature using a subset of these same stimuli, where a delay was introduced between the two stimuli in each pair (Ittyerah & Marks, 2007). In Experiment 1 of that study, for instance, haptic-haptic memory and haptic-visual memory (initial haptic exposure followed by either haptic or visual comparison) were nearly equivalent. Given the findings of the present experiment, showing cross-modal performance to exceed haptic performance, we infer that the introduction of a delay may affect cross-modal performance and unimodal haptic performance in different ways.
26 Figure 2 shows that the accuracy of responses to same pairs of stimuli (step 0) is much smaller with haptic comparison than with intramodal visual or with cross-modal comparison. And accuracy of responses to different pairs is also smallest, by and large, with intramodal haptic comparison. As Figure 3 shows, over the three smallest physical differences, where the measures of d’ are most reliable and least susceptible to variability associated with extreme proportions, unimodal visual performance exceeds cross-modal performance by about one d’ unit, essentially, one standard deviation unit, and cross-modal performance similarly exceeds unimodal haptic performance by about one d’ unit.
27 Differences between haptic and visual perceptual processing might underlie, at least in part, differences in mnemonic representations for haptic and visual curvature (Ittyerah & Marks, 2007). Haptic processing depends on movements of the hand or finger over the stimulus and therefore is subject to spatial constraints on motion, whereas visual processing is often global (Navon, 1977), with sensory detection and discrimination limited by the physical properties of the stimuli themselves, such as photon counts at low intensities (De Vries, 1943). In tasks requiring linear movement, demands on memory involve recognition of starting and ending locations of the movement (Laabs & Simmons, 1981; Millar, 1994). For haptically felt curvature, discrimination appears to be based largely on slope differences at the far ends of the stimulus (Gordon & Morison, 1982; Pont et al, 1997; Pont et al, 1998), for both static and dynamic touch of curved strips (Pont et al, 1999). Louw, Kappers, and Koenderink (2000) measured haptic discrimination between flat surfaces and curved surfaces with both concave and convex Gaussian profiles. When the spatial width of the Gaussian profile exceeded 1 mm, discrimination threshold (amplitude) increased as the 1.3 power of spatial width with both concave and convex stimuli. Louw et al. concluded that the haptic system is sensitive to the local slope of the stimulus (see also Pont et al., 1998, 1999).
28 The findings of Ittyerah and Marks (2007) suggest that memory of haptic and perhaps of visual representations of curvature can be particularly disrupted by tasks that involve spatial processing and movement – as assessed with a dual task paradigm of the sort often used to test for the demands of attention on a primary task (Brown, 1958; Peterson & Peterson, 1959). Filling the delay interval, between the initial exposure to a curved stimulus and subsequent comparison stimulus, with a spatial or a movement task produces modality-specific interference (e.g., Logie, 1986).
29 The susceptibility of visual and haptic representations of curvature to intervening tasks of spatial processing and movement may reflect general characteristics of haptic processing. To be sure, the encodings and representations depend on perceptual processing capabilities of the visual and haptic systems (Millar, 1999). It is possible, however, that perceptual encodings and representations themselves also depend at least in part on the task (Millar, 1994). It is conceivable that people deploy different strategies to encode stimulus information when the task at hand requires immediate comparison, as in the present study, and when the task requires comparison later in time. More complex experimental designs would be needed to tease apart the strategic from the perceptual and mnemonic factors.