Navigation – Plan du site

AccueilNuméros22, Vol. 2, 2007How do gesture and speech product...

How do gesture and speech production synchronise?

Pierre Feyereisen

Résumés

Les relations temporelles entre le début d’un geste et d’un énoncé ont été analysées dans deux expériences utilisant un paradigme de doubles tâches. Dans les conditions de tâches simples, les participants répondent à la présentation d’images soit par un geste référentiel prédéfini, soit par un énoncé oral. Dans les conditions de doubles tâches, les deux réponses doivent être fournies. La difficulté de la tâche orale est manipulée lors de la sélection de la réponse (Expérience 1: ensembles de 2 ou de 8 stimuli) ou de sa préparation (Expérience 2: énoncés de 1 ou 5 mots). Comme prévu, ces manipulations affectent les latences des réponses orales. Les effets d’interférence (différences entre tâches simples et doubles) se produisent dans les deux modalités de réponse. Le début du geste précède celui de l’énoncé oral mais il est retardé quand la difficulté de planification de cet énoncé augmente. Ces résultats sont discutés en relation avec les modèles de la performance en doubles tâches; ils sont en accord un modèle de l’administrateur central qui suppose une postposition stratégique des réponses dans les conditions de doubles tâches.

Haut de page

Texte intégral

Introduction

1Human beings are endowed with a rich, multi-modal system of communication including gestures and speech. The invention of sign languages indicates that gestures can also display the elaborate properties found in spoken languages. Usually however, speech conveys most of the information and co-verbal gestures do not fully exploit their communicative potential. Gesturing while speaking is a natural instance of dual-task performance and hence, speakers might have difficulties in planning sentences and complex gestures simultaneously. Actually, there are various ways to see the combination of the two modalities. According to McNeill (1985; 1992; 2005), gestures and speech are closely synchronised and integrated within a single production system in which they play complementary rolesi. In this model, there is no preliminary planning of the message, which growths dialectically from the available forms of expression, gestures and words of the spoken language. Alternatively, in information processing models (e.g., Butterworth & Hadar, 1989; de Ruiter, 2000; Feyereisen, 1987), hand and speech movements are elicited by a preliminary intention during a conceptualisation phase and are planned before execution. This temporal organisation may be conceived either as a strict succession of stages (e.g., Levelt, 1989) or as a more graded accumulation of activation. In either case, response planning (i.e., selection and preparation) requires a certain amount of time, which may vary depending on facilitating or interfering conditions. More particularly, speech responses may be delayed when rival formulations or concurrent tasks compete or they may be facilitated when information from diverse sources converges toward the target.

2Doing two things at once is obviously more difficult than performing only one task. Several models have been proposed in experimental psychology to explain dual-task interference effects, i.e. the difference between dual and single task processing (e.g., Lien & Proctor, 2002; Marois & Ivanoff, 2005). The scope of these models is purposively general and encompasses different kinds of responses (e.g., vocal naming and key pressing).

3The dominant view assumes that there is a central bottleneck stage during which the two responses are planned in succession (Pashler, 1994; see also the Figure 8 of Fagot and Pashler, 1992). Response latencies in concurrent tasks 1 and 2 result from the addition of three components: A1–A2 (pre-bottleneck), B1–B2 (central bottleneck), and C1–C2 (post-bottleneck). It is assumed that in dual task conditions, interference results from the postponement of the B2 planning after the completion of B1 (serial response selection model). Input processing stages A1 and A2 may occur in parallel, as may response executions C1 and C2. In other variants of the model, however, multiple bottlenecks are assumed in the perceptual stages A1–A2 or at response initiation C1-C2, yielding functional binding of the two responses. Thus, in some conditions, the faster process is delayed to ‘await’ the completion of the slower process, and the initiation of the two responses at C1–C2 are synchronised (response coupling model: De Jong, 1993; Fagot & Pashler, 1992).

4A second class of models, the capacity sharing models, explain interference effects by assuming a pool of divisible resources allocated to the concurrent tasks (Navon & Miller, 2002). Responses may be planned in parallel, albeit more slowly than in the single task conditions, and queuing is not the rule, even if it is a convenient solution to the problem of managing multiple tasks. Thus, increasing the demands of the second task may slow down the performance of the first one, an outcome that is not predicted by the classical central bottleneck model.

5In a third direction, Meyer and Kieras’s (1997; 1999) EPIC (executive-process interactive control) model gives prominence to control mechanisms that manage flexible adaptations. No capacity limitation is assumed, but one task may receive priority over the other, depending on strategic choices. Thus, diverse patterns of interaction, depending on individual differences, can be expected. Practice, in particular, is a factor that attenuates interference in dual task conditions.

6Two studies have already analysed the competition between gestures and speech in dual-task paradigms (Levelt, Richardson, & La Heij, 1985; Feyereisen, 1997). These experiments simulated in the laboratory the ordinary production of speech and gestures as expression of thought by presenting a single visual stimulus to which vocal, manual, or dual responses were required. Levelt et al. (1985) elicited pointing gestures whereas Feyereisen (1987) instructed participants to perform predefined iconic gestures. The findings consistently indicated that the initiation times of these gestures were generally faster than the voice onset times, and that interference effects were larger in the vocal than in the manual modality.

7How can this asymmetry be explained by models of dual-task interference? On the basis of the serial selection model, we can assume that manual and vocal responses are selected in succession. However, in his Experiment 3, Feyereisen (1997) facilitated response selection through a priming procedure but the interference effects did not decrease in the easiest condition (high response predictability, long preparation interval). Thus, slowing in the dual task was not due to the postponement of the vocal response planning until after completion of the manual response planning (serial response selection without overlap of the B1–B2 stages of response planning). Alternative explanations are needed.

8The aim of the present study was to better understand the temporal organisation of vocal and manual responses made to the same signal, and to test two hypotheses. These hypotheses were presented in Figure 1 of Feyereisen (1997), redrawn from Fagot and Pashler (1992). According to the first hypothesis (response coupling), the two responses are synchronised before execution, at the beginning of the C1–C2 stage (Figure 8C in Fagot & Pashler, 1992). In dual task conditions, the presumed fast planning of the vocal response is followed by a time lag during which the manual response is planned. Accordingly, increasing the duration of the vocal planning phase should delay the gesture onset and eliminate the interference effect in the vocal modality. Indeed, in this model, the interference effect should be absorbed by the lengthening of the planning phase. Shared capacity models (i.e. the conjoint selection model of Fagot and Pashler, 1992, see their Figure 8D) also predict that increasing the difficulty of vocal planning will affect the timing of manual responses but, in this case, the interference effects should be amplified. Interference effect in the manual modality should increase with lengthened response latencies in the most demanding conditions. Some support for such a hypothesis was found by Levelt et al. (1985) in their Experiment 3, which manipulated the number of response alternatives (one or two verbal expressions and two or four different pointing gestures). The initiation times of both vocal and manual responses were affected by the two factors. As the interaction was super-additive, these investigators concluded that the two production systems were competing for common resources until response execution.

9In the present study, two experiments were designed in parallel to evaluate these hypotheses by manipulating response selection and preparation. In the first study, the difficulty of selecting the vocal response varied with the number of response alternatives (two or eight), a factor whose influence on response latencies has been known for a long time (e.g., Hick, 1952). In the second experiment, the preparation time of vocal responses of different lengths was compared. The more complex the sentence, the longer the preparation time (e.g., Ferreira, 1991). The manual response was constant across the conditions. In Experiment 1, the choice was between a simulated precision grip (thumb-finger opposition) for some objects and a simulated force grip (fist clenching) for another set of objects. In Experiment 2, four gestures were used: the same two as in Experiment 1 plus palm opening and index finger extension.

Experiment 1: Selecting vocal responses among sets of various sizes

10The main objective of the first experiment was to examine the effect of increasing the difficulty of selecting the vocal responses (by increasing the choice from 2 to 8 items) on the initiation times of gestures and the patterns of interference effects.

Method

11Participant. Twenty-four female students (aged 19 to 26) received course credits for their participation.

12Materials and apparatus. We selected pictures of 8 objects from the Snodgrass and Vanderwart series updated and digitalised by Cycowicz, Friedman, Rothstein, and Snodgrass (1997). The criteria were to have bisyllabic French nouns beginning with a consonant and to allow one of the two the actions described by Klatzky, McCloskey, Doherty, Pellegrino, and Smith, (1987) to be performed. The gestures were either a ‘pinch’ for the pictures of a first set of objects: a candle (bougie), a bolt (boulon), a paintbrush (pinceau), and a sandwich (tartine) or a ‘clench’ for pictures of another set: a banana (banane), a basket (panier), a broom (balais), and a casserole dish (poêlon).The participants were instructed to learn these associations.

13The 2-D pictures were approximately the same size (5 by 4 cm) irrespective the object’s actual size. They were presented on the screen of a computer (Macintosh Performa 450) running the PsyScope program (Cohen, MacWhinney, Flatt, & Provost, 1993). Gesture onset time was recorded through the key release on the button box, and voice onset time through activation of the microphone. Errors were noted manually.

14Procedure. Six blocks of 40 randomised trials were constituted, comprising either 20 presentations of the casserole dish and the paintbrush (set size = 2) or five presentations of each of the objects (set size = 8). In the single vocal task condition, the participants were instructed to name the objects with the target nouns. In the single manual task conditions, the experimenter drew participant’s attention to the size of the objects and showed the pinch and clench gestures, with the instruction to perform with the preferred hand one of these gesture depending on the presented picture.  In the dual task conditions, both vocal and manual responses were required. Participants were invited to be as fast and as accurate as possible but no particular instruction was given about the relative timing of the two responses. A warning signal of 500 ms preceded the presentation of the pictures. In the manual and the dual tasks, the trials started with key pressing and thus, stimulus presentation was self-paced. In the single vocal task, the inter-stimulus interval was fixed at 750 ms. The order of the conditions and the set sizes was counter-balanced across subjects. Twelve practice trials preceded the experimental blocks.

Results and discussion

15The latencies of incorrect responses and those faster than 150 ms or slower than 2000 ms were excluded from the analyses. In the dual task conditions, the elimination of one response time entailed the elimination of the corresponding time from the other task. Errors were not frequent (less than 1%), but about 10% of latencies were eliminated as outliers (single manual 4%, single vocal 3%; dual 16%).

16Table 1

17Results of Experiment 1: latencies of vocal responses (in ms)

Task

Single

Dual

Experiment 1

M

SD

M

SD

Set size = 2

518

67

667

190

Set size = 8

617

62

749

156

18The influence of the set size (2, 8) and condition (single, dual) were analysed in separate ANOVAs for correct vocal and manual responses. First, we compared the mean latencies of the correct vocal responses through a three-factor model, with set size and condition as within-subject variables and task order (single vocal first, dual first) as a between-subject factor. The results showed two significant main effects: set size (F(1, 20) = 64.84, p < .001) and task (F(1, 20) = 33.18, p < .001): retrieving a name among eight alternatives was about 100 ms slower than among only two, and the vocal response was delayed by about 150 ms when a concurrent gesture was required (see Table 1). These effects were additive (non-significant two-way interaction, F < 1) and the order of the tasks did not have any influence. Thus, increasing the duration of the response-selection stage did not absorb the interference effect, and the results did not confirm this particular prediction of the response-coupling model.

19However, in a second analysis, we considered the latencies of the vocal responses in the dual task condition after sorting the corresponding manual response times into four quartiles. We found a significant linear increase in these latencies from quartile to quartile (F(3,69) = 14.53, p < .001); and longer latencies of vocal responses were associated with longer latencies of manual responses.

20Table 2.

21Results of Experiment 1: initiation times for manual responses (in ms)

Task

Single

Dual

M

SD

M

SD

Set size = 2

340

 89

363

 78

Set size = 8

398

104

414

104

22In a third ANOVA, we analysed the mean latencies of the correct manual responses in a three-factor model (2 orders: single manual first, dual first; 2 stimulus set sizes; 2 conditions). The responses were slower when pictures of 8 objects were used than when only two objects were presented (Table 2). The main effect of stimulus set size was significant (F(1, 22) = 14.22, p = .001), although the manual response to be given (a simulated pinch or clench) were unchanged.

23The main effect of condition (single or dual task) was not significant, but this factor interacted with the order of the tasks (F(1, 22) = 9.38, p < .01). Dual-task interference was only observed in the subgroup of participants performing the dual task before the single task. Neither the two-way interaction between set size and condition (F<1) nor the three-way interaction (F<1) were significant. Thus, increasing the difficulty of selecting the vocal response delayed the execution of the manual response, but only in the subset of participants who had not already practiced the single task. In addition, we found that the dual-task interference effect and the set-size effect were additive. Thus the predictions of the shared capacity model were disconfirmed. The results are more compatible with an alternative model, such as the EPIC model (Meyer & Kieras, 1997; 1999), assuming multiple sources of interference.

Experiment 2: Preparing short versus long utterances

24The main aim of the second experiment was to examine the effect of increasing the difficulty of planning the vocal responses (long or short phrases) on the initiation times of gestures and the patterns of dual-task interference effects.

Method

25Participants. Twenty-four young adults (aged 21–28; 12 women and 12 men) volunteered for the study (Vanden Bulke, 2001).

26Materials. The four hand shapes shown in Figure 1 in Klatzky et al. (1987) and called the ‘pinch’, poke’, ‘clench’ and ‘palm’ gestures by these authors were scanned for computer presentation. The participants were shown these pictures and instructed to learn to associate them with the names of four objects: a necklace (collier), a button (bouton), a hammer (marteau) , and a ball (ballon) respectively. The same apparatus and PsyScope software was used as in Experiment 1, and the trials were structured in a similar way.

27Procedure. All the participants were presented with five blocks of 40 pictures of hand shapes (4 shapes x 10 presentations in a random order). Each block was preceded by 12 practice trials to allow the participants to learn the associations between the stimuli and the responses. In the simple manual condition, the participants were asked to imitate the gesture shown in the picture. In the short vocal condition, they were to name the object associated with the gesture with the bisyllabic French noun, for instance, collier (necklace) in response to the picture of the pinch gesture. In the long vocal condition, they had to utter a predefined 5-word phrase (8 or 9 syllables) consisting of a verb, article, object noun, preposition and complement noun (for instance, ‘attacher un collier de perles’ (the grammatical structures are different in English, but this phrase means ‘to fasten a necklace of pearls’). In the dual conditions, they had to imitate the gesture and say either the short or the long phrase. The order of the blocks (simple/dual conditions and long/short utterances) was balanced across participants but the two short (simple/dual) and the two long utterance conditions were grouped to avoid confusion in response selection.

Results and discussion

28The latencies of incorrect responses (less than 1%) and those faster than 150 ms or slower than 2000 ms (3%) were excluded from the analyses.

29Table 3.

30Results of Experiment 2: latencies of vocal responses (in ms)

Task

Single

Dual

M

SD

M

SD

Short utterance

718

133

810

175

Long utterance

805

150

834

199

31The latencies of correct vocal and manual responses were analysed in separate ANOVAs. In a first three-way ANOVA, we analysed the latencies of correct vocal responses with utterance length (long, short) and condition (single, dual task) as within-subject factors and the two orders (vocal first, dual first) as a between-subject factor (Table 3). As expected, main effects of the utterance length (F(1, 22) = 20.19, p < .001) and the condition (F(1, 22) = 5.01, p < .05) were found (see Table 3). There was also a two-way interaction between order and condition (F(1, 22) = 4.41, p < .05), due to the interference effects being larger in the dual-first than in the vocal-first order. Another significant two-way interaction was observed between length and condition (F(1, 22) = 16.78, p < .001), showing under-additivity. The interference effects were larger in the short utterance condition, especially in the dual-first subgroup, yielding a significant three-way interaction (F(1, 22) = 5.91, p < .05). Thus, the manipulation of utterance length to increase preparation time was effective. Yet, the interference effect was absent in the vocal-first subgroup of participants (not shown).

32As in Experiment 1, we also compared the mean vocal response latencies in dual task conditions after sorting the corresponding manual response latencies into four quartiles. This quartile effect was highly significant (F(3, 69) = 56.39, p < .001), as slower initiations of vocal responses were associated with longer manual response latencies.

33Table 4.

34Results of Experiment 2: initiation times of manual responses (in ms)

Task

Single

Dual

M

SD

M

SD

No vocal response

413

64

Short utterance

487

127

Long utterance

522

129

35In a third ANOVA, we compared the mean latencies of the correct manual responses in a two-factor model: two orders (manual first, dual first) and three conditions (single, dual short, dual long). The results showed a significant main effect of condition (F(1, 22) = 12.38, p < .001), qualified by a two-way interaction (F(1, 22) = 3.18, p = .051): Table 4. The interference effects were also larger in the dual-first subgroup and in the dual-long condition (not shown). The condition effect was further analysed by means of orthogonal contrasts. We found a significant difference between single and dual tasks (pooled short- and long-utterance conditions) (F(1, 23) = 14.56, p = .001). The difference between short- and long-utterance conditions in the dual task was only marginally significant (F(1, 23) = 3.81, p = .063). The difference between the latencies in the single, manual task and the dual-task, one-word utterance condition was also significant (F(1, 23) = 7.39, p = .012). There was an interference effect of 73 ms in that contrast, and the difference was only slightly larger (109 ms) when the vocal response was longer.

36Thus, increasing the preparation time for spoken utterances delayed the initiation time of concurrent manual gestures. However increased difficulty of vocal response preparation did not increase the interference effects. Furthermore, with sufficient practice (in the vocal-first, dual-second subgroup), the increased preparation time for longer utterances absorbed the interference effect due to the concurrent preparation of the manual response.

37These outcomes were inconsistent with the capacity sharing models. Despite the significant effects of utterance length on response latencies, it is possible that, in the long multi-word utterance conditions, articulation can begin before complete preparation of the whole phrase and thus, some preparation may still be taking place after the initiation of the vocal response (see e.g. Griffin, 2003). This could explain some of the differences in the interference effects in Experiments 1 and 2.

General discussion

38The aim of the present study was to understand why, in some dual-task conditions (Levelt et al., 1985; Feyereisen, 1997), interference effects are larger in the vocal than in the manual modality. The explanation provided by the response-coupling model (see Figure 8C in Fagot & Pashler, 1992), would suggest that when two responses have to be given to a single signal, the faster production system (presumed to be speech) ‘awaits’ the completion of response planning in the slower production system before starting execution, so as to synchronise the two responses. Thus, increasing the planning time for vocal responses should eliminate or reverse the asymmetry by delaying the onset of concurrent gestures, and absorbing the interference effect in the vocal modality. The results of Experiments 1 and 2 were not consistent with these predictions, although slower vocal responses were associated with slower manual responses. The main problem with such an explanation is that response initiation is usually faster in the manual modality and thus, there is no strong reason to think that speech planning precedes gesture planningii.

39The results may be more compatible with some alternative models. Capacity-sharing models can explain variation in the amount of interference in relation to task requirements. Interference effects in the faster task (presumed to be gesture production) might only be observed when the resource demands of the slower task increase. Increased interference effects were found in the most demanding conditions of Levelt et al.’s (1985) Experiment 3. By contrast, in the present study, interference and task difficulty effects were either additive (Experiment 1) or under-additive (Experiment 2) and thus, the results were inconsistent with the prediction of the capacity-sharing model.

40By elimination, we are led to accept a variant of the executive-control model, although in the present case this does not allow us to make as precise predictions about the temporal relationships between vocal and manual responses as do the simulations of the EPIC architecture presented by Meyer and Kieras (1997, 1999). As this model assumes multiple sources of interference, individual variability and flexibility in adaptive control, and strategic postponement of response execution in dual-task performance, it is able to fit the results of the present experiments more closely than rival models. The interference effects found in the vocal and manual modalities in Experiment 1 might relate to the number of stimulus-to-response mapping rules stored in working memory. The task demands differed in Experiment 2, in which the gestures were simple copies of the pictures, whereas the vocal responses involved the re-activation of the associations between hand shapes, objects, and phrases referring to these objects. The EPIC model can accommodate the fact that different procedures produce different results, which the response-coupling and resource-sharing models cannot.

41At first sight, the present findings are at odds with two claims made in recent years in the literature on gestures and speech. The first is that the production of a representational gesture can facilitate access to its lexical affiliate (see Krauss & Hadar, 1999 for a review and Feyereisen, 2006 for a discussion). Evidence in favour of this hypothesis remains limited and the facilitative effect might be more likely occur earlier, during the conceptualisation stage (de Ruiter, 2000; Kita, 2000). The other claim is that gesture production alleviates the cognitive load in problem solving tasks by liberating space in working memory (Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001). The interference effects found in dual-task conditions seem inconsistent with any of these ‘facilitative’ hypotheses.

42However caution is needed in attempts to bring together observations made on different time scales and with different dependent variables. In the present experiments, we found that latencies of vocal responses were about 100 ms longer in the dual- than in the single-task conditions. This additional delay might improve performance by allowing the speakers to encode their verbal representations in more distinctive or more elaborate ways. In their current formulations, the facilitative hypotheses do not explicitly predict shortening of verbal response latencies. Let us add that, in studies of lexical retrieval using priming paradigms, the presentation of a semantic prime sometimes delayed, rather than sped up the production of the target word, depending on the temporal relations between the prime and the stimulus (see Bock’s 1996 review). Speech production models assume competition among the alternative response, and thus presenting the prime ‘fruit’ together with the picture of a banana may cause interference rather than facilitation. From such a perspective, gestural facilitation can only be expected if the gesture precedes the word it relates to by a significant intervaliii.

43On a different tack, the relevance of the present findings for the study of synchrony between gestures and speech in more natural discourse production conditions can be questioned. Unlike research on language production (reviewed by Bock, 1996), research on gestures has rarely relied on mental chronometry. Yet, just as picture naming enables investigators to analyse the time course of lexical access, experimental elicitation of manual movements in the present study can simulate the production of everyday gestures. The careful naturalistic observations reported by Kendon (2004) indicated that, in spontaneous conversation, empty-handed pinch and clench movements may serve communicative functions. More fundamentally, Kendon assumed the same kind of strategic control of the vocal and gestural aspects of the utterances as in the present study. To achieve synchrony, silent pauses are sometimes introduced to ‘await’ the onset of gesture execution, whereas in other circumstances, gestures are frozen to allow the insertion of parenthetical comments. In most cases, however, gesture onset precedes the vocalisation of the words they relate to (Bernardis & Gentilucci, 2006; Morrel-Samuels & Krauss, 1992; Kendon, 2004). Nonetheless, despite these similarities, it must be admitted that the manual responses elicited in the present study differed in some important respects from spontaneous conversational gestures. Notably, they were repetitive, non-communicative, and performed in relation to instructions and pictures rather than to self-generated images.

44This limitation of the current study actually relates to several issues. One is the question of whether actual conversational gestures would interfere with speech planning in the same way as nonverbal actions in other settings (driving, walking, key pressing during psychology experiments, etc.). Generally speaking, interference effects are significantly reduced when the two effectors are used in a highly coordinated way, for example in some bimanual activities (Swinnen & Wenderoth, 2004). Further studies are needed to analyse the synchronisation of rhythmic gestures with peaks of vocal intensity, but a preliminary investigation has indicated that beat gestures do not consistently coincide with stressed syllables (McClave, 1994). Another question concerns the use of the term ‘gesture’ to refer to simulated object manipulations. In the present experiments, stimuli were displayed on a computer screen, and thus the movements elicited were more akin to mime than to real object grasping or touching. Recent findings from neuro-imaging studies have confirmed previous kinematic analyses showing that the cerebral control over actual and imagined object use was different (Kroliczak, Cavina-Pratesi, Goodman & Culham, 2007).

45The purpose of the present study was to examine how gestures and speech synchronise, by manipulating the form and content of the verbal responses. There was an unavoidable trade-off between experimental control and ecological validity. Studying genuine discourse situations, Pine, Lufkin, Kirk and Messer (2007) distinguished semantic and temporal asynchrony (different content conveyed in speech and gesture at different times). By analogy, we could distinguish semantic and temporal synchrony. Semantic synchrony is probably what Kendon (2004) and McNeill (2005) are referring to when they say that gestures and speech often express the same underlying idea simultaneously, i.e., within the same utterance. Temporal synchrony is achieved at a more fine-grained level, and we found that longer initiation periods for manual responses were associated with longer vocal-responses latencies. These verbal responses were faster when no movement was required, because planning a manual movement requires time, a limited resource.

46The notion of capacity limitation is central in cognitive psychology. Human beings display remarkable planning abilities, in speech production and in manual activity for instance, but they have only a limited ability to perform multiple tasks simultaneously. One solution to the problem is to delay one process to give priority to another, for example, by interrupting speech planning to prepare a co-verbal gesture, or by using a gesture in the silent interval that is caused by a failure in word retrieval. Alternatively, speakers may elaborate more complex plans that integrate manual and vocal events to reach their communicative goal more effectively, but involve a longer planning phase than is needed for simpler actions need; the length of such a planning phase co-varies with the demands of the conjoined tasks. Finally, speakers may organise the production of hand and speech movements flexibly by adapting their time allocation to their intentions. The findings of the present study are more consistent with this final suggestion than with the other ones.

Acknowledgements

47Pierre Feyereisen was funded as Research Director by the National Fund for Scientific Research, Belgium. Gratitude is expressed to Laurence Vanden Bulke who conducted Experiment 2 under the author’s supervision.

Haut de page

Bibliographie

Bernardis, P., & Gentilucci, M. (2006). Speech and gesture share the sale communication system. Neuropsychologia, 44, 178-190.

Bock, K. (1996). Language production: methods and methodologies. Psychonomic Bulletin & Review, 3, 395–421.

Butterworth, B., & Hadar, U. (1989). Gesture, speech, and computational stages: a reply to McNeill. Psychological Review, 96, 168–174.

Cohen, J.D., MacWhinney, B., Flatt, M., & Provost, J. (1993). PsyScope: a new graphic interactive environment for designing psychology experiments. Behavioral Research Methods, Instruments, and Computers, 25, 257–271.

Cycowicz, Y. M., Friedman, D., Rothstein, M., & Snodgrass, J. G. (1997). Picture naming by young children: norms for name agreement, familiarity, and visual complexity. Journal of Experimental Child Psychology, 65, 171–237.

De Jong, R. (1993). Multiple bottlenecks in overlapping task performance. Journal of Experimental Psychology: Human Perception and Performance, 19, 965–980.

de Ruiter, J.-P. (2000). The production of gesture and speech. In D. McNeill (Ed.), Language and Gesture (pp. 284–311). Cambridge: Cambridge University Press.

Fagot, C., & Pashler, H. (1992). Making two responses to a single object: implications for the central attentional bottleneck. Journal of Experimental Psychology: Human Perception and Performance, 18, 1058–1079.

Ferreira, F. (1991). Effects of length and syntactic complexity on initiation times for prepared utterances. Journal of Memory and Language, 30, 210–233.

Feyereisen, P. (1987). Gestures and speech, interactions and separations: a reply to McNeill (1985). Psychological Review, 94, 493–498.

Feyereisen, P. (1997). The competition between gesture and speech production in dual-task paradigms. Journal of Memory and Language , 36, 13–33.

Feyereisen, P. (2006) How could gesture facilitate lexical access? Advances in Speech-Language Pathology, 8, 128–133.

Goldin-Meadow, S., Nusbaum, H., Kelly, S., & Wagner, S. (2001). Explaining math: gesturing lightens the load. Psychological Science, 12, 516–522.

Griffin, Z.M. (2003). A reversed word length effect in coordinating the preparation and articulation of words in speaking. Psychonomic Bulletin and Review, 10, 603–609.

Hick, W.E. (1952). On the rate of gain of information. Quarterly Journal of Experimental Psychology, 4, 11–26.

Kendon, A. (2004). Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.

Kita, S. (2000). How representational gestures help speaking. In D. McNeill (Ed.), Language and Gesture (pp. 162–185). Cambridge: Cambridge University Press.

Klatzky, R. L., McCloskey, B., Doherty, S., Pellegrino, J., & Smith, T. (1987). Knowledge about hand shaping and knowledge about objects. Journal of Motor Behavior, 19, 187–213.

Krauss, R.M., & Hadar, U. (1999). The role of speech-related arm/hand gestures in word retrieval. In L. Messing & R. Campbell (Eds.), Gesture, Speech, and Sign (pp. 93–116). New York: Oxford University Press.

Kroliczak, G., Cavina-Pratesi, C., Goodman, D.A., & Culham, J.C. (2007). What does the brain do when you fake it? An fMRI study of pantomimed and real grasping. Journal of Neurophysiology, 97, 2410–2422.

Levelt, W. J.M. (1989). Speaking: From Intention to Articulation. Cambridge, MA: MIT Press.

Levelt, W. J. M., Richardson, G., & La Heij, W. (1985). Pointing and voicing in deictic expressions. Journal of Memory and Language, 24, 133–164.

Lien, M. C., & Proctor, R.W. (2002). Stimulus-response compatibility and psychological refractory period effects: implications for response selection. Psychonomic Bulletin & Review, 9, 212–238.

Marois, R., & Ivanoff, J. (2005). Capacity limits of information processing in the brain. Trends in Cognitive Sciences, 9, 296–305.

McClave, E. (1994). Gestural beats: the rhythm hypothesis. Journal of Psycholinguistic Research, 23, 45–66.

McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92, 350–371.

McNeill, D. (1989). A straight path – to where? Reply to Butterworth and Hadar. Psychological Review, 96, 175–179.

McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: Chicago University Press.

McNeill, D. (2005). Gesture and Thought. Chicago: Chicago University Press.

Meyer, D.E., & Kieras, D.E. (1997). A computational theory of executive cognitive processes and multiple task performance. Part 1: basic mechanisms. Psychological Review, 104, 3–65.

Meyer, D.E., & Kieras, D.E. (1999). Précis to a practical unified theory of cognition and action: some lessons from EPIC computational models of human multiple-task performance. In D. Gopher & A. Koriat (Eds.), Attention and Performance XVII. Cognitive Regulation of Performance: Interaction of Theory and Application (pp. 17–88). Cambridge, MA: MIT Press.

Morrel-Samuels, P., & Krauss, R. M (1992). Word familiarity predicts temporal asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 615–622.

Navon, D., & Miller, J. (2002). Queuing or sharing? A critical evaluation of the single-bottleneck notion. Cognitive Psychology, 44, 193–251.

Pashler, H. (1994). Dual-task interference in simple tasks: data and theory. Psychological Bulletin, 116, 220–244.

Pine, K.J, Lufkin, N., Kirk, E., & Messer, D. (2007). A microgenetic analysis of the relationship between speech and gesture in children: evidence for semantic and temporal asynchrony. Language and Cognitive Processes, 22, 234–246.

Swinnen, S.P., & Wenderoth, N. (2004). Two hands, one brain: cognitive neuroscience of bimanual skill. Trends in Cognitive Sciences, 8, 18–25.

Vanden Bulcke, L. (2001). Compétition entre gestes et parole [Competition between Gestures and Speech]. Unpublished master’s thesis, University of Louvain, Louvain-la-Neuve, Belgium.

Haut de page

Note de fin

i McNeill (1992: pp. 26 and 375) distinguished the preparation phase of gesture, which anticipates co-expressive speech from the ‘stroke’ phase, which synchronises with speech. Evidence was based on the coding of videotapes by marking gesture phases on speech transcriptions. The stroke phase was defined by semantic and kinesic features. It may last until the end of the utterance. “Thus, the gesture as an organised expression of meaning can extend through the whole stretch of parallel speech. Time in milliseconds is not, in and of itself, important for questions of internal mental processes; what is crucial is time in relation to meaningful parallels between gestures and speech” (McNeill, 1989, p. 176). The philosophy underlying the present study is quite different.
ii The difference between the latencies of manual and vocal responses may be due to their physical characteristics (key release versus activation of the voice key) or to the possibility of selecting hand shapes after initiating the movement.
iii In a pilot experiment, we were unable to produce this expected result, partly because repetition priming effects masked the differences in vocal response latencies following performance of matching and mismatching gestures, as the same target words had to be given after consistent and inconsistent primes. The second response was significantly faster than the first one, whatever the preceding gesture. In addition, identical gestures were used to prime different target words, which might thus compete for output.
Haut de page

Pour citer cet article

Référence électronique

Pierre Feyereisen, « How do gesture and speech production synchronise? »Current psychology letters [En ligne], 22, Vol. 2, 2007 | 2007, mis en ligne le 09 juillet 2007, consulté le 29 mars 2024. URL : http://journals.openedition.org/cpl/1561 ; DOI : https://doi.org/10.4000/cpl.1561

Haut de page

Auteur

Pierre Feyereisen

Université catholique de Louvain at Louvain-la-Neuve, Belgium

Haut de page

Droits d’auteur

Le texte et les autres éléments (illustrations, fichiers annexes importés), sont « Tous droits réservés », sauf mention contraire.

Haut de page
Search OpenEdition Search

You will be redirected to OpenEdition Search