01  The question

Speakers plan in chunks. Nobody has asked what the hands do to the chunk.

Speaking means deciding what to say before saying it, but speakers do not always plan a whole sentence first. Often they plan only the first chunk, start talking, and plan the rest on the way (Levelt, 1989). The size of that chunk, the planning scope, depends on time pressure (Ferreira & Swets, 2002), sentence complexity (Smith & Wheeldon, 1999) and mental load (Wagner et al., 2010), and it differs from speaker to speaker.

These studies treat speech on its own. People gesture when they talk, and gesture is tied to speech at two levels.

Meaning. A gesture often carries the same idea as the words it goes with, such as a raised hand while saying a plane is “above” a tree. One gesture spans several words, and its most forceful part, the stroke, falls on the word whose meaning it shares (Krivokapić, 2014). The arm is slower than the mouth, so it has to start moving hundreds of milliseconds before that word. That is only possible if the speech plan already reaches that far.

Movement. A gesture pushes against the trunk. It disturbs breathing and the voice (Pouw et al., 2020) and shifts forces on the ground. Postural and breathing muscles brace before the arm moves (Bouisset & Zattara, 1987), and that bracing has to fit around the breath taken for speech.

So the time course of a gesture, and of the body’s preparation for it, may show how far a speaker has planned.

02  Three outcomes

Each possible result says something different about how speech and gesture are linked

  1. Constraint My hypothesis

    Without gesture, speakers may plan the whole sentence but do not have to. With gesture, they have to plan at least through the word the gesture goes with, so the stroke lands on time. Gesture would then be part of speech production itself: the plan for an utterance says when the hands move as well as which words to say.

    Gesture → more whole-sentence planning

  2. Competition

    Preparing a precise movement competes with speech for limited resources, and scope shrinks the way it does under mental load.

    Gesture → shorter planning scope

  3. Independence

    Gesture leaves scope unchanged. Speech and gesture would be separate systems that are only coordinated during execution.

    Gesture → no change in planning

03  The task

A deadline that arrives at an unpredictable time

To test these outcomes I need to know, on each trial, what the body did and how the sentence was planned. Response time says something about planning but nothing about the arm. Video tracks the hand coarsely in two dimensions and misses the forces and breathing behind it. The task below makes planning visible in response time, and the recording system captures the rest.

  1. A picture appears. A park scene has five landmarks: a tree, a cloud, a hill, a city and a pond. An object shows up next to, above, below or inside one of them.
  2. A delay of 0 to 2 seconds. The speaker can plan during it but cannot predict its length.
  3. A tone says go. The speaker describes the picture as fast as possible: “A plane is above the city.”
  4. Two kinds of block. Speech only, or speech with a gesture that shows where the object is. The stroke should land on the relation word.

Response time is the interval from the tone to the start of speech. After a long delay the sentence is already planned, so response time is short whatever the speaker’s habit. After a short delay the speaker has to choose: wait until the whole sentence is ready, or start with the first chunk and plan the rest while talking. Planning more takes longer, so the two choices leave different response times.

Fig. 1  One trial, five signalsSchematic
Schematic of one trial in the scene description task Five aligned signals over about three seconds. A picture appears, a go-cue tone follows after a short delay, and speech begins about 700 milliseconds later with the sentence "a plane is above a tree". Breathing shows an inhale before speech. Posture shows a brief shift just before the hand starts to rise. The hand rises through a preparation phase and its stroke peaks during the word "above". a plane is above a tree pre-speech inhale shift before the hand moves preparation stroke retraction target word Breathing belt Posture force plate Hand height Speech audio Words picture go-cue speech onset delay response time 0 0.5 1 1.5 2 2.5 s
What is recorded on every trial. An illustration, not data. The hand has to leave early for its stroke to land on the relation word, and posture shifts before the hand moves.
04  What the model predicts

A stage model of the task, with every number sourced

Before collecting data I wrote down what a trial should look like if speech is planned in stages. The scene is worked out in one step. The sentence is then encoded one phrase at a time: [a plane] [is above the tree]. A speaker can start once the first phrase is ready, or wait for both.

StageMeanWhere the number comes from
Working out the message250 ms175 ms for one object (Indefrey & Levelt, 2004), plus about 75 ms for the relation, from the pilot
Encoding the first phrase280 msRetrieving and encoding one word (Indefrey & Levelt, 2004)
Encoding the second phrase too+150 msA second clause adds 142 ms (Smith & Wheeldon, 1999); two pilot estimates near 150 ms. The least certain value
Getting started after the tone300 ms267–303 ms in tone-cued naming (Kawamoto et al., 2008); pilot floor of 293–306 ms
Motor cost of gesturing90 msPilot: gesture minus speech only at a 1 s delay (+93 ms)

response time = max(planning time − delay, 0) + getting started + gesture cost Planning time is 530 ms for the first phrase and 680 ms for both, at average speed. Stage times vary from trial to trial with a long right tail, and speakers differ in speed and in how often they plan both phrases.

Fig. 2  Predicted response timeModel and pilot
Both phrases planned before speaking First phrase only Pilot means, speech only

Response time from the tone (ms)

The two planning scopes differ only at short delays. Lines are the model at average speed. Dots are the pilot’s mean response time in 0.2 s bins of delay, speech only, both speakers. The pilot keeps falling until about 1 s, later than the model.
Show the values
Delay (s)First phrase onlyBoth phrasesPilot mean
0.1730880726
0.3530680602
0.5330480515
0.7300300418
0.9300300389
1.1300300270
1.3300300322
1.5300300296
1.7300300303
1.9300300285

The test

If gesture forces the second phrase to be planned first, it adds up to 150 ms, but only while planning is still running. At long delays any gesture cost is just the cost of moving the arm. So the test is a difference of differences: each speaker’s gesture cost at delays under 0.5 s, minus their gesture cost at delays of 1 s or more. The general motor cost cancels out.

How the model got here

  • Version 1Two strategies. Every trial is either planned ahead or planned on the go, and a mixture model sorts them. Simulations showed that unless gesture gets its own direct effect on speech onset and pauses, the model reads an ordinary gesture cost as a change of strategy: 28–58% false positives, against 6–8% once corrected.
  • Version 2Continuous scope. Planning runs word by word and scope varies smoothly. When data are generated this way, the two-strategy model breaks down in 68–86% of simulated studies and can give the wrong sign. A plain mixed model on response time is safer.
  • Version 3Stages per word. Stage times from the literature, 280 ms for each prosodic word. The pilot ruled this out: naming one, two or three words straight away took 587, 614 and 660 ms, not 913, 1,193 and 1,473.
  • CurrentStages per phrase. One conceptual step, then one phrase at a time, with a faster start after a tone and a motor cost for gesture. Checked against the pilot.
05  How many speakers

Thirty speakers detect the effect in four simulated studies out of five

I simulated 1,000 complete studies for each sample size, with 200 trials per speaker in each condition, and ran the planned test on every one. If the constraint hypothesis is true at the model’s default values, the average effect is about 73 ms.

Fig. 3  Power1,000 simulated studies per point
The effect is real: studies that detect it No effect: studies that report one anyway

Share of simulated studies with a significant gesture effect

80% at 30 speakers, 89% at 40, 97% at 60. The grey line marks 80%. With no real effect, 2–4% of studies report one.
Show the values
SpeakersDetect a real effectFalse alarms
1030.5%2.7%
1547.1%3.9%
2057.0%1.6%
2569.3%2.4%
3079.9%2.1%
4089.1%2.2%
5094.9%2.4%
6097.2%2.7%

What could break it

Power depends most on something two pilot speakers cannot pin down: how much speakers differ in when the motor cost of gesturing shows up. A cost that sits at short delays for one speaker and at long delays for another looks exactly like the effect, or hides it.

Speaker-to-speaker spread in the motor costPower at 30 speakersAt 60
None100%100%
SD of 50 ms100%100%
SD of 100 ms, the default79%98%
SD of 150 ms52%78%
SD of 200 ms35%59%

The cost of the second phrase matters too. If it is 75 ms instead of 150, power at 30 speakers falls to 29%. If it is 300 ms, power is near 100%. I report these because they are the weak points of the design.

06  Pilot results

Two speakers, to check the task and the system

The pilot was run in November 2026 to check that the task works and that the recordings line up. It was not meant to test the hypothesis, and it does not.

2speakers, each in both conditions
373trials kept for analysis
~300 msresponse time after long delays, speech only
~1 sdelay at which response time stops falling
Fig. 4  Response time against delayEvery kept trial
Speech with gesture Speech only

Response time from the tone (ms)

Speaker 1

Speaker 2

Response time falls as the delay grows, then levels off. Each dot is one trial. Lines are least-squares fits of a falling segment that meets a flat floor; the bend is where they meet. Speaker 1’s gesture fit has no clear bend (95% interval 0.26–1.80 s).
Show the fitted values
SpeakerConditionTrialsBend (s)95% intervalFloor (ms)Slope (ms per s)
1Speech only990.890.52–1.19306−454
1With gesture991.800.26–1.80305−271
2Speech only821.040.59–1.20293−512
2With gesture930.940.64–1.00394−438
  • The curve bends near one second

    Without gesture, response time stops falling at 0.89 s for one speaker and 1.04 s for the other. That is close to what planning the whole sentence predicts, though the intervals reach down to about 0.5 s.

  • The amount left to plan varies

    Response time falls by about half a millisecond for each millisecond of delay. If every trial had the same planning left at the tone, it would fall by one.

  • It is planning, not expectancy

    The delay on the previous trial does not predict response time, so the curve is not just speakers guessing when the tone will come.

  • Words inside a phrase are nearly free

    Naming one, two or three words with no delay took 587, 614 and 660 ms. Onset barely moves, which is why the model now plans by phrase.

  • Gesture costs about 90 ms, in different places

    At a 1 s delay, gesture trials started 93 ms later. For one speaker the cost sat at short delays and for the other at long delays.

  • Two ways to handle an unfinished plan

    After early tones, one speaker started on time and stretched the sentence by about 150 ms. The other waited and started later.

  • The stroke arrives just before its word

    The fastest point of the stroke came before the relation word on 169 of 190 gesture trials, and the stroke ended within about 150 ms of that word’s start.

Read with care There are two speakers, and I am one of them, so one speaker knew the hypothesis. The gesture block always came last, so gesture and practice are mixed together. The analyses were exploratory. I use these results to set up the model and the study, not as evidence for or against the hypothesis.

07  Reading the body

What each kind of planning should look like outside response time

Once the deadline task says how a trial was planned, I can ask what the voice and the body were doing on that trial. These are predictions. If they hold, the same markers can classify planning in ordinary speech, where there is no tone and no response time.

SignalWhole sentence plannedPlanned on the go
SpeechFluent, with few filled pauses and longer phrasesMore “um”, shorter phrases
BreathingA deeper breath before starting (Fuchs et al., 2013)A shallower breath
GestureOne smooth stroke that lands on its wordA jerky, stop-start preparation and a later stroke
PosturePostural muscles adjust 50–150 ms before the arm moves (Bouisset & Zattara, 1987), so force plates may show a planned gesture before the hand does

First check In an early force-plate test with one speaker, weight shifted by 45–100 N around the start of speech on three takes with a gesture, and by about 7 N on one take without. Four takes show that the signal is there to measure, and no more than that.

08  Natural speech

Taking the paradigm outside the task

Lab tasks show what speakers can do. The second aim asks whether the same relationships hold when people just talk. Speakers will recall a personal experience while I record the same signals, once with their hands free and once with their hands restrained.

I will test whether fluent stretches go with smooth, well-timed strokes and disfluent ones with jerky preparation, and whether restraining the hands shifts speakers toward planning on the go. That revisits an old question, whether gesturing helps speech production, which has so far been tested with audio alone (Rauscher et al., 1996).

09  Why it matters

Models of speaking leave out the moving body

  • For theory

    Speech production models describe how far ahead speakers plan, not how that planning makes room for a moving body. Whichever outcome holds, accounts of how speech and gesture are linked will need revising.

  • For other researchers

    The project will release a public dataset of synchronized motion capture, force, respiration, audio and video for structured and natural speech, with the pipelines needed to use it. Few labs can record this.

  • For generated gesture

    Models that generate gesture for virtual agents and robots learn from limb positions alone. Planning markers could help them time gesture to speech the way people do.

  • For the clinic

    Stroke and Parkinson’s disease can disrupt speech, gesture and balance together. Knowing how they are planned together could inform gesture-based speech therapy.

References
  1. Bouisset, S., & Zattara, M. (1987). Journal of Biomechanics.
  2. Ferreira, F., & Swets, B. (2002). Journal of Memory and Language.
  3. Fuchs, S., et al. (2013). Journal of Phonetics.
  4. Indefrey, P., & Levelt, W. J. M. (2004). The spatial and temporal signatures of word production components. Cognition, 92(1–2), 101–144.
  5. Kawamoto, A. H., Liu, Q., Mura, K., & Sanchez, A. (2008). Articulatory preparation in the delayed naming task. Journal of Memory and Language, 58(2), 347–365.
  6. Krivokapić, J. (2014). Philosophical Transactions of the Royal Society B.
  7. Levelt, W. J. M. (1989). Speaking: From intention to articulation.
  8. Pouw, W., et al. (2020). Journal of the Acoustical Society of America.
  9. Rauscher, F. H., et al. (1996). Psychological Science.
  10. Smith, M., & Wheeldon, L. (1999). High level processing scope in spoken sentence production. Cognition, 73(3), 205–246.
  11. Wagner, V., et al. (2010). Journal of Experimental Psychology.