Twelve cameras around a corner of the room
The capture area sits in the south-east corner of an 18 by 16.5 foot room. Two 12 foot rails on each wall hold the cameras, at 10 feet and at 5 feet, and one more stands on a tripod at the open corner. Six cameras look down from the high rails, five sit at 5 feet, and the tripod makes twelve.
Five signals from one speaker
| Signal | Instrument | Details |
|---|---|---|
| Movement | 10 OptiTrack Prime 13 cameras | A full-body skeleton including every finger joint: 51 points in three dimensions, 250 times a second. |
| Video | 2 OptiTrack Prime Color cameras | Color video on the same frame clock as the infrared cameras, for checking what the tracking shows. |
| Ground forces | AMTI force plates | Forces under the feet, sampled through the motion capture system at a whole multiple of the camera rate. |
| Breathing | Respiration belt | Chest movement, for the breath taken before speaking. |
| Speech | 3 microphones into TDT hardware | Recorded by the master clock itself, then combined and cleaned before transcription. |
| Task | PsychoPy with TDT Synapse | The task shows the scene, triggers the tone, marks each trial, and starts and stops the motion capture recording. |
Everything is placed on the audio hardware’s clock
The question depends on timing: whether the hand left before or after a word was planned can turn on tens of milliseconds. Each device keeps its own time, so the streams have to be mapped onto one clock, and the mapping has to be checked on every trial.
- The audio hardware is the master. The microphones are recorded by it, so anything measured from the audio is already on the master clock.
- Every camera frame sends a pulse. The motion capture system emits one pulse per frame, and the master clock records when each arrives.
- A line is fitted for each trial. Frame number is mapped to master time with a straight-line fit over that trial’s pulses, which absorbs drift and jitter.
- Force samples ride on the frame clock. The plates are sampled at a whole multiple of the camera rate, so each sample takes its time from the fitted line.
Raw files to word-aligned, gesture-annotated trials
- Clean the audio. The three microphone channels are combined and denoised, and the go tones are muted.
- Transcribe and align. Each trial is transcribed by a speech recognition model (Parakeet) and aligned with the Montreal Forced Aligner, giving the start and end of every word and sound.
- Build the session. Trials, movement, forces and speech are written as tables on the master clock, with response time, pauses, fillers and speech rate for each trial.
- Annotate the gestures. Each hand’s movement is split into preparation, stroke, hold and retraction, and the stroke is fitted with a minimum-jerk profile.
- Check by eye. Every trial gets a video of the point-light figure beside its spectrogram and hand kinematics, with a frame-by-frame viewer.
Pressure insoles looked promising and did not hold up
Before settling on force plates I tested pressure-sensing insoles, which would let a speaker walk around. Against the plates they followed the shape of each weight shift closely (r ≈ .99), but they reported only 55–69% of its size, read about 120 N with the foot off the ground, and drifted over a take.
The postural adjustments this project looks for are small and brief, so the plates stay.