01  The room

Twelve cameras around a corner of the room

The capture area sits in the south-east corner of an 18 by 16.5 foot room. Two 12 foot rails on each wall hold the cameras, at 10 feet and at 5 feet, and one more stands on a tripod at the open corner. Six cameras look down from the high rails, five sit at 5 feet, and the tripod makes twelve.

Fig. 1  Camera layoutDrag to rotate
3D view of the capture area inside the full roomDrag to rotate. The 12 by 12 foot capture area sits in the south-east corner of the room with a 4 foot storage strip at the far end. Twelve numbered cameras sit on two 12 foot rails at 10 feet and 5 feet, plus a tripod camera in the north-west corner of the capture area. Click a camera to show its field of view, dimensions and settings.
Click a camera to see where it points. The dashed box is the tracked volume, and the standing figure is a 6 foot person for scale. Dimensions are in feet.
02  What is recorded

Five signals from one speaker

SignalInstrumentDetails
Movement 10 OptiTrack Prime 13 cameras A full-body skeleton including every finger joint: 51 points in three dimensions, 250 times a second.
Video 2 OptiTrack Prime Color cameras Color video on the same frame clock as the infrared cameras, for checking what the tracking shows.
Ground forces AMTI force plates Forces under the feet, sampled through the motion capture system at a whole multiple of the camera rate.
Breathing Respiration belt Chest movement, for the breath taken before speaking.
Speech 3 microphones into TDT hardware Recorded by the master clock itself, then combined and cleaned before transcription.
Task PsychoPy with TDT Synapse The task shows the scene, triggers the tone, marks each trial, and starts and stops the motion capture recording.
03  One clock

Everything is placed on the audio hardware’s clock

The question depends on timing: whether the hand left before or after a word was planned can turn on tens of milliseconds. Each device keeps its own time, so the streams have to be mapped onto one clock, and the mapping has to be checked on every trial.

  1. The audio hardware is the master. The microphones are recorded by it, so anything measured from the audio is already on the master clock.
  2. Every camera frame sends a pulse. The motion capture system emits one pulse per frame, and the master clock records when each arrives.
  3. A line is fitted for each trial. Frame number is mapped to master time with a straight-line fit over that trial’s pulses, which absorbs drift and jitter.
  4. Force samples ride on the frame clock. The plates are sampled at a whole multiple of the camera rate, so each sample takes its time from the fitted line.
0.02 mslargest gap between a pulse and its fitted line
12pilot sessions checked
1,000trials in those sessions
250 Hzcamera rate measured on the master clock
04  From recording to data

Raw files to word-aligned, gesture-annotated trials

  1. Clean the audio. The three microphone channels are combined and denoised, and the go tones are muted.
  2. Transcribe and align. Each trial is transcribed by a speech recognition model (Parakeet) and aligned with the Montreal Forced Aligner, giving the start and end of every word and sound.
  3. Build the session. Trials, movement, forces and speech are written as tables on the master clock, with response time, pauses, fillers and speech rate for each trial.
  4. Annotate the gestures. Each hand’s movement is split into preparation, stroke, hold and retraction, and the stroke is fitted with a minimum-jerk profile.
  5. Check by eye. Every trial gets a video of the point-light figure beside its spectrogram and hand kinematics, with a frame-by-frame viewer.

The software behind each step

05  Tested and set aside

Pressure insoles looked promising and did not hold up

Before settling on force plates I tested pressure-sensing insoles, which would let a speaker walk around. Against the plates they followed the shape of each weight shift closely (r ≈ .99), but they reported only 55–69% of its size, read about 120 N with the foot off the ground, and drifted over a take.

The postural adjustments this project looks for are small and brief, so the plates stay.