Tools  /  Software

Tools and software

Software I have written to make speech and gesture research faster to do and easier to check. What is public is linked. The rest is in daily use and will be released with the project’s dataset.

01  Public

Gesture annotation without special equipment

github.com/walterdych

Co-speech gesture annotation from video

Goes from a plain video of someone talking to gesture annotations that open in ELAN. A free body-tracking model finds the speaker’s hands in every frame, and from how the hands move the notebooks find where each gesture starts and ends and the key moments inside it.

It runs on an ordinary laptop, the videos never leave the computer, and every step is explained in plain language, so it can be used with no programming experience. I built it so labs without motion capture can still study gesture.

This line of work began as the MultiGest tool, presented at ICPhS 2023, CorpusPhon 2024 and the International Multimodal Communication Symposium 2024. The current public version is packaged as a hands-on workshop.

cospeech-gesture-workshop on GitHub

Keypoints for more than one person

Extracts body keypoints for several people in the same video, for studies of conversation where both speakers are in frame.

multipose-keypoint-extraction on GitHub

02  In use, release planned

The pipeline behind the doctoral project

How the steps fit together

Session builder

Turns one recording session into one aligned dataset. Movement, forces, audio and word timings are all placed on the master clock, split by trial, and written as tables, with response time, pauses, fillers and speech rate computed for each trial and a quality report for the synchronization.

Gesture annotation from motion capture

The video tool rebuilt for three-dimensional data. It finds gesture units from rest to rest, names the phases inside them (preparation, stroke, hold, retraction), groups them into phrases, and marks candidate apexes. Every result records which rule produced it, so an annotation can be traced and argued with.

The stroke on each trial is the wrist’s fastest movement that is not the return to rest, fitted with a minimum-jerk speed profile to give its onset, duration and how well it fits.

Point-light and trial videos

Renders every trial as a point-light figure, front and side, next to the spectrogram with its aligned words and the speed, acceleration and jerk of each wrist. A small viewer steps through trials a frame at a time. The figure on this site’s home page is drawn from the same data.

Experiment control

The scene description task. It draws the scene, triggers the go tone and the trial marker on the audio hardware, and starts and stops each motion capture take, with generators for the trial lists.

Planning simulations

Notebooks that simulate the task under each hypothesis, run the planned analysis on thousands of simulated studies, and report power and false-positive rates.

What they found