Reconstructing the Vocal Tract with Differentiable Acoustic Simulation

We present a differentiable, GPU-accelerated acoustic simulator for the vocal tract that synthesizes speech by propagating sound along an acoustic tube model, and via its gradients can reconstruct the shape of the vocal tract solely from the sound it produces. Gradient descent succeeds despite the notoriously non-convex inverse mapping thanks to a frequency-domain formulation that is 70x more GPU-parallelizable than finite differences in time, a differentiable turbulence model for synthesizing consonants, and a neural-field parameterization of the geometry that accelerates convergence and escapes local minima. We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and couple our simulator with a generative model of MRI images to reconstruct one’s moving vocal tract from only their speech, without paired data.