Can AI read an animal’s footsteps?

When I look through vibration recordings, I spend a lot of time looking at plots. That made me wonder: could a language model with vision help with this part of the work?

We tried it with desert recordings from nodal geophones. The inputs were waveform and spectrogram images. After reading expert-reviewed examples, the agent proposed event labels such as vehicle, sheep flock, Tarim hare, and gerbil. It also allowed mixed and uncertain cases.

The useful part was asking it to show its reasoning on the plot: where an event starts and ends, what visual features support the label, and what else might explain it. One long recording can contain several different sources, so we moved to an event-by-event review instead of assigning one label to the whole image.

This is still a visual interpretation experiment. A label that sounds convincing is not proof that an animal was present. The blind predictions have not yet been scored against the answers, and image appearance can change with filtering, color scale, and background noise. These recordings are from geophones, not DAS.

I would like to use this as a first pass through a large collection of plots, with the uncertain cases kept visible for human review. The next useful result would be a careful comparison with independent labels, including the events the model misses.