[FEATURE] Allow Claude to hear and process audio

Hey, I am intrested in working on this! Can you assign it to me?

+1 with a concrete production workflow that shows how much scaffolding this gap forces today.

I run an AI-assisted music studio through Claude Code (voice conversion, mixing, mastering, vocal coaching) on a local RTX 4090. Because Claude cannot hear audio, every "listen" in my sessions is approximated by a locally-orchestrated tool chain:

1. faster-whisper transcription to check sung/rapped vocals against lyric sheets (word-error-rate gates)
2. librosa pyin/yin pitch tracking to diagnose "the vocal sounds too high / nasal / unstable"
3. ECAPA speaker embeddings to score whether a converted vocal actually sounds like the artist
4. ffmpeg/librosa spectral balance, LUFS, and transient measurements to approximate "how does this mix sound"

That's four models and hundreds of lines of glue to poorly imitate one capability: Claude listening to a WAV and saying "the vocal is muffled, the 808 is drowning the kick, the second verse drifts off-beat." In practice the metric proxies repeatedly missed defects that human listening caught instantly (garbled words, pitch flutter, crushed drum transients) until purpose-built detectors were written for each failure mode — a treadmill that native audio understanding would end.

Even a first step of transcription + paralinguistics (tone, energy, pitch contour) delivered natively through a Read-like tool for wav/mp3/flac would collapse most of this scaffold. Full audio understanding would unlock review workflows for music production, podcast editing, voice-app development, and accessibility QA that currently cannot live inside a Claude Code session at all.

Windows 11, Claude Code CLI + desktop app, local models attached via Bash/MCP.

Adding a concrete production case for this, because I hit the wall hard tonight and the shape of it might be useful.

I'm an independent artist running an AI-native music operation, and Claude Code is my engineer. Tonight it pulled my full 4,000+ clip catalog local, measured every track, clustered them, and ranked them for commercial readiness. It identified my strongest record without being told which one it was. Genuinely excellent work.

Then it hit the wall, and I had to tell it something it could not possibly know:

most of those aren't different songs — they're regenerations of the same song with something slightly changed, sometimes under a different name.

Any human hears that in two seconds. Claude had to take my word for it, then build chroma/MFCC fingerprinting with union-find clustering to approximate the judgment — which collapsed 882 clips into 395 actual songs. The engineering was impressive. It was also a prosthetic for a missing sense.

That pattern repeats through the entire workflow:

• It writes a voice-conversion chain but can't hear that the output sounds nasal.
• It masters a track to spec but can't hear the pitch drift I catch instantly.
• It once measured a "3-octave vocal range" that was actually a library bug. I said "the rapper isn't whispering" — and I was right. The meters were wrong and the ears were right.

We built an entire local perception stack specifically to work around this. It measures well. Measurement is not hearing. Every audio judgment still round-trips through me, and that round trip is the bottleneck in the whole operation.

Even a limited version would remove most of my manual loop:

1. Transcription with timing (does the vocal actually say the lyrics?)
2. Obvious-artifact detection (clipping, dropouts, garbled words)
3. Same-voice comparison (is this the same singer as the reference?)

Images unlocked screenshot debugging. PDFs unlocked document work. For everyone building in music, audio, and video, ears are that same unlock — and right now that's the one domain where Claude Code has to be told what things sound like.