Sanskrit ASR

Whisper ships with a generic vocabulary. I replaced it with a custom 8,192-token tokenizer built on phoneme units, which also removed the intermediate English transliteration step. That swap is what took word error rate from 41.7% down to the 28–32% range.

Some recordings have a drone running underneath. A dual-stream feature extractor with 160 frequency channels (log-Mel plus harmonic separation) pulls the vowels out from under that interference. Training ran on a laptop RTX 5060 with 8 GB of VRAM.

I added phoneme-level validation at inference. Phonologically invalid outputs dropped from 3.2% to 0.1%. The finished model is compressed to 8 MB as INT8 ONNX and transcribes a 3-second chunk in a median of 38 ms on a CPU.