Build Nepali speech-to-text with Whisper

Published 12 October 2026 · prices on this page are live

Whisper already understands some Nepali, but not well enough for most products. In our own test, Whisper large-v3 got about one character in four wrong on the FLEURS Nepali test set (a character error rate of 25.6%). After fine-tuning on 10,000 clips of public Nepali speech, about ten hours, the error rate fell to 13.9%: almost half the mistakes gone, trained on a single 16 GB GPU.

Nepali speech data you can use

Check every licence before you train a model you will sell. CC BY asks only for attribution; CC BY-SA also asks you to share changed versions of the dataset itself under the same licence; NC means non-commercial use only.

Some Nepali sets, such as OpenSLR 143, are for non-commercial use only. Keep a test set the model never trains on: FLEURS is a good one, because your results can be compared with published ones.

Which GPU to rent

LoRA fine-tuning of Whisper large-v3 fits on a 16 GB card with a small batch and gradient accumulation; our 13.9% run used a 16 GB RTX A4000. A 24 GB card such as the RTX 3090 or RTX 4090 takes bigger batches and finishes sooner, and an A100 is faster still. Today's lowest prices:

Lowest price per hour today, billed by the second.

Step by step

1. Rent the machine. Pick a 16–24 GB GPU with the PyTorch + JupyterLab software and at least 100 GB of disk: OpenSLR 54 alone is about 9 GB of zip files.
2. Install the libraries in a JupyterLab terminal:

   pip install transformers datasets peft accelerate evaluate jiwer soundfile librosa

3. Download the data. OpenSLR 54 comes as 16 zip files, asr_nepali_0.zip to asr_nepali_f.zip, plus utt_spk_text.tsv, which lists each clip's id, speaker and transcript. Each clip is at asr_nepali/data/ followed by the first two characters of its id, then the id with .flac on the end.
4. Prepare the model with LoRA, so you train a few million weights instead of all 1.5 billion:

   from transformers import WhisperForConditionalGeneration, WhisperProcessor
   from peft import LoraConfig, get_peft_model
   
   processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3", language="nepali", task="transcribe")
   model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3", torch_dtype="auto")
   model = get_peft_model(model, LoraConfig(r=32, lora_alpha=64, lora_dropout=0.05, target_modules=["q_proj", "v_proj"]))
   model.print_trainable_parameters()

5. Train with Hugging Face's Seq2SeqTrainer, as in their guide to fine-tuning Whisper for multilingual speech recognition. A learning rate of 1e-4 worked for us; if the loss jumps around, halve it. Save a checkpoint every few hundred steps.
6. Pick the best checkpoint by character error rate on your held-out test set, not by training loss.
7. Serve it fast. Merge the LoRA weights into the model and convert it with CTranslate2 for faster-whisper, which runs up to four times faster than the original code, and faster still with 8-bit weights.

Phone calls need phone-quality training audio

Phone recordings are narrow-band (8 kHz) and compressed, and a model trained only on clean audio does much worse on them. No public Nepali telephone corpus exists, so simulate one: downsample a share of your clips to 8 kHz, pass them through G.711 μ-law encoding, add a little line noise, and resample them back. In our test, degrading about two thirds of the training clips this way cut the error rate on phone-quality audio from 31.0% to 18.7%, while clean audio still improved to 13.9%.

Quality beats quantity. When we added thousands of noisier, machine-labelled clips, the model got worse, not better: a few thousand clean, correctly transcribed clips beat twice as many doubtful ones.

What it costs

Training time depends on how many clips and steps you run. As a guide, a four-hour session costs, on today's cheapest hosts:

Using the finished model is cheap too: Whisper large-v3 transcribes about 100 hours of audio in roughly three hours on an RTX 4090. See speech-to-text on a rented GPU.

Questions

More guides