whisper-tiny-grpo-0029b82-cv22-tiny-sft

A Whisper model fine-tuned with GRPO (Group Relative Policy Optimization) using a blended error-rate reward. Trained with whisper-rl.

Best validation (overall across 77 languages): WER 0.602, CER 0.253

Training data

Fine-tuned on cv22_index — all 77 Common Voice locales, streamed and decoded on the fly from the full training split. Each clip's language is pinned from its Common Voice locale during training.

Performance per language

Validation WER and CER at the best checkpoint, per Common Voice locale (also in the Evaluation Results metadata above):

Language	WER	CER
af	0.911	0.302
am	0.865	0.548
ar	0.647	0.253
as	0.344	0.193
az	0.744	0.257
ba	0.812	0.278
be	0.556	0.141
bg	0.600	0.165
bn	0.375	0.223
br	0.707	0.349
ca	0.820	0.368
cs	0.625	0.130
cy	0.450	0.142
da	0.346	0.132
de	0.224	0.089
el	0.409	0.130
en	0.512	0.267
es	0.652	0.242
et	0.963	0.233
eu	0.583	0.142
fa	0.900	0.361
fi	0.500	0.128
fr	0.400	0.167
gl	0.281	0.048
ha	0.516	0.108
he	1.000	0.670
hi	0.302	0.171
ht	0.952	0.411
hu	1.056	0.720
hy-AM	0.880	0.277
id	0.769	0.338
is	0.968	0.543
it	0.421	0.107
ja	1.000	0.438
ka	0.762	0.383
kk	0.650	0.224
ko	0.652	0.524
lo	0.788	0.704
lt	0.783	0.303
lv	0.667	0.143
mk	0.424	0.151
ml	0.349	0.245
mn	0.735	0.262
mr	0.171	0.104
mt	0.562	0.181
ne-NP	0.360	0.188
nl	0.667	0.198
nn-NO	0.889	0.483
oc	0.720	0.213
pa-IN	0.286	0.143
pl	0.462	0.177
ps	0.917	0.520
pt	0.222	0.113
ro	0.167	0.037
ru	0.636	0.229
sk	0.522	0.144
sl	0.667	0.103
sq	0.926	0.357
sr	0.444	0.258
sv-SE	0.586	0.198
sw	0.567	0.170
ta	0.255	0.138
te	0.679	0.373
tg	0.800	0.200
th	0.621	0.218
tk	1.000	0.503
tt	0.625	0.220
uk	0.833	0.186
ur	0.600	0.234
uz	0.727	0.192
vi	0.391	0.228
yi	0.905	0.520
yo	0.903	0.360
yue	1.000	0.235
zh-CN	1.000	0.229
zh-HK	1.000	0.214
zh-TW	1.000	0.250

How it was trained

Instead of cross-entropy against a single reference, for each audio clip the policy samples a group of num_generations transcriptions, scores each by a negated blend of word error rate, character error rate, and length / repetition penalties, and is nudged toward the better candidates with a clipped policy-gradient objective regularized by a per-token KL penalty to the frozen base model. Advantages are the group-relative, standardized rewards (A = (r - mean) / (std + eps)), so no value network is needed. The clip's language is pinned from its Common Voice locale, and the policy's own greedy transcriptions are scored as validation WER and CER.

Hyperparameters

Field	Value
Base model	openai/whisper-tiny
Dataset	/data/cv22_index
Learning rate	1e-05
Sampling temperature	0.7
Group size (generations/clip)	8
Reward weights	{'cer': 1, 'wer': 1, 'length': 0.5, 'repetition': 0.5}
KL penalty (β)	0.04
Batch size (clips/step)	16
Max optimizer steps	1000000
Warmup steps	20

Training curves

Pulled from the Weights & Biases run (static snapshot):

Usage

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="wrice/whisper-tiny-grpo-0029b82-cv22-tiny-sft")
print(asr("audio.wav")["text"])

Limitations

A proof-of-concept GRPO recipe, not a tuned production system. WER and CER are reported on a held-out Common Voice validation slice after text normalization; real-world performance varies by domain, accent, language, and audio quality.