🇦🇲 ArmBench-ASR: Benchmarking Speech-to-Text Models on Armenian

v1.0

JavaScript is turned off. The default leaderboard is available below, but turn on JavaScript to use filters, sorting, charts, and the other interactive views.

WER across all datasets

Rank is based on Overall WER. Scores are percentages; lower is better.

34 models View versions

Dataset

Metric across all datasets

Rank	Model	Availability	Combined	Common Voice 26	FLEURS	Poems	Movies	News
#1	google/gemini-2.5-pro	Closed	13.80	8.92	7.91	20.08	39.19	11.26
#2	hispeech/model-23012026	Closed	16.56	10.94	13.13	22.69	41.35	14.36
#3	google/gemini-3-flash-preview	Closed	16.64	10.77	8.99	24.36	51.38	11.06
#4	google/gemini-2.5-flash	Closed	17.35	12.23	10.10	24.84	45.87	13.47
#5	hispeech/model-01052025	Closed	17.63	10.05	14.41	22.96	49.49	17.75
#6	talk2edit	Closed	17.66	12.30	10.07	24.97	46.11	15.29
#7	elevenlabs/scribe_v2	Closed	17.74	12.10	9.45	26.07	49.82	13.24
#8	wavam/default	Closed	19.13	11.42	11.91	30.41	53.00	15.67
#9	google/gemini-3.6-flash	Closed	19.42	16.04	11.24	25.83	46.64	12.93
#10	google/gemini-3.5-transcribe-preview	Closed	19.59	15.68	10.95	28.04	46.46	13.26
#11	nvidia/stt_hy_fastconformer_hybrid_large_pc	Open	19.86	7.70	15.00	31.28	60.78	23.55
#12	wavam/streaming	Closed	20.18	12.29	14.00	31.01	53.75	16.84
#13	tbb-asr	Closed	20.19	8.15	15.52	31.24	61.67	23.33
#14	google/gemini-3.1-flash-lite	Closed	20.27	16.33	10.51	26.08	54.86	12.92
#15	openai/gpt-transcribe	Closed	22.40	17.28	15.37	30.17	51.95	17.24
#16	NCCAIT/model-2024	Closed	25.17	16.16	19.49	33.63	61.76	25.83
#17	google/gemini-3.5-flash-lite	Closed	26.06	23.69	14.32	33.72	56.01	16.74
#18	deepgram/nova-3	Closed	28.46	7.31	33.56	48.35	75.81	36.66
#19	facebook/seamless-m4t-v2-large	Open	28.94	15.15	14.44	43.93	80.16	37.43
#20	chillarmo/whisper-large-v3-turbo-armenian	Open	33.00	12.94	24.69	55.09	82.01	48.80
#21	arampacha/whisper-large-hy-2	Open	33.17	23.16	17.37	44.48	77.82	39.48
#22	modulate-velma-2-stt-batch-multilingual	Closed	33.71	27.35	23.95	37.52	72.73	33.77
#23	facebook/omniASR-LLM-3B	Open	37.76	27.45	26.68	49.84	87.43	35.18
#24	facebook/omniASR-LLM-1B	Open	38.10	28.09	27.18	49.06	87.64	35.78
#25	facebook/omniASR-CTC-3B	Open	39.85	30.55	28.75	51.76	86.64	36.14
#26	facebook/omniASR-CTC-1B	Open	40.50	33.78	31.01	47.05	79.37	38.35
#27	facebook/omniASR-LLM-300M	Open	41.10	30.75	29.40	53.61	91.83	38.06
#28	cast/whisper-large-v3-mwa-hy	Open	41.41	29.59	30.69	52.89	75.04	55.75
#29	facebook/mms-1b-all	Open	47.50	43.09	34.52	53.04	83.38	44.79
#30	facebook/omniASR-CTC-300M	Open	50.61	44.92	40.25	57.79	86.53	47.25
#31	google/gemma-4-e4b-it	Open	50.65	48.48	37.11	57.62	79.72	44.14
#32	google/gemma-4-e2b-it	Open	53.46	50.44	42.76	59.88	80.19	48.92
#33	modulate-velma-2-stt-batch-multilingual-vfast	Closed	64.23	62.42	52.80	63.00	107.30	53.91
#34	openai/whisper-large-v3	Open	65.12	61.67	51.84	71.10	106.17	55.25

Model comparison

Compare model error rates for one dataset and metric. Lower is better.

DatasetMetricAvailability

Cite ArmBench-ASR

If you use ArmBench-ASR in your research, please cite it as follows:

@misc{armbench-asr,
  title={ArmBench-ASR: Benchmarking Speech-to-Text Models on Armenian},
  author={Metric-AI-Lab},
  year={2026},
  howpublished={\url{https://metric-ai-armbench-asr.static.hf.space}},
  note={Benchmark for Evaluating Speech-to-Text Models on Armenian}
}

Five datasets, one Armenian benchmark

10,113 clips and 20.7 hours of speech spanning read, literary, cinematic, and conversational Armenian. Every model is evaluated on the same fixed test material.

01Public

Common Voice 26 ↗

Mozilla Common Voice Scripted Speech 26.0 for Armenian, using the full hy-AM test split released in June 2026.

Samples: 6,892
Audio: 10.8 h
Style: Read speech

02Public

FLEURS ↗

Google FLEURS Armenian (hy_am) test split. Raw transcriptions retain punctuation and capitalization for strict scoring, with some manual corrections applied.

Samples: 932
Audio: 3.0 h
Style: Read speech

03Private

Poems

Read Armenian poetry with background music, literary vocabulary, expressive phrasing, and text structures underrepresented in general ASR sets.

Samples: 969
Audio: 3.3 h
Style: Literary

04Private

Movies

Armenian movie dialogue with conversational delivery, produced-media acoustics, and occasional background noise.

Samples: 820
Audio: 1.7 h
Style: Dialogue

05Private

News

A compact Armenian news dataset recited by a speaker, designed to evaluate clear narrated speech beyond standard public test sets.

Samples: 500
Audio: 1.9 h
Style: Narrated news

Cite ArmBench-ASR

If you use ArmBench-ASR in your research, please cite it as follows:

@misc{armbench-asr,
  title={ArmBench-ASR: Benchmarking Speech-to-Text Models on Armenian},
  author={Metric-AI-Lab},
  year={2026},
  howpublished={\url{https://metric-ai-armbench-asr.static.hf.space}},
  note={Benchmark for Evaluating Speech-to-Text Models on Armenian}
}

Reading the benchmark

Armenian ASR Benchmark compares automatic speech recognition systems on a varied Armenian evaluation suite. Every score is an error rate, so lower values are better.

The benchmark aims to bring clarity to the Armenian ASR landscape, make model comparisons easier, and help model creators identify strengths, gaps, and opportunities for meaningful improvement.

WERWord error rate with number normalization.

CERCharacter error rate with number normalization.

WER-normNormalized word error rate with number normalization.

CER-normNormalized character error rate with number normalization.

Availability

Open means model weights are publicly available. Closed means access is provided through a hosted product or API.

API settings

Hosted systems use fixed model identifiers and transcription-only prompts. Evaluated Gemini runs use temperature 0, a 1,000-token output cap, and audio limited to 40 seconds. Thinking is disabled or set to the lowest applicable level: budget 0 for Gemini 2.5 Flash, budget 128 for Gemini 2.5 Pro, and MINIMAL for Gemini 3.1 Flash Lite, 3.5 Flash Lite, and 3.6 Flash.

Text cleaning

References and predictions are processed with the same deterministic pipeline. Number normalization is applied before scoring; normalized metrics additionally reduce orthographic and formatting differences while preserving Armenian, English, Russian letters, digits, and recognized clock times. Both WER / CER and WER-norm / CER-norm are reported.

About Us

Armenian ASR Benchmark is a Metric AI Lab project. The applied AI research lab builds production systems across custom language and vision-language models, multilingual voice agents, Physical AI, retrieval, and analytics—while publishing frontier research and open benchmarks.

Visit Metric AI Lab ↗

Acknowledgments

We gratefully acknowledge HiSpeech, Talk2Edit, TBB ASR, NCCAIT, and Wav.am for making their systems available for evaluation at no cost, and Infocom for providing Armenian news data used in this benchmark.