Measured, not promised

Mixed language, real rooms, and the numbers behind both.

Here is what the tables below say, before you read them: a routed mixture of models beats a single general-purpose model exactly where speech gets hard — other languages, mixed languages, a phone on a meeting-room table. Every number we measured ourselves, on public datasets anyone can download, and every one of them comes with the sample size next to it. Nothing on this page is anybody's press release.

Every figure is corpus-level word error rate — total word errors over total reference words — and it is re-measured whenever the routing changes. Lower is better.

Measured on real audio

Public datasets, named — download them yourself.

Google FLEURS, Multilingual LibriSpeech and the AMI Meeting Corpus. All three are published under Creative Commons licences, so we can tell you exactly which clips we used and you can fetch the same ones. These are the numbers we are willing to be held to.

Sinsonte — a mixture of models

Sinsonte does not run one transcription model. Every recording goes through a mixture — some of the models ours, some licensed — and the router picks between them by the language being spoken and the room it was spoken in. A mixture of languages is what forces a mixture of models: one model per recording commits to one language and writes the rest of the conversation as though it had heard that language too. And no single model wins everything either — the one that leads on read Spanish is not the one that survives a phone left on a meeting-room table. Choosing correctly, recording by recording, is the product. So what this column reports is not a model's score. It is what the routed mixture delivers under that condition.

Whisper large-v3 — the open-source standard

The only model we name, and on purpose. It is what a large share of transcription tools are actually running underneath, it is free to download, and you can reproduce this column yourself on the same clips in an afternoon. A baseline you can check is worth more than five you have to take on trust — so it is the one we ask to be held against.

The recipe stays ours

Which models are in the mixture, and how the router chooses between them, we do not disclose — la receta es nuestra. Everything else stays reproducible: the datasets, the splits, the sampling seed, the clip identifiers, the text normalisation and the metric are all published, and the baseline column is a free download. Anyone can re-run it and confirm our number was measured the same way, on the same audio, on the same day.

Word error rate, %, corpus level. Lower is better. The amber chip marks the Sinsonte figure.
Condition clips min Sinsonte Whisper large-v3
FLEURS es-419 — Latin American Spanish 22445.1 3.33.4
FLEURS pt-BR — Brazilian Portuguese beta 21045.2 5.15.7
FLEURS en-US — English 27745.1 4.85.7
MLS Spanish — audiobook narration 17745.1 2.84.1
MLS Portuguese — audiobook, European pt beta 17645.2 3.86.8
AMI-SDM — English, real meeting room, single far mic 19418.0 12.823.9

Reading those in plain words: at 3.3% on Latin American Spanish, about 97 words in every 100 come back exactly as they were spoken. On audiobook Portuguese, 3.8% against the open baseline's 6.8% means close to half the errors. And the further the audio gets from a studio, the wider that margin opens — which is the whole reason we route at all.

12.8%

The room is where the free tools fall over.

AMI-SDM is a real meeting recorded by a single microphone sitting on a table, several people, no headsets — the condition Sinsonte is built for and the one benchmarks usually skip. On 194 clips (18 minutes) Sinsonte scores 12.8% word error rate against 23.9% for Whisper large-v3: the same audio, nearly half the errors. It is the widest gap on this page, and it is the one that matters, because a meeting room is not an edge case — it is Tuesday.

What 12.8% feels like: roughly one word in eight worth a second look, in the hardest condition there is — a phone lying across the room from whoever is talking. The open-source standard misses about one in four on the same audio. And you never have to hunt for which word: every line carries its ▶ mm:ss, so checking one costs a single tap.

The gaps are not uniform, and that is the argument. Clean, well-articulated audio is a solved problem — on Spanish read speech anyone competent lands in the same place, us included. The mixture earns its keep where a single general-purpose model stops coping: other languages, mixed languages, and rooms. Three points on audiobook Portuguese; eleven in a real meeting room. That is where your recordings actually live.

Speaker separation — measured

Telling who is speaking, measured the same way as what they said.

Getting the words right means nothing if the wrong person gets credit for them. Every recording goes through a voice-separation stage before transcription starts, and we hold it to the same standard as the tables above: public datasets, known-correct reference labels, the same scoring harness every run.

DER — diarization error rate — is the share of a meeting's total time assigned to the wrong speaker, missed as speech, or invented as speech that was not there. It says nothing about what was said, only whose turn it was. Lower is better.

Sinsonte's voice-separation stage

Every recording goes through a voice-separation step before transcription, chosen the same way the transcription mixture is: by what measures best on the kind of room the audio came from. We do not name which engine that is, for the same reason we do not name the transcription mixture — the routing is the product.

The leading open-source diarizer

The most widely deployed open-source speaker-separation tool — free to download and run yourself on the same public datasets. Like Whisper on the tables above, it is the one baseline on this page we name a category for rather than a product, because anyone can reproduce it.

Diarization error rate, %, collar 0.25s (the NIST/DIHARD convention). Lower is better. The amber chip marks the Sinsonte figure.
Condition min Sinsonte Open-source diarizer
AMI Meeting Corpus — real meeting, English (sample 1) 5.0 23.131.1
AMI Meeting Corpus — real meeting, English (sample 2) 5.0 11.016.7
CallHome — Spanish phone call 2.5 15.130.5
VoxConverse — English, online audio (sample 1) 1.4 14.411.4
VoxConverse — English, online audio (sample 2) 0.6 5.110.9
Mean (n=5) 14.5 13.720.1

In plain words: at a mean of 13.7% DER, roughly seven minutes in eight of a meeting land with the correct speaker; the open-source baseline gets that right for about four minutes in five. The gap is widest on the one Spanish sample — a real phone call, not a synthetic one — where Sinsonte's stage scores 15.1% against 30.5%, essentially half the confusion.

15.1%

The one Spanish sample says what the tables above say too.

callhome-spa-0 is a real telephone call in Spanish, not a studio dialogue. Sinsonte's voice-separation stage scores 15.1% DER against 30.5% for the leading open-source diarizer on the same audio — half the confusion, on the language this whole page exists to prove out.

n=1. One real Spanish call is evidence, not a finished study — the same caveat this page makes everywhere else. Two more CallHome-Spanish clips exist and were excluded outright: both failed a basic clipping check before any engine touched them, corrupted at the source, not a result either side earned.

This does not win everywhere, and we are showing the row where it does not: on one of the five samples the open-source baseline scores lower (11.4% against 14.4%, both English, both VoxConverse). Five samples, one language beyond English, is real evidence and not a finished study — more Spanish diarization samples are the next thing on this list, same as the far-field Spanish claim above.

Measured in the studio · validation under way

The mixtures — where the specialty shows.

Spanglish, portunhol, pt-en: sentences that change language halfway through and do not warn anyone first. This is what Sinsonte is built for, and it is where the distance between a routed mixture and a single model is widest on the whole page.

Measured on study dialogues we generated ourselves; validation with real recordings is under way.

Word error rate, %, on our own reference set — study dialogues, clean variant. Small samples: n is per condition. Same two columns as above.
Condition clips Sinsonte Whisper large-v3
Spanglish, switching mid-sentence 3 1.13.1
Spanish/English, switching between sentences 4 7.349.6
Portuguese + English, mixed beta 1 3.2–
Portunhol — Portuguese + Spanish beta 1 9.8–

The one to look at is the second row. A conversation that switches language between sentences is where a single model commits to one language and writes the rest as though it had heard that language too: it gets roughly one word in two wrong. The routed mixture gets about one in fourteen. On portunhol — two languages close enough to be confused with each other — it is one word in twelve against one in five.

FFER — False-Friend Error Rate

How often an engine swaps a word for its false friend in the neighbouring language instead of writing what was actually said. We built the metric because nobody else measures it.

Spanish and Portuguese sit close enough together that transcription engines quietly “fix” one into the other. Someone speaking portunhol says the status is em rojo — a Spanish word inside a Portuguese sentence — and the engine writes em vermelho, because that is what a well-behaved Portuguese sentence ought to say. Fluent. Plausible. Not what was said.

That costs one word of error rate and changes a fact. Sinsonte writes decisions and action items straight off the transcript, so we built a metric for it: the false-friend error rate. Our reference clips carry 25 marked traps; for each one a model either kept what was spoken, swapped it for the other language's word — that swap is the FFER — or garbled it, which we deliberately keep out of the numerator so ordinary noise doesn't inflate the score.

20%

Even the best single model rewrites one false friend in five.

That is the best one we have measured, on clean portunhol, with the trap marked in advance — four words in five survive, and the fifth quietly becomes the other language's word. The worst engine we tested rewrites seven in ten while barely garbling a syllable: fluent, confident, and not what was said.

Which is exactly why Sinsonte does not hand portunhol to a single model and take the answer at face value. Where a false friend is at stake we flag the word instead of quietly picking a side, and the person reading decides — with the ▶ right there to hear what was actually said.

Methodology, datasets and limits

Scoring

Corpus-level word error rate — total word errors divided by total reference words for the condition, not an average of per-clip scores. Text is normalised the way the Open ASR Leaderboard does it, with one deliberate change: accents and ñ are preserved rather than stripped, because a transcript that loses them is not a Spanish transcript. Every hypothesis is stored, so scores can be recomputed later without re-running any audio.

Sampling and scale

Seed 42 throughout, roughly 45 minutes of audio per real-audio condition and 18 minutes for the meeting-room one: 1 258 clips, 4.1 hours. Each clip was put through six engines, which is where the 24.6 hours of evaluation on this page comes from — 4.1 × 6. FLEURS test splits are smaller than the shuffle buffer, so the sample is an exact uniform shuffle of the whole split. The identifier of every clip is written to a manifest, so the same sample can be re-fetched and re-scored by anyone. The round published here ran on 8–9 August 2026 and is re-run whenever the routing changes.

Known gaps in the sampling

The meeting-room index covers about 92.5% of the available split — seven pages of the catalogue kept returning rate-limit errors while it was being built, and we walked past them rather than retrying forever. The sample is still seeded and reproducible; it is just drawn from that 92.5%, and we would rather write this sentence than let you assume otherwise.

Whisper large-v3, reconfirmed

A follow-up round on 15 August 2026 re-tested Whisper large-v3 — alongside three other transcription models not shown on this page — against smaller samples from the same clip families used above: 3 to 5 clips per condition, macro-averaged rather than corpus-level. It confirmed Whisper large-v3 scoring worse on every condition it touched: 4.5% on Latin American Spanish and 3.6% on Spanish audiobook narration, both close to the 3.4% and 4.1% published above; 55.2% on Spanish/English code-switching between sentences, close to the 49.6% above; and 8.5% on mid-sentence Spanglish, well above the 3.1% above. Two results are worth stating plainly rather than averaging away: on clean English, Whisper large-v3 returned an empty transcript for a 4.3-second clip that is not silent — that alone accounts for most of a 34.6% raw mean; excluding just that one clip it drops to 18.2%, still worse than the 5.7% above. And on real meeting far-field audio, this smaller round's four clips scored 61.0%, well above the 23.9% in the larger round above — both are genuine measurements of the same open-source model on genuinely hard audio, not a contradiction to resolve. The same round also caught Whisper large-v3-turbo mistranslating a Spanish clip into English outright instead of transcribing it. None of this changes the Sinsonte figures published above — this was a Whisper-only confirmatory round, kept separate here because its sample was smaller than the rest of this page.

The datasets, and the licences that let us publish this

  • Google FLEURS

    Read speech in 102 languages; we use the es-419, pt-BR and en-US test splits.

    CC BY 4.0 · huggingface.co/datasets/google/fleurs

  • Multilingual LibriSpeech

    Audiobook narration; we use the Spanish and Portuguese test splits. The Portuguese one is European Portuguese, not Brazilian — which is part of why it scores differently.

    CC BY 4.0 · openslr.org/94

  • AMI Meeting Corpus

    Real multi-person meetings recorded in instrumented rooms. We use the single-distant-microphone test split — the closest public audio to a phone left on a table.

    CC BY 4.0 · groups.inf.ed.ac.uk/ami/corpus

  • VoxConverse

    Multi-speaker audio from online video, with speaker-turn ground truth. We use two clips from the test set.

    CC BY 4.0 · github.com/joonson/voxconverse

  • CallHome (Spanish)

    Real telephone conversations in Spanish, with speaker-turn ground truth. Research licence, not redistributable — we can tell you which call we used, not hand you the file.

    CC BY-NC-SA 4.0, research only · huggingface.co/datasets/talkbank/callhome

Limits, stated plainly

  • We do not compare Sinsonte against other products. Both columns are things we ran ourselves, on the same clips on the same day — a number scraped off a competitor's marketing page is not a measurement.
  • We do not disclose which models are in the mixture, or how the router picks between them. Deliberate: the routing is the product. What we publish instead is our own result next to a baseline anyone can download and re-run.
  • We do not claim to be the best at Spanish. Which approach wins changes with the condition, which is the whole reason we route rather than standardising on one model.
  • We make no far-field Spanish claim yet: AMI is English, and it is our only genuinely real room recording. Recording consented Spanish sessions in noisy rooms is the next thing on this list.
  • The mixed-language numbers are measured on study dialogues we generated, not on real-world recordings, and the samples are small — no public dataset of Spanish–English or Portuguese–Spanish conversational speech exists that we are licensed to publish results from. Three specifics worth knowing: the 1.1% on mid-sentence Spanglish rests on one clip (the other candidates land between 1.7% and 7.4%), the speech was generated by a system whose maker also sells transcription, and the “between sentences” row joins separate recordings end to end, which punishes any model that locks onto one language per clip. Validation with real recordings is under way; these are labelled as studio-measured everywhere they appear, including on our home page.
  • No number on this page comes from a user's recording. Not one, ever — not for benchmarking, not for training, not for a demo. Everything here is public data or audio we made for the purpose.
  • The Portuguese-mixed and portunhol rows above were updated on 15 August 2026 after a routing decision moved both conditions to a different engine in the mixture; the new figures are n=1 per condition, the standing caveat for this whole section. Whisper large-v3 was not tested on that round's clips, so we show no baseline number for those two rows rather than pair a new figure with an old one from a different round.
  • The voice-separation numbers in the new section above are diarization error rate, not word error rate, measured on their own five-clip set of real public recordings — a much smaller sample than the transcription tables use. Two further Spanish clips exist in that dataset and were excluded outright: both failed a basic clipping check before any engine touched them, corrupted at the source. The open-source baseline wins one of the five samples (VoxConverse, 11.4% against 14.4%), shown rather than dropped.

The harness, the scoring code and the clip-by-clip manifest are not public yet — the repository they live in also holds the product. They are available on request to anyone who wants to check a number: write to [email protected] and we'll send them.

← Back to Sinsonte