The Same AI Voice Ranks 3rd in English and 19th in Turkish

Summary

Most AI voice companies show you a demo page. Very few tell you how they tested, how confident they are, or how the voice holds up in a language other than English. We needed to pick a voice for a phone agent that talks to real patients, could not find that information anywhere, so we measured it ourselves.

Round 2026.2 covers 44 English voices and 28 Turkish voices from 16 companies. Two voices read the same sentence, a listener picks one, and nobody sees a brand name until voting closes. Every score ships with an error bar, and the full results file is a public download.

What Changed Since The Pilot

Our July pilot was honest about being too small: 2,675 votes across 40 English voices meant 21 of the 22 ranked entries overlapped the leader, and we said plainly that we could not rank English. Round two is 4.6 times bigger, and the picture is different in three specific ways.

First, English separates now — though not at the top. The error bars roughly halved, and 20 of the 44 voices are now measurably behind the leading pack instead of statistically tangled with it. The other 24 still overlap each other.

Second, the cross-language finding survived and got stronger. It was the one pilot result we said we believed, and more data has not softened it.

Third, and least comfortably, one pilot finding did not survive. We reported that the four listening questions measured genuinely different things, pointing at a near-zero correlation between “sounds human” and overall preference in Turkish. With four times the data that correlation is 0.89. It was noise. We were measuring our own sample size.

Splitting the questions is still worth doing — it is how you find out they agree, and it keeps the option open for a future round where they might not. But we no longer claim they measure different things, and anyone who quoted our pilot on that should stop.

How It Works

The point of the setup is to test voices, not brand loyalty or listener patience. Nothing about the method changed between rounds, which is what makes the two comparable.

Both voices read the same sentence

Every comparison uses identical text, from a fixed bank of 24 sentences — 12 English, 12 Turkish — covering plain statements, questions, lists, names, numbers and dates, abbreviations, long winding sentences, and emotional lines.

Nobody sees the brand names

The listener's browser gets an audio file and its length, and nothing else. File names are random IDs, so no company or model name appears in the page, the network traffic, or anywhere a curious listener could dig. Buttons say “First voice” and “Second voice”. The map from clip to company lives in a separate file that is only opened after voting closes.

One question at a time, and you have to listen

A listener gets 25 comparisons in a sitting, all asking the same one of four questions. The server rejects any vote where the listener did not play at least 60% of both clips, so a modified browser cannot fake it. The two players cannot run at once. Replays are unlimited. There is no skip — you pick a voice, say they sound the same, or report an audio problem. Nobody sees the same pair twice, and nobody votes more than once on a given question in a given language.

Hidden checks

Three of the 25 comparisons are traps: the same clip played on both sides, at positions 5, 13 and 21. They look completely normal, both sides getting their own download link for the identical file. The honest answers are “about the same” or “audio problem”; picking a winner counts as a miss. Traps are paid like any other comparison, which is what keeps them hidden, and they never count toward anyone's score.

How the scores work

We use a standard head-to-head method called Bradley–Terry, which turns a pile of pairwise results into a strength number per voice. A tie counts as half a win. Scores are centred so the average voice in each language sits at 1000. Error bars come from resampling listeners rather than individual clicks, because one person voting 25 times is not 25 independent opinions. A voice needs 20 votes to earn a rank; in this round every voice cleared that easily.

What We Collected

Voting ran from 20 to 25 July 2026 and closed at 556 completed sittings.

English Results

This is what the pilot could not give us, with one caveat. The field is 44 voices, the bottom 20 are now measurably behind, and the top 24 are still a crowd.

Google Gemini's gemini-3.1-flash-tts-preview leads at 1138, but its range overlaps every voice down to rank 24. The honest reading is that the English field splits in two: a broad leading pack of 24 voices we cannot order internally, and 20 that are clearly behind it. That is a real gain over the pilot, where only one voice was separable from the leader — but it is separation of the bottom half, not a winner at the top.

Turkish Results

Turkish has a smaller field and a much clearer top. 19 of 27 challengers are clearly behind the leader.

The Language Gap

Sixteen of the voices were put in front of listeners in both languages — identical company, model and voice ID. That is the cleanest possible test of whether quality travels, because nothing changes except the language being read.

It does not travel.

One important exception, stated plainly: Google Gemini's gemini-3.1-flash-tts-preview Puck is ranked 1st in both languages. A voice can be good in two languages at once. The point is that most are not, and you cannot tell which from an English score.

Pooling all four questions per company gives each one enough comparisons for the cross-language picture to resolve sharply. This is the most useful thing in the report.

The two highlighted companies move in opposite directions, and neither move is small.

Cartesia repeats its pilot result almost exactly, which is the best evidence we have that the effect is real rather than an artefact of one round. Speechify is the new and starker case: first in English, eighth of ten in Turkish.

Six of the sixteen companies fielded nothing in Turkish at all — Smallest, Gradium, StepFun, Hume AI, Rime and Inworld. Turkish speakers choose from 28 voices against 44, and four of the top ten Turkish slots belong to Google.

Voice Versus Model

People compare companies and models, but what you ship is one voice. Within a single model the spread between voices is still large enough to dominate the choice of model.

Fish Audio's English pairing is the extreme case: the same model, the same language, 333 points and ten ranks apart, with one of the two finishing last of all 44 voices. Test the voice you are going to ship, not the company that makes it.

Do Listeners Pay Attention

A listening test is only as good as the people doing the listening. Unlike the pilot, this round had the hidden traps running the whole time, so the answer is measured rather than assumed.

So roughly one time in ten, someone hears the exact same clip twice and still picks a favourite. That is the cost of asking humans to compare things, and it is why we put the traps in. Any listening test without checks like these is carrying about the same noise and does not know it.

Who We Excluded, And Why

Being transparent about this matters more than the result looking clean, so here is exactly what happened before we froze the numbers.

Two contributors tripped the automatic disqualification rule, which needs at least 6 controls seen, at least 3 misses, and a miss rate of 50% or higher. Their votes are out.

A further 51 sat on a softer hold, triggered by two misses. We looked at each one against the 9.3% population miss rate:

The 20 whose records are statistically consistent with honest listening were readmitted, returning 1,103 votes to the ranking. The 31 whose records are not — 23 of whom missed every single control they were shown — stay excluded, holding back 555 votes.

That line is a judgement call and we are showing our work rather than hiding it. Note that the automatic rule alone would have cleared all 31, purely because they had not yet been shown 6 controls each. A rule that needs six trials cannot catch someone who fails their first two.

What This Cannot Tell You

We would rather list these than have a reader find them.

Check It Yourself

The results file is public and signed. It is the same file that draws the leaderboard, so there is no private version.

The file carries its own checksum and signature, when it was generated, and a plain description of the method. Each leaderboard reports how many votes it rests on and how many people were behind them.

Working with us

We run these blind English and Turkish tests on unreleased models and tell you where the pronunciation and rhythm problems are before your users find them. If you build AI voices and want to be in the next round, or want a private evaluation, get in touch through patientdesk.ai. If you think we got something wrong, tell us — we publish corrections next to the original rather than quietly editing it.