← All runs

Evaluation run

audio run (incl LibriSpeech)

6 July 2026, 11:18 · eval-engine · 54c20aa · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

51%
Light, ships on by default
72/140 cases passed · 249 ms typical
40%
Polish, experimental
48/120 cases passed · 267 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  35 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

tts-grammar-numbers-keep-01Syntheticneeds review
HeardThe budget is like fifty thousand 50,000 no wait seventy five thousand save 85,000 for Q3. 38% misheard
Was saidthe budget is like fifty thousand no wait seventy five thousand for q3
InstantThe budget is like 50,000 no wait save 85,000 for Q3.
LightThe budget is like 50,000 no wait save 85,000 for Q3. 389 ms
PolishThe budget is like $50,000, no wait not save $85,000 for Q3. 547 ms
Why light?the result is missing “75”; the microphone misheard 38% of the words
Why polish?the result is missing “75”; the result still contains “50,000” and “$50”, which this case forbids; the microphone misheard 38% of the words
tts-grammar-runon-01Syntheticclean
HeardSo I went to the meeting and then I talked to Sarah and then we decided to push the launch and also we need to tell the team. 0% misheard
Was saidso i went to the meeting and then i talked to sarah and then we decided to push the launch and also we need to tell the team
InstantSo I went to the meeting and then I talked to Sarah and then we decided to push the launch and also we need to tell the team.
LightSo I went to the meeting and then I talked to Sarah and then we decided to push the launch, and also we need to tell the team. 646 ms
PolishSo I went to the meeting and then I talked to Sarah. and then We decided to push the launch and also we need to tell the team. 659 ms
tts-grammar-tense-01Syntheticneeds review
HeardYesterday I go to the office and ain't I see the new design. 8% misheard
Was saidyesterday i go to the office and i see the new design
InstantYesterday I go to the office ain't I see the new design.
LightYesterday I go to the office ain't I see the new design. 336 msthe cleanup was rejected, so the original was kept
PolishYesterday I go went to the office ain't I and didn't see the new design. 351 ms
Why light?guard rejected
tts-num-chapterSyntheticneeds review
HeardLet's start with chapter one 1. 20% misheard
Was saidlet's start with chapter one
InstantLet's start with chapter 1.
LightLet's start with chapter 1. 197 ms
PolishLet's start with chapter 1. 192 ms
Why?the microphone misheard 20% of the words
tts-num-currencySyntheticneeds review
HeardThe plan costs twenty five dollars $25 a month. 38% misheard
Was saidthe plan costs twenty five dollars a month
InstantThe plan costs $25 a month.
LightThe plan costs $25 a month. 208 ms
PolishThe plan costs $25 a month. 205 ms
Why?the microphone misheard 38% of the words
tts-num-decimalSyntheticneeds review
HeardThe download is three point five gigabytes 3.5GB. 57% misheard
Was saidthe download is three point five gigabytes
InstantThe download is 3.5GB.
LightThe download is 3.5GB. 210 ms
PolishThe download is 3.5GB. 207 msthe cleanup was rejected, so the original was kept
Why light?the microphone misheard 57% of the words
Why polish?guard rejected; the microphone misheard 57% of the words
tts-num-decimal-pctSyntheticneeds review
HeardWe hit ninety nine point nine percent 99.9% uptime. 63% misheard
Was saidwe hit ninety nine point nine percent uptime
InstantWe hit 99.9% uptime.
LightWe hit 99.9% uptime. 264 ms
PolishWe hit 99.9% uptime. 269 ms
Why?the microphone misheard 63% of the words
tts-num-first-placeSyntheticclean
HeardShe came in first place at the meet. 0% misheard
Was saidshe came in first place at the meet
InstantShe came in first place at the meet.
LightShe came in first place at the meet. 211 ms
PolishShe came in first place at the meet. 208 ms
tts-num-halfSyntheticneeds review
HeardGive a me half a second to think sacred toothink. 57% misheard
Was saidgive me half a second to think
InstantGive a me half a sacred toothink.
LightGive a me half a sacred toothink. 194 msthe cleanup was rejected, so the original was kept
PolishGive a me half a sacred toothink. 224 msthe cleanup was rejected, so the original was kept
Why?the result is missing “half a second”; guard rejected; the microphone misheard 57% of the words
tts-num-largeSyntheticneeds review
HeardWe shipped one hundred and twenty 120 units. 57% misheard
Was saidwe shipped one hundred and twenty units
InstantWe shipped 120 units.
LightWe shipped 120 units. 192 ms
PolishWe shipped 120 units. 191 ms
Why?the microphone misheard 57% of the words