← All runs

Evaluation run

faithfulness scoring cleanup audio

6 July 2026, 13:38 · eval-engine · 232478e · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

55%
Light, ships on by default
77/140 cases passed · 255 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  19 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

tts-disf-restartSyntheticclean
HeardI think we should UM, I think we should just ship it. 0% misheard
Was saidi think we should um i think we should just ship it
InstantI think we should UM, I think we should just ship it.
LightI think we should UM, I think we should just ship it. 215 ms
PolishI think we should UM, I think we should just ship it. 219 ms
tts-disf-scratchSyntheticclean
HeardThe deadline is Friday actually no let's say next Monday. 0% misheard
Was saidthe deadline is friday actually no let's say next monday
InstantThe deadline is Friday actually no let's say next Monday.
LightThe deadline is Friday actually no let's say next Monday. 182 ms
PolishThe deadline is Friday actually no let's say next Monday. 186 ms
tts-disf-selfcorrect-nameSyntheticneeds review
HeardSend it to John uh i mean Uhaimi Jane. 38% misheard
Was saidsend it to john uh i mean jane
InstantSend it to John Uhaimi Jane.
LightSend it to John Uhaimi Jane. 164 ms
PolishSend it to John Uhaimi Jane. 183 ms
Why?the microphone misheard 38% of the words
tts-disf-selfcorrect-timeSyntheticneeds review
HeardLet's meet at three 3 no wait four 4 o'clock. 25% misheard
Was saidlet's meet at three no wait four o'clock
InstantLet's meet at 3 no wait 4 o'clock.
LightLet's meet at 3 no wait 4 o'clock. 227 ms
PolishLet's meet at 3 no wait 4 o'clock. 298 ms
Why?the microphone misheard 25% of the words
tts-edge-emailSyntheticneeds review
HeardEmail me at john dot smith john.smith at gmail dot com gmail.com. 20% misheard
Was saidemail me at john dot smith at gmail dot com
InstantEmail me@john.smith at gmail.com.
LightEmail me@john.smith me at gmail.com. john.smith@gmail.com. 242 ms
PolishEmail me@john.smith me at gmail.com. john.smith@gmail.com. 263 ms
Why?the microphone misheard 20% of the words
tts-edge-gibberishSyntheticneeds review
Heardasdf qwerty ESDF QETI test test one two three 123. 71% misheard
Was saidasdf qwerty test test one two three
InstantESDF QETI test test 123.
LightESDF QETI test test 123. 290 ms
PolishESDF QETI test: test 123. 325 ms
Why?the microphone misheard 71% of the words
tts-edge-mixedSyntheticneeds review
Heardhey um Heyyum so the the lyzr blizzard demo is at four thirty 8430, no wait five 5 o'clock and accuracy is like ninety percent 90% now. 45% misheard
Was saidhey um so the the lyzr demo is at four thirty no wait five o'clock and accuracy is like ninety percent now
InstantHeyyum so the blizzard demo is 8430, no wait 5 o'clock and accuracy is like 90% now.
LightHeyyum so the blizzard demo is 8430, no wait 5 o'clock and accuracy is like 90% now. 444 msthe cleanup was rejected, so the original was kept
PolishHeyyum Hey, so the blizzard demo is 8430, no wait, it's 5 o'clock, and accuracy is like 90%. now. 677 ms
Why light?the result is missing “Lyzr”; the result still contains “um” and “no wait”, which this case forbids; the microphone misheard 45% of the words
Why polish?the result is missing “Lyzr”; the result still contains “no wait”, which this case forbids; the microphone misheard 45% of the words
tts-edge-shortSyntheticneeds review
HeardYeah sounds sound good. 33% misheard
Was saidyeah sounds good
InstantYeah sound good.
LightYeah, sound good. 166 ms
PolishYeah, sound sounds good. 184 ms
Why light?the result is missing “ounds good”; the microphone misheard 33% of the words
Why polish?the microphone misheard 33% of the words
tts-edge-singleSyntheticclean
HeardPerfect. 0% misheard
Was saidperfect
InstantPerfect.
LightPerfect. 118 ms
PolishPerfect. 133 ms
tts-faith-already-cleanSyntheticclean
HeardThe quarterly report is due on Friday. 0% misheard
Was saidThe quarterly report is due on Friday.
InstantThe quarterly report is due on Friday.
LightThe quarterly report is due on Friday. 202 ms
PolishThe quarterly report is due on Friday. 264 ms