← All runs

Evaluation run

self-correction chains audio

6 July 2026, 13:56 · eval-engine · 0169abd · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

55%
Light, ships on by default
77/140 cases passed · 288 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  19 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

noisy-disf-false-startsNoisyneeds review
HeardSo the the main my thing is uh the main that map point is we need to ship shift. 47% misheard
Was saidso the the main thing is uh the main point is we need to ship
InstantSo my thing is that map point is we need to shift.
LightSo my thing is that map point is we need to shift. 273 msthe cleanup was rejected, so the original was kept
PolishSo my thing is that map point is we need to shift. 341 ms
Why?the result is missing “we need to ship”; the microphone misheard 47% of the words
noisy-disf-heavy-stutterNoisyclean
HeardI I I think we we should just go go now. 0% misheard
Was saidi i i think we we should just go go now
InstantI I I think we we should just go go now.
LightI I I think we we should just go go now. 249 ms
PolishI I I think we we should just go go now. 298 ms
noisy-disf-repeatNoisyclean
Heardthe Tha the report is is ready to go. 13% misheard
Was saidthe the report is is ready to go
InstantTha the report is is ready to go.
LightTha The report is is ready to go. 211 ms
PolishTha The report is is ready to go. 199 ms
noisy-disf-restartNoisyclean
HeardI think we should UM, I think we should just ship it. 0% misheard
Was saidi think we should um i think we should just ship it
InstantI think we should UM, I think we should just ship it.
LightI think we should UM, I think we should just ship it. 241 ms
PolishI think we should UM, I think we should just ship it. 219 ms
noisy-disf-scratchNoisyclean
HeardThe deadline is Friday actually no let's say next Monday. 0% misheard
Was saidthe deadline is friday actually no let's say next monday
InstantThe deadline is Friday actually no let's say next Monday.
LightThe deadline is Friday actually no let's say next Monday. 196 ms
PolishThe deadline is Friday actually no let's say next Monday. 182 ms
noisy-disf-selfcorrect-nameNoisyneeds review
HeardSend it to John uh i mean jane Uhai Mijane. 50% misheard
Was saidsend it to john uh i mean jane
InstantSend it to John Uhai Mijane.
LightSend it to John Uhai Mijane. 174 msthe cleanup was rejected, so the original was kept
PolishSend it to John Uhai Mijane. 167 msthe cleanup was rejected, so the original was kept
Why?the result still contains “John”, which this case forbids; the microphone misheard 50% of the words
noisy-disf-selfcorrect-timeNoisyneeds review
HeardLet's meet at three 3 no wait four 4 o'clock. 25% misheard
Was saidlet's meet at three no wait four o'clock
InstantLet's meet at 3 no wait 4 o'clock.
LightLet's meet at 3 no wait 4 o'clock. 254 ms
PolishLet's meet at 3 no wait 4 o'clock. 227 ms
Why?the microphone misheard 25% of the words
noisy-real-emailNoisyclean
HeardHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around three 3. 4% misheard
Was saidhey sarah just wanted to follow up on the proposal um do you have time to hop on a quick call tomorrow around three
InstantHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around 3.
LightHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on a quick call tomorrow around 3? 602 ms
PolishHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on a quick call tomorrow around 3? 554 ms
noisy-real-longNoisyneeds review
HeardSo I talked to the team today and we decided UM, that we're we are going to push the launch to next week because the the testing isn't done and we want were to make sure everything is solid. 11% misheard
Was saidso i talked to the team today and we decided um that we're going to push the launch to next week because the the testing isn't done and we want to make sure everything is solid
InstantSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were to make sure everything is solid.
LightSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were to make sure everything is solid. 892 msthe cleanup was rejected, so the original was kept
PolishSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were need to make sure everything is solid. 720 ms
Why light?the result still contains “um”, which this case forbids
noisy-real-messageNoisyclean
HeardRunning like ten 10 minutes late traffic is crazy be there as soon as I can. 7% misheard
Was saidrunning like ten minutes late traffic is crazy be there as soon as i can
InstantRunning like 10 minutes late traffic is crazy be there as soon as I can.
LightRunning like 10 minutes late, traffic is crazy, be there as soon as I can. 442 ms
PolishRunning like 10 minutes late. Traffic is crazy. I'll be there as soon as I can. 428 ms