← All runs

Evaluation run

self-correction chains audio

6 July 2026, 13:56 · eval-engine · 0169abd · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

55%
Light, ships on by default
77/140 cases passed · 288 ms typical
49%
Polish, experimental
59/120 cases passed · 273 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  19 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

hfp-corr-money-noBluetoothneeds review
HeardThe plan is twenty five 35 no forty dollars $40 a month. 40% misheard
Was saidthe plan is twenty five no forty dollars a month
InstantThe plan is 35 no $40 a month.
LightThe plan is 35 no $40 a month. 337 ms
PolishThe plan is $35 no $40 a month, not $40. 320 ms
Why?the microphone misheard 40% of the words
hfp-corr-name-chainBluetoothneeds review
HeardCall John no Jane, no, actually, Mike. 0% misheard
Was saidcall john no jane no actually mike
InstantCall John no Jane, no, actually, Mike.
LightCall John no Jane, no, actually, Mike. 185 msthe cleanup was rejected, so the original was kept
PolishCall John no Jane, no, actually, Mike. 155 msthe cleanup was rejected, so the original was kept
Why?the result still contains “John” and “Jane”, which this case forbids
hfp-corr-num-chainBluetoothneeds review
HeardWe need twenty 20, no thirty 30, no forty 40 units. 38% misheard
Was saidwe need twenty no thirty no forty units
InstantWe need 20, no 30, no 40 units.
LightWe need 20 no 30, no 40 units. 245 ms
PolishWe need 20, no 30, no 40 units. 373 ms
Why light?the result is missing “40”; the result still contains “20”, which this case forbids; the microphone misheard 38% of the words
Why polish?the result still contains “20” and “30”, which this case forbids; the microphone misheard 38% of the words
hfp-corr-time-noBluetoothneeds review
HeardLet's meet at three 3 no four thirty 4.30. 43% misheard
Was saidlet's meet at three no four thirty
InstantLet's meet at 3 no 4.30.
LightLet's meet at 3 no 4:30. 288 ms
PolishLet's meet at 3 no 4.30. 3:30. 259 ms
Why light?the microphone misheard 43% of the words
Why polish?the result is missing “4:30”; the result still contains “3:3”, which this case forbids; the microphone misheard 43% of the words
hfp-disf-dayBluetoothclean
HeardLet's do it on Monday, sorry I mean Tuesday. 0% misheard
Was saidlet's do it on monday sorry i mean tuesday
InstantLet's do it on Monday, sorry I mean Tuesday.
LightLet's do it on Monday, sorry I mean Tuesday. 214 ms
PolishLet's do it on Monday, sorry I mean Tuesday. 199 ms
hfp-disf-false-startsBluetoothneeds review
HeardSo the the main thing is uh the main that map point is we need to ship. 27% misheard
Was saidso the the main thing is uh the main point is we need to ship
InstantSo the main thing is that map point is we need to ship.
LightSo the main thing is that map point is we need to ship. 312 ms
PolishSo the main thing is that map point is we need to ship. 362 ms
Why?the microphone misheard 27% of the words
hfp-disf-heavy-stutterBluetoothclean
HeardI I I think we we should just go go now. 0% misheard
Was saidi i i think we we should just go go now
InstantI I I think we we should just go go now.
LightI I I think we we should just go go now. 258 ms
PolishI I I think we we should just go go now. 296 ms
hfp-disf-repeatBluetoothclean
HeardThe the report is is ready to go. 0% misheard
Was saidthe the report is is ready to go
InstantThe the report is is ready to go.
LightThe the report is is ready to go. 205 ms
PolishThe the report is is ready to go. 213 ms
hfp-disf-restartBluetoothclean
HeardI think we should UM, I think we should just ship it. 0% misheard
Was saidi think we should um i think we should just ship it
InstantI think we should UM, I think we should just ship it.
LightI think we should UM, I think we should just ship it. 244 ms
PolishI think we should UM, I think we should just ship it. 219 ms
hfp-disf-scratchBluetoothclean
HeardThe deadline is Friday actually no, let's say next Monday. 0% misheard
Was saidthe deadline is friday actually no let's say next monday
InstantThe deadline is Friday actually no, let's say next Monday.
LightThe deadline is Friday actually no, let's say next Monday. 197 ms
PolishThe deadline is Friday actually no, let's say next Monday. 185 ms