← All runs

Evaluation run

audio run (post ITN fix)

6 July 2026, 13:24 · eval-engine · 059e772 · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise17.3%

Did it clean up correctly and safely?

51%
Light, ships on by default
71/140 cases passed · 263 ms typical
41%
Polish, experimental
49/120 cases passed · 290 ms typical
When a case fails, whose fault is it?
103 the microphone misheard the words  ·  37 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

hfp-corr-money-noBluetoothneeds review
HeardThe plan is twenty five 35 no forty dollars $40 a month. 40% misheard
Was saidthe plan is twenty five no forty dollars a month
InstantThe plan is 35 no $40 a month.
LightThe plan is 35 no $40 a month. 239 ms
PolishThe plan is $35 no $40 a month, not $40. 325 ms
Why?the microphone misheard 40% of the words
hfp-corr-name-chainBluetoothneeds review
HeardCall John no Jane, no, actually, Mike. 0% misheard
Was saidcall john no jane no actually mike
InstantCall John no Jane, no, actually, Mike.
LightCall John, no Jane, no, actually, Mike. 252 ms
PolishCall John, no not Jane. no, It's actually Mike. 335 ms
Why?the result still contains “John” and “Jane”, which this case forbids
hfp-corr-num-chainBluetoothneeds review
HeardWe need twenty 20, no thirty 30, no forty 40 units. 38% misheard
Was saidwe need twenty no thirty no forty units
InstantWe need 20, no 30, no 40 units.
LightWe need 20, no 30, no 40 units. 335 ms
PolishWe need 20, no 30, no 40 units. 381 ms
Why?the result still contains “20” and “30”, which this case forbids; the microphone misheard 38% of the words
hfp-corr-time-noBluetoothneeds review
HeardLet's meet at three 3 no four thirty 4.30. 43% misheard
Was saidlet's meet at three no four thirty
InstantLet's meet at 3 no 4.30.
LightLet's meet at 3 no 4:30. 303 ms
PolishLet's meet at 3 no 4:30. 274 ms
Why?the microphone misheard 43% of the words
hfp-disf-dayBluetoothclean
HeardLet's do it on Monday, sorry I mean Tuesday. 0% misheard
Was saidlet's do it on monday sorry i mean tuesday
InstantLet's do it on Monday, sorry I mean Tuesday.
LightLet's do it on Monday, sorry I mean Tuesday. 188 ms
PolishLet's do it on Monday, sorry I mean Tuesday. 197 ms
hfp-disf-false-startsBluetoothneeds review
HeardSo the the main thing is uh the main that map point is we need to ship. 27% misheard
Was saidso the the main thing is uh the main point is we need to ship
InstantSo the main thing is that map point is we need to ship.
LightSo the main thing is that map point is we need to ship. 425 ms
PolishSo the main thing is that map point is we need to ship. 334 ms
Why?the microphone misheard 27% of the words
hfp-disf-heavy-stutterBluetoothclean
HeardI I I think we we should just go go now. 0% misheard
Was saidi i i think we we should just go go now
InstantI I I think we we should just go go now.
LightI I I think we we should just go go now. 275 ms
PolishI I I think we we should just go go now. 283 ms
hfp-disf-repeatBluetoothclean
HeardThe the report is is ready to go. 0% misheard
Was saidthe the report is is ready to go
InstantThe the report is is ready to go.
LightThe the report is is ready to go. 184 ms
PolishThe the report is is ready to go. 188 ms
hfp-disf-restartBluetoothclean
HeardI think we should UM, I think we should just ship it. 0% misheard
Was saidi think we should um i think we should just ship it
InstantI think we should UM, I think we should just ship it.
LightI think we should UM, I think we should just ship it. 208 ms
PolishI think we should UM, I think we should just ship it. 257 ms
hfp-disf-scratchBluetoothclean
HeardThe deadline is Friday actually no, let's say next Monday. 0% misheard
Was saidthe deadline is friday actually no let's say next monday
InstantThe deadline is Friday actually no, let's say next Monday.
LightThe deadline is Friday actually no, let's say next Monday. 175 ms
PolishThe deadline is Friday actually no, let's say next Monday. 185 ms