← All runs

Evaluation run

audio run (post ITN fix)

6 July 2026, 13:24 · eval-engine · 059e772 · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise17.3%

Did it clean up correctly and safely?

51%
Light, ships on by default
71/140 cases passed · 263 ms typical
41%
Polish, experimental
49/120 cases passed · 290 ms typical
When a case fails, whose fault is it?
103 the microphone misheard the words  ·  37 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

hfp-disf-selfcorrect-nameBluetoothneeds review
HeardSend it to John uh i mean jane Ohai Mijane. 50% misheard
Was saidsend it to john uh i mean jane
InstantSend it to John Ohai Mijane.
LightSend it to John Ohai Mijane. 159 msthe cleanup was rejected, so the original was kept
PolishSend it to John Ohai Mijane. 161 msthe cleanup was rejected, so the original was kept
Why?the result still contains “John”, which this case forbids; guard rejected; the microphone misheard 50% of the words
hfp-disf-selfcorrect-timeBluetoothneeds review
HeardLet's meet at three 3 no wait four 4 o'clock. 25% misheard
Was saidlet's meet at three no wait four o'clock
InstantLet's meet at 3 no wait 4 o'clock.
LightLet's meet at 3 no wait 4 o'clock. 220 ms
PolishLet's meet at 3 no wait 4 o'clock. 272 ms
Why?the microphone misheard 25% of the words
hfp-real-emailBluetoothclean
HeardHey Sarah just wanted to follow up on the proposal UM, do you have time to hop on a quick call tomorrow around three 3? 4% misheard
Was saidhey sarah just wanted to follow up on the proposal um do you have time to hop on a quick call tomorrow around three
InstantHey Sarah just wanted to follow up on the proposal UM, do you have time to hop on a quick call tomorrow around 3?
LightHey Sarah, just wanted to follow up on the proposal. UM, Do you have time to hop on a quick call tomorrow around 3? 539 ms
PolishHey Sarah, just wanted to follow up on the proposal. UM, Do you have time to hop on for a quick call tomorrow around 3? 632 ms
hfp-real-longBluetoothneeds review
HeardSo I talked to the team today and we decided UM, that we're we are going to push the launch to next week because the the testing isn't done and we want were to make sure everything is solid. 11% misheard
Was saidso i talked to the team today and we decided um that we're going to push the launch to next week because the the testing isn't done and we want to make sure everything is solid
InstantSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were to make sure everything is solid.
LightSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were to make sure everything is solid. 714 msthe cleanup was rejected, so the original was kept
PolishSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the testing isn't done and we were want to make sure everything is solid. 887 ms
Why light?the result still contains “um”, which this case forbids; guard rejected
hfp-real-messageBluetoothclean
HeardRunning like ten 10 minutes late traffic is crazy be there as soon as I can. 7% misheard
Was saidrunning like ten minutes late traffic is crazy be there as soon as i can
InstantRunning like 10 minutes late traffic is crazy be there as soon as I can.
LightRunning like 10 minutes late, traffic is crazy, be there as soon as I can. 382 ms
PolishRunning like 10 minutes late. Traffic is crazy. I'll be there as soon as I can. 517 ms
ls-1272-128104-0000Real speechclean
Heardmister Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. 0% misheard
Was saidmister quilter is the apostle of the middle classes and we are glad to welcome his gospel
Instantmister Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel.
LightMister Quilter is the Apostle of the Middle Classes, and we are glad to welcome his gospel. 466 ms
ls-1272-128104-0001Real speechclean
HeardNor is mister Quilter's manner less interesting than his matter. 0% misheard
Was saidnor is mister quilter's manner less interesting than his matter
InstantNor is mister Quilter's manner less interesting than his matter.
LightNor is Mister Quilter's manner less interesting than his matter. 325 ms
ls-1272-128104-0002Real speechclean
HeardHe tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. 0% misheard
Was saidhe tells us that at this festive season of the year with christmas and roast beef looming before us similes drawn from eating and its results occur most readily to the mind
InstantHe tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind.
LightHe tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. 764 ms
ls-1272-128104-0003Real speechclean
HeardHe has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of rocky Ithaca. 0% misheard
Was saidhe has grave doubts whether sir frederick leighton's work is really greek after all and can discover in it but little of rocky ithaca
InstantHe has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of rocky Ithaca.
LightHe has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of rocky Ithaca. 715 ms
ls-1272-128104-0004Real speechneeds review
HeardLinnell's pictures are a sort of up guards and at em atom paintings, and Mason's exquisite idylls are as national as a jingo poem. mister birket Burkitt Foster's landscapes landscapepes smile at one much in the same way that mister Carker used to flash his teeth, and mister John Collier gives his sitter a cheerful slap on the back. before he says, like a shampooer in a Turkish bath, next man! 6% misheard
Was saidlinnell's pictures are a sort of up guards and at em paintings and mason's exquisite idylls are as national as a jingo poem mister birket foster's landscapes smile at one much in the same way that mister carker used to flash his teeth and mister john collier gives his sitter a cheerful slap on the back before he says like a shampooer in a turkish bath next man
InstantLinnell's pictures are a sort of up guards and atom paintings, and Mason's exquisite idylls are as national as a jingo poem. mister Burkitt Foster's landscapepes smile at one much in the same way that mister Carker used to flash his teeth, and mister John Collier gives his sitter a cheerful slap on the back. before he says, like a shampooer in a Turkish bath, next man!
LightLinnell's pictures are a sort of up guards and atom paintings, and Mason's exquisite idylls are as national as a jingo poem. mister Burkitt Foster's landscapepes smile at one much in the same way that mister Carker used to flash his teeth, and mister John Collier gives his sitter a cheerful slap on the back. before he says, like a shampooer in a Turkish bath, next man! 2320 msthe cleanup was rejected, so the original was kept
Why?guard rejected