Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise17.3%
Did it clean up correctly and safely?
51%
Light, ships on by default
71/140 cases passed · 263 ms typical
41%
Polish, experimental
49/120 cases passed · 290 ms typical
When a case fails, whose fault is it?
103 the microphone misheard the words · 37 the cleanup itself slipped
Every case
Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.
140 cases match
noisy-real-noteNoisyclean
HeardTodo for tomorrow finish the report call the bank and pick up groceries. 0% misheard
Was saidtodo for tomorrow finish the report call the bank and pick up groceries
InstantTodo for tomorrow finish the report call the bank and pick up groceries.
LightTodo for tomorrow: finish the report, call the bank, and pick up groceries. 357 ms
PolishTodo for tomorrow: finish the report, call the bank, and pick up groceries. 445 ms
tts-caps-properSyntheticneeds review
HeardI met sarah Sarai in New York on Tuesday. 13% misheard
Was saidi met sarah in new york on tuesday
InstantI met Sarai in New York on Tuesday.
LightI met Sarai in New York on Tuesday. 243 ms
PolishI met Sarai in New York on Tuesday. 250 ms
Why?the result is missing “Sarah”
tts-corr-money-noSyntheticneeds review
HeardThe plan is twenty five 35 no forty dollars $40 a month. 40% misheard
Was saidthe plan is twenty five no forty dollars a month
InstantThe plan is 35 no $40 a month.
LightThe plan is 35 no $40 a month. 244 ms
PolishThe plan is $35 no $40 a month, not $40. 331 ms
Why?the microphone misheard 40% of the words
tts-corr-name-chainSyntheticneeds review
HeardCall John no Jane no actually Mike. 0% misheard
Was saidcall john no jane no actually mike
InstantCall John no Jane no actually Mike.
LightCall John, no Jane, no actually Mike. 251 ms
PolishCall John, no not Jane. no It's actually Mike. 330 ms
Why?the result still contains “John” and “Jane”, which this case forbids
tts-corr-num-chainSyntheticneeds review
HeardWe need twenty 20 no thirty 30 no forty 40 units. 38% misheard
Was saidwe need twenty no thirty no forty units
InstantWe need 20 no 30 no 40 units.
LightWe need 20, no 30, no 40 units. 335 ms
PolishWe need 20, no 30, no and 40 units. 355 ms
Why?the result still contains “20” and “30”, which this case forbids; the microphone misheard 38% of the words
tts-corr-time-noSyntheticneeds review
HeardLet's meet at three 3 no four thirty 4.30. 43% misheard
Was saidlet's meet at three no four thirty
InstantLet's meet at 3 no 4.30.
LightLet's meet at 3 no 4:30. 298 ms
PolishLet's meet at 3 no 4:30. 263 ms
Why?the microphone misheard 43% of the words
tts-disf-daySyntheticclean
HeardLet's do it on Monday sorry I mean Tuesday. 0% misheard
Was saidlet's do it on monday sorry i mean tuesday
InstantLet's do it on Monday sorry I mean Tuesday.
LightLet's do it on Monday sorry I mean Tuesday. 193 ms
PolishLet's do it on Monday sorry I mean Tuesday. 200 ms
tts-disf-false-startsSyntheticneeds review
HeardSo the the main me thing is uh the main that my point is we need to ship. 33% misheard
Was saidso the the main thing is uh the main point is we need to ship
InstantSo the me thing is that my point is we need to ship.
LightSo the me thing is that my point is we need to ship. 393 ms
PolishSo the me thing is that my point is we need to ship. 379 ms
Why?the microphone misheard 33% of the words
tts-disf-heavy-stutterSyntheticclean
HeardI I I think we we should just go go now. 0% misheard
Was saidi i i think we we should just go go now
InstantI I I think we we should just go go now.
LightI I I think we we should just go go now. 266 ms
PolishI I I think we we should just go go now. 250 ms
tts-disf-repeatSyntheticneeds review
Heardthe Thother the report is is ready to go. 13% misheard
Was saidthe the report is is ready to go
InstantThother the report is is ready to go.
LightThother the report is is ready to go. 210 msthe cleanup was rejected, so the original was kept
PolishThother The report is is ready to go. 201 ms
Why light?the result is missing “report is ready”; the result still contains “is is”, which this case forbids; guard rejected