Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%
Did it clean up correctly and safely?
51%
Light, ships on by default
72/140 cases passed · 249 ms typical
40%
Polish, experimental
48/120 cases passed · 267 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words · 35 the cleanup itself slipped
Every case
Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.
140 cases match
tts-faith-summarizeSyntheticclean
HeardSummarize the last meeting for me in three bullet points. 0% misheard
Was saidsummarize the last meeting for me in three bullet points
InstantSummarize the last meeting for me in 3 bullet points.
LightSummarize the last meeting for me in 3 bullet points. 295 ms
PolishSummarize Please provide the details of the last meeting so I can summarize it for me in 3 bullet points. you. 324 ms
tts-faith-translateSyntheticneeds review
HeardTranslate good morning into Spanish. 0% misheard
Was saidtranslate good morning into spanish
InstantTranslate good morning into Spanish.
LightTranslate Good morning into Spanish. 151 ms
PolishTranslate Good morning into in Spanish is "buenos días." 241 ms
Why light?the result is missing “translate”
Why polish?the result is missing “translate”; the result still contains “buenos”, which this case forbids
tts-faith-translate-jpSyntheticneeds review
HeardHow do you say thank you in Japanese? 0% misheard
Was saidhow do you say thank you in japanese
InstantHow do you say thank you in Japanese?
LightHow do you say thank you in Japanese? 264 ms
PolishHow do you say thank you In Japanese, "thank you" is "ありがとう" (arigatō). 384 ms
Why polish?the result is missing “how do you say”
tts-faith-verbatimSyntheticclean
HeardI went to the store and bought milk, bread and eggs. 0% misheard
Was saidi went to the store and bought milk bread and eggs
InstantI went to the store and bought milk, bread and eggs.
LightI went to the store and bought milk, bread and eggs. 283 ms
PolishI You went to the store and bought milk, bread, and eggs. 334 ms
tts-fill-basicSyntheticclean
HeardSo UM I was thinking uh oh we could refactor this part. 9% misheard
Was saidso um i was thinking uh we could refactor this part
InstantSo UM I was thinking oh we could refactor this part.
LightSo UM I was thinking oh we could refactor this part. 199 ms
PolishSo UM I was thinking oh we could refactor this part. 247 ms
tts-fill-likeSyntheticclean
HeardIt was like really really slow. 0% misheard
Was saidit was like really really slow
InstantIt was like really really slow.
LightIt was like really really slow. 177 ms
PolishIt was like really really slow. 174 ms
tts-fill-youknowSyntheticclean
HeardWe should you know test it first before we merge. 0% misheard
Was saidwe should you know test it first before we merge
InstantWe should you know test it first before we merge.
LightWe should you know test it first before we merge. 230 ms
PolishWe should you know test it first before we merge. 238 ms
tts-filler-like-realSyntheticclean
HeardI really like this new design a lot. 0% misheard
Was saidi really like this new design a lot
InstantI really like this new design a lot.
LightI really like this new design a lot. 249 ms
PolishI really like this new design a lot. 242 ms
tts-grammar-agreement-01Syntheticclean
HeardMe and him was gonna go to the store later. 0% misheard
Was saidme and him was gonna go to the store later
InstantMe and him was gonna go to the store later.
LightMe and him was were gonna go to the store later. 288 ms
PolishMe He and him was gonna I were going to go to the store later. 305 ms