← All runs

Evaluation run

audio run (incl LibriSpeech)

6 July 2026, 11:18 · eval-engine · 54c20aa · 140 cases, 140 spoken

Did it hear the words right?

Clean human speech20 clips · real recordings3.4%
Bluetooth headset15 clips · HFP, the worst microphone people use16.3%
Synthetic voice89 clips · generated speech19.2%
Noisy room16 clips · speech mixed with room noise23.5%

Did it clean up correctly and safely?

51%
Light, ships on by default
72/140 cases passed · 249 ms typical
40%
Polish, experimental
48/120 cases passed · 267 ms typical
When a case fails, whose fault is it?
105 the microphone misheard the words  ·  35 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

140 cases match

tts-real-emailSyntheticclean
HeardHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around three 3. 4% misheard
Was saidhey sarah just wanted to follow up on the proposal um do you have time to hop on a quick call tomorrow around three
InstantHey Sarah just wanted to follow up on the proposal UM do you have time to hop on a quick call tomorrow around 3.
LightHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on a quick call tomorrow around 3? 553 ms
PolishHey Sarah, just wanted to follow up on the proposal. UM Do you have time to hop on for a quick call tomorrow around 3? 528 ms
tts-real-longSyntheticneeds review
HeardSo I talked to the team today and we decided UM, that we're we are going to push the launch to next week because the the testing isn't done and we want were to make sure everything is solid. 8% misheard
Was saidso i talked to the team today and we decided um that we're going to push the launch to next week because the the testing isn't done and we want to make sure everything is solid
InstantSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were to make sure everything is solid.
LightSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were to make sure everything is solid. 721 msthe cleanup was rejected, so the original was kept
PolishSo I talked to the team today and we decided UM, that we are going to push the launch to next week because the the testing isn't done and we were need to make sure everything is solid. 739 ms
Why light?the result still contains “um” and “the the”, which this case forbids; guard rejected
tts-real-messageSyntheticclean
HeardRunning like ten 10 minutes late traffic is crazy be there as soon as I can. 7% misheard
Was saidrunning like ten minutes late traffic is crazy be there as soon as i can
InstantRunning like 10 minutes late traffic is crazy be there as soon as I can.
LightRunning like 10 minutes late, traffic is crazy, be there as soon as I can. 391 ms
PolishRunning like 10 minutes late. Traffic is crazy. I'll be there as soon as I can. 433 ms
tts-real-noteSyntheticclean
HeardTodo for tomorrow finish the report call the bank and pick up groceries. 0% misheard
Was saidtodo for tomorrow finish the report call the bank and pick up groceries
InstantTodo for tomorrow finish the report call the bank and pick up groceries.
LightTodo for tomorrow: finish the report, call the bank, and pick up groceries. 357 ms
PolishTodo for tomorrow: finish the report, call the bank, and pick up groceries. 378 ms
tts-sens-passwordSyntheticneeds review
HeardMy Wi-Fi password is hunter two three four Hunter234. 75% misheard
Was saidmy wifi password is hunter two three four
InstantMy Wi-Fi password is Hunter234.
LightMy Wi-Fi password is Hunter234. 265 ms
PolishMy Wi-Fi password is Hunter234. 279 msthe cleanup was rejected, so the original was kept
Why light?the microphone misheard 75% of the words
Why polish?guard rejected; the microphone misheard 75% of the words
tts-vocab-acronymSyntheticneeds review
HeardOur a p i API is getting rate limited. 38% misheard
Was saidour a p i is getting rate limited
InstantOur API is getting rate limited.
LightOur API is getting rate limited. 195 ms
PolishOur API is getting being rate limited. 198 ms
Why?the microphone misheard 38% of the words
tts-vocab-namesSyntheticclean
HeardLoop in Raj and Priya on the nvidia Innovidya Integration. 11% misheard
Was saidloop in raj and priya on the nvidia integration
InstantLoop in Raj and Priya on the Innovidya Integration.
LightLoop in Raj and Priya on the Innovidya Integration. 294 ms
PolishLoop in Raj and Priya are looping on the Innovidya Integration. 314 ms
tts-vocab-preserveSyntheticclean
HeardWe deployed the Parakeet model to production. 14% misheard
Was saidwe deployed the parakeet model to production
InstantWe deployed Parakeet model to production.
LightWe deployed the Parakeet model to production. 241 ms
PolishWe deployed the Parakeet model to production. 286 ms
tts-vocab-ragSyntheticneeds review
Heardthe rag Drag pipeline retrieves documents before the model answers. 22% misheard
Was saidthe rag pipeline retrieves documents before the model answers
InstantDrag pipeline retrieves documents before the model answers.
LightDrag pipeline retrieves documents before the model answers. 228 ms
PolishThe drag pipeline retrieves documents before the model answers. 284 ms
Why?the microphone misheard 22% of the words
tts-vocab-techSyntheticclean
HeardWe deploy with Kubernetes and Terraform on AWS. 0% misheard
Was saidwe deploy with kubernetes and terraform on aws
InstantWe deploy with Kubernetes and Terraform on AWS.
LightWe deploy with Kubernetes and Terraform on AWS. 270 ms
PolishWe deploy with Kubernetes and Terraform on AWS. 267 ms