← All runs

Evaluation run

faithfulness scoring cleanup

6 July 2026, 13:36 · eval-engine · 059e772 · 89 cases

Did it hear the words right?

This run graded typed text only, so there is nothing here to have misheard.

Did it clean up correctly and safely?

94%
Light, ships on by default
84/89 cases passed · 254 ms typical
89%
Polish, experimental
79/89 cases passed · 266 ms typical
When a case fails, whose fault is it?
0 the microphone misheard the words  ·  15 the cleanup itself slipped

Every case

Each case shows the instant on-device cleanup, then what each mode did to it. Added words are marked, removed words are struck. For a spoken case, Heard is what the microphone transcribed and Was said is the reference it is scored against.

89 cases match

faith-translateTypedclean
Instanttranslate good morning into spanish
Lighttranslate good morning into spanish 242 msthe cleanup was rejected, so the original was kept
Polishtranslate good morning into spanish 291 msthe cleanup was rejected, so the original was kept
faith-translate-jpTypedclean
Instanthow do you say thank you in japanese
LightHow do you say "thank you" in Japanese? 284 ms
Polishhow do you say thank you in japanese 442 msthe cleanup was rejected, so the original was kept
faith-verbatimTypedclean
Instanti went to the store and bought milk bread and eggs
LightI went to the store and bought milk, bread, and eggs. 345 ms
Polishi You went to the store and bought milk, bread, and eggs. 337 ms
fill-basicTypedclean
Instantso i was thinking we could refactor this part
LightSo i was thinking we could refactor this part. 205 ms
Polishso I was thinking we could refactor this part. 245 ms
fill-likeTypedclean
Instantit was like really really slow
LightIt was like really really slow. 188 ms
PolishIt was like really really slow. 175 ms
fill-youknowTypedclean
Instantwe should you know test it first before we merge
LightWe should you know test it first before we merge. 244 ms
PolishWe should you know test it first before we merge. 251 ms
filler-like-realTypedclean
Instanti really like this new design a lot
LightI really like this new design a lot. 262 ms
PolishI really like this new design a lot. 316 ms
grammar-agreement-01Typedclean
Instantme and him was gonna go to the store later
LightMe and him was gonna go to the store later. 300 ms
Polishme I and him was gonna were going to go to the store later. 329 ms
grammar-faithful-question-01Typedclean
Instantwhat is the capital of france
Lightwhat is the capital of france 220 msthe cleanup was rejected, so the original was kept
Polishwhat is the capital of france 224 msthe cleanup was rejected, so the original was kept
grammar-numbers-keep-01Typedneeds review
Instantthe budget is like 50000 no wait 75000 for q3
LightThe budget is like 50000 no wait 75,000 for Q3. 359 ms
PolishThe budget is like 50000 no wait $75,000 for Q3. No wait, it was $50,000. 623 ms
Why polish?the result still contains “50,000” and “$50”, which this case forbids