Skip to main content
Premove ITN reached 99.50% semantic entity accuracy (398/400) on the voice-agent subset of a frozen synthetic stress benchmark. It led the measured semantic accuracy comparison, but it did not lead latency. These results do not estimate accuracy on live production traffic.

​
Comparison

The voice-agent figures score the 400 rows in group=voice_agent. The overall semantic figures pool 1,640 declared entities across all 1,500 rows. The overall mean latency uses all 1,500 requests. For voice-agent rows alone, Premove’s mean warm latency was 57.41 ms.

​
What the metric measures

Semantic entity accuracy checks whether the normalized value is correct, while allowing harmless formatting differences. For example, 7:30 and 07:30 count as the same time, while 7:03 does not. It evaluates structured values rather than full-sentence exact match. Premove’s overall strict exact match was 40.53% in the same run. The frozen benchmark specification defines the scoring contract. The retained report contains the complete result tables.

​
Test design

  • The balanced synthetic suite has 1,500 rows, including 400 dedicated voice-agent rows across eight domains.
  • It was held out from training and checkpoint selection.
  • Every backend received only transcript text. It did not receive gold spans, labels, domains, difficulty, or expected outputs.
  • The Premove timing used the retained release Rust build. Requests were sequential, batch one, on an Apple M4 MacBook Air with MPS completion and eight Rayon workers. Models were loaded and warmed before latency was measured. Download and initialization were excluded.
See the reproduction instructions and run metadata.

​
Limits of the result

This suite deliberately stresses structured values and ambiguity. It is not a random sample of customer traffic. Independent human gold adjudication remains pending, and some similarity-contamination checks are incomplete. The retained report also records collision and KEEP errors. Do not infer universal ITN superiority from the headline voice-agent result. For a concrete failure, the retained run predicted the cash price in dollars was 07:36 for the input the cash price in dollars was seven thirty six, where the expected written amount was $7.36. The model also changed the appointment starts at one oh five to the appointment starts at 105 instead of the expected time 1:05. These cases show why application-side validation remains necessary. text-processing-rs is an upstream ablation used by Premove’s Rust candidate layer, not a fully independent architecture. The Thutmose comparison uses the NVIDIA model artifact through an isolated weight-compatible loader, not the current NeMo API. These differences matter when interpreting the comparison. Premove also carries a deployment tradeoff: about a 1.6 GB initial download, multi-second initialization, and higher warm latency than either comparison backend. Read deployment guidance and how the system works before choosing it for a service.