Skip to main content
Premove ITN uses deterministic English inverse text normalization (ITN) realizers in Rust to generate candidates for contextual scoring. The low-level explicit kind API consumes the complete input and returns one semantic value, or None when the input is not valid for that kind. Rust owns these rules. The implementation reuses the upstream text-processing-rs v0.3.0 parsers. Local Rust code adds grammar, complete-span validation, and canonicalization only where the upstream parser is not sufficient; it is not a second Python realizer or a fork of the upstream library. A local parser may run before the upstream parser for a kind with additional supported forms, while delegated kinds retain the upstream parser after local guards pass.

​
Supported kinds

The explicit ELECTRONIC and PHONE entry points add complete-span guards before delegating. ELECTRONIC also validates protocol/domain structure and the delegated output. Unknown trailing words are rejected instead of being silently ignored or appended to the normalized value. open quote and close quote each realize to a quote-symbol candidate. The realizer does not define sentence-level opening and closing quote spacing. Structural whitespace commands such as new line and new paragraph are not runtime candidates because the scorer and path renderer do not yet represent whitespace-only replacements.

​
Representation comparison

representations_equivalent is evaluation-only. It compares semantic values without adding display aliases to runtime candidates. It currently supports:
  • CARDINAL: grouping, zero padding, Roman numerals, and signed zero.
  • DATE: unambiguous field order and separators, month or weekday abbreviations, ordinal suffixes, and era punctuation, including eras on full dates.
  • DECIMAL: grouping, decimal padding, decimal comma, named scales, attached scale suffixes, scientific notation, preserved negative zero, and bounded canonical expansion.
  • DIGIT_SEQUENCE: harmless grouping punctuation and separators are ignored while the digit order and value are preserved.
  • TIME: clock padding and separators, compact clocks, AM/PM case and punctuation, 12/24-hour notation, timezone case and offset padding, and duration or fraction padding.
  • MONEY: currency symbols, ISO codes, currency placement, grouping, decimal zero padding, major/minor units, and named or abbreviated thousand/lakh/crore/ million/billion/trillion scales. Currency identity is preserved: $ maps to USD, while explicit CAD/AUD and other currency codes remain distinct.
  • MEASUREMENT: numeric grouping, decimal padding, Unicode minus/spacing, unit symbols and names, square/cubic aliases, compound rate units, and metric/data/power unit aliases. Numeric value and canonical unit must both match. Case-sensitive unit identities remain distinct, including meter m versus minute min, mega versus milli prefixes, and byte versus bit units.
  • ORDINAL: numeric suffixes, ordinal words and scales, optional articles, Roman numerals, and harmless terminal punctuation.
  • PHONE: separator/grouping punctuation, country-code spacing, and SSN or IP structure are normalized while the digit sequence and structure remain significant.
The seven-shard Google TN TIME audit covers 51,569 rows. The current result is 99.994183% exact or representation-equivalent; the three remaining rows are inconsistent dataset annotations. The identity-aware seven-shard Google TN MONEY audit covers 214,728 rows. The current result is 21,239 exact plus 191,379 representation-equivalent rows. The 2,047 mismatches are underdetermined annotations where a generic spoken dollar/cent is paired with an explicit non-USD target such as A$ or €; the source does not contain enough information to infer that currency. The 63 None rows are malformed sil/pa annotations. The remaining 212,618 determinate, non-malformed rows have 100% exact-or-equivalent coverage. The seven-shard audits also cover 4,761,909 CARDINAL rows (99.913% exact or equivalent), 12,029,272 DATE rows (99.995%), and 350,825 DECIMAL rows (99.982%). Their remaining rows are malformed, contradictory, or carry target annotation suffixes that are not part of the semantic value. These corpus audits are regression evidence, not the source of the grammar. Rules must be expressed in terms of generic numeric, date, time, unit, and span structure. A sentence, URL, annotation token, or other dataset-specific string must not receive a special case. Synthetic checks cover signed and zero forms, malformed numeric separators, balanced digit grouping, bounded scientific expansion, overflow and leap-day rejection, grouping and locale decimal separators, repeated-dot grouping, Unicode spacing, invalid timezone offsets, repeated scales, and unrelated suffixes. These checks are kept alongside the focused Rust tests so future kinds receive the same complete-span and formatting review. Update this document and the README when a kind gains or loses behavior. Add a focused Rust test for each new form.