None when the input is not valid for that kind. Rust owns these rules. The
implementation reuses the upstream
text-processing-rs
v0.3.0 parsers. Local Rust code adds grammar, complete-span validation, and
canonicalization only where the upstream parser is not sufficient; it is not a
second Python realizer or a fork of the upstream library. A local parser may
run before the upstream parser for a kind with additional supported forms,
while delegated kinds retain the upstream parser after local guards pass.
From a sentence to scored options
The public runtime does not ask Rust to normalize a whole sentence. Python enumerates every contiguous token span and sends each distinct span string torealize_candidate_batch() once. Rust evaluates the 13 kinds for each
string in parallel. A kind can return no output, one output, or multiple
outputs; CARDINAL, for example, can preserve distinct interpretations of a
spoken number. Rust drops an output only when it is byte-for-byte identical
to the source span.
The batch groups identical written outputs across kinds into a bitmask. For
one oh five, 105 is one candidate with several kinds, while 01:05 is a
separate TIME candidate. Grouping avoids charging the scorer multiple times
for the same edit while retaining the kind evidence. A sorted map gives each
span a stable output order; Python then attaches the original token and
character offsets to every occurrence. See the
architecture for token alignment and
scoring, and structured prediction
for path selection.
The explicit realize(kind, text) API remains a useful complete-span probe,
but it returns only one option. Use realize_options() when a kind has multiple
valid readings, and the batch API when inspecting the runtime candidate set.
representations_equivalent() below is an evaluation comparator; it does not
add outputs to the runtime graph. This distinction matters when diagnosing a
normalization error: a candidate absent from the batch cannot be chosen by a
better contextual score.
Supported kinds
The Rust
DATE realizer generates possible written date candidates, including
spoken numeric fields. It does not decide whether an ambiguous value such as
03/04/2026 means March 4 or April 3. The Python temporal resolver applies
DateOrder after the model and decoder have selected a candidate. This keeps
candidate generation, contextual selection, and calendar semantics separate.
The explicit ELECTRONIC and PHONE entry points add complete-span guards before
delegating. ELECTRONIC also validates protocol/domain structure and the
delegated output. Unknown trailing words are rejected instead of being
silently ignored or appended to the normalized value.
open quote and close quote each realize to a quote-symbol candidate. The
realizer does not define sentence-level opening and closing quote spacing.
Structural whitespace commands such as new line and new paragraph are not
runtime candidates because the scorer and path renderer do not yet represent
whitespace-only replacements.
Representation comparison
representations_equivalent is evaluation-only. It compares semantic values
without adding display aliases to runtime candidates. It currently supports:
CARDINAL: grouping, zero padding, Roman numerals, and signed zero.DATE: unambiguous field order and separators, month or weekday abbreviations, ordinal suffixes, and era punctuation, including eras on full dates.DECIMAL: grouping, decimal padding, decimal comma, named scales, attached scale suffixes, scientific notation, preserved negative zero, and bounded canonical expansion.DIGIT_SEQUENCE: harmless grouping punctuation and separators are ignored while the digit order and value are preserved.TIME: clock padding and separators, compact clocks, AM/PM case and punctuation, 12/24-hour notation, timezone case and offset padding, and duration or fraction padding.MONEY: currency symbols, ISO codes, currency placement, grouping, decimal zero padding, major/minor units, and named or abbreviated thousand/lakh/crore/ million/billion/trillion scales. Currency identity is preserved:$maps to USD, while explicit CAD/AUD and other currency codes remain distinct.MEASUREMENT: numeric grouping, decimal padding, Unicode minus/spacing, unit symbols and names, square/cubic aliases, compound rate units, and metric/data/power unit aliases. Numeric value and canonical unit must both match. Case-sensitive unit identities remain distinct, including metermversus minutemin, mega versus milli prefixes, and byte versus bit units.ORDINAL: numeric suffixes, ordinal words and scales, optional articles, Roman numerals, and harmless terminal punctuation.PHONE: separator/grouping punctuation, country-code spacing, and SSN or IP structure are normalized while the digit sequence and structure remain significant.
dollar/cent is paired with an explicit non-USD target such as A$ or €;
the source does not contain enough information to infer that currency. The 63
None rows are malformed sil/pa annotations. The remaining 212,618
determinate, non-malformed rows have 100% exact-or-equivalent coverage.
The seven-shard audits also cover 4,761,909 CARDINAL rows (99.913% exact or
equivalent), 12,029,272 DATE rows (99.995%), and 350,825 DECIMAL rows
(99.982%). Their remaining rows are malformed, contradictory, or carry target
annotation suffixes that are not part of the semantic value.
These corpus audits are regression evidence, not the source of the grammar.
Rules must be expressed in terms of generic numeric, date, time, unit, and
span structure. A sentence, URL, annotation token, or other dataset-specific
string must not receive a special case.
Synthetic checks cover signed and zero forms, malformed numeric separators,
balanced digit grouping, bounded scientific expansion, overflow and leap-day
rejection, grouping and locale decimal separators, repeated-dot grouping,
Unicode spacing, invalid timezone offsets, repeated scales, and unrelated
suffixes. These checks are kept alongside the focused Rust tests so future
kinds receive the same complete-span and formatting review.
Update this document and the README when a kind gains or loses behavior. Add a
focused Rust test for each new form.