Italian

Status: Current Last updated: 2026-07-28 13:27 EDT

Scope

Italian is a Romance language with productive MWT (multi-word token) phenomena that Stanza’s neural tokenizer and MWT processor expand natively:

• Preposition+article contractions: al → a + il, del → di + il, nel → in + il, sul → su + il, della → di + la
• Clitic-article elisions: l'amico → l' + amico, dell'opera → di + l'opera, all'amore → a + l'amore

All BA2-inherited per-language MWT-override rules were audited and removed in 2026-04 (see History). Since 2026-07-28 Italian has a principled tokenize-stage MWT POLICY (not an override table): every split Stanza proposes is validated against the four multi-word patterns Italian actually has, and splits it wrongly withholds are forced. Design and evidence: Stanza Limitations.

Intervention status: the complete arc (as of 2026-07-28)

Italian has had three generations of intervention, each subsuming part of its predecessor. All three are documented on this page; this section is the authoritative summary of what is CURRENT.

Generation 1, BA2-inherited tokenize overrides: REMOVED (2026-04-21). The per-language string-manipulation rules ported from BA2 were audited with paired probes and deleted (dormant, redundant, or harmful on modern Stanza). See History.

Generation 2, the downstream %mor reconciler (2026-04/05): per-surface allowlists. Hand-curated tables in crates/batchalign-transform/src/morphosyntax/lang_it.rs repairing specific observed damage after Stanza produced it: IT_MIS_SPLIT_OVERRIDES (Defects 6/7, 23 entries), IT_COMPOUND_IMPERATIVES (Defect 8, 11 entries), IT_COMPONENT_REWRITES (Defects 9/10, 3 entries). Grown one production incident at a time; each entry documented below.

Generation 3, the tokenize-stage MWT policy (2026-07-28): validate every split, both directions. Implemented in batchalign/inference/_italian_mwt.py and applied in the tokenizer postprocessor, BEFORE Stanza’s MWT processor runs. Italian has exactly four legitimate multi-word patterns (preposition+article, ecco+enclitic, clitic cluster, verb+enclitic); a forced probe previews Stanza’s split for every candidate, the four patterns judge it, and the pipeline is told to suppress an illegitimate split or force a withheld genuine one. Full design, structural guards, and the measured evidence: Stanza Limitations.

Per-defect status

Corpus repair status

Measured 2026-07-28 with a language-resolving Rust audit over the typed CHAT AST (all 106,158 corpus files): 657 Italian-primary files; 1,608 Italian-resolved single-word utterances across 338 files carry verb+enclitic %mor (committed damage plus genuine imperatives). Regeneration of all Italian-primary files with the fixed pipeline is the repair step; outputs are diffed before entering the data repos.

What Stanza handles natively

Paired probes (free-tokenize vs our postprocessor) were run on 60+ Italian constructions on Stanza 1.11.1. All of the following produce identical output on both paths and satisfy the morphotag 1-to-1 invariant:

Probes in batchalign/tests/investigations/_cases/italian.py (typed ProbeCase fixtures consumed by the matrix harness at test_stanza_mwt_probe_matrix.py).

Known Stanza limitations

All three issues below are content-quality defects. The %mor count invariant holds, Stage 3’s assemble_mors in crates/batchalign-transform/src/morphosyntax/mapping_helpers.rs collapses Stanza’s MWT Range tokens into a single compound %mor entry per CHAT word, but the emitted entry carries linguistically wrong content: fake lemmas, spurious features, or the wrong POS.

Each row below shows the %mor that actually ships downstream from a minimal ita probe CHAT run through batchalign3 morphotag.

Defect 6: words with clitic-shaped endings split into fake verb+clitic compounds

Italian words whose last one-to-two characters look like a clitic (-la, -lo, -le, -li, -ne, -no, -ni, -mi, -ti, -ci, -vi, -si) get wrapped by Stanza in an MWT Token and analyzed as verb stem + enclitic pronoun with a bogus stem lemma , regardless of actual part of speech. The defect fires on:

• Verbs in imperative position: parla forte → verb|par~pron|la (should be verb|parlare-Imp-S2).
• Common nouns: arancione (orange) → verb|arancio~pron|ne with Part Past; seggiola (chair) → verb|seggio~pron|la; gomitolo (ball of yarn) → verb|gomito~pron|lo; divano (sofa) → verb|diva~pron|no; bottone (button) → verb|botto~pron|ne; cavallone (big horse) → verb|cavallo~pron|ne; cielo (sky) → verb|cie~pron|lo (cie is not a word).
• Adjectives: piccolo/piccola (small, m/f) → verb|picco~pron|lo / verb|picco~pron|la with Part Past.
• Baby-talk diminutives: coccole, babbolo, pettole.

Most non-verb hits carry Part Past features, Stanza confidently treats the whole surface as a past participle plus clitic.

Every row ships one %mor item per CHAT word (Stage 3’s assemble_mors collapses the MWT Range correctly), so the count invariant holds. Every row’s linguistic content is wrong.

A corpus-wide audit of committed %mor content (pre-parsed JSON snapshot of the TalkBank CHAT corpora) found 65 Defect-6 hits across 417 Italian files and 15 distinct surface forms. Mid-sentence position does protect verbs in context, la storia parla di ... gets correct verb|parlare-Fin-Ind-Pres-S3: but the noun/adjective pseudo-analyses fire independent of position.

Pinned as stanza-it-verb-clitic-pos-split in Stanza Limitations, Defect 6. Prevented at source since 2026-07-28 by the tokenize-stage MWT policy (the spurious split is suppressed, so Stanza analyzes the whole word); the IT_MIS_SPLIT_OVERRIDES reconciler entries remain as a dormant backstop.

Defect 7: sentence-initial article la gets junk il + i MWT expansion

Input:

*CHI:	la storia parla di un bambino .

Current %mor:

%mor:	det|il-Masc-Def-Art-Sing~det|il-Masc-Def-Art-Plur noun|storia-Fem
        verb|parlare-Fin-Ind-Pres-S3 adp|di det|uno-Masc-Ind-Art-Sing
        noun|bambino-Masc .

One %mor item per CHAT word, count is right. But the first item has lemma=il with masc-singular + masc-plural compound features, for a feminine-singular article la. Correct would be det|la-Fem-Def-Art-Sing.

parla mid-sentence gets its proper analysis (verb|parlare-Fin-Ind-Pres-S3), confirms Defect 6 is position-sensitive and unrelated to Defect 7. Position sensitivity of Defect 7 itself (whether mid-sentence la also gets the junk expansion) has not yet been characterized.

Pinned as stanza-it-la-sentence-initial-split in Stanza Limitations, Defect 7. Prevented at source since 2026-07-28: the policy rejects il + i because it is not a genuine preposition+article fusion (no preposition in the base and la is not in the contracted paradigm), so the split never happens.

Defect 8 (candidate): mid-sentence dammela tagged as ADJ, lemma normalized to dammelo

Input:

*CHI:	per favore dammela .

Current %mor:

%mor:	adp|per noun|favore-Masc adj|dammelo-S1 .

One %mor item per CHAT word, correct count. But dammela in mid-sentence position gets tagged ADJ with lemma dammelo (wrong gender) and no clitic decomposition at all. The bare-compound case (dammela alone as a single utterance) is handled correctly, verb|dare-Inf-Ind-Imp-S2~pron|me-Prs-S1~pron|la-Prs-S3: so this is a context-dependent Stanza misclassification, distinct from Defect 6.

Not yet pinned as a named Defect in the registry (proposed slug stanza-it-dammela-mid-sentence-adj); a corpus scan for -(la|lo|le|li|mi|ti|ci|vi|si|ne)$ verb+clitic compounds in mid-sentence position would quantify prevalence and inform whether it warrants its own entry.

Defect 9: Range-expansion with wrong head POS (dative -glie- stack)

Input:

*CHI:	per favore dagliela .

Before Defect 9 reconciler:

%mor:	adp|per noun|favore-Masc adp|da~pron|gli-Prs-S3~pron|la-Prs-S3 .

Stanza expands dagliela (2sg imperative of dare + 3sg.dat + 3sg.f.acc) as a structurally-correct 3-piece MWT, but tags the head component da with ADP/da instead of VERB/dare. The preposition da (“from, by”) is homographic with the imperative verb form, and Stanza’s POS layer prefers the preposition reading even though the clitic stack only makes sense under the verb reading.

This is distinct from Defect 6 (where Stanza spuriously creates an MWT split for a non-compound word) and Defect 8 (where Stanza omits MWT expansion entirely mid-sentence). The expansion shape is right; only component 0’s POS/lemma/feats are wrong.

Sibling forms in the same dative stack (digliela → di/VERB/dire, portagliela → porta/VERB/portare, prendigliela → prendi/VERB/prendere) are Stanza-correct, verified by direct probe. The defect is specific to surfaces where the head clitic-stripped form is homographic with a non-verb word.

Pinned observation-only case in the probe matrix: dagliela_mid_sentence in _cases/italian.py.

Defect 10: head-lemma-only rewrite for genuine imperative+clitic MWTs

Input:

*CHI:	posala .

Before Defect 10 reconciler:

%mor:	verb|posa~pron|la-Prs-S3 .

Stanza expands posala as a 2-piece MWT (posa/VERB + la/PRON) , structurally correct: this IS a genuine imperative (2sg of posare, “put down”) + accusative clitic. However the head component’s lemma is posa (surface-echo) rather than the canonical infinitive posare.

Unlike Defect 9 (dagliela), the head POS is correct (VERB); only the lemma is wrong. The component-rewrite mechanism in IT_COMPONENT_REWRITES handles both shapes because rewriting a field that Stanza already got right is idempotent, no new allowlist or new reconciler path was needed.

Defect 10 is verb-specific to posare: the cross-verb probe confirmed guardare, toccare, aspettare, mangiare, chiamare, lasciare, cambiare, provare, giocare, portare, suonare, chiudere all lemmatize correctly in this position. Stanza’s Italian model has a specific weakness on the posare paradigm. Allowlist entries: posala, posalo (IT_COMPONENT_REWRITES).

Singleton audit hits deliberately NOT added to the allowlist

A fleet JSON audit surfaced 5 singleton Defect 6 surfaces that were deliberately skipped despite confirming as mis-splits under current Stanza:

• soffioni (plural of soffione, dandelion), obscure
• coccolo: dialectal / obscure
• pettole: dialectal / obscure
• babbolo: likely a typo or child-speech approximation
• tecala: non-standard; likely corrupted

If any of these recur in new corpus processing, promote to the allowlist at that point. The probe matrix records each as observation-only (<surface>_alone in _cases/italian.py), so a Stanza upgrade that fixes them would flip those probes from silent-pass-with-stanza_words=2 to stanza_words=1.

Correctly-handled constructions (preserve, do not rewrite)

Bare single-utterance imperative+clitic compounds are produced correctly and must not be touched by any future content-quality rule. Pinned as counterexamples in the probe matrix:

Any future content-quality rule for Italian must leave these analyses untouched, they’re the correctness control group.

Reconciler architecture

This section is the overview. The per-defect subsections below are the detail. A successor who has never seen this subsystem should be able to read this section alone and understand what the Italian reconciler does and where the code lives.

Why the reconciler exists

Stanza 1.11.1’s Italian model has several distinct defect shapes in its output for imperative+clitic compounds and clitic-shaped-ending nouns. Each defect type produces malformed %mor content even when the structural (1-to-1) invariant between CHAT words and %mor items holds. The reconciler is a closed curated set of per-surface overrides that post-processes Stanza’s output to produce correct %mor content without retraining Stanza or writing a full Italian morphological analyzer.

It is explicitly a hack layer: every entry is expected to be retired eventually as Stanza improves upstream. The retirement workflow is to empty the allowlists in lang_it.rs, rerun the Italian integration tests (crates/batchalign/src/chat_ops/nlp/mapping/tests/italian_defects.rs), and remove any entry whose dependent test now passes without it.

Defect taxonomy

Eight numbered defect shapes have been observed in Stanza’s Italian output. Five are actively reconciled; three are deliberately not (documented below).

flowchart TD
    Start["Italian input word from Stanza"]
    MWT{"Stanza emits MWT Range?"}
    Range["UdId::Range(start, end)<br/>+ N component UdWords"]
    Single["UdId::Single(id)<br/>(one UD word)"]

    SpuriousSplit{"Is the Range a<br/>spurious split of<br/>a non-compound?"}
    Def6["Defect 6<br/>Range collapse to 1 Mor<br/>(parla, arancione, piccolo, …)"]
    Def7["Defect 7<br/>Sentence-initial la → il+i<br/>Range collapse"]
    HeadMisPOS{"Head component<br/>POS wrong?"}
    Def9["Defect 9<br/>Component head rewrite<br/>POS+lemma (dagliela)"]
    HeadLemmaOnly{"Head lemma<br/>surface-echo?"}
    Def10["Defect 10<br/>Component head rewrite<br/>lemma only (posala)"]
    NormalRange["Normal assemble_mors<br/>(correct MWT)"]

    SingleMisPOS{"Single mis-tagged<br/>ADJ/NOUN/VERB<br/>as compound?"}
    Def8["Defect 8<br/>Multi-chunk emit<br/>(dammela, aprila, finila)"]
    Def12["Defect 12<br/>VERB correct lemma<br/>missing MWT<br/>(aprilo)"]
    Def13["Defect 13<br/>VERB fabricated lemma<br/>(leggila)"]
    NormalSingle["Normal map_ud_word_to_mor"]

    Start --> MWT
    MWT -->|"yes"| Range
    MWT -->|"no"| Single

    Range --> SpuriousSplit
    SpuriousSplit -->|"yes: parla/arancione/…"| Def6
    SpuriousSplit -->|"yes: sentence-initial la"| Def7
    SpuriousSplit -->|"no"| HeadMisPOS
    HeadMisPOS -->|"yes: dagliela → da/ADP"| Def9
    HeadMisPOS -->|"no"| HeadLemmaOnly
    HeadLemmaOnly -->|"yes: posala → posa lemma"| Def10
    HeadLemmaOnly -->|"no"| NormalRange

    Single --> SingleMisPOS
    SingleMisPOS -->|"ADJ: dammela, finila"| Def8
    SingleMisPOS -->|"NOUN: aprila, aprili"| Def8
    SingleMisPOS -->|"VERB missing MWT: aprilo"| Def12
    SingleMisPOS -->|"VERB fabricated lemma: leggila"| Def13
    SingleMisPOS -->|"no defect"| NormalSingle

Verified against: crates/batchalign-transform/src/morphosyntax/lang_it.rs (allowlist definitions), crates/batchalign-transform/src/morphosyntax/sentence_mapping.rs::map_ud_sentence (dispatch), _cases/italian.py probe observations (defect signatures).

Defects 11 (unused number in current taxonomy), and other per-verb anomalies surfaced during probing but deemed too rare to reconcile individually, live in the “Singleton audit hits” and “Open work” sections below.

Levels of processing

The reconciler operates in two logical layers over Stanza’s output. The first layer rewrites UD content (mutating UdWord-equivalent state before %mor synthesis); the second layer fixes up the chunk-index and GRA accounting so the reconciled output aligns with CHAT’s per-chunk %gra convention.

flowchart LR
    subgraph Input
        direction TB
        Stanza["Stanza UD sentence<br/>(UdWord[], UdId::Range|Single)"]
    end

    subgraph "Layer 1: UD rewrite (lang_it.rs)"
        direction TB
        MisSplit["IT_MIS_SPLIT_OVERRIDES<br/>(Defect 6 + 7)<br/>Range → synthetic single UdWord"]
        Compound["IT_COMPOUND_IMPERATIVES<br/>(Defect 8 + 12 + 13)<br/>Single → verb UdWord + N clitic UdWords"]
        Component["IT_COMPONENT_REWRITES<br/>(Defect 9 + 10)<br/>Range components mutated in place"]
    end

    subgraph "Layer 2: Mor synthesis + GRA accounting (morphosyntax)"
        direction TB
        MapWord["map_ud_word_to_mor<br/>per UdWord"]
        Assemble["assemble_mors<br/>(Range → multi-chunk Mor)"]
        WithClitic["Mor::with_post_clitic<br/>(Single → multi-chunk Mor)"]
        BuildGra["build_gra_and_validate<br/>, chunk index<br/>, GRA relations<br/>, count invariant"]
    end

    subgraph Output
        direction TB
        Mors["Vec&lt;Mor&gt;<br/>(each chunk countable)"]
        Gras["Vec&lt;GrammaticalRelation&gt;<br/>(1 per chunk + terminator PUNCT)"]
    end

    Stanza --> MisSplit
    Stanza --> Compound
    Stanza --> Component
    MisSplit --> MapWord
    Compound --> MapWord
    Compound --> WithClitic
    Component --> Assemble
    MapWord --> Mors
    Assemble --> Mors
    WithClitic --> Mors
    Mors --> BuildGra
    BuildGra --> Gras

Verified against: crates/batchalign-transform/src/morphosyntax/lang_it.rs (three allowlists + helpers), crates/batchalign-transform/src/morphosyntax/sentence_mapping.rs::map_ud_sentence (orchestration), crates/batchalign-transform/src/morphosyntax/mapping_helpers.rs::assemble_mors (Range reassembly), talkbank-model::model::dependent_tier::mor::Mor::with_post_clitic (clitic stacking).

Chunk accounting via provenance (a later refactor)

The most subtle part of the reconciler is how a single UdId::Single can produce multiple %mor chunks (e.g. dammela → verb|dare~pron|me~pron|la is 3 chunks from 1 UD word). The original approach used two language- specific side-tables (reconciled_ranges, reconciled_singles) threaded through build_gra_and_validate. That coupling leaked per-language reconciliation detail into a language-neutral helper.

The current design uses a ChunkProvenance data structure produced by every Mor synthesis site. map_ud_sentence now returns two parallel vectors internally: Vec<Mor> and Vec<MorProvenance>. Each MorProvenance carries one ChunkProvenance per chunk of its Mor (main first, post-clitics after). Each ChunkProvenance records:

1. source_ud_ids: which UD word ids map to this chunk (one for a normal Single, N for a collapsed Range, zero for a synthesized post-clitic).
2. head: how to resolve the GRA relation’s head index (Root, FromUd(ud_id), or OwningMorMain).
3. deprel: pre-normalized relation string.

build_gra_and_validate is now language-neutral: it takes (mors, provenance) plus a TerminatorPolicy enum and emits GRA relations by walking provenance. No side-tables, no language awareness.

sequenceDiagram
    participant MUS as map_ud_sentence
    participant CI as check_italian_compound_imperative
    participant ACIO as apply_compound_imperative_override
    participant Mor as Mor::with_post_clitic
    participant Prov as ChunkProvenance
    participant BGV as build_gra_and_validate (language-neutral)
    participant P1 as Pass 1, build ud_to_chunk_idx
    participant P2 as Pass 2, emit GRA relations

    MUS->>CI: ud.text + ud.upos (for dammela)
    CI-->>MUS: Some(&override) with 2 clitics
    MUS->>ACIO: override + ud.head + ud.deprel
    ACIO->>Mor: main verb Mor
    ACIO->>Mor: with_post_clitic(me)
    ACIO->>Mor: with_post_clitic(la)
    Mor-->>ACIO: Mor{main, post_clitics: [me, la]}
    ACIO-->>MUS: multi-chunk Mor
    MUS->>Prov: 1 main chunk (source_ud_ids=[ud.id], head=FromUd, deprel=ud.deprel)
    MUS->>Prov: 1 clitic (source_ud_ids=[], head=OwningMorMain, deprel=me.deprel)
    MUS->>Prov: 1 clitic (source_ud_ids=[], head=OwningMorMain, deprel=la.deprel)
    MUS->>BGV: mors, provenance, TerminatorPolicy
    BGV->>P1: walk provenance; ci += chunks per Mor; map source_ud_ids → ci
    BGV->>P2: walk provenance; emit one relation per chunk; resolve head by ChunkHead variant
    P2-->>BGV: Vec<GrammaticalRelation>
    BGV->>BGV: validate: gras.len() == sum(count_chunks) + terminator offset

Verified against: map_ud_sentence (synthesis sites at each reconciler branch), map_ud_sentence_expanded (uniform push_ud helper), build_gra_and_validate (language-neutral in crates/batchalign-transform/src/morphosyntax/sentence_mapping.rs, re-exported through crates/batchalign/src/chat_ops/nlp/mapping/mod.rs), provenance.rs (data types), helpers.rs (normalize_deprel + assemble_mors + provenance_for_ud_word), apply_compound_imperative_override in lang_it.rs, and the test test_italian_defect8_dammela_emits_multi_chunk_mor.

Invariants checked at the end of build_gra_and_validate:

• mors.len() == provenance.len()
• For every i, mors[i].count_chunks() == provenance[i].len()
• A chunk with ChunkHead::Root was encountered
• gras.len() == sum(mor.count_chunks()) + terminator_offset

Violations surface as MappingError::ChunkCountMismatch / InvalidRoot / InvalidHeadReference. Because provenance is produced alongside the Mor at a single site, the two can’t drift , the chunk count check is a tripwire if a new synthesis site ever produces mismatched counts.

Allowlist design invariants

Three allowlists, each corresponding to a distinct reconciler mechanism:

Shared invariants across all three:

1. Closed set. Each allowlist is a hand-curated &'static []. No runtime extension, no auto-detection. Every entry corresponds to a specific Stanza output shape observed via direct probe (_cases/italian.py).
2. Gate + lookup. Each reconciler has both a POS/context gate AND a surface-text lookup. Both must match for the override to fire. Controls tests pin that legitimate (non-mis-classified) surfaces pass through unchanged.
3. Idempotent rewrite. If Stanza already got a field right (e.g., POS=VERB for Defect 10), the allowlist’s override still specifies the “correct” value, the rewrite is harmless. This lets shape-9 and shape-10 entries share IT_COMPONENT_REWRITES even though their Stanza-error signatures differ.
4. Retirement workflow. Empty all three allowlists in lang_it.rs and rerun the reconciler-dependent integration tests. Entries whose tests still fail are load-bearing; entries whose tests now pass without the reconciler are retirement candidates (Stanza fixed the defect upstream).

Reconciler for Defect 6 / 7 / 8 / 9 / 10 / 12 / 13

Implementation at crates/batchalign-transform/src/morphosyntax/lang_it.rs; plumbed into crates/batchalign-transform/src/morphosyntax/sentence_mapping.rs::map_ud_sentence.

Integration inside map_ud_sentence

The reconciler is not a standalone pass, it is two targeted branches inside the UD → CHAT mapping function, each guarded by an allowlist lookup. The diagram below shows the two hook points relative to the normal Range / Single handling:

flowchart TD
    Start["map_ud_sentence(ud_sentence, lang)"]
    PerTok{"per UD token id"}
    Range["UdId::Range(start, end)\n(Stanza emits MWT components)"]
    Single["UdId::Single(idx)"]
    ChkMis{"check_italian_mis_split?\n(IT_MIS_SPLIT_OVERRIDES)"}
    ApplyMis["apply_mis_split_override\n→ 1 MOR with corrected POS/lemma\n+ record reconciled range"]
    AssMor["assemble_mors()\n→ per-component MOR (default)"]
    ChkCmpd{"check_italian_compound_imperative?\n(IT_COMPOUND_IMPERATIVES)"}
    ApplyCmpd["apply_compound_imperative_override\n→ 1 MOR with verb POS + compound lemma\n+ record reconciled index"]
    Pass["normal Single handling"]
    Gra["build_gra_and_validate(reconciled_ranges)\n→ single %gra relation per collapsed word"]

    Start --> PerTok
    PerTok --> Range
    PerTok --> Single
    Range --> ChkMis
    ChkMis -->|"match"| ApplyMis
    ChkMis -->|"no match"| AssMor
    Single --> ChkCmpd
    ChkCmpd -->|"match + ADJ gate"| ApplyCmpd
    ChkCmpd -->|"no match"| Pass
    ApplyMis --> Gra
    AssMor --> Gra
    ApplyCmpd --> Gra
    Pass --> Gra

reconciled_ranges: Option<HashSet<(usize, usize)>> is allocated lazily, it stays None for utterances where neither allowlist fires, so the common path pays no allocation cost. When set, it is threaded into build_gra_and_validate so the GRA builder collapses per-component relations into a single relation on the rebuilt parent word.

BA3 carries a per-language reconciler hack that collapses Stanza’s known-bad Italian MWT mis-splits back to a single %mor entry with corrected POS / lemma / features. This is explicitly a hack, an allowlist of specific Stanza mis-splits we know about, not a principled morphological analyzer.

Where the hack lives. crates/batchalign-transform/src/morphosyntax/lang_it.rs (mirrors the pattern of lang_en.rs / lang_fr.rs / lang_ja.rs). The reconciler is called from crates/batchalign-transform/src/morphosyntax/sentence_mapping.rs inside map_ud_sentence’s UdId::Range branch, before the normal assemble_mors join. When it fires, the affected Range is also recorded so build_gra_and_validate emits a single %gra relation for the collapsed word rather than one per component.

The allowlist. Maintained in IT_MIS_SPLIT_OVERRIDES in lang_it.rs. Each entry:

How to extend. When a new Italian Defect 6 case surfaces in corpus data, add one row to IT_MIS_SPLIT_OVERRIDES plus a regression test in morphosyntax/tests.rs. The reconciler will fire on the new allowlist entry without further plumbing.

Defect 8 allowlist (separate hook)

Defect 8, mid-sentence compound imperatives mis-classified without MWT expansion, fires on UdId::Single, not UdId::Range, so it uses a separate allowlist IT_COMPOUND_IMPERATIVES:

All entries carry Mood=Imp|Number=Sing|Person=2|VerbForm=Fin.

Gate accepts both ADJ and NOUN : the original Defect 8 signature was ADJ-only, but -ire family probes surfaced NOUN mis-classifications (aprila tagged aprila/NOUN/aprila; aprili tagged aprili/NOUN/aprile , the latter is Stanza homographing the form onto the month name April). The allowlist is still a closed curated set; legitimate nouns pass through unchanged, pinned by test_italian_defect8_genuine_noun_stays_noun.

The prendere family was surfaced by a one-off corpus scan for CHAT main-tier words with verb+enclitic shapes; the same scan identified diglielo (already correctly handled by Stanza) and mettilo/mettila/mettili/mettiti (tagged VERB but not decomposed, a lemma-quality issue rather than a pure Defect 8 signature; deferred to a future investigation).

Scope : multi-chunk output is now emitted. Each allowlist entry specifies the post-clitic stack via CliticSpec entries, so dammela emits the full verb|dare-Imp-S2~pron|me-Prs-S1~pron|la-Prs-S3: matching Stanza’s own analysis for the bare-compound case. Previously (earlier) the reconciler emitted only the single-chunk main verb; that scope limit came from map_ud_sentence’s UdId::Single branch assuming 1 chunk per UD word. The a later refactor added a reconciled_singles side-table threaded through build_gra_and_validate so multi-chunk emission and its corresponding GRA relations stay consistent with the chunk-count invariant. See “Reconciler architecture” above for the full story.

How to extend Defect 8: same pattern, add a row to IT_COMPOUND_IMPERATIVES plus a regression test. The ADJ-POS gate limits false positives to words Stanza actively mis-classifies.

Defect 9 allowlist (Range component rewrite)

Defect 9, Range-expansion with wrong head POS, fires on UdId::Range, like Defect 6, but takes a different action: instead of collapsing the Range into a single Mor, it rewrites component 0’s POS/lemma/feats in-place and falls through to the normal assemble_mors path. The 3-chunk ~-joined Mor shape is preserved; only the head’s lexical analysis changes. Hook point lives right after the Defect 6 check in map_ud_sentence’s UdId::Range branch. Because the Range still produces multiple chunks, the entry is NOT recorded in reconciled_ranges: GRA reindexing proceeds as for a normal multi-chunk MWT.

Separate allowlist IT_COMPONENT_REWRITES:

All entries carry Mood=Imp|Number=Sing|Person=2|VerbForm=Fin as head feats. The rewrite is idempotent, if Stanza already had a field right (e.g. the POS for Defect-10 entries), re-setting it is harmless.

Scope. The allowlist is minimal and closed. Forms Stanza analyses correctly must stay off it, and control tests pin those:

• digliela, portagliela, prendigliela (Defect 9 controls) , Stanza analyses correctly; test_italian_digliela_stays_correctly_merged guards against regression.
• guardala, toccala, aspettala, mangiala, chiamala, lasciala, cambiala, provala, giocala, portala, suonala, chiudila (Defect 10 controls), all probed Stanza-correct. Only the posare paradigm mis-lemmatizes, hence the narrow allowlist.

How to extend Defect 9: add a row to IT_COMPONENT_REWRITES plus a regression test. A sibling control test should pin at least one neighboring form that Stanza handles correctly, to catch overzealous entries.

What the reconciler does NOT do.

• It does NOT touch genuine verb+clitic compounds like dammela, dammelo, portalo when they arrive via Stanza’s MWT Range (i.e., standalone, where Stanza gets them right). Those are correctly merged by Stanza; the allowlists are closed sets gated on Stanza’s mis-classification signatures.
• It does NOT change raw Stanza output, the xfail probes in _cases/italian.py continue to document what Stanza emits directly.
• It does NOT auto-detect new Defect 6 cases. Each must be observed in corpus data and explicitly added.
• It does NOT fix Italian’s Stanza model upstream, a future Stanza release that repairs the defect will render the corresponding allowlist entries redundant. Periodic re-audits (e.g., once per Stanza major version) should retire entries whose Stanza-raw output no longer mis-splits.

Constraints validated by tests at three layers:

• Unit tests in crates/batchalign-transform/src/morphosyntax/lang_it.rs , lookup semantics (case-insensitivity, positive and negative matches, exclusion of genuine compounds like dammela).
• Synthetic UD integration tests in crates/batchalign/src/chat_ops/nlp/mapping/tests/italian_defects.rs (search for test_italian_defect6_), confirm the reconciler collapses known mis-split UdSentence shapes into the correct single Mor.
• End-to-end golden in batchalign/tests/pipelines/morphosyntax/test_italian_defect6_end_to_end.py , runs batchalign3 morphotag on a CHAT fixture whose @Languages: header declares ita; the fixture contains all six allowlist words in context. Asserts no junk verb|STEM~pron|CLITIC pattern appears in the output %mor tier. (Morphotag has no --lang flag; language is per-file.) This closes the loop: real Stanza output flowing through the full production pipeline.

The specific contracts:

• Each allowlist entry produces the overridden single %mor with no ~ clitic: parla → verb|parlare, arancione → noun|arancione, piccolo → adj|piccolo, gomitolo → noun|gomitolo, divano → noun|divano.
• dammela continues to produce its correct Stanza-native merged %mor (verb|dare…~pron|me…~pron|la…), the correctness control group.
• Allowlist lookup is case-insensitive.
• Non-Italian MappingContext sees no behavior change (explicit lang2(&ctx.lang) == "it" gate).
• Unit tests in lang_it.rs cover the allowlist semantics directly (check_italian_mis_split for positive and negative inputs).

Future work

The tokenize-stage policy changed what remains open. Items from the reconciler era that the policy resolved outright: multi-chunk Defect 8 decomposition (the split now arrives as a Range and decomposes natively), per-surface allowlist expansion for Defects 6/8 (items that proposed scanning for more surfaces to list are OBSOLETE: the open verb+enclitic class is covered generally, including 2pl forms like prendetelo, which need no listing). What is genuinely left:

1. Regeneration-diff adjudication. The corpus regeneration diff is the corpus-scale verification instrument for the whole policy; any surprise it surfaces becomes the next item on this list.
2. In-context committed-damage signature scan. The language-resolved audit enumerates the single-word signature; committed in-context damage (det|il~det|il for la, verb+enclitic items on multi-word utterances) needs its own signature in lang_audit for exact enumeration. Regenerating all Italian-primary files cures it regardless.
3. Reconciler allowlist retirement. IT_MIS_SPLIT_OVERRIDES (23 entries, Defects 6/7) and IT_COMPOUND_IMPERATIVES (11 entries, Defect 8) are shadowed by the source-level policy and now no-op; removal needs per-entry verification (empty the table, run the dependent integration tests, retire entries whose tests still pass). The Defect 9/10 component rewrites (IT_COMPONENT_REWRITES) remain LOAD-BEARING and must not be retired.
4. Defect 9 breadth. The policy validates split SHAPE, not component analyses, so head mis-POS on a legitimate Range (dagliela -> ADP/da) is still repaired by a 1-entry rewrite. Whether more homograph heads mis-POS is unmeasured; a components-level audit over regenerated output would bound it.
5. Stanza-upgrade re-audit. On every Stanza upgrade: the lexicon extraction seam is pinned by test_italian_mwt_lexicon.py (a moved private attribute fails CI, not silently); the probe matrix and the Italian ml_golden tests re-verify policy behavior; the reconciler retirement workflow above re-classifies remaining entries.

Known limitations and flaws

Stated plainly, so nobody rediscovers them as surprises:

• Lexicon-gap over-splits. A form satisfying all four tests that is absent from Stanza’s lexicon is wrongly split (pentolo -> pento + lo: exact reconstruction, real clitic, attested verb base, not a dictionary entry). Invisible to every test the rule has; a real morphological analyzer is the only principled cure.
• Reflexive-imperative under-splits. Forms Stanza’s lexicon lists as words in their own right are not split (svegliati, vestiti), which is genuinely ambiguous context-free (vestiti = “get dressed!” or “clothes”). The policy deliberately under-splits when it must guess: a lost split leaves a word coarsely analyzed, a false split invents a verb.
• Lemma quality is out of scope. The policy fixes tokenization, not the lemmatizer: an unsplit unknown word can still receive a fabricated lemma (tecala -> tecalare), and a correct split can carry a wrong lemma (finila -> fini + la lemmatizes to fine, not finire).
• The lexicon is a private Stanza attribute. Deliberate trade (hand-maintained tables drift; Stanza’s own lexicon tracks its model), isolated at one loudly-failing seam and pinned by tests, but a Stanza refactor will require re-plumbing it.
• Probe cost. One extra tokenize,mwt pipeline pass per batch, on candidates only (Stanza-marked tokens plus lexical-pre-filter hits). Lazily loaded; negligible against model inference, but nonzero.
• Residual RATES are not stated. The regeneration diff is the measuring instrument; per-form rates before it would be guesses.
• Language routing is trusted, not audited. The policy assumes the words reaching the Italian pipeline are Italian-attributed. The corpus data is properly marked (language-resolved audit, 2026-07-28: zero unresolved words); whether the L2 routing layer honors every marker end-to-end has not been separately audited.

History

Rules that existed and were removed

BA2 carried two per-language hacks for Italian in ud.py:662-695, ported into BA3’s crates/batchalign-transform/src/tokenizer_realign.rs (the historical mwt_overrides.rs sub-module has since been consolidated into the single tokenizer_realign.rs file) and then emptied after a paired empirical audit:

Both rules were dead weight or actively wrong. Both were removed in favor of Stanza’s native MWT behavior plus the always-on character-DP realigner in align_tokens.

Why the DP alignment is the load-bearing piece

When Stanza’s Italian tokenizer occasionally mis-splits a CHAT word (e.g., emits [il, i, ami] for l'ami), the character-level Hirschberg alignment in align_tokens merges the N Stanza tokens back to 1 token per CHAT word based on surface-character identity. That rescue is unconditional (not per-language) and does not depend on any MWT-override table, so Italian benefits from it with zero configuration.

Tests

• Probe matrix cases: batchalign/tests/investigations/_cases/italian.py: typed ProbeCase fixtures covering l'X, preposition+clitic, lei variants, adjacent le i, and the Defect 6/7 xfails. Important: the xfails are UD-word-count observations that pin Stanza’s POS/MWT misbehavior. They do NOT indicate %mor injection failures, injection succeeds with junk content. See each case’s XfailMark.reason for the distinction.
• Matrix harness: batchalign/tests/investigations/test_stanza_mwt_probe_matrix.py runs every case through paired (free-tokenize vs postprocessor) pipelines. Invoke with uv run pytest batchalign/tests/investigations/ -m golden.

References

• Stanza Limitations, Defect 6
• Stanza Limitations, Defect 7
• Morphotag Invariants

This page last changed: 2026-09-16 (commit 197c81e6). The whole book last changed: 2026-09-16 (commit 34d249d8).