Language-Specific Processing Overview

Status: Current Last updated: 2026-09-07 07:04 EDT

This page is the single entry point for understanding how batchalign3 handles non-English languages. It maps every stage of the processing pipeline to the language-specific behavior at that stage.

Pipeline Stages and Language Divergence

Every audio file flows through the same pipeline. At each stage, the pipeline checks the language code and may take a different path:

flowchart TD
    input["Audio + lang code"]
    resolve["Model Resolution\n(lang → fine-tuned model)"]
    asr["ASR Transcription"]
    compound["Compound Merging"]
    numexp["Number Expansion\n(12 table langs + Chinese/Japanese)"]
    cantonorm["Cantonese Normalization\n(lang=yue only)"]
    rtlpunct["RTL Punctuation\nNormalization"]
    retok["Retokenization\n(punctuation-based)"]
    utseg["Utterance Segmentation\n(3 dedicated models)"]
    morpho["Morphosyntax\n(Stanza + per-lang workarounds)"]
    fa["Forced Alignment\n(Whisper/Wave2Vec/Cantonese FA)"]

    input --> resolve --> asr --> compound --> cantonorm
    cantonorm --> numexp --> rtlpunct --> retok --> utseg --> morpho --> fa

    style cantonorm fill:#f9e2b0
    style numexp fill:#d4edda
    style rtlpunct fill:#d4edda
    style morpho fill:#cce5ff
    style fa fill:#e2d5f1

Where Each Language Diverges

Stage 1: Model Resolution

The default --asr-engine whisper loads openai/whisper-large-v3 across every language (batchalign/inference/asr.py:120). UTR is engine-based: --utr-engine whisper loads openai/whisper-large-v3 at the commit the manifest pins, --utr-engine rev uses Rev.AI’s cloud API, and --utr-engine tencent routes to Tencent. There is no per-language fine-tune resolver wired into --asr-engine whisper or the UTR engines; per-language fine-tunes are opt-in through the separate --asr-engine whisper_hub engine (see Whisper Hub ASR for the seeded entries, today only mal → thennal/whisper-medium-ml).

See Language Code Resolution and Whisper ASR for the full picture.

Stage 2: Number Expansion

Digit strings in ASR output are converted to language-appropriate word forms.

The table-driven languages are enumerated in crates/batchalign-transform/data/num2lang.json (46 entries today; re-derive via python3 -c "import json; print(sorted(json.load(open('crates/batchalign-transform/data/num2lang.json'))))" rather than maintaining a parallel list here).

See Number Expansion for details on the Chinese character conversion algorithm and the table-based approach.

Stage 3: Cantonese Text Normalization (yue only)

This stage only activates when lang=yue. It applies two transformations:

1. Simplified → Traditional Chinese via ferrous-opencc (embedded OpenCC s2hk conversion tables)
2. 31-entry domain replacement table for Cantonese-specific character corrections (e.g., 系→係, 呀→啊, 中意→鍾意)

This runs in the core Rust pipeline (batchalign), not in a separate plugin package. Every ASR engine’s output benefits from it automatically.

See Cantonese Processing for the full replacement table and architecture.

Stage 4: RTL Punctuation Normalization

Arabic/Persian/Urdu punctuation is normalized to ASCII equivalents:

Additionally, Japanese full-width period (。) is normalized to ., and Spanish inverted punctuation (¿, ¡) is removed.

Stage 5: Utterance Segmentation

Three languages have dedicated BERT-based utterance segmentation models:

All other languages fall back to punctuation-based splitting (., ?, !, and CHAT-specific terminators like +..., +/.).

See Utterance Segmentation.

Stage 6: Morphosyntax (Stanza + Workarounds)

Stanza is the backbone for POS tagging, lemmatization, and dependency parsing. Language-specific workarounds correct systematic errors:

Cross-language infrastructure:

Stage 7: Forced Alignment

The Cantonese FA engine converts Chinese characters to tone-stripped jyutping romanization before alignment, because Wave2Vec MMS was trained on romanized text. See Cantonese Processing.

Language Code Flow

flowchart LR
    cli["CLI: --lang=yue"]
    iso3["ISO 639-3\n(3-letter, internal)"]
    resolve["Model resolver\n(yue → fine-tuned model)"]
    stanza["Stanza mapping\n(yue → zh)"]
    pipeline["Pipeline stages\n(yue triggers Cantonese norm)"]

    cli --> iso3
    iso3 --> resolve
    iso3 --> stanza
    iso3 --> pipeline

batchalign3 uses ISO 639-3 (3-letter codes) internally everywhere. Conversion to 2-letter codes only happens at the Stanza boundary. See Language Code Resolution.

Cantonese Normalization Is Now Core

Older Python-only code paths did not apply Cantonese normalization uniformly across every ASR path. Current batchalign3 implements simplified-to-traditional conversion plus the Cantonese replacement table once in Rust core and applies it as a shared ASR post-processing stage, so every ASR engine benefits from the same normalization contract.

Since 2026-09-16 “once” is literal: AlignedNormalization (crates/batchalign-transform/src/asr_postprocess/cantonese.rs) is the only route to normalized Cantonese text, and the server applies it a single time per monologue, before any stage splits the words. The provider bridges and the character tokenizer used to normalize as well, which mattered because the transformation is not idempotent: 聯繫 normalized twice becomes 聯係.

Related Pages

• Language Code Resolution, ISO mapping, model resolution
• Cantonese Processing, normalization, char tokenization, FA
• Hebrew Morphosyntax, HebBinyan, HebExistential
• Japanese Morphosyntax, verb forms, combined package
• Number Expansion, num2chinese, NUM2LANG tables
• Utterance Segmentation, per-language models
• Non-English Workarounds, workaround and convention catalog
• Whisper ASR, engine selection, model IDs
• Cantonese Language Support, engines, normalization, word segmentation, FA
• Cantonese and CJK, Architecture, engine dispatch, normalization pipeline, segmenter selection

This page last changed: 2026-09-16 (commit 197c81e6). The whole book last changed: 2026-09-16 (commit 34d249d8).