benchmark

Status: Current Last updated: 2026-09-07 07:04 EDT

Transcribe audio via ASR and evaluate word error rate (WER) against gold .cha transcripts in the same directory. A composite command that runs transcribe followed by compare internally.

Outputs per audio file:

1. A hypothesis .cha transcript
2. A .compare.csv with WER metrics

Quick start

# Benchmark a directory of audio files against gold .cha companions
batchalign3 benchmark input/ -o output/ --lang eng

# Use a specific ASR engine
batchalign3 benchmark input/ -o output/ --lang eng --asr-engine whisper

# Use the remote server
batchalign3 --server http://your-server:8001 benchmark input/ -o output/ --lang eng

Pipeline

flowchart TD
    start([benchmark invoked]) --> resolve[Resolve audio file + companion gold .cha]
    resolve --> transcribe[Rust transcribe workflow\nProduce hypothesis CHAT]
    transcribe --> compare[Rust compare workflow\nDP alignment + WER metrics]
    compare --> merge_check{--merge-abbrev?}
    merge_check -->|Yes| merge[Merge abbreviations in hypothesis CHAT output]
    merge_check -->|No| output
    merge --> output[Write hypothesis .cha + .compare.csv]
    output --> done([Output results])

Options

Path options

benchmark options

Gold file convention

For each audio file FILE.mp3, the gold companion must be FILE.cha in the same directory. If the gold file is missing, the audio file is reported as failed.

Pass only the audio. The gold transcript is found for you, by taking each recording’s own path and replacing the extension, so it is not an input you submit. Handing a .cha file to benchmark as a source is refused when the job is submitted, with a message naming the file:

command 'benchmark' takes media recordings as its sources, but "session.cha" is a CHAT transcript. Submit only the recording; benchmark finds each recording's gold transcript beside it by replacing the extension, so the gold must not be passed as an input.

This refusal is specific to benchmark, because benchmark is the command that derives a gold companion from each source. It does not apply to transcribe, opensmile, avqi or diarize.

This used to be accepted. Every submitted source became a recording to transcribe, so the transcript became a work unit whose “audio” was the transcript and whose gold was itself, and it was passed to ffmpeg to be decoded as a recording. Pointing benchmark at a directory is unaffected: directory expansion selects media by extension and never picks up the golds.

What gets created

• FILE.cha: hypothesis transcript produced by ASR
• FILE.compare.csv: WER metrics: aggregate row plus per-POS breakdown

The hypothesis .cha contains a main-annotated view (unlike compare, which outputs the projected reference). The %xsrep and %xsmor tiers are injected on the hypothesis utterances showing how the hypothesis deviates from the gold.

Gotchas

benchmark prefers the local daemon when auto_daemon is enabled. Use explicit --server to override.

Gold files are not passed through the network with --server. The server must be able to find the gold .cha files on its own visible filesystem alongside the audio.

Related documentation

• Benchmarks, WER metrics and evaluation methodology
• compare, standalone transcript comparison
• transcribe, ASR transcription pipeline
• Command I/O: benchmark, I/O patterns
• Command Flowcharts: benchmark, full architecture flowchart

This page last changed: 2026-09-16 (commit 197c81e6). The whole book last changed: 2026-09-16 (commit 34d249d8).