Worker Tuning

Status: Current Last updated: 2026-09-15 18:27 EDT

This page explains how the server decides how many workers to run, how memory budgets work, and how to tune the server for your hardware.

The --workers flag

Control how many files are processed in parallel:

batchalign3 --workers 1 transcribe corpus/ -o output/    # One file at a time (safest)
batchalign3 --workers 4 morphotag corpus/ -o output/     # Four files in parallel
batchalign3 transcribe corpus/ -o output/                 # Auto-tune (default)

All commands now use a two-stage policy: the runner computes a requested worker count from file count, CPU, and category caps, then the host-memory coordinator clamps that request to what the machine can safely fit right now. GPU-heavy commands (transcribe, align, benchmark) are capped by both max_gpu_workers and gpu_thread_pool_size.

CPU-only machines (Apple Silicon, no CUDA): the host-facts recommendation now sets gpu_thread_pool_size = 1 automatically when no functional GPU is detected. Leave the field absent in server.yaml (or set it to 0: the legacy “auto” sentinel that still deserializes to “no override”). The recommendation also sets force_cpu = true on the same hosts so workers skip GPU detection entirely.

PyTorch releases the GIL only during CUDA/MPS native calls; with MPS excluded for batchalign3, every Whisper inference is GIL-bound CPU work and there is no compute parallelism to gain. A higher value lets multiple execute_v2 calls into a single Python process where they fight for cores, slowing each other down by the contention factor. Verify your host’s resolved values with batchalign3 doctor --check (see Doctor).

The Rust-side dispatch_semaphore mirrors gpu_thread_pool_size permit-for-thread, so Rust dispatch and Python serving share one ceiling. Set this knob to the parallelism your device actually has: 1 on CPU-bound platforms, 2-4 on real GPU. See MPS Exclusion Decision and Worker Protocol V2 § The dispatch semaphore contract.

Measured Apple behavior on dev-machine confirms that the biggest win is not a larger CPU thread pool. Warm loopback-daemon reuse dominated everything else: align --no-utr --fa-engine wav2vec dropped from 17.65s direct / 9.64s sequential to 1.06s warm-daemon, and transcribe --asr-engine whisper dropped from 93.92s direct to 13.58s warm-daemon. On Apple CPU-only hosts, preserving warm workers matters much more than tuning gpu_thread_pool_size.

Override with --workers N when you want explicit control, or set max_workers_per_job in server.yaml for a persistent override.

How worker planning works

When you submit a job, the server decides how many parallel file workers to assign.

1. Compute a requested worker count from file count, CPU, and category caps:
   • GPU commands: min(max_gpu_workers, gpu_thread_pool_size)
   • CPU/IO commands: max_thread_workers
2. Ask the host-memory coordinator for a job execution plan.
3. The coordinator subtracts active local reservations, preserves memory_gate_mb as host headroom, and grants the largest safe worker count.
4. If nothing safe fits, the job is re-queued instead of speculatively running.

For a single file, the server always uses 1 worker, no parallelism needed.

If max_workers_per_job is set in server.yaml, it overrides auto-tuning (still capped by file count and the category max).

Why GPU commands allow parallelism: GPU-heavy commands share a single SharedGpuWorker process with a thread pool. While file N’s ASR runs on the GPU, file N+1 can do post-processing, utseg, or morphosyntax on CPU. The GPU itself serializes inference, but pipeline stages overlap. On a machine with 256 GB RAM, the coordinator may grant 4-8 parallel files for transcribe.

Worker profiles and host bootstrap mode

The server groups related commands into three worker profiles that share loaded models within a single process:

On large machines, this means running align followed by transcribe reuses the same GPU worker process, the ASR model loaded for transcription stays in memory and the FA model for alignment lives in the same process. On a 64 GB machine, this saves roughly 3 GB compared to loading each model in a separate process.

On small-memory hosts, the server now resolves a different host execution policy: local workers use task bootstrap instead of full profile bootstrap. That lets a weak laptop load only infer:asr or infer:morphosyntax instead of speculatively loading every model in a profile. The machine trades some reuse for a much lower idle footprint.

GPU workers handle multiple requests concurrently via internal threading only on CUDA-capable hosts. On CPU-only hosts they stay sequential to avoid oversubscribing OpenMP threads. Stanza and IO workers handle one request at a time but can run multiple processes in parallel for CPU-bound workloads.

Per-command memory profiles

Each command loads different ML models with different memory footprints. These values come from runtime_constants.toml (generated from crates/batchalign-types/src/command_spec.rs via xtask gen-runtime-toml; shared between Rust and Python at compile/import time):

These are the thread worker values (shared-model mode). Process worker values are higher because each worker loads its own copy of the models.

Commands in the same profile share a worker process, so the total memory for a mixed job (e.g., align + transcribe) is roughly the sum of their models loaded once, not separately. The GPU profile typically uses ~5 GB total for all its models (ASR + FA + Speaker), regardless of how many commands run.

Worker pre-spawning

There is no startup warmup. Workers are created on demand, by two paths:

Per-job pre-scaling. When a job runs more than one file concurrently, the runner pre-spawns its workers before file dispatch begins, so a batch pays one cold start rather than one per file. The pre-spawned worker is keyed on the job’s own engine selection, so the dispatches that follow reuse it rather than spawning a second process.

Registry adoption. A TCP worker daemon started outside the server registers itself in workers.json; the server adopts it at startup and routes to it like any other worker. Each entry records the build that started the daemon (build_identity, from BATCHALIGN_BUILD_IDENTITY, which batchalign3 worker start and the server’s own daemon spawner set). A server refuses to adopt a daemon from another build, or one whose entry names no build, and logs the remedy: stop the daemon (batchalign3 worker stop) and start it again with the current build. The refused daemon is left running and listed in /health under refused_registry_workers, with its worker key, pid and reason (foreign_build or unreported_build).

Idle workers are then reclaimed by memory-pressure eviction (largest resident set first) rather than by a fixed timeout.

Retired: a --warmup flag and a warmup_commands config key used to pre-load a list of commands’ models when the server started. They were disabled on every real server from 2026-03-26 (following the 2026-03-11 finding that warmed models stay resident for the server’s whole lifetime) and removed on 2026-07-30. Passing --warmup is now an unrecognised-argument error, and warmup_commands in a server.yaml is rejected as an unknown field; delete the key. Nothing else changes: neither had any effect on a real server for the four months before removal.

On-demand loading

No workers are pre-loaded at startup; they spawn lazily on the first job that needs them. Lazy startup does not mean the first real command is allowed to run against unknown infer-task metadata. The current server resolves that by forcing a live capability probe from the worker it is actually about to use. In practice:

• /health may still show an optimistic command surface immediately after boot
• the first real job pays the worker startup cost and records the detected infer-task view
• later jobs reuse that detected capability view instead of the cold-start placeholder
• align is advertised whenever a worker supports forced alignment, even before its FA model has loaded; an align job loads the model first and fails only if the worker still names no FA engine after that
• a worker whose capability report was refused is never used, and is listed with the reason in /health under worker_capability_admissions

server.yaml reference

Key tuning parameters:

# Worker parallelism
max_workers_per_job: 0          # 0 = auto-plan from files, CPU, and category caps
max_concurrent_jobs: 0          # 0 = CPU-based runner slot cap
gpu_thread_pool_size: 4         # Concurrent execute_v2 per shared GPU worker.
                                 # Rust dispatch_semaphore + Python ThreadPoolExecutor
                                 # share this ceiling: set to the device's real
                                 # parallelism (1 on Apple Silicon CPU-only;
                                 # 2-4 on CUDA where the GIL is released).
max_concurrent_worker_startups: 1

# Memory tier override: force a specific tier instead of auto-detecting
# from total RAM. Values: small, medium, large, fleet. Omit to auto-detect.
# memory_tier: small

# Host-memory reserve/headroom (MB) preserved after reservations
# 0 = disable explicit reserve. Default: 2048 MB (one absolute floor on
# every host; the worker-pool admission gate enforces the same number
# live on every spawn attempt, see book/src/batchalign/developer/memory-safety.md)
memory_gate_mb: 4000

# Per-profile startup reservation overrides (MB). 0 = use tier default.
# These control how much memory the coordinator reserves while a worker
# loads its models. Reduce on small machines if the tier defaults are
# too conservative for your actual model sizes.
# gpu_startup_mb: 6000
# stanza_startup_mb: 3000
# io_startup_mb: 2000

# Worker lifecycle
# Idle workers are evicted by host memory pressure (largest-RSS first
# when available memory drops below the eviction threshold); there is
# no fixed idle timeout. Health checks run every `worker_health_interval_s`.
worker_health_interval_s: 30    # Health check frequency

Scenarios

16 GB laptop / shared developer machine

# The Small tier (<24 GB) auto-detects these defaults, so this config
# is only needed if you want to further customize on a small machine.
memory_tier: small              # Force small tier (auto-detected on <24 GB)
max_workers_per_job: 1
memory_gate_mb: 2000            # Small tier default
stanza_startup_mb: 3000         # Actual Stanza RSS is ~2-3 GB
gpu_startup_mb: 6000            # Whisper float32 is ~4-5 GB
max_concurrent_worker_startups: 1
gpu_thread_pool_size: 1

The Small tier now also switches local workers to task bootstrap and clamps eligible file-parallel commands to one file at a time. That keeps the execution shape honest for 16 GB laptops: no speculative profile preload, no multi-file GPU stampede, and no assumption that the machine can afford idle models it is not about to use.

32 GB desktop

Default settings usually work well. The coordinator will clamp jobs as host pressure changes.

256 GB server (production)

max_workers_per_job: 0          # Coordinator-backed auto planning
max_concurrent_jobs: 8
max_concurrent_worker_startups: 1
memory_gate_mb: 8000            # Operator override; the default is 2048 MB (MIN_FREE_MEMORY_MB)

With this much RAM, worker profiles let the server run multiple concurrent jobs efficiently. A GPU worker handling an align job and a Stanza worker handling a morphotag job run in parallel without duplicating models, leaving plenty of headroom for additional jobs.

Testing with test-echo workers

For quick iteration during development:

batchalign3 serve start --foreground --test-echo

Workers start instantly (no ML models loaded). Useful for testing server infrastructure without waiting for model initialization.

Troubleshooting

“Job deferred due to memory pressure”

The host-memory coordinator could not fit the requested execution plan. Possible causes:

1. Too many concurrent workers. Reduce max_workers_per_job or gpu_thread_pool_size.
2. Other processes using RAM. Check system memory usage.
3. Idle workers holding memory. Workers that haven’t been used in a while still hold their loaded models. The pool’s pressure-driven eviction releases idle workers automatically when host available memory drops below the eviction threshold; if pressure is genuine but eviction isn’t firing fast enough, restart the server to reclaim immediately.
4. Another local batchalign3 server or test run is already holding leases. Check /health for host_memory_* fields.

Jobs are re-queued when the plan does not fit. /health now reports host_memory_pressure, current reservations, and active lease labels.

Only 1 worker running

The coordinator decided that only 1 worker currently fits. Check:

• /health host_memory_pressure and host_memory_reserved_mb
• memory_gate_mb
• gpu_thread_pool_size for GPU commands
• other local batchalign3 servers or ML tools on the same host

Override with max_workers_per_job if you know your system can handle more.

The first job takes too long

The first job of a given kind loads its ML models from disk, or downloads them on first run. To speed up:

• The first run after installation is slowest (model downloads)
• Subsequent starts load from the model cache (~5-20 seconds per model)
• Keep the daemon running (batchalign3 serve start) to avoid repeated cold starts

See also Performance and Server Mode.

This page last changed: 2026-09-16 (commit 197c81e6). The whole book last changed: 2026-09-16 (commit 34d249d8).