servatus

Servatus is a zero-dependency Python library and CLI for resumable application work. Campaign keeps an append-only roster of opaque tasks and its durable attempt history, submits reviewed plans to Slurm through native OpenSSH and Apptainer, and observes their scheduler state. Workspace retains private checkpoints and atomically publishes application-validated outputs.

Resilience is concentrated at submission, observation, and POSIX publication. Durable intents and receipts prevent silent duplicate submission, retries require explicit operator choices, and no-replace commits prevent concurrent corruption or overwrite. Applications retain task meaning, result schemas, validation, and completion decisions behind an opaque result probe.

Named profiles in SERVATUS.toml resolve a Slurm target and one homogeneous resource request. Planning collects scheduler and result evidence, packs eligible tasks into balanced single-node allocations with exact CPU, memory, and GPU totals, and binds the decision to a campaign revision and integrity digest; submission records intent before contacting Slurm and receipts after acceptance.

Inspection batches bounded squeue and sacct queries per original route, matches immutable allocation identities, and reports result readiness separately from quiescence. Workspace binds hidden state to an opaque identity, and drafts, file stages, and child workspaces commit through kernel no-replace renames that expose one absent-or-complete destination.