State Persistence¶

Where does my task graph go between orchestrator runs?

Everything Bernstein needs to resume after a crash lives in the project's .sdd/ directory. Task specs are YAML files, every state transition is appended to a hash-chained write-ahead log before being applied, and large artifacts are stored content-addressed. On restart the orchestrator reads the WAL, replays uncommitted entries through an idempotency filter, and resumes the tick loop where it left off.

If you only have time for one sentence: kill bernstein run, restart it in the same workdir, and your task graph comes back. The rest of this document explains why.

The .sdd/ directory layout¶

Bernstein writes everything under <workdir>/.sdd/. The split between durable (commit-worthy and crash-recoverable) and ephemeral (regenerated each run) is intentional:

Rule of thumb: anything under .sdd/runtime/ other than wal/ is disposable. Anything under .sdd/backlog/, .sdd/runtime/wal/, .sdd/metrics/, .sdd/cas/ or .sdd/audit/ is needed to reconstruct state.

The repo's .gitignore excludes the volatile pieces; the durable pieces can be checked in or shipped to remote storage.

WAL invariants¶

The write-ahead log is a hash-chained append-only JSONL file. There is one file per orchestrator run:

.sdd/runtime/wal/<run-id>.wal.jsonl

Every line is a JSON object representing one orchestrator decision. WALEntry fields:

@dataclass(frozen=True)
class WALEntry:
    seq: int  # 0-based monotonic sequence
    prev_hash: str  # SHA-256 of previous entry; first is GENESIS_HASH
    entry_hash: str  # SHA-256 over (seq, prev_hash, timestamp, ...)
    timestamp: float  # Unix seconds
    decision_type: str  # e.g. "task_spawn_intent", "task_spawn_confirmed"
    inputs: dict[str, Any]  # what we decided on
    output: dict[str, Any]  # the resulting action's primary key(s)
    actor: str  # which orchestrator component wrote it
    committed: bool = True  # False = pre-execution intent

Source: src/bernstein/core/persistence/wal.py.

Three invariants make the WAL load-bearing for recovery:

1. Append-only. WALWriter.append() only ever calls `f.write(...)
2. f.flush() + os.fsync(...)(wal.py). No mutation, no truncation. A torn write that leaves a partial trailing line is tolerated by the reader (wal.py) and by the tail-reader (wal.py`).
3. fsync per entry. Every successful append() returns only after the line is on stable storage (wal.py). A process crash immediately after append() returns cannot lose the entry.
4. Hash chain integrity. prev_hash of entry N+1 equals entry_hash of entry N. WALReader.verify_chain() walks the file and reports any break (wal.py). A tamper, a torn write, or an out-of-order append is detectable.

The pattern for a state transition that has external side effects (e.g. spawning an agent process) is:

WALWriter.append(decision_type="task_spawn_intent", committed=False)
   ↓ fsync returns
spawner.spawn(task)         ← real side effect
   ↓
WALWriter.append(decision_type="task_spawn_confirmed", committed=True)
   ↓
WALWriter.mark_committed(intent_seq)  # remove from sidecar index

If the process crashes between the intent and the confirm, the intent remains in the file with committed=False and the sidecar index points to it. Recovery picks up there.

The sidecar index (UncommittedIndex) is what lets recovery decide which WALs to open at all. A run it does not name holds no uncommitted entry, so that run's file is never read: 200 runs of 500 entries stop being 100 000 JSON parses per boot.

Recovery therefore has to be able to trust it, and the rule is that the index is either absent or right. Three paths keep that true (wal.py):

• Absent or unparseable. Recovery walks every WAL and rebuilds the index from what it found. One slow boot, never a wrong answer.
• Present and parseable, including empty. Taken as authoritative. Empty means "nothing is uncommitted", not "I do not know".
• A row that could not be written. WALWriter.append records an uncommitted entry in the index after the WAL line is durable. If that write fails, the append still succeeds - the entry is on disk - but the index is now well-formed and short, which is the one state a reader must never trust. The writer deletes it, turning the case back into "absent". If the file cannot be deleted either, that is logged at error, because recovery may then skip the run until the index is removed by hand.

The index is created by seeding it from one scan, not by writing an empty file, so a project whose WALs predate the index does not get told there is nothing to recover.

Loss of the index still never costs correctness, only one slow boot.

Recovery on restart¶

When bernstein run starts in a workdir that already contains .sdd/, the recovery sequence is:

1. Load durable backlog
   read .sdd/backlog/{open,claimed,closed}/*.yaml
   reconstruct in-memory task store

2. WALReplayEngine.scan_and_replay()
   a. WALRecovery.scan_all_uncommitted(sdd_dir, exclude_run_id=current)
      → list[(run_id, WALEntry)] where committed=False
   b. For each entry:
        if entry.decision_type in _SKIP_DECISION_TYPES → mark informational
        if (now - entry.timestamp) > MAX_REPLAY_AGE_S (1 h) → mark stale
        if IdempotencyStore.was_executed(key) → skip
        else                                     → replay_handler(entry)
   c. Append "wal_replay_completed" entry to current run's WAL

3. recover_stale_claimed_tasks()
   any task left in CLAIMED state by the dead orchestrator is reset to
   OPEN so a new agent can pick it up
   (core/tasks/task_store_core.py)

4. Begin tick loop

Source files: src/bernstein/core/persistence/wal_replay.py, src/bernstein/core/tasks/task_store_core.py.

The idempotency store is a JSONL log at .sdd/runtime/wal/idempotency.jsonl mapping (decision_type, entry_hash) → executed. It survives crashes and is consulted before every replay so the same intent is never executed twice (wal_replay.py).

The 1-hour age cap (_MAX_REPLAY_AGE_S in wal_replay.py) means WAL entries older than an hour are marked stale and skipped; the operator must re-trigger the action manually if it is still wanted. This prevents accidental replays of work the operator already abandoned.

CAS store and Merkle integrity¶

Content-addressed artifacts¶

CASStore (src/bernstein/core/persistence/cas_store.py) deduplicates arbitrary bytes by SHA-256 digest. Two agents that emit the same patch or the same screenshot store one blob, not two. Layout mirrors git's object store:

.sdd/cas/{first-2-hex-chars}/{full-sha256-hex}
.sdd/cas/{first-2-hex-chars}/{full-sha256-hex}.meta.json

API (cas_store.py):

store = CASStore(Path(".sdd/cas"))
digest = store.put(b"hello world", content_type="text/plain")
assert store.get(digest) == b"hello world"

Each blob has a .meta.json sidecar containing a CASEntry (cas_store.py) with size, content type, creation timestamp, and arbitrary user metadata. CASStats tracks how many put() calls hit an existing blob (dedup_saves).

Merkle seals over audit logs¶

merkle.py builds a binary Merkle tree over daily HMAC-chained audit log files. Each file's last-line HMAC becomes a leaf; the root hash is written to .sdd/audit/merkle/seal-<ISO-timestamp>.json and proves no file was deleted, inserted, reordered, or tampered with between seals (src/bernstein/core/persistence/merkle.py).

This is independent of the WAL hash chain - the WAL protects orchestrator decisions, the Merkle seal protects compliance audit evidence (auth events, policy denials, identity revocations).

Persistence boundary¶

Durable (survive a crash, can be backed up):

• All YAML in .sdd/backlog/{open,claimed,closed,issues}/
• .sdd/runtime/wal/*.wal.jsonl and idempotency.jsonl
• .sdd/metrics/*.jsonl (cost ledger, cascade chain reports, file health, custom metrics)
• .sdd/cas/** (artifact blobs and .meta.json sidecars)
• .sdd/audit/** (HMAC-chained audit logs and Merkle seals)
• .sdd/config/** (MCP server catalog, model policy, etc.)

Ephemeral (regenerated by the next run):

• .sdd/runtime/ PID files, sockets, agent signal queues
• .sdd/runtime/logs/ per-session log fragments
• .sdd/worktrees/ per-agent git worktrees
• .sdd/runtime/wal/uncommitted.idx.json (sidecar - rebuildable)

Never commit .sdd/runtime/ or .sdd/worktrees/. The shipped .gitignore excludes both.

Code pointers¶

Run-time helpers:

# verify WAL integrity
bernstein doctor

# back up the durable parts (used by `bernstein dr`)
bernstein dr backup --to ./backup.tar.gz

# inspect the replay log
jq -r 'select(.decision_type=="wal_replay_completed") | .inputs' \
  .sdd/runtime/wal/<run-id>.wal.jsonl

Related¶

• architecture/LIFECYCLE.md - task and agent state machines that drive the WAL entries.
• architecture/storage.md - how .sdd/ writes can be redirected to S3/GCS/R2 instead of the local filesystem.
• operations/disaster-recovery.md - the bernstein dr CLI.