Operations

This runbook describes how to operate a Pantheon Blueprint deployment after its first successful installation. It is intentionally conservative: recovery evidence matters more than a green dashboard, and an observed restart is not the same as self-healing.

Values such as <PRIVATE_CONFIG_REPO>, <LOCAL_TIMEZONE>, <BACKUP_BUCKET>, and <VERSION> are placeholders. Never commit live secrets, private addresses, account identifiers, or environment-specific credentials to the public repository.

Operating principles

Git records desired state; running containers are not the source of truth.
The public Pantheon Blueprint repository supplies architecture, sanitized examples, and human and agent setup guidance only; it is not a live deployment source.
A private desired-state repository records the production deployment's inventory, version pins, service configuration, and secret references.
Komodo applies the reviewed private desired state. Reconcile production drift in that repository, not by editing configuration in the Komodo interface.
Every stateful change has a pre-change backup and a tested rollback path.
Backups are append-oriented and recoverable without the live application.
Backup creation credentials cannot delete prior backups.
Agents may request an update, but they do not receive a host shell, Docker socket, Komodo administrator credential, or 1Password vault browser.
Monitoring data is diagnostic evidence, not an authorization decision or authoritative action audit.
A self-healing claim is valid only for a named failure, version set, and successful test.
Assistant-led bug fixes require an explicit, scoped, expiring maintenance session; diagnosis alone grants no mutation authority.

Desired-state ownership

Public reference repository

The public repository should contain:

architecture, security, data-flow, installation, and operations documents;
sanitized Compose and service examples when they are stable enough to share;
placeholder environment files containing no real values;
human and agent setup and validation guidance;
example policies; and
contribution and disclosure guidance.

It must not contain:

live DNS names, addresses, email accounts, phone numbers, or user IDs;
1Password item or vault details that reveal private organization structure;
tokens, passwords, private keys, OAuth secrets, recovery material, or rendered secret values;
internal firewall rules copied without sanitization;
backup bucket names or provider account identifiers; or
logs, database samples, conversation content, or tool arguments from the private deployment.

Private desired-state repository

The private repository managed by Komodo is the desired-state authority for one production deployment. It should contain:

inventory/
  hosts and roles
  public and private service names
  network intent
versions/
  pinned tags
  resolved image digests
  compatibility notes
stacks/
  deployment-specific Compose overlays
  Traefik labels
  health checks
config/
  non-secret application configuration
  1Password secret references
  Grafana Alloy configuration
policies/
  Heimdall catalogues and approval classes
  Tailscale and ingress intent
runbooks/
  deployment-specific recovery notes
  exceptions and validation records
evidence/
  redacted test results
  restore drill summaries

The private repository still must not contain decrypted secrets. A private Git repository is not a secret manager.

Prevent configuration drift

Komodo stacks are created from the private desired-state repository. Review and commit every production configuration change there, then use Komodo to apply that state. Do not edit production configuration in the Komodo interface or treat its recorded stack definition as an alternate configuration authority. If production differs from the reviewed state, investigate the difference and reconcile it through a reviewed change in the private repository.

At least nightly, compare:

committed Compose and rendered non-secret configuration;
Komodo's recorded stack definition;
running image digests;
enabled routes;
mounted paths;
container privileges and socket mounts; and
current workload identities.

Alert on drift. Do not automatically overwrite unexplained drift until an operator has determined whether it is an intrusion, emergency fix, or failed deployment.

Boot and service ownership

Komodo Core runs outside the three Pantheon Blueprint hosts. Each host runs Komodo Periphery, preferably as the system service described by Komodo's server connection documentation.

Host boot order

The intended order on agent-01, knowledge-01, and tools-01 is:

local filesystems and persistent volumes;
network and time synchronization;
Tailscale;
Docker Engine;
Komodo Periphery;
Grafana Alloy;
application containers; and
post-boot validation.

If Traefik is deployed as a container, it returns with the application containers. If it is a host service, define its dependency on network readiness and its configuration files explicitly.

Periphery under systemd

Use the current upstream installer or unit definition for the pinned Komodo release. Do not copy a historical unit file without comparing it with the installed version.

The unit should provide the equivalent of:

startup after network readiness;
automatic restart after an unexpected Periphery exit;
bounded restart backoff;
a root or dedicated service identity appropriate to its documented host operations;
a configuration file outside the repository working tree;
logs available to Alloy; and
no embedded Core credential in the unit file itself.

Command-shaped verification pseudocode:

service-manager verify periphery-unit
service-manager enable periphery-unit
service-manager start periphery-unit
service-manager status periphery-unit
komodo-core verify-server <HOST_ROLE>

Use the real service-manager and Komodo commands documented for the installed releases.

Boot validation

After every operating-system or deployment-plane change:

Reboot one host at a time.
Confirm Tailscale returns with the expected tagged identity.
Confirm Docker returns.
Confirm Periphery reconnects to the external Komodo Core.
Confirm required containers return.
Confirm dependencies become healthy in the expected order.
Run a harmless application transaction.
Confirm Alloy resumes export without exposing startup secrets.
Record recovery time and any manual step.

Do not call the deployment self-starting until this test has passed on all three hosts.

Health checks, restarts, and reconciliation

Docker documents container restart policies and Compose health checks. They solve different problems.

What each mechanism does

Mechanism	Detects or handles	Does not prove
Process exit	Main container process stopped	Application correctness or dependency health
Restart policy	Restarts a stopped container under defined conditions	That an unhealthy but running process will restart
Health check	Runs a bounded readiness/liveness test	Automatic recovery from an unhealthy result
Compose dependency health	Can delay a dependent service until a dependency is healthy	Continuous reconciliation after startup
Komodo deployment	Applies a requested stack definition	Continuous Kubernetes-style reconciliation unless explicitly configured and tested
Grafana alert	Tells an operator or automation that a condition exists	That remediation happened or was safe
Recovery controller	Performs a defined remediation	Recovery from failures outside its tested scope

A container can be running and unusable. A container can also be unhealthy because a check is wrong while the service remains usable.

Health-check requirements

Every long-running service should have a check that:

has a strict timeout;
does not require a broad credential;
tests a meaningful local dependency;
does not create or mutate user data;
distinguishes startup delay from ongoing failure;
cannot hang indefinitely;
has a documented expected response; and
emits enough context to diagnose a failure without exposing secrets.

For stateful services, use separate checks for process liveness and functional readiness where the upstream application supports them.

Restart policy

Use a restart policy appropriate to a daemon, but avoid infinite rapid restart loops. Add backoff at the service manager, application, or remediation layer where supported.

Do not use blind restart as the only response to:

schema migration failure;
disk exhaustion;
corrupt database state;
authentication or certificate failure;
repeated connector denial;
unexpected configuration drift; or
suspected credential compromise.

Those conditions require diagnosis or a purpose-built recovery action.

True reconciliation

If automatic remediation is added, define it as a finite state machine:

observe symptom
confirm symptom over a bounded interval
collect redacted evidence
check maintenance and deployment locks
run one approved remediation
verify the user-level transaction
stop or escalate after the retry budget

The remediation identity should have only the permission required for that action. For example, restarting one named Compose service does not justify general Docker administration from an agent.

Test each claimed recovery separately:

Failure	Example expected recovery
Container process exits	Docker restart policy returns it
Host reboots	Docker, Periphery, and pinned stacks return
Dependency starts slowly	Readiness-aware startup prevents false success
Temporary external outage	Bounded retry, then alert; no duplicate write
Failed deployment	Komodo reports failure and operator rolls back
Full disk	Alert and controlled capacity response; not restart loop
Database corruption	Stop writes, restore or repair under incident procedure
Heimdall unavailable	Agents fail closed without direct connector fallback

Record the exact test, versions, expected recovery time, result, and last test date. “Docker restarts it” is not a general self-healing strategy.

Release discovery and controlled promotion

Nightly automation discovers releases; it does not install them automatically.

Example local schedule

Use an explicit IANA timezone, represented here by <LOCAL_TIMEZONE>. Do not rely on an undocumented container default or UTC conversion.

00:30 local  verify backup service and available capacity
01:00 local  discover new pinned-component releases
01:15 local  create or update one change proposal per component
02:00 local  run read-only drift and certificate checks
03:00 local  run index freshness and checkpoint checks
weekly       run extended dependency and restore-readiness checks
monthly      perform an isolated restore drill

Stagger expensive jobs. Muninn review schedules, n8n collections, database maintenance, backups, and image pulls should not all start on the hour.

Discovery job

The discovery job may:

read the current version inventory;
query official release feeds or registries;
resolve tags to immutable digests;
collect release notes and migration notices;
identify known compatibility constraints;
create a reviewable proposal; and
notify an operator.

It must not:

rewrite production pins;
run a migration;
pull and activate arbitrary branch content;
delete a rollback image;
change a secret; or
grant itself deployment credentials.

Generic pseudocode:

for component in pinned_inventory:
    candidate = official_source.latest_supported_release(component)
    if candidate != component.current:
        proposal = compare(component.current, candidate)
        record(proposal, immutable_digest, release_notes, migrations)
        notify_operator(proposal_id)

Ody-requested updates

The user can ask Ody to update the system, but Ody remains an orchestrator.

sequenceDiagram
    participant U as User
    participant O as Ody
    participant H as Heimdall
    participant R as Private desired-state workflow
    participant K as Komodo
    participant V as Validation

    U->>O: Update component or system
    O->>H: Request update proposal
    H->>R: Read pins and official release metadata
    R-->>O: Candidate, changes, risk, backup and rollback plan
    O-->>U: Present bounded approval
    U->>O: Approve proposal ID
    O->>H: Submit approved proposal ID
    H->>R: Create reviewed desired-state change
    R->>K: Deploy approved pinned revision
    K->>V: Run smoke tests
    V-->>O: Success or rollback result
    O-->>U: Report outcome

Ody receives neither a shell nor the Docker socket. Its Heimdall catalogue may contain semantic actions such as:

check_for_updates;
prepare_update_proposal;
request_update_approval;
deploy_approved_revision; and
read_deployment_status.

Each action accepts bounded identifiers, not arbitrary commands. The deployment identity verifies that the approved commit, version, digest, and target match before asking Komodo to deploy.

Updates and bug fixes are related but distinct. A bug fix may prepare a branch, tests, and pull request only inside an owner-approved maintenance session. Merge and deployment remain separate one-use approvals bound to reviewed commits, pinned artifacts, health checks, and rollback evidence. See Scoped maintenance sessions.

Promotion gates

Before promotion:

release source is official;
version and digest are recorded;
release notes and migrations are reviewed;
compatibility with coupled components is checked;
an immutable pre-change backup exists;
the restore path is available;
disk capacity is sufficient for old and new images;
the current version remains available as a rollback target; and
the maintenance window and notification route are active.

Staged deployment

Deploy one failure domain at a time.

Recommended order for ordinary updates:

observability collectors and non-authoritative tooling;
tools-plane components that do not change connector semantics;
Mem0 and rebuildable indexing components;
AFFiNE supporting services, migration, and application as one release-matched procedure;
Hermes/Muninn workers while Ody remains available where possible;
user-facing Ody runtime and interfaces; and
connector permissions or write enablement last.

Change the order when an upstream migration guide requires it. Preserve compatibility at every step.

Deployment procedure

Announce or record the maintenance state.
Pause affected schedules and write-producing workflows.
Confirm the latest backup and upload completion marker.
Confirm no previous deployment or restore is running.
Pull the exact images and verify resolved digests.
Render the Compose configuration without printing secret values.
Run upstream preflight and migration checks.
Deploy the smallest affected stack through Komodo.
Wait for process, health, and application transaction checks.
Check logs, metrics, audit correlation, routes, and identities.
Resume schedules gradually.
Record the result and close the maintenance state.

Rollback

Rollback is a planned deployment, not an improvised reverse edit.

Before every promotion, record:

previous desired-state commit;
previous image digests;
previous configuration schema;
whether the database migration is backward compatible;
backup identifier and checksum;
restore procedure;
maximum tolerable data loss; and
the decision point beyond which restore is required instead of image rollback.

If a migration is not backward compatible, rolling the image back without restoring the database may make the outage worse. Follow the upstream migration guide.

Generic rollback decision:

if no_persistent_schema_change:
    deploy(previous_desired_state)
    verify()
else:
    stop_writes()
    restore(pre_change_backup)
    deploy(previous_desired_state)
    verify()

After rollback, keep the failed release evidence. Do not delete it as part of the rollback workflow.

Backup architecture

The backup system is independent of Grafana Cloud, Komodo, and the live application databases. It targets S3-compatible object storage in a separate failure and administrative domain.

Required coverage

Priority	Component	Required backup content
1	AFFiNE	Database, blob/upload storage, release/config metadata, consistency manifest
2	Hermes/Ody	Conversation/session state, approved memory/state, skills, profile configuration, identity references
3	Private desired state	Git history, version pins, non-secret configuration, policies, recovery notes
4	Heimdall	Tool catalogue, policy, connector-to-identity mapping, approval/audit data not held elsewhere
5	n8n/Huginn	Database, workflow definitions, encryption key reference, capture manifests, binary data if retained
6	Muninn	Checkpoints, candidate ledger, provenance, schedules, profile configuration
7	Source archive	Completed conversation exports and accepted external captures required for provenance
8	Mem0	Configuration and schema; data backup optional because the index must be rebuildable
9	Host edge/config	Traefik, Alloy, system units, firewall intent, Periphery configuration references

1Password remains its own secret system. Do not export vault contents into the ordinary Pantheon Blueprint backup merely to make restore simpler. Back up the references, required vault/account recovery procedure, and independent emergency access material according to 1Password's guidance and your organization policy.

Application-consistent backup order

For a coordinated backup:

allocate a globally unique backup ID;
record versions, desired-state commit, and start time;
pause or checkpoint affected writers;
create application-consistent database dumps or snapshots;
capture the matching blob, upload, binary, and source-archive state;
capture non-secret configuration and identity references;
calculate checksums and sizes locally;
upload each object under the unique backup prefix;
verify object metadata and checksums remotely;
upload the signed or checksummed manifest last as the completion marker;
resume writers; and
emit a redacted success or failure event.

If the application cannot be quiesced, use its documented online backup mechanism and record the consistency boundary. Copying a live database data directory is not automatically a valid backup.

Append-oriented object keys

Never reuse a key for a new backup. A suggested shape is:

<DEPLOYMENT_ID>/<COMPONENT>/<YYYY>/<MM>/<DD>/<BACKUP_ID>/
  manifest.pending.json
  database.dump
  blobs.archive.part-0001
  configuration.archive
  checksums.txt
  manifest.complete.json

<BACKUP_ID> should be collision-resistant and include time plus a random or monotonic component. A backup is complete only when manifest.complete.json exists and every referenced object passes verification.

Do not update a latest object as the only way to find backups. A generated catalog may point to immutable backup IDs, but it is convenience data and can be rebuilt by listing manifests.

Versioning, Object Lock, and WORM

Amazon documents that S3 Object Lock uses a write-once-read-many model and requires versioning. S3-compatible providers may implement different subsets or semantics.

Before relying on a provider:

confirm versioning behavior;
confirm whether Object Lock must be enabled at bucket creation;
test governance and compliance retention behavior;
test how simple deletes, version-specific deletes, and delete markers work;
verify which credentials can bypass governance retention;
verify legal hold support if required;
test restore tooling against versioned and locked objects; and
record the provider-specific evidence.

Versioning alone does not prevent an identity with version-deletion permission from removing history. Object Lock is not useful if the backup writer can bypass it or disable the protection. “S3 compatible” is not evidence of WORM semantics.

No automated deletion

The default Pantheon Blueprint retention rule is:

The system may create new backup objects, but it does not decide to delete old backup objects.

Therefore:

do not configure lifecycle expiration in the initial deployment;
do not grant the backup writer DeleteObject, version-deletion, retention bypass, or bucket-administration permissions;
do not allow Ody, Muninn, Huginn, Heimdall connectors, Komodo, or routine maintenance jobs to delete backups;
do not prune prior backup prefixes after a successful upload; and
treat any future retention policy as a separate, human-approved governance project.

Storage growth must be monitored. “Never delete automatically” transfers the problem from retention automation to capacity planning; it does not eliminate it.

Separate credentials

Use at least three identities:

Identity	Permission
Backup writer	Create objects under new prefixes, list only what verification requires; no delete or retention bypass
Restore reader	Read selected backup objects; normally disabled or stored separately
Bucket administrator	Configure versioning, retention, and access; no routine application use

If possible, use a fourth monitoring identity that can list metadata and retention status but cannot read backup contents.

Store these credentials in separate 1Password items and scope service accounts to the minimum items. Compromising the live backup writer must not grant backup deletion or unrestricted restore access.

Restore drills

A backup is unproven until it has been restored.

Schedule

monthly: restore one recent complete backup into an isolated environment;
quarterly: restore the full critical path, including identity and routing substitutions;
after a schema migration: restore both the pre-change and post-change backup;
after backup-tool or storage-provider changes: run an immediate drill; and
annually: perform a documented loss-of-site exercise.

Isolated restore procedure

Create a clean, isolated network and fresh target volumes.
Select a backup by immutable backup ID, not by a mutable latest pointer.
Verify the completion manifest, object inventory, retention metadata, and checksums.
Obtain restore credentials through the emergency or approved path.
Restore configuration without production secrets.
Restore databases and matching blob/binary state.
start the pinned application versions recorded in the manifest;
substitute test DNS, OIDC, email, messaging, and external connector targets;
run application-level integrity checks;
rebuild Mem0 from restored AFFiNE content;
run canonical retrieval and provenance tests;
record duration, manual steps, missing dependencies, and result; and
destroy the isolated environment through a separately approved cleanup process.

Never let a restore drill send real email or Signal messages, execute production tools, invoke production webhooks, or overwrite live DNS.

Recovery objectives

Define for each component:

recovery point objective (maximum acceptable data loss);
recovery time objective (maximum acceptable outage);
maximum tolerable outage;
restore dependency;
responsible operator; and
last successful drill.

An RPO shorter than the backup interval requires replication, journal shipping, or more frequent backups. Writing a smaller number in a runbook does not create that capability.

Monitoring and alerting

Grafana Alloy runs on each host and exports redacted metrics, logs, and traces to Grafana Cloud. Start with Grafana's Alloy installation and Docker monitoring documentation, then apply Pantheon Blueprint's data-minimization policy.

Required signals

Hosts

reachability and last telemetry time;
CPU, load, memory, swap, disk, inode, and filesystem error state;
clock synchronization;
operating-system reboot required;
Docker and Periphery service state; and
Tailscale connection state.

Containers and services

desired versus running container count;
restart count and restart-loop detection;
health-check state and age;
image digest drift;
request rate, latency, and error rate;
database connections, size, checkpoint/replication health where applicable;
queue depth and oldest item age;
certificate expiry; and
dependency reachability.

Pantheon Blueprint workflows

Ody request success by interface;
Heimdall decisions, approval latency, denial rate, and downstream result classification;
uncorrelated or duplicate approval events;
Muninn checkpoint age, reviewed conversation count, candidate count, and duplicate suppression;
Huginn workflow status, capture count, deduplication, and staging failures;
AFFiNE-to-Mem0 index lag and source revision mismatch;
release discovery age and unapplied critical release proposals;
configuration drift; and
backup start, completion marker, remote verification, and restore-drill age.

Redaction

Do not export by default:

prompts, conversation bodies, knowledge-page contents, or external captures;
tool arguments or results containing user data;
headers, cookies, tokens, passwords, OAuth codes, or environment dumps;
full email addresses, phone numbers, or downstream account identifiers; or
database rows and file contents.

Prefer stable request IDs, classifications, counts, hashes, durations, and bounded error codes.

Alert routes

Use at least two independent operator routes, for example:

a push-notification service for urgent events; and
email for durable notification and lower urgency.

Do not make Ody the only alert route. If Ody or Heimdall is down, the operator must still receive the alert.

Suggested urgency:

Severity	Example	Route
Critical	Backup deletion attempt, suspected secret exposure, canonical database unavailable, Object Lock disabled	Immediate push and email
High	Heimdall bypass path, failed backup, disk exhaustion imminent, repeated restart loop	Immediate push
Medium	Index lag, one failed scheduled workflow, certificate within warning window	Email or operations digest
Low	New release discovered, capacity trend, successful restore drill	Digest/dashboard

Alert notifications should link to a redacted runbook and correlation ID, not embed sensitive logs.

Incident response

General sequence

Detect: preserve the alert, request IDs, timestamps, versions, and affected identities.
Triage: determine whether confidentiality, integrity, availability, or backup recoverability is affected.
Contain: disable the smallest affected route, connector, workload identity, or service.
Preserve: copy relevant audit and system evidence to protected storage.
Recover: follow the tested rollback or restore procedure.
Verify: run user-level transactions and negative security tests.
Notify: report impact and current limitations through an independent route.
Learn: document cause, detection gap, recovery result, and permanent action.

Do not destroy containers, rotate every credential, or restore databases automatically before preserving evidence and understanding the failure, unless continued operation creates greater harm.

Heimdall bypass or credential exposure

Stop or isolate the affected agent and connector.
Deny its egress at the network boundary.
Revoke the narrowest affected workload and downstream identities.
Preserve Heimdall, connector, application, and access logs.
Search for unauthorized downstream actions using service-native audit data.
Rotate exposed credentials through 1Password.
Verify that old credentials fail.
Restore service only after the bypass path is blocked and negative tests pass.

Knowledge integrity incident

Pause Muninn, Huginn promotion, indexing, and canonical writes.
Preserve the affected AFFiNE revisions, Mem0 results, source captures, and correlated tool audit.
Determine the last trusted AFFiNE revision.
Correct or restore AFFiNE first.
Rebuild Mem0 from the trusted canonical content.
Resume read-only retrieval and validate results.
Re-enable writers one at a time.

Do not repair canonical knowledge by editing Mem0.

Backup incident

Disable the suspected writer credential.
Preserve bucket access and retention evidence.
Confirm whether objects, versions, locks, or manifests changed.
Test an unaffected restore using the separate restore identity.
Create a fresh backup with a new writer identity when safe.
Treat unexplained retention or deletion changes as a security incident.

Failed update

Pause additional deployments.
Keep the failed containers and logs long enough to diagnose.
Determine whether persistent schema changed.
Roll back desired state or restore the pre-change backup as planned.
Run the full affected smoke-test set.
Resume schedules only after idempotency and backlog behavior are understood.

Routine maintenance

Daily

confirm the latest backup has a valid remote completion marker;
review critical and high alerts;
check host disk and inode headroom;
check unhealthy/restarting containers;
check Heimdall denial and approval anomalies;
check Muninn checkpoint and index freshness; and
review pending update proposals.

Weekly

review configuration and image drift;
review failed n8n and Muninn executions;
inspect Tailscale, Pangolin, and OIDC identity changes;
verify certificate expiry horizons;
test one harmless approval from every enabled Ody interface;
inspect backup storage growth and retention state;
review workload and service-account access; and
confirm rollback targets are still available.

Monthly

perform and record an isolated restore drill;
apply reviewed operating-system and application updates;
review firewall, Tailscale ACL, Traefik, and Pangolin exposure;
inspect secret rotation age and service-account scope;
review capacity forecasts;
test one documented self-recovery scenario per host;
rebuild Mem0 from canonical content in a test environment; and
review open exceptions and unvalidated integrations.

Quarterly

run a full critical-path restore;
rotate selected non-human credentials;
review emergency access and restore-reader custody;
test failure of Heimdall, AFFiNE, and the external Komodo Core;
review audit retention and redaction;
verify S3 versioning/Object Lock behavior with the current provider; and
update recovery objectives from measured results.

Before and after every change

Before:

identify scope and owner;
record current versions and health;
create and verify a pre-change backup;
define success, rollback, and stop conditions;
pause conflicting schedules; and
notify through an independent route.

After:

run smoke and negative tests;
compare desired and running state;
check new logs for secrets;
resume schedules gradually;
record versions, digests, timing, and evidence; and
confirm alerting and backup still work.

Capacity management

Track capacity by trust zone, not only by total free disk.

`agent-01`

Monitor:

conversation/session growth;
attachment and workspace growth;
Hermes/Muninn concurrency;
skill and update staging space;
model request latency and quota; and
Signal or WebUI backlog.

`knowledge-01`

Monitor:

AFFiNE database and blob growth separately;
database transaction logs and temporary space;
Mem0 vector count, index size, and rebuild duration;
index lag;
backup staging space; and
restore space sufficient for a parallel test copy.

`tools-01`

Monitor:

n8n database, execution history, binary data, and queue growth;
external capture staging;
browser-worker concurrency and ephemeral disk;
Executor catalogue, audit, and approval backlog; and
outbound API quotas and rate limits.

Backup storage

Because routine deletion is prohibited, forecast:

projected monthly growth =
    full backup size × full backups per month
  + incremental change volume
  + source/capture retention growth
  + manifest and audit overhead

Alert at multiple horizons, such as capacity projected to exhaust within 90, 60, and 30 days. Increasing storage is the normal response; deletion requires a separate governance decision.

Keep enough local free space to:

stage one complete backup;
pull a new and retain the previous image set;
perform database maintenance;
write logs during an external telemetry outage; and
recover from a failed migration.

Disaster recovery priority

Recover in dependency order:

operator identity, emergency access, private desired-state repository, and 1Password recovery;
network, DNS, Tailscale, host firewall, and trusted time;
Docker, Komodo Periphery, Traefik, and Grafana Alloy;
1Password-based secret provisioning and Heimdall policy boundary;
AFFiNE database and matching blob/upload storage;
Hermes/Ody state and user-facing read-only interface;
canonical retrieval through Heimdall;
Mem0 rebuilt from AFFiNE;
approval workflows and controlled writes;
Muninn checkpoints and draft curation;
n8n/Huginn collection workflows; and
nonessential browser, messaging, and automation integrations.

Keep writes disabled until canonical data, caller identity, policy, audit, and backup paths have been verified. A degraded read-only assistant is preferable to a fully automated system with uncertain state.

Operating evidence

Use the shared maturity labels, readiness gates, and evidence-record structure defined in Readiness and assurance. For each update, incident, restore drill, or self-healing test, add the operating details this runbook requires: desired-state commit, component versions and image digests, affected hosts and services, backup and approval IDs, measured recovery time, manual interventions, alert results, and rollback or restore outcome.

Store only redacted evidence in Git. Put sensitive evidence in a separately protected incident or audit store.

Official upstream references

For the shared maturity model, readiness gates, and evidence semantics, see Readiness and assurance. For the initial deployment path, see Getting started. For trust assumptions and negative tests, see Security model. For request, approval, indexing, and review sequences, see Data flows.