Rotating the passphrase — runbook

The passphrase is what the runtime libraries use to decrypt the vault. It lives in your platform's secret store (Vercel sensitive variables, AWS Secrets Manager, GCP Secret Manager, host file mounted as VSYNC_PASSPHRASE_FILE). If it leaks — into a log, a Slack thread, an ex-employee's terminal — you rotate.

This page is the step-by-step incident-response procedure. For the design rationale and atomic-flow details, read the Runtime token guide and spec v0.10 §3.

When to rotate

Rotate the passphrase when any of the following happens:

Trigger	Urgency
Passphrase appears in a chat log, CI log, Sentry event, screenshot, recording, ticket	rotate now
A teammate with vault access offboards	rotate before revoking their bucket IAM, so the audit log of their last activity is preserved
Routine schedule (e.g. quarterly hygiene rotation for prod)	scheduled — pick a low-traffic window
You suspect a host with a VSYNC_PASSPHRASE_FILE was tampered with	rotate now, then audit the host

You do not need to rotate when:

• The IAM key in VSYNC_CONFIG leaks — that's IAM rotation (separate runbook).
• A teammate's laptop is lost but they had pull-ed already — they have plaintext locally, which the passphrase doesn't protect against. Rotate the upstream secrets (DB password, API keys), not the vault passphrase.
• A bundle from S3 leaks — the bundle is ciphertext. Without the passphrase + a way to read it, it's noise. (If you're not sure they have both, rotate anyway.)

Prerequisites

You need:

• A machine that already has vsync initialized for this (repo, env) — your own laptop is typical.
• The current passphrase (you'll type it once).
• A new passphrase, ideally generated by a password manager. The CLI will prompt twice + confirm; minimum 12 characters.
• Write access to the platform secret store (Vercel UI, aws secretsmanager update-secret, host SSH for VSYNC_PASSPHRASE_FILE).
• The ability to trigger a rolling restart of apps in the affected env.

The race window — understand it before rotating

Between when the CLI updates the S3 manifest and when you update VSYNC_PASSPHRASE in your secret store, apps that boot will fail to decrypt. They'll pull the new bundle but try to decrypt with the old passphrase from their environment.

T0:  vsync rotate-passphrase succeeds → S3 manifest now points at the new bundle
                                       → audit log shows the rotate row
T0..T1: WINDOW — any app boot in this gap gets WrongPassphraseError
T1:  You update VSYNC_PASSPHRASE in the platform secret store
T1..T2: Roll-restart in progress; old containers running with old (in-memory) bundle still serve traffic
T2:  All instances are restarted, reading the new bundle with the new passphrase

The window is operator-owned. Two-passphrase grace-period rotation (accept either passphrase during overlap) is parked for vsync v2.

How to shrink the window:

1. Have the new passphrase staged in the platform secret store before running rotate-passphrase (some platforms let you upload a new version without activating it).
2. Run rotate-passphrase and the secret-store update back-to-back — script it if you do this often.
3. Trigger the rolling restart immediately after the secret-store update.
4. Apps that boot during the window get WrongPassphraseError from open() and exit; the orchestrator restarts them; by then your secret-store update has propagated.

The procedure

0. Pre-flight — write down the gen

vsync status --check-remote
# Look for the line for <env>; note the gen column.

You're about to bump gen by 1. If you see anything unexpected (LOCAL IS BEHIND, an orphan, a dangling profile), resolve that first — rotating on top of a stale or torn state will make the post-mortem harder.

1. Pre-stage the new passphrase

Generate the new passphrase in your password manager. Don't type it from memory or read it aloud.

• AWS Secrets Manager: aws secretsmanager update-secret --secret-id <id> --secret-string '<new-pw>' — but don't activate it as the platform default yet, just upload a new version.
• Vercel: open the env-var UI, change the value, but don't redeploy yet.
• GCP Secret Manager: gcloud secrets versions add <name> --data-file=- (you'll switch the alias after step 3).
• VSYNC_PASSPHRASE_FILE on a VPS: stage at /etc/vsync/passphrase.new, owned and readable only by the service user, mode 0600.

The point is to have the new passphrase in place so step 3 is one command, not "now I need to remember to update Vercel".

2. Run vsync rotate-passphrase

From a laptop with the env already configured:

vsync rotate-passphrase --env=prod

Prompts:

Old passphrase: ********
New passphrase: ********
Confirm new passphrase: ********
Re-encrypt bundle and push? [y/N] y

For automation, the new passphrase can come from a flag (the old passphrase cannot — by design, so it never appears in shell history):

vsync rotate-passphrase --env=prod --new-passphrase="$(pbpaste)"
# Still prompts for the old one.

Success output (to stderr; stdout is empty for scriptability):

✓ Bundle re-encrypted with new passphrase (gen=3 → gen=4)
✓ Manifest pointer updated atomically (ETag-conditional)
✓ Audit log entry written

Next steps:
  1. Update VSYNC_PASSPHRASE (or contents of VSYNC_PASSPHRASE_FILE)
     in your secret store / host file
  2. Roll-restart apps in prod

⚠ Apps booting between this moment and step 1 will fail to decrypt.
  This rotation race window is operator-owned; vsync cannot bridge it in v1.

The S3 bundle is rotated. The race window starts now.

3. Activate the new passphrase in the secret store

You staged this in step 1. Activate it now:

• AWS Secrets Manager: the platform reads the AWSCURRENT staging label by default; aws secretsmanager update-secret-version-stage --secret-id <id> --version-stage AWSCURRENT --move-to-version-id <new-version-id>. ECS / Lambda will pick this up on next task start.
• Vercel: click Save / Redeploy in the env-var UI. Vercel kicks off a new build; depending on your project, that may automatically roll instances.
• GCP Secret Manager: if you used :latest, the activation was automatic in step 1 (skip ahead). If you pinned a version, update the alias: gcloud secrets versions ....
• VPS: mv /etc/vsync/passphrase.new /etc/vsync/passphrase (atomic rename; reads on /etc/vsync/passphrase see either the old contents or the new, never a torn read).

4. Roll-restart

Trigger a rolling restart immediately:

# Kubernetes
kubectl rollout restart deployment/<your-app> -n <ns>

# AWS ECS
aws ecs update-service --cluster <c> --service <s> --force-new-deployment

# GCP Cloud Run
gcloud run services update <name> --region <r>  # no real change; forces a new revision

# Fly.io
fly deploy --strategy=rolling

# Systemd
sudo systemctl restart myapp@*.service

Apps that booted during the window will have already exited with WrongPassphraseError; the orchestrator was restarting them anyway. Apps that were already running are unaffected — their in-memory bundle is still valid for the lifetime of that process.

5. Verify

# 5a. Audit log shows the rotate row
vsync audit prod --limit=5
# → 2026-05-23T11:42:01Z  rotate  ...  meta={"event":"rotate","gen":4,"prev_gen":3}

# 5b. status confirms gen alignment
vsync status --check-remote
# → all envs ok, no drift

# 5c. App healthcheck reports the new gen
curl https://yourapp.com/healthz
# → {"status":"fresh","gen":4}

If /healthz reports gen=3, you have stragglers still running on the old bundle. Force-restart them.

If /healthz reports WrongPassphraseError after step 4, the secret-store update didn't propagate. Re-check step 3, then restart.

Failure modes and recovery

Failed step	State on disk	Recovery
Old passphrase wrong (step 2 fails before re-encrypt)	Bundle + manifest untouched	Re-run; check your password manager. Exit 1.
Encrypt with new (step 2 internal)	Bundle + manifest untouched	Bug in vsync — file an issue with the stack trace. Exit 2.
PUT of new bundle (S3 transient)	Old bundle still active; orphan v=<newTs> object	Re-run rotate-passphrase. The orphan is harmless (manifest never named it). Exit 2.
Manifest swap — 412 conflict	Old bundle still active	Concurrent rotation. Find out who else ran rotate-passphrase; reconcile before retrying. Exit 3.
Manifest swap — other 5xx	Old bundle still active	Transient bucket failure. Safe to retry. Exit 3.
Audit append failed	Rotation succeeded — new bundle is live	Exit 4. The CLI prints a copy-pasteable "manual audit row" block. Append it via aws s3 cp / the S3 console so the team sees the row. Do not roll back the manifest.

The atomic invariant: the bundle and manifest can never disagree. If you got a non-zero exit and vsync status --check-remote shows the old gen, the rotation didn't take — your apps are unaffected. If it shows the new gen, the rotation took — proceed with step 3.

Notify the team

A passphrase rotation is a visible event. After step 5:

1. Audit log — vsync audit prod --limit=5 should already show the rotate row with whatever --note you passed (e.g. --note="quarterly hygiene rotation" or --note="incident #427").
2. Post in #ops (or wherever your team coordinates) — "rotated passphrase for prod env, gen 3 → 4, race window closed at 14:32 UTC". Include the audit log row.
3. Update the incident ticket if this is a leak response. Link the audit row.

Schedule

There is no enforced rotation schedule in vsync. Recommendations:

• Prod: quarterly, on a low-traffic day. Schedule a maintenance window.
• Staging / dev: ad-hoc on leak. Routine rotation is overkill.
• After offboarding a teammate with vault access: rotate before revoking their bucket IAM.

Where to go next

• IAM key rotation (separate from passphrase): IAM rotation runbook
• Full leak response checklist: Incident response
• Background: Runtime tokens · spec v0.10
• Audit log details: Audit log