Deploy / docs/DEPLOY-RUNBOOK.md

Deploy runbook (sandbox / production relay)

Step-by-step procedure for updating a running nostream host after a new ghcr.io/cameri/nostream:latest (or release tag) image is available.

Image delivery vs relay recreate

GitHub Release published (tag v*)
  → CI publishes ghcr.io/cameri/nostream:latest (and the version tag)
  → the release webhook pulls :latest onto the host (see below)
  → operator runs this runbook (migrate + recreate relay(s))
  → Cloudflare Tunnel / reverse proxy keeps pointing at 127.0.0.1:8008

Nothing publishes an image on a push to main any more, so a host that still pulls :main will never see a new image. The on-host pull stack must track the release instead:

1. NOSTREAM_TAG in /opt/nostream/webhook/.env set to latest.
2. The GitHub webhook subscribed to releases as well as push, because the receiver ignores anything that is not a push to the watched branch.

Both live in containers/nostream-dev-webhook/ (the versioned source) with its own README. Until both are done, deployments are manual: pull the tag by hand and continue at step 2 below.

The webhook pull stack loads images only. It does not run migrations or recreate relay containers. This runbook closes that gap.

The release event reaches the webhook at the same moment the image build starts, so the pull fires before the tag exists. The puller retries on a growing backoff and re-queues a message it exhausts, but there is no signal that says the pull landed. Before you recreate, confirm the image is on the host:

docker images --filter=reference='ghcr.io/cameri/nostream' --format '{{.Tag}} {{.ID}}'
# expect the tag the release carried, and a digest that is not the previous one

A recreate with a stale latest starts the previous release from a brand-new build: the migrate step runs the old schema, nothing fails, and the deploy looks successful. When in doubt, docker pull ghcr.io/cameri/nostream:latest and re-run the check above.

Choose your stack

Do not run both stacks at once — both bind 127.0.0.1:8008. Initial HAProxy install and trusted-proxy settings are in deploy/README.md.

During any deploy, wait for /readyz to return 200 before treating the relay as good. Endpoint behavior, fan-out checks, drain on SIGTERM, and probe timeouts are documented under Health checks in deploy/README.md.

Prerequisites

• Host bootstrapped (deploy/bootstrap.sh /opt/nostream)
• New relay image on the host (docker pull or docker load)
• Shell access (cd /opt/nostream)
• Before loading a new image or recreating containers, record the running relay image (not docker image inspect :latest after pull — that tag may already point at the new image)

cd /opt/nostream
RELAY_CID="$(docker compose ps -q nostream 2>/dev/null || true)"
if [[ -n "$RELAY_CID" ]]; then
  docker inspect "$RELAY_CID" --format '{{.Image}}' | tee /tmp/nostream-pre-deploy-image-id
else
  echo "warn: nostream is not running; save a known-good image tarball or ID manually" >&2
fi

HAProxy stack: capture from a running backend, e.g. docker compose -f docker-compose.haproxy.yml ps -q nostream-blue.

Shared steps (both stacks)

1. Refresh release-managed files (when release notes say so)

./deploy/bootstrap.sh /opt/nostream

Bootstrap refreshes the single-relay docker-compose.yml and postgresql.conf only. On HAProxy hosts, also copy stack-specific files from your checkout (same paths as initial install in deploy/README.md):

REPO=/path/to/nostream   # git checkout at the release tag or commit
cp "$REPO/deploy/docker-compose.haproxy.yml" "$REPO/deploy/rolling-relay-recreate.sh" /opt/nostream/
cp -r "$REPO/deploy/haproxy" /opt/nostream/
chmod +x /opt/nostream/rolling-relay-recreate.sh

Existing .env and .nostr/settings.yaml are preserved.

2. Load the new image (skip if webhook pull already did)

docker pull ghcr.io/cameri/nostream:latest

On hosts that cannot reach GHCR over IPv4, use docker save / docker load. See deploy/README.md.

Single-relay deploy

Bootstrap copies compose files to /opt/nostream but not recreate-relay.sh. Run the script from a git checkout, or install it once on the host:

# From repository checkout:
chmod +x deploy/recreate-relay.sh
./deploy/recreate-relay.sh /opt/nostream

# Or on the host (after copying from the repo):
cp /path/to/nostream/deploy/recreate-relay.sh /opt/nostream/
chmod +x /opt/nostream/recreate-relay.sh
/opt/nostream/recreate-relay.sh /opt/nostream

Equivalent manual steps:

cd /opt/nostream
docker compose up --no-deps --force-recreate --exit-code-from nostream-migrate nostream-migrate
docker compose up -d --force-recreate --no-deps nostream

HAProxy blue/green deploy

After the new image is loaded:

cd /opt/nostream
docker compose -f docker-compose.haproxy.yml run --rm nostream-migrate
./rolling-relay-recreate.sh

The rolling script replaces blue and green one at a time; the peer must stay up and /readyz healthy. See deploy/README.md.

Verify

cd /opt/nostream
docker compose ps   # or: docker compose -f docker-compose.haproxy.yml ps
curl -sf http://127.0.0.1:8008/readyz && echo
curl -sf http://127.0.0.1:8008/healthz
curl -s -H 'Accept: application/nostr+json' http://127.0.0.1:8008/

Expected: migrate exited 0, relay(s) running, /readyz HTTP 200 with database and redis "ok": true, NIP-11 JSON at /.

If admin is enabled, also check /admin/health through your normal auth path.

Watch logs (first few minutes)

docker compose logs -f --tail=100 nostream
# HAProxy stack: nostream-blue and/or nostream-green

Rollback

When the new relay fails readiness or behaves incorrectly:

1. Stop the bad relay(s)

Single stack:

cd /opt/nostream
docker compose stop nostream

HAProxy stack: stop the relay you just replaced; leave the healthy peer running.

2. Restore the previous image

PREVIOUS_IMAGE="$(cat /tmp/nostream-pre-deploy-image-id)"
docker tag "$PREVIOUS_IMAGE" ghcr.io/cameri/nostream:latest

Or docker load -i /path/to/nostream-latest-backup.tar.gz.

3. Recreate on the old image (skip migrations)

Do not run the migrator from a downgraded image against an already-upgraded database. Recreate the relay without migrations:

Single-relay:

cd /opt/nostream
SKIP_MIGRATE=1 /opt/nostream/recreate-relay.sh /opt/nostream
# or from checkout: SKIP_MIGRATE=1 ./deploy/recreate-relay.sh /opt/nostream

HAProxy: run migrate only when moving forward on a new image. For rollback, retag the old image to :latest, then recreate relays without migrations.

rolling-relay-recreate.sh will not replace a running relay unless its peer is up and /readyz healthy. If step 1 left one backend stopped, running the rolling script first fails with “peer … is not running” and neither backend is restored.

1. Recreate the stopped backend on the old image (substitute blue/green as needed):

   cd /opt/nostream
   STOPPED=nostream-green   # the relay you stopped in step 1
   docker compose -f docker-compose.haproxy.yml rm -f "$STOPPED" 2>/dev/null || true
   docker compose -f docker-compose.haproxy.yml up -d --no-deps --force-recreate --wait "$STOPPED"

2. If the other backend still runs the bad image, roll it back with the rolling script (peer is healthy again; the script does not re-run nostream-migrate):

   cd /opt/nostream
   ./rolling-relay-recreate.sh

   Or recreate that service alone the same way as step 1.

If both backends were stopped, bring one up with step 1, then step 2 for the other.

Troubleshooting

/readyz stays 503

• Check Postgres and Redis: docker compose ps
• Inspect relay logs: docker compose logs nostream --tail=200 (or blue/green)
• Confirm .env credentials match the running database and cache
• On HAProxy backends, confirm relayBroadcast.ok in /readyz when fan-out is enabled

nostream-migrate exits non-zero

• Read migrate logs: docker compose logs nostream-migrate
• Do not recreate relay(s) until migrate succeeds
• Escalate if a migration is destructive — deploy compatible image order for expand/contract migrations

Image pull fails (GHCR / IPv6)

• Use docker save / docker load from a machine that can reach GHCR
• Keep pull_policy: never on nostream services when using pre-loaded images

Relay up but clients cannot connect

• Confirm tunnel or reverse proxy still targets 127.0.0.1:8008
• Webhook image delivery alone does not replace this runbook

Automation (out of scope here)

Wiring migrate + recreate into the webhook pipeline after this manual procedure is proven on sandbox is a separate follow-up.