Production operations

This page covers production deployment patterns for the standalone Node collector backed by Postgres and S3-compatible object storage. The Cloudflare Workers path remains supported, but the Node collector is the clearest fit for container platforms and private infrastructure.

1. Network and reverse proxy

The recommended layout places the collector behind TLS, keeps OTLP ingest routes reachable, and protects dashboard/internal query routes with your normal auth boundary.

Nginx

server {
    listen 443 ssl http2;
    server_name obs.my-app.com;

    ssl_certificate /etc/letsencrypt/live/obs.my-app.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/obs.my-app.com/privkey.pem;

    gzip on;
    gzip_types application/json application/x-protobuf text/plain text/css;

    location /v1/ {
        proxy_pass http://127.0.0.1:8790;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        client_max_body_size 10m;
    }

    location /internal/ {
        proxy_pass http://127.0.0.1:8790;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location / {
        proxy_pass http://127.0.0.1:5173;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        try_files $uri $uri/ /index.html;
    }
}

Set TRUST_PROXY=true on the collector (and only when it is reachable solely through this proxy). Without it the dashboard login rate limit and client-IP handling key on the proxy's address instead of the real client, so one noisy client can lock everyone out of login.

CORS: the all-in-one image allows localhost and 127.0.0.1 on ports 5173, 5174, and 3000 by default. The standalone Node collector has no default; set ALLOWED_ORIGINS (comma-separated) to your dashboard and app origins, for example https://obs.my-app.com.

Caddy

obs.my-app.com {
    reverse_proxy /v1/* 127.0.0.1:8790 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }

    reverse_proxy /internal/* 127.0.0.1:8790 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }

    reverse_proxy /* 127.0.0.1:5173 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }
}

Set TRUST_PROXY=true on the collector (and only when it is reachable solely through this proxy). Without it the dashboard login rate limit and client-IP handling key on the proxy's address instead of the real client, so one noisy client can lock everyone out of login.

2. Postgres tuning

Each collector instance uses a persistent client-side pool. Start with the default PG_POOL_MAX=10 for low to moderate traffic (1–50 spans/second). For high-throughput environments (>500 spans/second), scale to 30 or 50, making sure the total across replicas stays below the database server's max_connections in postgresql.conf.

Set a statement timeout so expensive dashboard or analysis queries cannot lock the database indefinitely:

PG_STATEMENT_TIMEOUT=30000

3. Object storage retention

Replay chunks (.json) and pprof profiles (.pprof) live in S3-compatible object storage. Match bucket lifecycle retention to the database retention window (RETENTION_HOURS=72). For a 72-hour hot-debugging window, save the following three-day expiration rule as lifecycle-policy.json:

{
  "Rules": [
    {
      "ID": "AutoDeleteOldTelemetryBlobs",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "Expiration": {
        "Days": 3
      }
    }
  ]
}

Apply it with:

aws s3api put-bucket-lifecycle-configuration \
  --bucket obs-unified-storage-bucket \
  --lifecycle-configuration file://lifecycle-policy.json

4. Kubernetes orchestration manifests

Run collectors as stateless replicas against managed Postgres and S3-compatible storage.

Every collector process ships an in-process scheduler for the three recurring jobs the Cloudflare Worker gets from Cron Triggers (due analyses every minute, alert evaluation every 5 minutes, retention cleanup hourly). It is on by default, so a plain multi-replica Deployment would run each job once per replica. The manifests below keep exactly one owner: serving replicas set MAINTENANCE_ENABLED: "false" via the shared ConfigMap, and a separate single-replica obs-maintenance Deployment re-enables it. See the collector-node README for the MAINTENANCE_* settings.

ConfigMap and Secrets (obs-config.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
  name: obs-config
  namespace: observability
data:
  BLOB_STORE: "s3"
  S3_REGION: "us-east-1"
  S3_BUCKET: "my-production-obs-bucket"
  PG_POOL_MAX: "30"
  PORT: "8790"
  RETENTION_HOURS: "72"
  # Serving replicas must not run background maintenance; the dedicated
  # obs-maintenance Deployment below overrides this to "true".
  MAINTENANCE_ENABLED: "false"
  # Behind the reverse proxy above: login rate limit and client IP use X-Forwarded-For.
  TRUST_PROXY: "true"
---
apiVersion: v1
kind: Secret
metadata:
  name: obs-secrets
  namespace: observability
type: Opaque
data:
  # Base64 encoded connection strings, AWS keys, and passwords
  DATABASE_URL: cG9zdGdyZXM6Ly91c2VyOnBhc3NAcGctaG9zdDo1NDMyL29ic191bmlmaWVk  # postgres://user:pass@pg-host:5432/obs_unified
  S3_ACCESS_KEY_ID: QUtJQVhYWFhYWFhYWFhYWFhYWFg=                           # AKIAXXXXXXXXXXXXXXX
  S3_SECRET_ACCESS_KEY: c2VjcmV0LWtleS12YWx1ZS1nb2VzLWhlcmU=               # secret-key-value-goes-here
  INGEST_KEY: bXktc2VjdXJlLXdyaXRlLWtleQ==                                 # my-secure-write-key
  DASHBOARD_PASSWORD: bXktc3VwZXItc2VjdXJlLXBhc3N3b3Jk                    # my-super-secure-password
  # Programmatic read/admin bearer token for /internal/* (MCP server, CLI).
  # Generate a fresh random value per environment and base64-encode it:
  #   openssl rand -hex 32 | tr -d '\n' | base64
  DASHBOARD_TOKEN: <base64 of your generated token>

DASHBOARD_TOKEN is the one secret shared with programmatic clients, so keep it distinct from DASHBOARD_PASSWORD and INGEST_KEY:

• The MCP server (@obsunified/mcp-server) sends OBS_DASHBOARD_TOKEN as Authorization: Bearer; it must equal this DASHBOARD_TOKEN exactly.
• The CLI (@obsunified/cli) sends OBS_ADMIN_TOKEN the same way for keys mint, keys list, and doctor; it must also equal this DASHBOARD_TOKEN.
• Rotate all three together. The collector compares the token in constant time and only on /internal/* and GET /auth/check; ingest keys never grant read access, and the token plays no part in ingest or background maintenance.

Both Deployments below mount the same obs-secrets, so the serving replicas and the single obs-maintenance replica accept the same token. See Identity & External Access Recipes for the full header-forwarding and proxy patterns.

Serving Collector Deployment (obs-deployment.yaml)

Handles ingest and dashboard traffic only. MAINTENANCE_ENABLED is inherited as "false" from the ConfigMap, so scaling replicas up or down never changes how many times a job runs.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: obs-collector
  namespace: observability
  labels:
    app: obs-collector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: obs-collector
  template:
    metadata:
      labels:
        app: obs-collector
    spec:
      containers:
      - name: collector
        image: ghcr.io/dev-sawan-ruparel/obs-unified:2.3.0  # pin a tested version; avoid :latest
        ports:
        - containerPort: 8790
        envFrom:
        - configMapRef:
            name: obs-config
        - secretRef:
            name: obs-secrets
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "250m"
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /health
            port: 8790
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8790
          initialDelaySeconds: 15
          periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: obs-collector-service
  namespace: observability
spec:
  selector:
    app: obs-collector
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8790
  type: ClusterIP

Maintenance Deployment (obs-maintenance.yaml)

The single owner of the scheduled jobs. Same image, ConfigMap, and Secret as the serving Deployment; the only differences are replicas: 1, the explicit MAINTENANCE_ENABLED: "true" override (a container env entry takes precedence over envFrom), a Recreate strategy so a rollout never has two maintenance pods alive at once, and a distinct app label so obs-collector-service never routes traffic to it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: obs-maintenance
  namespace: observability
  labels:
    app: obs-maintenance
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: obs-maintenance
  template:
    metadata:
      labels:
        app: obs-maintenance
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: maintenance
        image: ghcr.io/dev-sawan-ruparel/obs-unified:2.3.0  # pin a tested version; avoid :latest
        ports:
        - containerPort: 8790
        envFrom:
        - configMapRef:
            name: obs-config
        - secretRef:
            name: obs-secrets
        env:
        - name: MAINTENANCE_ENABLED
          value: "true"
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "250m"
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /health
            port: 8790
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health
            port: 8790
          initialDelaySeconds: 15
          periodSeconds: 20

Verify the split after applying both manifests:

kubectl -n observability logs deploy/obs-maintenance | grep '\[maintenance\] scheduler started'
kubectl -n observability logs deploy/obs-collector   | grep 'maintenance scheduler disabled'

On SIGTERM the maintenance pod stops arming new ticks, waits up to 10 s for an in-flight job, then closes its Postgres pool, which fits inside the 30 s grace period above. The serving Deployment's /health endpoint is unaffected by the scheduler, so the probes are identical.

Never run two or more replicas of the same image without MAINTENANCE_ENABLED=false on all but one of them. The all-in-one image and the Node collector default to MAINTENANCE_ENABLED=true, so every replica would otherwise run each scheduled job.

If you prefer a scheduler you already operate, set MAINTENANCE_ENABLED: "false" everywhere, omit obs-maintenance, and run the jobs from your own cron; do not leave the default (true) on a multi-replica serving Deployment.

5. Upgrades and database migrations

During software updates, execute database migrations before recycling rolling container deployments to prevent schema mismatch errors inside the collector instances.

Pin the image to the version you are rolling out (:2.3.0 here; use your target release) rather than :latest. Run the migrations securely on your database using a temporary migrations job container:

docker run --rm \
  -e DATABASE_URL="postgres://user:pass@pg-host:5432/obs_unified" \
  ghcr.io/dev-sawan-ruparel/obs-unified:2.3.0 \
  node /repo/apps/collector-node/scripts/migrate-pg.mjs

6. Operational checks

Before opening ingest traffic:

1. Run database migrations for the target collector storage.
2. Confirm /health responds through the reverse proxy.
3. Verify CORS from the browser origin with pnpm dlx @obsunified/cli doctor.
4. Send one synthetic trace, one log, and one usage event.
5. Confirm dashboard login and internal query routes are not publicly exposed without your intended auth.

7. Backup and restore

Applies to the all-in-one image, which keeps Postgres 15 (cluster 15/main, database obs_unified, role obs) in /var/lib/postgresql and file blobs in /data/blobs. Managed Postgres plus S3 deployments should use the provider's snapshot and bucket-versioning tools instead.

Back up (container named obs; the embedded Postgres only accepts password auth over TCP, so pass -h 127.0.0.1):

docker exec -e PGPASSWORD=obs obs \
  pg_dump -h 127.0.0.1 -U obs --clean --if-exists obs_unified > backup.sql

docker run --rm -v obs-unified-blobs:/data -v "$PWD":/backup alpine \
  tar czf /backup/blobs.tgz -C /data .

--clean --if-exists makes the dump replace the schema the new container creates on first boot. /data only holds replay recordings and pprof profiles; skip the tarball if you do not need them.

Restore into empty volumes (a new container on fresh volumes, same image tag or newer):

# 1. Restore blobs into the empty blob volume before the first start
docker run --rm -v obs-unified-blobs:/data -v "$PWD":/backup alpine \
  tar xzf /backup/blobs.tgz -C /data

# 2. Start the container once so Postgres initializes and migrations run
docker run -d --name obs -p 5173:5173 -p 8790:8790 \
  -e INGEST_KEY -e DASHBOARD_PASSWORD -e DASHBOARD_TOKEN \
  -v obs-unified-db:/var/lib/postgresql -v obs-unified-blobs:/data \
  ghcr.io/dev-sawan-ruparel/obs-unified:<version>

# 3. Load the dump, then restart so the collector drops cached state
docker exec -i -e PGPASSWORD=obs obs psql -q -h 127.0.0.1 -U obs obs_unified < backup.sql
docker restart obs

Ingest keys live in the database, so they come back with the dump (a key you revoked stays revoked). INGEST_KEY, DASHBOARD_PASSWORD, and DASHBOARD_TOKEN come from the new container's environment; changing DASHBOARD_PASSWORD invalidates existing browser sessions.

8. Alerts and webhooks

Alert rules are evaluated by the maintenance scheduler every 5 minutes (see the maintenance split in the Kubernetes section). Webhook channels are admin-supplied URLs that the collector POSTs to from inside its own network, so they are validated:

• Only plain http(s) URLs to public hosts are accepted. Loopback, RFC 1918 ranges, link-local (cloud metadata) addresses, and embedded credentials are rejected. Hostnames are checked as written, so a name that resolves to a private address is not caught at validation time.
• To post to an internal receiver (for example an in-cluster Alertmanager or a local webhook sink), set ALERT_WEBHOOK_ALLOW_PRIVATE=true on the collector. The Workers collector honours it today; the Node/Docker image requires image 2.2.2 or newer, which forwards the flag (earlier Node images ignored it even though the rejection message mentions it).
• Webhook URLs are redacted in logs and span attributes (only the origin and first path segment are kept), since Slack and PagerDuty style hooks carry secrets in the path.
• The seeded sample data (OBS_LOCAL_SEED=true) creates alert rules that point at a placeholder hooks.example.com webhook URL. Since collector 2.2.2 the seeder creates them disabled (enabled: false), so they do not fire. Enable them only after pointing them at a real receiver.

9. Monitoring the collector

• Health: GET /health is the only probe endpoint. There is no /metrics and no /ready route, so use /health for both readiness and liveness (as in the Kubernetes manifests above).
• Docker: the image ships a HEALTHCHECK (every 15s) that needs both the collector /health on :8790 and the dashboard on :5173 to answer. docker ps shows healthy or unhealthy.
• Logs: written to stdout/stderr as [component] message followed by an optional fields object, for example [maintenance] scheduler started { ... }. Startup lines use the [obs-unified] prefix. Fields are printed with Node's console, so treat them as readable key/value output rather than strict JSON when building log parsers.

10. Identity and external access

obs-unified does not contain native per-user RBAC or SSO. To deploy behind Cloudflare Access or oauth2-proxy with proper path boundaries, secrets management, header forwarding, and MCP service tokens, see Identity and external access.