AutoSRE causal ontology

When a production incident pages an AI agent or human SRE, finding where an error surfaced is only the first step. Authorizing an automated rollback, feature-flag flip, or config revert requires proving what change caused the incident—and knowing when telemetry is insufficient to prove causality safely.

obs-unified's AutoSRE causal ontology (RFC 0013) adds three missing layers to the unified telemetry graph:

1. Change Event Ledger (change_events) — Every deploy, rollback, feature-flag evaluation, config update, and agent prompt release is recorded and linked to runtime spans via OpenTelemetry resource and span attributes (service.version, vcs.commit.id, obs.config.digest, feature_flag.*).
2. Dual Epistemic Model (relationshipCertainty vs. causalProofTier) — Graph link provenance (how sure we are two nodes are connected) is strictly separated from causal proof strength (how sure we are a candidate change caused the symptom).
3. Incident Memory & Pre-Execution Remediation Gate — Findings are fingerprinted across incidents, counterfactual eval runs verify baseline-fail + candidate-pass pairs, and a stateful lease/grant gate enforces Compare-And-Swap (CAS) target states and post-execution watchdog recovery before any autonomous write executes.

Real-world production scenarios AutoSRE solves

Standard APM tools and naive LLM incident responders fail in five recurring production scenarios. AutoSRE is architected specifically to solve each of them:

1. Progressive rollout vs. the traffic-share fallacy

The challenge: Suppose release v2.4.1 is rolled out to handle 95% of production traffic, while v2.4.0 serves the remaining 5%. During an outage caused by an upstream payment gateway or expired certificate, v2.4.1 will naturally account for 95% of all errors. A naive agent that checks "which version has the most errors?" will blame and roll back v2.4.1, even though both versions have the exact same per-request error rate.

How AutoSRE solves it: The blast-radius evaluator (GET /internal/changes/:id/blast-radius / get_change_blast_radius) computes traffic-normalized cohort error-rate lift across a minimum sample size (N >= 50 requests per cohort by default, configurable via minSample):

ErrorRate(v) = error_requests(v) / total_requests(v)
RelativeRisk = ErrorRate(v_new) / max(ErrorRate(v_prev), 0.001)

• If v2.4.1 has a 12% error rate and v2.4.0 has a 0.08% error rate (absolute rate difference >= 5% or p95Ratio >= 2.5, and RelativeRisk >= 5.0 or p95Ratio >= 2.5), the cohort verdict is isolated_to_new_version, promoting the finding to cohort_rate_isolated.
• If an instant 100% cutover occurs (oldCohort.requestCount < N but pre-change baseline preBaseline.requestCount >= N) and the same thresholds hold against pre-change baseline, the cohort verdict is sequential_post_cutover_regression (corroborated_candidate).
• If both v2.4.1 and v2.4.0 fail (errorRate >= 5% on both), AutoSRE distinguishes between noisy-neighbor contention (shared_degradation_new_version_aggressor) and an external upstream outage (shared_degradation_external_cause).

2. Noisy-neighbor shared database contention

The challenge: Sometimes a bad deploy (v2.4.1) introduces an unindexed query or connection pool leak that exhausts a shared PostgreSQL instance (db:prod:postgresql:orders-primary:5432/orders). Once the database saturates, requests on the old version (v2.4.0) also start failing with connection timeouts. If an agent treats "old version is also failing" as proof that the deploy is innocent, it will miss the real culprit.

How AutoSRE solves it: Instead of refuting the deploy when errors are shared across versions, AutoSRE runs a shared-resource contention join across canonical resource identities (environment + namespace + kind + name). When v2.4.1 exhibits a >= 3x downstream call fanout or duration amplification (fanoutRatio >= 3.0 || durationRatio >= 3.0) on a shared database or queue node while v2.4.0 fails downstream on that same node, AutoSRE emits cohortVerdict: "shared_degradation_new_version_aggressor" (corroborated_candidate) and populates sharedResourceAggressors with the contending trace/span (contended_on_shared_resource / induced_shared_resource_contention).

3. Structural code overlap without causal proof

The challenge: A deploy touches 40 files including src/checkout/charge.ts, which appears in a failing stack trace (code.filepath = "src/checkout/charge.ts"). That overlap is an exact structural link (relationshipCertainty: "exact"), but it does not prove the commit caused the outage—the commit might have only edited a comment or log line in charge.ts.

How AutoSRE solves it: AutoSRE separates Link Provenance (relationshipCertainty: "exact" | "inferred" | "heuristic") from Causal Proof Tier (causalProofTier):

Code-path overlap (touched_same_code_path) alone is capped at causalProofTier: "corroborated_candidate". It can never authorize an autonomous rollback or flag flip without cohort rate isolation (cohort_rate_isolated) or counterfactual replay (counterfactual_verified).

4. Paired baseline-fail + candidate-pass counterfactual verification

The challenge: Running a synthetic check that passes after a fix is applied—or seeing error rates drop when traffic subsides—does not prove the fix solved the bug. If the reproducer didn't fail against the broken baseline first, a passing test is meaningless.

How AutoSRE solves it: Upgrading a finding to counterfactual_verified (POST /internal/findings/:id/counterfactual) requires a paired evaluation:

1. Baseline failure (failingRunId): An eval_runs record against the incident's broken serviceVersion / commitSha / candidateAgentVersion that reproduces the failure (passed = 0) with a matching input_payload_hash and failure_fingerprint.
2. Candidate pass (passingRunIds): K passing eval_runs (passed = 1, requiredTrials >= 1) of the exact same eval_case_id and input_payload_hash against the candidate remediation state.

5. Competing alerts, state drift, and post-remediation regressions

The challenge: During an outage, webhook retries or multi-symptom alerts can spawn duplicate agent runs that race to roll back the same service. Or a human operator may manually deploy v2.4.2 while an agent is still deciding to roll back v2.4.1.

How AutoSRE solves it:

• Durable incident lease (incident_investigations): Deduplicates firing alerts by (project_id, environment, dedup_key) (enforced at the database level via partial unique index uq_incidents_active_dedup) and enforces an exclusive runner lease (POST /internal/incidents/:id/lease, with expired leases reaped back to open).
• Pre-execution CAS policy gate (remediation_proposals): propose_remediation_action binds an action to expectedCurrentState (e.g. { version: "2.4.1", commitSha: "8f3a1b2" }), evaluates the action-specific policy matrix (plus a 3-proposal/hour circuit breaker scoped to finding fingerprint + action kind), and either parks the proposal for 1-Click Human Approval in the dashboard (/#/autonomous-review, status: "pending_human_approval") or issues a 5-minute single-use grantToken (status: "grant_issued").
• External runner claim & collector watchdog (claim_remediation_grant): The collector acts as the stateful authorization gate, while your external agent or CI/CD runner claims the single-use grantToken (operation: "claim", verifying expectedCurrentState against ontology_nodes), executes the actual infrastructure rollback/flag-toggle/PR, and records operation: "complete" (which must be called by the claiming runnerId within the 5-minute claim window). Completing a proposal arms the post-remediation watchdog (+2m / +5m / +15m anchored to completion time, evaluated on the 1-minute analyses-runner cron tick or via POST /internal/remediations/watchdog), which requires two consecutive healthy checkpoints (errorRate < 5% with N >= 50 comparable requests) before transitioning recoveryStatus to recovered and causalVerificationStatus to operationally_corroborated (or still_degraded / refuted / inconclusive_low_traffic), never conflating operational recovery with counterfactual_verified.

Recording changes and runtime lineage

1. Stamp lineage and feature flags in your SDK

In TypeScript / JavaScript (@obsunified/telemetry-sdk, @obsunified/analytics-sdk, or @obsunified/sdk), pass environment, serviceVersion, commitSha, and configDigest (or set OBS_ENVIRONMENT, OBS_SERVICE_VERSION, OBS_COMMIT_SHA, OBS_CONFIG_DIGEST) when initializing, and call recordFeatureFlag(...) inside request scope:

import { initObservability, recordFeatureFlag } from "@obsunified/telemetry-sdk";

initObservability({
  serviceName: "checkout-api",
  collectorUrl: process.env.OBS_COLLECTOR_URL,
  environment: "prod",
  serviceVersion: "2.4.1",
  commitSha: "8f3a1b2c",
  configDigest: "cfg_99a1",
});

// Inside a request handler or span scope:
recordFeatureFlag("enable_legacy_tax_fallback", false, {
  providerName: "openfeature",
});

In non-JS OpenTelemetry SDKs (Python, Go, Rust, Java, .NET, PHP, Ruby, Elixir, Swift, Kotlin), set standard OpenTelemetry resource/span attributes directly (service.version, deployment.environment.name, vcs.commit.id, obs.config.digest, and feature_flag.key + feature_flag.variant or obs.feature_flags.<key>).

2. Record deploys, config changes, or flag flips from CI/CD

Use the CLI or POST /v1/changes from your deployment pipeline:

pnpm dlx @obsunified/cli changes record \
  --url https://collector.example.com \
  --api-key "$OBS_INGEST_KEY" \
  --environment prod \
  --kind deploy \
  --service checkout-api \
  --version 2.4.1 \
  --previous-version 2.4.0 \
  --commit-sha 8f3a1b2c \
  --previous-commit-sha 7d2e0a1b \
  --summary "Switch tax rounding to strict decimal mode" \
  --touched-paths "src/checkout/charge.ts"

Agent workflow with MCP tools

The @obsunified/mcp-server registers seven read tools for AutoSRE by default, plus four opt-in write tools that are only registered when OBS_MCP_ENABLE_WRITES=true:

1. get_evidence_bundle (intent: "triage_incident", or any incident / change / finding anchor) — Returns a token-budgeted incident bundle with bundle.autosre (faultLocus, candidateChanges, contentionCandidates, and priorFindings). priorFindings match the fault locus by entity id and by recurrence fingerprint, so a failure that recurs on a different trace of the same service, route and error signature surfaces the findings and fixes recorded before.
2. list_recent_changes — Lists recent deploys, flag flips, config updates, and prompt releases scoped by environment and serviceName.
3. get_change_blast_radius — Computes traffic-normalized cohort error-rate lift (isolated_to_new_version, sequential_post_cutover_regression, shared_degradation_new_version_aggressor, shared_degradation_external_cause, no_regression_detected, or insufficient_sample), noisy-neighbor contention, and stack-frame touchedPaths overlap.
4. search_similar_findings — Queries historical InvestigationFinding records by stable fingerprint (environment + originatingNodeId + normalizedOperation + errorSignature + downstreamTargetNodeId) so recurring incidents immediately surface proven prior fixes.
5. list_incidents, get_incident, get_remediation_proposal — Read incidents (filter by environment, status, limit), one incident with its findings and proposals, and one proposal's status. None of them ever returns a grant token.
6. open_incident (write) — Opens an incident, or joins the active one for the same symptom (deduplicated: true). Its id is the incidentId that proposals require.
7. record_investigation_finding (write) — Persists a structured finding with causalChain triples (degraded_after_change, isolated_to_version_cohort, touched_same_code_path, contended_on_shared_resource, induced_shared_resource_contention, originated_fault_in_span, verified_by_counterfactual, mitigated_by_change, remediated_by_agent_run), citedEvidenceIds, and optional paired counterfactual eval run IDs. The collector derives causalProofTier server-side from contributingChangeId's blast radius and counterfactual proof so uncorroborated client claims cannot inflate autonomy.
   • Eval runs named in counterfactual are checked first (GET /internal/evals/runs/:id); a missing run fails the call before anything is stored.
   • If verification fails after the finding was stored, the call still succeeds and returns data.finding plus data.counterfactualError instead of an error that hides the finding id.
   • Pass an idempotencyKey so a retry returns the already-recorded finding instead of inserting a duplicate.
   • When fingerprintInput is omitted and faultLocusEntityId is a span id, the collector resolves that span's service, operation (span name) and error signature, so every incident failing the same way on the same service shares one fingerprint (and one circuit-breaker budget) instead of minting a new one per span id. Unknown spans fall back to treating the id as a service name.
8. propose_remediation_action & claim_remediation_grant (write) — Evaluate the pre-execution policy matrix and manage the CAS execution grant lifecycle (claim, complete, fail). propose_remediation_action also accepts optional webhookUrl and webhookSecret (see below).

Policy matrix and statuses

causalProofTier and the requested autonomy decide effectiveAutonomy:

• A caller can only lower the result: requestedAutonomy: "human_approved_write" turns an autonomous grant into a human-approval proposal, and read_only / suggested_action never produce a grant (a human can still approve the parked proposal).
• A circuit breaker blocks the fourth live proposal of the same action kind for one finding fingerprint within an hour (rejected and expired proposals do not count).
• effectiveAutonomy: "blocked_by_policy" is the policy decision. The resulting proposal status is rejected; a proposal that a human rejects later is also rejected, but its effectiveAutonomy keeps the original decision.
• The compare-and-swap check of expectedCurrentState happens when the grant is claimed, not when it is proposed.

Statuses a proposal can actually reach: pending_human_approval, grant_issued, claimed_executing, completed, failed_execution, execution_timed_out (the runner missed the 5-minute claim window; set by the watchdog), expired (the grant was not claimed within 5 minutes), and rejected. proposed exists in the schema but is not emitted by the current creation path.

Approval → execution handoff

A grant is a single-use grantToken (rgt_…) minted by the collector. Only its hash is stored, so the token exists in exactly one place: the response of the request that mints it. That is the propose response when the policy grants autonomy, or the POST /internal/remediations/:id/approve response after a human approves (the dashboard approval card shows it once, with a copy button). GET /internal/remediations/:id, the list route, get_remediation_proposal, and evidence bundles never return it.

The runner must claim the token within 5 minutes (grantExpiresAt) and then complete or fail within 5 minutes of the claim. There are three ways to get the token to the runner:

1. Human pastes it. Approve in the dashboard, copy the token from the card, give it to the runner, which calls claim_remediation_grant (operation: "claim").
2. Callback stored at propose time. Give propose_remediation_action (or POST /internal/remediations/propose) a webhookUrl and optional webhookSecret (secret in the HTTP body). When the proposal is parked for human approval the callback is stored on it; the approval card shows "Grant will also be delivered to <host>" and GETs expose only callbackHost, never the URL path, query or secret. When a human approves without supplying a callback, the collector posts the grant to the stored callback. The stored callback is cleared as soon as the proposal is approved or rejected. The secret is stored as supplied (the collector has no at-rest encryption helper), so treat the database as sensitive; it is never returned by any route.
3. Callback on the approve request. POST /internal/remediations/:id/approve with {"webhookUrl": "...", "secret": "..."} (a human dashboard session is required). The web approval card sends these too: its confirm step has an optional "Deliver grant to webhook URL" field and a password-type secret field, and shows the stored callback host as read-only text ("stored at propose time"). It wins over a stored callback. The same fields on an autonomous propose deliver the grant immediately.

Webhook delivery is an HMAC-SHA256 signed POST (X-Obs-Signature-256: sha256=<hex of the raw body> when a secret is set) with event: "autosre.remediation.grant_issued", proposalId, status, grantToken, and the incident context. It is SSRF-guarded: private or unresolvable hosts are refused unless the operator sets ALERT_WEBHOOK_ALLOW_PRIVATE=true, redirects are never followed, and each attempt times out after 5 seconds.

Retries and grantDelivery. A grant delivery is attempted up to 3 times (500 ms, then 1500 ms backoff) on a network error, timeout or 5xx; a 4xx or a redirect is final and not retried. The approve and propose responses carry grantDelivery: { attempted, delivered, attempts, host, error? } (the host only, never the URL query or secret). The last outcome is also stored on the proposal and returned by GET as callbackDelivery: { status, attempts, lastError, at }; the approval queue shows a "delivery failed" badge when status is failed. The stored callback itself is still cleared on approve.

When delivery fails. The approval is not undone and the response still contains the token. The approval card shows "Webhook delivery failed after N attempts: <error>. Copy the token below and hand it to the runner." Copy the token and give it to the runner within the 5-minute grant window. If the window passes (or the response was lost), reject or let the grant expire and propose again with a new idempotencyKey.

Lost responses. Replaying propose with the same idempotencyKey returns the existing proposal and its current status but grantToken: null; approving a proposal that is already grant_issued returns 409. If the token never reached the runner, reject the grant_issued proposal (or wait for it to become expired; both free the circuit-breaker slot) and propose again with a new idempotencyKey. An unclaimed grant never executes on its own.

Verifying locally with the E2E testbed

You can run the full 10-stage live D1 + Wrangler testbed (Scenario D) locally with a single command:

pnpm e2e:autosre

The script needs jq, curl, and wrangler on your PATH and builds @obsunified/cli first. It creates an isolated local D1 database under .dev/e2e-autosre/d1 (your normal apps/collector/.wrangler state is left untouched), applies every migration, starts wrangler dev on port 8795 against it, seeds Scenario D (canary rollout v2.4.0 vs. v2.4.1, stack-trace overlap on src/checkout/charge.ts, and paired counterfactual eval runs; change chg_scenario_d_deploy_241, incident inc_scenario_d, finding fnd_scenario_d), and verifies the AutoSRE endpoints end-to-end. The ingest key is read from apps/collector/.dev.vars.