# AutoSRE causal ontology (/docs/autosre)



When a production incident pages an AI agent or human SRE, finding *where* an error surfaced is only the first step. Authorizing an automated rollback, feature-flag flip, or config revert requires proving *what change caused the incident*—and knowing when telemetry is insufficient to prove causality safely.

obs-unified's **AutoSRE causal ontology** (RFC 0013) adds three missing layers to the unified telemetry graph:

1. **Change Event Ledger (`change_events`)** — Every deploy, rollback, feature-flag evaluation, config update, and agent prompt release is recorded and linked to runtime spans via OpenTelemetry resource and span attributes (`service.version`, `vcs.commit.id`, `obs.config.digest`, `feature_flag.*`).
2. **Dual Epistemic Model (`relationshipCertainty` vs. `causalProofTier`)** — Graph link provenance (how sure we are two nodes are connected) is strictly separated from causal proof strength (how sure we are a candidate change caused the symptom).
3. **Incident Memory & Pre-Execution Remediation Gate** — Findings are fingerprinted across incidents, counterfactual eval runs verify baseline-fail + candidate-pass pairs, and a stateful lease/grant gate enforces Compare-And-Swap (CAS) target states and post-execution watchdog recovery before any autonomous write executes.

***

## Real-world production scenarios AutoSRE solves [#real-world-production-scenarios-autosre-solves]

Standard APM tools and naive LLM incident responders fail in five recurring production scenarios. AutoSRE is architected specifically to solve each of them:

### 1. Progressive rollout vs. the traffic-share fallacy [#1-progressive-rollout-vs-the-traffic-share-fallacy]

**The challenge:** Suppose release `v2.4.1` is rolled out to handle **95% of production traffic**, while `v2.4.0` serves the remaining **5%**. During an outage caused by an upstream payment gateway or expired certificate, `v2.4.1` will naturally account for **95% of all errors**. A naive agent that checks "which version has the most errors?" will blame and roll back `v2.4.1`, even though both versions have the exact same per-request error rate.

**How AutoSRE solves it:** The blast-radius evaluator (`GET /internal/changes/:id/blast-radius` / `get_change_blast_radius`) computes **traffic-normalized cohort error-rate lift** across a minimum sample size (`N >= 50` requests per cohort by default, configurable via `minSample`):

```text
ErrorRate(v) = error_requests(v) / total_requests(v)
RelativeRisk = ErrorRate(v_new) / max(ErrorRate(v_prev), 0.001)
```

* If `v2.4.1` has a 12% error rate and `v2.4.0` has a 0.08% error rate (absolute rate difference `>= 5%` or `p95Ratio >= 2.5`, **and** `RelativeRisk >= 5.0` or `p95Ratio >= 2.5`), the cohort verdict is **`isolated_to_new_version`**, promoting the finding to `cohort_rate_isolated`.
* If an instant 100% cutover occurs (`oldCohort.requestCount < N` but pre-change baseline `preBaseline.requestCount >= N&#x60;) and the same thresholds hold against pre-change baseline, the cohort verdict is &#x2A;*`sequential_post_cutover_regression`** (`corroborated_candidate`).
* If both `v2.4.1` and `v2.4.0` fail (`errorRate >= 5%&#x60; on both), AutoSRE distinguishes between noisy-neighbor contention (&#x2A;*`shared_degradation_new_version_aggressor`*&#x2A;) and an external upstream outage (&#x2A;*`shared_degradation_external_cause`**).

### 2. Noisy-neighbor shared database contention [#2-noisy-neighbor-shared-database-contention]

**The challenge:** Sometimes a bad deploy (`v2.4.1`) introduces an unindexed query or connection pool leak that exhausts a shared PostgreSQL instance (`db:prod:postgresql:orders-primary:5432/orders`). Once the database saturates, requests on the old version (`v2.4.0`) *also* start failing with connection timeouts. If an agent treats "old version is also failing" as proof that the deploy is innocent, it will miss the real culprit.

**How AutoSRE solves it:** Instead of refuting the deploy when errors are shared across versions, AutoSRE runs a **shared-resource contention join** across canonical resource identities (`environment + namespace + kind + name`). When `v2.4.1` exhibits a `>= 3x` downstream call fanout or duration amplification (`fanoutRatio >= 3.0 || durationRatio >= 3.0`) on a shared database or queue node while `v2.4.0` fails downstream on that same node, AutoSRE emits `cohortVerdict: "shared_degradation_new_version_aggressor"` (`corroborated_candidate`) and populates `sharedResourceAggressors` with the contending trace/span (`contended_on_shared_resource` / `induced_shared_resource_contention`).

### 3. Structural code overlap without causal proof [#3-structural-code-overlap-without-causal-proof]

**The challenge:** A deploy touches 40 files including `src/checkout/charge.ts`, which appears in a failing stack trace (`code.filepath = "src/checkout/charge.ts"`). That overlap is an exact *structural* link (`relationshipCertainty: "exact"`), but it does **not** prove the commit caused the outage—the commit might have only edited a comment or log line in `charge.ts`.

**How AutoSRE solves it:** AutoSRE separates **Link Provenance** (`relationshipCertainty`: `"exact" | "inferred" | "heuristic"`) from **Causal Proof Tier** (`causalProofTier`):

| Dimension                   | Values                                                                                          | Meaning                                                                                              |
| --------------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| **`relationshipCertainty`** | `exact`, `inferred`, `heuristic`                                                                | How was the graph edge constructed? (`exact` = shared trace/span ID or exact `code.filepath` match). |
| **`causalProofTier`**       | `counterfactual_verified`, `cohort_rate_isolated`, `corroborated_candidate`, `unverified_locus` | Has the root-cause hypothesis been proven via paired counterfactual replay or cohort rate isolation? |

Code-path overlap (`touched_same_code_path`) alone is capped at `causalProofTier: "corroborated_candidate"`. It can never authorize an autonomous rollback or flag flip without cohort rate isolation (`cohort_rate_isolated`) or counterfactual replay (`counterfactual_verified`).

### 4. Paired baseline-fail + candidate-pass counterfactual verification [#4-paired-baseline-fail--candidate-pass-counterfactual-verification]

**The challenge:** Running a synthetic check that passes after a fix is applied—or seeing error rates drop when traffic subsides—does not prove the fix solved the bug. If the reproducer didn't fail against the broken baseline first, a passing test is meaningless.

**How AutoSRE solves it:** Upgrading a finding to `counterfactual_verified` (`POST /internal/findings/:id/counterfactual`) requires a **paired evaluation**:

1. **Baseline failure (`failingRunId`):** An `eval_runs` record against the incident's broken `serviceVersion` / `commitSha` / `candidateAgentVersion` that reproduces the failure (`passed = 0`) with a matching `input_payload_hash` and `failure_fingerprint`.
2. **Candidate pass (`passingRunIds`):** `K` passing `eval_runs` (`passed = 1`, `requiredTrials >= 1`) of the exact same `eval_case_id` and `input_payload_hash` against the candidate remediation state.

### 5. Competing alerts, state drift, and post-remediation regressions [#5-competing-alerts-state-drift-and-post-remediation-regressions]

**The challenge:** During an outage, webhook retries or multi-symptom alerts can spawn duplicate agent runs that race to roll back the same service. Or a human operator may manually deploy `v2.4.2` while an agent is still deciding to roll back `v2.4.1`.

**How AutoSRE solves it:**

* **Durable incident lease (`incident_investigations`):** Deduplicates firing alerts by `(project_id, environment, dedup_key)` (enforced at the database level via partial unique index `uq_incidents_active_dedup`) and enforces an exclusive runner lease (`POST /internal/incidents/:id/lease`, with expired leases reaped back to `open`).
* **Pre-execution CAS policy gate (`remediation_proposals`):** `propose_remediation_action` binds an action to `expectedCurrentState` (e.g. `{ version: "2.4.1", commitSha: "8f3a1b2" }`), evaluates the action-specific policy matrix (plus a 3-proposal/hour circuit breaker scoped to finding fingerprint + action kind), and either parks the proposal for **1-Click Human Approval** in the dashboard (`/#/autonomous-review`, `status: "pending_human_approval"`) or issues a 5-minute single-use `grantToken` (`status: "grant_issued"`).
* **External runner claim & collector watchdog (`claim_remediation_grant`):** The collector acts as the **stateful authorization gate**, while your external agent or CI/CD runner claims the single-use `grantToken` (`operation: "claim"`, verifying `expectedCurrentState` against `ontology_nodes`), executes the actual infrastructure rollback/flag-toggle/PR, and records `operation: "complete"` (which must be called by the claiming `runnerId` within the 5-minute claim window). Completing a proposal arms the post-remediation watchdog (`+2m / +5m / +15m` anchored to completion time, evaluated on the 1-minute `analyses-runner` cron tick or via `POST /internal/remediations/watchdog`), which requires **two consecutive healthy checkpoints** (`errorRate < 5%` with `N >= 50` comparable requests) before transitioning `recoveryStatus` to `recovered` and `causalVerificationStatus` to `operationally_corroborated` (or `still_degraded` / `refuted` / `inconclusive_low_traffic`), never conflating operational recovery with `counterfactual_verified`.

***

## Recording changes and runtime lineage [#recording-changes-and-runtime-lineage]

### 1. Stamp lineage and feature flags in your SDK [#1-stamp-lineage-and-feature-flags-in-your-sdk]

In **TypeScript / JavaScript** (`@obsunified/telemetry-sdk`, `@obsunified/analytics-sdk`, or `@obsunified/sdk`), pass `environment`, `serviceVersion`, `commitSha`, and `configDigest` (or set `OBS_ENVIRONMENT`, `OBS_SERVICE_VERSION`, `OBS_COMMIT_SHA`, `OBS_CONFIG_DIGEST`) when initializing, and call `recordFeatureFlag(...)` inside request scope:

```ts
import { initObservability, recordFeatureFlag } from "@obsunified/telemetry-sdk";

initObservability({
  serviceName: "checkout-api",
  collectorUrl: process.env.OBS_COLLECTOR_URL,
  environment: "prod",
  serviceVersion: "2.4.1",
  commitSha: "8f3a1b2c",
  configDigest: "cfg_99a1",
});

// Inside a request handler or span scope:
recordFeatureFlag("enable_legacy_tax_fallback", false, {
  providerName: "openfeature",
});
```

In **non-JS OpenTelemetry SDKs** (`Python`, `Go`, `Rust`, `Java`, `.NET`, `PHP`, `Ruby`, `Elixir`, `Swift`, `Kotlin`), set standard OpenTelemetry resource/span attributes directly (`service.version`, `deployment.environment.name`, `vcs.commit.id`, `obs.config.digest`, and `feature_flag.key` + `feature_flag.variant` or `obs.feature_flags.<key>`).

### 2. Record deploys, config changes, or flag flips from CI/CD [#2-record-deploys-config-changes-or-flag-flips-from-cicd]

Use the CLI or `POST /v1/changes` from your deployment pipeline:

```bash
pnpm dlx @obsunified/cli changes record \
  --url https://collector.example.com \
  --api-key "$OBS_INGEST_KEY" \
  --environment prod \
  --kind deploy \
  --service checkout-api \
  --version 2.4.1 \
  --previous-version 2.4.0 \
  --commit-sha 8f3a1b2c \
  --previous-commit-sha 7d2e0a1b \
  --summary "Switch tax rounding to strict decimal mode" \
  --touched-paths "src/checkout/charge.ts"
```

***

## Agent workflow with MCP tools [#agent-workflow-with-mcp-tools]

The `@obsunified/mcp-server` registers seven read tools for AutoSRE by default, plus four opt-in write tools that are only registered when `OBS_MCP_ENABLE_WRITES=true`:

1. **`get_evidence_bundle`** (`intent: "triage_incident"`, or any `incident` / `change` / `finding` anchor) — Returns a token-budgeted incident bundle with `bundle.autosre` (`faultLocus`, `candidateChanges`, `contentionCandidates`, and `priorFindings`). `priorFindings` match the fault locus by entity id **and** by recurrence fingerprint, so a failure that recurs on a different trace of the same service, route and error signature surfaces the findings and fixes recorded before.
2. **`list_recent_changes`** — Lists recent deploys, flag flips, config updates, and prompt releases scoped by `environment` and `serviceName`.
3. **`get_change_blast_radius`** — Computes traffic-normalized cohort error-rate lift (`isolated_to_new_version`, `sequential_post_cutover_regression`, `shared_degradation_new_version_aggressor`, `shared_degradation_external_cause`, `no_regression_detected`, or `insufficient_sample`), noisy-neighbor contention, and stack-frame `touchedPaths` overlap.
4. **`search_similar_findings`** — Queries historical `InvestigationFinding` records by stable `fingerprint` (`environment + originatingNodeId + normalizedOperation + errorSignature + downstreamTargetNodeId`) so recurring incidents immediately surface proven prior fixes.
5. **`list_incidents`*&#x2A;, &#x2A;*`get_incident`*&#x2A;, &#x2A;*`get_remediation_proposal`** — Read incidents (filter by `environment`, `status`, `limit`), one incident with its findings and proposals, and one proposal's status. None of them ever returns a grant token.
6. **`open_incident`*&#x2A; &#x2A;(write)* — Opens an incident, or joins the active one for the same symptom (`deduplicated: true`). Its id is the `incidentId` that proposals require.
7. **`record_investigation_finding`*&#x2A; &#x2A;(write)* — Persists a structured finding with `causalChain` triples (`degraded_after_change`, `isolated_to_version_cohort`, `touched_same_code_path`, `contended_on_shared_resource`, `induced_shared_resource_contention`, `originated_fault_in_span`, `verified_by_counterfactual`, `mitigated_by_change`, `remediated_by_agent_run`), `citedEvidenceIds`, and optional paired counterfactual eval run IDs. The collector derives `causalProofTier` server-side from `contributingChangeId`'s blast radius and counterfactual proof so uncorroborated client claims cannot inflate autonomy.
   * Eval runs named in `counterfactual` are checked first (`GET /internal/evals/runs/:id`); a missing run fails the call before anything is stored.
   * If verification fails after the finding was stored, the call still succeeds and returns `data.finding` plus `data.counterfactualError` instead of an error that hides the finding id.
   * Pass an `idempotencyKey` so a retry returns the already-recorded finding instead of inserting a duplicate.
   * When `fingerprintInput` is omitted and `faultLocusEntityId` is a span id, the collector resolves that span's service, operation (span name) and error signature, so every incident failing the same way on the same service shares one fingerprint (and one circuit-breaker budget) instead of minting a new one per span id. Unknown spans fall back to treating the id as a service name.
8. **`propose_remediation_action`*&#x2A; & &#x2A;*`claim_remediation_grant`*&#x2A; &#x2A;(write)* — Evaluate the pre-execution policy matrix and manage the CAS execution grant lifecycle (`claim`, `complete`, `fail`). `propose_remediation_action` also accepts optional `webhookUrl` and `webhookSecret` (see below).

## Policy matrix and statuses [#policy-matrix-and-statuses]

`causalProofTier` and the requested autonomy decide `effectiveAutonomy`:

| Action                                   | `counterfactual_verified` | `cohort_rate_isolated` | `corroborated_candidate` | `unverified_locus`     |
| ---------------------------------------- | ------------------------- | ---------------------- | ------------------------ | ---------------------- |
| `rollback_deploy`, `toggle_feature_flag` | `autonomous_write`        | `autonomous_write`     | `human_approved_write`   | `blocked_by_policy`    |
| `open_code_pr`                           | `autonomous_write`        | `autonomous_write`     | `autonomous_write`       | `human_approved_write` |
| `apply_config`, `promote_agent_prompt`   | `autonomous_write`        | `human_approved_write` | `human_approved_write`   | `blocked_by_policy`    |

* A caller can only lower the result: `requestedAutonomy: "human_approved_write"` turns an autonomous grant into a human-approval proposal, and `read_only` / `suggested_action` never produce a grant (a human can still approve the parked proposal).
* A circuit breaker blocks the fourth live proposal of the same action kind for one finding fingerprint within an hour (`rejected` and `expired` proposals do not count).
* `effectiveAutonomy: "blocked_by_policy"` is the policy decision. The resulting proposal `status` is `rejected`; a proposal that a human rejects later is also `rejected`, but its `effectiveAutonomy` keeps the original decision.
* The compare-and-swap check of `expectedCurrentState` happens when the grant is **claimed**, not when it is proposed.

Statuses a proposal can actually reach: `pending_human_approval`, `grant_issued`, `claimed_executing`, `completed`, `failed_execution`, `execution_timed_out` (the runner missed the 5-minute claim window; set by the watchdog), `expired` (the grant was not claimed within 5 minutes), and `rejected`. `proposed` exists in the schema but is not emitted by the current creation path.

## Approval → execution handoff [#approval--execution-handoff]

A grant is a single-use `grantToken` (`rgt_…`) minted by the collector. Only its hash is stored, so the token exists in exactly one place: the response of the request that mints it. That is the `propose` response when the policy grants autonomy, or the `POST /internal/remediations/:id/approve` response after a human approves (the dashboard approval card shows it once, with a copy button). `GET /internal/remediations/:id`, the list route, `get_remediation_proposal`, and evidence bundles never return it.

The runner must claim the token within **5 minutes** (`grantExpiresAt`) and then complete or fail within 5 minutes of the claim. There are three ways to get the token to the runner:

1. **Human pastes it.** Approve in the dashboard, copy the token from the card, give it to the runner, which calls `claim_remediation_grant` (`operation: "claim"`).
2. **Callback stored at propose time.** Give `propose_remediation_action` (or `POST /internal/remediations/propose`) a `webhookUrl` and optional `webhookSecret` (`secret` in the HTTP body). When the proposal is parked for human approval the callback is stored on it; the approval card shows "Grant will also be delivered to `<host>`" and GETs expose only `callbackHost`, never the URL path, query or secret. When a human approves without supplying a callback, the collector posts the grant to the stored callback. The stored callback is cleared as soon as the proposal is approved or rejected. The secret is stored as supplied (the collector has no at-rest encryption helper), so treat the database as sensitive; it is never returned by any route.
3. **Callback on the approve request.** `POST /internal/remediations/:id/approve` with `{"webhookUrl": "...", "secret": "..."}` (a human dashboard session is required). The web approval card sends these too: its confirm step has an optional "Deliver grant to webhook URL" field and a password-type secret field, and shows the stored callback host as read-only text ("stored at propose time"). It wins over a stored callback. The same fields on an autonomous `propose` deliver the grant immediately.

Webhook delivery is an HMAC-SHA256 signed POST (`X-Obs-Signature-256: sha256=<hex of the raw body>` when a secret is set) with `event: "autosre.remediation.grant_issued"`, `proposalId`, `status`, `grantToken`, and the incident context. It is SSRF-guarded: private or unresolvable hosts are refused unless the operator sets `ALERT_WEBHOOK_ALLOW_PRIVATE=true`, redirects are never followed, and each attempt times out after 5 seconds.

**Retries and `grantDelivery`.** A grant delivery is attempted up to **3 times** (500 ms, then 1500 ms backoff) on a network error, timeout or 5xx; a 4xx or a redirect is final and not retried. The approve and propose responses carry `grantDelivery: { attempted, delivered, attempts, host, error? }` (the host only, never the URL query or secret). The last outcome is also stored on the proposal and returned by `GET` as `callbackDelivery: { status, attempts, lastError, at }`; the approval queue shows a "delivery failed" badge when `status` is `failed`. The stored callback itself is still cleared on approve.

**When delivery fails.** The approval is not undone and the response still contains the token. The approval card shows "Webhook delivery failed after N attempts: `<error>`. Copy the token below and hand it to the runner." Copy the token and give it to the runner within the 5-minute grant window. If the window passes (or the response was lost), reject or let the grant expire and propose again with a new `idempotencyKey`.

**Lost responses.** Replaying `propose` with the same `idempotencyKey` returns the existing proposal and its current status but `grantToken: null`; approving a proposal that is already `grant_issued` returns 409. If the token never reached the runner, reject the `grant_issued` proposal (or wait for it to become `expired`; both free the circuit-breaker slot) and propose again with a new `idempotencyKey`. An unclaimed grant never executes on its own.

***

## Verifying locally with the E2E testbed [#verifying-locally-with-the-e2e-testbed]

You can run the full 10-stage live D1 + Wrangler testbed (`Scenario D`) locally with a single command:

```bash
pnpm e2e:autosre
```

The script needs `jq`, `curl`, and `wrangler` on your PATH and builds `@obsunified/cli` first. It creates an isolated local D1 database under `.dev/e2e-autosre/d1` (your normal `apps/collector/.wrangler` state is left untouched), applies every migration, starts `wrangler dev` on port 8795 against it, seeds `Scenario D` (canary rollout `v2.4.0` vs. `v2.4.1`, stack-trace overlap on `src/checkout/charge.ts`, and paired counterfactual eval runs; change `chg_scenario_d_deploy_241`, incident `inc_scenario_d`, finding `fnd_scenario_d`), and verifies the AutoSRE endpoints end-to-end. The ingest key is read from `apps/collector/.dev.vars`.