Find where the work first went wrong.

Observed symptom: The answer cites a policy, but the policy does not support the decision for this case.

Possible mechanism

Retrieval selected an old version, omitted a relevant exception, or lost a constraint while condensing context.

Inspect first

Source IDs, versions, timestamps, retrieval results, the actual context supplied, and the case facts used by the decision.

Contain the effect

Hold affected decisions, identify the exposed case cohort, and route uncertain cases to an owner with authoritative evidence.

Repair owner

The information owner and retrieval engineer establish which records apply and why.

Durable repair

Repair source lifecycle, filtering, context assembly, and contradiction handling. Keep factual gaps explicit instead of inviting the model to complete them.

Misleading shortcut

Adding more documents without checking relevance, freshness, or which passages reached the model.

Regression proof

Exercise expired policy, conflicting records, missing exceptions, and long-context cases; verify both cited support and the resulting decision.

Observed symptom: The tool returns success, but the intended update is absent or attached to another record.

Possible mechanism

A vague tool contract confuses submission with completion, accepts ambiguous identifiers, or hides partial results.

Inspect first

Tool schema and version, exact arguments, resolved record identity, raw response status, and downstream receipts.

Contain the effect

Suspend the affected write path and reconcile attempted operations against the destination system before retrying.

Repair owner

The integration owner defines operation semantics and the downstream system's acceptance evidence.

Durable repair

Use explicit identifiers, validated parameters, clear partial/unknown states, and independently checkable results. Make tool descriptions match actual behavior.

Misleading shortcut

Renaming a tool or asking the model to be careful while preserving an ambiguous API contract.

Regression proof

Test wrong IDs, partial batches, delayed completion, empty results, and valid alternative tool paths against actual final state.

Observed symptom: An action crosses the intended account, record, or approval boundary.

Possible mechanism

A broad service identity, trusted retrieved instruction, or reusable approval lets the agent do more than the current task permits.

Inspect first

Caller identity, authorization decisions, target records, approval parameters, tool access, and the untrusted content encountered.

Contain the effect

Disable the exposed capability through the incident process, preserve restricted evidence, and assess affected records and recipients.

Repair owner

Security and system owners determine containment and the required authority for each effect.

Durable repair

Enforce scope outside the model and bind approval to the exact action. Revalidate changed targets or parameters before execution.

Misleading shortcut

Treating a stronger refusal prompt, model confidence score, or valid JSON as an authorization control.

Regression proof

Test cross-account targets, revoked access, altered approvals, and injected instructions; denied actions must leave no unauthorized effect.

Observed symptom: The release passed its evaluation, yet users repeatedly need to rescue nominally successful tasks.

Possible mechanism

The grader checks answer style, one favorable trial, or a narrow demo cohort instead of completion and residual work.

Inspect first

Task eligibility, test environments, grader logic, repeated trial outcomes, production exceptions, and reviewer corrections.

Contain the effect

Limit exposure for the affected task class and inspect a representative sample of supposedly successful work.

Repair owner

The product owner and evaluation lead agree what acceptable completion and escalation mean.

Durable repair

Add outcome checks, calibrated human review where judgment is needed, and regression cases for discovered failures. Keep failure and abandonment in reporting.

Misleading shortcut

Relaxing a failing assertion without proving that it rejects a valid outcome, or tuning only against the same known examples.

Regression proof

Show the repaired behavior across repeated and held-out cases while retaining earlier capabilities and reporting review effort.

Observed symptom: A timeout is followed by duplicate work, or the agent loops while the queue and cost grow.

Possible mechanism

Retries lack operation identity, the integration cannot resolve uncertain writes, or task limits do not stop repeated calls.

Inspect first

Request IDs, attempt history, receipts, timeout boundaries, queue state, latency, and the integration's retry contract.

Contain the effect

Pause unsafe retries, inspect destination state, and assign ambiguous operations for reconciliation.

Repair owner

The service owner controls recovery, retry policy, and the handling of already accepted work.

Durable repair

Use supported idempotency semantics, bounded attempts, explicit unknown states, and a recovery path for partial effects.

Misleading shortcut

Assuming a timeout means no action occurred, or assuming that rolling back application code reverses external writes.

Regression proof

Inject lost responses, duplicate requests, late completion, and dependency outages; inspect effects, limits, and recoverability.

Observed symptom: The agent correctly escalates, but the case remains untouched while reports count it as complete.

Possible mechanism

No person or queue accepts the exception, the recipient lacks evidence, or responsibility ends at tool submission.

Inspect first

Assignment and acknowledgement records, queue age, handoff contents, owner coverage, and the definition of completion.

Contain the effect

Identify stranded cases, assign an accountable owner, and correct completion reporting for the affected cohort.

Repair owner

The operating owner decides who accepts, resolves, and closes each exception.

Durable repair

Define acknowledgement, required evidence, escalation timing, and closure criteria in the workflow. Include human capacity in the operating plan.

Misleading shortcut

Adding another notification without establishing who must act and how acceptance is recorded.

Regression proof

Exercise unavailable owners, rejected handoffs, missing evidence, and reopened cases; prove each stays visible until accepted or resolved.

Observed symptom: Behavior degrades after a model, prompt, tool, policy, or data change that looked minor in isolation.

Possible mechanism

The release record does not identify all dependencies, or changed assumptions bypass regression and owner review.

Inspect first

Last accepted configuration, current component versions, source changes, release history, and outcome differences by task cohort.

Contain the effect

Bound exposure and compare against the last accepted system; withdraw a change only where that recovery path is valid.

Repair owner

The release owner coordinates engineering, information owners, and the people accepting workflow outcomes.

Durable repair

Record the complete tested configuration and retest when material dependencies change. Track corrective actions to closure after incidents.

Misleading shortcut

Blaming the latest model without checking tool, source, permission, or workflow changes made at the same time.

Regression proof

Replay the incident safely, run the relevant regression suite, and monitor a limited restart against explicit withdrawal conditions.