The first hour after an AI agent makes a damaging change

TL;DR

Imagine an alert that says an AI agent changed a customer record it should not have touched. The first instinct may be to shut everything down. Sometimes that is necessary. But a response also has to answer what the agent did, what authority it used, whether it is still acting, and how to restore the affected systems without erasing the evidence. The first hour is about stopping ongoing harm while building a reliable account of what happened.

The first hour of an agent incident. Incident runbook: Stop the run, Preserve the trail, Find the blast radius, Recover via owner, Re-enable when safe

Start with containment, while preserving the trail

Treat this as a tabletop scenario. The exact response depends on the systems involved, but the sequence below gives a team something concrete to rehearse.

If the agent may still be acting, stop the specific run or block the affected tool path first. Disable the agent’s active session or narrowly revoke the credential it used if that is the fastest reliable containment. If the identity or credential is shared across multiple workflows, the blast radius may require broader containment; make that decision with the system owner and incident lead.

At the same time, record the details needed to reconstruct the activity: agent and session identifiers, user or service identity, tool calls, arguments and results, timestamps, credential or connection used, and the state of affected records. Preserve the relevant application logs as well as the agent trace. Keep a copy in a location where the agent cannot modify it. Containment matters, but so does knowing what you are containing.

NIST’s incident response guidance places response within ongoing cybersecurity risk management, with preparation and recovery treated as part of the same work. For AI systems, the NIST AI Risk Management Framework also calls for planned incident response, recovery, change management, and clear responsibilities. An agent incident needs those ordinary operational disciplines, plus a clear view of the chain from instruction to tool call.

Establish what happened before you retry anything

Identify the exact action that raised the alert. Was it a write, a message, a permission change, or a tool call that returned an error? Which account or system record changed? Was the action initiated by a person, an agent, or an upstream workflow? Did the agent retry, call a second tool, or hand work to another agent?

Do not assume the first visible action is the only one. A single user request may create several tool calls, and a result can trigger a follow-up action. Search the surrounding time window and related identities for the same session, credential, workflow, and target system. Where the logs allow it, separate reads from writes and completed calls from failed or retried calls.

Avoid replaying the original request while the scope is unclear. If a tool call was partially successful, repeating it may duplicate the effect: sending a second message, creating another record, or applying a change twice. Check the state of the downstream system before deciding whether to retry, reverse, or leave the operation alone.

Work out the blast radius

The first question is not only “what changed?” It is also “what could this identity change?” Review the permissions attached to the agent, the user or service account behind it, the integration connection, and any shared credentials. Check whether the same identity was used by other runs or workflows. A narrow incident can become an organization-wide one if several agents share a credential or an over-broad integration.

For each affected system, identify an owner who can confirm the actual state and advise on recovery. Preserve before-and-after values where available. If sensitive information may have been read or sent, involve the organization’s security and privacy leads and follow its notification procedures. Do not infer exfiltration from a tool call alone; confirm what left the system and where it went.

Recover through the system that owns the data

Recovery belongs with the team responsible for the affected system. Use its supported rollback or correction mechanism, and verify the restored state there. If an agent updated a CRM record, for example, the CRM owner should confirm whether restoring the previous value is safe and whether downstream automations already acted on the change.

Keep the agent disabled or limited until the cause is understood. A rollback fixes state; it does not fix a permission that allowed the unintended action. Before re-enabling the workflow, narrow its access if possible, remove or repair the triggering instruction or integration, and run a safe test that covers the failure path.

Make the runbook answer these questions

Before putting an agent into production, an incident responder should be able to find:

Who can stop an active run, revoke its access, and contact the owner of each connected system?
Where are the agent’s session trace, tool arguments and results, identity, and timestamps recorded?
Can responders disable one agent or connection without taking unrelated workflows offline?
How can they check whether a tool call partially succeeded or was retried?
Who decides whether data should be restored, and how is that restoration verified?
What conditions must be met before the agent is enabled again?

If those questions take an hour to answer during a tabletop exercise, the incident will take longer when it is real. An agent that can change company data needs an owner, a way to stop its access, and a record detailed enough to reconstruct its actions. That preparation is less dramatic than an incident plan written after the first bad change. It is also what makes a response possible.