Automate remediation post AWS DevOps Agent investigation

Reducing the time between incident detection, investigation, and remediation is a critical priority for organizations running production workloads on AWS. When an issue arises, on-call engineers often need to quickly diagnose the problem across application components, identify the root cause, and apply the fix, often in the middle of the night. AWS DevOps Agent , an AI powered agent that autonomously triages incidents all day based on correlated metrics, logs, and application topology, addresses the first part of this priority by providing root cause analysis (RCA) and recommended actions for resolution. However, to retain control and help prevent unintended changes, organizations typically keep their observability agents, including AWS DevOps Agent, in an observe-and-report mode, where the agent diagnoses issues but doesn’t modify production resources directly. In this post, we demonstrate how to use AWS Lambda Durable Functions , a capability of AWS Lambda, Amazon EventBridge , and Amazon Bedrock to create an automated remediation workflow that complements AWS DevOps Agent to complete the issue resolution step. This workflow transforms investigation summaries into pre-validated fixes ready for single approval action, helping you reduce mean time to resolution (MTTR) and free your on-call engineers from repetitive diagnostic work. Solution overview With AWS Lambda Durable Functions , you can build resilient multi-step applications and AI workflows that can run for up to one year without requiring you to manage additional infrastructure or write custom state management and error handling code. These functions automatically checkpoint progress, suspend execution during long-running tasks, and recover from failures while maintaining reliable progress despite interruptions. The following diagram illustrates the solution architecture. Figure 1: Automated remediation workflow from an AWS DevOps Agent investigation through Amazon EventBridge, AWS Lambda, and Amazon Bedrock, with optional human approval before changes reach the infrastructure The workflow consists of the following steps: AWS DevOps Agent completes an incident investigation and emits an event containing symptoms, findings, and root cause analysis. Amazon EventBridge receives the investigation completion event and triggers the devops-agent-trigger function with the investigation content. The Lambda function packages the investigation summary and invokes the devops-agent-remediation-durable durable function. The durable function sends the investigation context to Amazon Bedrock, which analyzes the findings and looks for applicable remediations. Amazon Bedrock identifies and lists the available remediation tools from a curated allowlist of approved Lambda functions: devops-agent-lambda-tool . Amazon Bedrock proposes specific remediation actions based on the investigation findings and the available tools. For read-only actions, the durable function runs the remediation tools autonomously. For infrastructure changes, the workflow suspends and waits for human approval before proceeding. After approval, the durable function applies the remediation actions to the infrastructure using the selected tools. The durable function runs as an agentic loop , iteratively calling Amazon Bedrock, executing approved tools, and feeding results back into the conversation until the remediation is complete. To keep automated actions safe and auditable, the orchestrator enforces a curated allowlist of remediation tools. Each tool is a purpose-built Lambda function that performs a specific, well-scoped action, such as reading a Lambda function configuration or updating an AWS Identity and Access Management (IAM) policy statement. Amazon Bedrock can only select and invoke tools from this approved set, which keeps the scope of automated actions controlled. The workflow further distinguishes between read-only operations and mutating operations. Read-only tools run autonomously without human intervention. Mutating actions that would modify infrastructure state cause the durable function to suspend execution and wait for human approval. This is where AWS Lambda Durable Functions provide a key advantage. The function checkpoints its progress and pauses for minutes, hours, or even days without consuming compute resources, then resumes exactly where it left off after it receives the approval signal. By the time the on-call engineer engages, the system has already gathered relevant configurations, correlated the root cause with available remediation actions, and prepared a set of pre-validated changes ready for one-click approval. The current implementation uses an approve or reject signal. Because the callback accepts an arbitrary JSON payload, you can extend the approval to carry parameter overrides or reviewer observations. These can be fed back into the Bedrock conversation to refine the proposed remediation before execution. In the following sections, we walk through the implementation details, including the Amazon EventBridge rule configuration and the durable function orchestration logic. We then deploy the solution using the AWS Cloud Development Kit (AWS CDK). Prerequisites Before deploying this solution, verify that you have the following prerequisites: The AWS Command Line Interface (AWS CLI) installed and configured. Python 3.14 or later. The AWS CDK installed. An active AWS DevOps Agent space . (Optional) Kiro with the Agent Toolkit for AWS . The Agent Toolkit gives Kiro secure access to AWS APIs through a managed MCP Server with IAM-based access controls. If you use Kiro, the incident simulation, deployment, and cleanup steps in this post can be completed with natural language prompts instead of running CLI commands manually. To set it up, add the AWS MCP Server to ~/.kiro/settings/mcp.json ( setup instructions ). The repository includes a Kiro rules and agents file that gives Kiro the project context, deployment sequence, and safety conventions automatically. Simulate the incident To demonstrate the end-to-end workflow, we simulate a common scenario: a Lambda function that exceeds its configured timeout. This gives AWS DevOps Agent a real incident to investigate and sets the remediation workflow in motion. To keep the focus on the remediation solution itself, the steps to create and invoke this test function are kept in the repository . It includes a ready-to-use devops-agent-timeout function and step-by-step instructions to deploy it, invoke it, and confirm the timeout error in Amazon CloudWatch Logs. For the full walkthrough, see the “Simulate the incident” section of the README . After the function is deployed and has produced at least one timeout error, you’re ready to start an investigation with AWS DevOps Agent. Deploy the solution using the AWS CDK Complete the following steps to deploy the remaining solution resources: Kiro: If you have Kiro with the Agent Toolkit for AWS configured (see Prerequisites), open the cloned repository in Kiro and ask: “ Set up the Python environment and deploy the CDK stack. Show me what resources will be created before deploying. ” Kiro reads the project rules from the repository, sets up the virtual environment, installs dependencies, and shows you the planned resources before deploying. It confirms each infrastructure change before executing, following the same human-in-the-loop pattern that the remediation solution itself uses. To deploy manually, follow these steps. Clone the AWS CDK code hosted on GitHub: $ git clone https://github.com/aws-samples/sample-automate-remediation-post-devops-agent-investigation.git Navigate to the directory sample-automate-remediation-post-devops-agent-investigation : $ cd sample-automate-remediation-post-devops-agent-investigation Bootstrap the AWS CDK. This is required the first time you use the AWS CDK in a specific AWS environment (a combination of an AWS account and AWS Region). $ cdk bootstrap Deploy the stack: $ cdk deploy The AWS C