How Circle K accelerated DevOps incident triage to under 15 minutes with Amazon Bedrock

This post was co-written with Marek Klatowicz, Cloud DevOps Engineer at Circle K. Many DevOps teams know the incident triage pattern: An alert fires, and before anyone can fix the problem, an engineer spends 45–60 minutes figuring out what happened. They pull logs, check service health, and correlate symptoms across a handful of tools. All of this happens before remediation can begin. At Circle K, this challenge scaled with the business. The company operates more than 17,200 convenience and fuel retail stores across 27 countries. Behind the storefronts, a cloud-native infrastructure supports point-of-sale systems, loyalty applications, and electric vehicle (EV) charging applications running on AWS. As the company expanded its digital operations, the triage overhead grew with it. More services meant more alerts, and each one required the same manual investigation before remediation could begin. To address it, Circle K built an automated triage system using Amazon Bedrock , a fully managed service that makes foundation models (FMs) available through a single API. For model availability by AWS Region, refer to Supported models by AWS Region in Amazon Bedrock . The system uses an agentic AI architecture to investigate each incident autonomously and deliver a structured summary to Slack within minutes of an alert firing. Instead of raw alerts, on-call engineers now receive actionable context, so they can act immediately rather than spending an hour figuring out what happened. Since deploying this system, Circle K has reduced mean triage time to under 15 minutes, a 67–75 percent reduction based on the team’s internal incident-response metrics. In this post, we describe the architecture for both the initial Amazon Bedrock Flows implementation and the migration to Amazon Bedrock AgentCore , and share lessons learned from scaling the system. The challenge of triage overhead Circle K’s DevOps team monitors infrastructure across multiple AWS accounts using Grafana. A typical day brings alerts for container restarts, elevated error rates, latency spikes, and failed deployments. Each alert is a useful signal, but as the system expanded, the team wanted to help on-call engineers quickly see which signals needed attention, with the right context. When an alert came in, the triage involved a familiar sequence of manual steps: reviewing recent logs in Amazon CloudWatch , examining application-level traces in Amazon OpenSearch Service , checking Amazon Elastic Container Service (Amazon ECS) task status, and cross-referencing symptoms with internal runbooks in GitHub. This gave engineers the information they needed, but it also showed how much repetitive context gathering would be required as more services were onboarded. The team recognized two areas where a proactive investment could help. First, the same log queries, health checks, and runbook lookups would be repeated for every incident. Second, the depth and speed of triage could vary with the on-call engineer’s familiarity with the affected service. As alert volume and service ownership grew, switching between multiple tools for each investigation could slow response and make it harder to apply the team’s best practices consistently. Circle K’s DevOps team therefore started a research and development (R&D) initiative to explore how AI could support monitoring and incident triage. The goal was to automate repetitive context gathering and provide structured, actionable findings, while supporting engineers’ operational judgment during an investigation. Why Circle K chose Amazon Bedrock The team evaluated several approaches before building, including open source agent frameworks on self-managed infrastructure. That approach would have meant owning model hosting, prompt versioning, tool orchestration, and security controls. With Amazon Bedrock, the team avoided that overhead. They could access foundation models through a single API and swap foundation models without rewriting code. Because Circle K already ran on AWS, Amazon Bedrock fit into existing AWS Identity and Access Management (IAM) policies, Amazon Virtual Private Cloud (Amazon VPC) configurations, and logging pipelines. Two managed capabilities covered orchestration and retrieval. Amazon Bedrock helped with agent orchestration, giving the agents native tool use to call AWS APIs, query databases, and retrieve documents without custom orchestration code. Amazon Bedrock Knowledge Bases, the fully managed Retrieval Augmented Generation (RAG) capability, handled ingestion and retrieval of internal product cards, so the team did not need to run a separate vector database. This let the DevOps team focus on prompt design and tool definitions. Data privacy completed the picture. Circle K’s incident data includes operational metrics, infrastructure details, and logs that can contain sensitive information. Amazon Bedrock does not share customer data with model providers or use it to train foundation models . For a team feeding production logs and internal service details into the system, this data handling approach met Circle K’s security requirements. Solution overview This section describes the initial architecture, which the team built on Amazon Bedrock Flows. A later section covers the migration to AgentCore. The solution uses Amazon Bedrock Flows as the orchestration layer, coordinating a set of specialized agents that each handle a different part of the analysis. The pipeline starts when a Grafana alert publishes a notification to Amazon Simple Notification Service (Amazon SNS) , which forwards it to an Amazon Simple Queue Service (Amazon SQS) queue. An AWS Lambda function polls the queue, extracts the alert payload, and invokes the Flow. Inside the Flow, two agents are invoked in parallel. The AWSResourceExplorerAgent queries infrastructure across multiple workload accounts managed with AWS Organizations . It checks Amazon ECS task status, Application Load Balancer target health, and CloudWatch metrics and logs. At the same time, the OpenSearchExplorerAgent queries Amazon OpenSearch Service for application-level logs and distributed traces. This agent runs separately because OpenSearch returns large volumes of log data. Isolating it keeps each agent’s context window manageable. It also has exclusive access to the knowledge base, which contains product cards describing each service and its dependencies. After both agents return their findings, the GitHubExplorerAgent runs. It receives output from the previous two agents and finds the affected service’s GitHub repositories. Then it searches their application code, infrastructure definitions, and recent commits to pinpoint the source of the problem and propose a concrete fix. Running it after the parallel agents means it knows exactly what to search for rather than performing a broad query. All three outputs feed into an AnalysisPrompt node, which is a single model call with no tools attached. It receives the AWS resource data, the OpenSearch logs, and the GitHub context, then produces a structured summary: probable root cause, supporting evidence, affected service and owner, and recommended remediation steps. The prompt output uses a consistent format that the on-call team can scan in under 2 minutes. The Flow returns this summary to the invoking Lambda function, which posts it to a Slack channel. The entire process from alert to Slack message takes 5–15 minutes depending on incident complexity. The following diagram shows this architecture end to end, from the initial Grafana alert to the final Slack summary. Figure 1: Circle K automated incident triage architecture using Amazon Bedrock Flows Example triage A Grafana alert: HTTP 5xx error rate on a service exceeds 5% for 3 consecutive minutes. The AWSResourceExplorerAgent checks ECS task health in the target account and finds the database connection pool exhausted, with the ECS service attempting restarts. The OpenSearchExplorerAgent returns application logs sh