Pluggable Agentic AI Framework: A seven-layer composable enterprise agentic AI reference architecture on Amazon Bedrock AgentCore

“How do we run our AI agents in production at scale?” is the question every customer eventually asks. It comes bundled with several questions. How do we handle thousands of concurrent user requests securely and confidently? How do we make sure the agents stay reliable under real traffic instead of failing silently? How do we know if an agent still meets the quality and safety standards it passed during testing? What are agent security best practices? Should that be deterministic or non-deterministic? And many more. A proof-of-concept answers none of these, because it is built to show the agent model can do the task, not to survive production traffic, cost pressure, and security review. That gap between a working prototype and a system that runs reliably, securely, frugally and responsibly at scale, is where most agentic AI initiatives stall, and it is rarely the model’s fault. Every team building an agent tends to rebuild the same scaffolding from scratch: its own routing logic, its own guardrails, its own observability, all applied after the fact rather than built in from the start. The teams need a shared, production-ready architecture that every agent can build on. This post walks through a seven-layer composable reference architecture designed to take multi-agent AI applications from prototype to production scale. Composable architecture The Pluggable Agentic AI Framework is a reference architecture for multi-agent applications organized into seven layers , each with a distinct responsibility and a stable contract. The services and technologies shown in each layer reflect the reference implementation, not mandatory choices. Organizations can substitute alternatives that align with their existing infrastructure and requirements. Together, these layers offer a blueprint for building, securing, and operating agentic AI applications at production scale. Figure 1: The seven layers of the Pluggable Agentic AI Framework Data and Knowledge is the foundation of Agentic AI. An agent is only as good as the data it can access. Whether an agent is checking out a healthcare formulary, tracking inventory, or interpreting a compliance rule, it needs relevant, up-to-date information to answer correctly. This layer owns retrieval, embeddings, and access to structured and unstructured data through a clear contract and protocol: accept a query and return the most relevant information that acts as context for an agent. Amazon Bedrock Knowledge Base , Vector Engine for Amazon OpenSearch Serverless , Amazon S3 Vectors and Amazon DynamoDB are example building blocks for this foundation. Model and Inference choices should be driven by configuration, not hardwired into agent or application logic. Use smaller, faster models for simple tasks and larger models for requests that require more nuanced reasoning. It is important to conduct a thorough model evaluation for the use case you are building with realistic and representative business data. A model routing layer keeps AI agents decoupled from the model hosting environment and provider-specific APIs by relying on a consistent request-and-response contract. A proxy such as LiteLLM provides a common interface across providers, so you can switch models based on your evaluations without rewriting application logic. Orchestration determines how work is distributed, and choosing the right approach is critical. For straightforward tasks, use a single agent to handle the work from end to end. For complex, cross-domain workflows, use a supervisor to delegate specialist agents, each scoped to a specific capability and tied to a specific set of tools. Avoid building one agent that does everything. As tools, instructions, and context accumulate, the context window grows, latency increases, and accuracy can suffer. Tool sprawl sets in, which can push agents beyond their intended boundaries, leading to unintended actions and security breaches. Splitting responsibilities keeps each agent’s context focused and makes latency more predictable. You can also test each agent’s behavior, scale it, and define its security model independently. Enforce bounded autonomy: specialists should execute strictly within the tools they have been granted, and nothing more. In addition to decomposing into specific specialists and setting its boundary, decouple the specialist agents from the supervisor agents. Allow the supervisor to discover the specialists from a centralized agent registry and invoke them through a standardized protocol like MCP (Model Context Protocol) and A2A (Agent to Agent). Maintain a centralized agent registry that catalogs every agent’s tools, skills, and owner. This gives your organization the visibility it needs for governance at scale, increases the velocity of AI agent implementation through reusability, and avoids the cost of repetitive development. Consider using Strands Agents for specialists, Amazon Bedrock AgentCore harness for the orchestrator, AWS Agent Registry and Amazon Bedrock AgentCore Gateway for centralized discovery and routing. Security and Guardrails should be a primary concern, built into the design from day one. Agentic AI exposes organizations to a new class of risks that conventional security frameworks were not built to address, from prompt injection and tool misuse to memory and data poisoning. Treat all external content as untrusted by default. Authenticate every request, whether it comes from a user, another agent, or an automated workflow, through a service like Amazon Bedrock AgentCore Identity . Then use fine-grained access control to determine whether each tool call is permitted before execution. Grant agents temporary, scoped credentials instead of persistent secrets. On the data-protection side, mask sensitive data like PHI (Personal Health Information) and PII (Personally Identifiable Information) in agent and tool requests and responses, so that sensitive customer information does not appear in agent outputs or lingers in memory. For accountability, capture every significant decision in a tamper-evident audit trail, route high-impact actions through a human approval gate, and deploy circuit breakers to halt anomalous behavior before it can escalate. Consider using Amazon Bedrock Guardrails for user inputs, Amazon Bedrock AgentCore Identity and Policy in Amazon Bedrock AgentCore for authentication and access control, along with AWS Security and governance controls to bring defense-in-depth security. Application is where the end-user experience lives, whether that is a chat interface, an API surface, or an embedded widget. Keep this layer intentionally thin. Its only responsibilities should be presenting the conversation and passing requests to the orchestration layer beneath it. Resist the urge to embed business logic here. The cleaner the separation, the more easily you can iterate on the user experience without destabilizing the backend. A decision model can sit at this boundary as a fast, inexpensive check: in one call it scores whether a request is in scope, needs a human, or is urgent, returning probabilities instead of free text. Your code acts on those scores with simple thresholds, answering off-topic requests directly and flagging escalations, so the full agentic loop runs only when it adds value. When building the application layer, think about what would be needed for another AI agent to consume and interact with your AI agents along with humans. Build your external interfaces accordingly. Observability should watch every layer from day one, not after the first incident. Along with the logs and metrics, end-to-end agent traces are critical to measure whether the agent performed its task correctly. Along with the trace, collect cost and token usage per request, time to first token, whether the call is secure, and surface dashboards for latency, error rates, and spend before you go live. Proactive monitoring is cheaper than reactive firefighting. Consider using Amazon CloudWatch with Op