Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

Developers want AI coding assistants with minimal setup. With a large language model (LLM) gateway and LiteLLM, they can quickly access foundation models (FMs) available on Amazon Bedrock while you gain visibility and usage control. Platform teams need to monitor latency and errors, finance teams need to attribute inference costs to teams and projects, and security teams need consistent access controls. An LLM gateway can help address these needs by providing a centrally managed access point to the models that power coding assistants. In this post, we show how to build a self-service LLM gateway using Amazon Bedrock Application Inference Profiles with LiteLLM running on AWS Fargate to abstract Amazon Bedrock behind an OpenAI-compatible API. With this pattern, developers can access foundation models through a single API with a dedicated key while you gain per-user cost controls and model performance observability. By the end, you will understand the pattern well enough to build your own version from the illustrative snippets and AWS Cloud Development Kit (AWS CDK) constructs. This post uses Claude Sonnet and Claude Haiku models for example purposes. For a list of models and availability by AWS Region, see Supported models by AWS Region in Amazon Bedrock . This LLM gateway pattern follows a straightforward request flow from the developer, through the LiteLLM layer, to Amazon Bedrock and the underlying FM. The following diagram illustrates the production topology behind that flow. For simplicity, we don’t cover the addition of Amazon Bedrock Guardrails, such as content filtering and grounding, but you would normally include appropriate responsible AI controls before deploying to production. A developer’s OpenAI-compatible client reaches the LLM gateway through Amazon Route 53, Amazon CloudFront, and an Application Load Balancer (ALB). It authenticates to LiteLLM on Fargate with a per-user API key. Behind that, the request draws on Amazon Relational Database Service (Amazon RDS) (PostgreSQL) for virtual-key and spend state. It also interacts with Amazon ElastiCache for Valkey as a cache, and AWS Secrets Manager for secrets. LiteLLM then ultimately invokes Amazon Bedrock Application Inference Profiles using a scoped AWS Identity and Access Management (IAM) task role. Amazon CloudWatch captures requests per minute (RPM) and tokens per minute (TPM) metrics and the quota alarm. Figure 1: LLM gateway architecture (production topology) The architecture consists of these core components: LiteLLM on AWS Fargate : LiteLLM provides a proxy that exposes an OpenAI-compatible API. We run it on an AWS Fargate task with target-tracking Auto Scaling on CPU utilization. Developers point their tools at this endpoint and don’t interact with Amazon Bedrock directly. Self-service portal : LiteLLM includes a lightweight portal for admin and developer self-service. This web app can integrate with your corporate single sign-on (SSO). Using the portal, operations engineers set the spend policy while developers self-provision API keys within those parameters. Persistent and cache storage : LiteLLM uses Amazon ElastiCache for Valkey for non-persistent response caching. It also coordinates rate-limiting across AWS Fargate tasks. A separate Amazon RDS (PostgreSQL) database is the database of record for persistent state such as API key metadata and spend tracking. Amazon Bedrock Application Inference Profiles : Each model (such as Claude Sonnet and Claude Haiku) gets its own inference profile, which you manage using the AWS Cloud Development Kit (AWS CDK). With this, you get per-model quota isolation and a clean abstraction layer between LiteLLM’s configuration and the model versioning of Amazon Bedrock. Langfuse : Open source LLM observability for prompt-level telemetry, latency tracking, and debugging. It complements the infrastructure-level metrics of Amazon CloudWatch with application-level insight. By using an OpenAI-compatible protocol, developers can configure their tools, such as Claude Code, Continue, and Cursor, with a single base URL and API key. Developers then operate at a level above the Amazon Bedrock backend and can swap models, add fallbacks, or migrate model providers without touching developer configurations. Prerequisites You will need an AWS account, Amazon Bedrock model access, and appropriate IAM permissions to deploy this project. You will also need LiteLLM. The LiteLLM AI Gateway component is open source and suitable for a small pilot with some limitations. LiteLLM documents the AI Gateway as free for up to five users with a license required beyond that. LiteLLM Enterprise is a scalable, production-minded licensed version that includes the open source gateway plus single sign-on, audit logs, delegated admin roles, and support. While the free tier supports virtual keys, the self-service key issuance described in this post requires the Enterprise version. End-to-end configuration: Left to right We walk through each configuration layer from the developer’s machine to the foundation model. Step 1: Developer IDE configuration (for example, Claude Code) Developers configure their coding assistant to point to the LLM gateway. This is the only configuration a developer needs to make to their tool: { "apiProvider": "openai-compatible", "apiBaseUrl": "https://llm-gateway.internal.example.com/v1", "apiKey": "sk-user-abc123", "model": "claude-sonnet-4-20250514" } With that configuration, each developer gets their own API key from the self-service portal and drops it into their tool configuration. Step 2: LiteLLM gateway configuration Deploy LiteLLM to a container service following the LiteLLM documentation . In this example we are using AWS Fargate. This will include the portal, which you can open at http://localhost:4000/ui during local testing. The LiteLLM gateway is where you will map and enforce model names to Amazon Bedrock inference profile ARNs with per-key spend limits: model_list: - model_name: claude-sonnet-4-20250514 litellm_params: model: bedrock/arn:aws:bedrock:us-east-1:111122223333:application-inference-profile/sonnet-profile-id aws_region_name: us-east-1 max_retries: 3 timeout: 900 # 15 min for extended thinking # REQUIRED for spend tracking: when `model` is a raw inference-profile ARN, # LiteLLM can't find it in its built-in cost map and defaults the cost to $0 # which means per-key `spend` stays 0 and budgets NEVER enforce. Set the # per-token costs explicitly. model_info: input_cost_per_token: 0.000003 # example cost output_cost_per_token: 0.000015 # example cost - model_name: claude-haiku-4-5 litellm_params: model: bedrock/arn:aws:bedrock:us-east-1:111122223333:application-inference-profile/haiku-profile-id aws_region_name: us-east-1 timeout: 120 model_info: input_cost_per_token: 0.000001 # example cost output_cost_per_token: 0.000005 # example cost litellm_settings: max_budget: 250 # example per key cost budget_duration: 7d # Rolling 7-day window max_parallel_requests: 10 set_verbose: false cache: true cache_params: type: redis # Valkey is protocol-compatible, so this stays "redis" host: elasticache-endpoint port: 6379 master_key: os.environ/LITELLM_MASTER_KEY database_url: os.environ/DATABASE_URL Set on the Fargate task from AWS Secrets Manager: LITELLM_MASTER_KEY=sk-platform-admin-... DATABASE_URL=postgresql://user:password@rds-endpoint:5432/litellm Note: Don’t hardcode the master_key or database credentials in config.yaml . LiteLLM resolves os.environ/VAR_NAME references at load time, so set LITELLM_MASTER_KEY and DATABASE_URL as environment variables on the Fargate task, sourced from AWS Secrets Manager. Set the Langfuse credentials the same way in Step 5. When a developer calls claude-sonnet-4-20250514 , for example, LiteLLM resolves it to the inference profile ARN. It then authenticates to Amazon Bedrock using the AWS Fargate task’s IAM role and streams the response back in OpenAI format. Step 3: Bedrock inference profiles (CDK) With Amazon B