Deploy decision models with llama.cpp on Amazon SageMaker AI

Applications that route requests, select agent actions, or assign policy outcomes often need scores over a fixed set of choices instead of free-form text. Deploying these decision models with llama.cpp on Amazon SageMaker AI gives you a managed path for serving quantized models on CPU or GPU infrastructure. This post continues a series on specialized AWS Deep Learning Containers (DLCs) for generative AI inference. Previous posts cover WhisperX , vLLM-Omni – Part 1 , and vLLM-Omni – Part 2 . In this post, you first use the AWS DLC for llama.cpp to serve OpenAI-compatible chat completions through SageMaker AI InvokeEndpoint . You then extend the same runtime with SemIf , formerly OpenJev, which returns probabilities over declared options without decoding an answer token. The accompanying code sample includes CPU and GPU deployment options. Solution overview Figure 1 shows two deployment paths from the same packaged model. The text-generation path uses the AWS DLC for llama.cpp and returns generated responses through the OpenAI-compatible API. The structured-decision path builds a child image that replaces the default server with a SemIf adapter and returns probabilities for declared options. Figure 1: Run the AWS DLC for llama.cpp or the SemIf child image on CPU or GPU, depending on the API and latency requirements of your workload For the standard path, package the model as model.tar.gz in Amazon Simple Storage Service (Amazon S3) and reference the published DLC image directly. The container finds the first .gguf file under /opt/ml/model , maps POST /invocations to llama-server at /v1/chat/completions , and passes SM_LLAMA_CPP_* environment variables to llama.cpp arguments. The llama-server HTTP API doesn’t expose the final-position logits and state operations that SemIf uses. The structured-decision path therefore uses AWS CodeBuild to build a child image and store it in Amazon Elastic Container Registry (Amazon ECR) . The child image keeps the DLC’s compiled CPU or CUDA libraries while replacing its entry point with a SageMaker-compatible adapter. Understand the input and output schemas The AWS DLC for llama.cpp accepts the OpenAI Chat Completions schema. Send a messages array with role and content , plus generation controls such as max_tokens and stream : { "messages": [ { "role": "user", "content": "A customer cannot receive a password reset email. Reply with the best support queue." } ], "max_tokens": 64, "stream": false } The response uses the OpenAI-compatible choices array. Read generated text from choices[0].message.content . SemIf uses a constrained decision schema instead of chat messages. The request contains a caller-defined ID, unstructured state, a question, and an ordered list of allowed options: { "decision": { "id": "support-route", "state": "The customer cannot receive a password reset email.", "question": "Which queue should handle this request?", "options": [ {"id": "account_access", "description": "Account access support."}, {"id": "billing", "description": "Billing support."} ] } } The response returns option_ids , probabilities , and raw option_logits in the same order, along with prompt, timing, and pinned model metadata. With stable option IDs, your application can apply thresholds, fallbacks, and policy checks without parsing generated prose. SemIf belongs to a broader class of decision models that score application-defined outcomes from unstructured state. Other implementations include Jev , TypeSafe’s decision model and /v1/systemone API; Kev , a model server designed for TypeSafe API compatibility; and Strands Decider 2B , a purpose-built model for agent workflows with calibrated outputs. Choose the right decision workflows Use this pattern when your application defines the allowed choices at request time and needs a model to interpret unstructured context. Examples include routing a case to a queue, extracting one value from a known set, assigning an incident priority, checking a policy outcome, or selecting an agent action. Keep authorization and deterministic business rules outside the model, and validate model probabilities before treating them as operational confidence. Use the standard SageMaker AI text-generation path when you need OpenAI-compatible chat completions through InvokeEndpoint . The SageMaker AI image maps POST /invocations to the llama-server /v1/chat/completions endpoint. Build a child image only when your application needs behavior that this interface doesn’t provide, such as SemIf’s direct logits and state access. Removing answer-token generation doesn’t remove model inference. The model still processes the complete state, question, and option descriptions before SemIf reads the final logits. Keep prompts focused, and evaluate the selected model with labeled decisions from your application before production use. Tune for your latency and quality targets The following benchmark applies to the SemIf extension, not the text-generation endpoint. Model size, quantization, prompt length, runtime, and CPU thread count affect each decision. To quantify these trade-offs, we sent the same 231 public JevBench tasks serially to one CPU endpoint and one GPU endpoint. Configuration AWS DLC for llama.cpp Valid responses Accuracy Client p50 Client p95 Qwen3.5-4B Q4_K_M , ml.c6i.8xlarge , 32 threads llama-cpp:server-sagemaker-cpu-v1 231/231 179/231 (77.5%) 1.715 seconds 19.291 seconds Qwen3.5-4B Q4_K_M , ml.g6e.xlarge , full GPU offload llama-cpp:server-sagemaker-cuda-v1 231/231 180/231 (77.9%) 0.352 seconds 0.599 seconds Compared with the CPU configuration, the GPU endpoint was 4.9 times faster at p50 and 32.2 times faster at p95. The sample uses the SageMaker AI instance pools feature to list compatible GPU instance types in priority order, so SageMaker can select an alternative when the first preference isn’t available. The pool ordered g6e , g6 , g5 , then g4dn . SageMaker selected ml.g6e.xlarge , and startup logs confirmed that llama.cpp offloaded all 34 model layers to an NVIDIA L40S. The CPU and GPU images used different llama.cpp builds and produced different predictions on six hard tasks. Treat the one-answer accuracy difference as runtime variation, not evidence that GPU offload improves model quality. Smaller models reduced median latency, but the 2B and 0.8B candidates achieved 66.7 percent and 58.4 percent accuracy. The sample therefore keeps the 4B Q4_K_M model as its quality-first default. These measurements include network latency and aren’t service guarantees. Benchmark the complete model and infrastructure configuration against your own decisions. Use CPU for background, offline, or low-throughput workloads that aren’t latency-sensitive and can tolerate seconds of response time. Use GPU for interactive decisions, long prompts, or workloads with tight tail-latency targets. Keep state and option descriptions concise, group decisions that share the same state into one request, and scale endpoint instances for concurrency. Shared-state scoring reuses the common prompt prefix, while the adapter returns retryable HTTP 503 responses instead of allowing an unbounded queue around its stateful llama.cpp context. Prerequisites You need: An AWS account and an S3 bucket in your deployment AWS Region. Python 3.10 or later, Git, and the AWS Command Line Interface (AWS CLI). A purpose-specific CodeBuild service role for the optional SemIf extension. An AWS Identity and Access Management (IAM) role that SageMaker AI can assume. Permissions for SageMaker AI, Amazon ECR, Amazon S3, CodeBuild, IAM pass-role actions, and Amazon CloudWatch Logs . Quota for a compatible x86 CPU or NVIDIA GPU SageMaker real-time endpoint. The text-generation example uses the CPU DLC and a fixed CPU instance. The SemIf extension keeps CPU as the default for workloads that aren’t latency-sensitive and offers an optional CUDA image with ordered instance pools. Review SageMaker AI pricing before creating an endpoint. Deploy a decision model