Deploy and evaluate models with Amazon SageMaker JumpStart

Choosing the right model for a specific application requires a practical way to test it with representative inputs. Amazon SageMaker JumpStart provides access to a broad catalog of already pretrained models. In this post, we show how to deploy an available model from SageMaker JumpStart, run a lightweight and repeatable performance test using the SageMaker Python SDK, and delete the endpoint when testing is complete. For supported models and available capacity, this workflow provides a repeatable evaluation path, although deployment time varies by model, instance type, quota, and AWS Region. In 2026, SageMaker JumpStart introduced two improvements that simplify model discovery and deployment. In the Model Hub section of the Amazon SageMaker AI console, you can browse the SageMaker JumpStart catalog, view sample notebooks, and open them directly in SageMaker Studio. Pre-configured endpoint deployment options provide named configurations optimized for priorities such as low cost, low latency, or general-purpose workloads across a range of generative AI models, including Neuron-optimized configurations. Additionally, you can now go from Hugging Face model discovery to SageMaker AI deployment in a single click using the new deep link integration. Solution overview Our approach uses a reusable Python script that performs three steps: Discover and deploy a model from SageMaker JumpStart. Run model-specific sample prompts and measure latency, approximate output length, and single-request generation rate. Clean up the endpoint. This pattern works for many compatible SageMaker JumpStart models, although payload formats, required instance types, end-user license agreement (EULA) requirements, and response schemas may require model-specific adjustments. Prerequisites An AWS account with SageMaker AI permissions. Python 3+ with SageMaker Python SDK v3 installed ( pip install "sagemaker>=3,<4" ). Sufficient GPU instance quota for the model you want to test or request a quota increase . Step 1.a: Discover available models UI path to find models in SageMaker JumpStart: Figure 1: Browsing the model catalog on the SageMaker JumpStart Models page in SageMaker Studio Code path using SageMaker Python SDK: from sagemaker.core.jumpstart.notebook_utils import ( list_jumpstart_models, ) REGION = "us-east-1" models = list_jumpstart_models( filter="task == textgeneration", region=REGION, list_versions=True, ) print(f"Found {len(models)} text generation models") for model_id, model_version in models[:10]: print( f" - {model_id} " f"(version {model_version})" ) Or search by model name: from sagemaker.core.jumpstart.notebook_utils import list_jumpstart_models ids = list_jumpstart_models( filter="model_id contains qwen", region=REGION, list_versions=True, ) print(f"{len(ids)} models matched in {REGION}\n") for model_id, version in ids: print(f"{model_id:<55} v{version}") Step 1.b: Deploy the model UI path to deploy models from SageMaker JumpStart: Figure 2: Deploying a model from its SageMaker JumpStart detail page in SageMaker Studio Code path using SageMaker Python SDK: import time from sagemaker.core.jumpstart.accessors import JumpStartModelsAccessor from sagemaker.core.jumpstart.configs import JumpStartConfig from sagemaker.core.training.configs import Compute from sagemaker.serve import ModelBuilder MODEL_ID = "huggingface-vlm-qwen3-5-27b" MODEL_VERSION = "1.13.0" REGION = "us-east-1" ACCEPT_EULA = False # Set True only after reviewing the model terms def get_field(value, name, default=None): if isinstance(value, dict): return value.get(name, default) return getattr(value, name, default) # Resolve the JumpStart-recommended deployment configuration. specs = JumpStartModelsAccessor().get_model_specs( region=REGION, model_id=MODEL_ID, version=MODEL_VERSION, ) if isinstance(specs, list): if not specs: raise ValueError( f"No JumpStart specification found for {MODEL_ID}" ) specs = specs[0] recommended_instance_type = get_field( specs, "default_inference_instance_type", ) if not recommended_instance_type: raise ValueError( f"No recommended instance type found for {MODEL_ID}" ) print(f"Model: {MODEL_ID}") print(f"Region: {REGION}") print(f"Recommended instance: {recommended_instance_type}") start = time.time() jumpstart_config = JumpStartConfig( model_id=MODEL_ID, model_version=MODEL_VERSION, accept_eula=ACCEPT_EULA, ) model_builder = ModelBuilder.from_jumpstart_config( jumpstart_config=jumpstart_config, compute=Compute( instance_type=recommended_instance_type, instance_count=1, ), ) model = model_builder.build() endpoint = model_builder.deploy() deploy_time = time.time() - start print(f"Deployed in {deploy_time:.1f} seconds") print(f"Endpoint: {endpoint.endpoint_name}") Step 2: Run sample prompts and measure performance from sagemaker.core.payloads import retrieve_all_examples import json import time MODEL_ID = "huggingface-vlm-qwen3-5-27b" MODEL_VERSION = "1.13.0" REGION = "us-east-1" example_payloads = retrieve_all_examples( region=REGION, model_id=MODEL_ID, model_version=MODEL_VERSION, serialize=True, ) #you can re-create the endpoint pointer using the endpoint name: #from sagemaker.core.resources import Endpoint #endpoint = Endpoint.get(endpoint_name="",region=REGION) def read_response_body(result): body = result.body if hasattr(body, "read"): body = body.read() if isinstance(body, bytes): body = body.decode("utf-8") if isinstance(body, str): try: return json.loads(body) except json.JSONDecodeError: return body return body def extract_output(response): """Handle common JumpStart text-generation response formats.""" if isinstance(response, list): response = response[0] if response else "" if isinstance(response, dict): choices = response.get("choices") if choices: choice = choices[0] message = choice.get("message") if isinstance(message, dict): return message.get("content", "") return choice.get("text", "") for field in ( "generated_text", "output_text", "text", ): if field in response: return response[field] return response results = [] for payload in example_payloads or []: start = time.perf_counter() result = endpoint.invoke( body=payload.body, content_type=payload.content_type, accept=payload.accept, ) latency = time.perf_counter() - start response = read_response_body(result) output = extract_output(response) if not isinstance(output, str): output = json.dumps(output, indent=2) # This is an estimate based on whitespace-separated words. output_words = len(output.split()) results.append( { "request_preview": payload.body[:50], "latency_s": round(latency, 2), "estimated_output_words": output_words, "words_per_second": round( output_words / latency, 1, ) if latency > 0 else 0, } ) print("Input:\n", payload.body, end="\n\n") print("Output:\n", output, end="\n\n") print( f"Latency: {latency:.2f} seconds\n" f"Estimated output words: {output_words}\n" f"Estimated single-request output generation rate: " f"{output_words / latency:.1f} words/second\n" ) print("-" * 80) print("\nSummary:") for result in results: print(result) Result: Input: {"max_tokens": 256, "messages": [{"content": "What is deep learning?", "role": "user"}], "model": "/opt/ml/model"} Output: Here's a thinking process that leads to the explanation of deep learning: 1. **Deconstruct the Request:** * **Question:** "What is deep learning?" * **Intent:** The user wants a clear, comprehensive, yet accessible definition and overview of deep learning. * **Target Audience:** Likely someone looking for a general understanding, possibly a student, tech enthusiast, or professional exploring the field. The tone should be informative, structured, and not overly jargon-heavy (though key terms need explanation). 2. **Initial Brainstorming & Key Concepts:** * What is it? A subset of machine learning, which is a subset of AI. * Core mechanism: Artificial Neural Networks (ANNs). * "Deep" means many layers (hidden layers). * How does it work? Learning representations from data (feature learning). * Key applicatio