Fine-tuning GPT-OSS-20B for multilingual reasoning on Amazon SageMaker HyperPod

GPT-OSS-20B is an open-weight reasoning model that thinks in English. Ask it to reason in French, German, Spanish, or Italian and it acknowledges the instruction, then produces its chain-of-thought in English anyway. On a 50-example multilingual reasoning test, the base model reasoned in the correct target language only 22 percent of the time. That 22 percent was almost entirely the English examples (English 100 percent, French 0 percent, German 10 percent, Spanish 0 percent, Italian 0 percent). For teams deploying reasoning models for non-English users, that gap matters. A user who asks a question in French expects the model to reason and answer in French, not to switch to English mid-thought. Closing that gap is a fine-tuning problem. It raises a question every team faces once they start customizing open-weight models: supervised fine-tuning (SFT), reinforcement learning with Group Relative Policy Optimization (GRPO), or both? In this post, we walk through how we fine-tuned GPT-OSS-20B for multilingual reasoning on Amazon SageMaker HyperPod . We show when SFT alone was sufficient and when GRPO added measurable value on top. To understand more broadly when GRPO helps, we also ran a controlled Qwen2.5-7B experiment across two tasks with different capability gaps. The short answer: SFT does most of the work, GRPO refines when there is headroom to refine, and matching the technique to the capability gap is what decides the outcome. Solution overview This section covers the multilingual reasoning task and dataset, how SFT and GRPO complement each other, and the training stack we ran on Amazon SageMaker HyperPod. The task and dataset The dataset is HuggingFaceH4/Multilingual-Thinking — a chain-of-thought reasoning dataset translated into French, Spanish, German, and Italian alongside English, roughly 949 training and 50 validation examples. The model receives a reasoning language: system-prompt instruction and must both reason and answer in that target language, overriding its strong English prior. The evaluation is 50 test cases: 10 math/logic problems across 5 languages (EN/FR/DE/ES/IT). Two metrics matter: Reasoning-language correct – is the chain-of-thought in the requested language? Answer-language correct – is the final answer in the requested language? How SFT and GRPO fit together The clearest way to think about the two techniques: SFT teaches a behavior the model doesn’t have. GRPO optimizes a behavior the model can already sometimes produce. GRPO works by sampling a group of responses per prompt and pushing the policy toward the above-average ones. If the desired behavior has zero probability in the base distribution, there is very little for GRPO to reinforce. That single idea explains almost every result in this post. The training stack We ran on Amazon SageMaker HyperPod with Amazon Elastic Kubernetes Service (Amazon EKS) orchestration. For the initial SFT and GRPO runs on GPT-OSS-20B, we used the PyTorch TRL library. For the Qwen control experiment, we switched to veRL as the trainer, with Ray and KubeRay for distributed orchestration and vLLM for fast rollout generation. Checkpoints and datasets lived on Amazon FSx for Lustre, and the inter-node fabric used Elastic Fabric Adapter (EFA). All fine-tuning used Low-Rank Adaptation (LoRA), which is parameter-efficient and well-suited to both SFT and GRPO. The following diagram shows the training data path end-to-end. A submission from the user lands on the Amazon EKS control plane of the Amazon SageMaker HyperPod cluster. Amazon EKS schedules a Ray head pod plus worker pods that run the veRL trainer and vLLM rollout workers. Both sets of pods mount Amazon FSx for Lustre for the base model, training data, and checkpoints. Inter-node collectives use EFA. Rollouts stream from vLLM back to the veRL trainer, and the trainer writes checkpoints back to Amazon FSx for Lustre. Figure 1: End-to-end training data path for SFT and GRPO fine-tuning on Amazon SageMaker HyperPod with veRL, Ray, KubeRay, vLLM, and Amazon FSx for Lustre for Qwen control experiments The following describes each hop in more detail. The submitter starts the run from Amazon SageMaker Studio or the AWS Command Line Interface (AWS CLI). The Amazon EKS control plane on Amazon SageMaker HyperPod places pods on GPU nodes and starts a Ray head with worker pods for the veRL trainer and the vLLM rollout workers. Amazon FSx for Lustre provides shared, high-throughput storage for the base model, the training data, and the checkpoints. When training completes, the trainer writes the LoRA adapter back to Amazon FSx for Lustre for downstream evaluation or serving. Prerequisites To follow along, you need: An Amazon SageMaker HyperPod cluster with Amazon EKS orchestration. GPT-OSS SFT used 4× ml.g6e.12xlarge (16× L40S 48GB). The Qwen control used 2× g6e.48xlarge (16× L40S). An Amazon FSx for Lustre file system mounted on the cluster with a PersistentVolumeClaim named fsx-claim at /fsx . EFA-enabled instances and a working NVIDIA Collective Communications Library (NCCL) and aws-ofi-nccl stack for multi-node SFT. A container registry such as Amazon Elastic Container Registry (Amazon ECR) to hold the training image built from the reference repository. A HuggingFace access token for openai/gpt-oss-20b and the HuggingFaceH4/Multilingual-Thinking dataset. The AWS Command Line Interface (AWS CLI), kubectl , docker , and envsubst available on your workstation. Set up the training environment All of the commands, manifests, and scripts referenced below live in the awsome-distributed-ai GitHub repository under 3.test_cases/pytorch/trl/gpt-oss-lora-grpo/ . Clone the repository and change into the test case directory: git clone https://github.com/awslabs/awsome-distributed-ai cd awsome-distributed-ai/3.test_cases/pytorch/trl/gpt-oss-lora-grpo Copy the environment template, then edit env_vars to set your ECR registry, image name, tag, and HuggingFace token: cp env_vars.example env_vars source env_vars Build the training image and push it to your registry. The image bundles PyTorch, TRL, HuggingFace Transformers, PEFT, and the training and inference scripts: cd artifacts && ./build_push.sh && cd .. Point kubectl at your cluster and apply the FSx storage manager pod. The storage manager gives you a shell into the shared file system for checkpoint conversion and eval-log inspection: aws eks update-kubeconfig --name --region us-east-2 envsubst '$REGISTRY $IMAGE $TAG $HF_TOKEN' \ < kubernetes/fsx-storage-manager.yaml | kubectl apply -f - Fine-tune GPT-OSS-20B with SFT and LoRA We begin by measuring the base model, then configure the SFT run and examine the results. The base-model gap Base GPT-OSS-20B on the 50-case eval scored 22 percent reasoning-language correct (11/50). Per-language: English 100 percent, French 0 percent, German 10 percent, Spanish 0 percent, Italian 0 percent. The model defaults to English reasoning regardless of the instruction. SFT configuration The SFT setup used LoRA r=8, α=16, trained for 1,000 steps on 4× ml.g6e.12xlarge (16× L40S 48GB) with Fully Sharded Data Parallel (FSDP) full-shard, bf16 precision, and gradient checkpointing. The job runs as an elastic HyperPodPyTorchJob managed by Kueue, launched with hyperpodrun . Apply the SFT manifest: envsubst '$REGISTRY $IMAGE $TAG $HF_TOKEN' \ < hyperpod-eks/train-lora-hyperpod-elastic-g6e.yaml | kubectl apply -f - Watch training progress: kubectl get pods -l app=gpt-oss-lora-elastic kubectl logs -f -l app=gpt-oss-lora-elastic Loss and gradient norm print each step. A healthy run looks like this: {'loss': 1.035, 'grad_norm': 0.163, 'learning_rate': 0.0002, 'mean_token_accuracy': 0.716, 'epoch': 3.25} {'loss': 1.003, 'grad_norm': 0.171, 'lr': 0.0002, 'acc': 0.722, 'epoch': 3.32} {'loss': 0.976, 'grad_norm': 0.159, 'lr': 0.0002, 'acc': 0.726, 'epoch': 3.57} When training completes, convert the FSDP checkpoint to the Parameter-Efficient Fine-Tuning (PEFT) format that TRL’s