AWS finds full fine-tuning improves coding results, but longer training erodes gains
The SageMaker experiments reached 90.0% correctness under an AI judge. AWS also found that training settings could undermine gains and that both customization methods lost ground on a math test.
In a Nov. 18 report, AWS found that full fine-tuning Qwen3-4B on SageMaker could outperform LoRA on an AI-judged coding test, but the result was conditional rather than a general case against LoRA. Performance peaked after three epochs; more training weakened every tested variant, and both fine-tuning methods scored below the base model on MATH. Managed compute does not remove the need to choose training settings carefully or check whether broader capabilities survive.
01
AWS trained on 99,000 examples from Magicoder-Evol-Instruct-110K and reserved 5,500 each for validation and final testing.
02
Amazon Nova Pro judged code for logical correctness; AWS also used PRIMECODE similarity metrics and MMLU to check broader knowledge.
03
A learning rate of 2e-5, batch size of eight and 5% warmup best preserved general knowledge in these experiments.
AWS’s latest code-generation experiments put full fine-tuning ahead of the lighter-weight LoRA approach, reaching 90.0% correctness as assessed by an AI judge. But the results, published in a November 18 post, also expose a limit: training beyond three passes through the data degraded every tested variant, and the method’s success depended heavily on training settings.
Fine-tuning continues a pretrained model’s training on task-specific data. Full fine-tuning updates all its learned numerical parameters, or weights. LoRA instead freezes the original weights and trains small added matrices. AWS argues that this restricted approach can limit adaptation to complex tasks.
One coding dataset, three evaluation lenses
For the detailed comparison, AWS customized Qwen3-4B using Magicoder-Evol-Instruct-110K, a dataset of 110,000 coding instruction-and-response pairs. It assigned 99,000 examples to training, 5,500 to validation during training and 5,500 to a held-out final test. Both methods received the same prompts, target responses and evaluation code.
LLM-as-Judge used Amazon Nova Pro to read generated code alongside its prompt and assess whether the solution was logically correct.
PRIMECODE measured text similarity to reference solutions using BLEU, ROUGE and F1 scores, assessing coding patterns, formatting and implementation style.
MMLU tested knowledge across 57 academic subjects to detect whether coding specialization came at the expense of broader abilities.
AWS’s reported AI-judged correctness
90.0%Full fine-tuning
The score peaked at three epochs—three passes through the training data. Longer training degraded every variant AWS tested.
A falling training loss can hide forgotten skills
AWS warns that training loss can keep falling and validation results can look healthy while abilities outside the training domain deteriorate. Validation examples resemble the training data, so those signals alone may not expose the damage. Updating every parameter also changes weights that encode knowledge learned before customization.
This is catastrophic forgetting: narrower training overwrites broader capabilities. A learning rate of 2e-5, batch size of eight and 5% warmup best preserved general knowledge in AWS’s tests. The company presents its settings as starting points, not a universal recipe.
Both full fine-tuning and LoRA underperformed the original model on MATH exact-match scoring in AWS’s experiments. AWS explicitly cautions that full fine-tuning is not universally superior to LoRA.
SageMaker handles compute, not the training recipe
SageMaker’s serverless service allocates compute and manages saved training checkpoints automatically. Customers specify the model, data and settings without provisioning GPUs or configuring clusters. Infrastructure management is automated; choosing how aggressively to retrain—and checking which skills survive—remains part of the customization work.
Sources
aws.amazon.comAmazon SageMaker AI serverless model customization now supports full fine-tuning - AWS
aws.amazon.comFull Fine-Tuning (FFT) on SageMaker AI Serverless Model Customization | Amazon Web Services
Reader comments
Newest comments first. Replies stay oldest first.