Microsoft says Foundry Local outpaced llama.cpp in local coding tests

Microsoft says developers can generate coding-model output faster with Foundry Local than with llama.cpp, based on its tests on two NVIDIA systems. Its November 5, 2026 comparison used Qwen3.8-27B and accompanied a guide to connecting the model to GitHub Copilot. That local setup requires at least 24GB of graphics memory.

The advantage grew with longer prompts

The evaluation used a dataset of real GitHub issues and a 4-bit version of Qwen3.8-27B. Microsoft compared the two software engines on a GeForce RTX 5090 with 32GB of graphics memory and a DGX Spark with 128GB of unified memory.

Foundry Local was faster at every tested prompt length, Microsoft says, with its generation-speed advantage widening beyond 131K tokens of input. The company connects that long-input result to coding workflows involving whole repositories.

Foundry Local uses ONNX Runtime to optimize model execution. One technique, DFlash2 speculative decoding, uses a lightweight drafting model to predict multiple tokens at once. Prefix caching reuses previously computed context, while continuous batching processes concurrent requests.

Memory savings do not erase the reasoning tradeoff

Compression is only part of fitting the workload onto a GPU. Microsoft also compresses the cache that stores working context, shares embedding memory and offloads work to the CPU. Configurable, memory-aware inference lets users balance memory consumption against performance.

Microsoft cautions that smaller local models may have less reasoning ability than advanced cloud models. It positions local execution for everyday coding and targeted subtasks, while recommending cloud models as a strong option for complex reasoning.

Connecting the local model to Copilot

Microsoft’s setup guide uses GitHub Copilot’s Bring Your Own Key, or BYOK, connection. It requires Foundry Local CLI 0.13.0 or later, plus separately installed CUDA and compatible cuDNN runtimes. Foundry Local does not download those dependencies.

After loading Qwen3.8-27B, users retrieve their local API endpoint with foundry server status. In Copilot’s Model providers settings, they add Foundry Local, enter that endpoint as the Base URL and add Qwen3.8-27B from Browse Catalog before selecting it in the model picker.

Microsoft plans to extend Qwen3.8-27B support to Macs and AMD GPUs, add more coding-optimized local models and deepen its Copilot integration.