Together AI puts agents to work tuning model-serving code, opens limited beta

Together AI is pitching AI agents that improve their own inference—the computing used to run models. Its November 4 Together Compiler announcement has a concrete starting point: the company says agents are already optimizing some production endpoints. But customer access is limited to a beta, with self-serve optimization still ahead.

Together calls this “recursive self-improvement,” describing a loop in which AI agents optimize the software used to serve models. The announced work concerns inference performance, not a newly trained model or a demonstrated increase in model intelligence.

An optimization search, not a fixed recipe

Together Compiler treats inference optimization as a search problem for agents, rather than only a traditional compiler’s task of translating code. The system identifies recurring optimization patterns and pairs multiple agents with what Together describes as a high-fidelity, persistent engine for rapid iteration. Its Frontier Performance team developed the system over several months.

The company calls the resulting process Continuous Compilation. Instead of requiring fresh hand-written kernels—small programs for particular computations—and other optimizations for each new workload, Together says the system can automatically adapt to changing traffic patterns. The optimization loop becomes part of operating the service, not just preparation before deployment.

Together says the compiler has found additional gains beyond already highly optimized code on some of its most popular serverless endpoints. The announcement provides no numerical speedup or cost reduction for those deployments.

Why the workload changes the answer

Together’s argument is that one carefully tuned configuration does not stay optimal across different models, traffic patterns and hardware. It calls the alternative Polymorphic Inference: model serving that adapts to those changing conditions. As one indication of model turnover, the company says 80% of OpenRouter traffic goes to models released in the previous three months.

The company sets out three goals for an adaptive inference system. These are research ambitions, not measured outcomes demonstrated for every Together Compiler deployment:

• Adapt quickly when product changes or user behavior alter traffic.
• Approach ideal hand-tuned performance as soon as possible when a new model arrives.
• Approach hardware performance limits quickly when a stable workload moves to a new platform.