LangChain adds ScaleDown to handle AI agents’ growing conversation history

An AI agent can now hand off the work of trimming its conversation history without changing the model doing its main job. On October 13, LangChain introduced langchain-scaledown for LangChain and Deep Agents. It supplies specialized models for summarization, tool-result compression and data extraction, alongside access through the LangSmith LLM Gateway.

The problem builds as an agent works. File reads, fetched pages and other tool outputs accumulate in its working context—the information available for its next decision. Old messages eventually need summarizing, while large tool results can keep adding input costs at later steps. Using another general-purpose model for that housekeeping adds cost and delay, LangChain says.

ScaleDown’s response is a set of constrained-decoding models: each is trained for a narrow task and limits its answers to that task’s allowed output. That can mean selecting a label or returning passages from the input, rather than generating an unrestricted answer.

Three handoffs inside the agent loop

The integration packages those handoffs as middleware, software that runs around the agent’s model and tool calls. Its summarization component replaces Deep Agents’ built-in summarizer while preserving controls for when to summarize and which recent messages to keep. Older messages become one summary. If the ScaleDown call fails, history stays unchanged and the agent continues running.

Tool compression starts with the main model describing what it needs through an optional compression_query parameter. The middleware removes that parameter before running the tool, then asks ScaleDown to trim the result to relevant material. Existing tools need no changes. A note attached to the shortened result tells the model it can request the full output by calling again without compression.

Extraction takes a different route. Developers describe fields such as an order number or exact error message, and ScaleDown copies values from user messages into the agent’s stored state before the first model call. It does not read the agent’s replies. Fields can also use predefined labels for classification. With saved conversation state, extraction runs again as the user adds details.

Large inputs, company-measured savings

ScaleDown says all four models—Summarize, Compress, Extract and Classify—accept up to one million input tokens, the units models use to process text. Extract, Summarize and Classify also accept images and scanned PDFs directly, extending the same task-specific approach beyond plain text.

The performance case comes from ScaleDown’s own benchmarks on public datasets. Against the average of GPT-5.5, GPT-5.4 Mini and GPT-5.4 Nano, it reports Summarize was 41 times cheaper and 2.7 times faster. Compress was 138 times cheaper and 1.8 times faster. The company also says both scored higher on accuracy. These are task-level comparisons, not measured savings for an entire agent run.

Shared controls, custom stop rules

ScaleDown is now a direct provider in the LangSmith LLM Gateway at the /scaledown path. An administrator adds the ScaleDown key to workspace secrets, then developers authenticate gateway requests with their LangSmith key. Each call appears in a trace—a record of its execution—and counts toward the workspace’s existing spend and rate limits.

uv pip install -U langchain-scaledown
export SCALEDOWN_API_KEY="..."

LangChain says new ScaleDown accounts receive 50 million free tokens. Beyond the packaged middleware, developers can use Classify to detect repeated failures or off-task activity. There is no prebuilt drift-detection middleware: LangChain supplies a custom circuit-breaker example. Classify returns label probabilities and a short reason, leaving developers to choose the confidence threshold and whether to stop the run or first nudge it back on course.