IBM open-sources a plugin to inspect and steer AI models while keeping vLLM’s fast mode

IBM Research announced on October 21 that it is open-sourcing vLLM-Hook MIA, a plugin for inspecting and steering AI models while they generate responses. Its key change is preserving vLLM’s fast graph-replay mode, which many monitoring tools must disable. IBM says that makes interventions possible with much less delay in its tests.

The plugin works inside vLLM, an open-source engine for running language models, rather than changing their trained weights. Developers can select internal states to observe or alter during inference—the process of running a model. IBM describes potential uses including refusing malicious prompts, flagging fabricated facts, and adjusting a response’s tone or accuracy. These are intended applications, not a claim that the plugin automatically performs those checks.

The obstacle is a serving optimization called graph capture and replay. vLLM records repeated computation steps, such as those used to generate successive pieces of a response, into a fixed execution graph. The runtime can then issue one replay command instead of launching GPU operations individually. That cuts the CPU work needed to keep the GPU busy.

Many interpretability tools instead need Python code to run inside individual model layers. Those tools inspect or modify activations—the intermediate values a model produces as it works. In vLLM, that approach requires switching off graph replay and using its slower, eager execution mode. MIA moves the capture and steering operations onto the GPU before the graph is recorded, keeping them within the optimized execution path.

1. A developer chooses which internal model states to capture or steer. The scheduler turns those selections into routing entries held at fixed GPU memory addresses.
2. During graph replay, GPU operations use those routing entries to carry out the developer’s request without interrupting the replay mechanism.
3. The courier transfers captured data into CPU memory so it can be delivered alongside the model’s output.

2%Llama 8B and Qwen 1.5B

IBM reports that steering these smaller models added 2% inference delay in some cases.

6%Llama 70B

IBM reports 6% steering overhead for the larger model.

IBM’s performance results put MIA at up to 20 times faster than the earlier vLLM-Hook and up to 14 times faster than a method that inserts code into the model. Those comparisons concern ways of accessing and steering model internals. They should not be read as a 20-fold improvement over ordinary model serving without monitoring. The low-overhead figures above describe selected experiments, not a universal cost for every intervention.

The researchers also report outperforming vLLM-Lens, a UK government plugin that uses Python hooks, and EasySteer. EasySteer retains graph replay but modifies a separate copy, or fork, of vLLM. That creates a maintenance obligation: developers must keep their copy synchronized with changes to the main project. MIA’s approach targets both the execution penalty of hooks and the upkeep associated with modifying the serving engine.

The project began with an attention-tracking tool developed by IBM researchers, including Irene Ko and Pin-Yu Chen. It looks for patterns suggesting a prompt may cause a model to disregard its guardrails. The tracker worked in research settings, but moving it into vLLM proved difficult. The team’s first vLLM-Hook release earlier this year ran that tracker 1.7 times faster than the Hugging Face transformers library.

MIA followed the team’s effort to recover graph replay’s performance benefits. IBM says vLLM-Hook is already deployed in its Steerability toolkit, which helps developers design and evaluate steering methods. The distinction is important: the plugin supplies access to a model’s internal activity; developers still select the monitoring or intervention method that uses that access.