IBM open sources vLLM plugin for runtime model controls

IBM is open sourcing vLLM-Hook MIA, a plugin that lets developers inspect and change model states during inference, the company detailed in a blog post. The software can support tools that reject malicious prompts, flag unsupported claims, or adjust responses for tone and accuracy.

The plugin works without disabling the graph capture and replay system that vLLM uses to reduce CPU bottlenecks. Developers select the model states they want to inspect or change, while a scheduler creates routing entries in fixed GPU memory locations and a courier transfers captured data to CPU memory.

IBM measured steering overhead of 2% in some tests with an 8B Llama model and a Qwen 1.5B model. Overhead reached 6% with Llama 70B. The researchers also reported that vLLM-Hook MIA was up to 20 times faster than the original vLLM-Hook and up to 14 times faster than a method that inserts code into the model.