AI21 releases Teel to run Python agent calls in parallel without an async rewrite

AI21 Labs has released Teel, an open-source Python runtime that lets independent calls run together even when the code describes them one after another. Rather than requiring a rewrite with async instructions, Teel reads ahead and starts eligible work early. The catch: developers must identify functions that are safe to execute before they are needed.

The target is time spent waiting, not time spent calculating. AI21 says profiling its agents revealed that much of their elapsed runtime went to waiting on model calls, retrievals, tools and sandbox commands.

Reading ahead without taking over

Developers mark expensive, parallelizable functions with @cacheable and orchestration functions with @jit, then activate the runtime inside a Teel context. Outside that context, decorated functions run as ordinary Python.

A limited “shadow interpreter” examines upcoming bytecode—the instructions Python executes—using a copy of the program’s execution state. If a marked call’s arguments are known, Teel sends it to a background thread. Two searches can therefore start together, while a summary that needs both results waits for them.

The shadow interpreter does not produce the final answer. Python still executes the function normally, finding some results already cached or underway.

Faster runs in AI21’s tests

AI21 reports an average runtime reduction of 83% when applying Teel externally to Princeton’s Tree of Thoughts code, without editing the repository. The test used GPT-4 on Game of 24 samples, where the program generates candidates, evaluates them and explores further choices.

A second AI21 experiment replaced model-written asynchronous JavaScript in a fork of LangChain’s Deep Agents with synchronous Python under Teel. Across ten prompts, Teel averaged 58 seconds, against 71 seconds for JavaScript and 150 seconds for sequential Python. That comparison covers distributions of generated programs, not identical implementations: the model can write different code each time.

The cost of starting the wrong branch

Teel can also start work on alternative branches before a decision resolves, including calls the program ultimately never needs. AI21 says a wrong guess can waste time and model tokens without changing the answer—but only if marked functions obey its safety contract. Sending an email or charging a card is unsafe: an unused branch could still trigger the action.

The @cacheable contract also excludes database writes, clock reads and hidden state. Arguments must be JSON-serializable because Teel uses their serialized form to identify reusable results. Developers must enforce these rules themselves.

There is a narrow installation requirement, too: Teel currently needs Python 3.12.13 exactly. Its C extension reads execution-stack details through internal CPython headers, tying the implementation to that interpreter version.