Earlier this year, leaders at Amazon Web Services delivered a new mandate to their engineers: They need to conserve CPU cycles at all costs. AWS has reportedly experienced an explosion in wait times for CPU server capacity as AI workloads strain the company’s cloud infrastructure.
The issue seemingly took AWS off guard, and for good reason. The AI boom led to a surge in demand for GPUs and, later, memory. CPUs were mostly left out of the story, as their relative lack of parallelization made them a poor fit for AI model inference, the process of running and serving large language models (LLM) to users.
But the rise of agentic AI systems, which allow AI models to operate autonomously and call on sub-agents, is changing the narrative.
Matt Kimball, vice president and principal data center analyst at Moor Insights & Strategy in Austin, Texas, says 2026 has brought a spike in CPU demand, much of it due to agentic AI. “It’s one thing to have this agentic workload, and let’s say it spawns 100 agents. If I’m going to roll this out across my enterprise, those 100 become tens of thousands, hundreds of thousands, or millions of agents,” Kimball says. “You have agents spawning sub-agents, making API [application programming interface] calls and talking to more agents through [Anthropic’s] model context protocol.”
AI agents need to use computers, and computers need CPUs
Kimball’s comments refer in part to “tool use,” which is shorthand for an LLM’s ability to access the internet, open files on a desktop, and generally use a variety of software to accomplish its task.
LLMs trained for tool use learn how to call on other software. While the LLM’s inference is still primarily executed on a GPU or similar AI accelerator, the tool calls that the LLM makes are typically pushed to the CPU.
“Many components of an agentic AI task are inherently CPU based jobs,” explains Souvik Kundu, senior staff research scientist at Intel. “The CPU does the job of parsing output, figuring out which tool to invoke, making the API call or running the code, collecting the result, and feeding it back.” Madhu Rangarajan, vice president of compute and enterprise AI products at AMD, makes a similar claim, saying, “In our testing, seven of the eight stages in realistic agentic AI pipelines run entirely on the CPU.”
An LLM tasked with programming software, for example, will likely make tool calls to write code to files, move or replace files, download required packages, and build the software once the LLM believes it’s complete.
Kundu co-authored a paper on agentic AI optimization alongside researchers from Georgia Tech in Atlanta. They found the CPU is often idle while LLM inference is executed on a GPU and that, conversely, the GPU is often idle when tool calls are executed on the CPU. To optimize this, Kundu and his colleagues propose scheduling optimizations that can cut end-to-end latency (the time between the start and finish of the agentic workload) by up to 1.8-times under sustained load.
It’s a start, but the gains chase a moving target. Agentic systems generate work at machine speed and multiply it as they go. OpenAI’s inadvertent hack of Hugging Face saw its model fire off as many as 300 actions an hour, and a single agent can spawn sub-agents that make tool calls of their own.
And there’s one more important complication that may increase the workload on a CPU as models become more complex: safety guardrails.
Safety and policy checks on an agent’s actions are often specific rules that inspect syntax and log files, Kundu says. Guardrails may also use small models (under a billion parameters) to analyze task complexity or intent. Though they could be executed on a GPU, they often aren’t, because their small size and the need to minimize latency keeps the work on the CPU.
Increasing the number of CPUs available significantly decreases the latency for Llama-8B responses over longer sequence lengths.Source: Euijun Chung, Yuxiao Jia, et al.
Tokenization adds to bottlenecks
Euijun Chung, a PhD student at Georgia Tech, recently co-authored another paper, with findings that complement Kundu’s work. Chung and his co-authors found that when a server has too few CPU cores, it falls behind on dispatching work to the GPUs. That causes the GPUs to stall as they wait for instructions.
In addition to that, the paper touches on another key element of LLM workloads: tokenization.
Tokenization is a key first step in LLM inference. It converts text into integer token IDs that can be processed by the model. Unlike the matrix math required for most LLM inference, tokenization is branchy, data-dependent sequential string manipulation. Though it can be parallelized by chunking text, it’s not massively parallel in the same way as the bulk of LLM inference.
Tokenization of small prompts is a relatively trivial task and won’t tax even an entry-level CPU. However, an agentic model that makes tool calls must parse and tokenize the results of the call.
“If you have an ongoing
[…]
Content was trimmed to protect the source. Please visit the original article for the full text.
Read the original article: