- US - English
- China - 简体中文
- India - English
- Japan - 日本語
- Malaysia - English
- Singapore - English
- Taiwan – 繁體中文
As AI adoption accelerates, inferencing is becoming the dominant workload behind many large language model (LLM) deployments. Every chatbot response, search result, code suggestion, and translation request depends on the real-time execution of AI models. As these workloads scale, the infrastructure required to support them is becoming just as important as the models themselves. In this blog, we look at how different LLM query types place different demands on GPU memory bandwidth, power, throughput, and energy efficiency.
Understanding AI inference
Unlike training, which teaches a model by processing large datasets,
inference is the operational workload that powers real-world AI
applications. Every inference request requires moving vast amounts of
data between compute, memory, and storage resources to deliver timely
results. As AI adoption continues to grow, inference has become one of
the primary drivers of infrastructure demand, making memory capacity,
bandwidth, performance, and power efficiency critical to enabling AI at
scale.
Large language model inference is the process of using a trained AI
model to generate outputs from new inputs. When a user submits a prompt,
the model analyzes the request and predicts the most likely sequence of
tokens to create a response. Every chatbot conversation, text summary,
translation, recommendation, or generated piece of content is powered by
LLM inference.
The hidden complexity of LLM inferencing
The energy cost of a single LLM query can vary by up to 7x depending on the type of prompt (see table 1 below). That kind of variation has real implications for infrastructure planning, and it all comes back to memory.
Most people think of LLM inference as a simple process: you feed in a prompt and receive tokens in return. However, under the hood, it is a complex and highly variable computational process. The type of query sent to an LLM fundamentally changes the resource requirements, including how much memory bandwidth gets consumed, how hard the GPU works, how much power is drawn, and how efficiently that power is converted into useful output over time.
That's a big deal for infrastructure planning. As our CEO, Sanjay Mehrotra, has noted, "Without data, there is no AI", and without the right memory and storage infrastructure, there is no efficient inferencing. Understanding how different query types drive different demands is what makes capacity planning, power budgeting, and cost optimization work.
Test methodology
To quantify these differences, we ran inference tests using GPT-OSS-120B across six distinct query categories (healthcare, technology, science, programming, translation and roleplay). We collected real-time metrics on GPU utilization, memory bandwidth, power draw, throughput, and energy efficiency.
- The model used was GPT-OSS-120B, deployed via NIM containers.
- Metrics were collected using a custom Python Gradio interface with real-time GPU telemetry via NVIDIA® nvidia-smi.
- Storage was provided by one 15TB Micron® 9550 NVMe™ SSD for high-speed model and dataset loading. The high sequential read throughput of the Micron 9550 SSD was essential for rapid model weight loading and reducing cold-start latency during deployment.
- The server platform used was the HPE ProLiant® DL384 Gen12.
The six query categories
Six categories were selected to represent common real-world LLM use cases, each with distinct computational characteristics:
Healthcare: Concise, evidence-based medical queries requiring structured factual recall.
Sample prompt: "List evidence-based factors influencing cardiovascular risk; keep concise."
Technology: Technical summarization requiring domain knowledge and structured output.
Sample prompt: "Summarize the key differences between HBM3E and HBM4 in bullet points."
Science: Explanatory queries demanding precise technical language without simplification.
Sample prompt: "Explain the core mechanism of CRISPR, no analogies."
Programming: Code optimization tasks requiring analysis, reasoning, and code generation.
Sample prompt: "Optimize this function for speed and explain only the bottleneck."
Translation: Multilingual translation requiring large context windows and extended generation.
Sample prompt: "Translate this paragraph into Japanese with no commentary."
Roleplay: Creative generation with persona constraints and stylistic requirements.
Sample prompt: "Act as an overloaded spacecraft AI... write a brief internal status log."
Key findings: Resource requirements vary dramatically
Performance summary across categories
LLM Inference Resource Requirements by Query Category
Figure 1: Comparative view of key inference metrics across all six query categories
| Category | Prompt tokens | Completion tokens | TTFT (sec) | DecodeTPS | Avg power (W) | Decode energy (J) | Efficiency (tokens/J) |
|---|---|---|---|---|---|---|---|
| Health | 64 | 257 | 2.286 | 55.35 | 170.5 | 792 | 0.3246 |
| Technology | 68 | 436 | 7.188 | 62.48 | 172.3 | 1203 | 0.3625 |
| Science | 65 | 259 | 1.513 | 69.76 | 179.5 | 666 | 0.3886 |
| Programming | 244 | 1082 | 2.004 | 57.12 | 170.6 | 3232 | 0.3348 |
| Translation | 1150 | 1588 | 3.373 | 58.81 | 171.6 | 4634 | 0.3427 |
| Roleplay | 93 | 255 | 1.294 | 64.88 | 182.5 | 717 | 0.3555 |
Table 1: Decode-phase performance summary across six query categories (excludes prefill)
Figure 2: Memory bandwidth utilization over time, spiking at prefill
Memory bandwidth: The backbone of inference
Memory bandwidth is often a primary constraint for LLM inference. Our measurements show peak memory bandwidth requirements ranging from 348 GB/s (Health) to 475 GB/s (Roleplay) across categories. This level of variation means that infrastructure should be provisioned for the most demanding workload in the mix, not the average.
The time-series data in Figure 2 depicts a highly bursty memory bandwidth usage during the prefill phase. Observations of the memory bandwidth usage shows spikes during the prefill (prompt processing) phase and a steadier pattern during the decode phase. Owing to this bursty behavior, average bandwidth utilization significantly understates the actual peak requirement.
This underscores the importance of memory technology. High-bandwidth memory solutions like Micron HBM3E are designed to deliver the sustained bandwidth that AI inference workloads demand. On platforms like the GH200, HBM works alongside Micron LPDDR5X, which serves as the CPU-side memory, to keep data flowing through the unified memory architecture. . A single Roleplay query peaks at 475 GB/s, about 10% of the GH200's 4.9 TB/s of HBM3e bandwidth, so roughly ten at once can saturate the memory bus. This helps illustrate why memory bandwidth can become a primary constraint on inference performance as models scale.
Key Insight: Up to 475 GB/s of memory bandwidth is required to handle the most demanding workloads. Even 'simple' queries like health Q&A require 348 GB/s at peak. Memory bandwidth is often the limiting factor in inference throughput.
Note: A single time-series graph is sufficient to depict the memory bandwidth utilization over time because, in memory-bound inference workloads like these, memory bandwidth, GPU utilization, and power draw are tightly correlated. They move in lockstep: when memory bandwidth spikes during prefill, GPU Streaming Multiprocessor (SM) activity and power draw spike with itrise accordingly. When bandwidth settles during decode, so do the other two. Tracking memory bandwidth alone gives you a reliable proxy for GPU utilization and power draw, because provides a reliable proxy for GPU utilization and power draw, since the temporal patterns are effectively identical. They share the same root cause: how hard the GPU is working at each moment during inference.
Figure 3: Distribution of GPU utilization values. Taller boxes indicate more variable, bursty workloads
GPU utilization: Bursty by nature
The box plot below shows how GPU utilization varies across query categories. Wider distributions indicate more bursty, less predictable workloads.
Key insight: GPU needs are bursty, not steady-state. If provisioning is planned based on average utilization, the system will be under-provisioned during peak demand. The box plot makes this clear: utilization swings widely within a single inference request, which means traditional 'average utilization' metrics can be misleading when it comes to capacity planning.
Power consumption and energy efficiency
Power consumption varies significantly by query type. While average GPU power draw stays relatively consistent across categories (170W to 183W), the total energy consumed per query varies by up to 7.0x, from 666J (Science) to 4634J (Translation), as seen in Figure 2. This gap is driven primarily by the duration of each query and the number of tokens it generates.
Figure 4: Token volume and time breakdown. Translation and Programming dominate resource consumption
Token volume and time: The real cost drivers
The biggest differences between categories show up in token volume and total processing time. Translation generates 6.2x as many completion tokens as Roleplay, along with 6.9x more decode time and 6.5x more decode energy consumption (Table 1). Programming queries generated 4.2x as many completion tokens as science queries.
Figure 5: Energy efficiency trends over time, where higher values are better
Energy efficiency: Tokens per joule
As Figure 6 shows, efficiency isn't flat during a query. It starts low during prefill, when the GPU is drawing power but hasn't generated any tokens, then improves as output builds. Longer queries like Programming and Translation end up most efficient (~0.33 tok/J), since the upfront prefill energy is spread across more tokens. Short or prefill-heavy queries land lower. For teams managing power budgets and sustainability targets, this metric keeps getting more important.
Energy efficiency, measured in tokens generated per joule of energy, varies from 0.3246 tok/J (Healthcare) to 0.3886 tok/J (Science). Science queries achieve the highest efficiency because they maintain high decode throughput with relatively lowerwhile incurring relatively low power overhead. This metric is becoming increasingly important for data center operators who are managing power budgets and sustainability targets.
Conclusion
This analysis makes one thing clear: LLM inference is far from a uniform workload. The type of query fundamentally shapes the resource profile, from memory bandwidth and GPU utilization patterns to power consumption and energy efficiency.
The data shows that workloads are complex and GPU needs are bursty. Planning for average utilization will lead to performance degradation during peak demand. Instead, here is what teams should focus on:
- Consider resource requirements of the most demanding workloads expected in the deployment mix
- Consider both burst GPU utilization and averages
- Consider power budget with worst-case query categories in the expected mix
- Monitor per-category metrics to identify optimization opportunities
- Consider worst-case power requirements and size power budgets appropriately
By understanding the resource fingerprint of each query type, organizations can build more efficient, cost-effective, and responsive AI infrastructure. As inference workloads scale, it's important to build on the right memory and storage foundation: high-bandwidth HBM3E, high-capacity LPDDR5X, and high-throughput NVMe SSDs.
Ready to optimize your AI infrastructure? Explore Micron's portfolio of data center memory and storage solutions designed for AI workloads at Micron AI solutions.
Technical Footnotes
1 All inference tests were conducted on an NVIDIA GH200 platform featuring an integrated 72-core Grace Arm CPU + Hopper H200 GPU with unified memory architecture (480GB LPDDR5X + 144GB HBM3E). Performance results are based on internal testing and may vary based on system configuration, workload characteristics, and software versions.
2 Storage performance enabled by the Micron 9550 NVMe SSD. Learn more at Micron 9550 NVMe SSD
