Surging HBM Demand in AI Data Centers
At the center of today's AI infrastructure race is a surprising technology: HBM (High Bandwidth Memory). As the AI boom continues, demand is exploding not only for GPUs but also for high-performance memory. Market analyses show the HBM market growing quickly as AI servers spread, and it is expected to expand significantly in 2026 compared with the previous year. This trend is clearly visible in the semiconductor market as well. As demand for data centers and high-capacity AI servers rises, overall memory demand is also climbing sharply, and memory demand centered on HBM, used in AI infrastructure, is driving market growth.
As investment in AI data centers grows, one question naturally comes up.
What really determines AI performance?
Many people think of GPUs, but the answer is not that simple. In a computer system, performance is always limited by the slowest point, the bottleneck.
1. Why bottlenecks never go away
A computer system is built as a hierarchy that runs from disk to memory, cache, and registers. In this hierarchy, each device is designed with its own trade-off between speed and price.
In general, registers and cache are very fast but expensive and small. Memory and disk, on the other hand, are relatively cheap but slower and larger. Because speed and cost move in opposite directions, performance gaps between layers are inevitable. And as long as there are speed gaps, bottlenecks will appear.
In other words, a bottleneck is less a limitation of a specific technology and more a natural result of devices with different speeds and costs working together.
2. How computer architecture has dealt with bottlenecks
Since the von Neumann architecture, computer systems have evolved continuously to address bottlenecks. The most common approach is to add an intermediate layer (a buffer) between two devices.
For example,
- Memory capacity was expanded to narrow the speed gap between disk and CPU,
- and L1, L2, and L3 cache layers were added inside the CPU to speed up data access
The same approach applies not only to hardware but also to software architecture. The key point is that it does not eliminate the bottleneck entirely. It adds an intermediate layer to soften the speed gap.
3. The same bottleneck problem in AI infrastructure
This problem is repeating in AI infrastructure.
Generative AI and large language models (LLMs) run on billions to hundreds of billions of parameters. Training and running inference on these models requires massive GPU-based parallel computation. That is why today's data centers are quickly being rebuilt around a GPU-centric architecture.
GPUs are designed to run parallel computation across thousands of cores, which makes them a great fit for AI workloads. In practice, most AI workloads, including model training, image generation, and natural language processing, run on GPUs. For this reason, companies have been adopting GPU servers at scale, and GPU monitoring to manage GPU utilization and resource status has become an important operational task.
But as GPU performance improved rapidly, a new problem appeared. Data movement speed started to matter more than compute speed.
3-1. The hidden variable in AI performance: data movement
AI computation is not just calculation. In practice, it goes through the following steps.
- Read model parameters from memory
- Run large-scale computation on the GPU
- Write the result data back to memory
Throughout this process, the GPU keeps reading and writing huge amounts of data. The problem is that GPU compute performance has improved so quickly that data supply can no longer keep up.
If memory cannot deliver data fast enough, the GPU sits idle, unable to compute. In other words, the AI performance bottleneck is moving from the GPU to memory.
3-2. Enter HBM (High Bandwidth Memory)
HBM (High Bandwidth Memory) emerged to solve this problem. HBM is memory designed to deliver much higher data transfer rates than conventional DRAM. It stacks multiple memory chips vertically to greatly increase data bandwidth, and it sits very close to the GPU to minimize data movement latency.
HBM keeps getting more important in AI accelerators too. For example, NVIDIA's latest AI accelerators keep expanding HBM capacity, and some products carry hundreds of GB of HBM or more. This is because memory bandwidth has to grow alongside GPU compute performance to make full use of it.
4. How AI data center architecture is changing
The spread of AI workloads is changing the infrastructure architecture of data centers themselves.
Traditional data center | AI data center |
CPU-centric compute | GPU-centric compute |
Standard DRAM | HBM-based memory |
Storage-centric processing | Built around data movement |
Per-server performance management | System-wide bottleneck management |
If CPU performance and storage throughput were the key factors in traditional data centers, in AI data centers the core is an architecture where GPUs, memory, and networks work together.
Because AI infrastructure runs massive parallel computation, the following elements matter as much as the GPU itself.
- Memory bandwidth
- High-speed networking between GPUs
- Data pipelines
- Distributed training environments
AI infrastructure is becoming an environment where the balance of the overall system architecture matters more than the performance of any single device.
5. The next stage of the AI infrastructure race
As AI technology advances, many companies are focused on the race to secure GPUs. There have even been GPU shortages. But AI data centers are now facing a different question.
"Are we making full use of our GPU performance?"
Simply acquiring GPUs does not automatically improve AI infrastructure performance. For GPUs to perform as intended, many other factors, such as memory bandwidth, network speed, and data pipelines, have to be optimized together.
In recent AI infrastructure in particular, the next bottleneck after the GPU is moving more and more clearly to memory. In the end, AI competitiveness is not decided just by getting more powerful GPUs. How you design and operate the entire AI infrastructure, with GPUs, memory, and networks working together, will be a key factor in data center competitiveness going forward.
AI data center performance is not determined by GPU performance alone.
But properly understanding GPU utilization is where AI infrastructure operations begin.
Sources
"AI and HBM demand explodes": chip stocks set for a second rally this year - The Financial News (2025.12.31) AI accelerators race to expand HBM capacity, long-term DRAM demand keeps growing - Maeil Business Newspaper (2025.12.07)
Related articles
