![[NPU Ops Guide #2] Why NPUs Run Short Even When Utilization Is Low](https://www.notion.so/image/attachment%3A341805b9-162c-4b5e-b212-e2ebe910c1fe%3Acover.jpg?table=block&id=7e4f5382-ffe8-4444-a1e0-fb677847dc23&cache=v2&width=1200)
Korean NPUs have started to power large-scale commercial services. On September 21, 2026, Rebellions announced that its NPUs had been deployed in four commercial AI services at SK Telecom. The services are A. (A dot) call summaries, streaming speech synthesis, an AI customer center and scam text detection, and the traffic handled by Rebellions infrastructure reaches about 14 million requests a day.
SK Telecom did not dedicate these NPUs to individual services. It linked dozens of NPU units and hundreds of accelerator cards into an NPU farm and runs it as a pool that multiple internal services share.
Running NPUs as a shared resource this way has made expansion decisions harder, because requests saying devices are short sometimes do not match the numbers on the monitoring screen. Service teams request more capacity when the NPU farm has no devices left to assign to them. Yet the NPU utilization on the monitoring screen can still read low.
The two values disagree because utilization and allocation rate measure different things. Utilization shows how busy the assigned devices are, while allocation rate shows the share of all devices that have been assigned to workloads. Whether any devices are left for a new workload is judged by allocation rate, so devices are short when allocation rate is high, even if utilization is low.
Being short, however, does not mean you need to add capacity right away. If some allocated NPUs are not actually being used, they can be reclaimed and assigned to new workloads. So when an expansion request comes in, start by checking how much the allocated devices are actually used. Only then can you tell whether you truly need more devices or whether the NPUs already allocated are underused.
1. Why allocation is high when utilization is low
Allocation rises while utilization stays low because Kubernetes allocates NPUs by device count, not by actual usage. For Rebellions NPUs, the device plugin registers them on each node as a resource called rebellions.ai/npu, and workloads request as many units of this resource as they need. According to the Kubernetes device plugin documentation, such extended resources can only be requested in whole numbers and cannot be overcommitted or shared between containers.
An NPU assigned this way cannot be used by other workloads until the Pod terminates. Even if the device does almost no computation in the meantime, the scheduler still counts it as an allocated resource. This is how allocation stays high while utilization reads low.
Allocation can be high while utilization is low. Allocation rate shows whether a device has been assigned to a workload, and utilization shows how much computation the assigned device actually performed.
1-1. Allocations that stay in place without inference requests
Inference servers are a typical case where the allocation remains but computation stops. An inference server Pod keeps running even when there are no requests so that it can take the next one. Computation stops, but because the Pod is still running, its NPU allocation is not released either.
The bigger the gap between busy and quiet hours for a service, the longer this waiting time becomes. As more of these Pods accumulate, you get closer to a situation where utilization is low, yet the scheduler sees no devices left. That said, this waiting may be what the operator intended so the service can respond as soon as a request arrives, so it cannot be dismissed as waste on its own.
1-2. Allocation only in whole-device units
Allocating by device count alone creates another kind of waste. Even if a workload does not need a whole device, it is assigned an entire one, so unused compute and memory are tied up along with it. GPUs are no different. In Kubernetes,
nvidia.com/gpu: 1 only means one GPU is assigned; it does not specify how much of its VRAM or compute will be used.A survey by Cast AI, a Kubernetes cost optimization company, gives a sense of how much accelerators assigned as whole devices are actually used. According to the 2026 State of Kubernetes Optimization Report, average GPU utilization in clusters without optimization was just 5%. Because this is one company's own analysis, it cannot be applied as-is to every environment, but it is a useful reminder that accelerators acquired at significant cost may be barely used for actual computation.
The problem is that this leftover capacity does not show up in Kubernetes resource information. Container memory is managed by specifying the amount needed, as in
memory: 4Gi, but there is no standard resource that plays the same role for VRAM or NPU device memory. So to see which process uses how much memory inside an assigned device, you have to check separately with vendor tools such as nvidia-smi or rbln-smi.1-3. Why the NPU card count differs from the device count
To read allocation rate correctly, you first need an accurate count of the denominator: the total number of devices. For NPUs, this number can differ from the number of cards. According to the guide to running Rebellions ATOM published by Red Hat, a single ATOM-Max card contains four chips, and each chip is recognized as one NPU device. So a server with two cards installed registers as eight NPU devices in Kubernetes.
In other words, the unit you buy and manage is the card, but the unit Kubernetes allocates is the device. With ATOM-Max, the number of devices Kubernetes can assign is four times the number of cards. If you calculate allocation rate from the card count in an equipment inventory, you will misjudge the resources actually available to assign.
2. Why 100% utilization does not mean saturation
Nor can you conclude that a device is saturated just because utilization reads high. The same 100% can mean different things depending on what utilization measures. GPUs, whose measurement method is documented, make this difference clear.
The nvidia-smi manual defines GPU utilization as the percentage of time during the sample period in which one or more kernels were running. By this definition, utilization can read close to 100% even if just one small kernel keeps running.
In other words, this value tells you only how long the device was working, not how fully it used its compute resources. That is why GPUs have a separate metric showing how widely the compute units were used. A representative one is SM_ACTIVE, provided by DCGM, NVIDIA's GPU management tool. According to the DCGM official documentation, SM_ACTIVE is the time work was active on each SM (Streaming Multiprocessor), averaged across all SMs.
SM_ACTIVE becomes clearly useful when read alongside GR_ENGINE_ACTIVE, the fraction of time the engine was active. Looking at both lets you separate how long the GPU worked from how widely it used its compute units during that time.
The DCGM documentation presents an SM_ACTIVE of 0.8 or higher as a necessary condition for using a GPU effectively and notes that a value below 0.5 likely indicates inefficient use. So if a GPU shows 100% utilization but a low SM_ACTIVE, you can consider consolidating work before adding capacity.
GPUs have a documented measurement method like this, but NPUs do not. Unlike the nvidia-smi manual, the documentation for rbln-smi, the command-line management tool for Rebellions NPUs, describes Util only as Utilization. So rather than judging by utilization alone, it is better to also check the performance state (P-state) that rbln-smi shows alongside it.
P-state (Performance State) divides a device's clock and power level into steps. In rbln-smi, it appears in the Perf column, and a higher number means a lower performance step.
The same utilization should be read differently depending on the performance state. According to the state table in the rbln-smi documentation, P2 is the normal clock step, and P10 is a step where the clock is halved due to heat. P12, where the clock drops to its minimum, indicates a system hang, and P14, where the clock is off, indicates an idle state.
Utilization at P10 is measured at half clock, so it is hard to treat it as the same load as the same number at P2. Even for a device with high utilization, then, check whether heat has lowered its clock before requesting more capacity.
3. How to find NPUs that are allocated but unused
Utilization and performance state show how hard a device is working, but to find NPUs that are allocated yet unused, you also need to know which processes are occupying the device.
This information is in the context list that appears below the device information when you run rbln-smi without options. rbln-smi breaks down the processes using an NPU by context and shows each one's process name, PID, context ID, allocated memory (Memalloc) and status (Status).
Status is shown as run or idle, and it can differ from context to context even within the same device. So even when the device's overall utilization is high, some contexts may be holding memory without doing any computation.
3-1. Idle contexts on a device at 98.7% utilization
The example output in the official rbln-smi documentation shows just such a device.
In the example output, the NPU with device ID 0 (Device 0) shows 98.7% utilization, so it looks like a device working almost nonstop. But of the three contexts on this device, only one, which allocated 1.9GiB, is computing in the run state. The other two remain idle while holding 468.0MiB and 88.0MiB of memory respectively. The utilization number alone does not reveal that part of this device's memory is tied up without computation.
Checking contexts also lets you pick out NPUs that are allocated but unused. Idle contexts appear even on a device at 98.7% utilization, and the likelihood is greater on a device with low utilization and only idle contexts left. Some idle contexts may be leftovers that were not cleaned up after a job finished. Others, of course, are simply waiting for the next request after a brief lull.
3-2. Finding the workload behind an idle context
To tell which kind an idle context is, first find the workload that created it. According to the Kubernetes documentation, the kubelet's PodResources API tells monitoring agents which Pod, namespace and container each device on the node is assigned to.
Matching rbln-smi's context list against these assignments device by device reveals the workload that created an idle context. Once you know the workload, you can ask its owner whether it is still in use and decide whether to reclaim it.
PodResources API assignment data is also needed for devices with no contexts. In the example output, Device 1 has 0.0% utilization, its performance state is P14 (idle), and no contexts have been created, so it looks like an empty device. But rbln-smi only shows processes that are using a device; it does not tell you whether Kubernetes has allocated it. Whether this device is assigned to a workload has to be checked separately through the PodResources API.
4. What to check before requesting more capacity
Looking at allocation rate, utilization and context status together, you can estimate what situation an expansion request reflects, as shown below. These metrics only narrow down what to check, though. An allocation may reflect the service operator's intent to respond immediately even when requests are few, so decide on reclaiming or reducing it only after confirming with the service owner, not from the metrics alone.
Overall allocation | Device utilization | Context status | Likely situation | First step |
High | Mostly high | Mostly run | True saturation | Consider expansion, adjust priorities |
High | Low | Idle for a long time | Idle occupancy | Check whether the job has finished or is held for standby, then consider reclaiming |
High | Low | Alternating idle and run | Intermittent requests | Confirm the intent of the allocation, then consider adjusting the number of allocated devices |
High | High on only some devices | Varies by device | Uneven load | Rebalance workloads |
Low | Low | Few contexts | Headroom | Assign new workloads without expansion |
If it is not true saturation, you can first look at whether existing devices can cover the need. But do not reduce an allocation just because utilization is low. For a service that must respond immediately even when requests are rare, holding a device while waiting may itself be the allocation the operator intended.
Intermittent requests in the table can suggest that more devices than currently needed have been allocated to an inference service. Each Pod holds the devices it requested, so devices stay tied up in proportion to the number of Pods even when there are no requests. However, the Pods may have been provisioned generously for performance or stability, so decide whether to reduce the number of allocated devices only after confirming the intent of the allocation with the service owner.
The tools available for reclaiming or reallocating differ between GPUs and NPUs. NVIDIA GPUs can be partitioned at the hardware level into multiple instances with MIG, and Time-Slicing lets multiple workloads share a GPU by dividing its time. So you can move low-utilization workloads into smaller units and free up an entire GPU.
RSD groups on Rebellions NPUs, by contrast, combine multiple devices into one to run large models, so they are not used to split a single card. Instead, Rebellions states in its Rebellions Inference Platform overview that it supports multi-tenant inference through a DRA (Dynamic Resource Allocation) driver.
Under the device plugin approach, however, allocation is still by device count. For NPUs, it therefore matters more to request the right number of devices from the start, and to find idle occupancy and confirm it with the owner.
5. Wrapping up
In environments where multiple services share NPUs, like SK Telecom's NPU farm, it is hard to decide on expansion from utilization alone. Looking at allocation rate and utilization separately and checking performance state as well is what lets you tell true saturation apart from idle occupancy.
On top of that, matching device context information against Kubernetes allocation data lets you find, workload by workload, resources that are allocated but unused. Reclaiming the allocations that turn out to be unnecessary after checking with service owners can sometimes let you serve workloads waiting for assignment without bringing in new devices.
That is why, for NPUs that have become a shared resource, seeing which workloads occupy how much matters more than checking utilization more often.
Sources
Rebellions NPUs adopted for four SKT commercial AI services, processing '4 billion tokens a day' - ZDNet Korea (2026.09.21, Korean) RBLN Kubernetes Device Plugin - Rebellions Device Plugins - Kubernetes Cast AI's 2026 State of Kubernetes Optimization Report Reveals GPU Utilization at 5% - Cast AI (2026.04.21) Running AI inference on Rebellions ATOM NPU with Red Hat AI - Red Hat Developer (2026.05.27) nvidia-smi Manual - NVIDIA Feature Overview - NVIDIA DCGM Documentation Device Monitoring (rbln-smi) - Rebellions Time-Slicing GPUs in Kubernetes - NVIDIA GPU Operator Rebellions Inference Platform - Rebellions
Related articles
![[NPU Ops Guide #1] Why Existing Tools Fall Short for NPU Monitoring | EXEM](https://www.notion.so/image/https%3A%2F%2Fex-em.com%2Fapple-icon.png%3Fd5f9e2f6269b07ff?table=block&id=0199f421-61cf-439e-b2d4-806059ff41f4&cache=v2&width=1200){kind=icon}
![[NPU Ops Guide #1] Why Existing Tools Fall Short for NPU Monitoring | EXEM](https://www.notion.so/image/https%3A%2F%2Fstorage.ex-em.com%2Fv1%2Fblog%2Fthumbnail%2F74%2F2026%2F09%2F10%2F09-43-03-663%2FNPU%2520Ops%2520Guide1_1789000987311_c380.jpg?table=block&id=0199f421-61cf-439e-b2d4-806059ff41f4&cache=v2&width=1200)
