![[NPU Ops Guide #1] Why Existing Tools Fall Short for NPU Monitoring](https://www.notion.so/image/attachment%3A5ca81492-2bd8-409f-bb86-e6a1dc14f9b1%3Acover.jpg?table=block&id=3e4a9953-f4c4-4b28-9ad4-9bce8606e9d1&cache=v2&width=1200)
As cloud services built on Korean NPUs launch one after another, more environments are running GPUs and NPUs side by side. Each vendor already ships tools to check device health, but seeing several kinds of devices on one screen means first aligning metric names and decision criteria. In this post, we look at where commercial Korean NPUs stand today, then explain why you need separate monitoring criteria when you run GPUs and NPUs together.
1. Inference spending overtakes training
In 2026, inference is expected to account for more cloud AI infrastructure spending than training. The reason is the growth of AI features that get called over and over every day, such as chatbots, anomaly detection, and real-time recommendations. These features run again every time a user sends a request, even after the model is finished. More requests mean more computation, so running a finished model continuously has become a bigger cost driver than training a new one.
Gartner's forecast shows the same shift. Of 2026 spending on cloud AI infrastructure services, inference is expected to take $23.3 billion and training $19 billion. Inference makes up 55% of the total and is projected to reach 59% in 2027. This does not mean training spending is shrinking. Over the same period, the overall market nearly doubles from $21.5 billion to $42.3 billion, and inference spending grows even faster than that.
Training and inference are hard to separate completely. Even after a service launches, additional training or retraining can follow, based on inference results and user feedback. But they use resources differently. Training processes large computations over a relatively long time. Inference repeats short computations every time a user request comes in. The more requests there are, the more computation there is to handle, and when responses slow down, users feel the service is slow right away.
Training adjusts a model's weights based on data. Inference feeds real requests into a finished model to get answers. At the service stage, inference results and user feedback can flow back into training.
Handling all of this inference load on GPUs can hurt cost efficiency. Large GPUs are designed to process large, varied parallel computations at once. Inference, by contrast, repeats relatively small computations in short bursts, so even on the same GPU the compute resources often go underused. The device is occupied, actual throughput is low, and power costs accrue for the full time it is occupied.
That does not mean the GPU's role is shrinking. Large-scale training and complex experiments still need GPUs. But for inference, where a finished model runs repeatedly, it now makes sense to evaluate other options on cost efficiency.
The device getting attention as that alternative is the NPU. An NPU is a chip designed to efficiently process the computations AI inference needs. If the GPU is a device that handles a broad range of parallel computations, the NPU focuses on raising throughput per watt for inference work. Rebellions, a Korean NPU maker, says its chips are built on an architecture optimized for inference and deliver up to 3.2x higher energy efficiency than conventional GPUs. So can you adopt these chips today?
2. 2026: Korean NPUs go commercial
2-1. Private sector: NPU services from 3 cloud providers
In Korea, 3 cloud providers launched services built on Korean NPUs over the course of 2026. Korean NPUs are no longer devices used only in pilot projects. They are now regular cloud services that companies can sign contracts for and use.
Provider | Service | NPU | Launch |
Gabia | NPUaaS (virtual machines) | Rebellions ATOM-Max | April 2026 |
KT Cloud | NPU Server (public sector only) | Rebellions ATOM Plus | June 2026 |
Samsung SDS | SCP NPUaaS (subscription) | FuriosaAI RNGD (Renegade) | July 2026 |
Even with the same Korean NPUs, the service models differ a little. Gabia NPUaaS is offered per virtual machine, so you can tailor the environment down to the OS kernel, and a single ATOM-Max card provides 128 TFLOPS of FP16 compute and 64GB of NPU memory. KT Cloud also uses virtual machines, but serves them from a data center dedicated to the public sector.
Samsung SDS turned RNGD into a subscription service on Samsung Cloud Platform, where you choose 1, 2, 4, or 8 cards.
The models differ, but all 3 let you use NPUs in the cloud instead of buying them and installing them in your own servers. You can validate inference load first without an upfront hardware investment, then scale as needed. It is hard to say every workload has been validated, but things have at least reached the point where a procurement team can request a quote and review a contract.
2-2. Public sector: procurement grounds and performance validation
To offer a cloud service to public institutions, the Cloud Security Assurance Program (CSAP) certification is a key requirement. The KT Cloud NPU Server mentioned above was the first NPUaaS in Korea to earn it. Companies taking part in public-sector AX (AI transformation) projects can meet security regulations and earn extra evaluation points under the policy favoring Korean-made chips at the same time.
Even before commercial private-sector services appeared, the government had been funding data centers built on Korean AI chips. The Ministry of Science and ICT is putting KRW 826.2 billion through 2030 into the K-Cloud Project to build data centers based on Korean AI chips. Phase 1, the Korean NPU data center build-out, started at a combined private and public scale of 39.9PF (PetaFLOPS, 1,000 trillion floating-point operations per second), and NHN Cloud took the largest share at more than 22PF.
Budget alone is not enough to make an adoption decision. You also need a way to check performance under real service conditions. K-Perf, created by the Ministry of Science and ICT, the Telecommunications Technology Association (TTA), and the Institute of Information & Communications Technology Planning & Evaluation (IITP), is an evaluation framework for exactly that. It tested Rebellions and FuriosaAI products on 4 use cases: chatbots, document search, report generation, and long-document summarization. For chatbots and document search, the first token mostly arrived within 1-2 seconds, and text generation ran at 2 to 9 times human reading speed. Token throughput per watt held steady even when requests piled up.
The two companies tested differently. Rebellions ran GPT-oss-120B, a 120-billion-parameter model, on a single card, while FuriosaAI split EXAONE 4.0 32B across several RNGD cards. So when you evaluate adoption, do not look only at the result numbers. Also check which chip, and how many of them, produced that performance.
Public institutions need grounds to justify both budget and performance. To get through security regulations and procurement, they have to explain why new equipment is needed and show that performance is sufficient in a real service. With K-Cloud and K-Perf providing both kinds of grounds, putting Korean NPUs on the shortlist has become much easier.
As NPU adoption grows, operations teams have to move beyond watching only GPUs. Existing GPU clusters handle training and complex experiments, while cost-sensitive inference load is split off to NPUs. This is not swapping GPUs for NPUs. It is a setup where both accelerators run together in one data center. At that point the next question is no longer whether to adopt, but what the monitoring criteria are. You need to decide what you use to watch GPU and NPU health, and on what criteria you set alerts.
3. What vendor tools show out of the box
3-1. Device status with a single command
First, you can check device status with each vendor's built-in tools. On GPUs, nvidia-smi is widely used. Rebellions ships rbln-smi and FuriosaAI ships furiosa-smi with their SDKs.
Both companies have also released Exporters for metric collection. Rebellions' Metrics Exporter and FuriosaAI's furiosa-metrics-exporter both export metrics in Prometheus format. So the collection itself is relatively easy to hook into an existing Grafana setup. On top of that, Rebellions provides NPU Feature Discovery, a firmware updater, and hardware diagnostic tools for Kubernetes environments.
So far, this does not look very different from running GPUs. The main Korean NPU suppliers come down to Rebellions, FuriosaAI, DEEPX, and a few others, but from an operations standpoint you have to learn each vendor's tools and metric scheme separately. What operations teams actually need, though, is not to look at each vendor's screen separately. The trouble starts the moment you try to bring metrics from several vendors into one dashboard.
3-2. One alert rule, 3 queries
Collecting metrics with Prometheus and viewing them in a dashboard such as Grafana is a common setup for AI infrastructure monitoring too. In NVIDIA GPU environments, dashboards based on DCGM Exporter let you see key metrics such as utilization, power, and temperature in a fairly consistent way. But once you try to view GPUs and several NPUs together, you may have to realign the criteria. Even for the same "utilization," each vendor uses different metric names and label structures.
For example, even for the same "utilization above 90%" alert, GPUs and NPUs cannot easily share one rule. For NVIDIA GPUs you can base it on DCGM metrics, but for Rebellions and FuriosaAI NPUs you have to write queries against each vendor's metric names. So even a single alert criterion can require separate per-vendor queries and conditions.
The metric lists themselves differ too. A metric available on one device may not exist on another. To view multiple devices on the same criteria, you first have to decide which metrics serve as the common baseline. Adding more devices from the same vendor just means copying the existing configuration, but when the vendor changes, you have to realign metric names and alert conditions.
This gap is not limited to monitoring. The same problem shows up in profiling and model porting. When an NPU compiles a model, it restructures the model's layers for that chip, so GPU profilers are hard to use as-is, and the compiled model ends up tied to a specific NPU. A model compiled for Rebellions ATOM will not run if you load it onto a Furiosa RNGD as-is.
4. The monitoring standard GPUs have and NPUs lack
This raises a natural question.
"Is there a way to read metrics under the same names across vendors?"
Right now, there is no standard for reading NPU metrics under the same names regardless of vendor. The lack of an NPU monitoring standard becomes clearer when you compare it with GPUs.
Dashboards and alerting settled in fairly naturally on GPUs because NVIDIA DCGM has served as the de facto standard for metric collection. DCGM exports values such as SM utilization, power, temperature, and NVLink traffic under fixed names. Grafana dashboards based on NVIDIA DCGM Exporter are also publicly available, so GPU cluster operations teams can manage key metrics and alerts on the same criteria as they add more devices.
DCGM (Data Center GPU Manager) is NVIDIA's data center GPU management tool. Because it exports values such as SM utilization, power, temperature, and NVLink traffic under fixed names, operations teams that standardize on DCGM Exporter can collect key GPU metrics under fairly consistent names and wire them into dashboards.
But this approach was built around the NVIDIA CUDA environment. NPUs differ in architecture, metric names, and collection method. So if you point a dashboard built for GPUs at NPU metrics as-is, the metric names will not match, and key panels may come up empty or fail to read the expected values.
Vendor-neutral standards do not change the picture much. If you check the OpenTelemetry hardware semantic conventions, the observability standard, the list of well-known values for the hw.type attribute, which indicates the device type, includes the following values.
battery, cpu, disk_controller, enclosure, fan, gpu, logical_disk, memory, network, physical_disk, power_supply, tape_drive, temperature, voltage
The list has a gpu value for GPUs, but no value for NPUs. Even in the vendor-neutral standard, the NPU is not yet defined as its own device type. In the end, there is still no clear common baseline for organizing NPU metrics under the same names regardless of vendor, the way there is for GPUs.
That is not to say standardization has made no progress. In v1.34, Kubernetes promoted the core features of DRA (Dynamic Resource Allocation) to general availability (GA). The remaining features are still in beta and alpha. Unlike the old approach, which only counted devices, DRA schedules based on device attributes and handles accelerators other than GPUs within the same framework.
But DRA only solves the scheduling part. Kubernetes can decide which device a workload runs on. Once scheduling is done, though, it is back on the operations team. Whether that device is running well still has to be judged from vendor-specific metrics. The monitoring screens operators check every day have not been standardized.
5. A 5-point checklist before adoption
Gaps in NPU monitoring criteria are hard to deal with if you only discover them after adoption. If you already run NPUs or are considering them, it is worth first checking whether you can answer the five questions below.
Pre-adoption NPU checklist
- Can you tell whether NPU core utilization means idle or saturated?
- Can you see which model is using how much NPU memory?
- When inference responses slow down, can you compare GPU and NPU metrics over the same time window?
- When you replace an NPU device, can you identify the affected services in advance?
- Can you find processes that hold an NPU but do no actual work?
In GPU environments, these are items teams have grown used to checking through existing dashboards and metric schemes. On NPUs, though, the same questions can be hard to answer right away. What you need here is not to collect more metrics, but to first define the criteria you need for operational decisions.
Incident analysis gets more complex when you run GPUs and NPUs together. When inference responses slow down, finding out whether the cause is on the GPU side or the NPU side means comparing metrics from different monitoring screens over the same time window. If the tools are separate, you have to repeat this process every time an incident occurs.
What matters in the end is not piling up metrics per device, but organizing metrics you can compare on the same criteria. Since each vendor's tools and metric schemes differ, settling on common criteria before adoption can be the starting point for stable operations.
Sources
Gartner Forecasts Worldwide AI-Optimized IaaS Spending to Grow 96% Through 2026 - Gartner (2026.08.10) Gabia launches cloud service based on Korean NPUs - Electronic Times (2026.04.09, Korean) KT Cloud opens public-sector-only NPU market - ZDNet Korea (2026.06.04, Korean) Samsung SDS to launch Korea's first Korean NPU subscription service in July - Newsis (2026.04.02, Korean) K-Cloud Project Phase 1 gets underway - Ministry of Science and ICT (2023.06.26, Korean) K-AI chip blueprint unveiled: large-scale references and a full-stack ecosystem - IT Daily (2026.06.05, Korean) Korean NPUs spread to public, finance, and manufacturing - Apple Economy (K-Perf pilot validation, Korean) Rebellions and Red Hat announce Red Hat OpenShift AI on Rebellions NPUs - Rebellions (2025.12.12, Korean) RBLN Metrics Exporter - Rebellions Installing Furiosa Metrics Exporter - FuriosaAI Semantic conventions for common hardware metrics - OpenTelemetry Kubernetes v1.34: DRA has graduated to GA - Kubernetes (2025.09.01) DCGM Feature Overview - NVIDIA NVIDIA DCGM Exporter Dashboard (ID 12239) - Grafana Labs
Related articles
