1. Kubernetes Now Runs AI Agents and LLMs
Over the past year, one change has stood out clearly in IT operations. Beyond simple internal chatbots, more and more organizations are running AI workloads such as customer-facing agents, internal RAG systems, and embedding and vector DBs on Kubernetes (K8s).
The CNCF 2025 annual survey, published in January 2026, shows this trend well. 82% of respondents using containers run K8s in production, and 66% of organizations hosting generative AI models run some or all of their inference workloads on K8s. CNCF titled the report "Kubernetes Established as the De Facto 'Operating System' for AI," and the message is clear: K8s has become the default infrastructure for running AI.
This shift is quickly turning into standardization work. CNCF announced the Certified Kubernetes AI Conformance Program and started building compatibility and reproducibility standards for AI workloads running on K8s. On the inference server side, tools like NVIDIA Triton and vLLM are now widely used on K8s, and more operations teams have to manage GPU resources and inference workloads themselves.
This trend raises a new question for operations teams. GPU resources, model response quality, token usage, inference call flows. Should data that teams have rarely handled in K8s environments now be treated as operational data too?
In How Far Have AI Agents Automated IT Operations?, we looked at how AI agents are entering operations work. This post takes the infrastructure view underneath that: how observability changes when operations teams run AI workloads on K8s themselves.
2. Why the Scope of K8s Observability Is Growing
K8s observability has traditionally been organized around 4 areas: infrastructure, platform, application, and network. Now that AI workloads are running in clusters, there is more data to track on top of these existing areas.
Compare the observability data of regular workloads and AI workloads, and the difference is clear.
Category | Regular workloads | AI workloads |
Core resources | CPU, memory, disk | GPU utilization, VRAM, tensor cores |
Per-request metrics | Response time, error rate | Response time + token usage + response quality |
Call flow | Distributed traces across services | Service traces + inference steps, tool calls, RAG retrieval |
Cost structure | CPU time, traffic pricing | GPU time + model token pricing + external API calls |
Definition of quality | Response codes 200/500 | Response accuracy, hallucination rate, user feedback |
If you cannot follow infrastructure data and AI workload data for a single user request on the same timeline, then when an incident happens it is hard to narrow down whether the cause is in GPU resources, the model itself, or an external API it called.
There is also a clear cost reason to look at both together. Gartner's September 2025 announcement estimated that worldwide AI spending will reach $1.5 trillion in 2025, and that more than 90% of AI server spending goes to GPUs. Whether for training or inference, GPUs take up a large share of AI infrastructure cost, so if operations teams cannot see GPU resources and model responses together, cost control is the first thing to suffer.
If you adopt AI observability and K8s observability separately, aligning the two data sets on a timeline during an incident can become an extra burden for the operations team. At adoption time, check in advance whether the structure lets you see both in the same analysis flow.
3. 4 Data Areas to Track When Running AI Workloads on K8s
Once you run AI workloads on K8s, observability built around infrastructure, platform, application, and network alone can no longer fully explain the state of your operations. It is more accurate to say that the set of things to observe in AI inference environments extends existing metrics, logs, and traces, rather than a completely separate area appearing.
The items that matter most in operations are GPU resources and inference infrastructure, model response quality, inference traces and token flow, and data pipelines including RAG.
3-1. GPU Resources and Inference Infrastructure
In AI workload environments, GPUs are expensive, and a GPU bottleneck has a large impact on the service. In operations, you need to look not only at GPU utilization but also at memory usage (VRAM), per-model usage patterns, load imbalance across nodes, and request queue buildup to find the real bottleneck. NVIDIA Triton Inference Server exposes a Prometheus-compatible metrics endpoint, which makes it easy to connect to an existing K8s metrics collection setup.
But GPUs should not be read the same way as CPUs. CPU health is often judged by average utilization, but in GPU inference environments, a momentary memory squeeze or a change in model loading state can immediately cause higher latency or failed requests. So rather than average utilization, it matters more to look at memory pressure during peaks, per-model resource usage, and response patterns after batch settings change.
3-2. Model Response Quality
In traditional application monitoring, response codes and latency were enough to judge service health to a reasonable degree. With AI workloads, a request that is processed normally does not guarantee a good answer. System metrics can look fine while the actual responses fall short on accuracy, consistency, or appropriateness.
Quality metrics you can use in operations include response length outliers, the share of suspected hallucinations, and user feedback. Guardrail bypass attempts and sensitive data pattern detection should also be tracked from a security and compliance standpoint. Metrics like hallucination rate, which require evaluating the content of responses, are hard to measure automatically in real time, so they are often handled with a separate evaluation model, sampled review, or checks against an evaluation set.
This kind of data is different in nature from infrastructure metrics like CPU utilization or response time, which you can read at a glance. Even so, for operations teams to catch anomalies early, model response quality data must be visible on the same timeline as infrastructure and application data.
3-3. Inference Traces and Token Flow
In LLM-based services, producing a response often takes more than a single API call. When a user request comes in, RAG searches the vector DB for relevant documents before the model is called, external tools or APIs are called if needed, and the results are fed back into the prompt to generate the final response. If this flow is not recorded as a traceable trace, it is hard to figure out afterward where the latency came from.
OpenTelemetry GenAI Semantic Conventions is a leading standard for organizing this inference data in a common format. For example, you can record which model was called, how many input and output tokens were used, and which systems or tools were involved as standard attributes. Attributes such as
gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens fall into this category.In production, you need to see inference latency, token usage, external tool call time, and retrieval latency linked in a single trace. Only then can you tell whether a slow response is a problem with the model itself, a bottleneck in retrieval, or a delay in an external API call. It also helps keep your data from being overly tied to one vendor's format when you adopt or replace tools.
3-4. Data Pipelines (RAG and Vector DBs)
In retrieval-augmented generation (RAG), the data path outside the model becomes a core part of operations, not just the model. This includes metrics such as vector database search time, relevance of search results, reranking latency, embedding call rate, cache hit rate, and dependence on external APIs. Even when users only see slow responses or lower quality, the real cause may lie in the retrieval step or an external data path rather than the model.
This is the area existing operations teams miss most easily. Model response time can look normal while slower vector DB searches or more cache misses drive up the cost of external embedding calls. So when operating AI workloads, you need to observe not only the model's own state but the entire data pipeline used to generate answers.
4. Where Infrastructure Observability Meets AI Observability
Looking at AI workload data on its own gives you only half the operational value. What matters in practice is interpreting the existing K8s observability data and AI workload data together, on the same timeline and in the same service context. Root cause analysis, performance tuning, and cost control only become practical once this connection is in place.
4-1. How GPU Resource Pressure Affects Inference Quality and Latency
GPU resource pressure does not end with higher utilization. It can lead to higher inference latency, failed requests, and signs of lower response quality. For example, when VRAM headroom shrinks, the share of large-context requests that get processed can drop, or the batching strategy can change, making responses slower and shifting the output length distribution for some requests.
At that point, infrastructure metrics alone may show GPU utilization that does not look very high, but looking at response length distribution, request failure rate, and token throughput together makes the anomaly at that moment much clearer. The key is not to judge the state from a single metric. You need to look at GPU utilization, VRAM pressure, queue buildup, token throughput, and output length distribution together. Only then can you tell whether the cause is a resource problem, a model configuration change, or a change in prompt characteristics.
4-2. How K8s Events Turn Into Inference Latency
On K8s, inference performance can degrade without any change to application code. When a GPU Pod is recreated by events such as a rolling update, node failure recovery, eviction, or autoscaling, loading model weights, initializing the runtime, and rebuilding caches can cause high initial latency. Users just see slow responses, but the real cause may be a K8s operational event rather than anything inside the application.
This pattern only shows up when you line up K8s events and inference latency data on the same timeline. If the two are scattered across different tools, the operations team can spot a latency spike and still fail to connect it right away to the infrastructure event that happened just before. In AI workload environments, distributed traces alone are not enough; you need to look at cluster events, deployment history, and node state changes too in order to narrow down the real cause.
4-3. Combining GPU Costs and Token Costs per Service
The cost of running AI workloads comes from infrastructure and model usage at the same time. Infrastructure costs such as GPU time, storage, and network are closer to K8s operational data, while input and output token usage and external API call costs are closer to AI workload data. If you look at them separately, you can know the total, but it is hard to see which service or which model path is driving up the cost.
5. What Teams Often Miss When Adopting
When teams actually try to adopt AI workload observability, the same problems come up again and again. Here are the items worth addressing early, during tool selection and data structure design, to cut down on trial and error.
1) The cost of adopting tools separately
If you adopt AI observability tools and infrastructure observability tools separately, linking the two data sets on the same timeline is left to the operations team as a separate task. Sometimes the cost of data integration ends up larger than the cost of the tools. Checking early on whether both data sets can flow into the same analysis engine helps reduce the workload later.
2) Who owns response quality metrics
Some teams see response quality as the domain of the AI team or the data team only. But from the user's point of view, lower quality is often indistinguishable from slow responses or errors. Operations teams need to see quality metrics such as response length outliers, the share of suspected hallucinations, and user feedback on the same screen as the infrastructure dashboard, so they can catch anomalies early and isolate the cause.
3) Ownership of cost data and the tagging scheme
If each team manages infrastructure costs and token costs separately, figuring out the total cost of a given service can mean a long round of checks across departments. Agreeing on common tags at the service, model, and tenant level early on makes later discussions about cost attribution and optimization much faster.
4) Retention policy for inference traces
Inference traces record a multi-step flow for every request, so the data volume grows fast. If you do not set retention periods and sampling rates before adoption, the data may already be deleted when you actually need it for incident analysis. If storing every trace is not feasible, a practical first step is to design a conditional sampling policy that always keeps requests that errored or exceeded a latency threshold.
Trying to cover every service from day one often makes the scope too large and delays adoption. A more practical approach is to integrate infrastructure and AI observability for one or two core services first, confirm the effect, and then expand gradually.
In Korea, you also need to review the requirements of the AI Basic Act. Check whether your customer-facing agents or internal RAG systems might qualify as high-impact AI, and if needed, design response logs, sensitive data detection, and audit trail data together with your observability pipeline to avoid building the same thing twice.
6. Before You Start Evaluating
The infrastructure, platform, application, and network data that K8s observability has always covered remains the foundation in AI workload environments. But once you run AI workloads on K8s, you also have to handle things the existing setup cannot fully explain, such as GPU resource state, model response quality signals, inference traces, and data pipelines including RAG.
The point is not to build an entirely new system on the side. It is more practical to extend your existing operational observability with AI-specific data and connect the two. Once you start viewing both on the same timeline and in the same service context, you can trace root causes more accurately and make operational decisions that weigh performance and cost together more quickly.
Demand for full-stack observability that connects DB, APM, infrastructure, log, GPU, and model response data into a single flow is growing in operations teams in Korea as well. EXEM's exemONE is an integrated observability platform worth considering in this direction, built to handle operational data from hybrid environments, including K8s, on a single platform.
If you are evaluating adoption, start by checking whether your operations environment matches any of the items below.
- You run AI inference workloads (LLMs, embeddings, vector DBs) on K8s, but GPU metrics are not connected to your existing infrastructure monitoring.
- Model response quality data (response length, hallucination rate, feedback) does not appear on your operations dashboard.
- You have no dashboard that compares GPU costs, token costs, and external API costs per service or per model.
- Your inference traces are not organized according to OpenTelemetry GenAI conventions, so data continuity may suffer when you switch tools.
Before comparing individual tools, it is a good idea to put the question of how to connect infrastructure observability and AI observability on your evaluation schedule first.
In AI workload operations, whether you can connect different kinds of operational data on the same timeline and in the same service context determines real operational efficiency as much as which tool you choose.
Sources
Kubernetes Established as the De Facto 'Operating System' for AI as Production Use Hits 82% in 2025 CNCF Annual Cloud Native Survey - CNCF (2026.01.20) CNCF Launches Certified Kubernetes AI Conformance Program - CNCF (2025.11.11) Cloud Native AI Whitepaper - CNCF TAG Runtime Semantic conventions for generative AI systems - OpenTelemetry Gartner Says Worldwide AI Spending Will Total $1.5 Trillion in 2025 - Gartner (2025.09.17) NVIDIA Triton Inference Server Documentation - NVIDIA Framework Act on the Development of Artificial Intelligence and the Establishment of a Foundation for Trust (Korean) - Korea Law Information Center
Related articles
