Plenty of data, hard to interpret
IT operations today are more complex than ever. As cloud, containers, and microservices spread quickly, companies now handle far more monitoring data than before. The logs, metrics, and traces this produces never stop pouring in. To deal with this, many companies have adopted observability.
Reality did not match expectations, though. According to Observability Pulse 2024 from Logz.io, many companies that adopted observability said their MTTR (Mean Time To Recovery) did not shrink as expected and actually grew. Companies have more data than ever, yet they still lack the ability to interpret it quickly and turn it into action. Alerts get buried in noise, dashboards keep getting more complex, and fixing problems still takes a long time.
1. What is observability?
So what exactly is observability? Observability is an approach that makes a system's internal state observable and interpretable through external signals. You can't look directly inside the system to see what's happening, but you can infer it from three kinds of data: logs, metrics, and traces.
Observability is usually discussed around three elements.
- Logs: Records of events that occur in the system. Used to understand the detailed context of a problem.
- Metrics: Numeric indicators such as CPU usage and response time. Show the system's performance state at a glance.
- Traces: Track the path a request takes through a distributed environment. Help identify bottlenecks quickly.
💡 Combining these three types of data takes you beyond simple monitoring to a much deeper understanding of what is happening inside the system.
If traditional monitoring only told you that "a problem occurred," observability goes one step further and can explain "why it occurred." This has let companies gain deeper visibility, diagnose problems faster, and maintain service quality. But as we saw above, adopting observability does not automatically guarantee operational efficiency. The burden of interpreting data still falls on the operations team, and a new approach is needed to close that gap.
2. What LLMs change
A change drawing attention in IT operations lately is the arrival of large language models (LLMs). An LLM is an AI model trained on vast amounts of data to understand and generate natural language; when a user asks a question, it gives an answer that fits the context. Combined with observability, this changes how operations work. Until now, interpreting logs, metrics, and traces meant writing complex query languages or analyzing several dashboards. With an LLM, an operator can get the insight they need just by asking a question.
- "What caused yesterday's payment service incident?"
- "Which server had the CPU usage spike?"
- "Where did response time get slower compared to last week?"
For questions like these, the LLM analyzes logs, metrics, and traces together and presents an answer. In other words, the LLM turns data gathered through observability into contextual insights people can interpret. Instead of spending time interpreting data, operations teams can focus on quick decisions and problem solving. The result is the potential to improve operational efficiency, incident response speed, and even team productivity.
3. LLM-based observability scenarios
LLMs take observability to a new level. So how can they be used in real operations?
3-1. Insight analysis through questions
First, when a user asks a question in natural language, the LLM can query the data and answer. For example, you can ask questions like these.
- "Show me the Top10 hosts by CPU usage last week."
- "Analyze the cause of the ORA errors that occurred yesterday."
This can go through an NL2SQL process (natural language → SQL conversion → query execution → result return), or use MCP (Model Context Protocol) to connect a monitoring solution's API as a tool. With this approach, the LLM doesn't "judge"; it finds the data, and the operator makes the final call. That makes it realistic to implement and lowers the risk of wrong judgments.
3-2. Root cause analysis of operational incidents
Say a user reports an error in the payment service. Previously, an engineer had to trace the cause by looking at logs, metrics, and traces one by one. Now, with a question like "Analyze the cause of the payment error at 14:00 today," the LLM queries the logs, metrics, and traces for that time window and explains the analysis. This dramatically shortens MTTR (Mean Time To Recovery) and directly improves service stability. There is a technical limit, though: context window constraints caused by huge volumes of logs, metrics, and service context. This can be addressed by combining it with ML-based anomaly detection to narrow the scope and pass only summarized information to the LLM.
3-3. Onboarding support for new engineers
Engineers who just joined an operations team take a long time to understand the system architecture and past issues. An LLM can learn from past incident response records, log patterns, and dashboard information to answer questions like "How did we handle it when a similar problem happened before?" New engineers don't have to dig through huge manuals and documents; they can get up to speed quickly by asking about past history.
4. Technical challenges
Of course, LLM-based observability doesn't only promise a bright future. Real adoption and operation come with the following challenges.
Hallucination
LLMs sometimes make up content that isn't in the actual data. If hallucination happens in an observability environment, the operations team may rely on a wrong root cause analysis, leading to unnecessary actions or wasted time. So rather than trusting LLM answers as is, you need a structure that presents them together with the supporting data.
Data privacy and security
Observability data can contain sensitive data such as service logs and user request information. Using it for LLM training or analysis risks violating privacy regulations (GDPR, Korea's Personal Information Protection Act, and others). Security and governance measures such as data minimization, anonymization, and access control must be in place.
Cost
Using LLMs incurs large compute resources and API call costs. In real-time monitoring especially, huge volumes of logs and metrics must be processed continuously, so cost optimization becomes a key challenge. Operations teams need to think about "which data to pass to the LLM, and when," and in some cases a hybrid approach such as pre-summarization → LLM analysis may be needed.
Organizational barriers to adoption
LLM-based observability requires changes to the existing work culture and processes of operations teams. Engineers need time to get used to natural-language interfaces, and teams have to figure out how to run it alongside existing dashboard- and query-based ways of working.
5. Outlook
LLM-based observability is still at an early stage, but its direction is clear. Until now, observability has mainly focused on collecting and visualizing data. Going forward, it will evolve into a decision-making partner that interprets data and helps operators make better decisions. The expected changes are as follows.
- Strategic advice: When an operator asks, the observability tool goes beyond summarizing data to offer interpretation and response strategies.
- Expansion of business observability: Goes beyond system performance to connect with customer experience and business results.
- Combination with automation: Evolves from analysis alone toward semi-automated operations that execute predefined responses.
- Ecosystem growth: Various LLM-based features expand on standards like OpenTelemetry and grow quickly.
Observability will go beyond a technical innovation to become a key factor in raising operations team productivity, organizational agility, and ultimately a company's digital competitiveness.
Sources
2024 Observability Pulse Report - logz.io
Related articles
