Translated from the Korean original. Some screenshots show the original Korean interface.
1. When one incident triggers dozens of alerts
The volume of alerts operations teams handle every day has grown quickly along with system scale. As microservices became the norm and hybrid cloud and SaaS came to be used together, it is now common for a single slow external dependency to set off APM, infrastructure, response time, and user feedback alerts all at once. According to the Catchpoint 2025 SRE Report, a global SRE industry report, 62% of on-call engineers said they had missed an important alert because there were too many alerts. It has reached the point where the Google SRE Workbook recommends no more than 2-3 incidents per shift.
Today, rather than just cutting the number of alerts, operations are shifting toward catching the signals before an alert fires. The technology area that catches these early signals is predictive AIOps.
2. Operations are moving from reaction to prevention
For the past few years, AIOps focused on how quickly you could find the cause after an incident. The standard flow was: an alert fires, the operations team opens the screens, runs RCA (Root Cause Analysis), restores service, and writes a postmortem. More recently, early anomaly detection is clearly becoming part of standard operations. The alerting design guide in the Google SRE Workbook recommends putting signals that catch problems before users notice them at the top of alert priority.
This is not just a global trend. In Korea, the shift is visible at the policy level too. Through the amendment to the Electronic Government Act that the Ministry of the Interior and Safety put into effect in July 2025, the government wrote into law 24-hour monitoring and specific inspection standards for incident prevention across some 16,000 public administration information systems. With incident response now managed at the level of a social disaster, the role of IT operations teams is also moving from recovery after an incident to blocking it before it happens.
3. How far can a 30-minute early alert go?
The short answer: in some areas it works, and in others it does not. To answer precisely, we first need to separate two different functions.
Forecasting, which predicts future load and anomalies and shows them as graphs, and predictive alerting, which sends an alert in advance when the forecast goes beyond a threshold or the learned normal range, are different functions. In an environment with forecasting alone, the operations team has to watch the graphs themselves to catch a signal. Only when the two are combined do alerts buy you time to act.
That said, not every incident can be predicted. Below is a breakdown of incident types that can and cannot be predicted.
Predictable incidents: Anomaly Detection catches early signals that build up in time-series data, giving you roughly 30 minutes to respond in advance.
- Incidents where resources fill up gradually, such as memory leaks and disk usage • Incidents where DB lock waits and WAS response times grow little by little • Incidents tied to recurring hourly traffic patterns
Unpredictable incidents: incidents that happen instantly are outside the scope of prediction.
- External API outages, network disconnections • Regression bugs right after a code deployment • DDoS attacks and security incidents
When you evaluate adoption, sorting the incident types that occur most often in your environment into these two groups gives you an early sense of where prediction will have the biggest effect.
4. The operating environment that makes prediction possible
For predictive AIOps to make a real difference in production, several conditions have to be in place together. Adopting a tool does not mean signals 30 minutes before an incident get caught automatically. The operating environment itself has to be ready in four areas: training data, unified collection, correlation analysis, and standard response.
4-1. Normal operating data to learn everyday patterns
Prediction starts with a baseline for judging whether now looks different from usual. Learning-based models train on operating data over a period of time and automatically set a normal range that differs by system and by time of day. Trusting predictions before enough data has accumulated is risky, so right after adoption it is safer to set an explicit training period. This is typically 4-12 weeks, and the exact length is agreed with the vendor based on the service's characteristics and data periodicity.
Variety matters as much as volume. A model trained only on normal traffic tends to produce false positives in unusual situations such as quarter-end closing, promotions, or restarts after maintenance. Match your operations event calendar against the training data period so that, if possible, it covers at least one quarter of varied operating situations.
Threshold-based alerts and learning-based alerts both notify the operations team, but they set the baseline in completely different ways.
Category | Threshold-based alerts | Learning-based alerts |
Baseline | Fixed values set manually by the operator | Automatically learns normal ranges by hour, day of week, and season |
False positives | Normal fluctuations are often classified as alerts | Alerts only when values leave the learned normal range |
Accuracy over time | Fixed (adjusted manually by the operator) | Improves gradually as data accumulates |
Operational burden | Staff time spent resetting thresholds and cleaning up alerts | Only meaningful signals remain, easing the load on the operations team |
The accuracy of a learning-based model is more sensitive to label accuracy than to data volume. If the operations team clearly marks when past anomalies occurred, model accuracy rises noticeably even with the same training data. The habit of managing alert history not as plain logs but with labels such as normal, false positive, and real anomaly becomes a core asset for predictive AIOps.
4-2. Unified collection of structured metrics and unstructured logs
A model trained only on structured metrics such as CPU, memory, and response time has clear limits. The same 1-second response time means different things depending on which API and which call flow it came from, and structured metrics alone lose that context. What raises prediction accuracy is a unified collection setup that looks at structured and unstructured data together. A common setup runs real-time metric collectors alongside streaming pipelines such as Kafka and Fluentd.
Building unified collection all at once from the start is hard. Monitoring tools are often scattered across departments, with different data formats and retention policies. The safer approach is to go step by step: start with one or two services that have incidents most often, and widen the scope once you confirm prediction accuracy improves.
Data retention is something to check at the same time. Learning-based models need data over a certain period to work reliably, so in environments where metric retention is kept short to save costs, there may not be enough data to learn from even with unified collection. If you do not sort out the retention policy first, unified collection loses much of its effect.
4-3. Multi-domain analysis that captures correlations across areas
The signals 30 minutes before an incident do not show up in just one area. Lock waits on the DB side grow, WAS response times stretch little by little, and network latency starts to fluctuate along with them. Small changes happen in several areas at once. Monitoring one area alone, each stays within normal variation and is not caught as a signal, but looking at correlations across areas reveals a clear anomaly pattern.
Multi-domain correlation analysis is one of the features recent AIOps tools focus on most. In environments that use separate monitoring tools per area, this correlation analysis is technically impossible, so bringing data into one analysis engine is a prerequisite.
Just as important as technical integration is how departments work together. Where WAS, DB, and network are run by separate departments, working out which area and whose responsibility something is often takes longer than bringing the data onto one platform. To build an environment where unified RCA is possible, you need to agree in advance on analysis permissions and responsibilities across departments, along with integrating the tools.
4-4. A standard response defined for each predictive signal
The last condition is that prediction has to lead to actual response. Getting a signal 30 minutes early means nothing if nobody has decided what to do with it. If you link predictive signals in advance to standard response procedures (runbooks and playbooks: documents that define the actions to take for each incident type), the operations team can move to the next action without further decision-making.
When you connect standard response to automation, though, set the scope carefully. The usual approach is to keep the scope of automatic action narrow early on with a human verification step, then widen automation gradually as prediction accuracy stabilizes. When writing new runbooks and playbooks, it works well to gather postmortems of incidents from the past year and extract common response patterns. Documenting the procedures the operations team has actually followed naturally yields verified response scenarios, while trying to design an ideal procedure from scratch tends to produce documents that are disconnected from the field.
5. The XAIOps approach: one platform for all four
Once you start running these 4 areas on different tools, each tool has its own collection interval and data format, and more time goes into keeping them consistent. That is why operations become more efficient when they are handled together on one platform. EXEM's XAIOps is an AIOps solution that handles all 4 operating conditions covered above, from training data and unified collection to multi-domain analysis and standard response, in one environment, and it has been deployed in large-scale environments in Korea, including major commercial banks.
Category | Traditional monitoring | Predictive AIOps |
Baseline setting | Set manually by the operator for each system | Confidence intervals generated automatically with deep/machine learning |
Alert timing | When a threshold is exceeded (after the fact) | Early signals from load forecasting 30 minutes to 1 hour ahead combined with Smart Alert |
Data scope | Single metrics per area | Unified analysis of WAS, DB, network, and logs |
RCA flow | Manual analysis while switching between tools | Correlation analysis on one screen + automatic topology extraction |
Link to response | Alert → human judgment → tool execution | Event correlation analysis + standard response procedures based on runbooks and playbooks |
Here is how each condition is implemented in XAIOps.
- Training data → learning-based baseline + Smart Alert It learns normal operating data for each system and sets confidence intervals automatically. Combined with Smart Alert, which automatically detects anomalies against the learned normal range and sends alerts, the operations team gets early signals without having to check the graphs themselves.
- Unified collection → single analysis engine A unified collection setup in which structured metrics from WAS, DB, and network flow into the same analysis engine as unstructured logs. The load forecasting feature, which predicts load 30 minutes to 1 hour ahead, also runs on this data.
- Multi-domain analysis → 3D node map topology You can see call paths and the time of an anomaly on one screen. Without piecing together cross-area correlation patterns by hand, you can see at a glance which path an anomaly started from.
- Standard response → runbook and playbook integration Event correlation results and standard response procedures (runbooks and playbooks) are handled together within one platform. The operations team does not have to analyze from scratch each time and can move quickly to the next action with verified procedures.
6. What 30 minutes means, and what to check when evaluating adoption
We have mentioned 30 minutes often, but the exact lead time varies by system environment. In some systems signals appear 5 minutes ahead, in others an hour ahead. What matters is not the exact number of minutes but that the operations team gets breathing room to respond after receiving an alert. Even 5 minutes of warning lets the person in charge open the screens and stand by, and with 30 minutes you can even spread the load out gradually.
The value of predictive AIOps goes beyond operational efficiency to reducing business risk. Preventing service disruptions from incidents in advance also prevents revenue loss, damage to customer trust, and compliance issues. Especially in sectors like finance and the public sector, where a single incident can be treated as a social disaster, the value of prevention keeps growing year after year.
If you are evaluating adoption, first check whether your operating environment matches any of the situations below.
- There are so many threshold alerts that even normal fluctuations are classified as alerts
- WAS, DB, and network monitoring are scattered across departments, so RCA takes a long time
- One incident spreads to several systems, and the same event produces dozens of alerts
- Even when predictions arrive, the operations team has no standard procedure for the next action
If these concerns sound familiar, see how EXEM's XAIOps can help improve your operating environment.
Sources
Alerting on SLOs - Google SRE Workbook - Google SRE The SRE Report 2025 - Catchpoint (2025) Raising trust in digital services: Ministry of the Interior and Safety overhauls information system incident response (Korean) - ZDNet Korea (2025.03.20) Electronic Government Act takes effect in July, opening the public IT automation market (Korean) - Newsis (2025.05.28)
Related articles
