Translated from the Korean original. Some screenshots show the original Korean interface.
What stays with QA after AI takes over repetitive verification
Hi, I'm Gyeongsu Jeong from Product Technology Research Team 1 at EXEM, where I handle QA for MaxGauge.
QAI, the term in the title, is one I made up. It may look like a familiar word, but it is not an established quality assurance term.
Since joining EXEM and working in QA, I have naturally come to use AI often. At some point I felt that QA, too, needs to shift to a way of working alongside AI. Here is what QAI means to me.
∙ Quality Assurance with AI
∙ Quality Automation Intelligence
∙ Quality Assurance Intelligence
If traditional QA has been about checking and assuring product quality, QA going forward should use AI to verify faster, more accurately, and across a wider scope. So I decided to use "QAI" as the word that captures the direction of my work.
1. The problem started with repetitive verification
Where I used AI most was verification work that is highly repetitive and heavy on manual effort. A lot of QA work demands accuracy and consistency even when the checks themselves are repetitive.
Comparing multiple datasets or organizing results looks simple, but in practice it takes considerable time and focus. With manual verification, human error can creep in no matter how careful you are, and there were limits to how reliably we could verify every item within a fixed timeframe.
Could AI and automation make this more efficient and more consistent? That question was where it all started.
2. Why I chose RTS agent verification as the automation target
During my OJT period after joining, I verified MaxGauge's RTS agent several times. Along the way I noticed how much of the verification work repeated itself, and I started to think, "Couldn't this part be automated?"
RTS (Real Time Server) is the MaxGauge agent that collects database performance data and sends it on. Verification is only done once you have confirmed that the collected data is accurate, that the processes and commands work correctly, and that everything displays properly on screen.
Later, while preparing my final presentation, I fleshed out the idea of automating the verification process.
Below is the idea sketch I made while designing the automation tool. I drew it out by hand to decide how many seconds apart to line up Target and Repository, how many records to batch per comparison, and how many panels to split the screen into.
After the presentation, I built out the features so the team could actually use it. The automation tool is now part of the team's verification work.
This work gave me a chance to connect my existing interest in automation directly to MaxGauge QA.
Verification starts by preparing the target DB. We create a table, create a logging job, confirm that logging is active, and delete the created objects once verification is done. Comparisons run on two metric families, STAT and WAIT, with a specified query period.
The screen places metrics from the same point in time side by side for the Target DB and the Repository DB. Collected metrics such as session logical reads and DB time are plotted by family, so you can see at a glance where the two values diverge.
That said, RTS verification does not end with this one screen. There are several areas to check, so we first had to decide where to start automating.
3. How we split verification into 3 stages
The goal of RTS verification automation is to automate, step by step, the key verification criteria the team runs repeatedly. Instead of wiring everything up at once, we judged it safer to set an order and fill in one area at a time, since each area differs in automation difficulty and pass/fail criteria.
The automation tool now fully implements (100%) all 3 verification areas below, and we are in the phase of refining details to improve stability and accuracy.
Implemented verification areas
- Collected data and consistency verification
- RTS process and command verification
- Screen display and usability verification
So rather than adding new features, we are now focused on confirming that results stay consistent when run repeatedly under the same conditions in real verification work, and on covering exception cases.
This feature compares data collected from Target and Repository in two ways. Delta compares diff_value at the same point in time, and Delta Sum compares data summed over time intervals.
The Delta screen shows both the raw row count and the unique timestamp count. The raw row count is the total number of rows actually collected, and the unique timestamp count is the number of timestamps compared, with duplicate rows in the same second counted once. Comparing the expected timestamp count with the unique timestamp count tells you whether data is missing or whether multiple rows were collected at the same time.
The result of comparing diff_value at the same timestamps is shown as a per-second match rate and a mismatch count. The cumulative time series shows how the Target and Repository values change together, so you can find the interval where a difference appeared. This screen therefore verifies not only the number of collected records but also whether the actual values match at each timestamp.
The Delta Sum screen compares the Target total and the Repository total for each time interval. diff_ratio is calculated as Repository total ÷ Target total × 100, and under the current criteria a result within 99-101% counts as normal on a total basis. It also shows the mismatch count per time bucket and the time-bucket match rate, so you can catch cases where the overall totals are similar but values differ in a specific time window.
So this check does not judge whether Target's absolute values are strictly equal to Repository's. It judges whether data collected over the same time range and under the same criteria compares consistently at the same timestamps and in the same time intervals.
4. The automation tool needs verification too
Of course, automation raised a new concern. The tool we built to verify the product also needed verification.
Does the tool behave as intended? Does it judge correctly in exception cases? Can we trust its verification results? We had to confirm all of this.
At first I wondered, "Is it really worth the resources to build an automation tool and then verify it again?" But in the long run, the time invested now will clearly pay off in less repetitive work and higher verification quality. I am already starting to feel that effect in my day-to-day work.
This verification checks whether PL/SQL tests run on the Target DB are properly collected into the Repository DB. It does not just check that the data exists; it compares execution count and execution time per SQL to verify whether anything was dropped or changed during collection.
Cases 1-5 on the screen are test procedures with different execution times. We repeatedly run everything from long-running SQL to SQL that finishes very quickly, to also check whether collection results vary with execution time.
When the test finishes, we query the following items from
V$SQL on the Target DB as baseline values.- SQL_ID
- Execution count
- Total execution time
- Average execution time per execution
Next, we query the results collected in the Repository DB for the same SQL_ID and compare them with the Target DB baseline. If the execution counts match and the execution time and average execution time fall within a set range, the data can be considered properly collected.
"Not collected (<1s)" appears when a very short-running SQL is not found in the Repository. It indicates the SQL may not have been collected because of its short execution time, and that label alone does not mean there is an error. Conversely, if the execution time is long enough but there are no results or the execution counts do not match, you need to also check the collection time, the collection interval, and the Repository load status.
Meanwhile, the data analysis results at the bottom show, for each statistic, the number of collected data points, the Target and Repository averages, and the average difference.
The average difference is calculated as follows.
Average difference = Target average - Repository average
A positive difference means the Target average is larger, and a negative one means the Repository average is larger. The colors on screen only distinguish the direction of the difference; color alone does not decide normal or error.
Target and Repository values may not be exactly identical because of the gap between collection time and storage time. So rather than looking at any single value, it is important to check together whether there is enough collected data, whether the two environments' averages follow a similar trend, and whether differences at specific points in time keep recurring.
In the end, the purpose of this feature is not to confirm that Target and Repository values are exactly the same, but to confirm that data collected under the same conditions compares consistently overall.
5. AI as a design partner, not a code generator
I also leaned heavily on AI while building the automation tool. From logic design and coding to reviewing exception cases and defining result verification criteria, I worked through it with AI to raise the quality of the tool.
I did not use AI just to generate code. We thought through the design stage together: which items to automate and what criteria to use for judging results.
I am still using AI to automate my work. With AI, I list the verification items that need repeated checks and work out which data to compare, what criteria separate normal from abnormal, and what exception cases might come up. Based on that, I implement the automation code and verification logic, then check again whether the results actually serve the verification goal, and keep refining.
Our team also runs a QA AI ChatBot to cut the time spent searching past issue-handling records.
My biggest takeaway from working with AI is that it can go beyond a tool that writes code and become a partner that helps organize the verification perspective and design the direction of automation.
6. Closing thoughts
In the end, QAI as I see it is not AI replacing QA. AI and automation run repetitive verification more consistently, while people design the verification direction, review the results, and own the final quality. That, I think, is the core of QAI.
The goal of automation is not simply to work faster. It is to cut the time and resources spent on repetitive verification, minimize human error, and reinvest the freed-up resources in higher-priority verification and quality improvement. That way I can focus more on work that raises product quality, such as additional verification, result analysis, and identifying improvements.
Going forward, I want to keep improving how we verify through AI and automation, and grow as a QA engineer who builds better quality.
Explore MaxGauge for database performance management.
👉🏻 Learn more about MaxGauge
Related articles
