Translated from the Korean original. Some screenshots show the original Korean interface.
Upgrading our AI chatbot for QA and engineering work to version 1.2
Hi, I'm Minseok Oh from Product Technology Research Team 1.
Our team handles QA for MaxGauge. One of our key roles is to quickly analyze issues that come up at customer sites and give engineers accurate guidance they can use on site right away.
But one difficulty keeps coming back in this work. Every time a similar issue comes in, the whole process of finding past history, checking logs, and putting together the relevant guide starts from scratch. Experience and knowledge are clearly piling up, but it has always bothered us that they are hard to pull up quickly at the moment we need them.
QA AI ChatBot grew out of that problem. Built on the ChatGPT API, it keeps gaining the features we need: issue history lookup, Q&A on installation and configuration guides, and image-based symptom checks. We hope it helps a little in the day-to-day work of QA and engineers, and we want to share how we built it.
1. It looks familiar, but checking always starts from scratch
1-1. Our issue response flow and its bottleneck
Our team's issue response flow looks roughly like this.
When an engineer visiting a customer site runs into something that needs urgent checking, they report the issue through our messenger. Once it is reported, we ask for the version, configuration details, and related logs. When the logs arrive, we check whether a similar case has come up before. If there is a past record or we can figure out how to fix it, we put together a guide and send it over.
The flow itself looks simple, but in practice every step adds time.
Finding past history is the worst of it. We know we have handled a similar issue before, but figuring out which version and environment it happened in, and how we solved it back then, means digging through multiple documents and records by hand.
Along the way, we end up relying on memory, searching old chat logs, or asking the person in charge again.
The same symptom can have a different cause in a different version, and the same setting can behave differently depending on the environment. Even an issue that seems familiar usually turns up something new to check once you look closely.
This difficulty is not limited to issue response. When verifying a new version, we need past history to see whether a similar case existed, and we need it again when analyzing cases that only reproduce in certain environments. Each time, we repeated the same cycle of searching documents and checking with the person in charge.
In the end, the problem comes down to one thing.
Experience and knowledge were clearly piling up, but there was no structure for pulling them up quickly when we needed them. Wondering whether we could shorten this flow was the starting point for QA AI ChatBot.
1-2. Why we started with the ChatGPT API
Once we decided to build the tool, the first question was how to implement it.
We considered building our own pipeline that searches documents and generates answers, and running a separate search database. But the more complex the structure, the bigger the maintenance burden, and above all, getting it into real work came first. We decided the priority was to build it fast and actually use it.
By that standard, the ChatGPT API was a good fit for us.
✅ File Search comes built in, so just uploading documents gave us search-based answers right away.
✅ Conversation history is managed automatically per Thread, so we got a Q&A flow that keeps context without building it ourselves.
✅ It also mattered that we could start quickly with just an API integration and no separate infrastructure.
The first data we loaded were the documents we look up most often in daily work: issue history, installation and configuration guides, release notes, and product manuals.
Just by uploading these documents, with no complex setup, we had the basic structure for the ChatBot to answer questions like "How did we handle this error before?" and "What's the install order for this version?" It was a small start rather than a finished tool, but even that began to make a noticeable difference in our work.
2. What we learned running 1.0
2-1. The features people actually used
While running 1.0, the most used features were issue history lookup and Q&A on installation and configuration guides.
Issue history lookup worked as we hoped. Just having the ChatBot find and answer "How did we handle this error before?" cut our checking time noticeably. The biggest change we felt was that we no longer had to ask the person in charge again or dig through documents ourselves.
Q&A on installation and configuration guides was used often too. Asking about the install order or configuration items for a given version returned answers based on the relevant documents, and engineers started using it as a channel to check things themselves on site.
2-2. Limits that surfaced in operation
That said, running it also exposed clear shortcomings.
The first was answer accuracy. Now and then it missed the intent of a question, or gave an off-target answer even when the documents had the relevant content. Instead of using answers directly in our work, we sometimes had to verify them again.
Next, response speed also fell short. When an engineer is waiting on site, a slow answer inevitably erodes trust in the tool.
Finally, image-based questions did not work smoothly either. On site, people often attach a screenshot of an error or a settings page and ask, "What is this?" Sending an image was technically possible, but there was no intuitive way to paste it straight into the chat and ask, so in practice it was hard to use.
3. What changes in 1.2
3-1. Model upgrade: from GPT-4o to GPT-5.2
The core change in 1.2 is the AI model upgrade. Switching from GPT-4o to GPT-5.2 improved answer quality across the board.
We measured the improvement as the correct-answer rate on a verification question set we prepared in advance. For MaxGauge ChatBot, the correct-answer rate rose from 86% in 1.0 to 100% after applying GPT-5.2. Off-target answers that used to appear now and then, and cases where it failed to find content that was in the documents, dropped sharply.
InterMax ChatBot was upgraded to GPT-5.2 as well, with similar improvements. These results mean more than just a better model. In 1.0, we sometimes had to double-check the ChatBot's answers. With higher accuracy, the answers are now reliable enough to use directly in our work.
3-2. Image analysis added
Image-based questions, which were hard to use in 1.0, now work naturally in 1.2. You can attach a screenshot directly in the chat and ask, "Analyze what's wrong on this screen." The answer now comes from looking at the actual on-site screen, which makes checking issues much more intuitive.
3-3. Faster responses
The perceived responsiveness that fell short in 1.0 also improved in 1.2. We kept streaming output, where the answer appears from the first character as soon as it is generated, and because better model quality means fewer follow-up questions, the total time to get the answer you want has gone down.
On site, fast answers tie directly to how much people trust the tool, so this speed improvement is a positive change for real-world use.
Wrapping up
QA AI ChatBot did not start with a big plan.
It started from small frustrations we felt while handling recurring issues, and from not being able to quickly pull up our accumulated experience when we needed it.
Version 1.0 was less a finished tool than a first attempt to fix that. Issue history lookup and Q&A on installation and configuration guides alone clearly improved our work. At the same time, running it exposed limits in answer accuracy, speed, and image support, and those became the direct motivation for 1.2.
In 1.2, the model upgrade to GPT-5.2 raised answer similarity from 86% to 100% (based on our internal similarity verification), and we added image attachment analysis and better perceived responsiveness.
The goal is not for AI tools to replace the work of QA and engineers. It is to cut down repeated checking and help people find the information they need quickly. That alone changes the workload.
We will keep fixing the limits we find as we run it, one at a time. We hope this post is a useful reference for anyone wrestling with similar problems.
Meet MaxGauge, verified through a real QA process.
Learn more 👉🏻
Related articles
