Hi, I'm Jinuk Jeong from the LLM Platform Team.
I design NL2SQL (Natural Language to SQL) and agentic workflows, where an LLM understands structured data and turns users' business requirements into SQL queries. Generative AI has lowered the barrier to data analysis, but giving an LLM real context about an actual database is not easy. Designing an agent that understands an enterprise database with tens of thousands of intertwined columns is a different problem altogether.
In this post, I want to share how I worked through complex join logic, going beyond simple prompt injection, and how I solved it at the architecture level.
1. QueryError: "Out of memory"
To match a real enterprise environment during NL2SQL development, we connected the database of our own platform, XAIOps, and tested against it. XAIOps is a monitoring platform, and its database stores system data collected at 5-second and 30-second intervals.
During testing, we asked, "Find the GC problems on the WAS servers last month," and the LLM produced a very plausible-looking SQL query. The result: Query Failed. And not because of a simple syntax error. It was a DB node memory limit exceeded error.
We had queried just 1 month of data, yet the query burned through its allocated memory (about 600MB) almost instantly and was killed.
1-1. Root Cause: Missing JOIN conditions
Here is the query the LLM generated:
At first glance it looks correctly joined on
target_id (the server ID), but there is a critical mistake: the join condition on the time column is missing.For a typical RDBMS master-detail pair (for example, users and orders), joining on the ID alone might be enough. But the data we deal with is time-series data.
In time-series data, joining only on the ID without a time (
time) condition matches far too many rows. This is a Cartesian explosion.- Scenario: Server A writes data every 1 minute.
- Data volume: 1 month = 30 days * 24 hours * 60 minutes = about 43,200 rows
- Bad JOIN: ON s.target_id = p.target_id
- Server A's row for November 1, 00:00 is matched with every row for the whole of November in the other table (43,200 rows).
- Server A's row for November 1, 00:01 is likewise matched with 43,200 rows.
- Result: hundred million rows
We only wanted 1 month of data for a single server, but internally the query built a temporary table of 1.8 billion rows (18 × 100 million) and ran out of memory.
1-2. Why domain knowledge matters
When building NL2SQL, it is tempting to think that providing the schema as context is enough. But an LLM knows the shape of the data, not its characteristics. So we added a system prompt like the one below to explicitly inject the characteristics of time-series tables.
2. Dynamic few-shot sampling: "Learning from past successes"
A common way to improve NL2SQL accuracy is few-shot examples: you provide real user questions and their matching SQL queries as examples. But reusing a fixed set of examples across different databases is not a good idea.
2-1. The trap of fixed examples
Fixed examples work well for questions in a specific domain, but when the intent of a question drifts even slightly, the LLM tries too hard to copy the structure of the example query and ends up with schema hallucinations. Unneeded examples also waste the context window and become noise that buries the schema descriptions that actually matter.
Schema hallucination is when a model, while converting a natural language question into SQL, references database tables or columns that do not exist, or maps phrases in the question to the wrong parts of the actual database schema.
2-2. Solution: A data flywheel driven by user feedback
A data flywheel is a virtuous cycle in which data and a learning model interact to keep improving performance.
We designed a RAG-based "personalized dynamic few-shot" structure that uses successful cases users rated positively as real-time learning guides.
- Positive Feedback Indexing: When a user clicks "Like," the
{ natural language question : generated SQL }pair is stored in a vector DB. We do not store only the query pair. We also store the information of theuserwho generated that query as metadata.
- Semantic Retrieval: When a new question comes in, we search in real time for up to 8 past successful cases whose vectors are most similar to the question. Metadata filtering retrieves query patterns that this user has used before.
- Dynamic Context Injection: The retrieved "verified examples" are injected into the prompt to give the LLM context.
The key point is that the LLM picks up each user's specific business context in real time (in-context learning) without any fine-tuning. For example, "revenue" can mean different things in different departments. Personalized few-shot makes the LLM look first at the "revenue" formulas that worked for this user in the past, so it can infer the business context that matches the user's intent.
- Overcoming the limits of zero-shot: It is impossible to write every domain-specific proper noun or complex aggregation rule into the system prompt. With this structure, though, the agent learns each user's way of working as their feedback accumulates, and query accuracy improves over time. That is the data flywheel effect.
After adopting this approach, we raised the success rate on questions involving complex business context by more than 20% over the previous approach. It was an empirical confirmation of the research finding that in-context learning works best when the examples provided are similar to the question.
3. How our schema context design evolved in 3 stages
How to describe the DB schema to the LLM was the most important research question for NL2SQL performance. Our platform evolved through three stages of trial and error.
3-1. Automatic injection of DDL and sample values
Naive Approach
At first, to keep things automated, we pulled the full DDL and sample values for each column and put them into the context.
- Pros: Little development effort and easy to automate.
- Cons: In real environments with hundreds of tables, we ran out of tokens, and the LLM was often confused by noisy information and picked the wrong columns.
3-2. Enriching categorical data
Data-Driven Approach
When a column was categorical (for example, evaluation metrics or instance types), we extracted its unique values in advance and included them in the prompt.
- Pros: For questions like "GC-related problems," this greatly reduced hallucinations where the LLM made up values that do not exist, such as
jvm_gc_issue. Instead, by referring to the list of unique values provided in advance, it could generate queries with the correct values,jvm_gc_timeandjvm_gc_count.
- Cons: With large data volumes, extracting these values in real time added too much latency, which made it hard to use in a real service.
3-3. Semantic Metadata Management layer
Human-in-the-loop
In the end, for the best performance, we built a layer at the platform level that manages "semantic descriptions" of tables and columns.
- Trino abstraction: Whatever database a customer uses, we put Trino in front of it so the LLM only has to focus on standard SQL.
- Explicit relationships: Users can add descriptions on the platform such as "this table and that table must be joined on these conditions." The Cartesian explosion from section 1 was also solved at the root by writing "a time condition is required when joining" into this metadata layer.
4. Prompts that carry domain knowledge produce the right answers
An LLM may understand SQL syntax, but it does not know the context of our data
Closing the gap between database systems, which run on deterministic logic, and LLMs, which give probabilistic answers, ultimately comes down to careful engineering. Designing hard rules to prevent Cartesian explosions, building a dynamic few-shot pipeline that turns user feedback into intelligence, and establishing a semantic metadata layer that finds meaning across hundreds of tables went beyond writing code. It was the work of turning data knowledge into architecture.
Next step: Building an internal dataset for real-world use
Our engine currently performs well on public datasets such as the BIRD benchmark. But as the cases in this post show, real enterprise databases are far less friendly and far more complex than benchmark datasets.
Going forward, rather than settling for general-purpose datasets, we plan to build an internal benchmark that reflects the situations our platform actually faces. With it, we aim to go beyond simple query generation so the agent fully understands the ambiguous business intent in user questions and retrieves the right data.
Sources
Related articles
