Translated from the Korean original. Some screenshots show the original Korean interface.
1. Why running an in-house LLM is harder than deploying one
IT operations teams across many industries are evaluating LLMs for air-gapped networks. Security policies rule out external chatbots, and more and more teams say they don't know where to start when they have to run a model themselves.
The good news is that the models themselves have become much easier to adopt. Small open-source models (such as Meta's Llama 3 8B and Mistral AI's Mistral 7B) can now run comfortably on a single GPU. The hard part comes after deployment: running the model reliably as an internal service.
Backing this up, Gartner predicted that 30% of enterprise generative AI projects would stall at the proof-of-concept (POC) stage and never become real internal services. In other words, many organizations can deploy a model but fail to settle it into operations.
This post sorts what IT operations teams should check before adopting an air-gapped LLM into 4 areas: infrastructure, data, security, and monitoring. Along the way, we look at problems that come up often in real deployments and the items worth checking.
2. 4 things to check before running an air-gapped LLM
In an air-gapped environment you can't resize resources on demand or hand things off to a vendor the way you can in the cloud, so the operations team owns a wide scope. In real deployments, operational issues show up most often in the following 4 areas. They are interconnected: a decision in one area affects the operating cost and stability of the others.
1. Infrastructure - GPU specs and data center environment
2. Data - RAG and fine-tuning
3. Security and operational ownership - permissions, guardrails, regulations
4. Monitoring - system metrics + LLM answer quality
2-1. Infrastructure requirements to settle before choosing GPU specs
The infrastructure an LLM needs varies widely with model size, response speed, and the number of concurrent users. Even with the same small model, an environment where 50 employees use it occasionally and a call center where 200 people call it at once need different GPU configurations from the start.
You also need to look at the data center itself. A typical server rack draws around 4-10kW, but racks running high-end GPUs at H100 class and beyond easily exceed 20kW each. Teams that don't check power, cooling, and rack space together often get stuck at the very first step. In environments handling high-end GPUs, liquid cooling adoption is growing fast, so it's a good idea to review data center facilities before you deploy.
Because you can't scale out instantly in an air-gapped network the way you can in the cloud, it becomes important to reserve headroom for 6 months to 1 year of expected usage at initial deployment. Settle the following before you deploy:
- Model size for internal use and whether to quantize
- Number of concurrent users and target response time
- Spare power and cooling capacity in the data center
- Spare slots for future expansion
In the end, you can only pick the right GPU specs once you've decided how the model will be used.
Before you request GPU quotes, put "who will use it internally, when, and how" on a single page. That one page determines how accurate the quote is and how satisfied you are after deployment.
2-2. Two ways to connect internal data to the model: RAG and fine-tuning
The answer quality of an air-gapped LLM depends on how you connect internal data. Most teams choose between two approaches: RAG (Retrieval-Augmented Generation) and fine-tuning (additional training).
Item | RAG | Fine-tuning |
Data update cycle | Suits frequently updated environments | Suits domains that change little |
Cost | Relatively low | Relatively high |
Answer consistency | Varies with search results | Consistent tone and style |
Preparation burden | Building and managing a vector DB | Building a training dataset |
Retraining needed | No (update documents only) | Yes |
Best fit | Manuals, policies, customer responses | When answer consistency and tone matter |
RAG stores internal documents in a vector DB and, when a question comes in, finds the relevant documents and reflects them in the answer. Since you only update documents without retraining the model, it suits manuals, policies, and customer responses where information changes often. However, if the document structure changes significantly or you replace the embedding model, you need to re-vectorize. Fine-tuning trains answer style or domain knowledge directly into the model using internal data. It costs more time and money, but teams choose it when answer consistency and tone matter.
Whichever approach you take, data cleansing comes first. Internal documents are plentiful, but in most cases they can't be used as is: duplicates and outdated versions, material mixed with sensitive information, and documents in different formats by department are all jumbled together. Classification, organization, and masking have to happen first.
In real deployment projects, data cleansing alone often takes 30-50% of the overall schedule, so it's safer to set aside generous time for it when planning the schedule.
In practice, more teams are also using a hybrid setup that combines RAG and fine-tuning: frequently updated manuals go through RAG, while answer tone and domain vocabulary are handled by fine-tuning.
2-3. Additional security and operational ownership for air-gapped LLMs
Bringing a model into an air-gapped network does not solve every security problem by itself. If the model was trained on internal data or references it through RAG, you need to manage, per user permission, who can get which answers to which questions.
Typical items are whether RBAC (role-based access control) and MFA (multi-factor authentication) are applied. Permission filtering is needed at the model response stage so that HR information only goes to the HR team and sales data only to the sales team. If your environment is ISMS-P certified, model call logs should also be included in audits.
A risk unique to LLMs that comes up often is prompt injection. Prompt injection has held the No. 1 spot in the OWASP Top 10 for LLM Applications since 2023. It covers cases where a user deliberately bypasses the system prompt, or malicious instructions hidden in documents that RAG references cause the model to produce unintended answers.
Operational ownership of an LLM doesn't end with the IT operations team. The department accountable for model output (service and planning teams), the department that cleanses data (data and business teams), and the department that reviews security (security and IT operations teams) may all be different. Only when it's clear who makes which decisions is accountability clear when an answer goes wrong. In Korea, the AI Basic Act (Framework Act on the Development of Artificial Intelligence and Establishment of Trust), in force since January 22, 2026, and the Personal Information Protection Commission's guide to developing and using generative AI are becoming the reference points for LLM operations, and many organizations schedule regular reviews by responsible department accordingly.
Security isn't just model guardrails. Permission filters, input/output validation, and audit logs all need attention at the same time. If you agree before deployment on which department owns each of these three, your response will be much faster when an incident occurs.
2-4. LLM monitoring needs more than system metrics
Traditional system monitoring centered on infrastructure metrics like CPU, memory, and response time. LLM operations add a new axis: the accuracy and hallucination rate of model answers. Teams frequently report cases where every system metric looks normal but answer quality keeps degrading.
The causes vary. Internal documents that RAG references aren't updated, so the model answers with old information. User question patterns drift from those at launch, so the model sees more types it has never seen. Or attempts to bypass model guardrails pile up. You can't catch these changes by watching CPU and response time alone. That's why an LLM observability perspective, which tracks the answers themselves beyond infrastructure metrics, has become necessary.
Evaluation frameworks like RAGAS are commonly used to assess answer quality automatically. They quantify how relevant an answer is to the question, how faithfully it reflects the referenced documents, and how much hallucination is mixed in, which helps catch changes in model performance periodically.
However, automated evaluation suits periodic checks. To catch the anomalies an operations team runs into every day in real time, you need separate operational metrics. Commonly used everyday metrics include:
- Call volume and response time
- Detection of sensitive information patterns in answers
- Ratio of user feedback (likes and dislikes)
- Number of guardrail bypass attempts
If these metrics aren't gathered on one screen, it's hard to decide where to look first when an alert fires.
Automated evaluation (RAGAS, etc.) and operational dashboards play different roles. Automated evaluation is for periodic model checks; operational dashboards are for daily anomaly detection. Design them separately, and you won't be confused about where to look first when an alert comes in.
3. Handling all 4 on one platform: integrated LLMOps
To sum up: once you start handling the 4 areas of infrastructure, data, security, and monitoring with separate tools, the number of tools grows quickly and departmental ownership boundaries tend to blur. In real deployments, teams often report a heavier operational staffing burden after bringing in model serving tools, a vector DB, a permission system, and monitoring dashboards one by one.
This is why more organizations are trying to run all 4 on a single platform. The integrated LLMOps platform eXemble is an air-gapped LLMOps solution designed in this direction, covering everything from LLM model deployment to security and auditing on a single platform in on-premises and air-gapped environments. Below, we look item by item at how the 4 areas covered above are actually tied together and operated on one platform.
3-1. From data sources to AI services in one flow
Internal data such as manuals, reports, and operational DBs often varies in form and location. Connecting them manually one at a time makes pipeline management alone a heavy operational burden. eXemble provides a data collection pipeline that automatically connects diverse sources and automatically applies OCR and vectorization to formats such as PDF, HWP, and images. When data changes, automatic re-vectorization keeps search quality up, and a distributed query engine lets you query and analyze large volumes of data.
3-2. Designing AI agent workflows without code
Building LLM-based workflows usually requires development resources, which makes it hard for an operations team to experiment quickly on its own. With eXemble, you can build workflows yourself by connecting AI agents and external tools on a drag-and-drop canvas without writing code. It reduces hallucination by exposing the sources and reasoning behind each AI response step by step, and it continuously improves workflows by using user chat feedback as an evaluation dataset.
3-3. An integrated operations dashboard on one screen
When model, data, and security metrics are scattered across different screens, finding the cause of an anomaly takes time. eXemble monitors the usage, cost, and response speed of AI services on one screen, and tracks GPU resource usage patterns so you can optimize operating costs. Every execution record is automatically stored as an asset that serves as the basis for improving service quality and analyzing user behavior.
3-4. Permission management that fits naturally with air-gapped security
Even in an air-gapped environment, if you have to build a new permission system for the LLM service separate from internal systems, coordination with the security team becomes a burden from the start. eXemble provides RBAC-based access permissions aligned with your organizational structure and data access scope limits by department and job level out of the box, so it fits naturally with existing air-gapped security policies. It also supports automatic detection and real-time masking of sensitive information, collection of policy violation logs, and real-time anomaly monitoring.
4. Before you start evaluating
Infrastructure, data, security, and monitoring are the foundation of running an in-house LLM. Readiness differs by organization, but most of the difficulties operations teams face after deployment stem from these 4. If you're evaluating an air-gapped LLM, we recommend settling these 4 before comparing GPU prices. It's the work that cuts trial and error after deployment the most.
If any of the situations below apply to you, you can see for yourself how the integrated LLMOps platform eXemble handles these 4 areas.
- A PoC where the model is deployed but stuck before operations
- Internal rollout delayed because security and permissions aren't clear
- An operating environment where system metrics are normal but answer quality keeps degrading
- Model, data, permissions, and monitoring spread across different tools, blurring operational ownership
Sources
"Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept by End of 2025" - Gartner (2024.07.29) "'A new wave sweeping into data centers': Keeping up with the liquid cooling trend" - CIO Korea (2024.04.04) Framework Act on the Development of Artificial Intelligence and Establishment of Trust (AI Basic Act) - Korea Law Information Center Guide to Personal Information Processing for Developing and Using Generative AI - Personal Information Protection Commission (2025.08) OWASP Top 10 for LLM Applications 2025, Korean edition - OWASP (CC BY-SA 4.0)
Related articles
