Hi, I'm Jeongsu Sim from the LLM Platform Team.
I design and build stable backend architecture and infrastructure for LLM-based services. In this post, I want to share how we prevented authentication fragmentation and kept the system stable in a polyglot environment where Java and Python live side by side, along with the problems we worked through on the way.
The service our team runs today is a polyglot environment, where we pick the best language for the needs of each domain.
What is a polyglot architecture?
As the literal meaning,
using multiple languages, suggests, it is a design approach where you build one large service without locking into a single language, and instead choose and combine the language and framework that best fit each microservice's purpose and business requirements.- Java (Spring Boot): Handles source data management, where data reliability matters, metadata governance, and the complex permission logic of RBAC (Role-Based Access Control) and ABAC (Attribute-Based Access Control).
- Python (FastAPI): Handles flexible business logic and the core feature, AI model serving integration, to keep our AI services as agile as possible.
With Java owning
data governance and security and Python owning intelligent service implementation, our biggest challenge was how to apply a consistent authentication/authorization policy across two different environments. Early on, each microservice handled authentication on its own. As the service grew, we ran into two problems: fragmented authentication logic and complex secret key management.In this post, I share how we turned Nginx (OpenResty) from a simple proxy into a Programmable Gateway to unify our authentication structure, and how we optimized SSE (Server-Sent Events) performance, which is central to LLM services.
1. Background: The limits of distributed authentication
In our initial architecture, client requests went straight to each application, and each server validated the JWT on its own. As the service grew, this caused the following problems.
- Fragmented authentication logic: We had to maintain Spring Security on the Java side and FastAPI auth logic on the Python side separately. Implementing the same validation logic twice, in two languages, also raised the risk of policy mismatches.
- Hard-to-manage security keys: Every backend server had to hold the secret key. This
Secret Sprawlmade security vulnerabilities more likely to be exposed. It was a structural weakness: compromising just one service could threaten the entire authentication system.
To fix this, we needed to pull authentication/authorization, a cross-cutting concern, out of the application layer and move it into the infrastructure layer: the Gateway.
2. Technical decision: Why OpenResty over OpenSource Gateway?
A team that knows the Spring ecosystem would usually consider
Spring Cloud Gateway(SCG), and a team that wants more powerful features would look at Kong or APISIX. We did not go by what was familiar or popular. Instead, we looked closely at our platform's specific infrastructure situation and performance requirements. In the end, we compared several proven open-source solutions and worked out the leanest structure that fit our platform.
2-1. Comparing API Gateway solutions
Criteria | Pure OpenResty (final choice) | Apache APISIX | Kong Gateway | Spring Cloud Gateway |
Underlying technology | Nginx (C, Lua) | Nginx (C, Lua) | Nginx (C, Lua) | Java (Netty/WebFlux) |
Configuration management | Static (Config File) | Dynamic (etcd-based) | Dynamic (Postgres-based) | Static/dynamic (Java Config) |
Additional infrastructure | None (Zero) | etcd cluster required | RDBMS required | None (Java runtime) |
Network structure | Single Hop (integrated) | Sidecar or Gateway | Gateway-centric (Heavy) | Double Hop (when run alongside Nginx) |
Resource efficiency | Excellent (tens of MB) | Low (etcd management burden) | Low (DB operations burden) | Moderate (JVM overhead) |
Decision | Final choice | Rejected: excessive operating cost | Rejected: infrastructure overload | Rejected: performance and extra hop |
2-2. Key reasons for rejection
- Kong & Apache APISIX
Both are excellent engines built on OpenResty, but running them means constantly managing a separate database or coordinator (etcd) cluster. Given our team's principle, "get maximum performance with minimum infrastructure complexity", we judged that overhead to outweigh the benefit.
- Spring Cloud Gateway
SCG fits the Java ecosystem very well, but in our environment, where Nginx already sits in front as a reverse proxy, a Double Hop (Nginx → SCG → WAS) was unavoidable. The serialization overhead and latency added with every extra network step were a poor fit for an AI streaming service.
2-3. Final choice: Custom Pure OpenResty
We chose OpenResty in its purest form, with unneeded features stripped out, and got the following benefits.
- Zero Infrastructure
We got gateway functionality right away by adding Lua scripts on top of our existing Nginx configuration, with no extra DB or servers.
- Tailored Optimization
We dropped the heavy features that general-purpose solutions ship with and focused only on what we actually needed,
lua_shared_dict-based caching and SSE buffering control, to get the most performance.- Single Hop Architecture
By folding the logic into our existing Nginx layer, we kept network hops to a minimum and held latency down.
Of course, putting scripts in the Nginx configuration can make maintenance harder. To prevent that, we limited the gateway to one clear cross-cutting concern, "authentication and authorization", and delegated complex business policies to the backend (Java/Python) to keep the architecture clean.
📌 Note: During design, we also reviewed Netflix Zuul and Python-based gateways, but dropped them from the final shortlist because they did not meet modern asynchronous processing requirements or our infrastructure performance bar.
3. Redesigning the architecture: Centralizing security and efficiency
We adopted OpenResty, which combines Nginx with a Lua scripting engine, to build a
Programmable Gateway. This let us keep Nginx's high-performance, event-loop-based processing while controlling logic flexibly with Lua scripts.3-1. Architecture diagram: Single Hop Infrastructure
The biggest change is that every external request now passes through the gateway, giving us a physical Single Hop structure.
The diagram above shows the Single Hop Infrastructure structure.
- Gateway(OpenResty)
As the front-line entry point, the gateway receives all external requests (:80). Lua scripts run here to verify the JWT signature, and only valid requests are routed to internal services.
- Java Backend
Handles data source management and the permission system, such as
/api/v1/metadata and /api/v1/auth. Based on the user information passed on by the gateway, it runs the final RBAC/ABAC logic implemented in Java.- Python Backend
Handles real-time AI inference requests such as
/ai/v1/generate. Because authentication is offloaded to the gateway, the Python server does not need heavy auth libraries and can spend its resources purely on model integration and business logic.- Static Files
Instead of running a separate web server, we serve the React build files from inside Nginx. This eliminates CORS issues at the source and makes it easy to apply authentication logic to static resources too.
3-2. How the authentication flow changed: Security Offloading
The core of the new architecture is Security Offloading. Only the gateway manages the secret key, the most critical piece of security, which draws a clear security boundary.
The diagram shows the authentication flow step by step.
- Token Validation
The gateway (Lua) intercepts the Bearer token sent by the client and verifies its signature. Invalid tokens are blocked immediately and never reach the backend.
- Claims Extraction
From the payload of the verified token, the gateway extracts the data the business logic needs, such as user ID and role.
- Header Injection & Proxy
It passes the extracted information to the backend in custom headers such as
X-User-ID and X-User-Role.- Trusted Zone Operation
The backend services (Java/Python) fully trust the headers sent by the gateway. With no auth logic of their own, they build the user context from the header values alone and run the business logic.
Building this Trusted Zone dramatically improved backend development productivity. We no longer have to set up auth libraries or sync security keys every time we build a new service.
3-3. Network isolation: Minimizing external exposure
Along with centralizing authentication logic, we strengthened security further through physical network isolation. We made Nginx the only point of contact and applied a DMZ/Private structure that fully hides the backend servers from the external network.
Public Zone (DMZ)
- Only the Nginx Gateway connects to the Public Internet.
- Narrowing the external attack surface down to Nginx alone lets us concentrate security monitoring and firewall policy in one place.
Private Zone
- The Java and Python backend servers, which handle data and business logic, sit in a Private Subnet that cannot be reached directly from outside.
- They accept traffic only from Nginx in the DMZ (Allow only from DMZ), which isolates them completely from malicious outside access and port scans.
This makes it physically impossible for a client to bypass the gateway and call backend APIs directly. Even if an application-level vulnerability turns up, nothing can be reached without going through the gateway, the "single entrance", so the infrastructure as a whole stays secure.
4. Performance: Caching with Shared Dictionary
In the cloud, you can add servers (scale out) when traffic grows. Our on-premises solution, however, has to run within the limited hardware resources of each customer.
The HS256 algorithm we use is lightweight, but in a chat service a single user sends hundreds of requests (SSE, polling, and so on) in a short time. Repeating the exact same hash computation on the same immutable token every time is an inefficient design from an engineering standpoint.
Following the principle that "AI gets faster only when the infrastructure doesn't waste resources", we introduced an Nginx shared-memory caching strategy that replaces computation with lookup. The goal was to improve the system's structural efficiency, not just a performance number.
4-1. Optimization design: Why lua_shared_dict?
For JWT caching, instead of adding an external in-memory DB such as Redis, we chose
lua_shared_dict, which lets Nginx worker processes share memory directly. This meets both of our goals: "minimize infrastructure complexity" and "maximize performance".- Zero network I/O
We removed even the network latency of a Redis call, reaching L1-cache-level speed.
- Data shared across processes
Nginx runs multiple workers, but Shared Memory lets every worker read the same verification result, which keeps memory use efficient.
4-2. Optimization flow and detailed logic
We split request handling into an Init phase and a Runtime phase to visualize how the design minimizes OS system calls.
- Phase 1. Init Phase (when a worker starts)
OS system calls such as
os.getenv are not free. When an Nginx worker starts, it reads the environment variable (Secret Key) once and stores it in a Lua local variable (upvalue). During request handling, the key comes straight from memory with no system call.- Phase 2. Runtime Phase (when a request is handled)
- Shared Memory lookup
- Lazy Verification
- Dynamic TTL
Look up memory using the incoming token as the key. → O(1)
Run the actual CPU-bound cryptographic operation (HMAC/RSA) only on a cache miss.
When caching, we compute the JWT
exp (expiration) field so the entry is evicted the moment the token expires, which keeps the cache secure.4-3. Lua implementation details
Our goal is not just saving CPU but guaranteeing deterministic response times. Cryptographic computation is an
O(N) operation whose time varies with input length, while a memory lookup is always a constant O(1) operation. With this logic, the gateway delivers consistent latency even during traffic spikes.Below is a simplified example of the core logic from our production code.
5. Troubleshooting: Fixing SSE streaming delays
The core of an LLM (large language model) chatbot is real-time output: showing the answer to the user as soon as it is generated.
In our first build, however, we saw response delays: answers arrived all at once, or stalled for 5~10 seconds before appearing.
5-1. Root cause: Nginx's default buffering mechanism
By default, Nginx collects backend responses up to a set size (
proxy_buffer_size) before sending them to the client, to use resources efficiently. Gzip compression also waits for data blocks to build up before it starts. That helps ordinary API requests, but it backfired in an SSE (Server-Sent Events) environment, where tokens must flow out one at a time without pause.5-2. Solution: Streaming-specific tuning
For the specific path that needs real-time streaming (
/ai/v1/generate), we applied the following dedicated settings.proxy_buffering off;
Responses from the backend (Python FastAPI) are flushed to the client immediately instead of being held in Nginx memory. This minimizes the time until the first token the model generates reaches the user (TTFT, Time To First Token).
gzip off;
We removed compression delay for streaming responses. (Alternatively, you can send
X-Accel-Buffering: no in the HTTP response header.)proxy_read_timeout 300s;
We raised the timeout thresholds so connections are not cut off when LLM inference takes longer.
- TCP optimization
With
tcp_nodelay on;, even small packets are sent immediately, which improves perceived speed.6. Conclusion and results
Going beyond simply adding a proxy, redesigning the architecture around a Programmable Gateway became the key to achieving security, performance, and productivity at the same time, goals that often conflict.
✅ Performance: Latency & UX Optimization
- Algorithmic Efficiency
By replacing the O(N) cryptographic work repeated on every request with an O(1) memory lookup, we kept response times consistent even during traffic spikes.
- UX Improvement
Removing SSE buffering cut the LLM’s TTFT (Time To First Token) to under 0.5 seconds, delivering a smooth real-time chat experience.
✅ Security & Governance: Achieving zero trust
- Minimized attack surface
We isolated every backend server in a private subnet and removed public IPs, closing off external attack paths.
- Policy Centralization
Complex permission policies managed in Java are now applied consistently at the gateway.
✅ Productivity: More focus on development
- Boilerplate removed
Python developers no longer need to implement
AuthMiddleware or manage the Secret Key in every project.- Focus on Core
The communication overhead around authentication disappeared, leaving the team free to focus on AI model serving and business logic.
Wrapping up
In a polyglot environment, Nginx is no longer just a web server. With the right programming (OpenResty), Nginx can be
the lightest and strongest glue holding a tangled distributed environment together. We chose a lean architecture that best fit our platform's requirements over the polish of off-the-shelf solutions, and the result is practical engineering that lowers operational complexity while maximizing performance.Related articles
