Hi, I'm Jinman Park from AI Solutions Team 1.
We design, build, and operate the backend architecture of Woodpecker, an MLOps platform that covers the entire AI/ML workflow, from data analysis to model training and serving. In this post, I want to share the questions we faced while splitting one large platform into 15 independent services, and how we tied the Spring Cloud ecosystem and Python ML services together.
Instead of just listing the result that "we adopted MSA," we will focus on why we designed it that way and what we learned while running it, down to the code level. If you are considering MSA (Microservices Architecture) or comparing the architectures of several solutions, you should find points here that resonate.
1. "Why did we start with MSA instead of a monolith?"
Woodpecker was designed as MSA from day one, because we had run into many problems developing and operating other products on a monolithic architecture. To avoid repeating a situation where very different functions, such as data analysis, model management, Kubernetes resource control, and file processing, get tangled in a single codebase, we chose an MSA structure that separated each function into an independent service from the start.
With a monolith, we would have faced three main problems.
- Deployment risk: Changing just the file upload logic would mean redeploying the entire platform, and there was always a risk that a small change would turn into a platform-wide incident.
- Language constraints: Data analysis and model training naturally belong in Python, but a monolith is built and deployed as a single Java unit. There is no way to fit Python inside it, so we would have had to split the ML code out into a separate Python service anyway.
- Inefficient scaling: Even if we only wanted to scale the analysis function when AI analysis requests spiked, a monolith would force us to scale out the whole thing.
Monolith vs. MSA
Aspect | Monolith (Before) | MSA(After) |
Deployment unit | Whole application at once | Independent deployment per service |
Tech stack | Single language enforced (Java) | Free choice per service (Java + Python) |
Fault isolation | One failure → everything down | Isolated per service |
Scalability | Scale out everything | Scale only the services you need |
Operational complexity | Bloated codebase | More network and communication complexity |
But MSA is not a silver bullet. The moment you split services, new complexity follows: network failures, distributed tracing, service discovery. Our strategy was to absorb that complexity with the Spring Cloud ecosystem. Also, "fault isolation" in the table above means partial isolation at the application layer. Today we share a single DB and have no circuit breaker, so we cannot stop data-layer failures or cascading failures. We cover this as future work in Chapter 9.
2. A Spring Cloud-based Polyglot structure
Our core strategy is a selective polyglot architecture. Rather than adding languages indiscriminately, we standardized on Java for areas where reliability matters and split out only the data science area into Python. Inter-service communication and operations are standardized on Spring Cloud.
A polyglot architecture is a design that, instead of being limited to one language, combines the language and framework best suited to the purpose of each microservice when building a single system.
- Java (Spring Boot): Handles the areas where platform reliability matters, such as authentication, authorization, Kubernetes control, and file/Git integration.
- Python (FastAPI): Handles areas that need the agility of the data science ecosystem, such as the no-code ML pipeline.
- Vue (Nuxt 3): The frontend is built with Vue 3 on Nuxt 3, which supports SSR.
2-1. Overall service layout
Service | Stack | Role |
web-front-service | Nuxt 3 / Vue 3 | Frontend SPA (PrimeVue + Tailwind) |
gateway-service | Spring Cloud Gateway | API gateway, JWT authentication |
discovery-service | Spring Cloud Eureka | Service discovery |
auth-service | Spring Boot | Authentication and token issuance |
batch-service | Spring Batch | Batch job processing |
kubernetes-service | Spring Boot | K8s resource management |
bff-service | Spring Boot | BFF hub (28 domains) |
file-service | Spring Boot | File upload and download |
license-service | Spring Boot | License management |
query-service | Spring Boot | Query editor |
git-service | Spring Boot | Git repository integration |
mlflow-service | Spring Boot | MLflow integration proxy |
work-service | Spring Boot | Workflow and job management |
custom-service | Spring Boot | Customer-specific services |
analysis-service | Python / FastAPI | No-code ML pipeline |
2-2. System architecture
Every external request passes through a single entry point, the gateway, and then follows a layered structure through the authentication, BFF, business, and AI/ML layers.
3. API Gateway - centralizing a single entry point
3-1. Problem: scattered entry points and fragmented authentication
Early in our MSA work, we considered having the client call each service directly, but the limits soon became clear. The client would need to know the addresses of all 15 services, authentication logic would be duplicated in every service, and every time a service address changed, we would have to modify the client too.
To solve this, we pulled the cross-cutting concerns of authentication and routing out of the applications and moved them to the gateway layer.
3-2. Solution: the three roles of gateway-service
① JWT authentication filter
It clearly separates protected paths from public paths.
The key is that
auth-service and the gateway share the same secret key. Thanks to this, the gateway verifies tokens locally and immediately instead of calling auth-service every time. With no extra network hop for authentication, the gateway does not become a bottleneck. As shown at the bottom of the diagram, invalid tokens are blocked with a 401 at the gateway before they reach the downstream (backend) services.② Path rewriting
Requests that pass verification are forwarded with their paths rewritten. The client requests
/gateway/bff-service/api/<resource>, but the actual service receives /api/<resource>. Services don't need to know which prefix they are exposed under, which keeps them independent.③ Centralized CORS handling
The browser-based web-front-service and the gateway have different origins, so CORS handling is needed. If each service configured CORS on its own, policies would scatter and consistency could break, so the gateway manages allowed origins, methods, and headers (
Authorization, etc.) in one place. As a result, downstream services don't have to worry about CORS. (The gateway also performs dynamic routing via lb://, but since that works hand in hand with Eureka, I'll cover it in the next chapter.)4. Service Discovery - managing dynamic locations
4-1. Problem: IPs are not fixed in Kubernetes
In Kubernetes, a Pod's IP changes every time it restarts. If you hardcode fixed IPs in config files, you have to redeploy every time. To solve this, we introduced an Eureka-based service registry.
4-2. Solution: the registration and lookup flow
4-3. Detail: why we turned off self-preservation
Eureka's self-preservation, which defaults to
true, is a protective feature that keeps services from being removed hastily when the heartbeat failure rate rises.Woodpecker runs on Kubernetes, so when a Pod dies, it must be removed from the registry immediately. Here is how it works. A missing heartbeat alone doesn't tell the Eureka server whether an instance is truly dead or alive but cut off from the network. So when the total incoming heartbeats drop sharply below a set share of what is expected (85% by default), Eureka assumes a large-scale network problem rather than individual instance deaths, and preserves the registry without removing even expired instances. It is a conservative "if in doubt, don't delete" strategy that keeps live instances from being wrongly removed and escalating into a large-scale outage.
On K8s, however, this protection works against you. In K8s, Pods disappearing and coming back through deployments (rolling updates), autoscaling, and node eviction is normal behavior, not an incident, and a dead Pod does not come back with the same IP. If self-preservation is on and several Pods disappear within a short time, incoming heartbeats fall below the expected level, Eureka enters self-preservation mode, and the registration entries of dead Pods that no longer exist stay in the registry. The gateway then sends traffic to those dead IPs, and requests fail or hang until they time out. In other words, a feature meant to prevent "wrongly deleting something alive" ends up, on K8s, "holding on to ghost instances that are already dead and sending traffic to them."
On top of that, kubelet probes already decide directly whether a Pod is alive and remove it from the Service endpoints, so there is little reason for Eureka to make a conservative call based on the indirect signal of a "missing heartbeat." So we turned off self-preservation and remove expired instances quickly instead of holding on to them, which shortens the time traffic leaks to dead services. (There is still a risk that a real network partition could remove live instances too. But on K8s, kubelet probes act as the 1st line of judgment on whether a Pod is alive and remove it from the Service endpoints, so the actual impact is limited, and we currently mitigate it with short connection timeouts. A more fundamental fix with a circuit breaker remains future work. For details, see Chapter 9.)
5. BFF - Backend for Frontend
5-1. Problem: N API calls for a single screen
If the frontend has to fetch user, project, model, and alarm data from different services to draw a single dashboard screen, the client must make 4 calls itself and also carry the logic to combine the responses.
5-2. Solution: managing 28 domains with bff-service
bff-service is the de facto business hub of this project. It owns many domains and, depending on the case, combines data queries with calls to downstream services.6. Authentication flow: JWT Security Offloading
6-1. Overall flow
6-2. Key point: only auth and the gateway share the secret key
Because
auth-service and gateway-service share the same Base64 secret, the gateway verifies tokens without calling the backend. Invalid tokens are blocked at the gateway before they reach the backend, so downstream services can focus on business logic with no authentication logic of their own.The trade-off is that with local symmetric-key (HS256) verification, it is hard for the server to revoke an issued token immediately (blacklisting). We compensate by keeping the access token's expiry short, at 30 minutes.
7. Distributed tracing: following a request's journey with Zipkin
✔️ Problem: where does a request slow down?
When a single request passes through several services, such as gateway → bff → kubernetes → batch, it is hard to find the bottleneck.
Distributed tracing is a technique that groups the full path of one request across multiple services into a single trace. It visualizes the time spent in each segment (span) to diagnose bottlenecks.
✔️ Solution: B3 propagation + Zipkin collection
Each service extracts the
traceId from the B3 headers and propagates it as is on downstream calls. All spans gather on the Zipkin server and are visualized as a complete call tree. In the local environment, we disabled it with probability: 0.0 to remove development overhead. Full sampling in prod (1.0), however, adds storage and network overhead as traffic grows. We set that value because we judged it manageable at current traffic levels, and we plan to switch to adaptive sampling as we scale.8. analysis-service(Python)
This service proves that the polyglot strategy declared in Chapter 2 is more than words. The only service written in Python/FastAPI among 13 Java services, it handles a no-code ML pipeline that goes from data loading to model training without writing code. We can use the Python ecosystem that data analysis needs (pandas, scikit-learn, LightGBM, etc.) as is because splitting the services lets each one freely choose the language and tools best suited to its domain. With a Java monolith, there would have been no way to bring that ecosystem in. In short, analysis-service is living proof of the claim that "MSA lets you mix languages."
✔️ Hide the language, expose only the contract
The essence of MSA is that whatever language a service uses internally, it only needs to communicate through a defined interface. analysis-service shows this well. bff-service doesn't need to know whether this service is Python or Java; it only needs to know the defined HTTP endpoint (for example, an analysis request contract in the form of
POST /api/analyze). The language is a private decision inside the service, and only the contract is exposed to the outside, which is why services built in different languages can be tied into one system.✔️ The price of language freedom: the cost of distribution
But the moment you cross a boundary, the costs specific to distributed systems come along too. When bff-service calls ML training, the training takes a long time, so at our current scale we chose a direct synchronous call without a message queue, with a read timeout of up to 35 minutes (connect 30 seconds). This approach has the limitation of holding threads for a long time, so we are considering switching to message-queue-based asynchronous processing as traffic grows. In a monolith this would be a function call, and timeouts or thread occupancy would never come up; they are costs specific to distribution that appear once it becomes a remote call. In other words, the benefit of language freedom and the cost of a network boundary are two sides of the same coin.
9. Limits we hit in operation and what to improve
To be honest, the current structure has room for improvement. Here are the trade-offs and future work we found while analyzing the code.
✔️ A single shared DB
All services share one PostgreSQL database. The MSA ideal is an independent DB per service, but in practice the benefits of Database per Service (independent scaling, fault isolation, schema independence) are mostly felt in large environments with heavy traffic or many teams touching schemas at the same time. At our current scale, a single PostgreSQL handles the load well. Forcing a split would eliminate cross-service joins and only increase the complexity of distributed transactions (Saga, etc.), so a single DB is the reasonable choice. That said, we are aware of the weakness: if code and deployment are split but the DB is shared, service boundaries break down at the data layer. So rather than an immediate task, we keep separation as an option to revisit when traffic grows or teams split.
✔️ analysis-service is not registered with Eureka
The Python service is currently referenced by a fixed address outside service discovery, so we are reviewing and working on integrating it into the discovery system using the K8s sidecar pattern. This should make service location management more consistent.
✔️ No circuit breaker
To further strengthen fault isolation, we are introducing a Resilience4j-based circuit breaker. Until then, response delays in downstream services can propagate upstream (cascading failure), so the plan is to mitigate with short timeouts for now and address it fundamentally with the circuit breaker.
Wrapping up
MSA was not about chopping services into small pieces. It was a process of lifting cross-cutting concerns (authentication, discovery, tracing) into the infrastructure layer so each service can focus on its core job. Its key value lies in polyglot design: Java handles reliability, Python handles data science, and Vue handles the frontend, while Spring Cloud Gateway and Eureka tie them together. Next, we plan to work step by step on splitting into Database per Service, introducing a circuit breaker, and bringing the Python service into discovery. We believe that running a complex distributed system reliably raises product quality, and a high-quality product leads to customer satisfaction. That is why we treat reliable operation in a distributed environment as a competitive edge for our product, and why we keep working toward a better architecture.
How are you building your AI/ML operations environment?
See how Woodpecker supports everything from data analysis to model serving.
Related articles
