Kubernetes is a system for running containers. That sentence is technically accurate and completely undersells how much it changes the way a team works.
I want to explain what Kubernetes actually is before I get into the experience, because when we started this project I had colleagues who had been in finance for 20 years and had never needed to know what a container was. That's not a criticism - it's just the reality. So let me try to explain it the way I wish someone had explained it to me.
What Kubernetes actually does
Imagine you have an application. Traditionally, you run it on a server. One application, one server (or a few servers). The server has specific amounts of RAM and CPU. The application uses what it needs.
Containers are a way of packaging an application with everything it needs to run - its code, its configuration, its dependencies - into a single bundle. The container can then run on any machine that has the container runtime installed. You stop saying "this application runs on Server A" and start saying "this container can run anywhere we point it."
Kubernetes is the system that decides where to run your containers, makes sure they are running, restarts them if they crash, scales them up when load increases, and scales them down when it doesn't. It is, in essence, a very sophisticated scheduler and supervisor.
The analogy I use: if your application is a guest at a hotel, Kubernetes is the hotel management system. It assigns rooms, moves guests around if a room needs maintenance, orders more rooms if the hotel gets busy, and keeps track of who is staying where.
Why we did it
We had a portfolio analytics service that calculated risk metrics for the trading desk. It ran on a single virtual machine. On most days it was fine. On volatile market days - the kind of days where everyone is recalculating everything at once - it fell over.
Kubernetes would let us run multiple copies of the service and route requests between them. When things got busy, we could spin up more copies automatically. When things calmed down, they would wind down. We would stop paying for compute we didn't need at 2am, and we would stop having angry traders at 10am when the service was struggling.
The technical case was clear.
What nobody tells you before you start
Here is what I wish someone had told me before we started.
The Kubernetes learning curve is steep and it is not the technology that bites you first - it is the concepts. Things like namespaces (isolated spaces within the cluster, like separate rooms in the hotel), pods (the thing that actually runs your container), deployments (the description of what you want running and how many copies), and services (the thing that routes traffic to your pods) all have specific meanings that are different from how those words are used in normal conversation.
This created real problems for us because we were working with developers who were smart and experienced and had zero Kubernetes background. When I said "can you check the pod logs" I might as well have been speaking a foreign language. This was my fault. I had become fluent in the vocabulary without noticing that the vocabulary was the barrier.
We fixed this by running a half-day internal workshop before the project properly started. Not a deep dive - just enough to get everyone speaking the same language. That half-day saved weeks.
The compliance conversation nobody wants to have
In financial services, before you move a workload anywhere new, you have a conversation with risk and compliance. This is not optional and it should not be treated as a formality.
The specific question that caused us the most difficulty was: "where does the data live?" Kubernetes can, in principle, run your containers anywhere in the cluster. If the cluster spans multiple datacentres, or if nodes are in different jurisdictions, data that was previously sitting in one known location can end up somewhere you didn't expect.
This is not a Kubernetes problem specifically. It is a distributed systems problem that Kubernetes makes very visible. We solved it with node labels - a Kubernetes feature that lets you mark specific servers with tags, and then instruct your workloads to only run on servers with certain tags. We tagged all nodes with their datacenter, and added a rule to our deployment that said "only run on nodes in datacenter X."
This satisfied compliance. But it took three weeks of conversations to get there, and none of that time showed up in our original project estimate.
How it turned out
The analytics service has been running on Kubernetes for about 18 months. On a normal day it runs 3 copies. On a busy day it has peaked at 11. The auto-scaling happens in the background and nobody on the trading desk notices it.
The original VM that it ran on is gone. The on-call burden for the analytics service is lower than it was because Kubernetes handles restarts and most common failure modes automatically.
What I didn't expect was how much the team learned along the way. Engineers who had never written a Dockerfile before this project now routinely containerize new services without thinking much about it. That knowledge transfer happened because the project forced it. That might have been the most valuable thing that came out of it.
