The original problem was simple to describe: every time an application team needed a new namespace on the OpenShift cluster, they opened a ticket. The ticket went into the queue. The queue had other things in it. The namespace showed up somewhere between two hours and four days later.
For context: a namespace in Kubernetes or OpenShift is an isolated space within the cluster where a team's applications live. Think of the cluster as a large apartment building. A namespace is one apartment. The platform team was the building manager. Every time someone needed an apartment, they had to ask us.
The application teams were frustrated. The platform team was spending a significant chunk of time on work that felt routine and mechanical. The obvious answer was self-service: build a portal where teams could provision their own namespaces without involving us.
What followed was eight months of work and three distinct versions before we had something that didn't create more problems than it solved.
Version one: fast, simple, wrong
The first version took about six weeks to build. It was a web form. You typed in a namespace name, your team name, and a cost centre code. You clicked submit. The portal called the OpenShift API. The namespace appeared.
It worked perfectly from day one. Within six weeks we had 93 namespaces. Within eight weeks we had a resource exhaustion event on two nodes.
Here is what happened. When we created namespaces manually, we also set resource quotas - limits on how much CPU and memory the workloads in that namespace were allowed to use in total. This prevents any one team's applications from consuming all the available resources and affecting everyone else.
When we built the self-service portal, we forgot to include quota creation. Every namespace was created with no limits at all.
One team deployed a development workload that had a bug causing it to spin in a tight loop consuming CPU. In a properly quota'd cluster, this would have hit the namespace limit and been throttled. In our unquota'd cluster, it kept consuming CPU until the nodes it ran on were effectively unusable.
We took the portal down. We put the ticket queue back. We started over.
The lesson: self-service without guardrails is just faster mistakes
The insight that drove version three (version two overcorrected by adding a manual approval step which defeated the entire purpose) was this:
Self-service needs to be opinionated at the moment of creation and permissive after that.
What that means in practice: the developer gets to choose a few things - the namespace name, the team it belongs to, which environment tier it is (development, staging, production). The platform determines everything else automatically based on those inputs.
When a developer clicks "provision," the system creates not just the namespace but a complete, validated configuration:
- A ResourceQuota calculated from the environment tier. Development namespaces get modest limits. Production namespaces get more, but still bounded.
- A LimitRange that sets default resource requests and limits for any container that doesn't specify its own. This protects against the "forgot to set limits" problem.
- NetworkPolicies that isolate the namespace by default. Applications in namespace A cannot talk to applications in namespace B unless a specific policy permits it.
- RBAC roles that give the requesting team admin access to their namespace and read-only access to platform monitoring tools.
- Labels that feed into our internal chargeback reporting, so finance can attribute cloud costs to the right team.
The developer sees a four-field form and a "Provision" button. In 30 seconds they have a namespace ready to use. What they don't see is a complete, tested configuration template that has been reviewed by the platform team and that we control entirely.
# What gets created automatically - the developer never sees this
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-quota
namespace: requested-namespace
spec:
hard:
requests.cpu: "4"
requests.memory: "8Gi"
limits.cpu: "8"
limits.memory: "16Gi"
count/pods: "20"The application deployment layer
Once teams had namespaces, they wanted to deploy applications. This is where it got genuinely difficult.
Giving developers the ability to deploy their own applications means giving them access to the cluster API with enough permission to create Deployments, Services, and Routes. The moment you do that, you discover some interesting things about how developers write Kubernetes manifests when left to their own devices.
We saw containers configured with 64GB memory limits for a Node.js application that served roughly twelve users. We saw applications with no liveness probes (a liveness probe is a health check - if the container doesn't respond to it, Kubernetes restarts it; without one, a frozen application stays frozen forever). We saw applications trying to run as root because the base image assumed it could.
None of these were malicious. They were the natural result of people who are good at their jobs applying skills from one context to a new environment they hadn't fully learned yet.
Our solution was an application catalogue. Instead of allowing raw manifest submission, we built a set of templates covering the cases that represent about 90% of what teams actually need: a web service, a batch job, a worker process, a database-backed API. Each template is parameterised - you provide the image URL, the port, the replica count, the environment variables. The system generates the manifests, runs them through a validation step, and applies them.
The validation step is the part I'm most proud of:
def validate_deployment(manifest: dict) -> list[str]:
errors = []
containers = (manifest
.get('spec', {})
.get('template', {})
.get('spec', {})
.get('containers', []))for c in containers:
name = c.get('name', 'unknown')
if not c.get('resources', {}).get('requests'):
errors.append(f"{name}: resource requests required")
if not c.get('livenessProbe'):
errors.append(f"{name}: liveness probe required")
sec = c.get('securityContext', {})
if sec.get('runAsUser') == 0:
errors.append(f"{name}: running as root not permitted")
return errors
```
If the validation fails, the developer gets a clear error message explaining what's missing. They fix it and try again. No human review needed for the common cases.
What I didn't expect about running a self-service platform
Three things surprised me that I want to share because I haven't seen them written about much.
Self-service generates more support questions, not fewer. The ticket volume for routine provisioning dropped by about 70%. The support burden didn't drop by anywhere close to 70%, because the questions became more complex. Instead of "can I have a namespace," we got "why is my application failing to start," "why can't I connect to the database," "what does this SCC error mean." Those questions require real understanding to answer. The routine work went away. The hard work stayed.
Developers will find ways around guardrails they find inconvenient. Within a month of launching the catalogue, three separate teams had discovered that we had left a raw manifest deployment endpoint open for edge cases. All three were using it to bypass the template validation. We closed it. There was friction. We explained why. They understood. But we had to catch it first.
Documentation is the actual product, not the portal. The teams that used the platform confidently and well were the ones we had personally walked through it. The teams that struggled were the ones we had pointed at the UI and said "here you go." I spent more time on the UI and not enough time on the documentation. The portal is a delivery mechanism. The documentation is what determines whether the delivery mechanism gets used well.
We've been running this version for 14 months. Around 200 active namespaces. The provisioning queue is empty because everything self-serves. The resource exhaustion events from version one have not recurred.
The thing I'm most proud of is that the quota system has never been manually adjusted because of a crisis. The guardrails work because they were designed carefully and because developers generally don't think about them. That invisibility is the goal. When infrastructure is working well, nobody notices it's there.
