Technical debt is a well-understood problem in software engineering. It refers to the future work created by taking shortcuts today. You build something quickly and imperfectly, and you accumulate "debt" that you have to pay back later in the form of refactoring, fixes, and rewrites.
Every engineer knows this concept. There's a whole vocabulary around it.
What I want to talk about doesn't have a good name. I call it operational debt, though I'm not sure that's quite right either. It's the knowledge that lives in people's heads and nowhere else. The processes that exist only as muscle memory. The system behaviours that the current team knows but couldn't explain if you asked them to write it down.
In asset management, this kind of debt is everywhere. And unlike technical debt, it doesn't show up in any audit, review, or risk register until the person carrying the knowledge leaves.
What it looks like in practice
Here is a real example, anonymised.
On the third Friday of every month, a derivatives pricing job runs at 11pm. It takes about 40 minutes normally. If it runs longer than 90 minutes, the morning settlement will fail, because another job depends on it completing before a certain window. If the morning settlement fails, the operations team has a bad day and so does everyone else.
I know this because a colleague who has since left the firm told me, over lunch, in 2023. One conversation. Not in a runbook. Not in a monitoring alert. Not in the job's documentation. In my head.
If I left tomorrow, nobody would know. The job would still run on the third Friday. And one month, it would run long, and the settlement would fail, and people would spend a frantic morning figuring out why.
That's operational debt.
Why financial services is particularly exposed to this
Three reasons, in my experience.
The first is stability bias. Financial firms keep systems running for a long time because changing them is risky and the cost of an outage is high. A trading platform built in 2015 and running stably since 2018 is a platform nobody wants to touch. The people who built it are still around, carrying the knowledge of its quirks, and there has never been enough pressure to extract that knowledge because nothing has broken.
The second is production pressure. When something breaks in a trading or settlement environment, you fix it fast. You do not, in that moment, open a documentation page and write a careful post-mortem. You tell yourself you'll do it later. You usually don't.
The third is the nature of the knowledge itself. A lot of operational knowledge is procedural and contextual - it's not just "do this," it's "do this because of this history, except on the third Friday, and except when this other system is in this state." That kind of knowledge is genuinely hard to document because it's intertwined with context that the person carrying it has accumulated over years.
The "weird Fridays" document
About two years ago I started keeping what I call a weird Fridays document. Every time something surprised me in production - even a small surprise, even something that resolved itself - I write it down. Five minutes. What happened, what I expected, what was actually happening, what I did.
It has become, over two years, the most useful single document I maintain. I've shared it with every engineer who has joined my team. When someone is on call for the first time, I send them the document before their first weekend.
It does not pay down operational debt quickly. But it stops it from compounding, which is the more dangerous problem.
The second thing that helps
When I fix something non-trivial in production, I spend 15 minutes writing the alert that would have caught it. Not a full post-mortem. Just: what metric would have indicated this was about to happen, and what threshold would have triggered a useful alert?
Sometimes I can't write that alert. The failure mode doesn't have a clean metric. But often I can, and the monitoring system is measurably better than it was two years ago specifically because of failures that would previously have left no trace.
The goal isn't to eliminate surprises entirely. It's to ensure that each surprise leaves the system slightly more observable than it was before.
The best time to document a runbook is five minutes after an incident, while you're still slightly annoyed and all the details are fresh. Not next week. Not in the quarterly infrastructure review. Now.
That's the only advice I have that I actually follow consistently. Write it while it's fresh. Everything else I've tried requires a discipline that I consistently fail to maintain. Writing it fresh doesn't require discipline - it's basically automatic if you build the habit, because you're annoyed and you want to make sure it doesn't happen again.
