About two years ago, I reviewed a firewall change request that looked, on the surface, completely routine. A developer on one of our trading applications needed their service to reach an external API endpoint. They had filled out the form correctly. The business justification was clear. The endpoint they wanted to reach was a legitimate third-party data provider we already had a relationship with.
I almost approved it in under five minutes.
I didn't, because something felt slightly off. I can't fully explain what. After enough years doing this work you develop a kind of peripheral vision for requests that don't quite fit. I'm not claiming it was instinct or special talent - it was just pattern recognition built up over a lot of change requests.
What I found when I looked more closely is the subject of this post.
What a firewall rule actually is
Before getting into the story, I want to explain what a firewall rule does in plain terms, because the concept matters here.
A firewall sits between your internal network and the outside world. By default, in most corporate environments, it blocks everything. When someone needs to make an outbound connection - say, a server inside the firm needs to talk to an external API - they submit a change request asking for a rule to be added. The rule says: "allow traffic from server X, to address Y, on port Z."
Ports are a way of distinguishing different types of traffic. Port 443 is HTTPS (normal web traffic). Port 22 is SSH (remote server access). The port you allow matters a lot because it determines what kind of conversation can happen.
The request I was reviewing wanted to allow traffic from our trading application server to a specific IP address on port 443. Normal HTTPS. Fine.
Except the IP address was only partially right.
The problem hiding in plain sight
The developer had correctly identified the hostname of the third-party API. But instead of submitting the hostname, they had submitted an IP address - which is sometimes required because our firewall rules work at the IP level rather than the hostname level.
The IP address they submitted was associated with the right organisation in the WHOIS records. But when I did a reverse DNS lookup and a bit of additional checking, I found that the IP was assigned to a different service operated by that organisation. Not the trading data API. A generic hosting environment they also ran.
This matters because if we had approved that rule, we would have allowed outbound traffic to an address that could, in principle, receive connections from a wide range of services hosted there - not just the specific API we intended to reach. It wasn't a hostile IP. But it was a much broader permission than we thought we were granting.
The developer hadn't done anything wrong. They had genuinely looked up the IP and thought it was correct. The error was technical and subtle.
What happened next
I sent the request back with a note explaining what I had found and asking the developer to work with the third-party vendor to confirm the correct, specific IP range for the API endpoint they needed. This is a real range, documented in the vendor's technical documentation, that only covers their trading data service.
A week later the request came back with the correct IP range. I approved it. The connection worked.
The $2M figure in the headline is the firm's internal estimate of what a regulatory finding around unauthorized data egress could have cost, given the regulatory environment we operated in at the time. I want to be clear that I don't know if we would actually have had a problem. The IP wasn't malicious. But the possibility was real enough that the risk team got involved when I flagged it, and their assessment produced that number.
What this changed
After this, we updated our firewall change process in two ways.
The first was that we required IP addresses to be accompanied by documentation from the vendor - a specific page in their technical docs, not just a lookup. This slows things down slightly and causes some friction, but it means the person submitting the request has had to verify the address against an official source.
The second was that we started including a brief peer review for any rules involving external financial data providers. Just a second set of eyes. Not a full security review - just someone other than me reading it before it goes through.
Neither change was dramatic. Neither required new tooling or a big budget. They were both just process changes that made the human review slightly more rigorous.
Most security improvements in a financial services firm look like this. Not a new product or a new technology. Just a small tightening of something that was already there, applied consistently.
