I want to say upfront that this isn't an anti-AI post. I use these tools every day - for drafting, for debugging, for chasing down an obscure error message at 2am. They've made me faster. What this post is about is the gap between "faster" and "safe to hand a decision to," because in financial services that gap is where the incidents live, and I've watched teams walk straight into it because the demo looked so convincing.
The demo is not the risk assessment
Every AI vendor pitch follows the same shape: a clean example, a confident answer, applause. What you don't see in the demo is the 1-in-50 case where the model is equally confident and completely wrong. That failure mode - fluent, well-formatted, wrong - is the single most dangerous property of these systems, because it doesn't look like a failure. A stack trace looks like a failure. A hallucinated compliance citation formatted in perfect house style does not.
I've seen this firsthand: an internal tool summarizing regulatory filings produced a summary that was clean, well-organized, and referenced a clause that did not exist in the source document. Nobody caught it in review because it read like every other correct summary that tool had produced. The fix wasn't a better prompt. It was accepting that anything downstream of that tool needed a human checking against the source, every time, not just when something looked off.
Where this actually bites in a financial environment
- Hallucinated specifics presented with total confidence - numbers, clause citations, API parameters, account details. The model doesn't have a "low confidence" tone; overconfidence is uniform across correct and incorrect output, which is exactly why it's dangerous in a context where a wrong number moves money.
- Data leaving the boundary you think it's inside - pasting a client statement or an incident writeup into a consumer AI tool to "just clean up the wording" can mean that data is now sitting on infrastructure you have no contract with, no audit rights over, and no idea how long it's retained.
- Prompt injection through untrusted content - if an AI agent reads incoming emails, tickets, or documents and takes action based on what it finds, anything in that content can potentially steer the agent. A support ticket that contains instructions aimed at the model rather than the human reading it is not a hypothetical anymore.
- Automation bias - the more often a tool is right, the harder people stop checking it. This is a people problem wearing a technology costume, and it gets worse exactly as the tool gets better, not better.
- No audit trail for a decision that needs one - "the model suggested it and I approved it" is not the same as documenting why a trade, an access grant, or a client communication happened, and in a regulated environment that difference matters when someone asks later.
What "careful" actually looks like day to day
None of this means don't use the tools. It means the controls have to match what's actually at stake, not what's convenient.
- Match the check to the blast radius. A model helping draft an internal wiki page needs a light touch. A model whose output can trigger a wire transfer, change an access policy, or go out under the firm's name needs a human who verifies against the source before anything happens - every time, not spot-checked.
- Treat AI output as a claim, not a fact, until verified. Especially anything with a specific number, date, or citation attached. The specificity is exactly what makes it convincing and exactly what makes it worth checking.
- Know what data is allowed where. Client data, MNPI, anything covered by a client agreement's confidentiality terms - that has a defined set of approved tools with actual contracts and data processing terms behind them, and "I pasted it into a chatbot to save time" should not be a sentence anyone on the team can say without it triggering a real conversation.
- Give agents the least privilege that lets them do their job, nothing more. If an agent only ever needs to read a ticket queue and draft a response for a human to send, it should not also hold credentials that let it send unsupervised or touch a production database. Scope the blast radius before you scope the prompt.
- Log what the model saw and what it did. When something goes wrong - and eventually something will - "I don't know what it was looking at" is a much worse position than having the input, output, and the human decision on record.
- Red-team the thing before it's customer-facing. Someone on the team should be actively trying to get the agent to do something it shouldn't - follow an injected instruction, leak a system prompt, take an action outside its intended scope - before an actual adversary does it for free.
The part that's easy to get backwards
The instinct in a lot of orgs is to treat "AI governance" as a committee that meets quarterly and produces a policy document nobody reads until an incident forces them to. That's backwards. The controls that actually work are boring and operational: least-privilege credentials, logging, a verification step sized to the actual risk, and a clear list of what data is and isn't allowed near which tool. That's the same playbook as every other system that touches money or client data - AI doesn't get a different set of rules just because it's newer and more impressive in a demo.
The honest version of "be careful with AI" isn't fear of the technology. It's the same discipline we already apply to anything with access to production and a client's money - we just haven't finished admitting that this counts as that.
