I ship production work with Claude almost every day. I also know, precisely, where it will lie to my face. Confidently, fluently, in complete sentences, with perfect grammar and zero hesitation.
Those two facts aren’t in tension. They’re the whole job. They make AI… interesting.
The mistake I see teams make isn’t using AI. It’s trusting it like a calculator, a tool that’s either right or throws an error. Claude isn’t a calculator. It’s the most capable, most confident, most plausible engineer you’ve ever hired, and it will never once tell you it’s unsure unless you build the moment for it to. If you know where it breaks, that’s the best leverage available right now. If you don’t, it’ll embarrass you on a timeline you don’t control.
Here’s where it breaks. None of this is random. Every failure below has the same root cause, which I’ll get to.
It invents sources
Ask for citations and you’ll get them: authors, titles, journals, years, sometimes a Digital Object Identifier (DOI). Beautifully formatted. A meaningful share of them won’t exist. Not a “slightly wrong page number.” The paper is not real.
This one trips people up because the format is flawless, and we’re trained to read a well-formatted citation as a real one. I’ll be honest about how it gets me: I have too much to get done. I’m always looking for ways to cut my own effort, and citations are the easiest thing in the world to skim past. They look right while you’re scrolling them. But it only takes one fabricated reference for someone to stop trusting everything else you wrote. So now I test every citation, every time.
It makes up numbers
Hand it a spreadsheet, ask for the summary, and where there’s a gap it will fill that gap… with a number that’s the right shape and the wrong value. It would rather give you a plausible figure than say “that cell is empty” or “I can’t compute this reliably.” Plausible-and-wrong is the single most expensive failure mode in business work, because it sails straight past the gut-check that catches obvious nonsense. It’s also hard to check values when a computer generates a new document from an existing one. It must be right. Right?
It writes confident, broken code
Code that reads beautifully and fails at runtime. Calls a method that doesn’t exist on that object. Handles the happy path and silently drops the edge case. Imports a package that was deprecated two versions ago.
For me, this is the least dangerous failure, although it can be an expensive one. I’ve learned to invest in and rely on a test-oriented safety net: I run the code, check the functional results, and fix what broke. That safety net is the entire point. Hold that thought.
It drifts over long documents
Ask for something long and page one will say one thing, page twelve will quietly contradict it, and nothing flagged the conflict. The model is optimizing to sound locally coherent. It keeps each paragraph consistent with its neighbors, but not the document as a whole. Over a short answer you never notice. Over a forty-page document, the contradictions compound.
Why all of it happens
Every failure above is the same failure wearing different clothes.
Claude is, under the hood, predicting the next most plausible token given everything so far. It is extraordinarily good at plausible. It has no separate, internal sense of true that it checks against before answering. There’s no little fact-checker in there going “wait, does that paper exist?” There’s just: what’s the most likely continuation of this text? A fabricated-but-plausible citation is, to the model, a great continuation. So is a confident wrong number. So is code that looks exactly like working code.
Once you internalize that LLMs optimize for plausibility, not truth, the failures stop being surprising and start being predictable. And predictable failures are the kind you can build around.
The gap nobody talks about
Here’s the part that matters most, and it’s the reason I worry more about business teams than engineering teams.
Engineers have a safety net for all of this. Tests. Code review. CI. A type checker yelling before anything ships. When Claude writes confident broken code on my machine, the broken-ness gets caught, early and usually via automated testing. The verification is baked into how software already gets built.
Business teams have none of that. When Claude writes the customer email, the board summary, the contract redline, the campaign copy, there’s no test suite. There’s no CI. There’s just a person, often a busy one, often one who’s been told this tool is magic, hitting send. So when a wrong output reaches a customer, a partner, the public, it’s not an if. It’s a when. The only variable is whether someone checked first.
That’s the real divide in who’s getting value from AI right now. It’s not the teams who trust it the most. It’s the teams who know exactly where not to.
How I actually ship with it
So given all that, why do I use it every day, on real work, with my name on the output? Because the failures are predictable, and you can build a workflow that assumes them. Here’s mine, stripped down:
I treat every AI step as needing a check before it goes anywhere external. Not bureaucracy. Just a rule: nothing the model produces reaches a customer, a repo’s main branch, or a decision-maker without a human verifying the part that would actually hurt if it were wrong. The verification is targeted. I check the claims and the numbers and the edge cases, not the grammar.
I ask it to show its work, and I check the work, not the conclusion. “Give me the three sources” is a trap. “Give me the three sources and quote the exact sentence from each that supports the claim” makes the fabrication fall apart immediately, because there’s no sentence to quote.
I make the failure cheap to catch. For code, that’s tests. The model’s confident wrongness gets caught by a red test bar, not by me squinting. For documents, that’s asking a second pass: “find every place this contradicts itself.” The model that drifts is often good at spotting drift when you point it at the job directly. I’ve heard some tell me they use a second model to test the output of the primary model. To me, that makes sense.
I never let it be the last set of eyes on anything that matters. It’s a fast, tireless first drafter and a useful second opinion. It is not the final authority on anything with my name attached. My own rule is simple: I write a lot: engineering docs, marketing, sales, internal notes. And I try to never send any of it without reading it first. Sometimes the draft is just right and I change almost nothing; sometimes I rework it so my voice comes through. Either way, I’m checking that there’s no obvious error and nothing in there I wouldn’t actually say.
None of this slows me down in a way I would change. In fact, for marketing, I call it marketing automation! It’s the difference between a power tool with a guard and a power tool without one. Same speed. Many fewer trips to the ER.
The honest version
The frontier LLMs are genuinely remarkable! I’m not hedging when I say I build things now that I couldn’t have, or couldn’t have afforded the time to, two years ago. But “remarkable” and “trustworthy” are different words, and the gap between them is exactly where the value is for someone who knows the terrain.
The companies winning with AI aren’t just the believers. They’re the ones who learned fast, and a little painfully, precisely where it lies. And then they built the automation that catches it before anyone else sees.
I’m Brian Fromme. I help mid-market teams get real results from AI. Built and shipped, whether that’s code or content. More at atelier.purpleblossom.ai.