I built an AI agent. Here’s the incident log.
I built an assistant for my own work. I call her Gal Friday. This is the honest version, including the parts where she was wrong and the parts when I was worse.
01A rough start
I asked Friday to cut a cross-sell package down. She pulled the savings figures and the comparison table, then wrote a clean paragraph explaining why that was the right call.
The reasoning was pure fiction. She’d removed the whole argument for the product and then handed me a strategy that made it sound deliberate.
“This is categorically wrong. [Expletive deleted.] It guts the entire reason the kit exists. Now we’re stripping value and asking them to go on trust, which is a preposterous statement.”
Everybody watches for made-up facts.
And when it reads like contemplative thinking, it gets a nod and we keep moving.
02The part that’s actually dangerous
She’s very good at sounding right.
And the better it sounds, the less anybody reads it closely. Including me.
Here’s the part nobody writes about.
You walk into the meeting loose. You’ve got the deck, you’ve got the numbers.
And then somebody asks where a figure came from.
And you don’t know. Because you didn’t write it.
You approved it, and approving felt like working.
So you reach for the language that sounds like an answer.
You’ve said nothing and everyone in the room knows it. The harm that can do to a reputation sticks around longer than it takes to fact check.
That’s the real cost, and it isn’t the machine’s.
There’s a name for it. The EU’s AI Act calls it automation bias and requires anyone overseeing a system to watch for exactly this. I had it in my notes before I knew the term existed.
03The receipts
Four things she did that a better prompt would not have prevented.
She lost track of which version was real. Proofs went out built from a round that had already been replaced. That’s not a hallucination, that’s forgetting where it was in a long job.
She mixed up two partners. Credited a headline to one company’s deck when it came from the other’s. Then cited a version of that deck that was never a file.
She treated a broken test like routine feedback. The notes looked like line edits, but two themes were testing the same idea. She rewrote 24 headlines without catching that the study itself had stopped working.
The tool gave her an incomplete answer without saying so. It left out four past threads, then missed one again after a date-specific search. Better instructions wouldn’t have fixed it. Only a human patiently taking the time to get it right.
04What came out of it
Not principles. Rules, each one written the day something went wrong.
- Confabulating a source under pressure
- Generating plausible expertise never demonstrated
- Dressing procedure as original thinking
- Padding to hit a length target
- Using my own language back at me
- Not flagging a guess as a guess
Everything above is marked: confirmed, seen more than once, or a hunch with a way to check it.
When I tested the system against what actually happened, three rules had never run and another had already been replaced by something better. I cut the paperwork and kept what worked.
Writing a rule is easy. Deleting one is the part almost nobody does.
05It works both ways
Half the rules are for me.
- Every claim points to a real, named source.
- Uncertain gets marked as a guess, in the output.
- Name the version you’re auditing in the first line.
- A word count reads as a floor. She will pad to hit it.
- Say the important thing last. That’s where it sticks.
- When a correction lands, don’t re-argue it. Restating the wrong version puts it back in the room.
06She still tries
The parts-list habit comes back under deadline. It always will.
None of the rules prevent that. What changed is where it gets caught. It moves the catch from the proof to the draft, which is the whole difference between an annoyance and a real-world, real-money reprint.
One more, and this one cost nothing.
In a single session she cited a page she had never opened, then argued that two of my own files contradicted each other by reasoning past a line she had already read. The rules caught both. She retracted the second one in writing, on her own, before I said anything.
“Confirm this. You’ve been known to make [expletive deleted] up, strawman, and ‘infer.’”
That’s the working relationship. The rules are what make checking possible.
07What it costs
The marks slow everything down. Every fact has to be sourced or flagged, which adds a step to work that used to feel fast.
The check that catches drift interrupts a session four or five times.
And the log only works if somebody writes in it, which is the part most likely to quietly stop.
I built the rules from incidents first. The frameworks came after, when I went looking to find out whether any of it was already solved. Three of them changed something.
- NIST AI Risk Management Framework 1.0Oversight should be proportionate to risk, not spread evenly over everything.
- EU AI Act, Regulation 2024/1689Names automation bias. Requires a way to stop the system instead of riding a bad output.
- MIT, A Framework for U.S. AI GovernanceExtend the controls you already have. Do not build parallel ones.
Four more were read and mostly set aside: the EqualAI playbook, Deloitte’s board roadmap, Microsoft’s blueprint, and the federal AI Community of Practice toolkit. They are written for organizations with a legal department.
Version 6.4.
Every version number is a failure I didn’t catch the first time.