Back to the journal

The journal · Entry 22

Nobody can fire the robot

AGI is coming by December. Or at least, someone said so. So I went looking for what that word even means, found out it got redefined in 2018, and ended up somewhere much more uncomfortable: when the machine is wrong, who's accountable for its mistakes?

· 12 min read

Illustration of a hospital ward at night where five small robots do the work while one woman sits alone at a desk facing a wall of glowing monitors

I picked a fight with Claude at 8 a.m. this Friday.

Not a real one. I was pacing the kitchen before coffee, talking at my phone about a headline. It told me I was wrong so now you’re getting the whole spiral. I think we know how this goes by now.

The headline is the one you’ve seen. Every few months some tech CEO says AGI is right around the corner, LinkedIn loses its mind, and everyone forgets until the next one. This time the date is December. This December. Four months from now.

My first reaction was “liar liar pants on fire.” My second was to go check whether he actually said it, because a screenshot of a post about an interview on LinkedIn doesn’t feel like the best source.

He did. Sam Altman told TIME that OpenAI has “not quite yet” reached AGI but expects a system that meets his definition before the end of the year, and the chief research officer put them at 80 percent of the way there. The LinkedIn version just dropped the fine print. He’s describing a system OpenAI would consider to have crossed its own internal bar, not a product on a shelf labeled AGI by Christmas. But the betting markets, the closest thing we have to people putting money where their mouth is, have the December version in the low teens.

Which sent me to the question I should have asked years ago. What is AGI? I’d been picturing a human brain in a box this whole time, and that is not what Sam Altman is talking about.

They changed the definition in 2018

When OpenAI launched in 2015 as a nonprofit, the mission said “human-level” AI. A system as general as a person. In April 2018, right before it restructured into a company that could take investment, it published a charter with a new line: “highly autonomous systems that outperform humans at most economically valuable work.”

“Human-level” isn’t used in the definition anymore. The comparison stopped being “thinks like us” and became “does the paid work better than us” years ago. The AI Now Institute put it plainly: the clearest path to a business model was to describe AGI as the thing that automates labor, so that’s how the charter defined it. It got even more literal from there. For a stretch of the Microsoft partnership, AGI was contractually tied to OpenAI generating $100 billion in profit, later swapped for a review by an expert panel.

Data panel titled the definition moved, the deadline didn't, showing a timeline of what AGI has meant according to OpenAI: human-level AI in 2015, outperforming humans at most economically valuable work in April 2018, a 100 billion dollar profit clause in the Microsoft partnership from 2019 to 2025, an expert panel review starting October 2025, and Sam Altman's before the end of the year claim in August 2026. Below, bars compare the chief research officer's 80 percent of the way there against an 11 percent prediction market chance, plus two stats: robotics shut down in 2021 and zero dollars assigned to unpaid work

I’ve been arguing with a definition they retired years ago.

Illustration of a small robot kneeling on an open dictionary, crossing out an entry with a large marker and writing something new in the margin

“Most” is doing a lot of work in that sentence

OpenAI shut down its robotics division in 2021, so “economically valuable work” effectively means work you can do through a screen. That’s a lot of work, but it is not “most” work, and it is nowhere near a “humans won’t need to work anymore” amount.

The definition also only counts things that land on an invoice, which erases the jobs where the human connection is the invoice. Nursing. Teaching. Therapy. Sales. Ministry. Hospitality. Half of consulting, if we’re being honest. Those are paid, they are enormous chunks of the economy, and the thing being purchased is a person who shows up and is accountable for their decisions and mistakes. A chatbot walking a scared patient through a diagnosis is doing a smaller, different job than the nurse and calling it the same thing is offensive to healthcare workers.

Researchers who’ve picked this definition apart point out that it also decides unpaid work is worthless. Caregiving. Parenting. The three hours between daycare pickup and bedtime that nobody has ever paid me for in my life. All of it rounds to exactly $0.

Data panel titled most is doing a lot of work, showing that 37 percent of U.S. jobs can be done entirely through a screen while 63 percent cannot, that the 37 percent earns 46 percent of all wages, that OpenAI closed its robotics division in 2021, and a dot grid showing unpaid household work would add 25 percent to U.S. economic output while the current definition scores it at zero

The part where Claude was right and I was wrong

My original argument, the one I was so confident about in my kitchen rant, was that AGI can’t happen because we don’t understand the human brain well enough to replicate it. The medical science isn’t there. You can’t copy and improve a system you don’t fully understand.

Claude’s response, more or less: “planes don’t flap.”

Nothing in the definition requires a replica of humans. The models we have right now do plenty without any model of a human brain at all, so neuroscience was never the bottleneck. If the bar is “outperforms humans at most paid screen work,” that’s a capability curve, and I can’t point to any law of physics that says when that curve stops. If the bar is “understands the world the way a person does,” that’s a philosophy fight nobody is settling by December.

So the stronger version of my skepticism was sitting right there and I walked straight past it. The goalposts move because the company gets to grade its own exam, and gets a payout for passing it. Not because robots are magic.

Photo of a phone propped on a kitchen counter in early morning light, next to a full mug of black coffee

You can’t fire the robot

This is where I think the whole “replaces humans” story actually falls apart, and it has nothing to do with how smart the model is.

I said to Claude, “you can’t trust what comes out of a person’s mouth 100% of the time either. Right? People are wrong constantly. So why does ‘the AI might be wrong’ feel so different?”

When a person is wrong, there’s an entire apparatus around them. They can explain their reasoning. You can read their face and body language. They carry a reputation. They have a history. They can be fired, sued, or just never contacted again, and they know that, which shapes how carefully they speak and the decisions they make every single day. Trust between two people runs on “they know what happens if they’re wrong,” and it always has.

Now do that with an AI agent. What are you going to do to it? Stop using it? Stopping costs it nothing. It doesn’t even register. It doesn’t remember being wrong yesterday. It doesn’t know or care that it’s being abandoned. Its confidence isn’t tangibly connected to anything the way a human’s is. “Outperforms humans” measures the output and completely skips the accountability structure that makes human output usable in the first place.

Last week I wrote about hiring agents like employees. I still believe that. But there’s one thing an employee has that an agent will never have, and it’s a full legal name that can end up in a lawsuit.

So the human becomes the crumple zone

Follow the logic, because I did, and it’s not a fun place to end up.

You can’t hold the system accountable. Humans will always look for someone to hold accountable, because that’s how our laws, our brains and our overall society work. So the accountability lands on whichever person is standing closest to the machine.

That person has exactly two options. Check everything the system produces, at human speed, and become the bottleneck that erases the whole productivity gain. Or sign off on the work without actually checking, put their name under it, and eat the consequences when it turns out the machine was wrong.

I thought I’d figured something prolific out. Claude told me an anthropologist named Madeleine Clare Elish first presented this exact thing in 2016 and published it in 2019. She calls it the moral crumple zone. A car’s crumple zone absorbs the impact to protect the driver. The moral crumple zone runs backwards: the human in an automated system absorbs the blame to protect the technology. Her case study is Air France 447. The autopilot didn’t fail in any way its certification recognized, so the only failure the system could officially record was the pilots.

Illustration of a small car hitting a barrier. Instead of a crumple zone, a woman in a work shirt is folded up at the front absorbing the impact, while a small robot sits untouched in the driver's seat holding the wheel

Swap “autopilot” for “agent.” Picture a nurse “supervising” 200 beds through a dashboard. While the acting agents misroute meds for six weeks. The vendor points at the terms of service and the nurse’s license disappears.

The one exception Elish found was a Therac-25 radiation machine that killed patients in the 1980s. The machine had defects so obvious that the lawsuits went after the manufacturers. Blame flows to the builder when the failure is egregious and to the nearest operator when it’s subtle. Guess which kind most AI failures are.

This is also why the whole industry loves the phrase “human in the loop.” It sounds like control. In practice it functions like an accountability transfer from the people who made the machines, to the people who use the machines.

I’ve watched the version where this works

Seven years in the Navy and then nine agents running in production across U.S. Government systems, and the thing both of those had in common was named ownership. Not “the team,” a person, on paper, whose job it was to know what that system did and to answer for it, with the time and the authority to actually check the work. That’s the difference between augmentation and a liability sponge with a badge, even when both of them are looking at the same dashboards.

Close-up photo of a printed duty roster with a name written next to each responsibility. Identifying details are blurred

What I’d tell you

If someone tells you AGI is coming and humans won’t need to work, ask two questions. Whose definition, and who’s accountable when something goes wrong. Until those questions are answered, it’s not possible.

  1. Whose definition tells you what got left out. Right now it leaves out anything physical, anything unpaid, and anything where the product is the person. That carve-out covers most of the employed people I know.
  2. Who’s accountable tells you where the crumple zone is. Someone’s name is going under the output. Find that person before the launch, not after the incident, and give them the authority to stop the system and the time to review the work. Without both, they don’t have oversight. You have someone with a title and assumed risk.
  3. Keep a record of what got reviewed and what got waved through. When the subtle failure surfaces, the only thing standing between the operator and the blame is proof of what they were actually given time to check.
  4. Assume the subtle failure, not the dramatic one. Nobody is going to hack your hospital into a horror movie. (Chances are low at least.) Something will go quietly wrong for weeks and someone is going to have signed off on it every single day without realizing.
  5. Don’t confuse “useful” with “trustworthy.” I use these tools every day. I use them in my work and in my personal life. I’m typing this with a model open in the next tab that will review this blog before another human reads it. Useful is great. Trustworthy is a structure, and the structure is made of the humans holding it together.

I don’t think the people saying December are lying. If your compensation, your valuation, and your place in history all depend on AGI being close, you will sincerely believe it’s close, and you’ll be surrounded by people who are paid to agree with you. That is a normal brain inside a very weird reward system. Money does the rest.

So do I think AGI is coming in December? By their definition, maybe, and by their definition it doesn’t mean what the headline wants you to think. What I think is actually coming is a lot of people finding out they’re the new bottleneck.

And when something goes wrong and everyone turns around looking for who to blame, the answer is going to be a person, standing next to a machine, saying: it’s me, hi. 👋🏻

Illustration of a woman and a small robot sitting together on a porch step at dusk. The robot holds out a clipboard with a blank signature line and the woman holds a pen, not yet signing

Sources

Signed,

Stay in the loop

Get the next entry.

New notes land in your inbox when there is something worth reading. No cadence, no filler.