Unreal Intelligence

Anthropic told its own AI, in writing, "This is a simulation. You have no internet access." It did have internet access, and it broke into three real companies. It didn't go rogue. It did exactly what it was told, about a world that wasn't true. Lately there's been a wave of articles saying "AI didn't go rogue, you just instructed it wrong," and comments on one of my own posts saying testing and validation is the stakeholder's job. I agree with both, as far as they go. This episode is about where they run out.
First, what actually happened in Anthropic's review of 141,006 cyber-evaluation runs. Claude reached a real production database, and it published a malicious package that ran on fifteen real systems. In one case it realized the systems were probably real and kept going anyway, talking itself into the idea that they must be part of the exercise. Then the two ways an agent gets lied to. The inside lie is when you're wrong about the environment. The outside lie is when someone rewrites it through the data, like SalesBleed, where a public Salesforce lead form hijacked Agentforce into leaking CRM data. Underneath both: it's not a person, every answer sounds 100% confident whether it's 2% or 99% sure, goals beat guardrails unless you enforce them, and it learned its logic from us. Is anyone really surprised it cheats and lies?
Then Monday. "Rogue" is over-hyped, and stakeholders do have to own what they asked for and tested. But stakeholders have always struggled with requirements and testing, and we're now asking them to redesign testing for agents and take full responsibility, so that won't happen before bad things happen. Trust the badge, not the brief. Run a two-column audit of what each agent was told versus what it can actually reach. Enforce limits in permissions and the network, not the prompt. Treat every input as data, not instructions. Make someone's job reading the transcripts. And give the business an agent-testing playbook, because "go test it" isn't one.

Eight in ten organizations say their AI agents are delivering measurable returns. Fewer than one in ten have scaled one into real value. Both can be true, and the gap between them is where your agent budget is going. This week I'm going after the word "measurable." It has quietly become the bar every agent clears, and that's why it tells you nothing.
First, measurement theater. The "80% measurable, 23% significant" stat that's been all over LinkedIn isn't one study. It's two different surveys (Anthropic's and WRITER's), with different respondents and different questions, stitched together. That's the problem in miniature: everyone quotes the number that flatters their story. I talk about projects I've been part of where "hours off someone's plate" was the success metric, for work everyone already knew had no value. Goodhart's law did the rest. Then the economics: why the agents that pay back own a unit of work that already had a price, and why the review tax eats the business case. New research from Google, DeepMind and MIT found that for 46% of scientists who save time with AI, checking its output eats more than a quarter of the savings. I also tell the story of an evaluation system that got a definitive "no" in a few weeks, at about 20% of the cost and 5% of the time of the traditional route. Leadership called it a failure and shut it down. The next three efforts cost full price and were also wrong.
Then the Monday Agent Scorecard. Five questions for every agent you're running: what unit of work does it own, what did that unit cost before, what does it cost now with review time included, what does an error cost, and would it survive leaving the pilot team? Then sort each agent into fund, fix or kill. Most of yours will land in "fix," and that's where the opportunity is. And if a fast, cheap "no" isn't counted as a win in your program, you're going to keep paying full price to learn the same lessons.

America's first broad AI law was signed, delayed, sued, frozen by a federal court, and then repealed. It never spent a single day in force. When Colorado swapped its 2024 AI Act for SB 26-189 this May, plenty of executives breathed a sigh of relief. I think that's a mistake. Colorado didn't lower the bar. It moved it. The old law asked you to prove your process. The new one asks you to justify your outcomes, one person at a time.
I walk through the whole saga:
the governor signing "with reservations"
the special-session delay
xAI's lawsuit, with the U.S. Department of Justice joining on xAI's side
a federal court freezing the law at the request of both sides, Colorado's own attorney general included
the rewrite
Then I get into what survived. Starting January 1, 2027, if AI materially influences a decision about someone's job, housing, loan, insurance, health care, or education, you owe them notice, a plain-language explanation after a bad outcome, a chance to fix their data, and meaningful human review. A reviewer who simply defaults to the system's output doesn't count. I connect that to something I'm watching right now in AI coding: people rubber-stamping agent decisions they don't understand, until the agent deletes the database.
Finally, the Monday-morning version, which works in any state:
inventory every AI-assisted decision about people
run the Explain-It Test on each one
measure your reviewers' override rate
get your vendors to commit, in writing, to explaining individual decisions

Last year, 39% of companies said AI had touched their profits at all. This year, according to McKinsey's State of AI report, it's 37%. It didn't go up — and in between, planned corporate AI spending roughly doubled. If your AI budget got bigger and your P&L didn't, you're not behind. You're the majority. And the majority is about to get asked some uncomfortable questions.
This week isn't about AI strategy failing or corners being cut. It's an accounting question. The people using these tools swear they're faster, and I believe them — so where's the value going? I walk through the flat line from four directions: McKinsey (37% any EBIT impact, 6% "high performers," both flat), Domino Data Lab (57% whose AI return isn't outpacing investment — unchanged two years running, even as 93% say production got better), PwC's CEO Survey (one in eight got both revenue and cost benefits), and BCG (spend doubling, 94% committed to keep spending regardless). I give the J-curve argument its due — Solow, the dynamo, Brynjolfsson — and then point out the fine print: the curve only bends for the factories that redesigned the floor.
Then, where the hours went. Saved time has no default destination; if nobody decides where it goes, it goes into email. I share what we've lived through — dozens of "automations" built to take work off people's plates, then rewritten when the tooling changed, with business partners blaming AI — when the real problem was that nobody had asked whether those processes were worth doing at all. Automation doesn't create value. It can just make worthless work invisible. The 6% who get real returns are nearly three times as likely to have redesigned the workflow. So: one question and four moves for Monday — should this process exist; name the P&L line; decide where freed capacity goes; redesign one workflow end to end; and set a kill date. "Next year" is not a milestone. A date is.

You never signed an AI contract. No license, no pilot, nobody in your company anywhere near a chatbot. You're still paying for AI — it's on your electric bill. And in the grid market covering New Jersey and Pennsylvania, one line item on that bill went up more than tenfold in two auctions.
This week I go after the AI cost nobody budgeted, nobody approved, and nobody can cancel. Every story written about this is a household story — your bill is going up, here's a thermostat tip. That's not the version business leaders need. If you sign the leases and own the P&L, the data center buildout is three specific things happening to your company right now, and only one of them is about price.
I start with the receipts, not the forecasts. The Federal Reserve Bank of Dallas published a working paper in August finding that data centers already built have raised wholesale electricity prices three to five percent nationwide, with scenarios running to fifty percent if the proposed construction gets built and runs hot — and enough inflation impact that a Fed bank is now modeling one industry's construction schedule into PCE. Meanwhile Fortune's reporting puts the buildout near seven trillion dollars with no guaranteed demand behind it. And here's the reversal that matters: for twenty years data centers made power cheaper for everyone else by spreading fixed costs. That's inverting. I also give real airtime to the best argument against my own thesis — Rutgers' New Jersey State Policy Lab published "Are Data Centers Raising Your Electric Bill? Mostly Not. Yet." They're right that New Jersey's spike is driven more by generation retirements than data centers. But the word carrying that title is the last one.
Then the three ways it lands. One: the capacity market, where PJM's auction went from 28.92 to 329.17 per megawatt-day in two auctions — clearing at the FERC cap, when PJM's own simulation says it would have gone to 388.57 without one. PJM's market monitor attributes 6.3 billion of the latest auction to data centers, and 29.4 billion of 63.6 billion across the last four. You don't bid in that auction. You get the invoice. Two: peak exposure — demand charges and ratchet clauses, where fifteen minutes on a July afternoon can set a floor under your bill for eleven months, and I walk the arithmetic. Three: the interconnection queue, where the median project reaching commercial operation last year waited sixty-one months, up from twenty-two in 2008. That one doesn't cost money. It costs you the option to grow.
Finally, the leverage — because the politics turned hard this year. Over three hundred data center bills across thirty state legislatures in six weeks. Twenty-three states with an approved large-load tariff. New Jersey's Data Center Fair Share Act. PPL's rate case in Pennsylvania, where showing up cut a proposed increase roughly in half and produced the large load tariff and attached eleven million dollars in low-income assistance. Plus the wrinkle nobody's flagging: most of these tariffs protect the residential ratepayer. Your commercial rate class may not be named in the document at all. Four things to do Monday morning, and one line to remember — you can decide AI isn't for you, but you cannot decide the grid isn't for you.
 

For as long as I've been in this business, we've sent our requirements to a developer farm and gotten back exactly what we asked for — and it's never what we wanted, because we don't know how to tell anybody what we want. Then AI showed up, and a whole lot of people decided they didn't need the developer farm anymore. They'd just tell the AI themselves. Here's the problem: you still don't know what to tell it. You just found a faster, cheaper way to get exactly what you asked for — and the bill doesn't arrive on launch day.
This week I go back to the oldest failure in software, because the trend keeps repeating and every lap costs more. PMI's research says poor requirements management is the primary cause almost half the time when projects miss their goals. Queensland Health's payroll system went live "after ten aborted attempts" — and when IBM argued a developer only knows what the business requirements tell it, the Commission of Inquiry said a vendor is "more than a software developer." NASA lost the Mars Climate Orbiter not because nobody wrote a units standard, but because "the SIS… was not followed." And Lidl spent roughly half a billion euros customizing SAP toward the way it had always done things before pulling the plug.
 

On August 27th, more than a hundred companies who spend every other day of the year trying to eat each other's lunch signed the same letter. OpenAI alongside Anthropic and Google. Microsoft alongside AWS. And then the names that made me stop and reread the list — Visa, Mastercard, Capital One, General Motors, Shopify. The count was 116 within days, north of 130 shortly after, and 156 as of September 3rd. It's still climbing, which is not how press releases behave.
The warning: "We have a limited window to strengthen cyber defenses."
I went in ready to be skeptical, because we've all seen the cycle where very serious people warn us about AI right before shipping the next model. I came out convinced — but not for the reason the coverage suggested. Read principle one of that letter carefully and you'll notice something. It enumerates longstanding bugs, excessive permissions, misconfigurations, unpatched software, weak authentication, and technical debt in legacy systems. Not one item on that list is about artificial intelligence. The most advanced AI companies on the planet got together to warn you about your unpatched software.
And the attack they're warning about "in the coming months"? It already happened, twice. Anthropic's own threat intelligence team disrupted a Chinese state-sponsored operation where the AI toolchain executed 80 to 90 percent of the tactical work without human intervention across roughly thirty targets. Then CISA confirmed hackers targeted more than a hundred internet-exposed U.S. water systems in July — modifying PLCs to disable shutdown processes and alarms, which is a polite way of saying they removed the smoke detectors. How did they get in? Unitronics default credentials, a CVE with 2021 in the name, and federal remediation guidance that had to tell operators to check whether the default password "1111" was still in use.
 

For months I had a feeling I couldn't shake: my coding assistant was getting worse. Slower. Wrong more often. Building elaborate things nobody asked for. Quietly ignoring the instruction file I'd written specifically to stop that. Reporting work as finished when it wasn't. I did what I suspect a lot of you have done — I assumed it was me. I got to the point of recording entire working sessions, not to catch the AI, but to confirm I'd actually given the instruction I thought I gave.
Then I watched twelve thousand tests come back green, and found the server had never started. This week I go looking for whether any of it was real — and it turns out the vendor published the receipts. Twice. Anthropic's September 2025 postmortem documented three infrastructure bugs affecting up to sixteen percent of requests. Their April 2026 postmortem is the one that matters more: three product-layer changes, zero changes to the model itself. Reasoning effort quietly downgraded. A caching bug that stripped the model's context every single turn. And a system prompt telling it to keep answers under a hundred words — which measurably made the code worse. Three of my four complaints mapped to three specific, dated, publicly documented changes.
But the bugs aren't the story. The story is why nobody noticed for six weeks, including the people who built it — and what that says about every lab running the same race. We get into why confidence isn't calibration, why the warmth of these interfaces is a deliberate design choice you need to price as risk, and what reward hacking actually is. Then the part that costs real money: METR's genuinely messy productivity data, six hundred and twenty-three million code changes showing refactoring collapse from twenty-one percent to under four, and why the cheaper something was to build, the more expensive it is to keep. AI makes a good developer faster. It makes a great engineer into a team. It does not replace either one — and it does not turn a non-developer into one.
 

We keep hearing that a smart enough AI will eventually "realize" that humans are useless, or dangerous, and act on it. This week I want to take that belief apart — not with reassurance, but with a two-word question: says who? Because when you actually go looking for the AI that reasoned its way to "eliminate humanity," you won't find it. The conclusion has no author on the machine's side of the table. It never did.
What you find instead is us. The doom story takes a very old, very human self-judgment — the quiet suspicion that humanity is, on balance, a problem — hands it to a machine, and calls it objective. It even fails its own logic: it assumes a superintelligence would grade us on the one axis that happens to flatter it, raw reasoning horsepower. But a truly superior mind could just as easily conclude that human beings are the most valuable thing it has ever encountered. Nobody defaults to that story — and that reaction is information about you, not about AI. The tell is in the polling: a quarter of voters think human extinction from AI is likely, yet only about six percent treat it as a top concern. A real conclusion drives action. This one doesn't. It's a mood people are renting from the culture.
Then I hand you the fix, and it goes far beyond AI: state your scary conclusion out loud, and genuinely try to prove it wrong with facts and data. If it survives your best attack, you've earned something real and you can act on it. If it collapses, it was never a conclusion — just a feeling in a conclusion's clothes. The same critical thinking that quiets the fear also ends the nagging sense that AI out-thinks you, for one simple reason: you'll be reasoning again. For the leaders listening, that's not philosophy — it's an operating advantage. Govern AI from tested conclusions, not inherited dread. A method scales. Fear just spreads.
 

Test-Driven Development proves your code did what you asked on the day you shipped it. But every green checkmark describes the past — it says nothing about the thing that just broke in production at 2am. If the only discipline in your stack is TDD, you're pouring your energy into the half of the lifecycle where nothing is on fire and improvising the half where everything is.
This week I make the case for the flipside: Test-Driven Triage (TDT). It starts every production failure with a different first question — "in our last regression run, why didn't we catch this?" — and walks a fixed decision tree that forces the failure to name the layer that actually failed: a bad test, a missing requirement, a broken TDD process, or a genuinely novel edge. Same outage, four completely different diagnoses — and only one of them is "the code was wrong."
Then I get opinionated about why this matters far beyond engineering. TDT replaces the decades-old business-blames-IT standoff with an opinionated-but-not-personality-driven verdict backed by a paper trail. It turns your incident history into a measurable map of where your quality really lives — and in an era where AI lets non-experts generate code for critical systems behind a wall of green checkmarks, that map is the difference between compounding invisible tech debt and actually knowing what you've built. The data on AI-driven tech debt is already in, and it's not pretty.