The Last Check

Two developments, arriving together, threaten to answer that question no for the first time.

The first is robotics — physical and embodied automation capable of labor and, eventually, force. The second is recursive AI — systems capable enough to improve themselves and to do the cognitive work that used to require a workforce of skilled humans. Each alone is disruptive. Together they remove, at the root, the dependency every prior concentration of power has rested on.

Consider the guard. A robotic enforcement apparatus does not defect. It develops no sympathy for the people it is pointed at. It does not wonder, at 3 a.m., whether the orders are right. It has no family in the crowd. The single most reliable circuit-breaker in oppression's history — that at some threshold of cruelty, the human instruments of power refuse — does not exist in a machine with no inner life to revolt. A regime that automates coercion has cut the wire that every previous regime, however brutal, was ultimately wired to.

Consider the worker. A robotic, automated economy does not unionize, does not strike, does not need to be paid enough to keep buying the things it makes. The leverage working people have held — sometimes weakly, sometimes decisively — came from being necessary. Withdraw necessity and you withdraw the leverage. Not gradually weaken it: remove its basis entirely.

And consider the gap. Recursive self-improvement, if it works even partially, has a property no prior technology had: the leader can use its lead to extend its lead. A system good enough to do frontier research improves at machine speed while everyone behind moves at human speed. The advantage doesn’t merely persist — it compounds. Past technological leads could always, eventually, be caught, because everyone ran on the same human clock. This one might not be, because the leader is no longer running on the human clock at all.

Now watch the two halves converge. The economic check — they need us to work and to buy — and the coercive check — they need us to enforce and to obey — were load-bearing precisely because they were the same people. Workforce, soldiery, and citizenry overlapped; you could not discard one role without weakening your grip on the others. Automation severs both at once. What remains, from the vantage of a sufficiently automated apex, is a population that is neither a workforce nor a market nor an instrument — only a cost, and a risk.

This is the genuinely unprecedented thing. Not that power concentrates; power always concentrates. It is that, for the first time, power might concentrate without remaining dependent on the people over whom it is exercised.

The reflex is to imagine this requires monstrous intent — some figure who wants the boot on the neck forever. That reflex is a comfort, because monsters are rare and identifiable. The truth is worse and more ordinary.

We have a recent, instructive case. OpenAI was founded explicitly on the premise that this kind of power was too dangerous to concentrate — that it must be open, distributed, kept from any single actor. The name was the thesis. And it became one of the most closed, most commercial, most concentrated actors in the field, its founding mission worn down to vestigial branding. This did not happen through villainy. It happened through a sequence of locally reasonable steps, each defensible on its own terms: we need capital to compete; we need revenue to fund the mission; we cannot unilaterally slow down or we hand the lead to someone worse. Every step sincere. The sum: the precise outcome the founders set out to prevent.

That is the actual shape of the danger. Not ego, not malice — structure. A concentration of power need not be seized by a bad person; it can be arrived at by good people responding rationally to their incentives, where the locally sensible move at every step sums to a globally catastrophic destination. This is more frightening than villainy, because you cannot fix it by finding better-charactered people. But it is also, in one crucial respect, more hopeful — because incentives and structures are things human beings can change, and character is not.

Which means the whole problem reframes. The question is not “how do we ensure the people in charge are good?” — that question is unanswerable and always has been. The right question is: how do we build a structure that does not depend on anyone being good? That is the only kind of safeguard that has ever actually held.

Against concentration, the most important counterweight is some form of openness — a frontier capability not owned by one or two private actors, so no single entity can charge infinite rents, close every door, or pull so far ahead it stops needing the rest of the world.

The standard objection is proliferation: an open model is downloadable by anyone, including the worst actors. This is real and must be taken seriously. But it is almost always deployed with a sleight of hand — it compares the two risks on vividness rather than on probability over time, and that comparison is rigged, because catastrophe is easier to picture than erosion. Set them side by side honestly:

A system that reliably corrupts over time is a worse bet than a tool that rarely enables catastrophe — because you can build immune responses to incidents, but not to the immune system itself going bad. Concentration is a disease of the body politic; proliferation is a wound to it. You recover from wounds. You do not recover from the organ meant to fight disease turning against you.

There is exactly one condition under which this calculus flips: if the proliferation harm is not bounded — if a single success is not a survivable atrocity but a self-amplifying, civilization-scale event. That is the real crux of the entire open-versus-closed debate, and most of the debate refuses to name it.

Here the closed argument has a hidden weakness. If open capability is unbounded for the attacker, it is unbounded for the defender too — and defenders outnumber attackers by factors of millions to one. Where defense aggregates — many people hardening systems into a stronger wall — diffusion of capability is net defensive. This is the well-worn lesson of cybersecurity: openness mostly favors the defense, because there are vastly more people patching holes than exploiting them.

But the symmetry breaks in a specific, nameable place: offense-dominant domains, where three things hold at once.

Where all three hold — engineered biology is the canonical example — more defenders do not save you, because the harm replicates faster than the defense can be deployed even when it exists. This gives the precise scope of the real danger. It is not “AI is dangerous”; it is a narrow slice of self-amplifying, offense-dominant capability. The decisive move the closed camp makes is to take that narrow slice and use it to justify closing everything — and the actor making that leap has every incentive to claim the slice is wide, because “it’s wide” is also the argument that protects a multi-billion-dollar asset.

Once the problem is scoped this precisely, the false binary of “open versus closed” dissolves — it was applied to the wrong unit. The choice was never whether a model is released, but to whom and when. The right structure is neither “post everything” nor “lock it all” but structured, tiered access — and it is not exotic. It is how every society already governs dangerous knowledge: select-agent pathogens, controlled nuclear materials, restricted chemistry. The capability exists; access to its dangerous form is gated to verified, accountable people; everyone else receives the benefits without the raw capability.

Applied to frontier AI, the design has several interlocking parts.

Assembled, this is a coherent position the simple binaries cannot reach: release as the default; the narrow offense-dominant slice gated to verified defenders through transparent multilateral institutions; a public frontier option; government as check and never as partner; a legal cadence keeping oversight abreast of private capability; and defense funded ahead of diffusion. It defuses proliferation without enabling concentration — the only structure on offer that addresses both failure modes instead of trading one for the other.

A design is only as trustworthy as its weakest seam, and once the architecture leans on timing and access rather than on cleanly subtracting capability, only one seam remains — and it is not technical. It is political: keeping the gate uncapturable. Every component above can be subverted by the same move — the gate becomes a gatekeeper, the public model becomes a state secret, the multilateral body is captured by its strongest member, the cadence mandate is quietly waived, the “defensive head-start” becomes a permanent excuse never to open at all.

No mechanism guarantees against this. The only defense is the principle the whole essay has been circling: distribute the power so that no single actor — corporate or state — has to be trusted, and build the accountability into the structure rather than the character of whoever holds the keys. This is the slow, unglamorous, permanent work of building institutions that check one another, and then defending them, indefinitely, against the constant pressure to fuse and concentrate.

Be clear about what kind of problem this is: naming it correctly is the difference between despair and direction.

The thing that has restrained power for all of human history — the dependence of the powerful on people who can refuse — is a natural check. It came for free. It required no design, because it was built into the physical fact that power ran on human cooperation. Recursive AI and robotics threaten to remove it, and nothing in nature will replace it. For the first time, a check on concentrated power will have to be built deliberately — because the one we inherited from the structure of the physical world is being engineered away.

That is the real meaning of this moment. Not that catastrophe is certain — it is not. But that the automatic safeguard is failing, and the only replacement is an intentional one: structures that distribute power on purpose, that don’t depend on anyone being good, that hold the gate open by rule rather than by luck.

This is hard, and slow, and boring, and political, and it will not be finished by any single elegant idea. That it is hard and boring is precisely the sign that it is the real answer — the dramatic alternatives, trust the careful lab, lock it all down, release everything and hope, are all easier to say and all worse. The boring answer is the true one.

We are the first generation that will have to consciously construct the restraint every previous generation received for free — from the simple fact that a king still needed his guards to stay loyal, and his guards were men. The guards are about to stop being men.

That is the work. It is worth doing, and the time to do it is while the gap is still narrow enough that building is still possible — because the one thing the whole analysis agrees on is that the window to act is early, and the danger is that no one treats it as urgent until it is already too late to be early.