The Systems Belong to Us

Years ago, I was sitting in a conference room at a television network when the head of the network asked us a simple question.

“Who do we work for?”

I said the viewer.

I was nearly laughed out of the room. I was told we worked for the advertiser. The viewer’s attention was inventory. Our job was to capture it and sell it.

Yesterday, an artificial intelligence researcher resigned from Anthropic because he believes the race toward self-improving superintelligence presents an explicit risk to humanity.

He wasn’t alone. Another senior scientist said he and his colleagues genuinely believe there is a one in ten chance that Super AI could kill all humans within the next decade. 10 percent chance of extintion. Imagine an airplane whose engineers tell you there is a greater than ten percent chance everyone aboard will die.

Nobody gets on.

Yet we are all on this one, and no one asked our permission and the political system that should manage it is similarly out of our control.

The distinction between artificial intelligence and artificial superintelligence matters

. Artificial intelligence has extraordinary utility. I use it every day. It can build a website, analyze a business, propose a marketing strategy, draft a legal document and give one person creative capabilities that once required a team of people.

That expands human agency.

The problem is when the tool becomes more important than the human being it was created to serve.

I have seen this before. At Bloomberg, the mandate had been clear. We worked for the reader and the viewer. Our job was to provide the best information possible so people could make the best decisions for themselves.

I loved that job. It made sense to me.

At this network, I was told that I had it completely wrong. We worked for the advertiser. The viewer’s attention was inventory. Our job was to capture as much of it as possible so it could be sold. Media has enormous utility when it informs people. It becomes destructive when manipulating people to capture their attention becomes more important than serving them.

The same inversion exists elsewhere.

Finance has enormous utility when it connects capital with people who can put it to productive use. In 2009, millions of people lost jobs, homes and savings while the institutions at the center of the crisis were protected. Politics has enormous utility when it represents people. It becomes destructive when it is political monopoly that serves special interests (like Super AI companies) at the expense of the citizens who created it.

Corporations can organize thousands of people to accomplish extraordinary things. They become destructive when extraction, valuation and growth become purposes unto themselves.

Now we arrive at superintelligence. The people closest to its development are telling us that they are racing toward self-improving systems that could become uncontrollable. The competition itself makes slowing down difficult because each company fears another company—or another country—will continue.

This is where the problem becomes more than technological.

It becomes spiritual.

If you genuinely believe there is a meaningful possibility that what you are building could destroy humanity, what exactly do you believe you are going to win? Another billion dollars has no value against that risk. Neither does a larger company, another election, more market share or victory over another country.

There is a profound spiritual void in believing otherwise.

I don’t mean religion. I mean the basic connection of one human being to another. The recognition that the person across from you is as alive as you are. Their children matter as much as yours. Their suffering is as real as yours. We are all here for a short time together. Lose that connection and pathological behavior begins to look rational.

The spreadsheet works. The incentives work. The strategy works. The valuation goes up.

The human being disappears.

That is the illusion underneath all of these systems. There is nothing of real value at the end of a competition that requires destroying other human beings to win it. I learned a small version of this in television. The stakes were human attention and information. I eventually walked away because I could no longer reconcile myself to a system that I believed had stopped serving the people it was created to serve.

The stakes surrounding superintelligence are infinitely greater. Regulation matters. Technical safeguards matter. Alignment research matters. But none of them can substitute for restoring the human connection that tells us what all of these systems are for in the first place.

That, increasingly, is what I mean by human agency: the capacity to look at a system demanding that you continue and decide for yourself whether what you are doing serves human life.

The system had become more important than the human being.

That is the inversion we have to correct. We have to restore our connection to ourselves, to one another and to the idea that being alive means something beyond accumulating more, winning the race and beating the other guy.

Human beings built these systems to serve human life.

We created markets, governments, corporations, media and technology because they could improve the human experience.

They are our tools.

The system belongs to us.

We do not belong to it.