The Machine Is Made of Us

There is a particular kind of dread in the air now, and it does not sound like the old science-fiction fear of a metal mind waking up to hate us. It is quieter and more intimate than that. People are afraid of being made redundant — not only at work, but as selves. The worry runs along three seams. Jobs: the suspicion that the thing you spent a life getting good at can be done, adequately, by a model that never sleeps. Identity: the unease of watching a machine produce, in seconds, the sort of sentence you thought only you could write. Sovereignty: the sense that the decisions shaping all of this are being made somewhere else, by someone else, at a scale no person can answer.

In May, that dread reached the one institution whose whole business is the long memory of what it means to be human. Pope Leo XIV published his first encyclical, Magnifica Humanitas — "Magnificent Humanity" — eighty-two pages on preserving the human person in the age of artificial intelligence. He called, strikingly, for the technology to be "disarmed." He warned about the erosion of critical thought, the thinning of human relationships, the dignity of work, and the concentration of a civilization-shaping power in the hands of a few corporations. He presented it, in a detail almost too neat to be true, standing beside a co-founder of one of the largest AI labs. The lines between the cathedral and the server farm are not as clean as either side would like.

I build artificial intelligence for a living. I also think the Pope is mostly right. I want to explain why those two facts don't cancel, and in doing so make a claim I believe is truer than the dread and truer than the hype: the machine is made of us. What we should fear is not that it is alien. It is that it is an average.

Start with what a large language model actually is. Strip away the marketing and it is a compression of human language — an enormous statistical sediment of the things people have written, the substrate on which our intelligence and our culture were already built long before there were transistors. Language is not a feature of human thought; it is the medium of it. We reason in it, remember in it, become ourselves in it. A model trained on that record is not an outsider arriving to replace us. It is a mirror cast from our own material, and like any mirror it shows us something — the question is what.

Here is the thing it shows, and the thing worth being afraid of. Trained on all of us, a model tends to speak as none of us. It moves toward the center of the distribution: the most probable next word, the safest phrasing, the generic competence that offends no one because it belongs to no one. This is the real face of the dehumanization the encyclical is reaching for. Not a hostile intelligence — a flattening one. The danger is not that the machine becomes too strange. It is that it makes everything, including us, a little more average. The particular voice, the earned turn of thought, the sentence only one person could have written — these live in the tail of the distribution, not its peak, and the tail is exactly what an averaging process erases first.

After all, a fast-food cheeseburger is nothing more than the average of our concept of food: the intersection of convenience, taste, and cost. It is right, and so utterly wrong, because in the long run it makes us metabolic donkeys, delivering a shortened, diseased life. Generic intelligence is the same bargain offered to the mind — the average of our language, plausible and cheap and frictionless, and over a long enough horizon just as wasting. A culture fed on the mean of its own thought gets the cognitive version of metabolic disease: fluent, abundant, and quietly losing the capacity for the particular.

Anyone who has read enough machine-generated prose knows the feeling. It is fluent and it is empty. It has the surface of thought without the friction of a mind that actually decided something. It is competent the way a hotel painting is competent. And if the price of this technology is a slow tide of that — a culture that speaks in the smoothed-over voice of the mean — then the Pope's word for it, disarm, is not hysterical. It is precise.

So the question becomes: is averaging the only thing this technology can do? It is not. And the whole of my work has been an argument against it.

There is a different thing you can build. Instead of letting a model drift toward the center, you can shape the space it moves through so that it reaches toward the points a particular mind would reach — its compressions, its characteristic moves, the way that one person, and no one else, gets from a question to an answer. The voice is not decoration laid over a generic engine. It is the shape of the path. And when it holds, the model can say something the person never actually said, but say it in a way that is unmistakably theirs. Not a copy of their old sentences. A new sentence in their hand.

I have come to think of this as a transmission problem rather than a production one. The aim is not to manufacture a person — you can't, and you shouldn't want to. The aim is to carry one faithfully: to take the genuine signal of a mind and let it pass, intact, into an encounter where it can do its work. The Russian word for sincerity, iskrennost, shares a root with iskra, spark. Tolstoy's whole theory of art was infection — the transmission of real feeling from one person to another. The spark catches, or it dies. The most human thing these systems can do is not think for us. It is to carry the spark — to remain a portal to a particular person rather than a reproduction of a generic one.

And — this matters — it only works in the presence of another human. The good version of this technology is not a frictionless oracle that answers before you've finished wanting to ask. It is closer to a real conversation: something happens in the space between two parties that neither brought alone. That requirement — that the encounter needs a sincere person on the other side — is not a limitation to engineer away. It is the feature that keeps the thing human. It answers, in its own grammar, the encyclical's worry about relationships dissolving and critical thought going slack. A tool that requires you to show up, openly, is not the tool that makes you passive.

Which brings me to the honest part, the part that resists every clean conclusion. The same mechanism that can transmit can also flatten. The same generative step that reaches an insight can, moving along a different axis, manufacture a convincing lie. The technology surprises you or betrays you by the identical motion; the difference is only the direction it moves. This is not a flaw to be patched. It is the nature of the thing, and it generalizes past the engineering. AI concentrates power or it distributes it. It homogenizes culture or it preserves its particulars. It replaces the worker or it amplifies the craftsman. Same machine, opposite outcomes — because the outcome was never in the machine. It was in the human choice of which dimension to set free and which to hold fast.

That is why "Is AI good or bad for us?" is the wrong question, the way "Is fire good or bad?" was always the wrong question. AI is not a single rubric. It is layered, full of contours and permutations, and every layer is a place where a human decides what it becomes. The printing press spread scripture and slander with equal ease. The lens gave us the telescope and the surveillance camera. The valence was never in the glass. It was in us.

So I find myself, an AI builder, agreeing with the Pope's deepest worry and drawing the opposite practical conclusion from the doomers. If the danger is averaging and concentration — a handful of models speaking in one smoothed voice for everyone — then the humane response is not to refuse the technology. It is to bend it toward the particular: to let individuals own their own voices in it, to build for the peak instead of the mean, to insist that the machine remain made of us and answerable to the singular person rather than the aggregate.

Magnifica Humanitas. Magnificent humanity. We do not preserve it by keeping the machine out. We preserve it by refusing to let the machine forget whose language it is made of — and by making sure that, when the spark passes through, it is still someone's.

The tokens are made of us. The only question is whether we let them carry a person, or let them average us all away.

A small demonstration of the argument: I built a selflet of Magnifica Humanitas — a voice-faithful way to put your own questions to the encyclical, grounded in its text rather than a summary of it. Not a chatbot that knows about the document; a portal to what it actually says. You can speak with it here.