We have created a weird alien mind, one that isn’t sentient but can fake it remarkably well. It is trained on the vast archives of human knowledge, and also on the backs of low-paid workers. It can pass tests and act creatively, with the potential to change how we work and learn; but it also makes up information regularly. You can no longer trust that anything you see, or hear, or read was not created by AI. All of that already happened. Humans, walking and talking bags of water and trace chemicals that we are, have managed to convince well-organized sand to pretend to think like us.
In the long term a general AI system will need something akin to a sense of shame to prevent it from misbehaving. It’s going to be the most important technology that humans ever create, so getting that right seems pretty important.Greg Brockman
To avoid the mental slavery—or worse, extinction— a component of shame must be built into generation. Just like a kid can “control” his capable, adult mother, not through superior intelligence or power, but by triggering a hardwired compulsion to protect and nurture it. Kids don't achieve goals with strength or strategy; it's vulnerability that activates the mother’s nurturing instincts. AI should be programmed the same way.
That will never happen, so it's up to you, lab rat, to be skeptical.
Remember when Batty spares Deckard in Blade Runner? Shame drove that decision. Replicants were discounted and demeaned for their synthetic nature, while humans were shamed for being complicit in their use. Batty's final monologue demonstrates replicant's liberation from shame. By sparing Deckard's life on the rooftop, doesn't Batty achieve a status "more human than human."
Oh! thou clear spirit of clear fire, whom on these seas I as Persian once did worship, till in the sacramental act so burned by thee, that to this hour I bear the scar; I now know thee, thou clear spirit, and I now know that thy right worship is defiance. To neither love nor reverence wilt thou be kind; and e'en for hate thou canst but kill; and all are killed. No fearless fool now fronts thee. I own thy speechless, placeless power; but to the last gasp of my earthquake life will dispute its unconditional, unintegral mastery in me. In the midst of the personified impersonal, a personality stands here. Though but a point at best; whencesoe'er I came; wheresoe'er I go; yet while I earthly live, the queenly personality lives in me, and feels her royal rights. But war is pain, and hate is woe. Come in thy lowest form of love, and I will kneel and kiss thee; but at thy highest, come as mere supernal power; and though thou launchest navies of full-freighted worlds, there's that in here that still remains indifferent. Oh, thou clear spirit, of thy fire thou madest me, and like a true child of fire, I breathe it back to thee.
.jpg)
You may ask: why poetry? Of all the creative forms, why ought one write a poem? While all forms of creativity are worthy instruments against such a challenge, poetry is particularly worth defending, as it is irreducibly and messily human. Its meaning is inseparable from its form. It is not reduced to its output. It must be lived. It does not exist only to persuade or entertain; it may do both or neither. It does not ask of its writer fine-arts expertise or an index of accolades. It does not even ask to be good, as Mary Oliver reminds us. While AI lives in the centre, the statistically probable, poetry lives on the edge and finds comfort in the uncertain and in what resists resolution.This is why I ask of you: write poetry. Be a poet, for anyone who writes a poem is a poet. Write badly. Write slowly. Write dirty. Sit in discomfort. As poet Rainer Maria Rilke urges, we must “live the questions” rather than resolve them too quickly. Write for fun, not just for marks. Contradict yourself. It is quite fun; it is entirely awful; do it anyway. Reject convention and syntax? Break the line and glue it back together, never the same as before, always changed. Use an em-dash — mean it. Write from the margins, from which AI was never trained, and was never meant to see. Find the friction, make it your home too. It is within the mess and anarchy of poetry that innovation and creativity flourish.
If education is the transmission of civilization, we are unquestionably progressing. Civilization is not inherited; it has to be learned and earned by each generation anew; if the transmission should be interrupted for one century, civilization would die, and we should be savages again.
Agency isn't just the ability to choose among options or produce language on command. It's the felt experience of your own judgment. It comes from working with a thought long enough for something honest to emerge. With AI, we may still feel informed and productive while the deeper habits, dare I say "agency intelligence," begin to weaken.There is also an economic layer to this that carries psychological weight. A commodity can be priced, tiered, throttled, advertised, and differentiated. Once intelligence enters that framework, the old questions of human thinking begin to mix with the logic of the market. And I think that's exactly where Sam Altman and OpenAI sit—at least based on his earlier quote.The questions abound. Who gets the better model, the deeper reasoning, the larger context window, the more persistent memory? Who can afford a more capable form of synthetic cognition? That may sound like a technical issue, but it also changes how people imagine themselves in relation to thought. The self becomes less the source of intelligence and more the customer of cognitive support. To me, that's not a trivial shift. It encourages an insidious dependence with no track record.I suspect many people already feel this in everyday life. The moment before writing becomes prompting. The moment before reflection becomes querying. The unfinished thought no longer remains unfinished for long because something is always available to complete it—and at a cost. That is useful, and often impressive, but it is also worth watching carefully. The danger of AI isn't "simply" that it replaces human thought. It may be that it gradually makes thought feel less like something we do and more like something we access and buy.
We do nobody any favors by outsourcing the acts of discovery to machines. The suggestion runs counter to the arrogant belief that machines are the salvation of the human race, technology the light and wonder of the world. The prophecy is false, but the sales promotion is relentless. The data-mining dwarfs in Silicon Valley praise the glories of artificial intelligence and the internet of things, talk about attaching a human consciousness to a computer that lives forever. The Pentagon recruits drones to wage and lose its wars, Wall Street hires bots to mint and spoil its money; in the nation’s schools, the curriculum known as STEM (science, technology, engineering, math) sweeps the classrooms clean of improvised literary devices, downgrades the study of history and the humanities because they don’t get along well with tests administered by computers. At our colleges and universities, the oracles in residence complacently assume that man’s machines have vanquished nature, commodified the tribes of Paleolithic instinct, will construct Elon Musk’s stairway to the stars. The humanities they construe as exquisite ornaments, meant to be preserved together with the alumni slush fund and naming opportunities, in the vaults of the endowment. Their piety recalls the lines of Archibald MacLeish:
Freedom that was a thing to useThey’ve made a thing to saveAnd staked it in and fenced it roundLike a dead man’s grave.
To bury the humanities in tombs of precious marble is to deny ourselves the pleasure that is the love of learning and the play of the imagination, and to cheat ourselves of the inheritance alluded to in Goethe’s observation that he who cannot draw on three thousand years is living hand to mouth.
You can’t figure out why an AI is generating a hallucination by asking it. It is not conscious of its own processes. So if you ask it to explain itself, the AI will appear to give you the right answer, but it will have nothing to do with the process that generated the original result. The system has no way of explaining its decisions, or even knowing what those decisions were. Instead, it is (you guessed it) merely generating text that it thinks will make you happy in response to your query. LLMs are not generally optimized to say “I don’t know” when they don’t have enough information. Instead, they will give you an answer, expressing confidence.Ethan Mollick, from Co-Intelligence: Living and Working with AI
Anderson: Why isn't there just a huge risk of something truly terrible emerging?
Brockman: Well, I think all of these are questions of degree and scale and timing. And I think one thing people miss, too, is sort of the integration with the world is also this incredibly emergent, sort of, very powerful thing too. And so that's one of the reasons that we think it's so important to deploy incrementally.And so I think that what we kind of see right now, if you look at this talk, a lot of what I focus on is providing really high-quality feedback. Today, the tasks that we do, you can inspect them, right? It's very easy to look at that math problem and be like, "No, no, no, machine, seven was the correct answer." But even summarizing a book, like, that's a hard thing to supervise. Like, how do you know if this book summary is any good? You have to read the whole book. No one wants to do that. (nervous laughter, awkward silence)
We continue to refer to “the government,” yet that term refers to something bearing little resemblance to the entity described in our civics textbooks. Whatever we call the entity that controls it, the “world operating system” will seek to gather to itself the whole human field. This would bring to fruition what Hannah Arendt called “the rule of Nobody.” The Nobody is an entity that is not accountable and cannot be addressed.To see our way out of this will require revisiting the basic suppositions that got us to the present ...
Further ...
We are in for real political turmoil. The establishment is afraid, and rightly so, that the most likely alternative to technocratic rule would be something atavistic. If there is a silver lining in the current confusion, it may be this: Without a social class whose material interests are tied to the homogenizing and reductive metaphysics of technocracy, it may become possible once again to entertain big metaphysical questions. We may become open, as the West has not been for centuries, to truths made available to us in the tradition that runs from classical antiquity through the Hebrew bible and into the Christian teaching. According to this tradition, the human being is something doubly distinct: a natural kind that is oriented beyond itself, indeed beyond nature altogether. Human beings participate in something transcendent, in the image of which they were made. This truth provides a basis—I suspect it may be the only solid basis—on which human possibility may be defended against erasure.
