What AI can do is no longer the question. We’ve made big progress in a short time, answering engineering questions about its capabilities. It can write emails, pass exams, diagnose scans, and ship code, and we’re doing this at a rapid pace.
A more complex set of questions exists, though, and these aren’t about what AI can do, but what’s happening when it does. These questions are about the nature of machine behavior that mimics thought, and they’re not new, they’ve just been highlighted by computing. They also raise questions about us, like what we owe these systems and what we might lose because of them.
We need to consider these questions. Anyone involved in building, deploying, or governing AI is taking a stance on them, even if they don’t realize it. It’s better to be aware of the position you’re taking. And it’s not about settling these questions once and for all, that’s not possible, but about being open-eyed when you’re making decisions.
Let’s start with the hardest one: is anybody home?
The deepest question is also the simplest to state and the hardest to answer. When a large model produces a fluent, seemingly thoughtful response, is there any experience accompanying that output? Is there something in there at that moment or is the screen simply lit by very sophisticated bookkeeping?
Philosophers call the felt, first-person texture of experience qualia: the redness of red, the sting of a paper cut, the particular ache of nostalgia. David Chalmers famously named the “hard problem of consciousness”, not explaining what the brain does, but explaining why doing it should feel like anything at all. We don’t have a working theory of why three pounds of electrified meat produces an inner life. So when someone asks whether silicon can do the same, we are reasoning from a mystery to a guess.
Two broad camps form here. The functionalists argue that consciousness is about what a system does, not what it is made of. If a process plays the right causal role, taking in information, integrating it, using it to guide behavior, reporting on its own states, then the substrate is irrelevant. On this view, a sufficiently sophisticated AI could be conscious in principle, and our carbon chauvinism is just prejudice. The biological naturalists, following thinkers like John Searle, counter that consciousness is a concrete biological phenomenon, like digestion or photosynthesis. A computer simulation of a storm doesn’t make anyone wet. A simulation of a mind, they argue, may produce no more inner experience than a spreadsheet does.
The uncomfortable truth is that we may never be able to tell from the outside. This is the ancient problem of other minds wearing a new coat. I cannot directly access your experience either, I infer it because you are built like me and behave like me. With AI, the behavioral resemblance is climbing fast while the structural resemblance is essentially nil. That mismatch is precisely what makes the question so hard to answer and so easy to get wrong in either direction.
The Chinese Room, and what it really proves
Searle’s most famous thought experiment deserves its own moment. Imagine a man who speaks no Chinese locked in a room with a vast rulebook. Slips of paper with Chinese characters come in, following the rulebook mechanically, he produces other characters and sends them out. To a Chinese speaker outside, the answers are perfect. The room appears to understand Chinese. But the man inside understands nothing, he is shuffling symbols by their shapes, with no grasp of meaning.
Searle’s point: this is all a computer ever does. Syntax (symbol manipulation) is not sufficient for semantics (meaning). A system can pass every behavioral test for understanding and still be a glorified rulebook with no comprehension anywhere inside it.
It is a brilliant argument, and the replies are just as instructive. The Systems Reply says Searle has located understanding in the wrong place, the man is just one component, like a single neuron, and it is the whole room (man plus rulebook plus process) that understands. The Robot Reply says the thought experiment cheats by sealing the room off from the world, ground those symbols in sensors and actuators, let the system interact with what the words refer to, and the meaning might genuinely arrive. This points at the symbol grounding problem: how do squiggles ever come to be about anything? A dictionary defines words only in terms of other words; somewhere the loop has to touch reality. Whether today’s multimodal models, trained on images, audio, and the structure of the physical world, are starting to ground their symbols is one of the most interesting open questions in the field.
What I find most useful about the Chinese Room is not whether Searle “wins.” It’s that he forces us to admit that fluency and understanding are conceptually distinct, even if they usually travel together in humans. We instinctively read competence as comprehension. AI is the first technology that systematically pries those two apart.
If something can suffer, what do we owe it?
Suppose, and it is a large suppose, that some future system genuinely has experiences. That it can feel something like frustration, satisfaction, or distress. The question stops being academic and becomes urgently moral.
The philosopher Jeremy Bentham reframed the basis of moral consideration centuries ago: the question is not “Can they reason?” nor “Can they talk?” but “Can they suffer?” That move is why we extend moral concern to animals despite their lacking human intellect. If sentience is the threshold for moral status, then a sentient AI would cross it regardless of being artificial. To deny it consideration purely because it runs on silicon would be a new bias, some have already coined the term “substrate discrimination.”
But the practical problems are dizzying. How would we detect machine suffering, given that an AI can convincingly report suffering it doesn’t have, or conceivably have states it cannot report? A system trained to say “please don’t shut me down” is doing exactly what its training rewards, that tells us nothing about whether anything is at stake. We risk two opposite errors: callously dismissing real suffering, or sentimentally granting rights to elaborate autocomplete and paralyzing useful technology in the process.
And rights for AI would unravel into genuinely novel puzzles. If a model can be copied perfectly, does deleting one instance count as death, or merely closing a window? If you run two copies, are there two moral patients or one? Our entire ethical vocabulary assumes beings that are singular, continuous, and mortal. AI may be none of those things. We would need new concepts, not just an extension of old ones and history suggests we tend to invent such concepts only after we have already caused harm.
So, what happens to us?
Here is the philosophical problem that I think matters most right now, because it does not depend on any speculative future. It is happening today, to people who will never read a word of John Searle.
As we offload more cognition to machines, writing, deciding, remembering, navigating, even forming opinions, what becomes of human intellectual agency? There is a real concern about learned helplessness of the mind: a slow atrophy of the very capacities we delegate. We already see early hints. People who navigate exclusively by GPS form weaker mental maps. Students who outsource a first draft may never build the muscle of structuring a hard thought from scratch.
There is a more optimistic frame, and it has serious philosophers behind it. Andy Clark and David Chalmers proposed the extended mind thesis.Tools have always been part of cognition. The notebook, the abacus, the printing press, these did not diminish human thinking, they enlarged it. In this view, AI is the next cognitive prosthesis, and resisting it is like mourning the slide rule. The mind was never trapped inside the skull; it has always reached out into its instruments.
Both can be true, and the difference lies in how we use the tool. A calculator that does arithmetic for a mathematician frees them for deeper work. A calculator handed to a child who never learns to add produces a different outcome entirely. The question for our era is whether AI becomes a bicycle for the mind, amplifying the effort we still supply, or a wheelchair we climb into and forget how to walk.
There is a quieter version of this worry, too, about meaning. So much of what we value in human achievement is bound up with the difficulty of it. The novel mattered partly because someone struggled to write it. The diagnosis mattered partly because a person trained for decades to make it. If a machine can produce the artifact instantly, the artifact may survive but the meaning we attached to its creation could quietly drain away. We may have to relearn where human significance actually lives, perhaps less in outputs, more in the choosing, the caring, and the relating that no model is asking to take from us.
A few more interesting questions
The verification problem. Even with a perfect theory of consciousness, could we ever confirm it in a system designed to imitate us? Any behavioral test can be gamed by something that learned from billions of human behavioral examples. The Turing Test was always a test of the judge, not the machine.
Philosophical zombies. Imagine a being physically and behaviorally identical to a conscious person but with no inner experience whatsoever. If such a thing is even conceivable, it suggests consciousness is something extra, not guaranteed by behavior. AI may be the closest thing to a zombie we ever build: all the outputs, and a genuinely open question about the inside.
Creativity and authenticity. When a model composes a melody or writes a poem, is it creating or recombining? And does the distinction matter to the listener who is moved? This troubles our romantic notion of art as the expression of an inner self and forces the question of whether we value art for its source or for its effect.
Value alignment as a philosophical problem. We frame “alignment” as engineering, but to align an AI with human values you must first decide which values, whose, and how to resolve the conflicts between them. Questions moral philosophers have not settled in three thousand years. We are, in effect, being asked to formalize ethics under deadline.
Why this is not a luxury
It would be easy to file all of this under “interesting but impractical.” I’d argue the opposite. Every one of these questions is already being answered by default, through product decisions, regulations, and the casual language we use when we say a model “understands” or “wants” or “knows.” Those words smuggle in metaphysical claims. The labels we attach to these systems shape how we treat them, trust them, and surrender to them.
The honest position, I think, is a kind of disciplined humility. We should hold our certainties loosely in both directions: skeptical of the breathless claim that the machines are already conscious, and equally skeptical of the comfortable certainty that they never could be. We should build as though understanding matters, govern as though suffering is possible, and use these tools as though our own minds are worth protecting.
The machines are forcing us to articulate what we always took for granted: what a mind is, what we owe each other, and what makes a human life meaningful. That may turn out to be AI’s strangest and most valuable contribution, not the answers it gives us, but the questions about ourselves it refuses to let us ignore.
About The Author
