John Searle’s Chinese room contains a person who does not know Chinese, a set of formal instructions, and working notes. Chinese characters arrive through an opening. The person identifies them by shape, applies the instructions, and returns other characters. Speakers outside judge the answers fluent; the person experiences meaningless marks.
Searle introduced the case in “Minds, Brains, and Programs” to challenge strong AI, the claim that instantiating the right program is sufficient for understanding. His example drew on Roger Schank’s story-understanding programs. The thought experiment stipulated answers indistinguishable from a native speaker’s. Later versions extend the setup to open-ended conversation, a breadth symbolic AI did not achieve.
LLM agents now approximate this organization. A language model proposes responses and actions. A harness assembles context, exposes tools, preserves state, checks results, and calls the model again. Because these components can be changed independently, we can test which configurations support particular capacities.
Where the competence lives
The systems reply attributes understanding to the combination of person, instructions, notes, and room. Searle answered that the person could memorize the whole procedure and continue to experience no Chinese meaning.
The exchange grants successful linguistic performance before asking where that capacity resides. Open-ended conversation requires context, generalization, relevance judgments, and recovery from misunderstanding. If the operator performs fixed shape-matching steps, the rules and state must determine each transition. If the operator interprets novel instructions, selects relevant facts, or constructs intermediate goals, those capacities reside partly in the operator. The required organization must appear in the operator, rules, memory, loop, or their coordination.
A working implementation exposes that allocation to component-level experiments.
The LLM and the harness
At the agent level, the LLM occupies the operator’s role. It receives the state assembled for one step and proposes text, a tool call, a question, or a final answer. Its learned representations support language-conditioned action selection across tasks.
The harness supplies the room and its procedural rules. It chooses instructions and memories, defines available tools, preserves state, validates actions, and decides when to retry or stop. The loop applies those rules: assemble context, ask for an action, execute it, record the result, and repeat.
A user message corresponds to the slip through the opening; context and task state supply the working notes; retrieval and persistent memory supply the filing cabinets. This is an inspectable analog of the room’s organization.
The mapping is level-dependent. Within model inference, architecture and weights encode a transition function while software and hardware execute it. Treating the whole model call as the operator isolates the agent-level question: what capacities belong to the model, and what capacities depend on the process around it?
Training prevents us from stipulating Searle’s premise that the operator is linguistically ignorant. An LLM enters the harness with dispositions learned from text. The Chinese-room question moves inside the agent: do those dispositions amount to understanding?
Two questions about understanding
An LLM is a learned parameter system executed by a program. Its parameters can encode many distinctions and relationships. Deciding whether those encodings have meaning requires a criterion of understanding.
Emily Bender and Alexander Koller argue in “Climbing towards NLU” that form alone provides no route to communicative intentions or referents. Geoffrey Hinton takes learned representation and generalization as evidence of understanding. In a Nobel Prize conversation, he describes LLMs as our best current computational model of human linguistic processing and treats errors as failures on particular cases.
These views become easier to compare after separating capacities. Kyle Mahowald and collaborators distinguish formal linguistic competence from functional linguistic competence. The first covers grammar and patterns within language. The second recruits world knowledge, reasoning, situation models, and social inference. Yann LeCun’s architecture for autonomous machine intelligence assigns world modeling, memory, perception, action, reasoning, and planning central roles. A harness can add memory and action channels without establishing that the model learned a world model.
The evidence can include internal structure as well as behavior. Ellie Pavlick argues for empirical study of representations and processing, including whether language produced by grounded speakers can transmit indirect relations to the world.
Subjective experience requires another claim. David Chalmers identifies in “Could a Large Language Model Be Conscious?” features the systems he assesses may lack under prominent theories of consciousness. Representing a distinction and experiencing its meaning call for different evidence.
The model question therefore divides into linguistic representation, generalization, grounding, world modeling, and experience. The harness creates a second question: can the organized loop have a capacity absent from one model call?
Software makes that question testable. Hold the model fixed and remove retrieval, tools, memory, retries, or verification. Hold the harness fixed and replace the model. In a coding agent, for example, the model proposes a patch; the loop applies it, runs tests, returns failures, and preserves the files for another attempt. Repair depends on the proposals, external state, feedback, and repetition policy. These interventions identify dependencies in the tested configuration, though interacting or substitutable components may prevent a unique attribution.
Move the wall
The physical room shapes the original intuition. A conscious person occupies the center, the instructions appear as furniture, and the wall suggests a boundary around the relevant system. We receive first-person evidence from the person and behavioral evidence from the whole.
Shrink the room while preserving every state transition and information channel. A building becomes a booth, then a cabinet holding the rules and notes. The operator sits outside and reaches in. Continue until the room becomes a box in the operator’s hands.
The causal process stays fixed as the wall crosses the operator’s body. The person first appears to inhabit the system and later appears to be an external mechanism that updates it. Spatial enclosure therefore does no work in locating the competence.
Current agent systems already have this small-room form. A compact harness can call a much larger model running elsewhere:
next_action = model(current_state)
The harness treats the model as an external state-transition service; the model receives a local world assembled by the harness. Their interaction supports the agent’s behavior. In Searle’s case, the clerk’s ignorance establishes a fact about that component. A claim about the larger system needs evidence at the larger boundary.
Searle’s internalization reply already removes the enclosure, so shrinking the room addresses the vignette’s intuitive presentation. It does not refute his claim about the sufficiency of programs.
Memory and tools
Agent memory applies the same boundary test. A model’s weights can remain frozen while retrieval, persistent state, or an explicit tool changes future behavior. A resource participates in a capability when it is readily available, routinely consulted, and able to change perception, decisions, or future action. Storage alone supplies potential access.
Clark and Chalmers argue in “The Extended Mind” that a reliably available external resource can play a role associated with internal memory. An LLM harness provides a concrete test case by controlling what enters working context and what persists.
Tools add channels to nonlinguistic observations and action consequences: images, measurements, execution failures, and results of interventions. These reach the model through mediated representations. The symbol-grounding problem remains because a channel alone does not explain meaning, but it lets us test which outputs and actions change under controlled observations.
Culture as a room
Culture resembles the room’s rule system at a larger scale. Languages, mathematical notations, legal procedures, and scientific methods supply inherited symbols and rules developed across many people. The analogy concerns the allocation of competence. Culture is distributed, revised through use, and connected to the practices of people who perceive and act in the world.
Education moves some of this organization into an individual. A student may first repeat a phrase or apply an algebraic rule by procedure. Practice and correction connect the rule to situations, consequences, and recognizable errors. The resulting ability can become a disposition the student exercises without consulting the original source. Teachers and earlier texts then belong to the ability’s causal history rather than to every later act that uses it.
Other cultural resources remain active. Books, colleagues, instruments, software, standards, and institutions can supply information, constraints, and correction during a task. LLMs exhibit the same temporal division. Training data shapes the model’s weights; during deployment, system prompts, retrieved documents, tool schemas, tests, and human feedback can place selected cultural practices inside the agent loop. Their contribution can be tested by removing or changing them.
This leads to debates over whether intelligence belongs to individuals or to an entire society. Attribution is clearest for a bounded social system with a defined task. Edwin Hutchins’s Cognition in the Wild analyzes ship navigation across sailors, instruments, procedures, and representations. Individual sailors possess relevant knowledge; the coordinated system completes a navigation process that no sailor performs alone. This supports an attribution of distributed competence to the group. Group agency and collective experience require their own evidence.
What building the room changes
An attribution of understanding needs three specifications: the capacity, the system boundary, and the evidence. The capacity might be grammar, reference, generalization, world modeling, planning, learning, or subjective experience. The boundary might enclose model parameters, one inference, the agent loop, the product, or the operating organization. Evidence might come from behavior across new cases, interventions on internal states, persistent changes in future behavior, or a theory connecting physical processes to experience.
Building an LLM agent makes competence and integration available to component-level experiments. We can replace the model, clear the memory, restrict the tools, alter the control procedure, or remove cultural resources from the context. Changes in behavior locate dependencies in a component or in its coordination with others. Grounding and subjective experience require criteria that connect those dependencies to meaning or experience.
The shrinking-room exercise explains why the boundary must be stated. Moving the wall across the operator changes the apparent owner of the behavior while preserving the causal process. A clerk’s report directly concerns the clerk; an agent’s end-to-end performance directly concerns the organized loop; a team’s achievement directly concerns the coordinated team. Moving from one attribution to another requires an account of the larger system.
Current LLM agents realize enough of the Chinese room’s stipulated organization to make these distinctions operational. Searle’s question now applies separately to the learned operator, the agent loop, and any cultural system active during the task. An answer at one boundary does not transfer automatically to another; the transfer is itself a claim about how the larger system produces the capacity.