Thinking Source

Continual Learning: What Becomes Part of the Learner?

2026-07-03T09:00:00-07:00

A deployed language model’s weights usually stop changing. The surrounding AI system often does not.

Within a context window—the information available for one response—a model can immediately adapt to a useful idea, an unfamiliar API, or a correction. The adaptation ends with the context unless the surrounding system integrates it into durable state.

An AI can accumulate far more experience than it can consult for a single decision. Continual learning maintains a bounded, coherent, and revisable center that turns selected experience into reliable changes in future behavior.

A bounded center

An archive can hold conversations, documents, tool results, and observations accumulated over a system’s lifetime. Context remains finite, and rereading the entire history for every decision is too slow and expensive. Archival storage therefore provides potential knowledge. The system must retrieve and interpret an item before it can affect a decision.

John Searle’s Chinese-room thought experiment, examined in the earlier essay “We Are Building the Chinese Room”, illustrates the gap between lookup and integration. It imagines a person unfamiliar with Chinese using an enormous rulebook to produce convincing Chinese replies. The room appears fluent from outside because the person follows the instructions for each input. For this argument, consider the limiting version in which the rulebook grows through case-specific instructions. It covers more familiar cases while its organization for interpreting novel situations stays fixed. Continual learning, as I use it here, concerns experience integrated into a reusable organization, whether that organization lives in memory, software, or weights.

A system that uses its history repeatedly needs a compact working representation of what that history has taught it. I use core memory to mean the bounded set of beliefs, preferences, procedures, and unresolved questions that the system uses and revises across sessions. It extends the idea of a computer system’s working set—the information repeatedly needed by an active process—by persisting across sessions and shaping later behavior. Its cognitive index maps important ideas to their relationships, supporting evidence, and revision status.

Integration concerns persistent change in how the system handles later situations. A phone number may shape every call in which it is retrieved while remaining a reference. Core memory has a wider effect: it changes what the system notices, how it interprets new evidence, what it retrieves, or how it updates itself across situations. An archive primarily preserves evidence for later retrieval. Core memory is designed to influence interpretation and action routinely, so removing it changes the system’s learned disposition. Core memory can therefore be physically external yet functionally part of the AI.

Bounding core memory is one condition for practical human oversight. Core memory should be compact and legible enough for reviewers to inspect what it contains, what changed, which evidence motivated the change, and which behavior each entry is intended to influence. Versioned diffs and provenance make this monitoring incremental: reviewers can focus on each update instead of rereading the archive. Review capacity becomes a system constraint, because timely auditing requires the rate of change to stay within what people and monitoring tools can examine.

Where learning can live

The behavior-shaping center can span core memory, tools, policies, learned components, and model weights. Core memory supplies compact, explicit state within this center. A system may also maintain compiled knowledge: reusable syntheses of raw sources that capture concepts, relationships, and contradictions. Andrej Karpathy’s LLM Wiki gives a concrete pattern: preserve raw sources, maintain a structured wiki derived from them, and consult a compact index before opening the relevant pages. At greater scale, that index can become hierarchical or searchable.

Experience can move among these representations. An episode may remain raw, be summarized, enter core memory, become a tool or policy, or be consolidated into a learned component or model weights. When knowledge becomes outdated, the system can revise its explicit representations and propagate the correction to their dependents. Changing knowledge already distributed through model weights requires a targeted update, unlearning, or retraining. The appropriate route depends on how often the knowledge is used, how quickly it changes, how precisely its origin must be tracked, and how easily it must be reversed.

Suppose an agent discovers through documentation and failed calls that an API now requires a different authentication flow. The documentation and traces belong in the archive. Compiled knowledge summarizes the change and links to that evidence. Core memory records the current rule and where to verify it, while a tool wrapper may enforce it automatically. If the API changes again, the system can revise the rule while retaining its history.

These representations produce different access and update patterns. Archives favor inexpensive capacity and search. Compiled knowledge needs dependency tracking so changes in a source can trigger revisions to affected summaries. Model weights are read at high bandwidth for every response and updated infrequently during use. Core memory is both continually revised and security-sensitive, producing small, irregular, time-sensitive reads and authenticated writes. It also requires persistence and version history. It might live in a database, persistent memory near the processors, a dedicated device, or a combination of them. A learned component would need versioned, restorable state plus an explicit update record to remain inspectable and reversible. These requirements make core memory a distinct hardware workload.

Learning through parameter updates creates the stability–plasticity dilemma because neural networks distribute capabilities across many interacting weights. A learner needs enough plasticity to absorb new experience and enough stability to preserve earlier capabilities. Excess plasticity can cause catastrophic forgetting, the loss of earlier capabilities after new training.

Explicit memory faces a corresponding tension: a rigid cognitive index becomes stale, while an overly mutable one loses useful structure. Weight updates are harder to inspect because their effects are distributed across a broad model. Methods such as elastic weight consolidation try to protect parameters important to earlier capabilities. Reliable continual learning across these representations remains an open problem.

Learning should leave a trace

Continual updates need an audit trail. I use learning-aware for a system whose durable changes leave records that observers can independently inspect, test, and trace. These records must exist independently of the model’s verbal explanation, which may be incomplete or inaccurate.

What changed, and from where? Every update needs provenance: a record of its source, evidence, time, and update mechanism. Suppose a post-deployment learner adopts a proof technique after reading a paper. The proof shows the technique; the update record links it to the paper, the time it was integrated, and its later uses.

How hard was it to integrate? Assimilation cost includes how many examples and updates were required, how much existing capability transferred, and how much reorganization occurred. Retention and transfer tests then assess whether the change generalized beyond the recent data that produced it.

What else changed? Earlier capabilities should be tested immediately after an update and again later. Work on the “stability gap” shows why: performance on earlier tasks may collapse immediately after new learning and recover later.

Can the update be revised? When evidence changes, the system should retain that it once believed A, that evidence B triggered an update, and that the new conclusion holds under conditions C. This history supports diagnosis. Correction and rollback additionally require restorable state and dependency tracking.

Fine-grained records and staged updates improve behavioral attribution by making controlled before-and-after comparisons possible. Observers can test whether new behavior or regressions appeared with a particular update and preserve the result for later experiments. The internal mechanism may remain obscure. Large batch updates make attribution harder because they combine many changes at once.

A practical update protocol has four steps:

Record a baseline of the system’s state and relevant capabilities.
Introduce a bounded learning experience.
Measure intended and unintended changes.
Accept, quarantine, or reverse the update, and record the outcome.

False or malicious experience can also become durable. Post-deployment learning therefore requires source validation, authenticated origin where available, staged consolidation, access control, monitoring, and restorable state.

Evaluation across a lifetime

Most AI benchmarks evaluate a fixed model on a fixed dataset. Measuring continual learning requires following a system through new skills, corrections, environmental changes, and malicious data. An endpoint score cannot show how acquisition, forgetting, and recovery unfolded.

A useful evaluation should cover four families of questions:

Learning quality: How quickly does the system acquire, retain, transfer, and revise knowledge?
Integration: Does learned state reliably shape behavior, and can the system use its cognitive index to reach relevant evidence without repeating the same discovery?
Accountability: Can people efficiently review core memory, trace behavior, detect collateral changes, isolate poisoned experience, and reverse harmful updates?
Economy: How much context, retrieval, storage, compute, data, time and human review effort does each durable improvement require?

A system may perform well after ten updates and fail after ten thousand. Its cognitive index may grow stale, and core memory may accumulate inconsistent entries. Later updates may depend on earlier ones, so rollback can damage dependent knowledge. Replaying selected old examples can resist forgetting, but the storage and computation required can grow with the history retained.

A coherent center through time

Continual learning is a process of selective integration. A system maintains its center by deciding which experiences remain archival, which enter core memory, which become tools or policies, and which are consolidated into learned components or weights. Update records preserve evidence and dependencies so people can inspect and challenge those decisions; versioned, restorable state makes reversal possible.

We Are Building the Chinese Room

2026-06-11T12:00:00-07:00

John Searle’s Chinese room contains a person who does not know Chinese, a set of formal instructions, and working notes. Chinese characters arrive through an opening. The person identifies them by shape, applies the instructions, and returns other characters. Speakers outside judge the answers fluent; the person experiences meaningless marks.

Searle introduced the case in “Minds, Brains, and Programs” to challenge strong AI, the claim that instantiating the right program is sufficient for understanding. His example drew on Roger Schank’s story-understanding programs. The thought experiment stipulated answers indistinguishable from a native speaker’s. Later versions extend the setup to open-ended conversation, a breadth symbolic AI did not achieve.

LLM agents now approximate this organization. A language model proposes responses and actions. A harness assembles context, exposes tools, preserves state, checks results, and calls the model again. Because these components can be changed independently, we can test which configurations support particular capacities.

Where the competence lives

The systems reply attributes understanding to the combination of person, instructions, notes, and room. Searle answered that the person could memorize the whole procedure and continue to experience no Chinese meaning.

The exchange grants successful linguistic performance before asking where that capacity resides. Open-ended conversation requires context, generalization, relevance judgments, and recovery from misunderstanding. If the operator performs fixed shape-matching steps, the rules and state must determine each transition. If the operator interprets novel instructions, selects relevant facts, or constructs intermediate goals, those capacities reside partly in the operator. The required organization must appear in the operator, rules, memory, loop, or their coordination.

A working implementation exposes that allocation to component-level experiments.

The LLM and the harness

At the agent level, the LLM occupies the operator’s role. It receives the state assembled for one step and proposes text, a tool call, a question, or a final answer. Its learned representations support language-conditioned action selection across tasks.

The harness supplies the room and its procedural rules. It chooses instructions and memories, defines available tools, preserves state, validates actions, and decides when to retry or stop. The loop applies those rules: assemble context, ask for an action, execute it, record the result, and repeat.

A user message corresponds to the slip through the opening; context and task state supply the working notes; retrieval and persistent memory supply the filing cabinets. This is an inspectable analog of the room’s organization.

The mapping is level-dependent. Within model inference, architecture and weights encode a transition function while software and hardware execute it. Treating the whole model call as the operator isolates the agent-level question: what capacities belong to the model, and what capacities depend on the process around it?

Training prevents us from stipulating Searle’s premise that the operator is linguistically ignorant. An LLM enters the harness with dispositions learned from text. The Chinese-room question moves inside the agent: do those dispositions amount to understanding?

Two questions about understanding

An LLM is a learned parameter system executed by a program. Its parameters can encode many distinctions and relationships. Deciding whether those encodings have meaning requires a criterion of understanding.

Emily Bender and Alexander Koller argue in “Climbing towards NLU” that form alone provides no route to communicative intentions or referents. Geoffrey Hinton takes learned representation and generalization as evidence of understanding. In a Nobel Prize conversation, he describes LLMs as our best current computational model of human linguistic processing and treats errors as failures on particular cases.

These views become easier to compare after separating capacities. Kyle Mahowald and collaborators distinguish formal linguistic competence from functional linguistic competence. The first covers grammar and patterns within language. The second recruits world knowledge, reasoning, situation models, and social inference. Yann LeCun’s architecture for autonomous machine intelligence assigns world modeling, memory, perception, action, reasoning, and planning central roles. A harness can add memory and action channels without establishing that the model learned a world model.

The evidence can include internal structure as well as behavior. Ellie Pavlick argues for empirical study of representations and processing, including whether language produced by grounded speakers can transmit indirect relations to the world.

Subjective experience requires another claim. David Chalmers identifies in “Could a Large Language Model Be Conscious?” features the systems he assesses may lack under prominent theories of consciousness. Representing a distinction and experiencing its meaning call for different evidence.

The model question therefore divides into linguistic representation, generalization, grounding, world modeling, and experience. The harness creates a second question: can the organized loop have a capacity absent from one model call?

Software makes that question testable. Hold the model fixed and remove retrieval, tools, memory, retries, or verification. Hold the harness fixed and replace the model. In a coding agent, for example, the model proposes a patch; the loop applies it, runs tests, returns failures, and preserves the files for another attempt. Repair depends on the proposals, external state, feedback, and repetition policy. These interventions identify dependencies in the tested configuration, though interacting or substitutable components may prevent a unique attribution.

Move the wall

The physical room shapes the original intuition. A conscious person occupies the center, the instructions appear as furniture, and the wall suggests a boundary around the relevant system. We receive first-person evidence from the person and behavioral evidence from the whole.

Shrink the room while preserving every state transition and information channel. A building becomes a booth, then a cabinet holding the rules and notes. The operator sits outside and reaches in. Continue until the room becomes a box in the operator’s hands.

The causal process stays fixed as the wall crosses the operator’s body. The person first appears to inhabit the system and later appears to be an external mechanism that updates it. Spatial enclosure therefore does no work in locating the competence.

Current agent systems already have this small-room form. A compact harness can call a much larger model running elsewhere:

next_action = model(current_state)

The harness treats the model as an external state-transition service; the model receives a local world assembled by the harness. Their interaction supports the agent’s behavior. In Searle’s case, the clerk’s ignorance establishes a fact about that component. A claim about the larger system needs evidence at the larger boundary.

Searle’s internalization reply already removes the enclosure, so shrinking the room addresses the vignette’s intuitive presentation. It does not refute his claim about the sufficiency of programs.

Memory and tools

Agent memory applies the same boundary test. A model’s weights can remain frozen while retrieval, persistent state, or an explicit tool changes future behavior. A resource participates in a capability when it is readily available, routinely consulted, and able to change perception, decisions, or future action. Storage alone supplies potential access.

Clark and Chalmers argue in “The Extended Mind” that a reliably available external resource can play a role associated with internal memory. An LLM harness provides a concrete test case by controlling what enters working context and what persists.

Tools add channels to nonlinguistic observations and action consequences: images, measurements, execution failures, and results of interventions. These reach the model through mediated representations. The symbol-grounding problem remains because a channel alone does not explain meaning, but it lets us test which outputs and actions change under controlled observations.

Culture as a room

Culture resembles the room’s rule system at a larger scale. Languages, mathematical notations, legal procedures, and scientific methods supply inherited symbols and rules developed across many people. The analogy concerns the allocation of competence. Culture is distributed, revised through use, and connected to the practices of people who perceive and act in the world.

Education moves some of this organization into an individual. A student may first repeat a phrase or apply an algebraic rule by procedure. Practice and correction connect the rule to situations, consequences, and recognizable errors. The resulting ability can become a disposition the student exercises without consulting the original source. Teachers and earlier texts then belong to the ability’s causal history rather than to every later act that uses it.

Other cultural resources remain active. Books, colleagues, instruments, software, standards, and institutions can supply information, constraints, and correction during a task. LLMs exhibit the same temporal division. Training data shapes the model’s weights; during deployment, system prompts, retrieved documents, tool schemas, tests, and human feedback can place selected cultural practices inside the agent loop. Their contribution can be tested by removing or changing them.

This leads to debates over whether intelligence belongs to individuals or to an entire society. Attribution is clearest for a bounded social system with a defined task. Edwin Hutchins’s Cognition in the Wild analyzes ship navigation across sailors, instruments, procedures, and representations. Individual sailors possess relevant knowledge; the coordinated system completes a navigation process that no sailor performs alone. This supports an attribution of distributed competence to the group. Group agency and collective experience require their own evidence.

What building the room changes

An attribution of understanding needs three specifications: the capacity, the system boundary, and the evidence. The capacity might be grammar, reference, generalization, world modeling, planning, learning, or subjective experience. The boundary might enclose model parameters, one inference, the agent loop, the product, or the operating organization. Evidence might come from behavior across new cases, interventions on internal states, persistent changes in future behavior, or a theory connecting physical processes to experience.

Building an LLM agent makes competence and integration available to component-level experiments. We can replace the model, clear the memory, restrict the tools, alter the control procedure, or remove cultural resources from the context. Changes in behavior locate dependencies in a component or in its coordination with others. Grounding and subjective experience require criteria that connect those dependencies to meaning or experience.

The shrinking-room exercise explains why the boundary must be stated. Moving the wall across the operator changes the apparent owner of the behavior while preserving the causal process. A clerk’s report directly concerns the clerk; an agent’s end-to-end performance directly concerns the organized loop; a team’s achievement directly concerns the coordinated team. Moving from one attribution to another requires an account of the larger system.

Current LLM agents realize enough of the Chinese room’s stipulated organization to make these distinctions operational. Searle’s question now applies separately to the learned operator, the agent loop, and any cultural system active during the task. An answer at one boundary does not transfer automatically to another; the transfer is itself a claim about how the larger system produces the capacity.