A deployed language model’s weights usually stop changing. The surrounding AI system often does not.
Within a context window—the information available for one response—a model can immediately adapt to a useful idea, an unfamiliar API, or a correction. The adaptation ends with the context unless the surrounding system integrates it into durable state.
An AI can accumulate far more experience than it can consult for a single decision. Continual learning maintains a bounded, coherent, and revisable center that turns selected experience into reliable changes in future behavior.
A bounded center
An archive can hold conversations, documents, tool results, and observations accumulated over a system’s lifetime. Context remains finite, and rereading the entire history for every decision is too slow and expensive. Archival storage therefore provides potential knowledge. The system must retrieve and interpret an item before it can affect a decision.
John Searle’s Chinese-room thought experiment, examined in the earlier essay “We Are Building the Chinese Room”, illustrates the gap between lookup and integration. It imagines a person unfamiliar with Chinese using an enormous rulebook to produce convincing Chinese replies. The room appears fluent from outside because the person follows the instructions for each input. For this argument, consider the limiting version in which the rulebook grows through case-specific instructions. It covers more familiar cases while its organization for interpreting novel situations stays fixed. Continual learning, as I use it here, concerns experience integrated into a reusable organization, whether that organization lives in memory, software, or weights.
A system that uses its history repeatedly needs a compact working representation of what that history has taught it. I use core memory to mean the bounded set of beliefs, preferences, procedures, and unresolved questions that the system uses and revises across sessions. It extends the idea of a computer system’s working set—the information repeatedly needed by an active process—by persisting across sessions and shaping later behavior. Its cognitive index maps important ideas to their relationships, supporting evidence, and revision status.
Integration concerns persistent change in how the system handles later situations. A phone number may shape every call in which it is retrieved while remaining a reference. Core memory has a wider effect: it changes what the system notices, how it interprets new evidence, what it retrieves, or how it updates itself across situations. An archive primarily preserves evidence for later retrieval. Core memory is designed to influence interpretation and action routinely, so removing it changes the system’s learned disposition. Core memory can therefore be physically external yet functionally part of the AI.
Bounding core memory is one condition for practical human oversight. Core memory should be compact and legible enough for reviewers to inspect what it contains, what changed, which evidence motivated the change, and which behavior each entry is intended to influence. Versioned diffs and provenance make this monitoring incremental: reviewers can focus on each update instead of rereading the archive. Review capacity becomes a system constraint, because timely auditing requires the rate of change to stay within what people and monitoring tools can examine.
Where learning can live
The behavior-shaping center can span core memory, tools, policies, learned components, and model weights. Core memory supplies compact, explicit state within this center. A system may also maintain compiled knowledge: reusable syntheses of raw sources that capture concepts, relationships, and contradictions. Andrej Karpathy’s LLM Wiki gives a concrete pattern: preserve raw sources, maintain a structured wiki derived from them, and consult a compact index before opening the relevant pages. At greater scale, that index can become hierarchical or searchable.
Experience can move among these representations. An episode may remain raw, be summarized, enter core memory, become a tool or policy, or be consolidated into a learned component or model weights. When knowledge becomes outdated, the system can revise its explicit representations and propagate the correction to their dependents. Changing knowledge already distributed through model weights requires a targeted update, unlearning, or retraining. The appropriate route depends on how often the knowledge is used, how quickly it changes, how precisely its origin must be tracked, and how easily it must be reversed.
Suppose an agent discovers through documentation and failed calls that an API now requires a different authentication flow. The documentation and traces belong in the archive. Compiled knowledge summarizes the change and links to that evidence. Core memory records the current rule and where to verify it, while a tool wrapper may enforce it automatically. If the API changes again, the system can revise the rule while retaining its history.
These representations produce different access and update patterns. Archives favor inexpensive capacity and search. Compiled knowledge needs dependency tracking so changes in a source can trigger revisions to affected summaries. Model weights are read at high bandwidth for every response and updated infrequently during use. Core memory is both continually revised and security-sensitive, producing small, irregular, time-sensitive reads and authenticated writes. It also requires persistence and version history. It might live in a database, persistent memory near the processors, a dedicated device, or a combination of them. A learned component would need versioned, restorable state plus an explicit update record to remain inspectable and reversible. These requirements make core memory a distinct hardware workload.
Learning through parameter updates creates the stability–plasticity dilemma because neural networks distribute capabilities across many interacting weights. A learner needs enough plasticity to absorb new experience and enough stability to preserve earlier capabilities. Excess plasticity can cause catastrophic forgetting, the loss of earlier capabilities after new training.
Explicit memory faces a corresponding tension: a rigid cognitive index becomes stale, while an overly mutable one loses useful structure. Weight updates are harder to inspect because their effects are distributed across a broad model. Methods such as elastic weight consolidation try to protect parameters important to earlier capabilities. Reliable continual learning across these representations remains an open problem.
Learning should leave a trace
Continual updates need an audit trail. I use learning-aware for a system whose durable changes leave records that observers can independently inspect, test, and trace. These records must exist independently of the model’s verbal explanation, which may be incomplete or inaccurate.
What changed, and from where? Every update needs provenance: a record of its source, evidence, time, and update mechanism. Suppose a post-deployment learner adopts a proof technique after reading a paper. The proof shows the technique; the update record links it to the paper, the time it was integrated, and its later uses.
How hard was it to integrate? Assimilation cost includes how many examples and updates were required, how much existing capability transferred, and how much reorganization occurred. Retention and transfer tests then assess whether the change generalized beyond the recent data that produced it.
What else changed? Earlier capabilities should be tested immediately after an update and again later. Work on the “stability gap” shows why: performance on earlier tasks may collapse immediately after new learning and recover later.
Can the update be revised? When evidence changes, the system should retain that it once believed A, that evidence B triggered an update, and that the new conclusion holds under conditions C. This history supports diagnosis. Correction and rollback additionally require restorable state and dependency tracking.
Fine-grained records and staged updates improve behavioral attribution by making controlled before-and-after comparisons possible. Observers can test whether new behavior or regressions appeared with a particular update and preserve the result for later experiments. The internal mechanism may remain obscure. Large batch updates make attribution harder because they combine many changes at once.
A practical update protocol has four steps:
- Record a baseline of the system’s state and relevant capabilities.
- Introduce a bounded learning experience.
- Measure intended and unintended changes.
- Accept, quarantine, or reverse the update, and record the outcome.
False or malicious experience can also become durable. Post-deployment learning therefore requires source validation, authenticated origin where available, staged consolidation, access control, monitoring, and restorable state.
Evaluation across a lifetime
Most AI benchmarks evaluate a fixed model on a fixed dataset. Measuring continual learning requires following a system through new skills, corrections, environmental changes, and malicious data. An endpoint score cannot show how acquisition, forgetting, and recovery unfolded.
A useful evaluation should cover four families of questions:
- Learning quality: How quickly does the system acquire, retain, transfer, and revise knowledge?
- Integration: Does learned state reliably shape behavior, and can the system use its cognitive index to reach relevant evidence without repeating the same discovery?
- Accountability: Can people efficiently review core memory, trace behavior, detect collateral changes, isolate poisoned experience, and reverse harmful updates?
- Economy: How much context, retrieval, storage, compute, data, time and human review effort does each durable improvement require?
A system may perform well after ten updates and fail after ten thousand. Its cognitive index may grow stale, and core memory may accumulate inconsistent entries. Later updates may depend on earlier ones, so rollback can damage dependent knowledge. Replaying selected old examples can resist forgetting, but the storage and computation required can grow with the history retained.
A coherent center through time
Continual learning is a process of selective integration. A system maintains its center by deciding which experiences remain archival, which enter core memory, which become tools or policies, and which are consolidated into learned components or weights. Update records preserve evidence and dependencies so people can inspect and challenge those decisions; versioned, restorable state makes reversal possible.