Four carriers, five duties, five operations: what memory means in autoregressive video generation
A deep dive into the first survey of memory in autoregressive video generation, which sorts the field into four carrier families, five preservation duties and five operations, and shows why a benchmark score counts as memory evidence only when the query cannot be answered without it.
An autoregressive video model builds a video the way a language model builds a sentence: one unit at a time, each unit conditioned on everything produced before it. The approach carries a few seconds comfortably. It strains at minutes, because the conditioning history cannot keep growing. Self-attention costs grow quadratically with sequence length, and accelerator memory, cache storage and latency bound what a running model can hold. Practical systems therefore condition on a bounded window of recent units, and a frame, clip or latent block leaves that window long before its relevance expires. 1
The Past Frames the Future: Memory for Autoregressive Video Generation treats what happens after that point as a memory problem. 2 Harold Haodong Chen, Rongjin Guo, Disen Lan, Wen-Jie Shu and Hongfei Zhang wrote it with twenty co-authors from HKUST, City University of Hong Kong, Fudan University, Carnegie Mellon, NYU, NUS, Georgia Tech, Peking University, MBZUAI, NVIDIA and four other institutions, and posted the 110-page survey to arXiv on 23 September 2026. 2 Hugging Face's paper page ranked it second among the papers of 24 September, and its companion repository of methods and benchmarks has collected 62 stars. 34 In a comment on that page the authors describe it as the first survey of memory mechanisms in autoregressive video generation, a claim the field's earlier surveys support: work on video diffusion and on world models covers the broader paradigm and leaves memory as a footnote under retrieval or long-context conditioning. 3
What counts as memory
The survey's first move is to make the word testable. Memory is a persistent state, maintained across autoregressive steps, that conditions later generation after the evidence it came from has left the immediate context. Operationally, a state counts as memory when removing or modifying it changes what the model generates later. 5
That test separates memory from a long context without drawing a line at any particular window size. Tokens, frames or cache states that every step reads directly are active context; once history is selectively retained, compressed, retrieved, consolidated or revised beyond that window, it is functioning as memory, whatever the module is called. 1 The definition also survives architecture changes, which matters for a field whose systems mix discrete-token predictors, diffusion generators and flow models inside the same causal rollout.
Six failures motivate the whole apparatus, and each is a familiar sight in long generations: an entity that leaves the frame and comes back as a different entity, appearance that drifts across chunks, a scene that rebuilds itself with incompatible geometry when the camera returns, motion that decays into freezing or physical impossibility, a narrative that repeats or contradicts its own earlier commitments, and an intervention that quietly reverts (a door opened earlier closing itself again). 6 The survey is careful about attribution: the same symptoms can come from weak dynamics modelling, control errors or rendering, so each is a behaviour in which lost history is a contributing cause rather than a diagnosis by itself. 6
Four things can carry it
The survey's central taxonomy sorts mechanisms by the object that holds the history. 7
Visual memory keeps frames, clips or observation-aligned latents, in pixel space or in a VAE's latent space. It preserves appearance and layout with little abstraction and stays traceable to a specific observation, at the price of redundancy that grows with the rollout unless entries are selected or compressed; because the stored evidence may itself be generated, an artifact written into memory can be replayed into later steps. Named examples include Context-as-Memory (RGB frames), WorldPack (VAE-frame tokens), FramePack (packed VAE-frame history) and WorldCam (pose-indexed latents). 9
Implicit state memory keeps the history inside model-native states with no declared meaning: attention and KV caches, recurrent or state-space states, and compressed encodings of the past. This is where most production systems already sit, since a persistent KV cache is the cheapest thing to carry. Its cost is that the content of the state is learned, so a memory cannot be read, checked or edited directly, and the information is tied to that model. 10 LongLive and Anchor Forcing hold persistent sink or anchor KV states beside a rolling local history, an approach whose weakness the survey names: an anchor becomes stale once appearance, viewpoint or world state legitimately changes. 8
Explicit state memory stores typed variables with declared world meaning: entity identities and attributes, camera pose, occupancy or depth fields, spatial relations, event states. It can be inspected and corrected directly, and it can hold an estimate of the current world rather than an archive of observations, but it omits rendering detail and needs a state estimator, a geometric fusion step, a simulator or a learned transition model to stay current. Representative entries in the survey's table include WorldStereo, Spatia, Memory Forcing, Voyager and SPMEM, several of them point-cloud or surfel maps indexed by pose. 8
Adaptive parametric memory writes the history into adapted parameters: fast-weight layers or selected backbone weights, or a separate module such as a temporary LoRA patch. The state persists beyond the adaptation event and influences generation through the forward pass. Ordinary pretraining and task fine-tuning do not qualify here; the parameter state has to carry sequence- or instance-specific information. The cost is isolation and reversibility, and interference between adapted parameters. 8
| Carrier | What it retains | What it costs | Named in the survey |
|---|---|---|---|
| Visual | Frames, clips, observation-aligned VAE latents | Redundancy grows with the rollout; generated artifacts stored as memory get replayed | Context-as-Memory, WorldPack, FramePack, WorldCam 9 |
| Implicit state | KV and attention caches, recurrent or state-space states, encoded history | Content is learned rather than declared, so it can be hard to inspect or edit; tied to the model | Persistent sink and anchor KV states in LongLive and Anchor Forcing 8 |
| Explicit state | Typed variables: entities, pose, depth, relations, events | Omits rendering detail; maintenance needs estimation or a simulator | WorldStereo, Spatia, Memory Forcing, Voyager, SPMEM 8 |
| Adaptive parametric | Adapted weights, fast-weight layers, LoRA modules | Hard to isolate or reverse; interference between adaptations | Temporary LoRA patches and fast-weight layers 8 |
Put the four side by side on one axis — how much of the original evidence survives, against how structured and correctable the state is — and the ordering inverts: the carriers that hold the most detail are the hardest to inspect and revise, and the ones that are easiest to edit carry the least rendering information. The judgment at that intersection is that this is a design trade-off between kinds of information rather than a ranking, and the mechanism follows from it: because no single carrier preserves both fine appearance and an editable world state, real systems stack them, and then fail on the seams. The survey names the seams — incompatible entity identifiers, temporal references, coordinate frames, access mechanisms and update rules, with little cross-carrier verification — and states the open problem as coordination between historical evidence and an evolving world estimate. 8
Five things it has to keep
Holding a state across steps is the means; the survey separates out what the state has to keep. 8
- Identity. The same individuated entity comes back after a gap, an occlusion or a cut. Recovering it matters more than freezing its pose or attributes. 8
- Spatial. A region visited earlier is recoverable when the camera returns, including its geometry. 8
- Dynamic. Motion states and evolving processes survive the moment their evidence leaves the window. 8
- Semantic. Roles, relations, events and unresolved goals established earlier keep constraining later output. 8
- Causal. The delayed consequences of an intervention persist after the action and its visual evidence are gone. 8
The governing principle is selective invariance: memory should hold what ought to stay fixed while letting pose, appearance conditions, motion, relations and world state change legitimately. A memory that freezes the world fails as badly as one that forgets it. 8
The functions also interact, and the interactions are where partial memory looks like success. Identity and semantics couple through reference: a visually stable character can carry the wrong role, or a correctly remembered relation can attach to the wrong instance. Spatial and dynamic preservation couple whenever something moves through a persistent environment, so locally plausible motion can cross incompatible geometry. An intervention pulls on all of them at once. The survey's example is transferring a tool: location, ownership or assignment, subsequent motion and available future interactions all change together, so a system can remember that the action happened and still render the pre-intervention state, or reproduce the immediate edit without its downstream consequences. 8
Five operations, and where each one loses information
The survey then follows the state through one step of generation, and the loop has five stages: write, read, update, manage, integrate. 8
Writing is the admission gate, and its policies fail in characteristic ways. Admitting everything (FIFO-Diffusion appending latent frames to a queue, dense frame or KV buffers) defers the decision rather than avoiding it, and carries redundant observations and self-generated artifacts forward. Admitting anchors instead keeps a sparse set of long-range references at low cost, as LongLive and Anchor Forcing do, and then depends on later updating to stop those anchors going stale; FreqForcing periodically commits new anchors to a persistent bank for that reason. Admitting by estimated utility — Grounded Forcing's diversity-updated anchor bank, RECAP-Forcing's novelty scoring — allocates capacity more selectively, with a limit the survey states plainly: future usefulness is only partly observable when the decision is made, so a peripheral object or briefly seen landmark dropped as redundant may be exactly what a later revisit needs. 8
Reading decides which retained entry is exposed to the current prediction; storage without access changes nothing. Updating decides whether that entry appends to, consolidates or corrects the state, and the tension is between stability and plasticity: too little updating leaves stale state, too much overwrites commitments that were still valid. Append-only extension keeps evidence intact and lets conflicting versions coexist. 8 Management decides what survives under a fixed budget. The simplest policy, a rolling window, treats age as a proxy for relevance, so evidence about identity, geometry or an earlier state change can disappear while it is still needed. 8 Integration decides whether the memory that was retrieved actually influences the output. Concatenating it to the input is the cheapest path and the weakest: distant entries may attract little attention, and the concatenated history consumes the very budget memory was meant to relieve, which converts retrieval into available context rather than used context. 8
The payoff of splitting the loop this way shows up in the survey's conclusion. A system can discard a crucial observation, hold a stale or incorrect state, keep a correct state out of reach, or retrieve the right content and then fail to use it. Those four failures produce similar-looking video and call for different fixes. 8
How the training target moved
A memory system is trained on histories that differ from the ones it will meet. Training typically builds the memory state from ground-truth visual units; deployment writes the model's own outputs into it. Two different distributions over persistent states follow, and the survey states the relation as an extension of exposure bias from generated outputs to the states that condition future generation: one small error written into memory shifts later conditioning, and repeated transitions move the rollout into regions the training distribution never covered. 8
The literature the survey organises maps onto three ways of choosing those training states. Teacher forcing builds them from ground truth, as ACDiT, Ca2-VDM and GPDiT do when they condition each noisy block on preceding clean blocks; Causal Forcing uses a teacher-forced model to initialise a causally conditioned student. History augmentation perturbs, masks or resamples reference histories to widen the distribution, as in Diffusion Forcing, GameNGen, AR-Diffusion and Resampling Forcing. Self-rollout builds the state from the model's own recursive predictions, as Self Forcing, Rolling Forcing, Self-Forcing++ and Causal-rCM do. Self-rollout matches deployment conditions most closely and costs the most to stabilise. 8
Alongside that, memory behaviour itself became a training target: learning to retain (LongLive, Reward Forcing, Sparse Forcing), to compress (FramePack, PackForcing, TinyHistory), to retrieve (Context-as-Memory, MosaicMem, Memory Forcing, LongLive-RAG, MemLearner) and to adapt (SlowFast-VGen). 8 The survey's own diagnosis of where this stands: teacher forcing still dominates because it is stable, hand-designed augmentation misses structured errors such as identity drift or stale state, and self-rollout exposes the model to realistic states at a cost that grows with rollout length. 8
A score is memory evidence only when the query needs history
The survey's most transferable judgment is about measurement. No score is memory-specific on its own: a DINO similarity between adjacent frames measures visible consistency, and the same similarity measured after an entity disappears and returns can measure delayed identity preservation. The protocol decides why the number is evidence, and the queried state decides what to measure. 8
Three setups create a clean information gap: an entity reappears after occlusion or a cut (EntityBench, MBench, MemoBench, LiveBench); the camera leaves and returns to a comparable view (MIND, iWorld-Bench, WBench, WorldRoamBench, R2M-Bench); or the query asks for the state that should have resulted from a hidden transition (MemoBench, StEvo-Bench, MBench). Two further setups are used but weaker: goal and narrative continuation suites (StoryEval, Long-CODE, StoryBench, LoCoT2V-Bench) generally keep the script available at query time, and delayed action effects have no standardised protocol — WBench, for instance, scores state-changing interactions within the current turn. 8
Each setup carries a prerequisite that separates memory failure from execution failure, and the survey asks for the success rate to be reported: the entity must actually leave and return, the camera must actually depart and come back to a comparable pose, the hidden transition must be one the model can generate when visible, and the intervention must have succeeded before its persistence is scored. 8 With the gap in place, the measurement follows the target rather than the metric family: presence first, then identity; perceptual similarity together with geometry, since matching pixels need not mean matching structure; state or progress labels when several visual realisations are valid; goal completion, ordering and contradiction for commitments; and action success before delayed-effect correctness. 8
| Query needs history | What to measure | Benchmarks the survey cites |
|---|---|---|
| Entity reappears | Presence, then identity and attributes | EntityBench, MBench, MemoBench, LiveBench 8 |
| Camera returns | Appearance plus geometry, camera and pose closure | MIND, iWorld-Bench, WBench, WorldRoamBench 8 |
| Hidden state evolves | State and progress accuracy, event order | MemoBench, StEvo-Bench, MBench 8 |
| An action's effect is queried later | Action success, then delayed-state correctness | MIND, WBench, WorldMark; VACT and What-If World for counterfactuals 8 |
Two further controls make the difference between correlation and attribution. Running paired histories with and without the relevant past shows whether prior history mattered; disabling, deleting or corrupting a specific memory bank, cache or retrieval module localises the dependence inside one architecture, which the survey notes cannot substitute for a model-independent protocol. Evaluator reliability is treated as part of validity, since judges with sparse temporal sampling miss brief failures. 8 The survey's recommendation for reporting is to plot the score against the history-query separation — recurrence gap, path length, hidden duration, interaction depth — with the memory budget controlled separately, because gradual drift and abrupt failure can end at the same number. 8
Before you trust a memory claim
- Ask what the query can still see. If the target remains on screen or the prompt restates it, the score measures consistency. A memory claim needs the answer to be absent from the local condition. 8
- Demand the prerequisite rate. Reappearance, pose-compatible return, a generatable transition and a successful action each have to be verified before the memory number means anything. 8
- Name the carrier and the stage. Ask which object holds the state and whether the failure was in writing, updating, reading or integration; four different repairs follow. 8
- Ask for separation, not an endpoint. A score at one recurrence gap says little; a curve against gap, path length or hidden duration says whether retention degrades gradually or collapses. 8
- Require a paired-history or component test for causal claims. Matched conditions with and without the relevant past, or an intervention on the memory module, are what tie a result to memory use. 8
- Report the cost with the score. Storage, active access, retrieval latency and computation are separate axes, and a method that wins at one budget can lose at another. 8
- Check the evaluator. Sampling density, failure coverage and calibration against human judgments decide whether a judge can see the failure being claimed. 8
Where the evidence stops
The survey classifies methods and it does not compare them. I looked for head-to-head figures across the sections and tables covering memory functions and operations and found none: no effect sizes, no benchmark deltas, no ranking. That is consistent with what it is — a taxonomy built to make heterogeneous work comparable in vocabulary — but it means a reader cannot use it to choose between two named methods, and the property labels in its carrier tables are graded qualitatively rather than measured. Method attributions throughout this piece are the survey's own classification.
Its protocol coverage is uneven by its own account. Identity recurrence and spatial revisitation have the most mature tests; correction, contradiction resolution, stale-state invalidation, selective forgetting, rollback and delayed intervention effects are thinly covered, and no standardised protocol exists for a delayed action effect or for a goal that has been removed from the local condition. Different studies vary separation, budget, prerequisite success and evaluator calibration, which limits comparison across systems. 8
The scope has edges the authors set deliberately: memory mechanisms for autoregressive generation, covering discrete-token and continuous generators, with short-video generation excluded and world-action models covered only where their memory mechanisms inform visual generation. 2 The paper is an unrefereed preprint submitted on 23 September 2026, and it invites corrections from a field that is moving faster than its reference list can. 3
What the survey does settle is the question a long-video demo invites and rarely answers. Capacity is a separate variable from memory. A system keeps a durable memory when the state that should still matter stays correct, stays reachable and still changes the output, and the way to find out is to hide the answer and ask for it back later. The full text, with the carrier tables, the operation taxonomy and the benchmark survey, is at The Past Frames the Future: Memory for Autoregressive Video Generation.
References
- 1
- 2
- 3Hugging Face, "The Past Frames the Future" paper page
huggingface.co
- 4"Awesome-AR-Video-Memory" repository
github.com
- 5
- 6
- 7
- 8
- 9
- 10
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.