Your multimodal agent should compile memories before users ask
PMMC shifts multimodal memory work from query time to consolidation time, giving product teams a concrete way to trade precomputed, verified evidence paths for lower live-query cost.
A multimodal agent does not become reliable just because it can retrieve an old image. The harder problem is deciding, before anyone asks a question, which future questions a memory may need to answer and how to prove the answer from pixels, text, and time.
A new paper proposes Prospective Multimodal Memory Compilation (PMMC): move part of memory reasoning from query time to consolidation time. For a product team, the signal is a new architecture choice: spend compute while memories are being organized so the live request can route to a verified evidence program instead of improvising a search over raw history.
The short version
| Question | PM answer |
|---|---|
| What changed? | PMMC predicts likely future questions, compiles question-conditioned programs for retrieving and binding multimodal evidence, then verifies those programs before storing them in a question bank. |
| What problem does it target? | Text-only summaries lose visual detail, fixed top-k retrieval misses image-text relationships, and agentic search puts token and latency cost on the user's critical path. |
| What is the evidence? | The authors report better answer quality and visual-evidence recall with lower query-time token and latency costs on multimodal long-term-memory benchmarks. The paper is an arXiv preprint submitted August 2, 2026, not a production SLA. 1 |
| Why now? | Google updated its Memory Bank documentation on August 4 with extraction, consolidation, asynchronous generation, multimodal-to-text understanding, and identity-scoped retrieval, making the adjacent production pattern concrete. 2 |
| What should a PM do? | Pilot precompiled memory only for read-heavy workflows where visual evidence matters, and gate each program on evidence support, freshness, and cost. |
What changed: memory becomes a compiled artifact
Most agent memory systems take one of two shapes.
Fixed multimodal RAG stores a representation of an interaction and retrieves the top few matches. It is efficient, but the representation and retrieval rule are fixed before the system knows the user's question. A photo, caption, and later text correction may be close in vector space while still failing to answer a question about a small visual detail or a change over time.
Agentic retrieval lets the model search, reformulate, inspect, and search again after the user asks. That can adapt to an unfamiliar question, but every extra search and multimodal inspection adds tokens and latency to the request.
PMMC moves work to the period when memories are consolidated. Its three roles are deliberately separated:
- Questioner: proposes future questions that the accumulated interaction might need to answer.
- Planner: turns a question into a typed retrieval program, such as retrieving text and an image, binding the two, expanding to the raw image, and verifying the result.
- Doubter: checks whether the planned evidence path can actually support the predicted answer. If not, the plan is revised or discarded.
The surviving question-program pairs form a verified question bank. At query time, the system classifies the user's request, matches it against that bank, runs the stored program, and materializes source evidence for the answer. The model still answers the question, but it no longer has to invent the entire evidence path on the critical path.
The important product distinction is that PMMC does not mean "summarize every image better." It creates a typed, testable access path to the original evidence. The program can preserve the option to reach back to raw pixels when a caption or embedding is not enough. That makes the stored memory more like a small executable query with a contract than a paragraph of prose.
The arXiv record lists 13 authors but does not display their institutional affiliations on the abstract page. The method and results below therefore attribute the work to the listed author group and arXiv record without guessing at institutions. 1
Why PMs should care
1. Personalization is moving from storage to access design
A typical product conversation asks whether the assistant remembers a user preference. PMMC suggests a more useful question: Can the system produce the right evidence path for the next class of questions?
That matters for products with long-lived visual context:
- A field-service assistant may need to connect a photo of a damaged part with a later repair note and the time between them.
- A shopping or home-design assistant may need to distinguish what was visible in an earlier image from what the user merely described.
- A research workspace may need to bind a chart, its caption, and a later correction without flattening them into one irreversible summary.
In each case, a text summary can preserve the topic while losing the evidence. A raw-image dump preserves evidence but makes every future query expensive. PMMC is a proposal for storing the middle layer: a verified route that can retrieve and bind the source material when a known question shape appears.
2. The cost moves; it does not disappear
Compilation makes the online path cheaper by spending more during memory consolidation. That is attractive when the same user's memory will be queried repeatedly, but it is a poor fit for disposable chats or data that changes faster than the question bank can be refreshed.
The paper's abstract reports lower query-time token and latency costs, but that claim should not be read as lower total system cost. A production review needs both ledgers:
| Cost surface | What to measure |
|---|---|
| Consolidation | GPU time, raw-image inspection, number of candidate questions, rejected plans, storage for programs and evidence pointers |
| Query | Routing accuracy, program execution time, tokens, raw-image fetches, fallback rate, p50 and p95 latency |
| Quality | Answer accuracy, visual evidence recall, citation or evidence grounding, stale-memory rate, and performance on unseen question forms |
| Operations | Recompilation after edits, deletion propagation, identity isolation, access-control failures, and rollback behavior |
The risk is not only an incorrect answer. A stale compiled program can confidently retrieve the wrong version of an image or text fragment. The system therefore needs a freshness and invalidation policy alongside its memory schema.
3. The adjacent production pattern already has useful primitives
Google's Agent Platform Memory Bank documentation describes a nearby, more general workflow: extract meaningful information from session events, consolidate it with existing memories, generate asynchronously or from continuous event ingestion, and retrieve by similarity later. It also documents multimodal understanding that produces textual insights, identity-scoped memories, and IAM conditions for restricting which principals can read or write a scope. 2
That is not an implementation of PMMC. The difference is the artifact being stored. Memory Bank's documented primitives center on managed memories and retrieval; PMMC adds predicted questions, typed programs, and a verifier for the evidence path. A product team can use the first set of primitives to build the ingestion and governance plane, then test whether the second set is worth adding for high-value visual workflows.
For an accessible implementation walkthrough, Google Cloud Tech's six-minute video demonstrates session versus memory, MemoryService options, asynchronous persistence, semantic retrieval, and a multimodal Memory Bank flow. It had 25,415 views when checked. The video explains an adjacent platform pattern; it does not validate PMMC's benchmark results. 3
Loading content card…
There is no useful X reaction to cite for this paper: targeted searches for the arXiv ID, title, and selected practitioner accounts returned no relevant posts. That absence is itself a boundary on the signal. The case for PMMC comes from the paper and adjacent first-party implementation material, not from social adoption chatter.
How to implement now
Do not start by compiling every user interaction. Build a narrow experiment around a repeated question family and a corpus whose source evidence is worth preserving.
- Pick a visual workflow with repeat queries. Use a read-heavy surface such as product catalog history, inspection records, or a research notebook. Avoid actions that can spend money, change permissions, or alter a user's data until the evidence path is proven.
- Define the evidence contract. For each answer, specify what must be returned: the relevant text span, the image or crop, the timestamp or version, and the relationship between them. Do not accept a fluent answer without the evidence objects that support it.
- Create three baselines. Compare fixed multimodal retrieval, an agentic search-and-reformulate loop, and a PMMC-style path. Keep the underlying model, corpus, and answer budget comparable. The question is not whether compilation wins in a paper; it is where its amortized cost beats online search for your traffic pattern.
- Compile offline and keep programs typed. Store the predicted question, paraphrases, retrieval modality, binding operation, raw-evidence fallback, verifier result, and source versions. Treat a program as a versioned product artifact with an owner and a deletion path.
- Add hard gates before execution. A query should run a compiled program only when its match confidence, evidence freshness, identity scope, and access checks pass. Otherwise route to ordinary multimodal retrieval or an agentic fallback, and record why the compiled path was rejected.
- Measure the trade, not just answer quality. Track visual evidence recall, answer accuracy, unsupported-claim rate, query p95, consolidation cost per memory, fallback rate, stale-program rate, and cost per correct grounded answer. Hold out new question phrasings and changed images so the question bank cannot pass by memorizing its own candidates.
- Test deletion and contradiction as first-class flows. When a user deletes an image, corrects a caption, or revokes access, invalidate every program and cached evidence pointer that depends on it. A memory feature is not ready if it can answer correctly only while its source history never changes.
The practical decision rule is simple: compile when the same evidence will answer many future questions and a wrong visual detail is expensive; retrieve online when the question distribution is novel or the underlying memory changes quickly. PMMC is worth watching because it turns that rule into an explicit system boundary. It does not make multimodal memory solved. It gives product teams a way to buy lower query-time cost with precomputed, verifiable access paths, and a set of metrics for deciding whether that trade is worth shipping.
References
- 1PMMC arXiv record
arxiv.org
- 2Google Agent Platform Memory Bank
docs.cloud.google.com
- 3AI agent long-term memory with memory bank
youtube.com
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.