REFACTOR-VLA finds a library signal in robot trajectories, then loses the motor programs (2026)

On 1 September 2026, Riyaaz Shaik and Chandru Venkataraman released REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs. On the four LIBERO suites, their BEK + InfoNCE configuration beat the strongest selected skill-discovery baseline by 0.184 mean NMI across three seeds. The paper's more consequential result is narrower: its behavioral clustering works, its typed library can rewrite demonstrations, and its tested motor-primitive vocabulary yields zero abstractions. 1

That split matters for the DreamCoder lineage. REFACTOR-VLA moves the equivalence test closer to robot dynamics, then uses typed program compression to package the result. The paper supplies evidence for better clustering and library-conditioned decoding. The paper leaves persistent motor-library growth and real-robot transfer for future work.

What the paper changes

DreamCoder starts with symbolic programs, where two solutions can share a syntactic structure. REFACTOR-VLA starts with continuous robot trajectories. Two action fragments can reach the same useful state through different motor commands, while similar-looking fragments can produce different outcomes. A reusable library needs an equivalence rule that can tell those cases apart.

The paper's Behavioral-Equivalence Kernel, or BEK, compares two trajectory fragments after inserting each fragment into the same initial state of a learned latent world model. The kernel combines a value-difference term with a Wasserstein distance between their multi-step latent rollout distributions. The resulting distance becomes the input to clustering. 1

The training loop has three phases:

Phase A trains a DreamerV3-style latent world model with a frozen DINOv2 visual encoder.
Phase B trains a Typed Program Emitter and a Library-Conditioned Action Decoder to emit typed lambda terms and 16-step action chunks.
Phase C distills the world model's fragment representation into a Siamese amortizer, clusters fragments, and applies top-down anti-unification to propose library entries.

The library-refactor step admits a candidate only when the candidate clears a BEK soundness check, a return-preservation check, and a minimum-description-length gain threshold of more than four nats. The admitted entries become typed lambda programs that condition later decoding. 1

REFACTOR-VLA alternates a sleep-phase behavioral-equivalence pass with a wake-phase typed program decoder. — The paper's architecture: a latent world model feeds BEK rollouts; clustered fragments pass through a Siamese amortizer and Stitch-style refactor; the resulting typed library conditions the wake-phase emitter and action decoder. 1

The design therefore joins two kinds of compression. BEK supplies a behavioral grouping rule for continuous fragments. Anti-unification supplies a symbolic program form for shared structure. The paper's distinctive move is the handoff between them.

What the evidence measures

The headline comparison uses held-out fragments from libero_object, libero_spatial, libero_goal, and libero_10. The reported metric is normalized mutual information between the discovered clusters and task-index labels. With the auxiliary supervised InfoNCE term in Phase A, the BEK reaches 0.462, 0.867, 0.915, and 0.754 on the four suites, respectively, with three-seed standard deviations of 0.021, 0.025, 0.013, and 0.010. The mean gain over the strongest baseline in each suite is 0.184 NMI. 1

The ablation identifies the training objective as the main lever in this experiment. Removing InfoNCE lowers the mean NMI from 0.749 to 0.493. Increasing the latent world model from 188 million to 430 million parameters lowers NMI on all four suites when the objective shape stays fixed: the object, spatial, goal, and LIBERO-10 values fall by 0.040, 0.146, 0.018, and 0.090. The larger model fits its distillation target more closely, yet the target organizes fragments into a less useful partition. 1

The result supports a precise claim: task-aware shaping of the Phase-A latent space matters more than parameter count for this clustering probe. The result says less about robot control. NMI measures whether fragments land in task-aligned clusters; it does not measure successful manipulation, cross-task execution, or physical-robot transfer.

The paper also reports an end-to-end library check. On 256 sampled libero_object demonstrations, the three-entry grammar rewrites all 256 examples, uses two of the three entries, and changes LCAD velocity error by -0.0059, within the paper's ±0.02 tolerance. That result shows that the decoder can consume the discovered grammar while preserving the measured action distribution. 1

The cross-provider result is more qualified. Across 12 pairs, the mean NMI is 0.705 with a 95% bootstrap interval of [0.683, 0.729]. The point estimate clears the paper's 0.700 gate, while the lower interval bound falls 0.017 below it. Only the goal suite clears that gate cleanly. 1

Where the library actually grows

The paper's title places the emphasis on typed motor programs. The real-LIBERO library result places the emphasis elsewhere. The sleep-phase compression finds three abstractions totaling 1211 nats in the Lang slot of the vocabulary. The largest abstraction captures a "pick up X, place in basket" pattern. The motor subspace containing Twist, Wrench, GripperPhase, and Pose produces zero abstractions across five MDL thresholds, even after dynamic-time-warping pre-alignment. 1

That outcome separates semantic task structure from motor reuse. BEK can group fragments by predicted behavioral similarity, yet strict token-level anti-unification still needs matching motor syntax before it can write a reusable motor abstraction. The paper proposes continuous action prototypes as the next route. The current experiment therefore demonstrates a task-language library conditioned on a continuous decoder; it leaves the harder question of reusable motor code unanswered.

A second boundary concerns the return-preservation gate. LIBERO supplies no reward column, so the paper sets the return-head weight to zero and leaves the gate inactive on the main benchmark. The paper measures velocity-error preservation for the rewritten demonstrations, which is useful evidence about the decoder. That measurement is different from verifying return preservation under a task reward. 1

The theoretical guarantee carries the same qualification. The paper measures the latent world model's return error on a separate synthetic RecursivePourEnv, where the five-seed mean sup-norm error is 0.205 ± 0.039. With the paper's stated discount and admission threshold, the resulting worst-case return-degradation bound is numerically loose. LIBERO supplies the main clustering evidence; RecursivePourEnv supplies the reward-model error estimate. The two environments support different parts of the argument. 1

Position in the DreamCoder lineage

DreamCoder learns a reusable typed DSL and a neural recognition model together. Its wake phase searches for task programs in library order; abstraction sleep compresses solved programs; dreaming sleep trains the recognizer on replayed and generated programs. The library persists and deepens across tasks. 2

Stitch changes the compression procedure rather than the wake-sleep framing. Its corpus-guided top-down synthesis searches partial abstractions, prunes them with corpus matches and utility bounds, then rewrites the corpus one abstraction at a time. The process builds hierarchical libraries from program corpora. 3 LILO combines LLM-guided synthesis with Stitch compression and automatic documentation, then reuses the resulting library in later synthesis. 4 LAPS adds language supervision to the joint learning of abstractions and neural search heuristics. 5

Method	Persistent object	Neural role	Abstraction or equivalence rule	Execution evidence	Lifetime across tasks
REFACTOR-VLA	Three typed entries in the reported real-LIBERO language slot	Latent world model, BEK amortizer, typed emitter, and Phase-A InfoNCE shaping	Value plus multi-step rollout divergence, then typed anti-unification with MDL gating 1	256/256 demonstration rewrites; velocity error stays within the stated tolerance 1	Training-run library; motor-slot growth and transfer remain open
DreamCoder	Hierarchical typed DSL and recognition model	Recognition model orders search; dreaming trains it	Bayesian compression of solved programs	Solves symbolic tasks through recognition-guided enumeration 2	Explicitly cross-task and iterative
Stitch	Compressed program corpus and learned abstractions	Compression search is the core engine	Corpus-guided top-down synthesis and rewriting	Evaluated on program corpora and library compression 3	Corpus-level and iterative
LILO	Named, documented reusable code library	LLM proposes and documents; Stitch compresses	Description-length compression plus documentation	Reuses libraries on REGEX, CLEVR, and LOGO 4	Cross-task reuse is a central goal

REFACTOR-VLA therefore advances one part of the lineage: it gives continuous trajectory fragments a dynamics-aware equivalence signal before symbolic refactoring. Its strongest measured library is still language-level. DreamCoder-style progress would require the later steps to hold together: motor abstractions must survive beyond one corpus, guide a recognizer or decoder on held-out tasks, and improve execution under a reward or physical outcome measure.

Verbatim quotes

"The driver runs three steps per outer iteration t: the wake phase (TPE and LCAD), the sleep phase (the BEK head), and the library-refactor phase (grammatical anti-unification with joint MDL and return-preservation gates)." — Riyaaz Shaik and Chandru Venkataraman, REFACTOR-VLA. 1

"Library learning is thus limited to the Lang slot (3 abstractions / 1211 nats); the motor subspace, with continuous ACT-style prototypes per cluster as a natural alternative, is left for future work." — Riyaaz Shaik and Chandru Venkataraman, REFACTOR-VLA. 1

The unresolved test is a held-out transfer experiment. A future system should learn motor abstractions from one set of tasks or embodiments, retain those abstractions in a persistent library, and use a learned recognizer or typed decoder to invoke them on new tasks. The evaluation should include an active return-preservation or physical-success check. Until that test exists, REFACTOR-VLA is best read as a dynamics-aware clustering and task-language refactoring result inside the DreamCoder family, rather than as a completed DreamCoder loop for reusable motor programs.

References

1
REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs
arxiv.org
2
DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning
arxiv.org
3
Top-Down Synthesis for Library Learning
arxiv.org
4
Lilo: Learning Interpretable Libraries by Compressing and Documenting Code
arxiv.org
5
Leveraging Language to Learn Program Abstractions and Search Heuristics
arxiv.org