SimSkill accumulates 150 simulator skills in plain text, but trades inference cost for task coverage (2026)

On 11 September 2026, Qi Liu, Qinzheng Wang, Can Li, Yiming Bie, and Wanjing Ma released the third revision of SimSkill: A Self-Evolving LLM Agent for Skill and Knowledge Accumulation in Traffic Simulation. Tested across two held-out 40-task benchmarks in the open-source SUMO simulator, SimSkill improved independently verified task success by up to 25 percentage points over vanilla Claude Code. The paper's more consequential finding is structural: explicit procedural and semantic memory expands the boundary of what an LLM agent can solve on hard, multi-step tasks, while increasing median monetary cost and execution time for capable models. 1

That division speaks directly to the DreamCoder lineage. DreamCoder pairs a learned domain-specific language (DSL) with a neural recognition model trained via wake-sleep dreaming. SimSkill swaps formal λ-calculus and parameter updates for external filesystem artifacts: natural-language instructions bundled with executable Python scripts, cross-linked Markdown knowledge pages, and frozen LLM retrieval. The paper provides evidence that an agent can autonomously discover and maintain a library of 150 procedural skills and 153 semantic pages over 80 hours of simulation. It also shows where natural-language library learning runs into retrieval overhead and backbone sensitivity.

What the paper changes

Classic program induction builds domain abstractions inside formal calculi. In DreamCoder, solved program traces are refactored into λ-calculus expressions through Bayesian compression, optimizing the Minimum Description Length (MDL) of the program corpus and the evolving library. 2 SimSkill operates in the Simulation of Urban MObility (SUMO) environment, where tasks require configuring road networks, running dynamic user equilibrium assignments, controlling signals via TraCI, and processing trajectory files. Such environments resist closed formal DSLs.

SimSkill formalizes the persistent state of the agent at iteration

t

as a tripartite memory tuple:

M_{t} = (E_{t}, P_{t}, S_{t})

where

E_{t}

is episodic memory,

P_{t}

is procedural memory, and

S_{t}

is semantic memory. 1 Each memory type fulfills a distinct operational role:

Episodic memory (episodic-memory/): Stores timestamped records of every attempted task, including initial task prompts, tool-call traces, intermediate code, critic verdicts, and final deliverables. Retaining failed attempts provides an auditable history of revisions.
Procedural memory (procedural-memory/): Formatted as Claude Code skill directories. Each skill contains a required SKILL.md file with YAML metadata and natural-language operating instructions, accompanied by optional deterministic Python utilities in scripts/, documentation in references/, and templates in assets/.
Semantic memory (semantic-memory/): Follows the Open Knowledge Format and LLM Wiki pattern. Each page represents a conceptual entity with YAML front matter (retrieval keywords, summary, timestamps, and links) and a Markdown body. A compact central index (index.md) exposes titles and summaries for rapid matching.

The operational engine runs three specialized sub-agents guided by five system skills (learn, infer, memory-retrieve, memory-ingest, memory-lint). The autonomous curriculum generator proposes a task designed to fill identified competence gaps. The retrieval skill matches the task against procedural front matter and semantic index summaries, selecting at most ten relevant candidates. The action agent writes and executes simulator code. The critic agent independently validates the outcome against simulation logs, edge-data outputs, and task constraints.

After an episode

e_{t}

concludes, an LLM-driven consolidation operator updates memory:

(P_{t + 1}, S_{t + 1}) = I (P_{t}, S_{t}, e_{t})

The operator searches for semantically related items before creating new files, preferring to refine existing skills or merge related concepts. A background linting skill validates cross-links, updates the semantic index, and removes orphaned references. The backbone model weights remain entirely untouched.

What the evidence measures

The empirical evaluation uses two frozen, held-out benchmarks of 40 standalone tasks each. Benchmark V1 covers the complete simulation lifecycle across four difficulty tiers (10 tasks per tier). Benchmark V2 doubles the difficulty, distributing 40 tasks across Tier 3 (20 tasks) and Tier 4 (20 tasks), requiring complex policy modeling such as long-run induced demand, multimodal transit priority, and origin-destination matrix estimation.

The authors evaluated three distinct backbone LLMs: DeepSeek-V4-Pro, GLM-5.2, and Qwen3.7-Max. Claude Opus 5 served as an independent judge, re-running simulations in fresh container sessions to verify reproducibility, numerical claims, and constraint satisfaction. For Benchmark V2, GLM-5.2 provided a second, continuous pointwise evaluation score in

[0, 1]

Benchmark	Backbone Model	Complete SimSkill (full-ver)	Vanilla Claude Code (vanilla-cc)	Success Difference (pp)	Median Cost F/V (USD)	Median Wall Time F/V (s)
V1	DeepSeek-V4-Pro	38 / 40 (95.0%)	34 / 40 (85.0%)	+10.0	$0.78 / $0.49	1056 / 650
V1	GLM-5.2	30 / 40 (75.0%)	31 / 40 (77.5%)	-2.5	$1.02 / $1.47	918 / 1918
V1	Qwen3.7-Max	23 / 40 (57.5%)	13 / 40 (32.5%)	+25.0	$1.84 / $1.03	851 / 683
V2	DeepSeek-V4-Pro	27 / 40 (67.5%)	19 / 40 (47.5%)	+20.0	$3.93 / $2.92	4623 / 4796
V2	GLM-5.2	10 / 40 (25.0%)	10 / 40 (25.0%)	0.0	$2.29 / $2.44	2008 / 2899
V2	Qwen3.7-Max	2 / 40 (5.0%)	0 / 40 (0.0%)	+5.0	$2.16 / $2.35	916 / 1969

Task-solving success, median cost, and median wall-clock time across Benchmark V1 and V2. 1

These metrics highlight an accuracy–resource trade-off. For models capable of utilizing the framework (DeepSeek-V4-Pro and Qwen3.7-Max), SimSkill substantially increases task completion, reaching +20.0 percentage points on the demanding V2 suite for DeepSeek-V4-Pro. However, median monetary cost rises from $0.49 to $0.78 on V1 and from $2.92 to $3.93 on V2. Multi-agent coordination, memory retrieval, and critic verification consume tokens and wall-clock time.

GLM-5.2 demonstrates the opposite pattern. The model achieved no accuracy gain on either benchmark (75.0% vs. 77.5% on V1; 25.0% vs. 25.0% on V2). Yet SimSkill reduced its median cost by 31% and its median time by 52% on V1. The retrieved memory allowed GLM-5.2 to terminate search earlier on tasks within its capability, without expanding the frontier of tasks it could solve.

Where memory actually helps

To isolate the individual contributions of the system components, the authors ran a five-condition ablation on Benchmark V1 using DeepSeek-V4-Pro and Qwen3.7-Max.

Ablation curves on Benchmark V1 showing fraction of tasks solved versus monetary cost and wall-clock time. — Ablation curves on Benchmark V1 for DeepSeek-V4-Pro and Qwen3.7-Max under the Claude Opus 5 binary judge, comparing full SimSkill, procedural memory only, semantic memory only, framework only, and vanilla Claude Code. 1

On DeepSeek-V4-Pro, the inference framework alone (infer-frame-only) solved 34 tasks, matching vanilla Claude Code. Adding semantic memory (sem-mem-ver) raised completion to 35 tasks. Adding procedural memory (proc-mem-ver) raised completion to 37 tasks. Combining both (full-ver) reached 38 tasks. On Qwen3.7-Max, the progression was 13 (vanilla-cc), 16 (infer-frame-only), 19 (sem-mem-ver), 20 (proc-mem-ver), and 23 (full-ver).

The factorial interaction contrast measures whether procedural and semantic memory amplify each other:

Δ_{P \times S} = A_{P S} - A_{P \overset{ˉ}{S}} - A_{\overset{ˉ}{P} S} + A_{\overset{ˉ}{P} \overset{ˉ}{S}}

Evaluating this formula yields

0.950 - 0.925 - 0.875 + 0.850 = 0.0

for DeepSeek-V4-Pro, and

0.575 - 0.500 - 0.475 + 0.400 = 0.0

for Qwen3.7-Max. The contributions of procedural skills and semantic knowledge pages are strictly additive at the aggregate accuracy endpoint.

The resource curves reveal an additional dynamic. For DeepSeek-V4-Pro, the complete system exhibited a lower median cost ($0.78) and median runtime (1056 s) than either single-memory condition ($1.41 / 2423 s for procedural alone; $1.31 / 2383 s for semantic alone). Having access to both instructions and declarative domain knowledge pruned unproductive search paths. For Qwen3.7-Max, the complete system was more expensive across the board, with its advantage appearing exclusively in the high-budget tail.

Position in the DreamCoder lineage

DreamCoder established that program induction becomes tractable when an agent alternates between solving tasks, abstracting common code into a library, and training a neural model to guide search. Subsequent iterations have modified individual components of that loop. Stitch replaced enumerative compression with top-down corpus synthesis over e-graphs. 3 LILO used large language models to write code and generate documentation strings, while retaining Stitch for formal compression. 4

SimSkill alters both the representation and the search guidance mechanism.

Method	Persistent Object	Neural Search Guidance	Abstraction / Refactoring Engine	Execution Substrate	Primary Scaling Bottleneck
SimSkill	Tripartite filesystem store: 150 procedural skills (`SKILL.md` + scripts) and 153 semantic Markdown pages 1	Frozen LLM reading retrieved metadata; lazy-loads full context on demand	LLM consolidation operator $I (P_{t}, S_{t}, e_{t})$ with merge and lint rules	SUMO traffic simulator, TraCI API, Python scripts, XML definitions	Retrieval context overhead, token budget, and backbone reasoning limits
DreamCoder	Hierarchical typed λ-calculus DSL and trained neural recognition model 2	Recognition model $Q (ρ ∣ x)$ trained on waking replays and self-supervised fantasies	Bayesian program refactoring via version-space algebras and e-graphs	Lambda-calculus evaluator across synthetic symbolic domains	Combinatorial explosion of enumerative search and λ-calculus expressivity
Stitch	Compressed symbolic corpus and learned abstraction primitives 3	None; corpus-guided top-down search over partial program graphs	Fast top-down anti-unification with non-linear variable binding	Static program corpora (LISP, λ-calculus)	Memory footprint on massive corpora; absence of neural guidance
LILO	Named, documented Python/λ-calculus code library 4	LLM prompted with library functions, docstrings, and task specifications	Stitch symbolic compression paired with LLM auto-documentation	Domain-specific Python environments (Regex, Clevr, Logo)	Stitch compression limits on arbitrary Python control flow

SimSkill proves that an agent can construct an extensive library in an engineering environment where formal λ-calculus cannot operate. Natural language serves as the connective medium: instructions describe edge cases and heuristics, while bundled Python utilities handle deterministic data transforms. The trade-off is the loss of formal equivalence guarantees. DreamCoder proves equivalence by evaluation; SimSkill relies on an action–critic loop and LLM linting to catch regressions.

Verbatim quotes

"At learning iteration $t$ , the persistent state of SimSkill is $M_{t} = (E_{t}, P_{t}, S_{t})$ , where $E_{t}$ , $P_{t}$ , and $S_{t}$ denote episodic, procedural, and semantic memory, respectively. The system evolves by adding, revising, linking, validating, and consolidating these explicit artifacts." — Qi Liu, Qinzheng Wang, Can Li, Yiming Bie, and Wanjing Ma, SimSkill. 1

"Sutton’s Bitter Lesson holds that general methods that scale with computation eventually outperform systems built on hand-crafted domain structure. SimSkill follows this principle with a small set of general operations—task proposal, memory retrieval, environment interaction, criticism, consolidation, and maintenance—rather than a dedicated chain-of-thought template, planner, or hard-coded workflow for each traffic-simulation problem." — Qi Liu, Qinzheng Wang, Can Li, Yiming Bie, and Wanjing Ma, SimSkill. 1

"Parametric knowledge is difficult to inspect, revise locally, attribute to evidence, or transfer between model families. Natural-language instructions and knowledge pages, by contrast, can be read, critiqued, combined, and shared by humans and heterogeneous LLMs." — Qi Liu, Qinzheng Wang, Can Li, Yiming Bie, and Wanjing Ma, SimSkill. 1

"Full SimSkill increases verified success for two of the three tested backbones and retains an advantage on hard compositional and novel tasks, but the effect depends on the backbone model. Both procedural and semantic memory contribute, with a modestly larger procedural contribution. Memory can reduce wasted exploration in some settings, yet its retrieval and orchestration overhead frequently raises total inference cost; the principal demonstrated benefit is expanded task coverage rather than universal cost savings." — Qi Liu, Qinzheng Wang, Can Li, Yiming Bie, and Wanjing Ma, SimSkill. 1

The unresolved test

SimSkill demonstrates that an LLM agent can accumulate hundreds of inspectable, modular skills in a complex simulator without modifying weights. The critical unresolved question is how this natural-language library scales when the skill collection grows by another order of magnitude.

In DreamCoder, the learned recognition model amortizes search: as the library deepens, the neural model learns sharper conditional distributions over primitive selection, counteracting search explosion. In SimSkill, retrieval relies on surface-level keyword matching against YAML descriptions and index summaries, followed by an in-context selection of ten candidates. As the library expands from 150 skills to several thousand, this metadata-matching layer faces severe collision and rank collapse. A future test must evaluate whether natural-language skill libraries can sustain continual accumulation without human linting or degradation in retrieval accuracy, and whether cross-simulator transfer succeeds when underlying simulator APIs and execution semantics diverge.

SimSkill accumulates 150 simulator skills in plain text, but trades inference cost for task coverage