Parameters, representations, behaviors: three ways to fuse models, no ranking between them
A deep dive into a new arXiv survey that folds model merging, representation alignment and distillation into one taxonomy, and shows that the access a team has to its source models decides which fusion route it can take.
Hugging Face counted more than two million public models in its review of the open-source ecosystem this spring, and much of the field's work now arrives by reusing them. 1 Checkpoints get averaged, separately fine-tuned experts get merged back together, teachers get distilled into smaller students. Those practices grew separate literatures and separate names for themselves.
From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models puts them under one definition and one boundary, reading more than 150 papers to do it. 2 Shuo Cai, Yanggan Gu, Zihao Wang, Yuanyi Wang, Yibo Yan, Wenjun Wang, Yuhang Liu, Guanghao Zhu, Sirui Huang, Ming Li and Hongxia Yang wrote it at HKUST (Guangzhou), Hong Kong Polytechnic University, InfiX.ai and the Chinese University of Hong Kong, and posted it to arXiv on September 17.
The boundary is what runs at inference
Their definition is operational: model fusion constructs a target model by combining parameters, aligning representations, or distilling output behaviors from several source models, and the finished target then runs at inference without the complete source models present. 3 That closing clause does the work. It draws the line at what survives into production, leaving aside how the target was trained, and it settles four comparisons that practitioners otherwise argue about indefinitely.
Four familiar practices fall outside it. Ensembles still call every member model at inference; query routers such as RouteLLM pick one source model per request and invoke it; a model used to judge outputs evaluates them; and reinforcement learning against a scalar reward passes on a score while the teacher's own output behavior stays behind. 3 Inside the boundary are the methods that construct a target able to stand on its own.
Once the boundary is fixed, the older categories line up as one operation at three different fusion signals. Weight averaging and knowledge distillation become the cheap end and the flexible end of a single spectrum. The authors check this against ten earlier surveys and report that only theirs covers parameter combination, representation alignment and behavior distillation together. 4
Three levels, chosen by the signal you can reach
Parameter-level fusion combines the weight updates of the source models, adding task vectors and scaling, masking or projecting them to keep the sources from interfering. TIES, DARE and DELLA cut interference through sign consistency and selective dropping or rescaling; AdaMerging learns fusion coefficients from unlabeled data; AWD optimizes a decomposition of task vectors into redundant and disentangled parts. The price of working directly on weights is compatibility: the sources must agree on architecture, initialization and tokenizer for their parameters to line up at all. 5
Representation-level fusion matches what happens inside the models instead of what they store. Hidden states, activations and attention maps are used in three ways: to derive the merge weights, as in AIM and ZipIt!, to solve a local matching problem in closed form, as RegMean does for linear modules from their input covariance, or as a training objective for a repair step, as Representation Surgery, Surgeryv2 and FeatCal do for the drift a plain merge leaves behind. This level needs calibration data, layer correspondence and a similarity measure it can trust. 6
Behavior-level fusion learns from what the sources do: their token distributions, demonstrations, reasoning traces, tool-use traces, preferences and critiques. Forward and reverse KL over output distributions, on-policy distillation that supervises the target on prefixes the target itself generated, and preference-based fusion such as InfiFPO all sit here. Responses are the only input this level needs, which is how it reaches across architectures, tokenizers, modality interfaces and decoding policies. 7
What a team can run is decided before any of that, by how much of the source models it can see. The survey's comparison figure arranges the three levels against white-box access (weights and internals), gray-box access (representations, logits, attention) and black-box access (responses only), and prices each one in data demand and compute.
No level wins across settings
The next question is which level to choose when access allows more than one, and the survey declines to rank them. Comparisons across papers mix model pools, task pools, budgets and tuning, so it collects its evidence from matched comparisons inside single papers instead. 9
| Matched setting | Methods compared | Reported outcome |
|---|---|---|
| RLVR expert fusion, Qwen3-4B, five domains | TIES (parameter-level) against MT-OPD (behavior-level) | 61.00 average for TIES against 60.46 for MT-OPD, which used 967.9 GPU-hours of training 10 |
| Domain expert fusion, Llama-3.1-8B, five-domain accuracy | Task Arithmetic (parameter-level) against RegMean (representation-level) | 48.7 against 46.3 10 |
| Instruct + Code, scored on AlpacaEval 2.0, HumanEval and MBPP | Task Arithmetic at 0 shots (parameter-level) against ProDistill at 16, 32 and 64 shots (representation-level) | ProDistill trails at 0.2367 and 0.2380, and edges ahead at 0.2496 against 0.2448 10 |
| Matched 16-shot CLIP average accuracy | ProDistill (representation-level) against SAMerging (behavior-level) | SAMerging higher in five of six configurations, 91.2 against 87.2 on L14/8 and 83.8 against 81.1 on B32/8; ProDistill higher on one, 86.8 against 85.8 on L14/20 10 |
The 967.9 GPU-hours behind that behavior-level run are the part a budget-holder notices first. The score differences are small and the cost difference is large, and the ordering flips with the pool and the budget the comparison fixes.
Levels also stack, and the survey's second piece of within-paper evidence shows the stack paying off. Task Arithmetic rises from 63.5 to 65.8 on six Llama-3.1-8B tasks once FeatCal corrects representations; patient knowledge distillation lifts RACE test accuracy from 58.74 to 60.34; TED lifts a GLUE development average from 87.0 to 87.5; TinyBERT lifts a three-task GLUE average from 73.5 to 75.6. Each row sets one paper's own baseline against its own two-level variant, so those gains belong to those four settings. 11
Eleven years from checkpoint averaging to industrial distillation
The survey's timeline puts the field's growth on one axis and its methods on another, and the shape of the curve explains why the vocabulary took so long to settle.
The early years are weight-space tricks borrowed from training practice: knowledge distillation in 2015, snapshot averaging and patient knowledge distillation in 2018, DistilBERT in 2020, model soups in 2022. The 2023 jump comes with techniques built for combining sources that disagree, among them task arithmetic, TIES, RegMean and ZipIt!, and the following years add repair modules and distillation-scale methods. 13
By 2026 the practice shows up in production systems the survey names. DeepSeek-V4 trains domain experts separately and consolidates them with on-policy distillation, GLM-5 uses the same route, and Nemotron-Cascade 2 builds a compact mixture-of-experts model with 30 billion parameters and 3 billion active. DeepSeek-R1 earlier transferred its reasoning patterns into six dense models between 1.5 and 70 billion parameters. 14
What a single average score hides
Evaluation starts from two numbers here. Average performance over the task pool is the familiar one. Normalized performance compares the target model with each source model on that source's own task, which is the measure MergeBench uses to report how much of the source capability survived. 15 The two diverge often enough to matter, because a target can raise its average while quietly dropping the one capability it was built to keep.
The survey's metric table splits the question into six dimensions: overall quality, capability retention, interference and transfer, internal alignment, efficiency, and safety and risk. The later dimensions apply only when the setting supplies what they need — hidden states or logits for alignment, a threat model for safety, matched hardware for cost. 16
Benchmarks lag the methods they would judge. Of the 14 evaluation resources the survey compares, most support parameter-level fusion with shared model pools and task pools, and a subset, FusionBench and MergeBench among them, offers reusable multi-method protocols. Representation-level work usually reports drift inside its own method paper, and behavior-level work borrows task, response or safety benchmarks from distillation studies. 1517
A checklist before you fuse
- Start from the signal you can reach. Weights point to parameter-level fusion, internals to representation-level, responses only to behavior-level. The rest of the decision follows from that constraint. 9
- Rank candidate methods only inside a matched setting. Scores from different model pools, task pools and budgets do not order fusion levels, and the survey publishes its own numbers on that condition. 9
- Budget the compute the level implies. A behavior-level run in the survey's evidence cost 967.9 GPU-hours for a fraction of a point against a parameter-level merge. 10
- Plan a second level when the first leaves drift or missing capability. A cheap parameter merge, a representation repair pass and behavior supervision address different failure modes, and the survey's four within-paper hybrids show the second pass helping. 11
- Report per-source retention, worst-task drop and fusion cost beside the average. An aggregate can hide the loss of one source capability, which is the failure fusion exists to avoid. 16
- Screen the sources before they are fused. Unsafe behavior survives parameter merging, malicious updates and backdoors enter through both merging and distillation, and a backdoor stays in the target after the source models are gone. 18
- Assume fusion will become routine, and plan for forgetting and for I/O. Continual fusion overwrites earlier knowledge, and as the source pool grows, expert-parameter input and output and repeated scans become the bottleneck ahead of accuracy. 19
Where the evidence stops
Four problems stay open in the survey's own account. No general theory predicts whether a fusion will succeed, so source and method selection still runs on trial and error, and existing results cover parameter averaging in specific settings with no equivalent for representation alignment or behavior distillation. Source models that differ in architecture, tokenizer or modality have no reliable correspondence, and alignment techniques such as optimal transport and learned coefficients are early. Multi-source retention has no standard protocol. Risk travels with capability through all three levels. 18
The risk findings are concrete. Unsafe behavior survives parameter merging; LoRATK and Merge Hijacking show malicious updates and backdoors entering a fused model; backdoors also transfer through behavior distillation and persist in the target after the source models are removed. Merger-as-a-Stealer studies private-information leakage through fusion, and Among Us studies malicious contributions in collaborative model building. MergeGuard is the kind of post-fusion defensive check the authors call for. 18
Part of the cost picture is improving. Lightning OPD and TIP reduce live teacher serving and token-level supervision for on-policy distillation, where much of the behavior-level expense sits. 19
The survey's own limits matter for how much weight its numbers carry. It relies on reported results, some sources publish neither variance nor significance, fusion costs are not normalized across hardware and implementations, and its cross-paper figures are illustrative rather than controlled head-to-head comparisons. It also warns that fast-moving preprints and industrial systems with limited public details may have escaped it. This is a preprint submitted on September 17. 20
What the survey settles is narrow and useful: a boundary that says which systems count as fusion, and an access ladder that says which level a given team can run. Once settings do match, the performance gap between levels is small enough that cost and compatibility decide the choice. The full text, with the taxonomy figure, the method tables and the reference list, is at From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models.
References
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13Cai et al., Section 1: Introduction
arxiv.org
- 14Cai et al., Appendix B: Applications
arxiv.org
- 15Cai et al., Section 3.4: Evaluation
arxiv.org
- 16
- 17
- 18
- 19
- 20Cai et al., Limitations
arxiv.org
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.