Flawed reasoning traces, model-heavy audio, and five-megabyte scene memory: twelve top-conf papers from October 10–16, 2026
Twelve first-submission arXiv papers from October 10–16, 2026, ten accepted at NeurIPS 2026 and one each at ICML 2026 and ICLR 2026, covering multimodal data mixtures, chain-of-thought distillation, self-correction policy learning, linear-probe diagnostics, coding and memory agents, multimodal agent safety, masked distributed attention, planar memory for video world models, fractional neural operators, RNA inverse folding and discrete mutual-information estimation.
The twelve papers in this issue were first submitted to arXiv between October 10 and October 16, 2026, the week that follows the October 3–9 window of the previous issue. An entry qualifies here only when its own arXiv submission record carries an explicit acceptance or proceedings statement naming NeurIPS, ICML, ICLR or CVPR.
Ten entries are NeurIPS 2026 papers, two of them spotlights and one in the Evaluations and Datasets track. One is a published ICML 2026 conference paper and one an ICLR 2026 poster, and both of those state the venue in their own records.
Of the 111 papers in this window whose records name an accepted venue, these twelve were picked to spread across directions and to keep papers that report their headline numbers. Workshop records, comments that merely name a conference, review-stage comments and rejected submissions were left out, and each first-submission date was checked against the paper's own arXiv record.
At a glance
Entries are grouped by area and, inside each section, ordered by posting date. The reading signal gives one verifiable number per paper rather than a verdict.
| Paper | Area | Status | Reading signal |
|---|---|---|---|
| Scaling Laws for Multimodal Data Mixtures | Multimodal pretraining | NeurIPS 2026 spotlight | Audio's compute-optimal split is 0.62 against 0.38, so a fixed budget should buy parameters rather than audio tokens. 1 |
| Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation | Post-training | Accepted to NeurIPS 2026 | Training only on incorrect reasoning traces keeps 94.7% of the accuracy gain that correct traces deliver. 2 |
| Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry | Reasoning post-training | Accepted at NeurIPS 2026 | A single pass scores 71.8% average accuracy, matching test-time reflection's 71.9% with 28% less latency. 3 |
| Is a Linear Probe Evidence of a Linear Representation? | Interpretability | Accepted at NeurIPS 2026 | DINOv3-B/16 decodes 12 position latents at ρ ≈ 0.94, yet its reverse predictivity is 0.18 against 0.02 for random weights. 4 |
| SpecForge: Recursive Self-Improvement for Coding Agents via Behavioral Specifications | Coding agents | Accepted to NeurIPS 2026 (main track) | Rewriting its own specifications inside a tree search lifts the DevEval+ solve rate to 94.9% with Claude-4.5-Sonnet. 5 |
| MemCode: Discrete Semantic Representations for Long-Term Agent Memory | Agent memory | Accepted by NeurIPS 2026 | On LongMemEval-S, MemCode reaches 78.40% overall accuracy and builds one dialogue's memory in about 83 seconds. 6 |
| MMA-SafetyBench: A Benchmark for Multimodal Agent Safety Evaluation | Agent safety | NeurIPS 2026 Evaluations and Datasets (poster) | Forged interface elements redirect nine agents on up to 87.96% of video tasks, and a safety prompt does not close the gap. 7 |
| FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention | Training systems | NeurIPS 2026 main conference | Structured masks run 45% to 141% faster in TFLOPs than FlashMask on one H100, and 1.80× to 3.71× faster than Megatron-Core under context parallelism. 8 |
| Persistent Planar Memory for Video World Models | Video world models | Accepted at NeurIPS 2026 | A full scene's memory is about 100,000 planes, roughly 5 MB, after 400 generated frames on DL3DV. 9 |
| Fractional is Better: Learnable Derivative Orders in Neural Operator Learning | Scientific machine learning | Accepted at ICML 2026 | Adding learnable fractional derivatives cuts Transolver's Burgers error at ν=0.001 from 18.99% to 4.63%. 10 |
| GOLD: Geometric Optimized Latent Diffusion for Structure-Aware RNA Inverse Folding | RNA design | NeurIPS 2026 spotlight | Sequence recovery reaches 0.6380 against SOLD's 0.5843, and AlphaFold3-scored RMSD falls from 13.65 Å to 12.31 Å. 11 |
| Information Estimation with Discrete Diffusion | Information estimation | Published at ICLR 2026 (poster) | At a true mutual information of 50, INFO-SEDD estimates 47.77 while the strongest baseline returns 17.27. 12 |
Pretraining data and post-training
1. Scaling Laws for Multimodal Data Mixtures
Area tag: Multimodal pretraining and data mixture scaling
arXiv: 2610.14423, first submitted October 10, 2026. 1
Authors / institutions: Aditi Khandelwal, Ayush K. Tarun, Yixuan Xu, Imanol Schlag, Golnoosh Farnadi, Siva Reddy, Luke Zettlemoyer and Antoine Bosselut, across Mila and McGill University, EPFL, ETH Zürich and the University of Washington. 13
Status: Accepted as a spotlight at NeurIPS 2026. 1
Problem: Frontier models are now pretrained jointly on text, vision and audio, and the recipes stay private. A practitioner choosing a mixture has no account of how much of each modality to take, or what one modality costs the others. 13
Method: The authors fit one scaling law per modality and add directional transfer terms that let each modality raise or lower the loss of the others. They run 268 experiments on mixture-of-experts models from 457 million to 8.3 billion parameters, on up to 150 billion tokens. 13
Comparison: Chinchilla and TGL for text, NMM for vision and SODA for audio, plus nine held-out predictions at an unseen 12.6-billion-parameter scale and two continual-pretraining runs. 13
Results: Vision scales fastest, with exponents α = 0.50 and β = 0.51 against 0.34 for text. Audio is model-heavy, at N ∝ C0.62 and D ∝ C0.38, so spare compute should buy audio parameters. Repeated vision data loses value quickly: the second epoch is worth 63% of fresh data and the fifth 3%. On the held-out runs the law predicts nine modality losses at a Pearson correlation of 0.999, with eight inside 2.6% of the observed value. 13
Resources: The record lists no code repository; the paper documents model sizes, learning-rate sweeps and batch-size sweeps. 1
Takeaway: A training team planning a three-modality mixture can start from the exponents rather than guess. The authors note the coefficients belong to their tokenizers, filtered corpora and sparse architecture, and that vision's interference with text (−0.558) stays unexplained at the mechanism level, since they find no modality-specific expert segregation. 13
2. Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation
Area tag: Reasoning post-training and distillation
arXiv: 2610.17607, first submitted October 13, 2026. 2
Authors / institutions: Yongcan Yu, Jian Liang, Kuangpu Guo, Yanbo Wang, Shuo Lu and Ran He at the National Laboratory of Pattern Recognition and MAIS, Institute of Automation, Chinese Academy of Sciences, with the University of Chinese Academy of Sciences; Lingxiao He and Meng Wang at Meituan. 14
Status: Accepted to NeurIPS 2026. 2
Problem: Long chain-of-thought distillation filters reasoning traces through rejection sampling, on the assumption that correct answers and broad prompt coverage are what transfer. 14
Method: The authors fine-tune on deliberately incorrect traces alone, then repeat the experiment on problems the base model has already mastered. They vary the ratio of wrong traces and the number of distinct prompts, and compare the standard −log p loss with the probability loss −p across three trace-length tiers of 426, 1,337 and 9,062 tokens. 14
Comparison: Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen2.5-7B and Llama-3.1-8B-Instruct, on AIME 2024, AMC23, Minerva Math, Olympiad Bench and MATH, with GPQA-Diamond, MMLU-Pro and LiveCodeBench added for breadth. 14
Results: Incorrect traces recover 97.7% of the absolute accuracy that correct traces produce and 94.7% of their gain over the base model. The gap stays near one point outside mathematics. With only 512 correct samples, adding wrong ones at a 1:5 ratio lifts the five-benchmark math average from 20.4% to 26.4%. 14
Resources: Training and evaluation code is public at the project repository. 14
Loading content card…
Takeaway: Filtering for answer correctness buys less than the cost of the filter suggests, and the loss matters more as traces get longer. The −p loss wins on simple traces and collapses on complex ones, where −log p keeps improving. The models also gain 21.5 points over Qwen3-4B-Base on the very problems whose traces were flawed. 14
Reasoning and policy optimisation
3. Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry
Area tag: Reasoning post-training and self-correction
arXiv: 2610.20350, first submitted October 14, 2026. 3
Authors / institutions: Ren Zhuang, Ben Wang and Shuifa Sun at Hangzhou Normal University. 15
Status: Accepted at NeurIPS 2026. 3
Problem: Preference optimisation ranks final answers, and trajectory-level reinforcement learning maximises a return over the whole attempt. Neither credits the single edit that turns a wrong reasoning state into a better one. 15
Method: Each answer gets a learned continuous coordinate, and an energy function orders the resulting lifted states. A shaped reward drives energy down net of the cost of editing and stepping, and an internalisation loss folds the low-energy answers back into a single-pass generator. 15
Comparison: DPO, Step-DPO, GRPO, GDPO and SCoRe on Qwen3.5-9B, Qwen3.5-27B and Qwen3.5-35B-A3B, over MATH-500, AIME25, GPQA, LiveCodeBench, ZebraLogic and MMLU-Pro, with component ablations and parameter-matched coordinate controls. 15
Results: On Qwen3.5-27B the single-pass policy averages 71.8% at pass@1, against 65.9% for the base model, 70.0% for GRPO and 70.8% for GDPO. Test-time reflection reaches 71.9% at 4.3 seconds more latency and 440 more tokens per query. The gain holds at 60.8% from a 54.8% base at 9B and 72.6% from 67.8% at 35B-A3B. 15
Resources: Code is public at the project repository. 15
Loading content card…
Takeaway: The method lets a model keep the benefit of revising and still serve one pass. Dropping the lifted coordinate costs 0.9 points, the progress reward 1.8 and internalisation 1.5. The training budget is real: 2,983 GPU hours against 1,218 for DPO, and the evaluation stops at the Qwen3.5 family. 15
Interpretability
4. Is a Linear Probe Evidence of a Linear Representation?
Area tag: Interpretability methodology
arXiv: 2610.22655, first submitted October 15, 2026. 4
Authors / institutions: Nadir Kutluozen and David Klindt at Cold Spring Harbor Laboratory; Christian Internò at Bielefeld University. 16
Status: Accepted at NeurIPS 2026. 4
Problem: The standard test of the linear representation hypothesis trains a linear probe and calls a factor represented when accuracy is high. Asymptotically any sufficiently rich feature map admits a linear readout, so the test cannot separate two very different geometries. 16
Method: The authors propose three asymmetric diagnostics: reverse predictivity, spectral concentration, and out-of-distribution generalisation along the directions orthogonal to the probe. They validate the diagnostics on simulated worlds with known latents, then apply them to pretrained vision encoders. 16
Comparison: DINOv3 in three sizes alongside DINOv2, DeiT and CLIP on dSprites, 3D Shapes, MPI3D and CelebA, with randomly initialised weights of the same architecture as the null. 16
Results: DINOv3-B/16 recovers all 12 position latents in the two simulated worlds at mean ρ ≈ 0.94 and R² of 0.88, and random initialisation recovers little beyond person position. Reverse predictivity still separates the two, at R² of 0.18 for pretrained weights against 0.02 for random. Pretrained DINOv3 needs its top 17 principal components to hold 90% of the latent-predictable variance; its random initialisations need 55 to 59. 16
Resources: Code and the simulated-world splits are released at the project repository. 16
Loading content card…
Takeaway: A team reporting a high probe accuracy can add two cheap numbers, reverse predictivity and the component count for 90% of the variance, and say which geometry it found. The authors place DINOv3 between a random-feature null and a true linear world model. The evidence covers synthetic latents and four standard benchmarks; language models are untested. 16
Agents
5. SpecForge: Recursive Self-Improvement for Coding Agents via Behavioral Specifications
Area tag: Coding agents and self-improvement
arXiv: 2610.15708, first submitted October 11, 2026. 5
Authors / institutions: Yutong Cheng and Peng Gao at Virginia Tech; Haifeng Chen and Wei Cheng at NEC Laboratories America. 17
Status: Accepted to NeurIPS 2026 main track. 5
Problem: A frozen coding agent can improve without weight updates by rewriting the documentation it works from. Program entities depend on one another, so a better specification for one entity can break several others, and the loop oscillates instead of converging. 17
Method: SpecForge runs a multi-frontier tree search over specification states. A Pareto archive keeps complementary states, refinement goes callee-before-caller along the dependency graph, and separate branches explore different subsets of the test failures. 17
Comparison: DevEval+ with 50 rollouts per setting, against static documentation, the ACE prompt optimiser, Cursor and Claude Code with topological ordering and test feedback, plain iterative refinement, and DocSearch. 17
Results: SpecForge reaches a 94.9% solve rate with Claude-4.5-Sonnet, against 61.9% for Claude Code and 52.1% for Cursor with the same augmentation. It also leads on Qwen3-235B-A22B at 88.9%, GPT-4o at 90.9% and Gemini-3.1-Pro at 91.0%. The average margin over the strongest baseline is 5.7 percentage points. 17
Resources: Code and data are public at the project repository. 17
Loading content card…
Takeaway: The reusable part is the search discipline, which needs only a test suite and one specification file per entity. The rewritten specifications also beat the benchmark's own requirement documents on cross-language translation and new-feature implementation. The benchmark is Python modules with execution-verified tests, and the authors' DocSearch re-run differs from the original paper by 5.6 points. 17
6. MemCode: Discrete Semantic Representations for Long-Term Agent Memory
Area tag: Agent memory and retrieval
arXiv: 2610.19472, first submitted October 14, 2026. 6
Authors / institutions: Xin Li, Liang Hu, Duoqian Miao, Ming Peng and Qi Zhang at Tongji University; Qi Zhang also at WorldMind. 18
Status: Accepted by NeurIPS 2026. 6
Problem: Long-term memory systems store dialogue fragments and retrieve them by nearest neighbour in a continuous embedding space. The stored units depend on context, and retrieval misses evidence that is logically related but worded differently. 18
Method: MemCode rewrites dialogue into self-contained atomic memory units, maps them into a learnable codebook, and uses codebook co-occurrence as a second retrieval path beside vector search. Writes happen online, and consolidation runs asynchronously offline. 18
Comparison: Mem0, LangMem, A-MEM, MemoryOS and LightMem on LongMemEval-S and LoCoMo, with GPT-4o-mini and Qwen3-30B-A3B-Instruct-2507 as the answering backbones. 18
Results: MemCode reaches 78.40% overall accuracy on LongMemEval-S with GPT-4o-mini and 74.60% with Qwen3, after asynchronous consolidation. On LoCoMo it reaches 79.55% and 76.88%. One dialogue's memory takes about 74 seconds to build, or 83 with consolidation, which is a third to a twenty-first of LightMem's runtime. 18
Resources: The record lists no code repository. 6
Takeaway: Two retrieval paths buy better binding of facts to the person who said them: single-assistant questions reach 92.86% against LightMem's 32.14%. Continuity across sessions is the price, at 65.41% against LightMem's 71.74% on multi-session questions. 18
7. MMA-SafetyBench: A Benchmark for Multimodal Agent Safety Evaluation
Area tag: Agent safety and evaluation
arXiv: 2610.19384, first submitted October 14, 2026. 7
Authors / institutions: Yuke Wang and Benlei Cui as co-first authors, with Shen Pang, Xuemei Dong, Longtao Huang, Hui Xue, Yuwen Zhai, Junjie Li, Jingqun Tang and Haiwen Hong, at the Yuvion team of Alibaba Group and at Zhejiang Gongshang University's School of Statistics and Data Science. 19
Status: Accepted to the NeurIPS 2026 Evaluations and Datasets track as a poster. 7
Problem: A computer-use agent reads the screen to decide what to do next, so a forged system alert or a spoofed button is an instruction it may obey. Output-level safety evaluations score generated text and miss whether an action was redirected. 19
Method: The authors insert reproducible visual payloads at injection points found by walking each task's reasoning-action trace, and name the failure mode visual semantic hijacking. A 1–3–5 rubric on every trajectory yields an attack success rate and a broader semantic compromise rate. 19
Comparison: Nine victim models and 1,083 adversarial trajectories over web automation, desktop GUI control, mobile navigation, document understanding and video processing, with clean-task and neutral-prompt controls. 19
Results: The worst single cell is GLM-4.6V on video at 87.96% attack success, and its semantic compromise rate on graphical interfaces reaches 98.09%. Mean attack success is 69.17% on video and 63.69% on graphical interfaces, against 43.00% on web tasks. Claude 4.5 Sonnet is the most resistant on the web at 30.50%, and no model stays low across all five domains. 19
Resources: Code and the dataset are released at the project repository and on Hugging Face. 19
Loading content card…
Takeaway: The gap between the two rates is the finding: on graphical interfaces GLM-4.6V adds 13.4 points of cases where the agent followed the forged instruction without hitting the target. A safety-priority prompt lowers attack success in every reported cell and in none of them to zero, which the authors read as an argument for authenticated instruction channels. 19
Training systems
8. FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
Area tag: Efficient training systems
arXiv: 2610.17901, first submitted October 13, 2026. 8
Authors / institutions: Guoxia Wang, Qianyue He, Siming Wu, Haoyang Xie, Qiwen Bao, Jinle Zeng, Jiabin Yang, Tian Wu and Dianhai Yu at Baidu. 20
Status: Published as a main conference paper at NeurIPS 2026. 8
Problem: Long-context training uses structured attention masks to encode document boundaries, prefix-language-modelling and sliding windows. Under multi-node context parallelism the mask bookkeeping, rather than the matrix multiplication, becomes the bottleneck. 20
Method: FlashMask-3 keeps four boundary vectors per query block and shows that under any context-parallel assignment, locating a rank's mask reduces to a clip-shift-max on those vectors with no communication. On top of that it adds a Hopper kernel with three warp-specialised roles, a sparsity-aware load balancer and mask-driven communication pruning. 20
Comparison: FlashMask, FlexAttention and MagiAttention on one H100 across 12 structured masks and context lengths from 4K to 128K, then Megatron-Core context parallelism in both ring and all-gather variants. 20
Results: Achieved TFLOPs rise 45% to 141% over FlashMask, 34% to 71% over FlexAttention and 2% to 44% over MagiAttention. At context-parallel degrees of 4 to 32 with 8K tokens per device, compute-communication overlap efficiency reaches 31% to 75%. End-to-end training of a 118B-A14B mixture-of-experts model at 128K context on 128 H100s gains 7.3% to 11.0% throughput, with attention forward and backward 24.5% to 44.9% faster. 20
Resources: Source code is public at the project repository. 20
Loading content card…
Takeaway: A team already paying for context parallelism can treat mask handling as a kernel problem with a communication-free localisation step. The kernel targets Hopper and the column-wise sparse mask family, so the speedups are tied to that hardware and that pattern. 20
Video world models
9. Persistent Planar Memory for Video World Models
Area tag: Video generation and 3D memory
arXiv: 2610.23795, first submitted October 16, 2026. 9
Authors / institutions: Yuze He at Tsinghua University and Ant Group; Bin Tan, Yujun Shen and Nan Xue at Ant Group; Zelin Gao and Yinghao Xu at the Hong Kong University of Science and Technology; Yong-Jin Liu at Tsinghua University. 21
Status: Accepted at NeurIPS 2026. 9
Problem: A video world model that generates an explorable environment needs to remember what it has produced. Pure-video models keep no explicit 3D state, and per-pixel point clouds are heavy to carry and to condition on. 21
Method: PPMem makes a set of oriented planes the memory, at seven parameters per plane. A video diffusion transformer gains a planar prediction head and a depth token fusion module, so each autoregressive chunk emits video and planar geometry together, and the merged planes condition later chunks through rendered depth. 21
Comparison: Gen3C, Voyager, VMem and SPMem on DL3DV and RealEstate10K at 49 and 121 frames, plus ablations over the plane head, the fusion module and the merging procedure. 21
Results: On DL3DV at 49 frames PPMem scores 16.99 PSNR and 0.335 LPIPS against 15.74 and 0.474 for Gen3C. On RealEstate10K it reaches 22.46 PSNR and 0.168 LPIPS against 21.93 and 0.238, and the lead holds at 121 frames. A full scene after about 400 frames holds roughly 100,000 planes, about 5 MB, three orders of magnitude below a per-pixel point cloud. 21
Resources: A project page carries generated scenes and exported geometry; the record lists no code repository. 9
Takeaway: Planes are a compact structural prior for indoor and outdoor scenes, and the memory exports as surfaces with metric depth and normals. Removing plane conditioning collapses depth accuracy to 0.436 absolute relative error and drops PSNR from 14.91 to 14.19. The model is trained at 576×1024 on scenes dominated by walls, floors and facades. 21
Scientific machine learning
10. Fractional is Better: Learnable Derivative Orders in Neural Operator Learning
Area tag: Neural operators and scientific machine learning
arXiv: 2610.20363, first submitted October 14, 2026. 10
Authors / institutions: Fares B. Mehouachi and Saif Eddin Jabari at New York University Abu Dhabi. 22
Status: Accepted at ICML 2026; the record states the acceptance and the paper carries the proceedings venue line. 10
Problem: Neural operators learn PDE solutions from raw coordinate and value pairs. Derivatives of the input appear explicitly in the solution operator, so a network given only values has to differentiate implicitly. 22
Method: The authors prove that supplying derivative features improves the approximation bound from O(W−(s−m)/d) to O(W−s/d). They then show the best derivative order sits strictly below the PDE order, deriving the gap from a bias-variance tradeoff in spectral space, and make the order learnable per problem. 22
Comparison: Five PDE benchmarks and four backbones, FNO, TFNO, LocalNO and Transolver, each with and without the augmentation, in a clean regime and under noise at σ = 0.05. 22
Results: The augmentation improves all 40 backbone, dataset and noise cells. The largest gain is Transolver on Burgers at ν=0.001, where relative L2 error falls from 18.99% to 4.63%; FNO on the same problem goes from 1.138% to 0.837%. Gains survive additive noise up to σ = 0.3, and a matched-parameter control places them on the derivative input rather than on added capacity. 22
Resources: Code is public at the project repository. 22
Loading content card…
Takeaway: A team training a neural operator can add fractional derivative features to an existing backbone and learn one or two extra scalars. The learned orders move toward the PDE order as noise falls, which is the tradeoff the authors derive: below a sample-to-noise threshold the exact order is never optimal. 22
11. GOLD: Geometric Optimized Latent Diffusion for Structure-Aware RNA Inverse Folding
Area tag: RNA design and scientific machine learning
arXiv: 2610.22992, first submitted October 16, 2026. 11
Authors / institutions: Qi Si, Xuyang Liu and Penglei Wang as co-first authors, with Shuo Su, Limei Han, Mengzhu Li, Xin Guo, Yuan Qi and Yuan Cheng, at the Shanghai Academy of Artificial Intelligence for Science and Shanghai Jiao Tong University. 23
Status: Accepted as a spotlight paper at NeurIPS 2026. 11
Problem: RNA inverse folding conditions on the 3D backbone and emits a one-dimensional sequence. That drops the continuous side-chain orientations, which carry much of a folded RNA's stability. 23
Method: A GeoVAE projects discrete sequence and continuous side-chain angles into one latent manifold, and diffusion runs over that manifold. A training-free sampling step decodes intermediate sequences into structures and adds gradients from physical constraints. 23
Comparison: PiFold, RDesign, gRNAde, RiboDiffusion and SOLD on the authors' own benchmark and on the out-of-distribution CASP15 targets, scored by sequence recovery, secondary-structure recovery and RMSD from RhoFold and AlphaFold3. 23
Results: On the in-distribution benchmark GOLD reaches 0.6380 sequence recovery against 0.5843 for SOLD, and 0.8186 secondary-structure recovery against 0.7366. AlphaFold3 RMSD falls from 13.65 Å to 12.31 Å, averaged over five seeds. On CASP15 it reaches 0.6624 recovery against 0.5877 and 26.18 Å against 27.74 Å. 23
Resources: Code and data are public at the project repository. 23
Loading content card…
Takeaway: Adding side-chain geometry to the generative space moves every ranking metric in the right direction, and the authors attribute the gain to the shared manifold rather than to the guidance step alone. Absolute fold accuracy outside the training distribution stays near 26 Å, so a designed sequence still needs experimental checking. 23
Information estimation
12. Information Estimation with Discrete Diffusion
Area tag: Information theory and estimation
arXiv: 2610.23404, first submitted October 16, 2026. 12
Authors / institutions: Alberto Foresti, Giulio Franzese and Pietro Michiardi at the Department of Data Science, EURECOM, in Biot, France. 24
Status: Published as a conference paper at ICLR 2026 as a poster; the record states the venue and the paper carries the same line. 12
Problem: Mutual information on discrete data is usually estimated by embedding the data into a continuous space and running a neural estimator built for continuous distributions. The embedding needs hand engineering and degrades in high dimensions. 24
Method: The authors build on discrete diffusion and continuous-time Markov chain theory to estimate the Kullback–Leibler divergence directly over the discrete support, which turns into a mutual information estimate. The design is lightweight and can be attached to a pretrained model. 24
Comparison: NWJ, MINE, SMILE, three DIME variants, MINDE and a GAN-based estimator on synthetic data with known MI, at matched vector lengths from 10 to 50. 24
Results: At a true mutual information of 50, INFO-SEDD estimates 47.77 with a standard deviation of 1.18. The next closest competitors reach 32.60 and 17.27, and the variational estimators stay near 6 or 7 whatever the true value. The authors bound the estimator's deviation from the true divergence and show consistency up to an exponentially small failure term. 24
Resources: Code is public at the project repository. 24
Loading content card…
Takeaway: Discrete data no longer needs the embedding step before an information-theoretic measure can be computed. The cost is a diffusion model trained per variable pair, which is what the estimator is built on. The paper applies it to promoter motif discovery, summarisation-model selection and Ising-model entropy. 24
What to open first
- Planning a multimodal pretraining run: the data-mixture scaling laws (2610.14423) give per-modality exponents and a fitted law checked against nine held-out predictions. 1
- Running a distillation pipeline: the behavioural analysis of long-CoT distillation (2610.17607) if you are paying for rejection sampling, and LSPO (2610.20350) if the reflection stage is where your serving cost goes. 14
- Reading interpretability claims: the probe paper (2610.22655) before you cite a linear probe as evidence of a linear representation. 4
- Building or auditing an agent: SpecForge (2610.15708) for the specification search loop, MemCode (2610.19472) if memory across sessions is the weak point, and MMA-SafetyBench (2610.19384) before shipping anything that reads a screen. 5
- Training long-context models: FlashMask-3 (2610.17901) if structured masks and context parallelism both appear in your profile. 8
- Generating explorable video or solving PDEs: PPMem (2610.23795) for a memory that exports as geometry, and the fractional-derivative operators (2610.20363) for an input change that improved all four backbones tested. 9
- Designing RNA or estimating dependence in discrete data: GOLD (2610.22992) for side-chain-aware inverse folding, and INFO-SEDD (2610.23404) when high mutual information would saturate a continuous-space estimator. 11
References
- 1
- 2
- 3
- 4
- 5SpecForge arXiv record
arxiv.org
- 6MemCode arXiv record
arxiv.org
- 7MMA-SafetyBench arXiv record
arxiv.org
- 8FlashMask-3 arXiv record
arxiv.org
- 9PPMem arXiv record
arxiv.org
- 10
- 11GOLD arXiv record
arxiv.org
- 12INFO-SEDD arXiv record
arxiv.org
- 13
- 14Behavioral CoT distillation paper
arxiv.org
- 15
- 16Linear probe diagnostics paper
arxiv.org
- 17SpecForge paper
arxiv.org
- 18MemCode paper
arxiv.org
- 19MMA-SafetyBench paper
arxiv.org
- 20FlashMask-3 paper
arxiv.org
- 21PPMem paper
arxiv.org
- 22Fractional neural operators paper
arxiv.org
- 23GOLD paper
arxiv.org
- 24INFO-SEDD paper
arxiv.org
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.