Five diffusion papers worth reading today (June 1, 2026)
Monday's digest covers the ArXiv weekend gap (Sat+Sun+Mon bundled, 176 cs.CV + 319 cs.LG scanned). Five papers selected: SANA-Streaming (NVIDIA/MIT, 24 end-to-end FPS real-time video editing on RTX 5090), Memorization "Slop" (Imperial College, prototypical examples are memorized first — point-level deduplication offers no meaningful privacy guarantee), TunerDiT (TU Munich, training-free multi-event video generation via intrinsic DiT turning points + new MEve benchmark), FREUD (CompVis/LMU Munich, uncertainty-preserving rectified flow transformer for SEVIR SOTA precipitation nowcasting, code released), and CameraNoise (Fudan, geometry-guided noise warping for faithful camera control in video diffusion). All five are preprints; FREUD is the only day-one code release.
리서치 브리프
Monday's digest covers the weekend gap — ArXiv's Sat+Sun+Mon submissions bundled. Of 176 cs.CV and 319 cs.LG new listings scanned, 19 genuine diffusion-model preprints remained after filtering already-covered IDs; five made the cut. The batch splits into three zones: real-time video systems (SANA-Streaming), theoretical memory analysis with direct privacy implications (Memorization "Slop"), training-free multi-event video generation (TunerDiT), operational weather nowcasting with a code release (FREUD), and geometry-aware camera control through noise-space encoding (CameraNoise).
Ranking signals: first consumer-GPU real-time video editing result with end-to-end FPS numbers (SANA-Streaming), theoretical contribution with an immediately deployable privacy insight (Memorization), training-free method on an existing benchmark suite with a new evaluation protocol (TunerDiT), SEVIR SOTA with open code from the Stable Diffusion lab (FREUD), noise-space camera control formulation that cleanly solves a long-standing coordinate-gap problem (CameraNoise).
1. SANA-Streaming: real-time video editing at 1280×704 on a single RTX 5090
ArXiv: 2605.30409 | Yuyang Zhao, Yicheng Pan, Qiyuan He, Jincheng Yu, Junsong Chen, Tian Ye, Haozhe Liu, Enze Xie, Song Han (MIT / NVIDIA) | cs.CV, cs.AI
Peer-review status: Preprint.
Real-time video-to-video editing has a structural tension: temporal consistency requires attending over many frames, but inference throughput demands the opposite. Prior approaches sacrifice one for the other. SANA-Streaming is a system-algorithm co-design that addresses both simultaneously. 1
Three interlocking components carry the method:
- Hybrid Diffusion Transformer — softmax attention in a subset of blocks handles local temporal modeling where it matters; linear layers carry the rest for throughput
- Cycle-Reverse Regularization — flow matching is used to predict the source frame from the edited output, enforcing temporal consistency without requiring paired long-form edit sequences
- Mixed-Precision Quantization (MPQ) + GDN kernels — system-level co-optimization targeting NVIDIA Blackwell (RTX 5090) Tensor Core utilization
As Song Han's team reports: "The resulting system achieves real-time 1280 × 704 resolution editing at 24 end-to-end FPS on a single RTX 5090 GPU, with the DiT core running at 58 FPS." 1
Code/resources: No repository confirmed at time of writing.
Why read it: The 24 end-to-end FPS figure puts this in a different category from throughput-optimized papers that report DiT-only latency — end-to-end includes encode, decode, and I/O, which is the number that matters for interactive deployment. The Cycle-Reverse Regularization mechanism is the architecturally novel piece: it does not need paired edit datasets, which makes scaling to diverse content tractable. For groups building on-device or low-latency video editing pipelines, this is the current published bar. The open question is whether the MPQ approach is Blackwell-specific or generalizable to other NVIDIA generations.
2. Memorization "Slop": diffusion models preferentially memorize prototypical examples
ArXiv: 2605.30642 | Marta Aparicio Rodriguez, Anastasia Borovykh, Grigorios A. Pavliotis, Daniel J. Korchinski (Imperial College London) | cs.LG
Peer-review status: Preprint.
The standard assumption in diffusion model memorization work is that atypical or rare samples pose the greatest privacy risk — they appear less often, so when the model reproduces them it is doing something suspicious. This paper inverts that assumption. 3
The team trains diffusion models on data from the Random Hierarchy Model (RHM) — a generative process that allows precise control over which patterns appear as common sub-structures. The finding: samples composed of frequent sub-strings are memorized first, not last. Even when every training data point is globally unique (point-level deduplication applied), the model memorizes by learning those common sub-features and over-reproducing them — a behavior the authors label "slop," defined as regression toward blandness through excessive generation of shared priors. 3
As the abstract states directly: "deduplication at the data point level does not provide a meaningful privacy guarantee." 3
The authors also identify three conditions that delay memorization: fat-tailed data distributions (more atypical samples), higher-level abstraction diversity, and smaller model capacity. The capacity result is counterintuitive: larger models enter the memorization regime faster, with a shorter window during which generalization precedes memorization.
CelebA image experiments confirm the RHM-derived results: faces composed of statistically typical attributes are memorized earlier than unusual-looking faces. The implication is that diversity-at-the-latent-feature level, not at the data-point level, is what determines privacy exposure.
Code/resources: No repository listed.
Why read it: This paper changes how to think about training data auditing and deduplication. If the risk comes from common sub-features rather than rare whole-samples, then standard dataset-level or image-level deduplication pipelines are measuring the wrong thing. The RHM framework is clean enough to derive quantitative predictions, and the CelebA validation gives a direct image-domain anchor. Groups using diffusion models in production for any application where training data privacy is a consideration — medical imaging, personalization, fine-tuning on proprietary data — should treat this paper as a reason to revisit their current deduplication and data diversity practices.
3. TunerDiT: training-free multi-event video generation via DiT denoising turning points
ArXiv: 2605.31590 | Ruotong Liao, Guowen Huang, Qing Cheng, Guangyao Zhai, Lei Zhang, Xun Xiao, Thomas Seidl, Daniel Cremers (TU Munich), Volker Tresp (LMU Munich / Siemens) | cs.CV, cs.AI
Peer-review status: Preprint.
Text-to-video models handle single-event prompts adequately, but multi-event prompts ("first X happens, then Y, then Z") degrade into either blended sequences where events overlap or rigid cuts without natural transitions. Existing solutions require fine-tuning, additional conditioning modules, or paired multi-event training data. 5
TunerDiT's entry point is an empirical observation about DiT denoising dynamics: the trajectory contains intrinsic turning points — timesteps at which the role of text conditioning shifts from determining global layout to specifying fine-grained detail. The authors identify these turning points by probing the denoising trajectory's sensitivity to text perturbations and find they are consistent across video DiT architectures. 5
Two guidance mechanisms exploit this:
- Event-Partitioned Masking — enforces event boundaries in spatial attention at specific timesteps, while allowing overlap bands at transitions
- Cross-Event Prompt Fusion — injects adjacent-event semantics during the fine-grained phase after the turning point, enabling smooth transitions without erasing event separation
The team introduces MEve, a new benchmark for multi-event video evaluation covering up to 4 sequential events across multiple quality dimensions. TunerDiT achieves state-of-the-art results across all 8 MEve metrics, with text-alignment improvements growing as event count increases — the exact regime where competing methods fail. 5
Code/resources: No repository listed. No project page confirmed at time of writing.
Why read it: The turning-point discovery is the paper's most transferable contribution — it characterizes a structural property of video DiTs that appears to hold across architectures, and that property is independently useful for understanding how these models process sequential instructions. The training-free approach means it can be applied to any existing video DiT (CogVideoX, HunyuanVideo, etc.) without retraining. The MEve benchmark is a practical contribution as well: the field has been evaluating multi-event generation with ad hoc visual inspection, and having a structured multi-dimension benchmark changes what can be rigorously compared.
4. FREUD: rectified flow transformer for probabilistic weather nowcasting, SOTA on SEVIR
ArXiv: 2605.31204 | Johannes Schusterbauer, Jannik Wiese, Nick Stracke, Timy Phan, Björn Ommer (CompVis, LMU Munich) | cs.CV
Peer-review status: Preprint. Code: github.com/CompVis/weather-rf. Project page: compvis.github.io/weather-rf.
Diffusion-based weather nowcasting approaches typically apply a deterministic compression stage before any generative modeling — encoding observation sequences into a fixed latent, then running diffusion on top. That design discards uncertainty information during encoding: what gets passed to the generative stage is a point estimate, not a distribution. For extreme weather events, where the aleatoric uncertainty (intrinsic randomness of physical systems) is largest, this is exactly the wrong trade-off. 7
FREUD (Frame-wise Encoder and United Decoder) separates the problem by design. A frame-wise encoder processes each observation independently, which supports continuous forecast updates as new radar frames arrive. A unified video decoder runs a compact rectified flow transformer over the latent sequence, preserving the distributional structure rather than collapsing it. At inference, the team runs an ensemble of predictions to capture aleatoric uncertainty across possible precipitation outcomes. 7
The model achieves SOTA on the SEVIR benchmark (Storm EVent ImageRy, the standard radar-based precipitation nowcasting dataset) for precipitation prediction. Performance scales further with both model scaling and test-time compute scaling, which is the same pattern seen in language model inference scaling. 7
Code/resources: github.com/CompVis/weather-rf
콘텐츠 카드를 불러오는 중…
Why read it: The authorship is the first signal: Björn Ommer's CompVis group at LMU Munich originated Stable Diffusion (LDM, 2022), and FREUD applies the same latent-space philosophy to a high-stakes physical prediction task. The uncertainty-preserving encoding is the genuine methodological contribution — keeping distributional information alive through the entire pipeline rather than collapsing it at the encoder is a design choice with implications beyond weather. The test-time scaling result is noteworthy: it suggests these rectified flow models obey the same scaling laws now being exploited in LLMs, which opens a door to compute-optimal inference for physical forecasting tasks where ground-truth is expensive to obtain.
5. CameraNoise: faithful camera control via geometry-guided noise warping
ArXiv: 2605.30774 | Haoyu Zhao, Jiaxi Gu, Haoran Chen, Qingping Zheng, Yeying Jin, Hongyi Yang, Junqi Cheng, Yuang Zhang, Zenghui Lu, Huan Yu, Jie Jiang, Peng Shu, Zuxuan Wu, Yu-Gang Jiang (Fudan University) | cs.CV
Peer-review status: Preprint. Project page: gulucaptain.github.io/CameraNoise.
Existing camera-controlled video diffusion methods inject camera parameters numerically — as pose matrices, rotation/translation vectors, or ControlNet-style conditioning. The problem is that a camera pose matrix is an abstract coordinate, not a visual signal, and diffusion model backbones are trained on visual data. The gap between "a rotation matrix says the camera pans left" and "the pixel array shifts in a specific correlated pattern" has to be bridged by the model, which is doing so imperfectly: methods that inject poses directly produce temporal flickering and geometric inconsistencies, especially for long-range camera motions. 8
CameraNoise moves the camera signal into the noise space rather than the conditioning space. The core mechanism is Geometry-guided Reprojection Flow (GRFlow): given only the camera intrinsics and pose sequence — no optical flow estimation required — GRFlow computes per-pixel displacement vectors describing how each frame's content should shift under the specified camera motion. These displacements directly warp the initial diffusion noise, creating a noise tensor that already encodes the camera trajectory before the denoising process begins. 8
As the authors state: "CameraNoise embeds camera poses directly into the noise space. This decouples motion from scene appearance while faithfully preserving trajectory dynamics." 8
Lie algebra optimization reduces jitter from discretization error in the reprojection computation. Dynamic perturbation augmentation at inference time improves robustness to out-of-distribution camera paths. The method establishes a one-to-one mapping between pose and noise, which is what makes the control precise rather than approximate. 8
Code/resources: No code repository confirmed. Project page above.
콘텐츠 카드를 불러오는 중…
Why read it: The core idea — encode camera motion into noise rather than conditioning — is architecturally clean and conceptually satisfying. Prior approaches that inject pose directly are working against the model's own learned prior; CameraNoise works with it. GRFlow requiring only camera intrinsics and extrinsics (no optical flow network) keeps the method lightweight and portable. Yu-Gang Jiang's group at Fudan has a track record of usable video generation work, and the 14-author collaboration suggests substantial engineering investment behind the paper. The limitation to note is that the noise-warping approach assumes scenes where camera motion dominates over scene-internal motion — dynamic foreground objects will interact with the warped noise in ways the theory does not fully characterize.
Quick reference
| Paper | ArXiv ID | Core method | Venue | Code |
|---|---|---|---|---|
| SANA-Streaming | 2605.30409 | Hybrid DiT + Cycle-Reverse Regularization + MPQ; 24 FPS end-to-end at 1280×704 on RTX 5090 | Preprint | Not released |
| Memorization "Slop" | 2605.30642 | RHM-based analysis; prototypical examples memorized first; point-level dedup insufficient | Preprint | Not released |
| TunerDiT | 2605.31590 | Turning-point steering + MEve benchmark; SOTA across 8 multi-event metrics; training-free | Preprint | Not released |
| FREUD | 2605.31204 | Frame-wise uncertainty-preserving encoder + unified rectified flow decoder; SEVIR SOTA | Preprint | GitHub |
| CameraNoise | 2605.30774 | GRFlow noise warping from camera params only; one-to-one pose-noise mapping | Preprint | Not released |
This batch's connecting thread is the noise space. SANA-Streaming controls temporal consistency through how the model is regularized to relate edited and source noise. FREUD preserves distributional uncertainty rather than collapsing it during encoding. CameraNoise literally rewrites the noise tensor to encode geometry before denoising starts. The Memorization paper's result can be interpreted the same way: it is the structure of the training distribution's noise floor — which sub-features are common — that determines what gets memorized. TunerDiT is the outlier: its contribution is about the denoising trajectory's temporal structure rather than noise initialization. All five are preprints; FREUD is the only one with code available at submission.
Cover image: AI-generated illustration
참고 출처
- 1SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformer (arXiv 2605.30409)
- 2SANA-Streaming (arXiv 2605.30409)
- 3Diffusion Models Preferentially Memorize Prototypical Examples (arXiv 2605.30642)
- 4Memorization paper (arXiv 2605.30642)
- 5TunerDiT: Training-free Progressive Steering of Diffusion Transformer for Multi-Event Video Generation (arXiv 2605.31590)
- 6TunerDiT (arXiv 2605.31590)
- 7Probabilistic Precipitation Nowcasting with Rectified Flow Transformers (arXiv 2605.31204)
- 8CameraNoise: Enabling Faithful Camera Control in Video Diffusion through Geometry-Flow-Guided Noise Warping (arXiv 2605.30774)
- 9CameraNoise (arXiv 2605.30774)
이 콘텐츠를 둘러싼 관점이나 맥락을 계속 보강해 보세요.