Ramen picks its neighbors for mixed-domain VLMs

Ramen is a test-time adaptation method for vision-language models that assumes the incoming stream may mix domains. Instead of adapting on every recent sample together, it builds a small support set that is close to the current image in embedding space and balanced across pseudo-classes. 1

The paper targets a failure mode in ordinary test-time adaptation: one shared update can average together incompatible domains, such as different lighting, capture styles, or platforms. Ramen stores image embeddings and sample-level gradients in class-split FIFO queues, retrieves nearby samples from each queue, aggregates the cached gradients, updates normalization layers, runs inference, and resets the model. 1

This matters when a classifier sees a mixed stream rather than a clean batch: camera feeds, photo libraries, inspection images, or user uploads can change domain between adjacent inputs. The useful product question is not "can the model adapt?" It is "can it adapt without letting the last domain distort the next prediction?"

Start with openai/clip-vit-base-patch32 on Hugging Face, a public CLIP checkpoint with a Transformers example. The model card describes it as a research output and warns that task-specific testing is needed before deployment. 2

On the paper's mixed-domain benchmarks, Ramen improves average accuracy over the strongest baseline by +1.3% on CIFAR-10-C, +3.4% on CIFAR-100-C, +2.5% on ImageNet-C, and +0.6% on DomainNet. It also reports a 490x speedup over a naive implementation on CIFAR-100-C. Those are paper results, not a guarantee for your stream. 1

The practical takeaway: measure domain drift first, then let adaptation see a deliberately chosen neighborhood rather than the whole recent stream.