Extending the Platonic Representation Hypothesis to dynamic inputs like videos and multi-caption text dramatically improves their alignment. This suggests that models are more aligned than previously thought, just that their inputs were impoverished. However, representations are not temporally-sensitive yet. We also analyze alignment as a zero-shot probe of video representations on downstream tasks.

VLMs achieve great performance on a wide array of benchmarks. However, the source of this generalization is poorly understood. How do we disentangle memorization from generalization? We evaluate train & val accuracy in 1-epoch setting (seen and unseen) and break examples into visual task-specific categories for analysis. This reveals inefficiencies in question format (Bounding) and misleading task accuracies (OCR).

We propose a framework for using many visual encoders covering broad visual categories like action recognition and spatial understanding as a unified visual encoder for video LLMs. This trend is exciting as it could allow our model to scale visual processing with the number of GPUs and run them all in parallel (sharding one expert per device) while still retaining similar runtimes to just a single expert.

A simple yet effective framework for adapting vision models trained on small, 224x224 images to larger images with larger context by using an LLM-style encoder to integrate context over larger regions than otherwise possible. We also proposed a set of effective benchmarks for reflecting such improvements on larger images.

We overcome the extra overhead of reversible transformers by parallelizing the backward pass using CUDA streams. This speeds up training for models in both vision and language, making them nearly as fast as the base models with incredible memory savings to boot.

Four new datasets measuring real-world distribution shifts, most well-known of which is ImageNet-R(enditions), as well as a new state-of-the-art data augmentation method that outperforms models pretrained with 1000x more labeled data.

In this work, we present an in-depth analysis of reversible transformers and demonstrate that they can be more accurate, efficient, and fast than their vanilla counterparts. We introduce a new method of reversible backpropagation which is faster and scales better with memory than previous techniques, and also demonstrate new results which show that reversible transformers transfer better to downstream visual tasks.

We propose a framework towards automating prompt tuning for learning preferences in text-to-image synthesis using reinforcement learning with human feedback.