on: Oct. 16, 2026
in: arXiv

Generative Representation Learning for Scalable Latent Tokenization

Alan Baade
Kyle Sargent
Vincent Sitzmann
Eric Ryan Chan ^†
Justin Johnson ^†
Li Fei-Fei ^†
Gordon Wetzstein ^†

†: equal advising

@article{baade2026repr,
    title = { Generative Representation Learning for Scalable Latent Tokenization },
    author = { Baade, Alan and 
               Sargent, Kyle and 
               Sitzmann, Vincent and 
               Chan, Eric Ryan and 
               Johnson, Justin and 
               Fei-Fei, Li and 
               Wetzstein, Gordon },
    year = { 2026 },
    booktitle = { arXiv },
}

Copy to Clipboard

We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional collapse and feature suppression in cases where commonly used methods like DINO and LeJEPA fail. Moreover, we show our representation learning technique is scalable; expending more compute leads to predictably better latent tokenization. For the first time, we perform a careful analysis of the combined flop efficiency of tokenizer and generative model and demonstrate that our approach is a compute-optimal improvement over direct diffusion for pixel-space generation, obtaining an FID-50K of 2.18 with JiT-L. Applied to ImageNet and the text-to-image dataset GPIC, we show tokenizer encoder scaling laws for diffusability, where extrapolating our approach achieves representations in 7x fewer FLOPs than DINOv2 on ImageNet and 8x fewer FLOPs than WebSSL on GPIC.