Dataset Distillation for Pre-Trained Self-Supervised Vision Models

We apply dataset distillation to pre-trained self-supervised vision models by directly optimizing just a single image per class to train an optimal linear classifier on top of these models. To accomplish this, we introduce Linear Gradient Matching, which optimizes synthetic images such that they induce similar gradients as real images when training a linear classifier on top of a pre-trained model. Our method out-performs all real-image baselines and further excels on fine-grained visual classification datasets. Furthermore, the distilled images offer interesting interpretability results by partially revealing what and how these models actually see, particularly when trained on data with spurious correlations.