Deep Learning: Reducing Dimensional Collapse of Contrastive Learning

In self-supervised visual representation learning, the goal is to learn valuable representations without the need for human annotations. Contrastive learning, a joint embedding approach, seeks to maximize the agreement between embedding vectors from different views of the same image. However, this approach can still suffer from dimensional collapse, where embedding vectors occupy a lower-dimensional subspace rather than the entire available embedding space. We propose DisCLR, a novel contrastive learning framework that uses an autoencoder to disentangle the dimensions of pre-trained encoders in contrastive learning to reduce dimensional collapse. Experiments demonstrate that DisCLR results in less dimensional collapse compared to SimCLR with linear finetuning on CIFAR-10.

Faculty Mentor: Peng Li

Project Mentor: Zihu Wang, Samuel Tian