Representation learning

Erik Ylipää (erik.ylipaa@liu.se)

NBIS

07-May-2026

Outline

• Contrastive learning
• Self-supervised learning
• Multimodal Learning

Overview

• Representation learning
• Breaking down classification
• Contrastive learning
• Self-supervised learning
• Multimodal learning

Welcome everyone. Today we cover self-supervised learning, a middle-ground between supervised and unsupervised learning. The key idea is to create labels from the data itself, allowing us to learn useful representations without manual annotation. Before we dive into this concept, we’ll first review some fundamentals of learning paradigms and representation learning. Then, we’ll explore autoencoders, which are a foundational architecture in self-supervised learning.

AI subfields

Deep learning is a subset of machine learning. But what makes deep learning special?

AI subfields: Representation Learning

Data representation is crucial for learning

The way we represent data has a huge impact on how easy a problem is to solve. This toy example illustrates that in a cartesian representation, the classes are hard to easily separate, while in a polar representation they become linearly separable.

Shallow representations – Principal Component Analysis (PCA)

Neural Networks learn representations from data

The fundamental idea of neural networks and deep learning is that we learn to map the input data into new representations which are more useful for the downstreams layers (helping them reduce the loss). Each layer learns a transformation of the data into a new space, and the final layers typically use a linear model (e.g. linear or multinomial logistic regression) to make predictions. In deep learning, we often talk about learning signals, the signal which drives the learning of representations. By crafting the network in different ways, and adjusting what the loss function penalizes we can drive the learning of different representations. For example, in supervised learning, the learning signal is the error between the predicted labels and the true labels.

Transfer learning

While there are many advantages to deep representation learning, one powerful consequence is that we can transfer the learned representations to new tasks. For example, a model trained on ImageNet learns general features that can be useful for many other vision tasks, even if the new task has limited data. This is known as transfer learning and is a key reason why deep learning has been so successful in practice.

The classification head

https://www.mathworks.com/discovery/convolutional-neural-network.html

The classification head typically has a fully connected layer followed by a softmax. The fully connected layer is what we’ll look at here.

What is the core mechanism of neural networks?

\vec{y} = \operatorname{softmax}(W \vec{x} + \vec{\beta})

\Leftrightarrow

\vec{y} = \operatorname{softmax}( \begin{bmatrix} \vec{w}_1^\top \\ \vdots \\ \vec{w}_n^\top \end{bmatrix} \vec{x} + \vec{\beta})

A matrix multiplication can be though of as a list of separate dot products.

The core mechanism of neural networks is the dot product

Properties of the dot product

\vec{a} \cdot \vec{b} = |\vec{a}||\vec{b}| \cos\theta

• Dot product is directly connected to the angle (alignment) between vectors
• Neural Network learns by aligning vectors

The cosine similarity is essentially the dot product of unit vectors. When two representations are identical in direction, cosine similarity = 1; orthogonal = 0; opposite = −1.

Graphical representations of neural networks

Different graphical representations of neural networks

Graphically representing softmax

Computational graph view of softmax

Graphically representing softmax

Computational graph view of softmax

MNIST example

Softmax angle alignment

Computational graph view of softmax

Contrastive learning

Using computed “softmax” weights

No fixed “class” vectors

Embeddings of other examples act as the “class” vectors

Contrastive learning - triplet loss

By Krishnachandranvn - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=138534332

The kind of loss used here was used a lot in face identification and referred to as triplet loss, three different data points are used where two consists of a positive pair which we try to align (maximize the dot product), while the negative pair should have maximum dot product. The idea was that we want to learn a metric over our data points which is semantic. Things which are close together are semantically similar (images of the same person).

What would happen if we didn’t have any negative pairs?

Contrastive learning

What negative samples?

• How do we select negative samples?
  • Easiest is randomly
  • But very likely to draw “easy” negative samples
  • The model doesn’t learn as much contrasting against these

The setup learns more from hard negatives

Why do we need negative samples?

While the idea of triplet loss and non-parametric softmax is to remove the fixed class vectors, the problem is still supervised in the sense that we need to know what data points we should bring together and which ones we should pull apart. Empirically it’s not necessarily better to use contrastive loss like this

Representation collapse

Why contrastive learning

• No need to decide on number of classes beforehand
• We only need to know that some inputs are associated and some are not
• By choosing what examples are negative vs. positive, we design what is learned

Supervised contrastive learning

Contrastive learning like we’ve seen here still needs labeled data, just like regular supervised learning

Most problems we want to solve are supervised, we want to learn a mapping from inputs to outputs. However, supervised learning relies on labeled data, which can be expensive and time-consuming to obtain. In contrast, unsupervised learning can discover structure in data without labels, but is often more useful for exploratory analysis than for solving specific tasks.

The practical meaning of supervised learning – labeling

The main challenge in supervised learning is obtaining labeled data, which can be expensive and time-consuming to collect and annotate. For certain problems, such as rare disease, we are fundamentally limited in how much labeled data we can get. This motivates the need for methods that can learn from unlabeled data.

Self-supervised learning

Self-supervised learning

What if we can use the data itself to drive the learning of representations?

Self-supervised learning - Yan LeCun

Transfer learning is promising, but requires large amounts of labeled data to learn useful representations from. Manually annotating data is expensive and time-consuming. Self-supervised learning is a paradigm that allows us to learn useful representations without manual annotation by creating labels from the data itself.

Basic selfsupervision – Autoencoders

An autoencoder is a neural network trained to:

1. Encode input x into a latent representation z
2. Decode z back to reconstruct \hat{x} \approx x

\mathcal{L} = \|x - \hat{x}\|^2

The key insight: the network learns a compressed representation without labels.

Shortcomings of autoencoders

From the Deep Learning book (Goodfellow, Bengio, Courville)

Minimizing mean squared error in pixel space is similar to PCA, it mainly captures the directions of most variation (in pixel space). This does not necessarily mean it captures the semantically meaningful features. Is there some other way we can learn representations without manual labelling?

Self-supervised examples - predict in context

Pathak, Deepak, et al. “Context encoders: Feature learning by inpainting.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

Noroozi, Mehdi, and Paolo Favaro. “Unsupervised learning of visual representations by solving jigsaw puzzles.” European conference on computer vision. Cham: Springer International Publishing, 2016.

Doersch, Carl, Abhinav Gupta, and Alexei A. Efros. “Unsupervised visual representation learning by context prediction.” Proceedings of the IEEE international conference on computer vision. 2015.

Figure 1

Constrastive self-supervised learning

• Instead of relying on existing positive pairs, we can create them
• As long as the pairs we create have a semantic association, there is something to learn
• SimCLR is a very successful example of this

Chen, Ting, et al. “A simple framework for contrastive learning of visual representations.” International conference on machine learning. PmLR, 2020.

Beyond contrastive learning

• The central part of contrastive learning is to bring associated examples close together
• Why do we need negative examples?
  • To avoid representational collapse

BYOL - Bootstrap your own latents

• Instead of avoiding representation collapse by using negative contrastive examples, what if we have two different models?
  • The main network we try to train – the online network
  • Another network we use to create the representations – the target network

Grill, Jean-Bastien, et al. “Bootstrap your own latent - a new approach to self-supervised learning.” Advances in neural information processing systems 33 (2020): 21271-21284.

DINO = BYOL + Vision Transformers

Caron, Mathilde, et al. “Emerging properties in self-supervised vision transformers.” Proceedings of the IEEE/CVF international conference on computer vision. 2021.

DINO v3

• Development of DINO v2 to increase model scale
  • Large data with auto curation
  • Regularized patch representations with Gram matching

Siméoni, Oriane, et al. “Dinov3.” arXiv preprint arXiv:2508.10104 (2025).

for images have been hard - Models are unstable in terms of performance - Coherence of attention maps degrade over training time - DINO v3 introduces a series of tricks to improve this - Automated data curation - train on larger sets of data with higher average quality - Starts from a pool of 17B images from Instagram, clusters and chooses balanced coverage to 1689 million images - Tweaks to architecture (positional embeddings) - Regularizing the attention maps (Gram matching) - Match matrix of pairwise dot products of all patches to a example from early on in training (regularizing the feature structure, keeping it spread oout) - Scaling up SSL methods like DINO (in terms of model size) for images have been hard - Models are unstable in terms of performance - Coherence of attention maps degrade over training time - DINO v3 introduces a series of tricks to improve this

DINO v3 - Gram matching

Figure 2: Siméoni, Oriane, et al. “Dinov3.” arXiv preprint arXiv:2508.10104 (2025).

\mathcal{L}_\text{Gram} = \lVert \mathbf{X}_S \cdot \mathbf{X}_S^\top - \mathbf{X}_G \cdot \mathbf{X}_G^\top \rVert_\text{F}^2

Constrain all the pairs of inner products to be similar to an earlier version of the model

for images have been hard - Models are unstable in terms of performance - Coherence of attention maps degrade over training time - DINO v3 introduces a series of tricks to improve this - Automated data curation - train on larger sets of data with higher average quality - Starts from a pool of 17B images from Instagram, clusters and chooses balanced coverage to 1689 million images - Tweaks to architecture (positional embeddings) - Regularizing the attention maps (Gram matching) - Match matrix of pairwise dot products of all patches to a example from early on in training (regularizing the feature structure, keeping it spread oout) - Scaling up SSL methods like DINO (in terms of model size) for images have been hard - Models are unstable in terms of performance - Coherence of attention maps degrade over training time - DINO v3 introduces a series of tricks to improve this

Multimodal Learning

Multimodal Machine Learning tutorial - ACL2017 – https://www.cs.cmu.edu/~morency/MMML-Tutorial-ACL2017.pdf

Use modalities to label each other

With supervised contrastive learning we need to label the data

Arandjelovic, Relja, and Andrew Zisserman. “Look, listen and learn.” Proceedings of the IEEE International Conference on Computer Vision. 2017.

With multimodal data we have “natural” associations in the separate modalities.

Use modalities to label each other

Figure 3: Arandjelovic, Relja, and Andrew Zisserman. “Look, listen and learn.” Proceedings of the IEEE International Conference on Computer Vision. 2017. za

Use modalities to label each other

Afham, Mohamed, et al. “Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

CLIP

https://github.com/openai/CLIP

CLIP

Radford, Alec, et al. “Learning transferable visual models from natural language supervision.” International Conference on Machine Learning. PMLR, 2021.

Summary

• Contrastive learning is a general technique for learning representations
• Self-supervised learning is about crafting ways of learning from unlabeled data
• Clever augmentations technique can use contrastive learning without labled data
• Multimodal data is perfect for contrastive learning, the modalities “label each other”