Suppressing Non-Semantic Noise in Masked Image Modeling Representations

Self-supervised learning (SSL) via Masked Image Modeling (MIM) objectives have become a popular source for strong, generalized vision backbones. However, there are key issues with artifacts and noise in the representations learned models that rely on MIM-based objectives.

We present a novel method to measure the amount of non-semantic noise in ViT tokens for state-of- the-art MIM-based models. We characterize non-semantic noise as components that are invariant to the semantic content in the input. This can for example be positional encoding, which are necessary for attention mechanisms but seldom useful in inference, or structural artifacts.

We discover that strong principal components exhibit high levels of non-semantic noise, and that this feature is pervasive in MIM-based models while nearly non-existent in other, non MIM-based SSL models. Importantly, this holds regardless of which positional embedding method is employed and whether predictions are conducted in latent or input-space, suggesting that this is an implicit issue in MIM.

To suppress non-semantic information, we introduce a Semantically Orthogonal Artifact Projection (SOAP) to remove unwanted artifacts that are not useful for inherently semantic tasks, such as instance level classification and salient segmentation—cf. Fig. 1. SOAP is flexible: It is computed directly from data using a Gram-Schmidt based projection, thus requiring no training, and can be attached as an external module to any pretrained SSL backbone.

Figure 1: Pipeline overview; a pretrained MIM encoder outputs dense representations zwhich are used for downstream tasks—we show salient segmentation as an example. By identifying and suppressing principal components encoding positional noise, our SOAP module improves the representations and enhances downstream performance in zero-shot settings.

Methodology: The TL;DR

Semantic invariance refers to the property of a component yielding consistent responses even when the semantic content of local representations varies. In other words, a component is semantically invariant if it produces similar activations regardless of whether the input carries meaningful semantic information.

We introduce a new score for measuring Semantic Invariance (SI) in the learned patch representation space, by comparing the representations of real images to those of synthetically generated “noise” images. The synthetic images are generated by a mixture of pink noise, modulated white noise, and random low-frequency gradient fields.

In short, given a model $f$ that encodes an image $x$ into into $N$ patch embedding representations $z_1,…,z_N$, we perform PCA of the patch representation space using Welford’s algorithm to estimate the covariance, and obtain the eigendecomposision $\text{Cov}(\mathcal{Z}) = V\Lambda V^\top \in \mathbb{R}^{D \times D}$, with principal component vectors $V=(v_1, …, v_D)$. Let $\Omega$ be the set of images with natural spatial frequency spectrum. For each principal component we calculate the aggrated activations of real and synthetic images by

where $A_{d,n} = \mathbf{1}[ z_n v_d \geq 0 ]$ is the activation of patch $z_n$ for component $v_d$. In practice, $\mathcal{X}$ can be instantiated as a set of natural images (for example ImageNet validation), while $\mathcal{X}^c$ is approximated by our synthetic image generator. The Semantic Invariance (SI) of component $d$ is then measured by

which assigns high scores when $P_d \approx Q_d$, and vice versa.

Finally, we introduce Semantically Orthogonal Artifact Projection (SOAP), a simple Gram-Schmidt-based post-hoc projection method to suppress components with high SI score:

Here, $\mu$ and $\tau$ are hyperparameters to control the cutoff and smoothness of suppression. This projection can be used to “correct” the patch representations $\hat z = P_\phi z$ by removing the contribution of components that encode non-semantic information such as positional noise.

Read our paper for a more in-depth formulation and explanation of the methodology!

Semantic Invariance in MIM models

We analyze popular SSL models by measuring their Semantic Invariance, and find a striking contrast between models trained with and without a MIM-style objective. The main findings are summarized by the plots below. Models with MIM are shown in solid lines; models without MIM are shown in dashed lines.

Figure 2: Semantic invariance (SI) score in descending order. All scores are shown in the left plot, while the right focuses on the top 10 semantically invariant scores.

The models trained with MIM have at least two principal components with high SI score ($\geq 0.75$). Upon manual inspection, we find that these components encode positional infomation about the patch location. Critically, DINO and DeiT3, which are not trained with MIM, do not exhibit the same behaviour and have lower SI scores in general ($\leq 0.75$).

Looking at the maxumim SI score of each transformer block, we see that while some MIM-models (DINOv2 and DINOv3) have lower SI scores in earlier layers, there is a sharp increase in the last layers, where all MIM-models land well above the $0.75$ threshold. This can be explained as the model saturating more positional information into the embeddings in preparation for solving the MIM task.

Figure 3: Maximum semantic invariance score for MIM models (solid lines) and non-MIM models (dashed lines) vs. model depth. Critically, MIM models show high SI-scores in the last layers. This can be explained by the MIM objective encouraging positional information in the patch embeddings of deeper layers.

SOAP improves zero-shot performance

We use SOAP to correct for semantically invariant components in local embeddings, and find that this improves performance in zero-shot downstream tasks for all MIM models in our study. Read our paper for more results!

Salient segmentation

We use TokenCut for zero-shot evaluation of salient information present in patch embeddings, and find that correcting with SOAP can significantly improve performance. Below are the results for the ECSSD dataset.

kNN segmentation

We evaluate zero-shot segmentation on ADE20k by performing per-patch k-nearest neighbors (kNN) and upsampling the predictions to full image resolution using nearest neighbor interpolation.

Citation

@inproceedings{hjelkremtan2025soap,
  title={Suppressing Non-Semantic Noise in Masked Image Modeling Representations},
  author={Hjelkrem-Tan, Martine and Aasan, Marius and Chakraborty, Rwiddhi and Arteaga, Gabriel Y. and Choi, Changkyu and Ram\'irez Rivera, Ad\'in},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}