A Smaller Transformer in Your Transformer

The problem

Vision Transformers (ViTs) often suffer from computational redundancy, meaning that consecutive layers in the network frequently settle into phases where they perform highly similar, repetitive tasks. While previous research has tried to address this inefficiency, existing methods generally fall short: they either fail to reduce the actual computational cost during inference (by using local recurrence) or rely on linear approximations that severely degrade the model’s expressivity and performance.

Objective

We aim to provide a unified theoretical view of block redundancy in ViTs and introduce a new compression method that better exploits this inefficiency. Our primary goal is to reduce both the number of model parameters and the computational load without losing the original architecture’s representational power.

A Unified View of Block Redundancy

A ViT applies a sequence of structurally similar layers to the input. Empirically, consecutive layers often organise into contiguous blocks whose representations remain highly similar. We interpret these blocks as learned approximations to broader computational phases: rather than every layer performing a distinct operation, several consecutive layers may instead make incremental updates within the same phase.

This view provides a common way to understand seemingly different approaches to exploiting block redundancy. Once a redundant block has been identified, the central question becomes: what surrogate should replace it? Recurrent approaches use a shared ViT layer repeatedly across the block, reducing parameters while retaining the original computation. Other approaches replace the block with a simpler non-mixing surrogate, such as a linear layer or a feed-forward network, which reduces computation but also gives up the token-mixing behaviour of self-attention, resulting in a drop in performance.

Figure 1. (Left) A standard ViT can take several small, uneven steps through a computational phase. (Middle) A recurrent approximation regularises those steps but still executes the phase iteratively. (Right) TWT instead asks whether the same phase can be traversed by a single learned macro-step while preserving its endpoint.

In this framework, these methods are therefore exploiting the same phenomenon. They identify the same kind of redundant computational phase, but make use of different surrogates to approximate it. This motivates a natural question: can we find a surrogate that reduces computation and retains the expressive power and performance of the original architecture?

Transformer-Within-Transformer (TWT)

Our answer is to use a standard ViT layer itself as the surrogate. If several consecutive layers primarily correspond to a single computational phase, we hypothesise that the intermediate representations in that phase are not all strictly necessary. Instead, a single appropriately initialised and trained Transformer layer may be able to map directly from the beginning to the end of a phase.

We call this Transformer-Within-Transformer (TWT). TWT replaces each identified redundant block with a single learned ViT layer. This preserves the dynamic token-mixing capability of self-attention while reducing both the number of parameters and the number of Transformer layers executed at inference.

Turning this idea into a practical compression method requires answering three questions: which consecutive layers form a phase, which candidate provides the best starting point for its surrogate, and how to train the resulting compressed network. TWT addresses these through block discovery, surrogate auditioning, and deep supervision.

Figure 2. TWT compresses a ViT in three steps: it first discovers contiguous redundant blocks, selects a suitable initialisation for each block’s surrogate, and then jointly trains all resulting surrogate layers using deep supervision.

1. Discovering computational phases

First, we need to determine which consecutive layers can be treated as a single phase. TWT measures the cosine similarity between representations produced by different layers and looks for contiguous regions where these representations remain highly similar. We then partition the network into blocks using a similarity threshold, preferring partitions that compress the network into as few phases as possible.

This block-like structure appears across different ViT families and training paradigms. The similarity matrices below show that layers do not change their representations uniformly with depth; instead, they tend to form contiguous regions of locally similar computation. These regions provide the candidate phases that TWT subsequently attempts to collapse.

Figure 3. Pairwise cosine similarity between intermediate representations reveals contiguous, block-like structure across supervised and self-supervised ViTs. The white boundaries indicate candidate computational phases.

2. Auditioning the surrogate

Once we identify a block of layers, we still need a good starting point for the single ViT layer that will replace it. Rather than initialising this surrogate randomly, TWT reuses what the teacher has already learned. We treat each layer inside the block, along with a surrogate obtained by averaging the block’s weights, as a candidate. We first train each candidate on a small number of minibatches to adapt it to the full block mapping, then evaluate it on a small calibration set. We select the candidate that best maps the inputs of the block to its output. We call this procedure surrogate auditioning.

3. Learning the macro-step

After replacing each redundant block with its selected surrogate, we train the resulting smaller Transformer as a single student network. During training, each surrogate is trained to reproduce the output of the corresponding teacher block and the final student output is also matched to that of the full teacher. For natural images, these intermediate supervision signals are introduced progressively during training so the compressed network learns the shorter path without reproducing every intermediate step taken by the original model.

Does Block Collapse Preserve the Representation?

TWT is based on the idea that intermediate layers in a redundant block largely refine the representation rather than modify it drastically, and that the surrogate reaches the same destination while compressing this refinement into a single block. If this is correct, the coarse representational trajectory of the network should also remain the same: the student should still arrive at similar kinds of representations at the boundaries between phases.

Of course, this is difficult to inspect directly in a high-dimensional representation space. One way to probe the geometry is through intrinsic dimension (ID), which estimates the effective dimensionality of the manifold the representations occupy. We therefore measure how ID changes with depth in both the original teacher and the compressed student, looking separately at the CLS token, the mean patch representation, and the full flattened token representation.

Figure 4. Intrinsic-dimension (ID) trajectories of the original teacher and block-collapsed student for histopathology (top) and natural images (bottom). The student preserves the teacher’s broad trajectory at the discovered phase boundaries. In histopathology, block collapse additionally produces lower-dimensional representations with reduced variance under stochastic augmentation.

Across the discovered block boundaries, the compressed student follows the same broad ID trajectory as the original teacher, suggesting that the surrogate layers preserve the coarse representational path despite skipping the intermediate steps. In the histopathology setting, however, the student has lower ID and shows lower variance under stochastic augmentations. This suggests that block collapse may be removing augmentation-sensitive degrees of freedom that are not essential to the task, resulting in a more compact and stable representation. This interpretation is consistent with the downstream results: the compressed histopathology models generally preserve performance and, in several settings, even improve over the uncompressed teacher.

Downstream Performance

We evaluate TWT across histopathology, natural-image classification, and dense prediction.

Histopathology

Histopathology is where block collapse is particularly effective. On H0-mini, TWT compresses the backbone from 85.74M to 36.11M parameters (from 12 layers to 5 layers) while cutting inference compute from 23.56 to 9.36 GFLOPs. Despite this reduction, performance on PANDA is essentially preserved, while performance on the external TCGA-PRAD cohort improves for both ABMIL and TransMIL. We observe a similar pattern on CAMELYON17, where the compressed models often improve over the full backbone. Together with the lower intrinsic dimension and reduced augmentation variance observed above, these results suggest that TWT can remove representational variation that is unnecessary for downstream histopathology tasks without sacrificing useful information.

Natural Images

On natural-image classification, TWT provides a compression trade-off. Reducing a 12-layer ViT to six layers roughly halves the inference computation while the performance remains close to the original model and competitive with recurrent alternatives. For example, the six-layer DINOv2 TWT model requires 11.71 GFLOPs compared with 23.42 GFLOPs for the recurrent surrogate, while achieving comparable or better classification performance. More aggressive compression to five layers leads to a larger accuracy drop, suggesting that compression eventually removes computation that is no longer redundant.

Dense Prediction

Dense prediction is less forgiving. On ADE20k semantic segmentation, TWT incurs a larger drop in mIoU than in image classification. This suggests that while an entire computational phase can often be compressed into a single macro-step for classification, dense prediction may require preserving more intermediate spatial processing.

Scope and Limitations

TWT is not a universal replacement for depth in Vision Transformers. The gains from block collapse depend both on the model and the downstream task. While histopathology benefits strongly in our experiments, natural-image models primarily preserve rather than improve performance, and dense prediction is noticeably more sensitive to compression. The quality of the compressed model also depends on how well we identify the redundant phases, and our block-discovery procedure remains an approximation. More broadly, TWT is a post-hoc method for exploiting redundancy that already exists in a trained model; understanding why these computational phases emerge in the first place, and whether we can mitigate or better control them during pre-training, remains an open question.

Citation

@inproceedings{Tomar2026,
  title     = {A Smaller Transformer in Your Transformer},
  author    = {Tomar, Dhananjay and Aasan, Marius and Kleppe, Andreas and Ramírez Rivera, Adín},
  booktitle = {Proceedings of the British Machine Vision Conference ({BMVC})},
  year      = {2026},
  publisher = {BMVA Press},
  doi       = {}
}