SPoT: Subpixel Placement of Tokens in Vision Transformers

Sparsity - the fine art of doing more with less - is an attractive prospect in systems design and modeling. As models grow ever larger, sparse features alleviates the computational demands of a model to provide lower latency, lower memory overhead, and higher throughput - all indispensable properties for real-time applications. Incidentally, sparse selection of features offers inherent interpretability and transparency for increasingly complex models

Vision Transformers (ViTs) can naturally be adapted to sparse inputs via their Bag-of-Visual-Words (BoVW) representation of images as a collection of patch tokens. However, in carefully studying the fine print of Dosovitskiy et al’s. work, one notes an insistence on aligning features with an underlying grid, mirroring the structure inherited by its language counterpart where inputs are naturally represented as sequences of discrete tokens. Our work posits that this adherence turns sparsity into an awkward dance; forcing the selection of entire tiles, even if the true optimal feature set hides in-between rigid lines. Like eating soup with a fork: possible, but decidedly inefficient and frustrating.

We propose a simple remedy via Subpixel Placement of Tokens (SPoT). By allowing patches to occupy continuous subpixel positions instead of constraining features to a discrete grid, we expand our modeling toolbox to include gradient based search and sampling for discovering optimal sparse feature sets.

Problem Statement: Sparse Feature Selection in ViTs

We denote a ViT encoder as $g_\theta : \mathcal{I} \times \Omega \to \mathbb{R}^d$, where $\mathcal{I}$ is a dataset of source images, and $\Omega$ is a space of positions from which to sample image features. With standard tokenization $\Omega_\text{grid}$ is a fixed, discrete set of non-overlapping square patches tiling the image with a fixed window size on a grid of pixels. The sparse feature selection (SFS) problem can then be formulated as

In other words, we are looking for a probability distribution $p_\phi$ over subsets of $\Omega$ that minimizes a loss function $\mathcal{L}$. We note that for the discrete non-overlapping case of $\Omega_\text{grid}$, there is an implicit assumption that sampling of $S$ is done without replacement, since sampling the same feature more than once is unlikely to improve model performance. Three specific issues arise from the ViT sparse sampling problem;

1. Interdependence: Transformers process tokens as a set. This means that the marginal distribution of one token is dependent on the inclusion of other tokens. Furthermore, the optimal distribution $p_\phi$ for a given image may vary depending on the choice of number of tokens.
2. Combinatorial search: The discrete nature of $\Omega_\text{grid}$ means that selecting a subset of tokens is combinatorial knapsack problem. This makes search difficult and gradient methods intractable, particularly since cardinality-constrained subset selection is NP-hard.
3. Misalignment: By quantizing patches to a fixed grid, key patterns for discriminating an image could be missed in SFS. Concretely, if the grid imposed by $\Omega_\text{grid}$ is misaligned with key features in the image, SFS could be challenging, as a central shape or texture may be spread over multiple patches, making subset selection more challenging.

These issues hinder efficient optimization of SFS under standard tokenization - in other words, we posit that grids cannot align every salient region.

Figure 1: A $5 \times 5$ patch grid (gray) with three optimal region placements for sparse feature selection. (a) The green patch is well aligned (A), yellow straddles two cells (B), and red lies on a corner (C) and leaks into four cells. Translating the grid only swaps which peak is misaligned—one patch is always bad. (b) Our subpixel tokenizer drops fixed-size windows (\textcolor{ok}{green} squares) directly on each peak, eliminating the alignment trade-off while still allowing conventional grid tokens when they \emph{are} well aligned.

Methodology: SPoT in a Nutshell

We propose a more flexible tokenization scheme to tackle SFS problems in ViTs. Instead of considering $\Omega$ as a fixed discrete partition, we instead imagine ${\Omega_\text{subpix} = [0,H-1] \times [0,W-1]}$ as a continuous space of subpixel positions from which to select features within a $H \times W$ image. We parametrize a subset of positions $S = { s_1, \dots, s_m }$ as a set of points of interest from which to extract features from within an image. By sampling tokens from continuous subpixel positions, our tokenizer directly addresses the intrinsic misalignment issue imposed by traditional grid‐based methods, as illustrated in \Cref{fig:sfs-misalignment}. To tackle the combinatorial search problem, we use a bilinear interpolation function $q$ and window size $k$, each subpixel position $s_i = (h,w)$ provides an extracted feature

This allows us to formulate SFS as a continuous, probabilistic optimization problem rather than an intractable discrete subset‐selection. The key insight is that our novel tokenizer allows us to (1) investigate placing different priors on $p_\phi$, and (2) use gradient based optimization to search for $S$ by way of gradients through $I_q$. Since we select $q$ to be bilinear, its partial derivatives w.r.t. $s$ exist everywhere except at pixel boundaries, so gradients propagate cleanly back to the placements ${s_1,\dots,s_m}$. Note that subpixel tokens do not impose any constraint on non-overlapping patches.

Since ${\Omega_\text{grid} \subseteq \Omega_\text{subpix}}$, patch tokenization is just a special case of our tokenization method. This means that models can be evaluated with the exact same features as a standard patch-based ViT by letting $S = \Omega_\text{grid}$.

Spatial Priors

By removing the strict adherence to grids in ViTs, we can leverage more continuous spatial priors for token placements for optimal feature extraction. We compare several spatial priors, each encoding different assumptions about feature importance and spatial distribution.

• Uniform: randomly samples locations with no spatial bias, assuming all regions are equally important.
• Gaussian: randomly samples locations with a central bias, which encodes a prior belief that subjects are typically centered in images.
• Sobol: provides quasirandom sampling aimed at uniform coverage while reducing overlap.
• Isotropic: deterministically distributes tokens evenly in a subpixel grid, emphasizing coverage.
• Center: deterministically distributes tokens evenly with slight central-bias.
• Salient: encodes object-centric bias by placing tokens based on regions identified as visually salient from a pretrained saliency model.

Figure 2: An illustration of different spatial priors investigated with SPoT.

Exploring Oracle Neighbourhoods with SPoT-ON

In addition to investigating spatial different spatial priors, we also look to directly explore differentiable optimization for token placement. To probe for ideal choices of $S = {s_1, \dots, s_m }$, we optimize a constrained version of the SFS problem directly for each image. We freeze the encoder $g_\theta$, and optimize

for a set number of tokens $m$ with initial positions ${S^0 \sim p_\phi}$ sampled from a chosen prior $p_\phi$. Gradient optimization provides an Oracle Neighborhood guided (ON) adjustments of the initial placements with SPoT.
SPoT-ON reveals locations are optimal for classifying each image, which allows us to ascertain the existence of an optimal set of positions $S$ for each image, and estimate an upper bound on performance gain from effective token sampling.

Figure 3: Illustration of oracle placements with 25 tokens with SPoT-ON. By optimizing our oracle-neighborhood search through the model, the oracle discovers optimal placement of points, yielding an accuracy of $90.9\%$ on ImageNet1k with only $\sim12.5\%$ of the tokens. Trajectories are colored with dark purple for initial points, and endpoints colored bright yellow.

Experiments and Key Findings

We design four case studies for SPoT to determine the effectiveness and properties of subpixel tokenization in sparse regimes.

1. Are Grids an Inherent Limitation of ViTs? Off-grid token placement enables greater flexibility and yields substantially better performance than grid-based approaches under sparse token settings.
2. Do Object-Centric Priors Improve Predictions? Object-centric priors yield higher performance in sparse regimes. In dense regimes, even and structured coverage provides better performance.
3. Does Oracle Guidance Prefer Salient Regions? While oracle gradient search yields a slight bias toward higher-saliency pixels, the results are not conclusive. Assessments of token trajectories suggest that token interdependency, rather than saliency alone, predominantly determines final placements.
4. Do Oracle Guided Placements Transfer? Discovered positions generalize between models; a set of placements optimized via a model will improve results with another independently trained model.

Full Experimental Results

We report classification top-1 and kNN accuracies for supervised and and self-supervised models using different token priors. Our results show that center-bias in spatial priors is beneficial in sparse regimes, while coverage becomes more important as token budgets increase.

Backbone: CLS-IN21k

Backbone: CLS-IN1k

Backbone: MAE-IN1k

Performance Gap

Figure 4: We show ImageNet1k accuracy vs throughput with 5 models at four sparsity levels. The ceiling area denotes performance unlikely to be achieved given the intrinsic label noise in ImageNet. The gap highlights the margin between SPoT with optimal configuration and SPoT-ON, illustrating possible performance gain through better token placement.

Figure 4 shows image throughput versus accuracy, comparing SPoT with the baselines across varying sparsity levels. As sparsity increases, throughput improves significantly, albeit with an associated trade-off in accuracy. Notably, SPoT achieves the most favorable trade-off, maintaining substantially more of the full-model accuracy while enabling higher throughput than competing approaches. We observe only slight variation in throughput between the models at each sparsity level, indicating that SPoT incurs very minimal computational overhead compared to baselines. Our oracle-guided variant SPoT-ON is included in the figure, and illustrates a ceiling on achievable performance in sparse regimes when placements are ideally sampled.

Summary

SPoT is a novel flexible tokenization framework for Vision Transformers that naturally adapts to sparse feature selection, and provides inherent methods for discovering optimal subpixel placements for extracting optimal features for prediction tasks. Our work highlights a gap for achievable improvements on classification tasks, and shows that sparse token selection can be significantly improved via more flexible tokenization schemes.

Limitations

In this study, we focus on analyzing the effects of subpixel tokenization under varying sparsity configurations with different spatial priors, the development of learnable spatial priors is a next step towards narrowing the oracle performance gap. By enabling continuous token positioning, SPoT facilitates gradient-based optimization of token placement, which can be advantageous in resource-constrained environments where sparsification is beneficial. Although we limit our scope to employ an oracle to determine optimal token placements, exploring oracle-independent strategies represents a compelling direction for future research. Specifically, integrating efficient saliency-driven objectives or heuristics during inference could potentially enhance throughput efficiency while maintaining competitive performance compared to models utilizing a full token budget.

Citation

@inproceedings{hjelkremtan2025spot,
  title={{SPoT}: Subpixel Placement of Tokens in Vision Transformers},
  author={Hjelkrem-Tan, Martine and Aasan, Marius and Arteaga, Gabriel Y. and Ram\'irez Rivera, Ad\'in},
  journal={{CVF/ICCV} Efficient Computing under Limited Resources: Visual Computing ({ECLR} {ICCVW})},
  year={2025}
}