Beyond Endpoints
A study of paths, not just destinations

Beyond Endpoints.

Bézier Paths for
Vision Foundation Model Adaptation

Chenzhi Zheng♠,† Xi Xiao♣,†,✉︎ Chih-Ting Liao♥ Qizhen Lan♣ Jingwen Lin♠ Tianyang Wang♣ Hao Xu♦ Xin Cao♥
♠ University of Birmingham♣ University of Alabama at Birmingham ♥ University of New South Wales♦ Harvard University

† Equal contribution · ✉︎ Corresponding author: Xi Xiao

Read the paper PDF Project on GitHub
The central idea Shared attention can rule out the ambient shortcut. Unrolling its attainable cone reveals a shortest realizable path. Schematic, not a measured ViT feature surface.
01 / The question

How prompts move
matters.

Reaching two good endpoints does not mean shared visual prompts can take the shortest path between them.

As prompts change, a frozen vision model’s image features move. Each image’s content competes for attention, so the same prompt update can move different images in different directions. We study the extra transport cost of keeping the entire journey realizable.

Read the abstract

A frozen vision foundation model can be adapted by learning a few prompt tokens shared across all images. When prompts are updated as data shift or tasks change, the distribution of image features moves from one endpoint to another. Prior theory asks which endpoints prompts can reach; we ask whether they can travel between them along the shortest path. Because shared prompts compete with each image’s own content for attention, they often cannot. We quantify this restriction as the extra transport cost that shared attention imposes over the shortest path. In an exactly solvable attention model, the reachable features form a cone; unrolling it into a plane gives the exact minimum action for every pair of endpoints and explicit optimal prompts on short arcs. The analysis reveals a sharp threshold in content normalization, which measures how much attention each image’s own tokens absorb: for same-sign endpoints, at most two distinct levels allow prompts to follow the shortest path, whereas three or more force a strictly positive extra cost unless the endpoints differ only by rescaling. This gap persists with more prompts, small content perturbations, and aligned multihead outputs. On ViT-B/16 with VTAB-1k, a quadratic Bézier prompt path that minimizes feature action reduces held-out transport action relative to linear and cross-entropy-trained paths between the same endpoints and improves intermediate and online accuracy under unlabeled distribution shift, while cross-entropy-trained paths retain higher minimum accuracy when labeled calibration data are available. Reachable endpoints do not guarantee a reachable shortest path: how prompts move matters, not only where they end.

02 / From geometry to adaptation

Understand the constraint.
Then learn the route.

Analyze the attainable geometry in an exactly solvable attention model; learn the control tensor of a Bézier prompt path in a frozen ViT.
i.

Exact path geometry

Derive the minimum transport action across endpoint regimes and construct optimal prompts on short arcs in a realizable attention head.

ii.

A sharp threshold

Three distinct content normalizers force an intrinsic gap for same-sign, non-rescaling endpoints in the core model.

iii.

Route and timing

Separate the cost of a longer route from the cost of uneven speed, then evaluate feature-action paths between fixed ViT prompts.

03 / What changes along the way

Less motion.
Same endpoints.

Between identical prompt endpoints, the training objective changes the journey through feature space.

Held-out transport action ↓ Lower is better

Cross-entropy
4.50
CE + feature action
3.42
Feature action
3.20
28.9%less action with feature action
relative to cross-entropy
Minimum accuracy along the path ↑
ObjectiveMin. accuracy
Cross-entropy68.8%
CE + feature action68.7%
Feature action67.4%

There is a tradeoff: pure feature action reduces motion, while cross-entropy retains higher minimum accuracy. The hybrid stays within 0.1 percentage points of CE.

ViT-B/16 · 19 VTAB-1k tasks · fixed endpoints and classifier · 128 held-out images per task · five endpoint pairs per task. Mean values from Table 2; action evaluated at 128 intervals.

When target labels are unavailable

Feature-action Bézier improves minimum shifted accuracy by +1.6 / +1.8 pp over unlabeled entropy Bézier for Shallow / Deep.

Fixed-endpoint transitions · 7 VTAB-Natural tasks · Gaussian noise, severity 3 · disjoint 128-image calibration and evaluation sets. Table 13.
04 / Visual comparisons

Saliency &
attention.

Matched animal photographs at the midpoint of two Bézier prompt paths: cross-entropy and feature action.

Oxford-IIIT Pet

Path midpoint
Saliency on six pet photographs. CE Bézier (top); feature-action Bézier (bottom).Open full size

Swipe across the figure to see all six photographs. Tap to enlarge.

Each column shares the same crop and scale across methods. These snapshots illustrate spatial diagnostics; full-path action and prediction quality are evaluated separately.

05 / A better route, a better clock

Retiming helps.
Geometry still matters.

Moving at a constant speed removes timing cost. It cannot straighten a curved route or remove the intrinsic gap.

Formula evaluations on 24 scalar-model cases, normalized by their attainable optimum. These diagnostics distinguish route cost from timing cost.
06 / Cite this work

Citation.

@misc{zheng2026beyondendpoints,
  title  = {Beyond Endpoints: B\'ezier Paths for
            Vision Foundation Model Adaptation},
  author = {Chenzhi Zheng and Xi Xiao and Chih-Ting Liao
            and Qizhen Lan and Jingwen Lin and Tianyang Wang
            and Hao Xu and Xin Cao},
  year   = {2026},
  url    = {https://xixiaouab.github.io/Beyond-Endpoints/}
}

Beyond Endpoints / Figure