FastViTHD is the high-resolution hybrid vision encoder inside FastVLM. It converts a large image into a compact set of strong visual tokens before those tokens enter the language model.

CORE IDEA

Downsample early, reserve self-attention for smaller feature maps, and send fewer visual tokens into the expensive language decoder.

HOW IT WORKS

01  Use RepMixer blocks in the first three stages.

02  Downsample before the attention-heavy stages.

03  Apply multi-head attention in the final two stages.

04  Combine multi-scale features and project the compact tokens into the LLM.

WHY IT MATTERS

VLM latency includes image encoding plus LLM prefilling over visual tokens. FastViTHD reduces both costs while retaining high-resolution information.

KEEP IN MIND

More image resolution is not always optimal. The best operating point depends on encoder resolution, visual-token count, and language-model size together.