FastViTHD
FastViTHD is the high-resolution hybrid vision encoder inside FastVLM. It converts a large image into a compact set of strong visual tokens before those tokens enter the language model.
Vision-language models
A photo of a restaurant menu is an awkward input for a vision-language model. At low resolution the heading might be clear while the prices blur together. At a higher resolution you can preserve the small print, but now there is more work for the image encoder and potentially a much longer sequence for the language model.
FastViTHD is the image encoder inside FastVLM. It gives the early layers a detailed image to work with, then reduces the spatial grid before running global attention. The final grid is small enough to hand off to a language model without making it process thousands of image tokens.
One image, two latency bills
A typical vision-language model first makes features from the image, maps them into the language model’s embedding space, and sends them to the language model alongside the text prompt. The language model processes that whole prefix before it can generate its first word. That prefix pass is called prefilling.
Image resolution and visual token count are separate quantities. An encoder can take in many pixels and return a relatively small grid. Keeping those quantities separate matters because a faster encoder can still leave the language model with a slow prefill if its output sequence is long.
Follow a 1,024-pixel image through the encoder
FastViTHD begins with a stem that reduces each spatial dimension by four. Between stages it halves the width and height again. The first three stages do local mixing with RepMixer blocks, and the last two use multi-head self-attention. By the time attention is introduced, the feature map is already 32 times smaller along each axis than the input.
For a square 1,024-pixel input, the final stride of 64 gives a 16 × 16 grid, which is 256 visual tokens. Earlier features can also be downsampled and combined at the output grid. That gives the projector access to information from a finer stage without handing the language model an extra sequence of positions.
Input 1024 × 1024 pixels
After the stem 256 × 256
Stages 2 and 3 128 × 128 → 64 × 64
Stages 4 and 5 32 × 32 → 16 × 16
Final visual sequence 16 × 16 = 256 tokensWhy the best resolution depends on the question
In the paper’s Table 4, FastViTHD goes from 64 visual tokens at 512 × 512 to 256 tokens at 1,024 × 1,024. With the Vicuna 7B setup used in that table, the DocVQA score rises from 25.7 to 35.6. Encoder time also rises from roughly 34 ms to 236 ms on the authors’ benchmark hardware, before adding the language model’s prefill. Doubling each input dimension is a real choice even with a compact output grid.
For a menu-reading task, the higher resolution might help locate tiny prices. For a question about whether the photo contains a menu at all, a smaller image might be enough. Both should be measured on the hardware you plan to use, with the language model you intend to pair with the encoder. The paper’s speed figures depend on those choices.