They sped up attention by 2.13× on Blackwell chips using FP4.

New Blackwell computer chips can do math with tiny 4-bit numbers called FP4, but using them does not automatically make attention calculations faster. Attention helps models focus on relevant data. Once the core math shrinks, converting values with a function called softmax and waiting on chip data transfers take up most of the time. To solve this, researchers built two methods. First, Direct-P turns attention scores straight into FP4 probabilities during inference, reaching up to 2.13$\times$ the bfloat16 speed on an NVIDIA GB200. Second, for causal tasks, their method saves quantized inputs to rebuild probabilities later. This speeds up a full single-GPU 8-billion-parameter model update by up to 1.14$\times$. However, training runs using MXFP4 probabilities and values failed and diverged.

Why it matters

This makes large AI models run faster and cheaper on new hardware, reducing the time and cost for companies and researchers who build and deploy them. It directly accelerates both inference and training, speeding up AI development cycles.

Questions answered