\n
As deep studying fashions develop bigger and datasets increase, practitioners face an more and more widespread bottleneck: GPU reminiscence bandwidth. Whereas cutting-edge {hardware} gives FP8 precision to speed up coaching and inference, most information scientists and ML engineers work with older GPUs that lack this functionality. <\/p>\n
This hole within the ecosystem is what motivated me to construct Feather<\/a><\/strong>, an open-source library that utilises a software-based method to ship FP8-like efficiency enhancements on broadly obtainable {hardware}. I created this software to make environment friendly deep studying extra accessible to the broader ML group, and I welcome contributions<\/p>\n FP8 processing has confirmed efficient within the Deep Studying group [1]<\/strong><\/a>; nonetheless, solely particular current {hardware} architectures (Ada and Blackwell) assist it, limiting its advantages for practitioners and researchers to utilise it. I actually have an `Nvidia RTX 3050 6GB Laptop computer GPU<\/em>`, which sadly doesn\u2019t assist FP8 operations on the {hardware} degree.\u00a0<\/p>\n Impressed by software-based options like (software-accelerated rendering on computer systems that don\u2019t assist native {hardware} acceleration for gaming), the article proposes an attention-grabbing resolution that may utilise the facility of FP8 datatypes One may argue that we\u2019re performing redundant computation, \u201cPack -> Load -> Unpack -> Compute<\/em>.\u201d Nevertheless, think about Deep Studying operations; More often than not, these operations are memory-bound reasonably than compute-bound. This is similar bottleneck that algorithms like FlashAttention tackle; nonetheless, FlashAttention utilises tiling to maintain information in quick SRAM, whereas Feather compresses information to scale back reminiscence visitors.<\/p>\n Check out this diagram. SRAM is the quickest accessible GPU reminiscence area and has the very best bandwidth (excluding the register itself), however is restricted to solely 20MB. HBM may be considered because the VRAM of the GPU itself, which has roughly 1\/seventh<\/strong> the bandwidth of SRAM.\u00a0<\/p>\n The GPU cores are quick sufficient to finish the computation immediately, however they spend most of their time sitting idle, ready for the info to complete loading and writing again. That is what I imply by memory-bound: the bottleneck right here isn\u2019t the maths, however the information switch between the hierarchy of reminiscence within the GPU.<\/p>\n Identical to AVX-512 directions, that are supported solely on a restricted variety of {hardware} platforms, FP8 and FP16 directions and registers are additionally restricted by {hardware} and can be found solely on the current ones. If you’re on an RTX-30 or RTX-20 sequence GPU from Nvidia, you then will be unable to make the most of this decrease precision FP8 kind. That is precisely the issue that Feather<\/a><\/strong> makes an attempt to unravel.<\/p>\n Utilizing bitwise operators, one can simply pack the FP16 kind right into a FP32. The algorithm is described beneath.<\/p>\n FP8 has two broadly used codecs \u2013 E5M2 & E4M3. <\/strong>One can not use the identical algorithm used for packing two FP16 into FP32 as a result of the CPU doesn\u2019t assist FP8 sorts natively, however does for FP16 (half precision); that is the rationale that np.float8<\/em> doesn\u2019t exist.\u00a0<\/p>\nNotation & Abbreviations<\/h3>\n
\n
\nMotivation<\/h2>\n
\nPacking FP8 & FP16 in FP32 containers<\/h3>\n
\nGPU Reminiscence Hierarchy<\/h2>\n
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/12\/feather_gpu_mem_chart-3-1024x512.png\")
\nDecrease Precision Varieties & Bandwidth<\/h3>\n
{Hardware} Stage Assist<\/h3>\n
\nPacking Technique<\/h2>\n
Packing FP16<\/h3>\n
\n
Unpacking FP16<\/h3>\n
\n
Packing FP8\u00a0<\/h3>\n
![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/12\/feather_fp_1-scaled.png\")