FluxBin Uses Binary Bases with a LUT CUDA Kernel, Reports Up to 5.92× Faster Inference on a Single A100

A team from the University of Hong Kong and Harbin Institute of Technology has introduced FluxBin, an attempt to solve a common problem in binary quantization: models become much smaller but do not actually run faster. Many 1-bit to 4-bit methods still need to dequantize weights into FP16 at runtime or incur irregular memory access because of unstructured high-precision exceptions. As a result, the theoretical benefits in memory usage and bitwise computation are consumed by conversion and kernel overhead.

FluxBin first approximates weights using multiple sets of ±1 binary bases and decomposes each set’s scaling factors into row and column vectors to capture weight distributions along different dimensions. It then uses a Hessian approximation to identify sensitive columns and assigns additional binary bases only to those columns. To prevent sparse sensitive columns from slowing down the GPU, Virtual Columnar Mapping rearranges logically scattered columns into physically contiguous matrices, allowing the global bases and sensitive branches to reuse the same type of dense GEMV kernel.

The key to the execution layer is a lookup table (LUT). The kernel precomputes partial sums for the possible binary sign combinations encountered by input subvectors, while the bit patterns of the compressed weights serve directly as indices. Column scaling is fused into LUT construction, and row scaling is applied after accumulation in registers. The weights can therefore be streamed in a compressed Uint32 format without being reconstructed into a floating-point matrix inside the main loop. The tests also use CUDA Graphs to fix the KV cache and replay the prefill and decode kernels, reducing interference from Python scheduling and launch overhead in the measurements.

On a single 80GB A100, the authors report up to a 5.92× speedup, a 10.19× improvement in energy efficiency, and roughly a 4× reduction in memory usage, and claim that the system can accommodate 70B-class models. For LLaMA-2 7B, a mixed 2.75-bit configuration reaches 250.87 tokens/s, compared with 42.94 tokens/s for FP16. However, the average zero-shot score falls from 67.07 to 59.15, so the throughput gain should not be interpreted as a lossless replacement. All results are also limited to the A100, a custom Transformers path, and static CUDA Graphs. The GitHub repository cited in the paper returned a 404 response at the time of verification, leaving engineers unable to independently validate the kernels, model compatibility, or end-to-end memory figures.