💡 Radix-3 Information Density: While 2-bit packing stores 4 values/byte, Radix-3 base-3 packing (3^5 = 243 < 256) fits 5 ternary values per byte, hitting the Shannon theoretical limit of log2(3) ≈ 1.585 bits/weight.
1. Input Signed 4-Bit Integers ([-8, +7])
Two 4-bit integers are packed into one byte: Low Nibble (Bits 0..3) stores even index, High Nibble (Bits 4..7) stores odd index.
Even Index (v_even):-3
Odd Index (v_odd):5
2. Packed INT4 Register
byte = (v_even & 0x0F) | ((v_odd & 0x0F) << 4)
0
Decimal uint8
0x00
Hexadecimal
4.00x
vs FP16 Footprint
✓ Sign Extension Check:
1. 4-Element Floating-Point Block
NVIDIA Ampere/Ada/Hopper Sparse Tensor Cores require exactly 2 non-zeros out of every consecutive 4 elements. The smallest 2 magnitudes are pruned to zero.
Pruning Decision:
2. Condensed Matrix & 2-Bit Metadata Packing
Dense Matrix Shape: (M, N/2) — 50% Weights Stored
Metadata: Two 2-bit column indices fit into 1 nibble (4 bits). Two blocks pack into 1 byte.
[0, 2]
Retained Positions
0x8
Packed 4-Bit Nibble
2.00x
Sparse Tensor Core FLOPs
Transformer Layer Sensitivity Analysis
Layer-by-layer Signal-to-Quantization-Noise Ratio (SQNR) retention & Outlier Resilience
Scheme:
💡 Architectural Insights
Attention Q & K Projections: Most vulnerable to quantization. Phase drift in query/key dot-products leads to exponential attention softmax dispersion.
MLP Gate & Up Projections: Most resilient. Non-linear activations (SwiGLU/GeLU) naturally dampen small magnitude quantization errors.
MLP Down Projection: Acts as an information bottleneck compressing representations back into d_model, requiring higher precision or finer group sizes.
Mixed Precision Strategy:
Keep Q, K, and Down-proj in INT8 or INT4 AWQ (G=32), while compressing Gate, Up, and Value into aggressive 2-bit or INT4 (G=128) to achieve optimal throughput with 0 PPL degradation.
Live Quantizer & SQNR Lab
Test real weight distributions, observe scale factors, discrete integer bins, and exact dB SNR
Preset Tensor:
0.0000
Scale (S)
0
Zero-Point (Z)
0.0 dB
Exact SQNR
Reconstructed Tensor vs Original Comparison
Index
Original (X)
Quantized Discrete (Q)
Dequantized (X̂)
Error (X - X̂)
Python SDK & CLI Usage
Reproduce every evaluation result locally in 2 lines of Python
# Run full automated Pareto frontier sweep on local model
quantbench eval --model meta-llama/Llama-3.2-1B --schemes int8,int4_awq,fp8_e4m3,bitnet --device cuda