TurboQuant-H is a method for 2-bit embedding weight quantization used during the cactus convert process. It uses Hadamard rotation to concentrate coordinates and Lloyd-Max codebooks for quantization.
Quantization (Offline/Conversion):
- Partitioning: Divide the PLI embedding matrix into positional groups of $G=128$ contiguous elements.
- Hadamard Rotation: Apply a normalized Hadamard rotation $\hat{\mathbf{x}}_{v,p} = \bar{\mathbf{H}}G \cdot \mathbf{x}{v,p}$ to each group. The normalization factor is $1/\sqrt{G}$.
- Codebook Training: For each positional group, train a Lloyd-Max codebook $\mathcal{C}_p$ containing 4 centroids (for 2-bit quantization) in FP16.
- Quantization: Map each rotated element to its nearest centroid and store the 2-bit index.
Dequantization (Inference/Runtime):
- Gather: Look up the 2-bit indices and the corresponding positional codebook.
- Scatter: Replace indices with their FP16 centroid values.
- Inverse Rotation: Apply the same Hadamard rotation (since $\bar{\mathbf{H}}_G$ is symmetric and self-inverse) to return to the original space.
- Output: Resulting FP16 embedding is fed to the transformer layer.
QUANTIZATION (offline, during cactus convert)
==============================================
PLI Matrix E (262K x 8190)
|
v
+-----------------------+
| Partition into |
| groups of G=128 |
+-----------+-----------+
|
v
+-----------------------+
| Hadamard rotation |
| x_hat = (1/sqrt(G)) * H_128 * x
| per group |
+-----------+-----------+
|
v
+-----------------------+
| Lloyd-Max codebook |
| Train 4 centroids (2-bit) per position
| across all 262K vocab rows
| C_p = {c1, c2, c3, c4} in FP16
+-----------+-----------+
|
v
+-----------------------+
| Quantize by |
| proximity |
| q = argmin_j |x_hat_i - c_j|
| Store 2-bit indices per element
+-----------+-----------+
Output: 2-bit index tensor + 64 FP16 codebooks
Effective: 2.125 bits/element
DEQUANTIZATION (at inference, per token)
=========================================
Token IDs
|
v
+-----------------------+
| Gather 2-bit indices |
| + codebook per pos. |
+-----------+-----------+
|
v
+-----------------------+
| Scatter codebook |
| Replace 2-bit indices with FP16
| centroid values from C_p
+-----------+-----------+
|
v
+-----------------------+
| Hadamard rotation |
| x_tilde = (1/sqrt(G)) * H_128 * scatter(...)
| (same as forward, |
| H_bar is symmetric: H_bar = H_bar^T = H_bar^-1
+-----------+-----------+
|
v
FP16 embedding -> feed to transformer layer
