TetraNet: Power-of-Two Quaternary Weights for Small Decoder-Only Language Models

1Introduction

Native low-bit language models seek to reduce the cost of storing and applying the many linear-projection weights in a transformer. BitNet b1.58 showed that ternary weights can be a useful large-scale training target [3]; efficient ternary inference systems have subsequently demonstrated that the kernel implementation is as important as the weight alphabet [7]. This work explores a deliberately narrow question: if a deployment allocates exactly two bits of storage per projection weight, can the fourth state improve the weight grid while preserving a shift-friendly inner product?

We answer this question with the symmetric quaternary set

Unlike a uniformly spaced four-level grid, both intermediate magnitudes are powers of two. Given integer activations, multiplication by $c$ in the core dot product is therefore a signed right shift. The proposal does not eliminate all floating-point work: per-layer scale application, normalization, attention, and the language-model head remain floating point in this prototype.

The empirical scope is intentionally modest. We train a 9.4M-parameter Llama-style decoder [6] on TinyStories [1], compare it with full-precision, ternary, and uniform-four-level baselines, and freeze the validation and energy artifacts in the repository. The principal findings are:

• Both fixed quaternary grids improve over the repository’s matched BitNet-style ternary baseline on the frozen validation subset; $c = 1/4$ is lower by 2.29 perplexity points.
• Under the stated analytical memory and operation model, fixed two-bit quaternary storage has essentially the same estimated total energy as ternary weights stored in a two-bit container, while using a fourth state.
• The evidence is a single-run, small-scale pilot. It does not establish statistical significance, scaling behavior, measured energy, or a wall-clock advantage over optimized dense or ternary libraries.

2Related Work

BitNet b1.58 is a native ternary language-model approach in which weights take values in $\{-1, 0, 1\}$ [3]. Its reported large-scale results are not directly comparable to this study: our ternary comparator is a compact, local implementation with abs-mean scaling and a clipped straight-through estimator (STE), not a reproduction of the complete BitNet b1.58 recipe. Bitnet.cpp illustrates the separate systems problem of mapping low-bit weights to efficient edge inference kernels [7]. We provide correctness-tested C++ and Triton proof-of-concept kernels, but do not compare them with Bitnet.cpp or highly tuned BLAS libraries.

The model uses a conventional small decoder design with rotary position embeddings [5] and SwiGLU feed-forward blocks [4]. TinyStories was designed as a controlled setting for studying small language models [1]. Finally, our energy accounting follows the order-of-magnitude operation and memory-access cost methodology popularized by Horowitz [2]; it should be read as a sensitivity model, not a measurement on contemporary hardware.

3Method

3.1 · Scaled quaternary quantization

For a latent projection weight matrix $W$, the implementation computes a detached layerwise scale

Let $b = (1 + c)/2$. The quantized weight used in the forward pass is

The backward pass uses a clipped STE, passing the incoming derivative to $x$ only when $|x| \le \tau$, with $\tau = 1$ in all fixed-$c$ experiments. The fixed variants register $c$ as a buffer rather than optimizing it.

3.2 · Shift-oriented inference representation

For $c = 1/2$ and $c = 1/4$, a two-bit code can store the four states as $00 \mapsto +1$, $01 \mapsto -1$, $10 \mapsto +c$, and $11 \mapsto -c$. The included specialized path quantizes activations to signed int8-derived integer codes, packs four weights per byte, accumulates in int32, and evaluates the intermediate states with one or two arithmetic right shifts. The final activation and weight scales are applied after accumulation. This describes the available inference path, not the floating-point training/evaluation operator used to produce the perplexities in Section 5.

4Experimental Setup

4.1 · Model and training protocol

The table below summarizes the frozen experiment configuration. The tokenizer is a byte-level BPE tokenizer trained with a 4,096-token vocabulary. All Q, K, V, output, gate, up, and down projections are replaced by the chosen linear class. Token embeddings, the tied output head, normalization, rotary embedding operations, and attention products remain full precision.

We did not set or record a random seed and store one final run per condition. Consequently, differences below are point estimates rather than multi-seed estimates with confidence intervals.

4.2 · Baselines and metrics

We compare five conditions. Full precision uses ordinary projection linears. Ternary uses abs-mean scaling, values $\{-1, 0, 1\}$, and the same clipped STE; we refer to it as “BitNet-style ternary” only as a local comparator. Uniform 2-bit uses values $\{-1, -\tfrac13, \tfrac13, 1\}$; it is an informative but imperfect ablation because its implementation uses an abs-maximum scale while the ternary and quaternary implementations use abs-mean scale. The remaining conditions are the proposed fixed $c = 1/4$ and $c = 1/2$ grids.

Validation perplexity is the exponentiated mean token cross entropy. The frozen evaluation concatenates tokenized stories, then evaluates 180 non-overlapping blocks; it records 92,160 nominal input tokens. We report no generation-quality evaluation.

For energy, we split per-token operations into quantized projections, floating-point attention score/value products, and the full-precision output head. The model assigns 4.6 pJ to an FP32 multiply-add, 0.13 pJ to a ternary select/negate plus integer add, 0.23 pJ to an int8 multiply plus add or the $c = 1/2$ shift path, and 0.33 pJ to the $c = 1/4$ shift path. It adds a 20 pJ/bit DRAM cost for one full decode-time read of all weights, including full-precision embeddings and one FP32 scale per quantized projection. These choices make memory dominant and should not be generalized to an on-chip-cache regime.

5Results

5.1 · Language modeling quality

The figure below and the table that follows use the frozen ppl 10m.json artifact. Both proposed variants have lower perplexity than the matched ternary run. The strongest point estimate is $c = 1/4$, at 76.73 versus 79.03 for BitNet-style ternary. Uniform 2-bit is also lower than the ternary comparator, but the differing scale statistic means this table does not isolate grid geometry alone. All low-bit models remain materially behind the full-precision condition.

The repository also contains a learned-$c$ exploration with an L1 multi-well snapping objective. It snapped all 56 scale parameters to $1/4$ or $1/2$ but gave 124.16 perplexity on the same evaluation. We treat this as a negative result, and do not use it as evidence for a learned heterogeneous design.

5.2 · Analytical compute and memory energy

The architecture has 8,388,608 quantized-projection MACs per generated token at context length 512, 2,097,152 full-precision attention MACs, and 1,048,576 full-precision output-head MACs. The figure below visualizes the resulting accounting. Under the DRAM re-read assumption, the two fixed quaternary variants have a total estimate of approximately 1.02 mJ/token, about 16.8% of the full-precision estimate. This ratio is principally caused by fewer bits read from DRAM, not by the small difference between a shift and an integer multiply in the compute term.

The distinction between the two ternary rows matters. The idealized 1.58-bit row gives ternary a 7.0% total-energy advantage over the fixed two-bit quaternary grids. At equal two-bit packing, the difference is below 0.2% in this model. This is a representation-storage comparison only; the current ternary implementation in the proof-of-concept kernel uses int8 codes, not an optimized entropy-coded ternary format.

6Implementation Status and Limitations

The official code implementation repository contains a C++ extension and optional Triton kernels that pack quaternary weights four per byte and implement the signed-shift accumulation path. Tests check these kernels against their integer reference computations. It also contains CPU microbenchmarks and end-to-end generation harnesses. Those measurements use equal-effort research kernels, only three end-to-end runs in the frozen artifact, and lack the hardware metadata and optimized baselines needed for a systems claim. We therefore do not report wall-clock speedups in this paper.

Several limitations materially constrain interpretation:

• The study has one seed, one small model size, one short training run, and one domain. Perplexity differences of one to two points require multi-seed and larger-scale confirmation.
• The uniform 2-bit comparator differs in its scaling statistic, so it does not cleanly separate the effect of four states from the effect of the quantizer geometry.
• Perplexity is evaluated with floating-point quantized weights. The specialized inference path adds int8 activation quantization and integer shift rounding, neither of which has been evaluated end to end for quality.
• The energy numbers use 45nm operation constants and a full DRAM weight re-read per token. Cache residency, hardware process, vectorization, scale handling, and memory traffic can all change absolute and relative energy.
• The local ternary baseline is BitNet-style, not a controlled reproduction of BitNet b1.58, and the prototype is not a comparison against Bitnet.cpp or vendor GEMM libraries.

The next revision should fix seeds, train multiple runs at substantially larger model and token budgets, align scale choices across ablations, evaluate the integer inference path for quality, and benchmark packed ternary and quaternary kernels against production implementations on documented hardware.

7Conclusion

This initial study evaluates a simple alternative to ternary weights under a fixed two-bit storage assumption. Fixed power-of-two quaternary grids retain a shift-oriented dot-product path and produced lower frozen validation perplexity than a matched local ternary baseline in a 9.4M TinyStories pilot. At equal two-bit storage, the analytical energy model places ternary and quaternary at near parity; the proposed grid’s motivation is therefore improved representation use rather than a claimed energy or throughput win. The result is promising enough to justify a larger controlled study, but its current evidence should be read with the limitations above.

TetraNet in one line: a two-bit register holds four states; with $c \in \{\frac14, \frac12\}$ the extra two cost nothing but a shift, and they beat the ternary baseline that leaves them idle.

Reproducibility note

Checkpoints and the TinyStories data are not included in the source tree, so full training or evaluation replay requires supplying them separately. The frozen validation (ppl 10m.json) and energy artifacts define the numbers reported above.

1. Ronen Eldan and Yuanzhi Li. TinyStories: How Small Can Language Models Be and Still Speak Coherent English? arXiv preprint arXiv:2305.07759, 2023.
2. Mark Horowitz. Computing’s energy problem (and what we can do about it). ISSCC Digest of Technical Papers, pp. 10–14, 2014.
3. Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv preprint arXiv:2402.17764, 2024.
4. Noam Shazeer. GLU Variants Improve Transformer. arXiv preprint arXiv:2002.05202, 2020.
5. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint arXiv:2104.09864, 2021.
6. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971, 2023.
7. Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, and Furu Wei. Bitnet.cpp: Efficient Edge Inference for Ternary LLMs. arXiv preprint arXiv:2502.11880, 2025.

@techreport{shinde2026tetranet,
title   = {TetraNet: Power-of-Two Quaternary Weights for
           Small Decoder-Only Language Models},
author  = {Shinde, Abhishek},
year    = {2026},
month   = jul,
note    = {Published on Oropis},
url     = {https://oropis.com/posts/tetranet/}
}