TezzLLM Architecture — Deep Dive

Overview

TezzLLM implements a decoder-only Transformer — the same fundamental architecture behind GPT, LLaMA, and other modern language models. Every operation is implemented from scratch in TezzNative with no external dependencies.

The Full Forward Pass

Input Token (integer 0-255) │ ▼ ┌─────────────────────────┐ │ Token Embedding (WTE) │ tok → 128-dim vector │ vocab=256, dim=128 │ └─────────────────────────┘ │ ▼ [x4 transformer layers] ┌─────────────────────────────────────────────┐ │ Transformer Layer │ │ │ │ x = x + Attention(RMSNorm(x)) │ │ x = x + FFN(RMSNorm(x)) │ │ │ │ ┌──────────────────────────────────────┐ │ │ │ Multi-Head Attention (4 heads) │ │ │ │ │ │ │ │ Q = xb @ Wq [128×128] │ │ │ │ K = xb @ Wk [128×128] │ │ │ │ V = xb @ Wv [128×128] │ │ │ │ │ │ │ │ RoPE(Q, K, pos) ← position info │ │ │ │ │ │ │ │ att = softmax(Q @ K^T / √32) │ │ │ │ out = att @ V │ │ │ │ out = out @ Wo [128×128] │ │ │ └──────────────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────────┐ │ │ │ SwiGLU Feed-Forward Network │ │ │ │ │ │ │ │ gate = xb @ Wgate [128→512] │ │ │ │ val = xb @ Wval [128→512] │ │ │ │ h = SiLU(gate) × val │ │ │ │ out = h @ Wproj [512→128] │ │ │ └──────────────────────────────────────┘ │ └─────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────┐ │ Final RMSNorm │ │ Logits = x @ WTE^T │ 128 → 256 logits │ Softmax → probabilities│ └─────────────────────────┘ │ ▼

Next token prediction

Key Components

1. RMSNorm (Root Mean Square Normalization)

Standard LayerNorm computes mean and variance. RMSNorm is faster — it only computes the RMS:

RMSNorm(x) = x / rms(x)  γ

where rms(x) = sqrt(mean(x²) + ε)

In TezzNative:

fn rmsnorm(out:int, x:int, w:int, n:int):
unsafe:
xp:float = x as float

wp:float = w as float

op:float = out as float

ss:float = 0.0
i:int = 0
while i < n:
ss = ss + xp[i]  xp[i]
i = i + 1
ss = 1.0 / fsqrt(ss / (n as float) + 0.000001)
i = 0
while i < n:
op[i] = wp[i]  (ss  xp[i])
i = i + 1

2. Multi-Head Attention with RoPE

Attention splits the embedding into H=4 heads (32 dims each). Each head independently attends to the past:

score[t1, t2] = dot(Q[t1], K[t2]) / sqrt(head_dim)
attn[t]       = softmax(scores[t, :t]) @ V[:t]

RoPE (Rotary Position Embedding) encodes position by rotating Q and K vectors:

Q[i], Q[i+1] = Q[i]cos(θ) - Q[i+1]sin(θ),
Q[i]sin(θ) + Q[i+1]cos(θ)
where θ = pos / 10000^(2i/head_dim)

This makes the attention pattern depend on relative position rather than absolute, enabling better generalization.

3. SwiGLU Activation

Most transformers use ReLU or GELU. SwiGLU (from LLaMA) is more expressive:

SwiGLU(x) = SiLU(gate) × value

SiLU(x) = x  sigmoid(x) = x / (1 + exp(-x))

This requires 2 weight matrices instead of 1, but consistently outperforms simpler activations.

In TezzNative:

tn
fn silu_act(x:float) -> float:
ret x / (1.0 + fexp_fast(0.0 - x))

4. Accurate Mathematical Functions

The key breakthrough in TezzLLM v2: range-reduced exp and log that work correctly for all inputs:

flog(x) — Natural Logarithm:

ln(x) = eln(2) + 2(z + z³/3 + z⁵/5 + ...)
where x = m  2^e, z = (m-1)/(m+1), 0.5 ≤ m < 2

8 terms gives full float64 precision.

fexp(x) — Exponential:

exp(x) = exp(r)  2^k
where x = kln(2) + r, |r| ≤ ln(2)/2 ≈ 0.347

Taylor series on the small residual r is always accurate.

Why this matters: Without range reduction, exp(-15) returns 93,770 instead of 0.0000003. This breaks softmax as the model becomes more confident, causing catastrophic divergence during training.

Training

Cross-Entropy Loss

loss = -log(p_target)

where p_target = softmax(logits)[target_token]

The model predicts a probability distribution over 256 tokens. We penalize it for assigning low probability to the correct next token.

Starting loss: -log(1/256) = 5.545 (uniform random)

Good loss: < 2.0 (model learning structure)

Excellent loss: < 1.0 (model understanding language)

Backpropagation

Gradients flow backward through every operation:

dL/dW = dL/dlogits · dlogits/dW

The full backprop includes:

logits_back — gradient through final matmul + softmax

rmsnorm_back — gradient through final RMSNorm

Layer by layer backward:

- ffn_back — SwiGLU gradient

- attn_back — Attention + RoPE gradient

- rmsnorm_back × 2 (per-layer)

Embedding gradient accumulation

AdamW Optimizer

m = β1  m + (1 - β1)  g           # 1st moment (momentum)
v = β2  v + (1 - β2)  g²          # 2nd moment (RMS)
lr_t = lr  sqrt(1 - β2^t) / (1 - β1^t)   # bias correction

w = w  (1 - lr  wd) - lr_t  m / (sqrt(v) + ε)  # weight update

Parameters used:

β1 = 0.9, β2 = 0.999, ε = 1e-8

Weight decay wd = 0.0 (disabled for Tiny v1)

Gradient clipping at 1.0

Cosine LR schedule with 100-step warmup

Federated Averaging

8 trainers start with different random seeds → learn different features → merge:

w_merged = (w_0 + w_1 + ... + w_7) / 8

This is mathematically equivalent to ensemble averaging. Each trainer sees the same data but from a different starting point, creating diversity that improves the final model.

Weight File Format (TEZW)

Offset Size Field ────────────────────────────────────── 0 4 Magic: 'T','Z','2','W' 4 4 Version: 1 8 4 Header size: 40 12 4 Padding 16 4 n_layer: 4 20 4 n_head: 4 24 4 n_embd: 128 28 4 vocab_sz: 256 32 4 max_seq: 128 36 4 ffn_dim: 512 40 8 each Weights (float64, 1,082,496 values) ──────────────────────────────────────

Total: ~8.66 MB

Why TezzNative?

Traditional ML is done in Python with PyTorch/TensorFlow. TezzNative gives us:

| Feature | Python+PyTorch | TezzNative |

|---------|---------------|------------|

| Setup | pip install (GB of deps) | 1.3MB installer |

| Binary | Needs Python runtime | Single .exe |

| Memory | GC pauses | Zero leaks, pre-allocated |

| Pointer control | None | Direct *float access |

| Compile time | N/A | < 50ms |

| GPU required | Often | Never |

| Min RAM | 4GB (CUDA) | 64MB |

| Understand code | Thousands of layers | ~400 lines visible |

TezzNative lets us see every multiplication, every memory access, every gradient — total transparency and control.