TezzLLM Dataset Preparation Guide

This guide explains how to prepare, clean, and structure training datasets for TezzLLM Tiny models.

The Tokenizer: Byte-Level Model

Unlike traditional LLMs (which use complex subword tokenizers like BPE or WordPiece with vocabularies of 32,000+ tokens), TezzLLM Tiny v1 uses a pure byte-level vocabulary.

How Byte-Level Tokenization Works:

1. Every character is converted to its raw ASCII/UTF-8 byte value.
2. The vocabulary size is exactly 256 (representing all possible byte values: 0 to 255).
3. The dataset file tezzllm_train_v2.bin stores token IDs as 4-byte integers (int32), prefixed with a 4-byte magic signature TEZD and 4-byte token count.

Advantages of Byte-Level Vocab:

• Zero OOV (Out Of Vocabulary): The model can read and write any text, language, symbol, emoji, or formatting character since all UTF-8 strings decompose into raw bytes.
• Extreme Simplicity: No external tokenizer model or vocabulary file is needed during compilation or runtime.
• Language Agnostic: You can train TezzLLM on English, Spanish, Hindi, Chinese, binary code, or music notation without changing a single line of compiler or engine code.

Dataset Formats

You can train TezzLLM on any text file, but structuring the text helps the model learn conversational flow. Here are the three primary formatting patterns:

1. Conversational Format (Q&A / Chat)

This is the recommended format for building chat assistants. Use explicit prefixes so the model learns when to take turns:

Q: Hello, who are you?
A: I am TezzLLM, an AI assistant built in pure TezzNative.
Q: What can you do?
A: I can answer questions, complete text patterns, and assist you with coding.
Q: Tell me a joke.
A: Why did the computer catch a cold? Because it had a bad windows setup!
Q: Goodbye!
A: See you soon! Have a great day.

2. General Knowledge / Prose

For general language modeling or text completion, use raw paragraphs. Separate distinct topics or paragraphs with double newlines:

TezzNative is a systems programming language designed by TezzCorp. It compiles directly to native machine code with zero external runtime requirements. TezzNative features manual memory management with safe blocks, direct pointers, and seamless assembly integration.

The TezzLLM project represents the flagship AI implementation using TezzNative. It includes a multi-layer transformer network, federated weight merging pipelines, and an interactive chat engine running entirely on local CPU resources.

3. Code / Scripting

For code assistance, write files containing comments, function signatures, and implementation blocks:

// TezzNative code snippet
fn add_numbers(a:int, b:int) -> int:
ret a + b

fn main() -> int:
result:int = add_numbers(10, 20)
ret result

Dataset Best Practices

For a high-quality model, follow these empirical guidelines:

| Parameter | Recommended Value | Why |

|-----------|-------------------|-----|

| Minimum Size | 10,000 characters | Below this, the model will overfit rapidly and memorize the training text verbatim. |

| Target Size | 50,000 to 200,000 characters | Excellent balance for Tiny v1 (1M parameters). Allows learning syntax and facts without saturation. |

| UTF-8 Encoding | Strict UTF-8 | Ensure your text editor saves in standard UTF-8 without a BOM (Byte Order Mark) to prevent extra header characters. |

| Newlines | Unix style (\n or LF) | Windows carriage returns (\r\n) consume double tokens per line. Converting to LF saves context window capacity. |

Cleaning Your Dataset:

• Remove Web Scrape Noise: Strip out HTML tags, Markdown links, system scripts, and JSON blocks unless you specifically want the model to learn them.
• Fix Typos: The model will learn and repeat typos in the training corpus.
• Diverse Content: If training a conversational agent, include diverse questions (e.g. greetings, science questions, coding questions, factual requests).

Building Your Dataset

Option A: Manual Compilation

Simply open a text editor (like VS Code or Notepad), paste your clean corpus, and save it as train_data.txt in:

TezzAI/LLM/train_data.txt

Option B: Automated Merging

If you have multiple files (e.g. greetings.txt, facts.txt, code_samples.txt), you can merge them using PowerShell before tokenizing:

Get-Content -Path greetings.txt, facts.txt, code_samples.txt | Out-File -FilePath train_data.txt -Encoding utf8

Step-by-Step Dataset Processing Pipeline

1. Place your dataset: Save your custom corpus to TezzAI/LLM/train_data.txt.
2. Run Tokenizer:

.\tezzllm_data.exe

This reads the text, maps characters to bytes, and writes tezzllm_weights/tezzllm_train_v2.bin.

1. Verify Tokenization:

1. Launch Training:

.\build_v2.ps1

The training script will automatically divide the tokenized dataset among the parallel trainers.