The weight format

The file

A model reaches the chain as one file, the artifact, written by curta convert from a llama2.c checkpoint and its tokenizer. The artifact is the model in integers: int8 weights with a scale per row, fixed-point coefficients, lookup tables for the nonlinear functions, and the tokenizer. Conversion is the one place floating point is used. Everything that runs afterwards, in the program, the CLI and the browser, reads these integers.

The format’s tag is CURTAW01. Every scalar in the file is little-endian. The file is a sequence of sections with no gaps between them, and each section becomes the bytes of one weight account. The first 4,096 bytes of section zero are the header.

section 0   header (4,096 bytes)
            normalization coefficients
            rotary coefficients
            exponential table
            SiLU table
            tokenizer entries
            sorted token ids
            token strings
section 1   matrices: the embedding, the classifier, and every layer of a small model
section 2…  one section per layer of a larger model

The header

The header is 4,096 bytes. It holds the configuration, the section table, the offsets of everything in section zero, and one descriptor per matrix. Bytes the tables below leave out are zero.

The offsets at 40 to 64 count from the start of section zero, which is the start of the file, and each points past the header. The head size is dimension divided by heads, an even number up to 64.

Sections

The section table starts at byte 128. Entry i is 16 bytes at 128 + 16 × i:

Section zero starts at offset zero and each section starts where the one before it ends. A section is up to 10,483,712 bytes: a Solana account’s 10,485,760 bytes less the 2,048-byte header of a weight account.

How the matrices are divided depends on the count of matrix weights, which for dimension d, hidden dimension h, key and value dimension k, vocabulary v and L layers is:

d × v  +  L × (2 × d × d  +  2 × d × k  +  3 × d × h)

• Under 1,000,000: the file has two sections, and section one holds every matrix.
• 1,000,000 or more: the file has L + 2 sections. Section one holds the embedding and the classifier, and section 2 + layer holds the seven matrices of that layer.

No matrix crosses a section, and every matrix starts at a multiple of eight bytes within its section.

Matrix descriptors and tensor order

The matrix table starts at byte 512. Descriptor i is 24 bytes at 512 + 24 × i:

A model has 2 + 7 × layers matrices, always in this order. Layer l starts at index 2 + 7 × l.

The key and value dimension is dimension times key and value heads divided by heads. When a checkpoint ties the classifier to the embedding, descriptor 1 is a copy of descriptor 0 and the bytes are stored once.

In stories15M, with dimension 288, hidden dimension 768 and vocabulary 32,000, a row of 288 columns is 296 bytes and a row of 768 columns is 776 bytes. The embedding is 32,000 rows of 296 bytes, the 9,472,000 bytes of section one. A layer is four matrices of 288 rows of 296 bytes, two of 768 rows of 296 bytes and one of 288 rows of 776 bytes, the 1,019,136 bytes of each layer section.

Rows and quantization

A matrix is its rows, one after another. Each row is:

Quantization is per row, with round-to-nearest:

1. The scale is the largest absolute value in the row divided by 127. A row of zeros takes the smallest scale the field holds.
2. Each weight is the float divided by the scale, rounded to the nearest integer and kept within -127 to 127.
3. The scale is stored as itself times 2²⁴, rounded, and at least 1.

scale    = max(|w|) / 127
q[i]     = clamp(round(w[i] / scale), -127, 127)
stored   = round(scale × 2^24) as u32, then 4 zero bytes, then (q[i] + 128) as u8 for each column

Storing the weight plus 128 makes every weight byte unsigned, which is what lets the forward pass multiply rows without sign-extending each weight. A row times a vector of activations is the sum of the stored bytes times the activations, less 128 times the sum of the activations, times the row’s scale, shifted right by 24 bits. The inference core describes that arithmetic.

Coefficients and lookup tables

Section zero carries everything the forward pass needs beside the matrices. Each block sits at the offset the header names for it, in this order after the header.

normalization coefficientsi32 eachoffset at header byte 40

The weights of every RMS normalization, in fixed point with 24 fractional bits. For each layer, dimension values for the normalization before attention, then dimension values for the one before the feed-forward block. After the last layer, dimension values for the final normalization. (2 × layers + 1) × dimension values in all.

rotary coefficientsi32 pairsoffset at header byte 44

For each position from 0 to context minus one, and for each pair j from 0 to half the head size minus one, the cosine and then the sine of position / 10000^(2j / head size), each with 24 fractional bits. context × head size values in all.

exponential tableu32 × 4,097offset at header byte 48

Entry i is e^(−i / 256) with 30 fractional bits, for i from 0 to 4,096. Softmax in attention and in sampling reads it.

SiLU tablei32 × 4,097offset at header byte 52

Entry i is SiLU of i / 128 − 16 with 16 fractional bits, covering inputs from -16 to 16 in steps of 1/128. Above 16 the function is its input, and below -16 it is zero.

The tokenizer section

The tokenizer is stored with the weights, in three blocks of section zero. The program tokenizes the prompt and decodes the output from these bytes.

Tokenizer entries

One 16-byte entry per token id, at the offset in header byte 56. Entry id is at that offset plus 16 × id:

The score key is made from the bits of the float score. A score with its sign bit clear takes its bits with the top bit set, and a score with its sign bit set takes its bits inverted. A higher score is a higher key, so merging compares integers.

Sorted token ids

One u32 per token, at the offset in header byte 60: every token id, sorted by the bytes of its token. Looking a piece of text up is a binary search over this list, comparing bytes.

Token strings

The bytes of every token, one after another in id order, from the offset in header byte 64 to the end of section zero, which is padded to a multiple of eight bytes.

How the ids are used

• Token 1 opens every prompt. Tokens 1 and 2 end a run when the model produces them: the id is kept in the output and no text is added.
• Token byte + 3 is the single byte byte. A character with no token of its own is encoded as its bytes this way, which is why a vocabulary has at least 259 entries.
• A token whose bytes read <0xXX> decodes to the one byte with that hexadecimal value.

Encoding follows llama2.c: the opening token, a space prefix, one lookup per UTF-8 character, then repeated merging of the adjacent pair whose merged token has the highest score. The inference core shows these steps inside a run.

The hash

The weights hash is computed over the sections, the same bytes that go into the weight accounts. It is a chain of SHA-256:

1. Start with 32 zero bytes.
2. Take each section in order, 65,536 bytes at a time. The last chunk of a section ends where the section ends.
3. For each chunk, the hash becomes SHA-256 of the previous hash followed by the chunk.

The value after the last chunk of the last section is the weights hash. curta convert prints it, the publisher declares it when creating the model, and sealing recomputes it on chain one chunk per transaction. Models describes sealing.

import { readFile } from 'node:fs/promises'
import { decodeArtifact, weightsCommitment } from '@curta/sdk'

const artifact = decodeArtifact(new Uint8Array(await readFile('stories260K.curta')))
const hash = await weightsCommitment(artifact.sections)
console.log(Buffer.from(hash).toString('hex'))

What convert reads

Conversion takes two files in the llama2.c formats and fails on any file that does not match them exactly.

The checkpoint

A 28-byte header of seven 32-bit integers: dimension, hidden dimension, layers, heads, key and value heads, vocabulary, context. A positive vocabulary means the classifier shares the embedding’s weights, and a negative one means the absolute value is the vocabulary and the classifier follows as its own tensor. Then 32-bit floats, in this order:

1. token embedding, vocabulary × dimension
2. attention normalization weights, layers × dimension
3. query matrices, layers × dimension × dimension
4. key matrices, layers × key and value dimension × dimension
5. value matrices, the same size
6. attention output matrices, layers × dimension × dimension
7. feed-forward normalization weights, layers × dimension
8. feed-forward gate matrices, layers × hidden dimension × dimension
9. feed-forward down matrices, layers × dimension × hidden dimension
10. feed-forward up matrices, layers × hidden dimension × dimension
11. final normalization weights, dimension
12. the checkpoint’s own rotary tables, context × head size floats, which conversion passes over and computes afresh
13. the classifier, vocabulary × dimension, when it is not shared

Every float is finite and the file ends with the last tensor.

The tokenizer file

A u32, the longest token’s length, up to 128. Then for each token id in order: the score as a 32-bit float, the length of the token’s bytes as a u32, and the bytes. The file ends with the last token.

Read an artifact

decodeArtifact in @curta/sdk checks the tag and returns the configuration and the sections as byte ranges of the file. The Rust side is Metadata in curta-core, which reads section zero in place.

import { readFile } from 'node:fs/promises'
import { decodeArtifact } from '@curta/sdk'

const bytes = new Uint8Array(await readFile('stories260K.curta'))
const { config, sections } = decodeArtifact(bytes)

console.log(config)                           // dim, hiddenDim, layers, heads, kvHeads, vocabSize, contextLength
console.log(sections.map((s) => s.length))    // one length per weight account

// Matrix descriptor i is 24 bytes at 512 + 24 × i of section zero.
const view = new DataView(sections[0].buffer, sections[0].byteOffset)
const matrix = (i: number) => ({
  section: view.getUint32(512 + 24 * i, true),
  columns: view.getUint32(516 + 24 * i, true),
  rows: view.getUint32(520 + 24 * i, true),
  rowBytes: view.getUint32(524 + 24 * i, true),
  offset: Number(view.getBigUint64(528 + 24 * i, true)),
})
console.log(matrix(0))                        // the embedding

The native core binary prints the same directory with curta-core inspect, described in the CLI reference.