Models and the weight format

The architecture family

Curta runs transformer language models in the llama2.c architecture: token embeddings, a stack of layers with RMS normalization, multi-head attention with rotary position encoding and a SiLU feed-forward block, then a final normalization and a classifier over the vocabulary.

The curta program runs any model of this family from the configuration in its Model account:

The integer weight format

A model on Curta is integer-only. curta convert turns a llama2.c checkpoint into this format:

• Weights are int8, quantized per row with round-to-nearest. A model has two matrices outside its layers, the embedding and the classifier, and seven per layer: query, key, value, attention output, and the feed-forward gate, down and up matrices.
• Each matrix row is a scale in fixed point with 24 fractional bits, then one byte per weight holding the weight plus 128, padded to a multiple of eight bytes.
• Activations are fixed point with 16 fractional bits in 64-bit words. The attention cache keeps keys and values at the same precision in 32-bit words.
• Normalization, softmax, the SiLU activation and the rotary position encoding run on integer arithmetic and lookup tables. The exponential table and the SiLU table have 4,097 entries each, and the tables are stored with the weights.
• The tokenizer is stored with the weights: every vocabulary entry with its score and its bytes, and the token ids sorted by their bytes for lookup.

The file is little-endian and opens with the tag CURTAW01 in a 4,096-byte header. The header carries the seven configuration values, the section count, the offset and length of each section, the offsets of the tables and the tokenizer, and one descriptor per matrix: section, columns, rows, row length and offset.

On the prompt “Once upon a time” at temperature zero, both genesis models produce the same first 32 tokens in this format as the floating-point llama2.c original. Replay and determinism tells what the integer format makes possible.

Weight accounts

Weights live in weight accounts: keypair accounts created at full size and owned by the curta program. A Solana account holds up to 10,485,760 bytes. A weight account opens with a 2,048-byte header that names its model and its section and carries the upload bitmap, and the section’s bytes follow.

The format divides a model into sections, one weight account each. Section zero holds the header, the tokenizer and the lookup tables. A model with under a million matrix weights keeps all its matrices in section one. A larger model keeps the embedding and the classifier in section one and each layer in a section of its own, and no matrix crosses a section.

A turn reads section zero and up to two matrix sections, the ones its position in the forward pass needs. Every run reads the weight accounts and no run writes them, so all runs on a model share them.

Sealing and the weights hash

A Model account records the hash of its weights. The hash is a chain of SHA-256 over the sections in order, 65,536 bytes at a time: it starts as 32 zero bytes, and each step hashes the previous value followed by the next chunk.

seal_model recomputes that chain from the bytes written to the weight accounts, one chunk per transaction, and checks the header, the matrix descriptors and every tokenizer entry on the way. The last step compares the result with the recorded hash and marks the model sealed.

A sealed model never changes. It appears in the model picker on /run, and its page on /models shows the weights hash with each weight account and its size.

The genesis models

The two genesis models are Andrej Karpathy’s TinyStories checkpoints for llama2.c, published under the MIT license and converted with curta convert.

Turns per token

Turns per token is how many turns one token’s forward pass takes on a model. It is a measured figure, taken from the program running on a validator, and each model’s page shows it for prompt tokens and for generated tokens. Runs and turns has the figures for the genesis models.

Behind it is the cost of one multiply-accumulate, the step a matrix row repeats for each weight: