How it works

A forward pass in turns

Producing one token from a transformer is a fixed sequence of integer operations: embed a token, pass it through every layer, score the vocabulary, pick the next token. One transaction on Solana may use 1,400,000 compute units, and a token of stories15M takes about 55.1 transactions of that size. Curta cuts the sequence wherever a transaction’s compute ends and keeps the cut in an account.

1. The caller creates a run account and sends open_run. The account receives the prompt, the parameters and the budget, and its position is set to the first stage.
2. Anyone sends turn. The program reads the position, computes forward from it until the turn’s compute allowance is used, and writes the new position and every intermediate value back into the run account.
3. The next turn continues from that position. Whoever sends it, the bytes it finds and the bytes it leaves are the same.
4. Each time the sampling stage finishes, a token id and its text are appended to the run account. The run is complete at the end token or at max tokens.

A turn carries no model input. Its arguments are the turn index it expects, a nonce that makes a resent transaction distinct, and the count of matrix weight accounts it names. Everything it computes comes from the run account and the model’s sealed weights.

The run account is the register

The run account is one account, created at its full size before the run opens. After a 256-byte header of run and budget fields comes the working state of curta-core, which holds:

• the cursor: stage, layer, row, position, and inside attention the stage and index within the current head
• the prompt bytes and the input tokens they become
• the output: one token id per generated token and up to 128 text bytes per token
• the working vectors of the current position: the activation vector, two scratch vectors, the query, the feed-forward gate and up vectors, the scores over the vocabulary and the attention scores
• the attention cache: a key vector and a value vector for every layer at every position the run has passed
• the sampler’s running maximum, sum and draw, and the generator’s state

Working vectors are 64-bit fixed-point words with 16 fractional bits. The attention cache stores the same precision in 32-bit words, and every store into it checks that the value fits. The cache has room for the positions the run can reach, one plus the prompt’s bytes plus max tokens, up to the model’s context length. The program reads and writes these buffers in place in the account.

Because every intermediate value is in the account, a turn can stop inside a matrix, between two rows, or inside one attention head, and the next turn resumes at that row. Runs and turns lists the header’s fields, and Accounts the byte layout.

The stages of the pass

The cursor’s stage is a number from 0 to 20. Stages 0 to 2 tokenize the prompt once. Stage 3 runs once for every position, and stages 4 to 16 once for each layer of it. Stages 17 to 19 run for every generated token.

The prompt is consumed position by position. After the last layer’s residual, a prompt position that has another prompt token after it goes straight to that token’s embedding: it fills the attention cache and skips the final normalization, the classifier and the sampler. The last prompt position runs all three and produces the first generated token. Each generated token is then embedded as the next position’s input.

Tokenizing follows llama2.c: the begin token, a space prefix, a lookup per UTF-8 character with a fallback to bytes, then merges of the highest-scoring adjacent pair. It runs in the program, in turns of its own that read only section zero of the weights, and ends at the embedding of the first token.

The turn planner

The turn planner is the part of curta-core that decides where a turn stops. It divides the forward pass by compute units and by nothing the sender supplies. It walks the stages from the cursor in chunks: a run of rows of one matrix, a run of positions or channels in one attention stage, a run of vocabulary entries in the sampler, or one whole small stage. Each chunk has a cost in compute units computed from the model’s configuration and the cursor.

A matrix row of n input weights costs 37 × floor(n / 8) + 8 × (n mod 8) + 34 compute units, or 42 in place of 34 when the result goes into the attention cache, and a matrix chunk adds 6 × n + 350 once. The planner takes as many rows as the remaining allowance holds and stops mid-matrix when it must. Inference core gives the cost of every stage.

A turn’s plan ends at the first of three conditions:

• the next chunk does not fit in what is left of the allowance
• the next chunk needs a third matrix weight account, where a turn names section zero and up to two matrix sections
• the run completes

The plan returns four things: the matrix sections the turn reads, the compute units it estimates, the most tokens the turn can complete, and the cursor where it ends. Each has a use.

• The sender runs the planner in WebAssembly to learn which weight accounts the turn’s transaction names. The SDK’s turn does this for you.
• The program runs the same planner on the same bytes, checks that the named accounts are the planned ones, and reserves the turn’s cost for the most tokens it can complete before it computes anything.
• The quote tokenizes your prompt and steps the planner from the first position to max tokens, counting turns. That count is the turns of quote in @curta/sdk and of the console’s quote. Fees turns it into a budget.

What one turn computes

How much of the pass fits in one turn depends on the model. A turn does not stop at a token or a position: when the allowance permits it carries the normalization, the attention, the sampling, the decoding and the next token’s embedding along, and it continues across a layer boundary.

• On stories260K, with dimension 64 and a vocabulary of 512, one turn usually finishes a whole position and begins the next. Its reference run of 12 tokens takes 20 turns, tokenizing and the four prompt positions included.
• On stories15M, with dimension 288 and a vocabulary of 32,000, the classifier alone is 32,000 rows of 288 weights, more than nine million multiply-accumulates for each generated token. A generated token takes about 55.1 turns and a prompt position about 22.25.

Attention grows with the run: each head attends over every position cached so far, so a later position costs more compute than an earlier one. Its four stages are separately resumable, which lets a turn stop and resume inside a head.

When a turn finishes, the program has also done the run’s bookkeeping in the same transaction: it pays the turn reward to the signer, accrues the royalty for each token the turn completed and the protocol share, counts the turn, records the cost of the next turn, and emits an event that carries every token id the turn completed.

Sampling with temperature and seed

The sampler runs in the program over the classifier’s scores. Temperature is a number from 0 to 16, stored in the run as fixed point with 16 fractional bits. The seed is an unsigned 64-bit integer.

Temperature zero

One pass over the vocabulary takes the entry with the highest score. The seed is not read.

Temperature above zero

Three passes over the vocabulary:

1. Maximum. Finds the highest score.
2. Sum. For each entry, divides its distance below the maximum by the temperature and reads the exponential of that from a lookup table stored with the weights. The table holds exp(-i / 256) for i from 0 to 4,096 in fixed point with 30 fractional bits. The pass sums these integer probabilities.
3. Draw. Advances the generator, reduces its output modulo the sum, and walks the vocabulary subtracting each entry’s probability until the remainder falls inside one. That entry is the token.

The generator is xorshift64* with its state in the run account, starting from the run’s seed and advanced once per sampled token:

x ^= x >> 12
x ^= x << 25
x ^= x >> 27
draw = (x * 0x2545f4914f6cdd1d) mod sum

The sampler is resumable like every other stage: on a large vocabulary a pass spans turns, and its running maximum, sum and remainder wait in the run account between them.

After the choice

The token’s id is appended to the output ids. Its text piece, decoded from the tokenizer in section zero, is appended to the output bytes. The begin token and the end token end the run: the id stays in the output ids and adds no text. Otherwise the run ends when the count of generated tokens reaches max tokens, or the token becomes the input of the next position.

Measured on a validator

Both genesis models ran the prompt “Once upon a time” at temperature zero with seed 42. The prompt is 16 bytes and becomes five tokens with the begin token, so four positions come before generation.

Compute units per turn, from the confirmed receipts of those runs:

Turns between one completed position and the next, in order:

The first prompt position’s count includes tokenizing. A turn uses about 1,340,000 of the 1,400,000 compute units of its transaction.

Why it is bit-exact

A run’s output is a pure function of the sealed weights, the prompt, the parameters and the seed. Six properties of the build make that hold on every target.

• Integers only. Weights are int8 with one fixed-point scale per row. Activations are fixed-point words. Normalization uses an integer square root, and the exponential, SiLU and rotary coefficients are lookup tables stored with the weights. Floating point is used once, by the converter that writes the weight file.
• One source. curta-core compiles into the program (SBF), the CLI (native), the browser (WebAssembly). The same functions tokenize, compute, sample and decode on all three.
• Sealed weights. A sealed model’s weight accounts take no further writes, and sealing checks their bytes against the hash the model declared. Every turn of every run reads the same weights.
• Nothing from the sender. A turn’s transaction supplies no token, no score and no stopping point. The program checks the weight accounts it names against the model.
• Deterministic cuts. The planner places each turn’s end from the configuration and the cursor alone. It reads no remaining-compute counter, so a turn ends at the same row on a validator, in a native build and in a browser.
• A seeded generator in the account. Sampling draws from the run’s seed through a generator whose state is run account data.

The reference runs above were recomputed by the native build and by the WebAssembly build, and every token and every text byte matched the chain’s. The float implementation the checkpoints come from, llama2.c, produces the same first 32 tokens and text for both models on that prompt at temperature zero.