Open research
Akeratos is an open lab for training language models on a Mac.
Akeratos trains language models from scratch on Apple silicon with MLX.
Its models pair a small transformer with a large learned n-gram table held in the Mac’s unified memory.
Code and weights are Apache-2.0, and the release publishes its data recipes, checkpoints, training logs and receipts.
Version 0.1.0, released 14 November 2026.
Plenty of memory, modest compute.
Apple’s unified memory lets one consumer machine hold far more trainable parameters than a consumer GPU can. The price is compute: a Mac’s GPU does far fewer calculations per second than a data-centre GPU. Akeratos asks what that much memory is good for.
Its test is a learned table that costs memory and almost no compute. The model reads two of the table’s rows per token, so the table can be far larger than the model while the compute per token stays that of the small model.
Transformer core 162 million active parameters
n-gram table 6.0 billion parameters
Source:Billet’s model card on Hugging Face, 14 November 2026
Hashed n-gram tables are not new. The question is what such a table does when it is trained resident in unified memory on one machine, with no offloading code.
Two rows per token, learned like any weight.
An n-gram is a run of neighbouring tokens: “New York” is a pair, “the United States” a triple. Classic language models counted them. Akeratos learns a vector for each instead.
1Hash
The CPU turns each pair and triple of token ids into a row number with a fixed, seeded hash. Different n-grams sometimes share a row, and frequent ones rarely collide.
2Read
At every position the model reads two rows, one for the pair ending at this token and one for the triple. It adds their sum, divided by √2, to the residual stream before every layer, each layer with its own learned strength.
3Update
A row changes when a batch reads it, under a sparse version of Adam that keeps its statistics per row rather than per entry. Every row starts at zero, so an untrained table adds nothing.
4Write back
Rows go back at 16 bits, rounded up or down at random so small updates survive on average. The proof of concept’s 6.0 billion numbers, 7,812,500 rows of 768, take 12.1 GB with each row’s optimiser state, and its release ships them as trained.
Token ids
The model is at position t.
Hashed on the CPU
bigram United States
trigram the United States
Each is mixed into a row number.
Table in unified memory
Billions of learned numbers, trained with the model.
Table vector
(bigram row + trigram row) / √2
Added before every block, scaled by a learned number per block
Block 1
attention, feed-forward
Block 2
attention, feed-forward
Block L
attention, feed-forward
Next-token logits
49,152 scores
Source:Akeratos’s n-gram memory page on GitHub, 14 November 2026
The table against its twin.
The proof of concept, Akeratos Billet, is a model with 162 million active parameters and a 6.0-billion-parameter table. It was trained on one MacBook Pro with an M4 Max and 48 GB of memory. Beside it is Billet’s twin: the same model on the same tokens, with the table switched off. The gap between the two measures the table’s effect.
| Model | Held-out bits per byte | Table parameters | Active core parameters | Training tokens |
|---|---|---|---|---|
| With the table | 0.8866 | 6.0 billion | 162 million | 3.2 billion |
| Twin, table off | 0.9030 | none | 162 million | 3.2 billion |
A step with the table takes 1.052 times as long as a step without it.
Training with the table peaked at 24.1 GiB of memory.
Bits per byte is how many bits the model needs, on average, to encode each byte of text it has not seen. Unlike raw loss, it compares across tokenizers.
The weights
- Akeratos Billet
The model with 162 million active parameters and its 6.0-billion-parameter table, trained on one Mac.
- Billet’s twin, table off
The same model, trained on the same tokens with the table switched off.
Equal wall-clock time is a separate question: an open problem.
Open enough to rerun.
- Code and weights
Apache-2.0. Weights go to Hugging Face under Cogvera-Labs, ungated: no request, no extra terms.
- Data
Permissively licensed or public domain, each dataset listed with its pinned revision and licence.
- Recipe
The commit, config, seeds, token count, and the commands that rerun training and evaluation.
- Training record
Training logs, the final checkpoint, and at least one intermediate checkpoint.
- Results
The log, config, and evaluation output behind each model’s scores. Scores are published in full, favourable or not.
- Pipeline
Pretraining and evaluation run from a fork of Akeratos on Apple silicon. Nothing needs a paid API, a gated model, or a closed binary.
Run it on your Mac.
You need a Mac with Apple silicon on macOS 14 or later, and uv 0.13.6 or newer, which installs the right Python itself. The smallest model trains on a laptop.
Set up a checkout
Train the smallest model
Run Billet
Rerun the proof of concept and its twin
A rerun needs about 57 GB of free disk to build the data, plus the checkpoints, and each of the two runs takes about four and a half days on an M4 Max.
Reproduce Billet and its twin, on GitHub
If your number does not match ours, open an issue. That is a finding.
Bring a Mac and a question.
Most contributions are a fork and a pull request. Each open problem names its baseline and what counts as an answer.
- Does the table pay off at equal wall-clock?two runs of 3.2 billion tokens
- How does held-out bits per byte change with table size at a fixed core?each point is a run of Billet’s length
- How much do 8-bit and 4-bit table storage hurt the least-trained rows?evaluation alone; Billet’s table takes 11.2 GiB in memory
- How long do the table’s gather, update and write-back take per step on each Apple chip and memory size?bake bench takes 10 minutes or less, on mains power
Source:Akeratos’s open problems on GitHub, 14 November 2026
Or set a record on the Mac speedrun: five minutes of training on a stated class of Mac, then a held-out score. A record stands once a second person reruns it.
The speedrun records, on GitHub
A negative result is an answer.
The work before it.
Akeratos’s table builds on published work; COUNT is listed for comparison.
Akeratos is written in Python on MLX, Apple’s open-source array framework, and trains on FineWeb-Edu with SmolLM2’s tokenizer.
Engram
January 2026
A hashed n-gram memory beside a mixture-of-experts model. It keeps the standard input embedding and serves its table from host memory at inference.
SCONE
February 2025
Produces n-gram embeddings with a second network and caches them for inference.
Over-Tokenized Transformer
January 2025
Adds hashed 2- and 3-gram rows to the input embedding, and reports lower loss as the table grows.
Hierarchical memories (Apple)
September 2025
A 160-million-parameter model that fetches an 18-million-parameter block from a 4.6-billion-parameter memory bank.
modded-nanogpt speedrun
records 62 and 92
Record 62 added a hashed bigram table, starting from zero, straight into the residual stream. Record 92 trained a bigram and trigram table of about 65 billion parameters across eight GPUs, with Adam statistics kept per row. Akeratos starts its rows at zero, as both records do, and adds the table at every layer, as record 62 does.
COUNT
October 2026
Keeps up to 797 GB of n-gram counts in host memory while training on eight H200 GPUs. Its tables hold counts; Akeratos’s rows are learned.
Short answers.
Why does Akeratos exist?
We wanted to do our own research on a Mac, not on rented data-centre GPUs, so the proof of concept trained on the maintainer’s own MacBook Pro.
Does it run on NVIDIA GPUs?
Untested. MLX has a CUDA build for Linux, but Akeratos targets Apple silicon, and no test or training run has used that build.
Who writes it?
Thomas Nicklin maintains Akeratos, and AI coding agents write most of its code.
Is Akeratos a product?
No. It is open research from Cogvera Labs. The proof of concept is a small model that tests the mechanism, as a starting point for other researchers.
If you use it, cite it.
The reference covers the code, the models, and the results.
Thomas Nicklin (2026). Akeratos: an open lab for training language models on Apple silicon, version 0.1.0. https://github.com/cogvera-labs/akeratos