Original title: Graph Machine: Towards Better Pretraining via Edges
Lintai Hou · September 2026
Language models need to track lots of context, but checking every piece of data takes too much compute. Researchers created the Graph Machine to fix this. It keeps an expanding memory state that matches the full input size instead of cutting it down to a fixed size. To search this memory, it uses dynamic routing with items called edges. Edges act like pointers that can be updated smoothly, following links in a way similar to pointer chasing. The team tested this by swapping out 75% of the dense layers in a Qwen3-0.6B model. When retrieving only 2 of 4,096 tokens per key-value head, performance got only slightly worse. Retrieving 4 tokens actually made the best model slightly better.
Imagine you are looking up a recipe in a huge cookbook. Normally, you might flip through every page to find what you need. But here, you have bookmarks that point directly to the relevant pages, and you only check a couple of pages instead of the whole book. In this paper, the GM uses edges like those bookmarks to quickly access only a few pieces of information from a large memory, making the process faster without missing much.
For AI developers, this could mean training large models faster and cheaper by using fewer computations. It shows that most dense layers can be replaced without hurting performance much.
The abstract doesn't specify limitations, but the work is limited to pretraining experiments on a particular model and token set, and only measures loss.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Graph Machine: Towards Better Pretraining via Edges — Lintai Hou
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.