Original title: Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
Zhiliang Chen, Sebastian Ament, David Eriksson, Maximilian Balandat +2 · September 2026
Training large language models requires picking settings called hyperparameters. Scaling laws predict the best settings for giant models, but finding these laws usually takes thousands of expensive test runs. The researchers created a method named Power-Law Entropy Search, or PLES, to solve this problem. Instead of trying to find the single best setting immediately, PLES measures how uncertain it is about the overall scaling law. In each round, it chooses experiments that reduce this uncertainty the most for the least computational cost. This trick naturally pushes the system to run informative small-scale tests rather than costly large ones. Across synthetic tests, surrogate models, and real training runs, PLES found accurate laws using less than one-tenth of the normal computing budget.
You are planning a multi-course dinner for a group where the ideal menu depends on the final number of guests. Cooking full meals for every possible guest count to test them is wasteful and impractical. In this paper, the system runs a few small sample dishes with different ingredient ratios, and uses those results to figure out the best menu proportions for the full party. It picks the samples that give the most reliable clues about how the menu should scale.
Training large language models requires finding optimal settings, which currently demands enormous computational resources. This method reduces that cost dramatically, making AI development faster and more accessible for researchers and companies.
The abstract does not explicitly state limitations. It reports evaluation on synthetic benchmarks, surrogate models, and actual LLM pre-training runs, but does not discuss potential constraints or generalizability beyond these settings.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search — Zhiliang Chen, Sebastian Ament, David Eriksson et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.