Original title: Gradient-Free Optimization for Matrix functions
Sawyer Allen, Cash Cherry, Aidan Eck, Stephen Becker +1 · September 2026
Training large AI models on normal computers is tough. Their weights are arranged in grids of numbers called matrices. Full gradients show the best direction to update every number at once, but calculating them takes too much memory. Computers can still find directional derivativesA way to measure how a function changes in a specific direction without calculating the full gradient., which measure slope along just one chosen path. The researchers treated finding the true gradient as a puzzle to rebuild from these limited clues. They built a new way to estimate the gradient, replacing an adjointA mathematical operator used in standard gradient estimation methods for differentiation. calculation with a pseudoinverseA generalized inverse of a matrix that helps in estimating gradients more accurately in this context.. When gradients have an approximate low-rank structure, meaning their rows or columns share repeating patterns, tools from matrix sensingTechniques from signal processing that recover a matrix from limited measurements, used here for gradient estimation. can estimate them with high accuracy. Although this takes extra computation, pairing it with spectral descentAn optimization method for matrix problems that uses spectral properties like eigenvalues to update variables efficiently. amortizes the cost. Testing this on synthetic functions with approximately low-rank gradients, they found that using this low-rank property leads to much faster convergence to good approximate solutions.
Imagine tuning a guitar by ear, but you can only pluck one string at a time. Normally, to get the whole guitar in tune, you'd need to adjust each string based on how it sounds alone, which might not account for how they harmonize together. In this paper, instead of checking every string individually, the method uses the sound from one string to estimate how all strings should be tuned, especially if the harmony is simple and predictable. This makes the tuning process faster and more efficient.
This helps people train large AI models on regular computers by reducing memory needs. It makes advanced AI more accessible without specialized equipment.
The abstract doesn't say if this works for real neural networks with non-low-rank gradients. It only demonstrates results on synthetic functions with approximately low-rank gradients. The computational cost is high but can be amortized with specific optimizers, so practical use depends on that.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Gradient-Free Optimization for Matrix functions — Sawyer Allen, Cash Cherry, Aidan Eck et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.