Original title: Correlated initialization of deep residual networks
Felix Benning, Ivan Nourdin, Giovanni Peccati · September 2026
Deep residual networks are computer programs with many layers. When people start setting them up, they usually pick random numbers, called weights, independently for each layer. But what happens if those starting weights are linked across layers instead? The researchers studied networks as they grow infinitely deep with weights correlated from layer to layer. They found that this setup bridges the gap between totally random motion and completely predictable, smooth paths. If you create the starting weights using a special feature function on a correlated Gaussian sequence, the deep network acts like a specific system driven by a Hermite process. How that final system behaves depends entirely on how quickly the correlations fade and a property called the Hermite rank. This shows that the exact pattern of starting connections gives engineers useful control settings that fundamentally reshape how deep networks behave.
Imagine you're painting a fence with a long roller, and each stroke depends on how wet the paint is from the previous stroke. If you paint each stroke independently, the coverage might be uneven, but if you coordinate the strokes based on the paint's condition, you can get a smoother finish. In this paper, correlated initialization is like coordinating the weight settings at each layer so that their effects build smoothly, whereas independent initialization is like random strokes that lead to noisier behavior in deep networks.
This helps neural network designers understand how to set starting weights to influence deep network training. By choosing correlations, they can affect how the network behaves as it gets deeper, which is crucial for building effective very deep architectures.
This work is purely theoretical and does not test these findings on actual neural networks with real data. It assumes specific mathematical conditions, like the initialization coming from a stationary Gaussian sequence, which may not hold in practice. The results are for infinite-depth limits, not finite-depth networks.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Correlated initialization of deep residual networks — Felix Benning, Ivan Nourdin, Giovanni Peccati
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.