Restricting residual mixing to pure rotations stops training instability in deep models.

In neural networks, information often flows through a single path called a residual streamresidual streamA pathway in a model that carries information without direct modification.. Hyper-ConnectionsHyper-ConnectionsA modification in neural networks where multiple data streams are mixed instead of one residual stream. add multiple parallel streams and mix them together. Previous mixing methods either grew too large, causing instability, or made the streams too similar over time. oHC mixes them by rotating the streams, which keeps their size steady and their differences distinct. This leads to better training stability and performance.

Why it matters

This helps people training large neural networks get more reliable results without instability. It means models can be deeper and more accurate across different tasks, which affects researchers and engineers building AI systems.

Questions answered