A new weight-initialization formula keeps deep networks stable when the activation function is randomized.

Deep neural networks learn by passing signals through layers of virtual connections. When these connections start with bad starting values, or weights, signals can shrink to nothing or explode, ruining training. Researchers designed a new starting weight formula for RReLU, an activation function that turns incoming signals into outgoing ones. RReLU uses a random slope for negative inputs. The researchers accounted for that random slope to create a rule that keeps signal strengths steady as data moves forward and backward through the network. They tested it on feedforward networks and convolutional neural networks. Compared to Xavier and He initialization schemes, their method provided more stable optimization and helped networks finish training, especially in deeper models. Testing multiple functions, they also found that standard ReLU ran fastest, PReLU reached the highest classification accuracy on ImageNet, and LReLU and RReLU showed competitive performance on CIFAR-10 and CIFAR-100.

Why it matters

Training deep neural networks often fails because bad initial weights make signals explode or vanish. This paper derives an initialization scheme matched to a specific activation function, RReLU, that keeps training stable as networks grow deeper. It also shows that the 'best' activation function depends on your goal: PReLU wins on ImageNet, but ReLU is fastest.

Questions answered