Processing math: 100%

Twitter proof: neural networks and the linear activation function


< change language

In this post we will see why it is not helpful to have two consecutive layers of neurons with linear activation functions in neural networks. With just a bit of maths we can conclude that n consecutive linear layers compute the exact same functions as 1 single layer.

Claim: having two fully connected layers of neurons using linear activation functions is the same as having just one layer with a linear activation function.

We just have to lay down some notation in order for the maths to be doable. Assume the two consecutive layers of linear neurons are preceded by a layer with n0 neurons, whose outpus are o1,o2,⋯,on0.
Let us say that after that layer, there is a layer of n1 neurons with linear activation functions fi(x)=aix+bi; the weight from neuron t of the previous layer to the neuron i of this layer is wt,i.
The second layer of neurons has n2 neurons, with linear activation functions f′i(x)=a′ix+b′i and the weight from neuron i of the previous layer to neuron j of this layer is w′i,j.

Twitter proof: we will compute the outputs produced by the second layer, in terms of the values o1,⋯,on0 of the 0-th layer and conclude we obtain a linear function of those. The output of the j-th neuron will be:
f′j[n1∑i=1w′i,jfi(n0∑t=1wt,iot)]=a′j×[n1∑i=1w′i,jfi(n0∑t=1wt,iot)]+b′j=a′j[n1∑i=1w′i,j(ai×(n0∑t=1wt,iot)+bi)]+b′j=a′j[n1∑i=1w′i,jai(n0∑t=1wt,iot)+n1∑i=1w′i,jbi]+b′j=a′j[n1∑i=1w′i,jai(n0∑t=1wt,iot)]+a′jn1∑i=1w′i,jbi+b′j=a′j[n0∑t=1n1∑i=1w′i,jaiwt,iot]+a′jn1∑i=1w′i,jbi+b′j now defining a″j=a′j, b″j=b′j+a′jn1∑i=1w′i,jbi and w″t,j=n1∑i=1w′i,jaiwt,i we get that the output of the j-th neuron really is f″j(n0∑t=1w″t,jot) where f″j(x)=a″jx+b″j, a linear function. This concludes our proof!

I wrote this post today because of the #100DaysofMLcode initiative I started (yesterday!)!

  - RGS

Twitter proof: neural networks and the linear activation function Twitter proof: neural networks and the linear activation function Reviewed by Unknown on September 18, 2018 Rating: 5

No comments:

Powered by Blogger.