Dev Log D-07 to D-09. My Backprop is/will sucking/suck.

Don't learn here. All code is implementing based on Noob Imagination and poor understanding of Articles.

import mpmath as mpMath 
mpMath.mp.dps = 5
class Neuron:
    def __init__(self,W):
        # Weight is an vector.
        self.W =  W
        self.B = np.random.uniform(-1, 1)
    def forward(self,X):
        self.wX = X.dot(self.W)
        self.f_out = self.wX + self.B
        return self.f_out
    def tanh(self):
        self.out_tanh = mpMath.tanh(self.f_out)
        return self.out_tanh
    def derv(self,w_index,step_v,X):
        u = self.W.copy()
        print(f'Printing old Weight \n{self.W[w_index]}')
        u[w_index] += step_v
        print(f'Printing update weight \n {u[w_index]}')
        _d = (self.f_out - X.dot(u))/step_v
        return _d

neuron = Neuron( np.random.uniform(-1.,1.,size=(784,)).astype(np.float16))

print(neuron.forward(train_data[0]))
print(neuron.derv(2,0.001,train_data[0]))

Slope when step is 0.00001

-707.2305512337132
Printing old Weight 
0.496826171875
Printing update weight 
 0.496826171875
39243.93562717796

As you can see output below when I update weight 0.001 there is 951 slope and If I go too small slope is even large. I don't understand.

-5.366328888592351
Printing old Weight 
-0.47216796875
Printing update weight 
 -0.47119140625
951.3314431337232

Turns out this noob brain of mine forgot to add bais.

 _d = (self.f_out - (X.dot(u)+self.B))/step_v)

so moving on.

Below is the code snippet how I am finding out how my model is predicting.

 def predict(self,X):
        self.inputL_out = self.inputL.forward(X)
        # print(f"Input Layer Output---> \n \t {self.inputL_out}")
        # self.inputL.display_output()
        for l in range(0,len(self.neuronsLayers)):
            if l == 0:
                self.neuronsLayers[l].forward(self.inputL_out)
            else:
                self.neuronsLayers[l].forward(self.neuronsLayers[l-1].f_out)
            print(f'Forward output of layer {l} is :--> {self.neuronsLayers[l].f_out}')
            # self.neuronsLayers[l].display_output1()
            if l == len(self.hidden_layer_dim_arr)-1:
                self.outL_f_out = self.outL.forward(self.neuronsLayers[l].f_out)
                print(f'output of outL \n \t{self.outL_f_out}')
        print("Predicting--->")
        max_index = np.argmax(self.outL_f_out)
        print(f"index of max confidente {max_index} and probability is {self.outL_f_out[max_index]}")

So, My last layer have 10 neurons. each index represent each digit.
like index 2 represent 2.

Now I am stuck on another bug.
I works drev method works for 0.001 but it is 0.0 when I change setup to 0.0001.
I think I should learn basic Numpy and Python. But I will not it is fun doing things this way. Getting stuck on silly bugs.

Blog

Dev Log D-07 to D-09. My Backprop is/will sucking/suck.

Amit

Join Our Newsletter. No Spam, Only the good stuff.

Related