Not sure if this helps, but: is really the indicator function , as described here. This forms the expression (j == y[i]) in the code. Also, the gradient of the loss with respect to the weights is: where which is the origin of the X[:,i] in the code.
I am assuming you have a 3-layer NN with W1, b1 for is associated with the linear transformation from input layer to hidden layer and W2, b2 is associated with linear transformation from hidden layer to output layer. Z1 and Z2 are the input vector to the hidden layer and output layer. a1 and a2 represents the output of the hidden layer and output layer. a2 is your predicted output. delta3 and delta2 are the … Read more
Try this: Then add a softmax layer in this way:
They’re both correct, but yours is preferred from the point of view of numerical stability. You start with By using the fact that a^(b – c) = (a^b)/(a^c) we have Which is what the other answer says. You could replace max(x) with any variable and it would cancel out.