Neural Networks and Deep Learning (10)

Deep Neural Networks

Deep Neural Network

728x90

Forward and Backward Propagation

012

Optional Reading: Feedforward Neural Networks in Depth

• Feedforward Neural Networks in Depth - Deep Learning Specialization / Deep Learning Resources - DeepLearning.AI

Parameters vs Hyperparameters

01

Parameters are the weights and biases, something deep neural networks optimize to get close to the answer.

Parameters are something we initialize randomly and don’t touch.

Hyperparameters are the criteria that we can change to ensure deep neural networks reach global optima.

Some examples of hyperparameters are learning rate, number of iterations, number of hidden layers, and number of hidden units.

Remember that deep learning is a highly iterative process, there is no one set of criteria to reach the global optima every time.

Clarification For: What does this have to do with the brain?

Note that the formulas shown in the next video have a few typos. Here is the correct set of formulas.

$dZ^{[L]}=A^{[L]}−Y$

$dW^{[L]}={1\over m}dZ^{[L]}A^{[L−1]^T}$

$db^{[L]}={1\over m}np.sum(dZ^{[L]},axis=1,keepdims=True)$

$dZ^{[L−1]}=W^{[L]^T}dZ^{[L]}∗g^{′[L−1]}(Z^{[L−1]})$

Note that * denotes element-wise multiplication)

⋮

$dZ^{[1]}=W^{[2]^T}dZ^{[2]}∗g^{′[1]}(Z^{[1]})$

$dW^{[1]}={1\over m}dZ^{[1]}A^{[0]^T}$

Note that $A^{[0]^T}$ is another way to denote the input features, which is also written as $X^T$

$db^{[1]}={1\over m}np.sum(dZ^{[1]},axis=1,keepdims=True)$

What does this have to do with the brain?

We usually use the analogy of conveying deep neural networks to human brains.

Using a logistic regression as an example, we get the outcome of either 1 or 0 depending on the probability of the output layer.

It’s more complex, but neurons in brain cells also send signals (electricity) and depending on the power level, the signals can reach or fade.

 

All the information provided is based on the Deep Learning Specialization | Coursera from DeepLearning.AI