Calculus for Machine Learning and Data Science (6)

Gradients and Gradient Descent

Gradients

Introduction to Tangent planes

01

Just like we had a tangent line in a 1-dimensional line by computing derivatives, we have a tangent plane, which is a slope or a derivative of the functions with multiple variables.

To find the tangent plane we use gradient descent to speed up the optimization.

A tangent is a straight line or plane that touches a curve or curved surface at a point, but if extended does not cross it at that point.

To get the tangent plane, we cut the planes/space to find the corresponding tangent lines to get the tangent plane that contains all tangent lines.

To cut the planes/space, we use partial derivatives.

Partial derivatives - Part 1

012

A partial derivative is a way to find a derivative of one variable and treat all other variables as a constant.

Partial derivatives - Part 2

What is it?

$6xy^3$

Correct! Keeping y constant and performing the calculation you got the result!

What is it?

$9x^2y^2$

Correct. Keeping the x constant and differentiating with respect to y gives you the result!

Gradients

A gradient is simply a collection of partial derivatives in a vector.

Let $f(x,y) = x^2+y^2$, calculate the gradient $f, \nabla f$, at (2, 3)

$\left[\begin{array}{c}4\\6\end{array}\right]$

Gradients and maxima/minima

To get the maximum/minimum point, we solve for the slope to be 0.

Optimization with gradients: An example

Sauna example to find the coldest spot

Quiz 1: Using the expanded form of the function $f(x, y) = 85 - {1\over 90}x^2(x-6)y^2(y-6)$ written as $= 85-{1\over 90}x^3y^3 + {1\over 15}x^3y^2+{1\over 15}x^2y^3 - {2\over 5}x^2y^2$, find ${\partial f\over \partial x}$

$-{1\over 30}x^2y^3 + {1\over 5}x^2y^2+{2\over 15}xy^3 - {4\over 5}xy^2$

Correct! This is the same expression as the lecturer is going to show, but it will be shown in a factored form, so it is easier to find the zeros!

Quiz 2: Using the expanded form of the function $f(x, y) = 85 - {1\over 90}x^2(x-6)y^2(y-6)$ written as $= 85-{1\over 90}x^3y^3 + {1\over 15}x^3y^2+{1\over 15}x^2y^3 - {2\over 5}x^2y^2$, find ${\partial f\over \partial y}$

$-{1\over 30}x^3y^2 + {2\over 15}x^3y+{1\over 5}x^2y^2 - {4\over 5}x^2y$

Correct! This is the same expression as the lecturer is going to show, but it will be shown in a factored form, so it is easier to find the zeros!

01

We can solve for x and y and then apply them to see the minimum point.

Optimization using gradients - Analytical method

01

To minimize the sum of squares cost, we need to solve for the derivatives.

Compute ${\partial E\over \partial m}$

$28m + 12b - 42$

Compute ${\partial E\over \partial b}$

$6b+12m-20$

012

Finding m and b

We use gradient descent to solve for partial derivatives quickly.

 

All the information provided is based on the Calculus for Machine Learning and Data Science |  Coursera from DeepLearning.AI