Introduction to Deep Learning with PyTorch > 1-dimensional Gradient Descent with torch Tensors | Python Programming (70053 Autumn Term 2022/2023) | Department of Computing

Introduction to Deep Learning with PyTorch

Chapter 4: PyTorch for Automatic Gradient Descent

1-dimensional Gradient Descent with torch Tensors

face Luca Grillotti

Remember the function gradient_descent(...) you implemented before?

def gradient_descent(initial_theta, learning_rate, number_steps):
    """
    Args:
        initial_theta (float): Initial value of theta
        learning_rate (float)
        number_steps (int): number of 1-step gradient descent to perform.

    Returns:
        final_theta (float): Final value of theta after multiple 1-step gradient descents
    """

Let’s make it torch-compliant!

Exercise: Gradient descent with torch Tensors

Write a function gradient_descent_torch that does the same thing as gradient_descent, but with tensor variables instead of floats. We are considering the same function as before $L(\theta) = \theta^2$ . where $\theta$ is a tensor of shape (1,)

def gradient_descent_torch(initial_theta, learning_rate, number_steps):
    """
    Args:
        initial_theta (torch.Tensor): Initial value of theta
        learning_rate (float)
        number_steps (int): number of 1-step gradient descent to perform.

    Returns:
        final_theta (torch.Tensor): Final value of theta after multiple 1-step gradient descents
    """

def get_gradient_tensor(tensor):
    # considering function x^2
    return 2 * tensor

def gradient_descent_torch(initial_theta, learning_rate, number_steps):
    """
    Args:
        initial_theta (torch.Tensor): Initial value of theta
        learning_rate (float)
        number_steps (int): number of 1-step gradient descent to perform.

    Returns:
        final_theta (torch.Tensor): Final value of theta after multiple 1-step gradient descents
    """
    tensor = initial_theta
    for _ in range(number_steps):
        tensor = tensor - learning_rate * get_gradient_tensor(tensor)
        print(tensor)

    return tensor

initial_theta = torch.Tensor([1])
gradient_descent_torch(initial_theta,
                       learning_rate=0.2,
                       number_steps=20)