Summary
In this module, we have provided a quick overview of:
- How Deep Learning Libraries work.
- How to implement & train a simple deep learning model from scratch.
How do Deep Learning Libraries work?
Most of the operations performed in deep learning libraries are differentiable. Thus, deep learning libraries can estimate automatically the gradient of the loss function \(\dfrac{\partial L}{\partial \mathbf{w}}\), where \(\mathbf{w}\) represent the parameters of the neural network. To do so, they use an algorithm called backpropagation.
After having estimated the gradient \(\dfrac{\partial L}{\partial \mathbf{w}}\), they use a gradient-based optimiser (such as stochastic gradient descent) to update the parameters of the neural network model \(\mathbf{w}\).
How do we implement & train a simple deep learning model from scratch?
- Define precisely what type of input do you want to consider. Does it require some data preprocessing?
- Consider what is the output of the neural network, and the loss function used to training the model.
- Decide on a complete internal structure of the neural network model.
- Decide on an optimiser to train it.