Training
Gradient Descent
Optimization algorithm to minimize loss functions
What is Gradient Descent?
Gradient Descent is an iterative optimization algorithm used to minimize a function by moving in the direction of steepest descent. In machine learning, it's used to minimize the loss function and find optimal model parameters.
Key Points
1
Iteratively updates parameters
2
Moves in direction of negative gradient
3
Has variants: SGD, Adam, RMSprop
4
Learning rate is crucial hyperparameter
Practical Examples
Stochastic Gradient Descent (SGD)
Mini-batch gradient descent
Adam optimizer
Momentum-based methods