Global Information Lookup Global Information

Stochastic gradient descent information


Part of a series on
Machine learning
and data mining
Paradigms
  • Supervised learning
  • Unsupervised learning
  • Online learning
  • Batch learning
  • Meta-learning
  • Semi-supervised learning
  • Self-supervised learning
  • Reinforcement learning
  • Curriculum learning
  • Rule-based learning
  • Quantum machine learning
Problems
  • Classification
  • Generative modeling
  • Regression
  • Clustering
  • Dimensionality reduction
  • Density estimation
  • Anomaly detection
  • Data cleaning
  • AutoML
  • Association rules
  • Semantic analysis
  • Structured prediction
  • Feature engineering
  • Feature learning
  • Learning to rank
  • Grammar induction
  • Ontology learning
  • Multimodal learning
Supervised learning
(classification • regression)
  • Apprenticeship learning
  • Decision trees
  • Ensembles
    • Bagging
    • Boosting
    • Random forest
  • k-NN
  • Linear regression
  • Naive Bayes
  • Artificial neural networks
  • Logistic regression
  • Perceptron
  • Relevance vector machine (RVM)
  • Support vector machine (SVM)
Clustering
  • BIRCH
  • CURE
  • Hierarchical
  • k-means
  • Fuzzy
  • Expectation–maximization (EM)

  • DBSCAN
  • OPTICS
  • Mean shift
Dimensionality reduction
  • Factor analysis
  • CCA
  • ICA
  • LDA
  • NMF
  • PCA
  • PGD
  • t-SNE
  • SDL
Structured prediction
  • Graphical models
    • Bayes net
    • Conditional random field
    • Hidden Markov
Anomaly detection
  • RANSAC
  • k-NN
  • Local outlier factor
  • Isolation forest
Artificial neural network
  • Autoencoder
  • Cognitive computing
  • Deep learning
  • DeepDream
  • Feedforward neural network
  • Kolmogorov–Arnold Network
  • Recurrent neural network
    • LSTM
    • GRU
    • ESN
    • reservoir computing
  • Restricted Boltzmann machine
  • GAN
  • Diffusion model
  • SOM
  • Convolutional neural network
    • U-Net
  • Transformer
    • Vision
  • Mamba
  • Spiking neural network
  • Memtransistor
  • Electrochemical RAM (ECRAM)
Reinforcement learning
  • Q-learning
  • SARSA
  • Temporal difference (TD)
  • Multi-agent
    • Self-play
Learning with humans
  • Active learning
  • Crowdsourcing
  • Human-in-the-loop
  • RLHF
Model diagnostics
  • Coefficient of determination
  • Confusion matrix
  • Learning curve
  • ROC curve
Mathematical foundations
  • Kernel machines
  • Bias–variance tradeoff
  • Computational learning theory
  • Empirical risk minimization
  • Occam learning
  • PAC learning
  • Statistical learning
  • VC theory
Machine-learning venues
  • ECML PKDD
  • NeurIPS
  • ICML
  • ICLR
  • IJCAI
  • ML
  • JMLR
Related articles
  • Glossary of artificial intelligence
  • List of datasets for machine-learning research
    • List of datasets in computer vision and image processing
  • Outline of machine learning
  • v
  • t
  • e

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data). Especially in high-dimensional optimization problems this reduces the very high computational burden, achieving faster iterations in exchange for a lower convergence rate.[1]

The basic idea behind stochastic approximation can be traced back to the Robbins–Monro algorithm of the 1950s. Today, stochastic gradient descent has become an important optimization method in machine learning.[2]

  1. ^ Bottou, Léon; Bousquet, Olivier (2012). "The Tradeoffs of Large Scale Learning". In Sra, Suvrit; Nowozin, Sebastian; Wright, Stephen J. (eds.). Optimization for Machine Learning. Cambridge: MIT Press. pp. 351–368. ISBN 978-0-262-01646-9.
  2. ^ Bottou, Léon (1998). "Online Algorithms and Stochastic Approximations". Online Learning and Neural Networks. Cambridge University Press. ISBN 978-0-521-65263-6.

and 27 Related for: Stochastic gradient descent information

Request time (Page generated in 1.7632 seconds.)

Stochastic gradient descent

Last Update:

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e...

Word Count : 6588

Gradient descent

Last Update:

of gradient descent, stochastic gradient descent, serves as the most basic algorithm used for training most deep networks today. Gradient descent is based...

Word Count : 5280

Online machine learning

Last Update:

out-of-core versions of machine learning algorithms, for example, stochastic gradient descent. When combined with backpropagation, this is currently the de...

Word Count : 4740

Federated learning

Last Update:

of stochastic gradient descent, where gradients are computed on a random subset of the total dataset and then used to make one step of the gradient descent...

Word Count : 5961

Stochastic gradient Langevin dynamics

Last Update:

Stochastic gradient Langevin dynamics (SGLD) is an optimization and sampling technique composed of characteristics from Stochastic gradient descent, a...

Word Count : 1370

Backtracking line search

Last Update:

Gradient descent Stochastic gradient descent Wolfe conditions Absil, P. A.; Mahony, R.; Andrews, B. (2005). "Convergence of the iterates of Descent methods...

Word Count : 4566

Backpropagation

Last Update:

can be derived through dynamic programming. Gradient descent, or variants such as stochastic gradient descent, are commonly used. Strictly the term backpropagation...

Word Count : 7493

Recursive neural network

Last Update:

for all nodes in the tree. Typically, stochastic gradient descent (SGD) is used to train the network. The gradient is computed using backpropagation through...

Word Count : 954

Sparse dictionary learning

Last Update:

being stuck at local minima. One can also apply a widespread stochastic gradient descent method with iterative projection to solve this problem. The idea...

Word Count : 3496

Gradient method

Last Update:

descent Stochastic gradient descent Coordinate descent Frank–Wolfe algorithm Landweber iteration Random coordinate descent Conjugate gradient method Derivation...

Word Count : 109

Multilayer perceptron

Last Update:

the first deep-learning feedforward network, not yet using stochastic gradient descent, was published by Alexey Grigorevich Ivakhnenko and Valentin...

Word Count : 1922

Feedforward neural network

Last Update:

the first deep-learning feedforward network, not yet using stochastic gradient descent, was published by Alexey Grigorevich Ivakhnenko and Valentin...

Word Count : 2320

Stochastic hill climbing

Last Update:

of selection can vary with the steepness of the uphill move." Stochastic gradient descent Russell, S.; Norvig, P. (2010). Artificial Intelligence: A Modern...

Word Count : 69

Learning rate

Last Update:

Hyperparameter (machine learning) Hyperparameter optimization Stochastic gradient descent Variable metric methods Overfitting Backpropagation AutoML Model...

Word Count : 1108

Simultaneous perturbation stochastic approximation

Last Update:

See the brief discussion in Stochastic gradient descent. Bhatnagar, S., Prasad, H. L., and Prashanth, L. A. (2013), Stochastic Recursive Algorithms for Optimization:...

Word Count : 1555

Stochastic optimization

Last Update:

Methods of this class include: stochastic approximation (SA), by Robbins and Monro (1951) stochastic gradient descent finite-difference SA by Kiefer and...

Word Count : 1083

Slope

Last Update:

gradient method, generalizes the conjugate gradient method to nonlinear optimization Stochastic gradient descent, iterative method for optimizing a differentiable...

Word Count : 2619

Least mean squares filter

Last Update:

(difference between the desired and the actual signal). It is a stochastic gradient descent method in that the filter is only adapted based on the error...

Word Count : 3045

Peter Richtarik

Last Update:

learning, known for his work on randomized coordinate descent algorithms, stochastic gradient descent and federated learning. He is currently a Professor...

Word Count : 874

Gradient boosting

Last Update:

introduced the view of boosting algorithms as iterative functional gradient descent algorithms. That is, algorithms that optimize a cost function over...

Word Count : 4188

Stochastic approximation

Last Update:

Stochastic gradient descent Stochastic variance reduction Toulis, Panos; Airoldi, Edoardo (2015). "Scalable estimation strategies based on stochastic...

Word Count : 4147

Coordinate descent

Last Update:

method – Method for finding stationary points of a function Stochastic gradient descent – Optimization algorithm – uses one example at a time, rather...

Word Count : 1649

Elo rating system

Last Update:

{if}}~{\mathsf {B}}~{\textrm {wins}},\end{cases}}} and, using the stochastic gradient descent the log loss is minimized as follows: R A ← R A − η d ℓ d R A...

Word Count : 11569

Feature scaling

Last Update:

only works for x ≠ 0 {\displaystyle x\neq \mathbf {0} } . In stochastic gradient descent, feature scaling can sometimes improve the convergence speed...

Word Count : 882

Huber loss

Last Update:

prediction problems using stochastic gradient descent algorithms. ICML. Friedman, J. H. (2001). "Greedy Function Approximation: A Gradient Boosting Machine"....

Word Count : 1037

Preconditioner

Last Update:

grids. If used in gradient descent methods, random preconditioning can be viewed as an implementation of stochastic gradient descent and can lead to faster...

Word Count : 3511

Variational autoencoder

Last Update:

|x)}}\right]} and so we obtained an unbiased estimator of the gradient, allowing stochastic gradient descent. Since we reparametrized z {\displaystyle z} , we need...

Word Count : 3158
PDF Search Engine © AllGlobal.net