Class AdamOptimizer

java.lang.Object
deepnetts.net.train.opt.AdamOptimizer
All Implemented Interfaces:
Optimizer, TrainingListener, Serializable, EventListener
public final class AdamOptimizer extends Object implements Serializable, Optimizer, TrainingListener
Implementation of Adam optimizer which uses an estimation of a gradient statistic (mean and variance) to adjust learning rate for each weight. Adaptive Moment Estimation (Adam) is a method that computes adaptive learning rates for each parameter/weight. Recommended(default) settings: learningRate=0.001 epsilon=1e-8 small constant to prevent division by zero / ensure numerical stability beta1=0.9 exponential decay rate for the first moment estimate(mean) beta2=0.999 exponential decay rate for the second moment estimate(mean) biasCorrection1=0.9 correct exp avg in initial iterations when it is 0 biasCorrection2=0.999 correct exp avg in initial iterations when it is 0 FORMULAS: g(t) is a gradient at step t m(t)=beta1*m(t-1) + (1-beta1)*g(t) estimate of a gradient mean using exponential moving average of gradients v(t)=beta2*v(t-1) + (1-beta2)*g(t)^2 estimate of a gradient variance using exponential moving average of grad squares -learningRate * m(t) deltaWeight(t) = __________________________ sqrt(v(t))+epsilon Paper: ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION https://arxiv.org/abs/1412.6980
See Also:

  • Constructor Details

  • Method Details