Class AdaGradOptimizer

java.lang.Object
deepnetts.net.train.opt.AdaGradOptimizer
All Implemented Interfaces:
Optimizer, Serializable
public final class AdaGradOptimizer extends Object implements Optimizer, Serializable
Implementation of ADAGRAD Optimizer , which uses sum of squared previous gradients to adjust a global learning rate for each weight. Recommended initial value for the learning rate is 0.01 Original paper1: https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf
See Also:

  • Constructor Details

    • AdaGradOptimizer

      public AdaGradOptimizer(AbstractLayer layer)

  • Method Details

    • calculateDeltaWeight

      public float calculateDeltaWeight(float grad, int... idxs)
      Description copied from interface: Optimizer
      Smoothing term to prevent division by zero if sqr grad sum becomes zero 1e-8 should be also tried https://d2l.ai/chapter_optimization/adagrad.html 1e-6 The value to use is 1e-6, 1e-8, Keras uses 1e-7 for adam
      Specified by:
      calculateDeltaWeight in interface Optimizer

    • calculateDeltaBias

      public float calculateDeltaBias(float grad, int idx)
      Specified by:
      calculateDeltaBias in interface Optimizer

    • setLearningRate

      public void setLearningRate(float learningRate)
      Specified by:
      setLearningRate in interface Optimizer

    • calculateDeltaWeight

      public TensorBase calculateDeltaWeight(TensorBase grad)
      Specified by:
      calculateDeltaWeight in interface Optimizer

    • calculateDeltaBias

      public Tensor1D calculateDeltaBias(Tensor1D grad)
      Specified by:
      calculateDeltaBias in interface Optimizer