AdaGrad¶

class AdaGrad(eps=1e-07, **kwargs)¶

AdaGrad optimizer.

This class implements the AdaGrad optimizer described in Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, and available at http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf.

This optimizer updates each weight by:

grad = clip(grad * rescale_grad, clip_gradient)
history += square(grad)
div = grad / sqrt(history + float_stable_eps)
weight += (div + weight * wd) * -lr

This optimizer accepts the following parameters in addition to those accepted by Optimizer.

Parameters:	eps: float, optional Initial value of the history accumulator. Avoids division by 0.