AdaDelta

class AdaDelta(rho=0.9, epsilon=1e-05, **kwargs)

The AdaDelta optimizer.

This class implements AdaDelta, an optimizer described in ADADELTA: An adaptive learning rate method, available at https://arxiv.org/abs/1212.5701.

This optimizer updates each weight by:

grad = clip(grad * rescale_grad + wd * weight, clip_gradient)
acc_grad = rho * acc_grad + (1. - rho) * grad * grad
delta = sqrt(acc_delta + epsilon) / sqrt(acc_grad + epsilon) * grad
acc_delta = rho * acc_delta + (1. - rho) * delta * delta
weight -= (delta + wd * weight)

This optimizer accepts the following parameters in addition to those accepted by Optimizer.

Parameters:

rho: float

Decay rate for both squared gradients and delta.

epsilon : float

Small value to avoid division by 0.