LayerNorm

class LayerNorm(axis=-1, epsilon=1e-05, center=True, scale=True, beta_initializer='zeros', gamma_initializer='ones', in_channels=0, prefix=None, params=None)

Applies layer normalization to the n-dimensional input array. This operator takes an n-dimensional input array and normalizes the input using the given axis:

(17)\[out = \frac{x - mean[data, axis]}{ \sqrt{Var[data, axis] + \epsilon}} * gamma + beta\]

Parameters:

axis : int, default -1

The axis that should be normalized. This is typically the axis of the channels.

epsilon: float, default 1e-5

Small float added to variance to avoid dividing by zero.

center: bool, default True

If True, add offset of beta to normalized tensor. If False, beta is ignored.

scale: bool, default True

If True, multiply by gamma. If False, gamma is not used.

beta_initializer: str or `Initializer`, default ‘zeros’

Initializer for the beta weight.

gamma_initializer: str or `Initializer`, default ‘ones’

Initializer for the gamma weight.

in_channels : int, default 0

Number of channels (feature maps) in input data. If not specified, initialization will be deferred to the first time forward is called and in_channels will be inferred from the shape of input data.

Inputs:

• data: input tensor with arbitrary shape.

Outputs:

• out: output tensor with the same shape as data.

Examples

>>> # Input of shape (2, 5)
>>> x = mx.nd.array([[1, 2, 3, 4, 5], [1, 1, 2, 2, 2]])
>>> # Layer normalization is calculated with the above formula
>>> layer = LayerNorm()
>>> layer.initialize(ctx=mx.cpu(0))
>>> layer(x)
[[-1.41421    -0.707105    0.          0.707105    1.41421   ]
 [-1.2247195  -1.2247195   0.81647956  0.81647956  0.81647956]]
<NDArray 2x5 @cpu(0)>