SyncBatchNorm

class SyncBatchNorm(in_channels=0, num_devices=None, momentum=0.9, epsilon=1e-05, center=True, scale=True, use_global_stats=False, beta_initializer='zeros', gamma_initializer='ones', running_mean_initializer='zeros', running_variance_initializer='ones', **kwargs)

Cross-GPU Synchronized Batch normalization (SyncBN)

Standard BN [1] implementation only normalize the data within each device. SyncBN normalizes the input within the whole mini-batch. We follow the implementation described in the paper [2].

Note: Current implementation of SyncBN does not support FP16 training. For FP16 inference, use standard nn.BatchNorm instead of SyncBN.

Parameters:

in_channels : int, default 0

Number of channels (feature maps) in input data. If not specified, initialization will be deferred to the first time forward is called and in_channels will be inferred from the shape of input data.

num_devices : int, default number of visible GPUs

momentum: float, default 0.9

Momentum for the moving average.

epsilon: float, default 1e-5

Small float added to variance to avoid dividing by zero.

center: bool, default True

If True, add offset of beta to normalized tensor. If False, beta is ignored.

scale: bool, default True

If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. nn.relu), this can be disabled since the scaling will be done by the next layer.

use_global_stats: bool, default False

If True, use global moving statistics instead of local batch-norm. This will force change batch-norm into a scale shift operator. If False, use local batch-norm.

beta_initializer: str or `Initializer`, default ‘zeros’

Initializer for the beta weight.

gamma_initializer: str or `Initializer`, default ‘ones’

Initializer for the gamma weight.

running_mean_initializer: str or `Initializer`, default ‘zeros’

Initializer for the running mean.

running_variance_initializer: str or `Initializer`, default ‘ones’

Initializer for the running variance.

Inputs:

• data: input tensor with arbitrary shape.

Outputs:

• out: output tensor with the same shape as data.

Reference:

[1]	Ioffe, Sergey, and Christian Szegedy. “Batch normalization: Accelerating deep network training by reducing internal covariate shift.” ICML 2015

[2]	Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. “Context Encoding for Semantic Segmentation.” CVPR 2018