Step 2: Create a neural network

In this step, you learn how to use NP on MXNet to create neural networks in Gluon. In addition to the np package that you learned about in the previous step Step 1: Manipulate data with NP on MXNet, you also import the neural network nn package from gluon.

Use the following commands to import the packages required for this step.

from mxnet import np, npx
from mxnet.gluon import nn
npx.set_np()  # Change MXNet to the numpy-like mode.

Create your neural network’s first layer

Use the following code example to start with a dense layer with two output units.

layer = nn.Dense(2)
layer

Dense(-1 -> 2, linear)

Initialize its weights with the default initialization method, which draws random values uniformly from \([-0.7, 0.7]\). You can see this in the following example.

layer.initialize()

Do a forward pass with random data, shown in the following example. We create a \((3,4)\) shape random input x and feed into the layer to compute the output.

x = np.random.uniform(-1,1,(3,4))
layer(x)

array([[-0.02524132, -0.00874885],
       [-0.06026538, -0.01308061],
       [ 0.02468396, -0.02181557]])

As can be seen, the layer’s input limit of two produced a \((3,2)\) shape output from our \((3,4)\) input. You didn’t specify the input size of layer before, though you can specify it with the argument in_units=4 here. The system automatically infers it during the first time you feed in data, create, and initialize the weights. You can access the weight after the first forward pass, as shown in this example.

# layer.weight.data() # FIXME

Chain layers into a neural network

Consider a simple case where a neural network is a chain of layers. During the forward pass, you run layers sequentially one-by-one. Use the following code to implement a famous network called LeNet through nn.Sequential.

net = nn.Sequential()
# Add a sequence of layers.
net.add(# Similar to Dense, it is not necessary to specify the input channels
        # by the argument `in_channels`, which will be  automatically inferred
        # in the first forward pass. Also, we apply a relu activation on the
        # output. In addition, we can use a tuple to specify a  non-square
        # kernel size, such as `kernel_size=(2,4)`
        nn.Conv2D(channels=6, kernel_size=5, activation='relu'),
        # One can also use a tuple to specify non-symmetric pool and stride sizes
        nn.MaxPool2D(pool_size=2, strides=2),
        nn.Conv2D(channels=16, kernel_size=3, activation='relu'),
        nn.MaxPool2D(pool_size=2, strides=2),
        # The dense layer will automatically reshape the 4-D output of last
        # max pooling layer into the 2-D shape: (x.shape[0], x.size/x.shape[0])
        nn.Dense(120, activation="relu"),
        nn.Dense(84, activation="relu"),
        nn.Dense(10))
net

Sequential(
  (0): Conv2D(-1 -> 6, kernel_size=(5, 5), stride=(1, 1), Activation(relu))
  (1): MaxPool2D(size=(2, 2), stride=(2, 2), padding=(0, 0), ceil_mode=False, global_pool=False, pool_type=max, layout=NCHW)
  (2): Conv2D(-1 -> 16, kernel_size=(3, 3), stride=(1, 1), Activation(relu))
  (3): MaxPool2D(size=(2, 2), stride=(2, 2), padding=(0, 0), ceil_mode=False, global_pool=False, pool_type=max, layout=NCHW)
  (4): Dense(-1 -> 120, Activation(relu))
  (5): Dense(-1 -> 84, Activation(relu))
  (6): Dense(-1 -> 10, linear)
)

Using nn.Sequential is similar to nn.Dense. In fact, both of them are subclasses of nn.Block. Use the following code to initialize the weights and run the forward pass.

net.initialize()
# Input shape is (batch_size, color_channels, height, width)
x = np.random.uniform(size=(4,1,28,28))
y = net(x)
y.shape

(4, 10)

You can use [] to index a particular layer. For example, the following accesses the first layer’s weight and sixth layer’s bias.

(net[0].weight.data().shape, net[5].bias.data().shape)

((6, 1, 5, 5), (84,))

Create a neural network flexibly

In nn.Sequential, MXNet will automatically construct the forward function that sequentially executes added layers. Here is another way to construct a network with a flexible forward function.

Create a subclass of nn.Block and implement two methods by using the following code.

  • __init__ create the layers
  • forward define the forward function.
class MixMLP(nn.Block):
    def __init__(self, **kwargs):
        # Run `nn.Block`'s init method
        super(MixMLP, self).__init__(**kwargs)
        self.blk = nn.Sequential()
        self.blk.add(nn.Dense(3, activation='relu'),
                     nn.Dense(4, activation='relu'))
        self.dense = nn.Dense(5)
    def forward(self, x):
        y = npx.relu(self.blk(x))
        print(y)
        return self.dense(y)

net = MixMLP()
net

MixMLP(
  (blk): Sequential(
    (0): Dense(-1 -> 3, Activation(relu))
    (1): Dense(-1 -> 4, Activation(relu))
  )
  (dense): Dense(-1 -> 5, linear)
)

In the sequential chaining approach, you can only add instances with nn.Block as the base class and then run them in a forward pass. In this example, you used print to get the intermediate results and nd.relu to apply relu activation. This approach provides a more flexible way to define the forward function.

The following code example uses net in a similar manner as earlier.

net.initialize()
x = np.random.uniform(size=(2,2))
net(x)

[[0.         0.         0.00014992 0.        ]
 [0.         0.         0.00025604 0.        ]]

array([[ 2.0575192e-06, -6.8181575e-06, -7.2768480e-06, -4.4091166e-06,
         8.4296198e-06],
       [ 3.5140570e-06, -1.1644797e-05, -1.2428200e-05, -7.5303733e-06,
         1.4397029e-05]])

Finally, access a particular layer’s weight with this code.

net.blk[1].weight.data()

array([[-0.0204011 ,  0.00283971, -0.03490724],
       [-0.05335442,  0.05835057, -0.03953977],
       [ 0.00657327,  0.04791138,  0.01269311],
       [-0.02781162,  0.04208799, -0.04010999]])

Next steps

After you create a neural network, learn how to automatically compute the gradients in Step 3: Automatic differentiation with autograd.