Python  - PyTorch - nn.Module                                     Home : www.sharetechnote.com

 

 

 

 

PyTorch - nn.Module

 

In most case, we implement a Neural Network in the form of a class.  What I am going to show you in this page is how to implement a neural network in Python class. I will be trying to keep the structure as simple as possible to make it easy to understand.  The network that I am going to create in this page is basically same as the one that I created without using Class in nn.Linear page.

 

 

You can implement a network in Class from scratch, but in most case we create a class that inherits from Pytorch class nn.Module because we can use many of the high level API that is provided by Pytorch without implementing them by ourselves. Every class that is created in this page inheritis nn.Module class. If you are not faimilar with the basic structure of Python Class or how it inheritis other class, check this basic Class tutorial.

 

 

 

Single Perceptron and Foward only

 

This section shows a example of the simplest neural net class that can be created from nn.Module. The minium requirement for nn.Module based class is to override __init__() method and forward() method. What this class does is exactly same as this example without using the nn.Module.

    import torch

    from torch import nn

     

In this class, I define a simple linear network with two inputs and one output and use Sigmoid() function as the activation function of the network.

    class Network(nn.Module):

        def __init__(self):

            super().__init__()

     

            self.linear = nn.Linear(2,1)     # self.linear is just a local variable. you can define any name

            self.activation = nn.Sigmoid() # self.activation is just a local variable. you can define any name

     

        def forward(self,x):                  # x in the forward() method is the input vector.

            o = self.linear(x)                  # the input vector x is transferred to the network

            o = self.activation(o)           # the result of the network output is transferred to the activation function

     

            return o                             # the outpuf of the activation function is returned as the result   

     

Now let's test if the network is working as intended.

    net = Network()                         # create an instance of the Network() that we defined

    print(net)                                 # print out the Network() class information

    ==> Network(

              (linear): Linear(in_features=2, out_features=1, bias=True)

              (activation): Sigmoid()

           )

     

     

    x = torch.tensor([[1.0,1.0]])        # define a test input vector

    print(x)

     ==> tensor([[1., 1.]])

     

     

    o = net.forward(x)                     # put the input vector into the forward() method of the class

    print('o = net.forward(x) :',o)       # print the output of the forward() method.

    ==> o = net.forward(x) : tensor([[0.3510]], grad_fn=<SigmoidBackward>)

 

 

 

Single Perceptron, Foward, Loss and Backpropagation

 

What we went through in previous section is to define a nn.Module class and follow through the forward path. In this section, I will add the second step meaning 'backward() and back propagation'.

 

    import torch

    from torch import nn

Class definition is the same as we looked at in previous section. So no further explanation.

    class Network(nn.Module):

        def __init__(self):

            super().__init__()

     

            self.linear = nn.Linear(2,1)

            self.activation = nn.Sigmoid()

     

        def forward(self,x):

            o = self.linear(x)

            o = self.activation(o)

     

            return o

     

Creating an instance of Network Class. Same as shown in previous section. No further explanation.

    net = Network()

    print(net)

    ==> Network(

                     (linear): Linear(in_features=2, out_features=1, bias=True)

                     (activation): Sigmoid()

                     )

     

Creating an input vector. Same as shown in previous section. No further explanation.

     

    x = torch.tensor([[1.0,1.0]])

    print(x)

    ==> tensor([[1., 1.]])

     

Following through the forward path. Same as shown in previous section. No further explanation.

    o = net.forward(x)

    print('o = net.forward(x) :',o)

    ==> o = net.forward(x) : tensor([[0.6695]], grad_fn=<SigmoidBackward>)

     

Now let's define a loss function and optimizer.   

    lossFunc = torch.nn.MSELoss()

    optimizer = torch.optim.SGD(net.parameters(), lr = 0.01)

     

Define the desired output for the test input vector created above.

    t = torch.tensor([[1.0]])

    print('t = ',t)

    ==> t =  tensor([[1.]])

     

Define a loss function using the output value 'o' and the desired output value 't' as shown below.

    loss = lossFunc(o,t)

    print('loss = ',loss)

    ==> loss =  tensor(0.1092, grad_fn=<MseLossBackward>)

     

Initialize the optimizer. This is to reset the gradient() value to zero to prevent any existing value from affecting the newly calculated value. Usually you need to do this for every epoch of the training process.

    optimizer.zero_grad()

Now calculate the gradient value for all the network parameters (i.e, weight and bias of each neuron in the network). This is done just by calling backward() function. The backward() just calculate the gradient value. It does not update the weight/bias value themselves yet.

    loss.backward()

Then, let's update the weight/bias using the gradient values calculated in previous step. This is done by calling step() function of the optimizer.

    optimizer.step()

Following is just to check if weight and bias has been updated. I put the same input x that I used before and then I get the different result. So I can guess weight/bias has been changed, but don't know exactly which values they are changed to. In next example, I will show you one way to show you exactly how the weight and bias has changed.

    net.printInfo()

    ==>  Weight of linear :

                  Parameter containing:

                         tensor([[-0.7002, -0.0495]], requires_grad=True)

            Bias of linear :

                  Parameter containing:

                         tensor([-0.6640], requires_grad=True)

            Grad of weight :

                         tensor([[-0.2524, -0.2524]])

            Grad of bias :

                        tensor([-0.2524])

 

 

 

Some Basics on the Class

 

    import torch

    from torch import nn

     

    class Network(nn.Module):

        def __init__(self):

            super().__init__()

            print("__init__() is called");

     

            self.linear = nn.Linear(2,1)

            self.activation = nn.Sigmoid()

     

        def forward(self,x):

            print("forward() is called");

            o = self.linear(x)

            o = self.activation(o)

     

            return o

     

        def printInfo(self):

            print("Weight of linear : \n     ", self.linear.weight)

            print("Bias of linear : \n     ", self.linear.bias)

            print("Grad of weight : \n     ", self.linear.weight.grad)

            print("Grad of bias : \n     ", self.linear.bias.grad)

 

 

    net = Network()

    ==> __init__() is called

 

    x = torch.tensor([[1.0,1.0]])

    o = net.forward(x)

    ==> forward() is called

 

    print('o = net.forward(x) :',o)

    ==> o = net.forward(x) : tensor([[0.3122]], grad_fn=<SigmoidBackward>)

 

    o = net(x)

    ==> forward() is called

 

    print('o = net(x) :',o)

    o = net(x) : tensor([[0.3122]], grad_fn=<SigmoidBackward>)

 

 

 

 

 

Reference :

 

[1] PyTorch 101: Building Neural Networks

[2] Linear Regression using PyTorch

[3] Linear Regression with PyTorch