How to implement dropout in Pytorch, and where to apply it

Question

I am quite unsure whether this is correct. It is really sad I can't find many good examples on how to parametrize a NN.

What do you think of this way of dropping out in those two classes. First I'm writing the original class :

class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

and then here, I found two different ways to write things, which I don't know how to distinguish. The first one uses :

self.drop_layer = nn.Dropout(p=p) 

whereas the second :

self.dropout = nn.Dropout(p) 

and here is my result :

class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.drop_layer = nn.Dropout(p=p)

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out


 class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.dropout = nn.Dropout(p) 

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.fc3(out)
      return out

could this work, if not how to improve that, and is it giving me the result I m expecting, meaning creating a neural network where I can dropout some neurons. Important detail, I only want to dropout the second layer of neural network, no touch to the rest!

Solution

The two examples you provided are exactly the same. self.drop_layer = nn.Dropout(p=p) and self.dropout = nn.Dropout(p) only differ because the authors assigned the layers to different variable names. The dropout layer is typically defined in the .__init__() method, and called in .forward(). Like this:

 class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, num_classes, p = dropout):
      super(NeuralNet, self).__init__()
      self.fc1 = nn.Linear(input_size, hidden_size)
      self.fc2 = nn.Linear(hidden_size, hidden_size)
      self.fc3 = nn.Linear(hidden_size, num_classes)
      self.dropout = nn.Dropout(p) 

  def forward(self, x):
      out = F.relu(self.fc1(x))
      out = F.relu(self.fc2(out))
      out = self.dropout(self.fc3(out))
      return out

You can do the test:

import torch
import torch.nn  as nn

m = nn.Dropout(p=0.5)
input = torch.randn(20, 16)
print(torch.sum(torch.nonzero(input)))
print(torch.sum(torch.nonzero(m(input))))

tensor(5440) # sum of nonzero values
tensor(2656) # sum on nonzero values after dropout

Let's visualize it:

import torch
import torch.nn as nn
input = torch.randn(5, 5)
print(input)

tensor([[ 1.1404,  0.2102, -0.1237,  0.4240,  0.0174],
        [-2.0872,  1.2790,  0.7804, -0.0962, -0.9730],
        [ 0.4788, -1.3408,  0.0483,  2.4125, -1.2463],
        [ 1.5761,  0.3592,  0.2302,  1.3980,  0.0154],
        [-0.4308,  0.2484,  0.8584,  0.1689, -1.3607]])

Now, let's apply the dropout:

m = nn.Dropout(p=0.5)
output = m(input)
print(output)

tensor([[ 0.0000,  0.0000, -0.0000,  0.8481,  0.0000],
        [-0.0000,  0.0000,  1.5608, -0.0000, -1.9459],
        [ 0.0000, -0.0000,  0.0000,  0.0000, -0.0000],
        [ 0.0000,  0.7184,  0.4604,  2.7959,  0.0308],
        [-0.0000,  0.0000,  0.0000,  0.0000, -0.0000]])

Approximately half the neurons have been turned to zero, because we had probability p=0.5 that a neuron is set to zero!