PyTorch ValueError: Target and input must have the same number of elements

Question

I am kinda new to PyTorch, but I am trying to understand how the sizes of target and input work in torch.nn.BCELoss() when computing loss function.

import torch
import torch.nn as nn
from torch.autograd import Variable

time_steps = 15
batch_size = 3
embeddings_size = 100
num_classes = 2

model = nn.LSTM(embeddings_size, num_classes)
input_seq = Variable(torch.randn(time_steps, batch_size, embeddings_size))
lstm_out, _ = model(input_seq)
last_out = lstm_out[-1]
print(last_out)

loss = nn.BCELoss()
target = Variable(torch.LongTensor(batch_size).random_(0, num_classes))
print(target)
err = loss(last_out.long(), target)
err.backward()

I received the following error:

Warning (from warnings module):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch/nn/functional.py", line 767
"Please ensure they have the same size.".format(target.size(), input.size()))

File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch/nn/functional.py", line 770, in binary_cross_entropy
"!= input nelement ({})".format(target.nelement(), input.nelement()))
ValueError: Target and input must have the same number of elements. target nelement (3) != input nelement (6)

This error definitely comes from the different sizes of last_out (size 3x2) and target (size 3). So my question is how can I convert last_out into something like target (of size 3 and containing just 0s and 1s) to compute loss function?

Solution

The idea of nn.BCELoss() is to implement the following formula:

Both o and tare tensors of arbitrary (but same!) size and i simply indexes each element of the two tensor to compute the sum above.

Typically, nn.BCELoss() is used in a classification setting: o and i will be matrices of dimensions N x D. N will be the number of observations in your dataset or minibatch. D will be 1 if you are only trying to classify a single property, and larger than 1 if you are trying to classify multiple properties. t, the target matrix, will only hold 0 and 1, as for every property, there are only two classes (that's where the binary in binary cross entropy loss comes from). o will hold the probability with which you assign every property of every observation to class 1.

Now in your setting above, it is not clear how many classes you are considering and how many properties there are for you. If there is only one property as suggested by the shape of your target, you should only output a single quantity, namely the probability of being in class 1 from your model. If there are two properties, your targets are incomplete! If there are multiple classes, you should work with torch.nn.CrossEntropyLossinstead of torch.nn.BCELoss().

As an aside, often it is desirable for the sake of numerical stability to use torch.nn.BCEWithLogitsLossinstead of torch.nn.BCELoss() following nn.Sigmoid() on some outputs.

Comments

[bbasaran]

Hello mbpaulus! I have seen your comment while looking around for my target-input size mismatch problem. I am trying to use BCELoss, but getting "Target size (torch.Size([64])) must be the same as input size (torch.Size([64, 256]))" error. How can I match these sizes? Thanks!