PyTorch - Element-wise multiplication between a variable and a tensor?

Question

As of PyTorch 0.4 this question is no longer valid. In 0.4 Tensors and Variables were merged.

How can I perform element-wise multiplication with a variable and a tensor in PyTorch? With two tensors works fine. With a variable and a scalar works fine. But when attempting to perform element-wise multiplication with a variable and tensor I get:

XXXXXXXXXXX in mul
    assert not torch.is_tensor(other)
AssertionError

For example, when running the following:

import torch

x_tensor = torch.Tensor([[1, 2], [3, 4]])
y_tensor = torch.Tensor([[5, 6], [7, 8]])

x_variable = torch.autograd.Variable(x_tensor)

print(x_tensor * y_tensor)
print(x_variable * 2)
print(x_variable * y_tensor)

I would expect the first and last print statements to show similar results. The first two multiplications work as expected, with the error coming up in the third. I have attempted the aliases of * in PyTorch (i.e. x_variable.mul(y_tensor), torch.mul(y_tensor, x_variable), etc.).

It seems that element-wise multiplication between a tensor and a variable is not supported given the error and the code which produces it. Is this correct? Or is there something I'm missing? Thank you!

Solution

Yes, you are correct. Elementwise multiplication (like most other operations) is only supported for Tensor * Tensor or Variable * Variable, but not for Tensor * Variable.

To perform your multiplication above, wrap your Tensor as a Variable which doesn't require gradients. The additional overhead is insignificant.

y_variable = torch.autograd.Variable(y_tensor, requires_grad=False)
x_variable * y_variable # returns Variable

But obviously, only use Variables though, if you actually require automatic differentiation through a graph. Else you can just perform the operation on the Tensors directly as you did in your question.

Comments

[golmschenk]

Do you happen to know the reasoning behind this implementation choice in PyTorch? Specifically, why isn't the tensor just treated like the scalar? In both cases, it's known that gradients aren't needed on them, so why isn't the multiplication allowed directly? In any case, thanks for the answer!

[mbpaulus]

Most operations exist twice in PyTorch, they are implemented once for Tensors and once for Variables. The operations on Variables require a backward method. Hence, non-differentiable operations are not supported by Variables, but you can apply them to tensors. I think they wanted to keep these two threads separate for this reason. And scalar operations on Variables are just implemented for convenience.