What does layout = torch.strided mean?

Question

As I was going through pytorch documentation I came across a term layout = torch.strided in many of the functions. Can anyone help me in understanding where is it used and how. The description says it's the the desired layout of returned Tensor. What does layout mean and how many types of layout are there ?

torch.rand(*sizes, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)

Solution 1

strides is number of steps (or jumps) that is needed to go from one element to next element, in a given dimension. In computer memory, the data is stored linearly in a contiguous block of memory. What we view is just a (re)presentation.

Let's take an example tensor for understanding this:

# a 2D tensor
In [62]: tensor = torch.arange(1, 16).reshape(3, 5) 

In [63]: tensor  
Out[63]: 
tensor([[ 1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10],
        [11, 12, 13, 14, 15]])

With this tensor in place, the strides are:

# get the strides
In [64]: tensor.stride() 
Out[64]: (5, 1)

What this resultant tuple (5, 1) says is:

• to traverse along the 0^th dimension/axis (Y-axis), let's say we want to jump from 1 to 6, we should take 5 steps (or jumps)
• to traverse along the 1^st dimension/axis (X-axis), let's say we want to jump from 7 to 8, we should take 1 step (or jump)

The order (or index) of 5 & 1 in the tuple represents the dimension/axis. You can also pass the dimension, for which you want the stride, as an argument:

# get stride for axis 0
In [65]: tensor.stride(0) 
Out[65]: 5

# get stride for axis 1
In [66]: tensor.stride(1) 
Out[66]: 1

---

With that understanding, we might have to ask why is this extra parameter needed when we create the tensors? The answer to that is for efficiency reasons. (How can we store/read/access the elements in the (sparse) tensor most efficiently?).

With sparse tensors (a tensor where most of the elements are just zeroes), so we don't want to store these values. we only store the non-zero values and their indices. With a desired shape, the rest of the values can then be filled with zeroes, yielding the desired sparse tensor.

---

For further reading on this, the following articles might be of help:

• numpy.ndarray.strides
• torch.layout
• torch.sparse

---

P.S: I guess there's a typo in the torch.layout documentation which says

Strides are a list of integers ...

The composite data type returned by tensor.stride() is a tuple, not a list.

Solution 2

For quick understanding, layout=torch.strided corresponds to dense tensors while layout=torch.sparse_coo corresponds to sparse tensors.

From another perspective, we can understand it together with torch.tensor.view. A tensor can be viewed indicates it is contiguous. If we change the view of a tensor, the strides will change accordingly, but the data will keep the same. More specifically, view returns a new tensor with the same data but different shape, and strides is compatible with the view to indicate how to access the data in the memory.

For example

In [1]: import torch
In [2]: a = torch.arange(15)

In [3]: a.data_ptr()
Out[3]: 94270437164688

In [4]: a.stride()
Out[4]: (1,)

In [5]: a = a.view(3, 5)

In [6]: a.data_ptr() # share the same data pointer
Out[6]: 94270437164688

In [7]: a.stride() # the stride changes as the view changes
Out[7]: (5, 1)

In addition, the idea of torch.strided is basically the same as strides in numpy. View this question for more detailed understanding. How to understand numpy strides for layman?