What exactly is timestep in an LSTM Model?

Question

I am a newbie to LSTM and RNN as a whole, I've been racking my brain to understand what exactly is a timestep. I would really appreciate an intuitive explanation to this

Solution

Let's start with a great image from Chris Olah's blog (a highly recommended read btw):

In a recurrent neural network you have multiple repetitions of the same cell. The way inference goes is - you take some input (x₀), pass it through the cell to get some output₁(depicted with black arrow to the right on the picture), then pass output₁ as input(possibly adding some more input components - x₁ on the image) to the same cell, producing new output output₂, pass that again as input to the same cell(again with possibly additional input component x₂), producing output₃ and so on.

A time step is a single occurrence of the cell - e.g. on the first time step you produce output₁, h₀, on the second time step you produce output₂ and so on.

Comments

[Steven Wang]

Thanks alot for the comment! I have a follow up question. If my code says timestep=40, does that mean that I start from the 40th timestep or I go upto the 40th timestep from the 1st timestep

[Ivaylo Strandjev]

it is impossible to answer this without additional context. It seems this is a variable in your code but could refer to either I guess

[Steven Wang]

The time timesteps were sliced between 10-40 and the 40th timestep was used. I have attached a link to the code for the LSTM model used. imgur.com/gallery/pejFKnY

[Ivaylo Strandjev]

I believe there are two different timestep variables in action here. When you train a recurrent model you typically unroll it for a fixed number of steps and backpropagate, I believe this is the timestep in build_model. The others to my understanding and from this limited code snippet look like referring to the current time step

[Steven Wang]

Thanks alot Ivaylo! what you've assumed checks out. really appreciate the help figuring this out!