Why do Ipython cells stop executing?

Question

I'm sure this is a very newb question, so I apologize in advance. I'm trying to use ipython notebook for a group project. The program we are building is fairly large and pulls in a large number of external datasets. Much of the time, Ipython seems to stop working. I'll try to run a cell or multiple cells and nothing will happen (except a little asterisk * will appear in the brackets [] to the left of the cell). Even if I try to just add a new cell and execute 2+2, nothing will happen. What is going on here? How do I fix this? Thanks!

Solution

The asterisk next to a cell [*] indicates that the cell is currently executing. While IPython provides each notebook with it's own kernel, there is only one kernel per notebook. When that kernel is busy executing code (either a cell, or a series of cells) it cannot accept or run any further code until what it is currently doing is finished. New executions sit in a queue, until the kernel is ready.

If you wait long enough after trying to execute 2+2 you should find that it will eventually execute (assuming your main code ever exits).

The solution to this depends on your code, and how long you're willing to wait to get the results. As a general rule try the following:

• Use a smaller data set to test the algorithm, then scale up gradually noting the increase in time. Is it going to be feasible with your full dataset?
• Is your algorithm reading/writing to the disk? Can you avoid it, or pre-load/post-save state?
• Is it possible to split your data into batches?
• If your algorithm is batchable, can you parallelize it to make best use of your CPU?

You can interrupt the kernel, however this will not work if the execution is currently out of the kernel's hands e.g. in external C modules (a lot of numpy for example). In these cases you may need to restart completely.

Comments

[Alex Riley]

Are you certain that the IPython kernel hasn't died?

[Matt]

The "Pulling large number of dataset" probably mean that IPython is just working. "*" means I'm currently thinking and will come back to you. Try smaller dataset first.