Step-by-step debugging with IPython
Solution 1
You can use IPython's %pdb
magic. Just call %pdb
in IPython and when an error occurs, you're automatically dropped to ipdb
. While you don't have the stepping immediately, you're in ipdb
afterwards.
This makes debugging individual functions easy, as you can just load a file with %load
and then run a function. You could force an error with an assert
at the right position.
%pdb
is a line magic. Call it as %pdb on
, %pdb 1
, %pdb off
or %pdb 0
. If called without argument it works as a toggle.
Solution 2
What about ipdb.set_trace() ? In your code :
import ipdb; ipdb.set_trace()
update: now in Python 3.7, we can write breakpoint()
. It works the same, but it also obeys to the PYTHONBREAKPOINT
environment variable. This feature comes from this PEP.
This allows for full inspection of your code, and you have access to commands such as c
(continue), n
(execute next line), s
(step into the method at point) and so on.
See the ipdb repo and a list of commands. IPython is now called (edit: part of) Jupyter.
ps: note that an ipdb command takes precedence over python code. So in order to write list(foo)
you'd need print(list(foo))
, or !list(foo)
.
Also, if you like the ipython prompt (its emacs and vim modes, history, completions,…) it's easy to get the same for your project since it's based on the python prompt toolkit.
Solution 3
(Update on May 28, 2016) Using RealGUD in Emacs
For anyone in Emacs, this thread shows how to accomplish everything described in the OP (and more) using
- a new important debugger in Emacs called RealGUD which can operate with any debugger (including
ipdb
). - The Emacs package
isend-mode
.
The combination of these two packages is extremely powerful and allows one to recreate exactly the behavior described in the OP and do even more.
More info on the wiki article of RealGUD for ipdb.
Original answer:
After having tried many different methods for debugging Python, including everything mentioned in this thread, one of my preferred ways of debugging Python with IPython is with embedded shells.
Defining a custom embedded IPython shell:
Add the following on a script to your PYTHONPATH
, so that the method ipsh()
becomes available.
import inspect
# First import the embed function
from IPython.terminal.embed import InteractiveShellEmbed
from IPython.config.loader import Config
# Configure the prompt so that I know I am in a nested (embedded) shell
cfg = Config()
prompt_config = cfg.PromptManager
prompt_config.in_template = 'N.In <\\#>: '
prompt_config.in2_template = ' .\\D.: '
prompt_config.out_template = 'N.Out<\\#>: '
# Messages displayed when I drop into and exit the shell.
banner_msg = ("\n**Nested Interpreter:\n"
"Hit Ctrl-D to exit interpreter and continue program.\n"
"Note that if you use %kill_embedded, you can fully deactivate\n"
"This embedded instance so it will never turn on again")
exit_msg = '**Leaving Nested interpreter'
# Wrap it in a function that gives me more context:
def ipsh():
ipshell = InteractiveShellEmbed(config=cfg, banner1=banner_msg, exit_msg=exit_msg)
frame = inspect.currentframe().f_back
msg = 'Stopped at {0.f_code.co_filename} at line {0.f_lineno}'.format(frame)
# Go back one level!
# This is needed because the call to ipshell is inside the function ipsh()
ipshell(msg,stack_depth=2)
Then, whenever I want to debug something in my code, I place ipsh()
right at the location where I need to do object inspection, etc. For example, say I want to debug my_function
below
Using it:
def my_function(b):
a = b
ipsh() # <- This will embed a full-fledged IPython interpreter
a = 4
and then I invoke my_function(2)
in one of the following ways:
- Either by running a Python program that invokes this function from a Unix shell
- Or by invoking it directly from IPython
Regardless of how I invoke it, the interpreter stops at the line that says ipsh()
. Once you are done, you can do Ctrl-D
and Python will resume execution (with any variable updates that you made). Note that, if you run the code from a regular IPython the IPython shell (case 2 above), the new IPython shell will be nested inside the one from which you invoked it, which is perfectly fine, but it's good to be aware of. Eitherway, once the interpreter stops on the location of ipsh
, I can inspect the value of a
(which be 2
), see what functions and objects are defined, etc.
The problem:
The solution above can be used to have Python stop anywhere you want in your code, and then drop you into a fully-fledged IPython interpreter. Unfortunately it does not let you add or remove breakpoints once you invoke the script, which is highly frustrating. In my opinion, this is the only thing that is preventing IPython from becoming a great debugging tool for Python.
The best you can do for now:
A workaround is to place ipsh()
a priori at the different locations where you want the Python interpreter to launch an IPython shell (i.e. a breakpoint
). You can then "jump" between different pre-defined, hard-coded "breakpoints" with Ctrl-D
, which would exit the current embedded IPython shell and stop again whenever the interpreter hits the next call to ipsh()
.
If you go this route, one way to exit "debugging mode" and ignore all subsequent breakpoints, is to use ipshell.dummy_mode = True
which will make Python ignore any subsequent instantiations of the ipshell
object that we created above.
Solution 4
You can start IPython session from pudb and go back to the debugging session as you like.
BTW, ipdb is using IPython behind the scenes and you can actually use IPython functionality such as TAB completion and magic commands (the one starts with %
). If you are OK with ipdb you can start it from IPython using commands such as %run
and %debug
. ipdb session is actually better than plain IPython one in the sense you can go up and down in the stack trace etc. What is missing in ipdb for "object inspection"?
Also, python.el bundled with Emacs >= 24.3 has nice ipdb support.
Solution 5
Looks like the approach in @gaborous's answer is deprecated.
The new approach seems to be:
from IPython.core import debugger
debug = debugger.Pdb().set_trace
def buggy_method():
debug()
Related videos on Youtube
Amelio Vazquez-Reina
I'm passionate about people, technology and research. Some of my favorite quotes: "Far better an approximate answer to the right question than an exact answer to the wrong question" -- J. Tukey, 1962. "Your title makes you a manager, your people make you a leader" -- Donna Dubinsky, quoted in "Trillion Dollar Coach", 2019.
Updated on July 08, 2022Comments
-
Amelio Vazquez-Reina almost 2 years
From what I have read, there are two ways to debug code in Python:
-
With a traditional debugger such as
pdb
oripdb
. This supports commands such asc
forcontinue
,n
forstep-over
,s
forstep-into
etc.), but you don't have direct access to an IPython shell which can be extremely useful for object inspection. -
Using IPython by embedding an IPython shell in your code. You can do
from IPython import embed
, and then useembed()
in your code. When your program/script hits anembed()
statement, you are dropped into an IPython shell. This allows the full inspection of objects and testing of Python code using all the IPython goodies. However, when usingembed()
you can't step-by-step through the code anymore with handy keyboard shortcuts.
Is there any way to combine the best of both worlds? I.e.
- Be able to step-by-step through your code with handy pdb/ipdb keyboard shortcuts.
- At any such step (e.g. on a given statement), have access to a full-fledged IPython shell.
IPython debugging as in MATLAB:
An example of this type of "enhanced debugging" can be found in MATLAB, where the user always has full access to the MATLAB engine/shell, and she can still step-by-step through her code, define conditional breakpoints, etc. From what I have discussed with other users, this is the debugging feature that people miss the most when moving from MATLAB to IPython.
IPython debugging in Emacs and other editors:
I don't want to make the question too specific, but I work mostly in Emacs, so I wonder if there is any way to bring this functionality into it. Ideally, Emacs (or the editor) would allow the programmer to set breakpoints anywhere on the code and communicate with the interpreter or debugger to have it stop in the location of your choice, and bring to a full IPython interpreter on that location.
-
Dmitry Galchinsky almost 11 yearspdb has
!
command that executes any python command at breakpoint -
user1914692 almost 11 yearsI am seeking a python debugger similar to Matlab, too! For example, I do a lot of prototyping in python shell. All variables are saved with the shell. Now I meet a problem. I hope to debug one small piece of code, with those calculated variables with the shell. However, a new debugger cannot access old variables. It is not convenient for prototyping.
-
Clément about 8 yearsFor Emacs users, RealGUD has an incredibly good interface.
-
Amelio Vazquez-Reina about 8 yearsThanks @Clément I have been following the repo over the last month and I am very excited about the project :) I haven't tried it yet, but once I do (or if you do) feel free to write an answer here that maybe shows how to accomplish what's requested. For others for reference, the URL is github.com/rocky/emacs-dbgr
-
Amelio Vazquez-Reina about 8 years@Clément Also, if you have any experience with RealGUD & ipdb, I tried using it as explained here github.com/rocky/emacs-dbgr/issues/96 without luck.
-
Clément about 8 yearsIt's great that you opened a ticket :) Looks like your problem was solved, too ^^
-
-
Amelio Vazquez-Reina almost 11 yearsThanks tkf. I am big fan of your Emacs-Jedi package. When you said that Emacs 24.3 has nice support for ipdb, would you mind elaborating? I usually start IPython in a separate
M-x ansi-term
buffer, and then I use isend-mode to bind my source buffers to the IPython buffer so that I can send code to the IPython interpreter with a keyboard shortcut that automatically sends the%paste
magic to the IPython buffer. This allows me to quickly test regions in IPython. I always run my programs from this IPython shell withrun
and useembed()
to stop. -
tkf almost 11 yearsFor example, when you step through code, the source code is opened in the other buffer with arrow pointing the current execution point. You also have send-region command, like you do with isend-mode.
-
Amelio Vazquez-Reina almost 11 yearsThanks @tkf How can I start an
ipdb
debugging session usingpython.el
and have things likesend-region
etc. to the corresponding shell? In general, where can I find more info on this? -
tkf almost 11 yearsI use
%run
or%debug
. I guessimport ipdb; ipdb.set_trace()
works too. I don't think you can send multiple lines to ipdb. That's ipdb's limitation. But probably%paste
trick works. You may want to send a feature request to IPython dev. -
Wilfred Hughes about 10 yearsThis is a nasty pdb gotcha, and well worth knowing about.
-
Phani over 9 yearsPlease update this whenever you find an even better solution. But this looks great. I will use it.
-
Phani over 9 yearswhere do we use
ipshell.dummy_mode = True
? Can we use it from with in an ongoing ipython session? -
Amelio Vazquez-Reina over 9 years@Phani -- yes, you can do this anytime during the "debugging session" (i.e. whenever the nested shell
ipsh()
is invoked and you are done with debugging). -
Phani over 9 yearsbut it says that it cannot find
ipshell
variable. -
Gordon Bean over 9 yearsWhere/how do you define/import cfg, banner_msg, exit_msg, and inspect?
-
karan.dodia over 9 years@GordonBean I found the context in another answer: stackoverflow.com/questions/15669186/…
-
karan.dodia over 9 yearsThis seems to work well for me: github.com/kdodia/snippets/blob/master/ipsh.py
-
j08lue over 9 yearsThis is the way to do it.
-
Pascal about 9 yearsI needed to add
import inspect
before it worked. The command line customization seems to be broken for me though. -
Amelio Vazquez-Reina almost 9 yearsThanks @tuner That's correct. I have fixed it in my answer.
-
luator over 8 yearsWhy not just using
import ipdb; ipdb.set_trace()
? Maybe I miss something, but I don't see the advantage of your much more complicated method. -
Berk U. over 8 years@luator I think that the difference is that this method allows you to use iPython right at the breakpoint and execute any Python command.
ipdb.set_trace
only gives access to theipdb
commands -
luator over 8 years@BerkU. I can execute any Python command with
ipdb.set_trace
. At least I never noticed any restrictions and I use it quite a lot. -
Lucas Alonso about 8 yearsWhen I use this method and write this code:
a = 3; cap = [a for x in range(1)]
the interpreter outputs:NameError: name 'a' is not defined
. I have no idea why. -
Amelio Vazquez-Reina about 8 years@LucasAlonso I had the same problem myself with embedded shells. I think there are some tickets about this problem in IPython (BTW, see my latest update on this answer).
-
Amelio Vazquez-Reina about 8 yearsMy answer now includes how to use
ipdb
in Emacs usingRealGUD
andisend-mode
to do exactly what the OP asks. -
Cesar almost 7 years^ This is the real answer
-
F-A over 6 yearsJupyter is not a replacement for IPython, but for IPython Notebook. Jupyter notebooks uses kernel in the background. For Python notebook, the kernel is usually an IPython kernel. The IPython project continues further.
-
Lucas over 6 yearsIs there something similar where pdb has IPython like functionality without being in IPython initially?
-
alpha_989 over 6 yearsDoes this allow you to use
ipdb
statements within ipython shell, while debugging? -
sebastian about 5 yearsYou can also start the program with
ipython --pdb file.py -- args
and are dropped to ipdb upon an exception. Might be worth adding to the answer. -
Romeo Valentin about 5 yearsThis should be higher up. If I need to insert a line into my code (within a module) I need to restart IPython to reload the module. WIth this, I can just
%save /tmp/foo.py x-y
The code I want to run and debug and then%run -d /tmp/foo.py
to set the breakpoint wherever I like. Great answer! -
Richard Möhn over 4 years
breakpoint()
is wonderful. In PyCharm it even drops you into the PyCharm debugger. It's also a quick way to enter the PyCharm debugger from functions that were pasted into the console. -
Hritik over 2 yearsAfter the update I could simply set the environment variable
PYTHONBREAKPOINT=IPython.embed
and thenbreakpoint()
will do the job -
John Greene over 2 years
ModuleNotFoundError: No module named 'IPython.Debugger'
-
gaborous about 2 years@JohnGreene This is a very old answer, nowadays IPython has been renamed to Jupyter and the API changed a lot.