VS Code / Python / Debugging pytest Test with the Debugger

Question

How can I get VS Code to put me into the debugger at the point of failure when running tests with pytest?

Pytest catches all errors and asserts, and VS code invokes the debugger only on uncaught errors (I can change this to raised exception, but then it stops on everything raised under a try).

I tried to set --pdb as pytest argument, but this leads to errors:

============================= test session starts =============================
platform win32 -- Python 3.8.1, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: c:\Projects\debugtest, inifile: pytest.ini
collected 1 item

test.py F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Traceback (most recent call last):
  File "C:\Projects\debugtest\test.py", line 4, in test_with_assert
    assert 42==2.71828
AssertionError: assert 42 == 2.71828
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>
> c:\projects\debugtest\test.py(4)test_with_assert()
-> assert 42==2.71828
(Pdb)

PYDEV DEBUGGER WARNING:
sys.settrace() should not be used when the debugger is being used.
This may cause the debugger to stop working correctly.
If this is needed, please check: 
http://pydev.blogspot.com/2007/06/why-cant-pydev-debugger-work-with.html
to see how to restore the debug tracing back correctly.
Call Location:
  File "C:\Program Files\Python38\lib\bdb.py", line 359, in set_quit
    sys.settrace(None)


- generated xml file: C:\Users\tzhgfma1\AppData\Local\Temp\tmp-24044mEWMyB1nPYAu.xml -
!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!
============================== 1 failed in 0.43s ==============================

I have a very simple project for testing this:

.vscode\settings.json

{
    "python.testing.pytestArgs": [
        "--pdb"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "git.ignoreLimitWarning": false
}

.vscode\launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        
        
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "internalConsole",
            "justMyCode":false
        },
        {
            "name": "Python: Attach using Process Id",
            "type": "python",
            "request": "attach",
            "processId": "${command:pickProcess}",
            "justMyCode": false
        },
        {
            "name": "Debug Tests",
            "type": "python",
            "request": "test",
            "console": "internalConsole",
            "justMyCode": false
        }
    ]
}

pytest.ini

[pytest]

python_files = test*.py
python_classes = Test
python_functions = test
addopts = --tb=native
console_output_style = classic
junit_duration_report = call
filterwarnings =
    ignore::RuntimeWarning

and test.py:

def test_with_assert():
    assert 42==2.71828

Is --pdb the correct way to do this? Or how do I enter the debugger upon an assert or error?

Solution 1

If you want to enter debugging when running pytest in VSCode and stay at a line of code, you could click 'Debug Test' at the top of the method after selecting the pytest test, as shown in the screenshot:

In addition, "python.testing.pytestArgs": [], in .vscode\settings.json is the folder path of the tested file, for example, my test file is in Test_cc under the a folder.

>      "python.testing.pytestArgs": [
>             "a/Test_cc"
>         ],

If this is not what you want, please let me know and describe the details of your needs.

Reference: Debug tests.

Update:

Usually, when we debug the test in VSCode, without setting a breakpoint for it, it will only display the test result. (success or failure). It will only display relevant test information in the console OUTPUT.

When I use the extension 'Python Test Explorer for Visual Studio code', it will display the debugging test information on the console, prompting the issue.

Solution 2

This Answer actually solves the problem. I wish there would be a more straight-forward way, either from Visual Studio Code or a simple raise flag from pytest.

Comments

[mgf]

So I have to set a breakpoint where the assert fails? In the real application, I have hundreds of asserts and so many errors (I'm porting a legacy code base to Py3), I thought I could enter the debugger at the first error or first failing assert, without knowing the condition firsthand when it will trigger (some asserts are correct for a thousand runs, then fail on one). But pytest always completes, reporting failures at the end, instead of breaking into the debugger. If I set Breakpoint to Raised Exceptions, it will break on any handled exception, which is far too much.

[Jill Cheng]

@mgf I added some content to my answer, and you could try to refer to it.

[mgf]

Thanks. I've installed the extension, but it still completes the tests instead of stopping in the debugger at the point of failure. I guess this is a problem of pytest, which cannot be made to raise the error, so it remains uncaught, triggering the VSC debugger.

[Jill Cheng]

@mgf Thank you for your feedback. At present, VSCode cannot automatically catch errors when debugging test scripts, unless we set breakpoints for it or obtain relevant information from the console. We have already reported this issue(the link), let us look forward to the realization of this feature. GitHub link: github.com/microsoft/vscode-python/issues/13709

[mgf]

Thank you very much Jill! Your support is greatly appreciated!

[brandonscript]

This was honestly pretty straightforward and doesn't require fiddling with a (not very well documented) VSCode extension.

[mgf]

@brandonscript Can you pls explain what is pretty straightforward? I agree with stackoverflow.com/questions/62419998/… that this should be trivial, but it isn't.

[brandonscript]

Adding a few lines of code your conftest.py, and an env variable to launch.json? Took me about a minute to copy paste it into my workspace.

[mgf]

@brandonscript Yes, the lines are few, but not intuitive. I agree with the author of the quoted question: "But I have a very hard time understanding how pytest does not have a flag or mode for this. It seems like the most normal natural thing in the world to want the debugger to break where you your test fails."

[teichert]

@mgf: I agree that it would be great to have an option built-in. here's a possibly more straight-forward, but certainly more cumbersome hack. :)