How to find out where a Python Warning is from

Question

I'm still kinda new with Python, using Pandas, and I've got some issues debugging my Python script.

I've got the following warning message :

[...]\pandas\core\index.py:756: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
return self._engine.get_loc(key)

And can't find where it's from.

After some research, I tried to do that in the Pandas lib file (index.py):

try:
    return self._engine.get_loc(key)
except UnicodeWarning:
    warnings.warn('Oh Non', stacklevel=2)

But that didn't change anything about the warning message.

Solution 1

You can filter the warnings to raise which will enable you to debug (e.g. using pdb):

import warnings
warnings.filterwarnings('error')

*The warnings filter can be managed more finely (which is probably more appropriate) e.g.:

warnings.filterwarnings('error', category=UnicodeWarning)
warnings.filterwarnings('error', message='*equal comparison failed*')

Multiple filters will be looked up sequentially. ("Entries closer to the front of the list override entries later in the list, if both match a particular warning.")

Solution 2

You can also use the commandline to control the warnings:

python -W error::UnicodeWarning your_code.py

From the man page:

-W argument
[...] error to raise an exception instead of printing a warning message.

This will have the same effect as putting the following in your code:

import warnings
warnings.filterwarnings('error', category=UnicodeWarning)

As was already said in Andy's answer.

Solution 3

If you enable logging in python, then when an exception is received you can use the method logging.exception to log when an exception has been caught - this method will print out a nicely formatted stack trace that shows you exactly in the code where the exception originated. See the python document on logging for more information.

import logging
log = logging.getLogger('my.module.logger')

try:
    return self._engine.get_loc(key)
except UnicodeWarning:
    log.exception('A log message of your choosing')

Alternatively, you can get a tuple that contains details of the exception in your code by calling sys.exc_info() (this requires you to import the sys module).

Solution 4

The most informative way to investigate a warning is to convert it into an error (Exception) so you can see its full stacktrace:

import warnings
warnings.simplefilter("error")

See warnings.

View more solutions

Comments

[Andy Hayden]

I think the easiest way is to force it to raise, then you can debug as normal.

[Bonswouar]

Thanks but that also didn't change the warning message, except the line number (still on return self._engine.get_loc(key)). That's weird, but I guess that would be because it's a lib so it's using the compiled (.pyc) file ? How do you usually debug warnings from libs ?

[robjohncox]

Ah so you don't have the source code - that makes life difficult! Can you get hold of the source code? If not, then I would start experimenting with passing different data into the closed library to see if you can isolate what it is about the data you are providing that causes the error. As a last resort, decompiling the library (for example using dis) and debugging into the decompiled source may be the only way to get to the bottom of it.

[Bonswouar]

I do have the source code (all .pyc .py and .pyo files together), but (how) should I recompile it to make it work ?

[robjohncox]

OK thats good - if you have the source code, then you should be able to use a debugger to step into the library code and examine what is happening in there. There should be no need to recompile anything - as long as the source code is in your PYTHONPATH then there should be no issue stepping into the library with a debugger.

[HaPsantran]

I'm observing cases where adding the broader warning (in a driver script) results in no error OR warning resulting in the module the driver uses, whereas when I do NOT use warnings, I DO see the warning. Any experience with this?

[Andy Hayden]

@HaPsantran please ask this as a new question, that way it'll get more attention. Feel free the CC me here with a link to the new question. I'm not sure I follow you, but the point here is that a library can warn and we want to make this an exception (so we can fix that warning more easily).

[jtorca]

Try to finely tune the warnings as suggested. At first I tried throwing an error for any warning and my code stopped executing after some obscure Numpy warning I didn't even seen in the previous runs. Then I was more specific and that solved the problem.

[LucG]

The extra filter warning raised an error for me: re.error: nothing to repeat at position 0 with python version 3.7.10 . It is because the regex in the message argument is not a valid regex as the * symbol should follow a character to repeat. Adding a . (dot) before each * solved the issue.