How to watch for a variable change in python without dunder setattr or pdb

Question

There is large python project where one attribute of one class just have wrong value in some place.

It should be sqlalchemy.orm.attributes.InstrumentedAttribute, but when I run tests it is constant value, let's say string.

There is some way to run python program in debug mode, and run some check (if variable changed type) after each step throught line of code automatically?

P.S. I know how to log changes of attribute of class instance with help of inspect and property decorator. Possibly here I can use this method with metaclasses...

But sometimes I need more general and powerfull solution...

Thank you.

P.P.S. I need something like there: https://stackoverflow.com/a/7669165/816449, but may be with more explanation of what is going on in that code.

Solution 1

Well, here is a sort of slow approach. It can be modified for watching for local variable change (just by name). Here is how it works: we do sys.settrace and analyse the value of obj.attr each step. The tricky part is that we receive 'line' events (that some line was executed) before line is executed. So, when we notice that obj.attr has changed, we are already on the next line and we can't get the previous line frame (because frames aren't copied for each line, they are modified ). So on each line event I save traceback.format_stack to watcher.prev_st and if on the next call of trace_command value has changed, we print the saved stack trace to file. Saving traceback on each line is quite an expensive operation, so you'd have to set include keyword to a list of your projects directories (or just the root of your project) in order not to watch how other libraries are doing their stuff and waste cpu.

watcher.py

import traceback

class Watcher(object):
    def __init__(self, obj=None, attr=None, log_file='log.txt', include=[], enabled=False):
        """
            Debugger that watches for changes in object attributes
            obj - object to be watched
            attr - string, name of attribute
            log_file - string, where to write output
            include - list of strings, debug files only in these directories.
               Set it to path of your project otherwise it will take long time
               to run on big libraries import and usage.
        """

        self.log_file=log_file
        with open(self.log_file, 'wb'): pass
        self.prev_st = None
        self.include = [incl.replace('\\','/') for incl in include]
        if obj:
            self.value = getattr(obj, attr)
        self.obj = obj
        self.attr = attr
        self.enabled = enabled # Important, must be last line on __init__.

    def __call__(self, *args, **kwargs):
        kwargs['enabled'] = True
        self.__init__(*args, **kwargs)

    def check_condition(self):
        tmp = getattr(self.obj, self.attr)
        result = tmp != self.value
        self.value = tmp
        return result

    def trace_command(self, frame, event, arg):
        if event!='line' or not self.enabled:
            return self.trace_command
        if self.check_condition():
            if self.prev_st:
                with open(self.log_file, 'ab') as f:
                    print >>f, "Value of",self.obj,".",self.attr,"changed!"
                    print >>f,"###### Line:"
                    print >>f,''.join(self.prev_st)
        if self.include:
            fname = frame.f_code.co_filename.replace('\\','/')
            to_include = False
            for incl in self.include:
                if fname.startswith(incl):
                    to_include = True
                    break
            if not to_include:
                return self.trace_command
        self.prev_st = traceback.format_stack(frame)
        return self.trace_command
import sys
watcher = Watcher()
sys.settrace(watcher.trace_command)

testwatcher.py

from watcher import watcher
import numpy as np
import urllib2
class X(object):
    def __init__(self, foo):
        self.foo = foo

class Y(object):
    def __init__(self, x):
        self.xoo = x

    def boom(self):
        self.xoo.foo = "xoo foo!"
def main():
    x = X(50)
    watcher(x, 'foo', log_file='log.txt', include =['C:/Users/j/PycharmProjects/hello'])
    x.foo = 500
    x.goo = 300
    y = Y(x)
    y.boom()
    arr = np.arange(0,100,0.1)
    arr = arr**2
    for i in xrange(3):
        print 'a'
        x.foo = i

    for i in xrange(1):
        i = i+1

main()

Solution 2

Try using __setattr__ to override the function that is called when an attribute assignment is attempted. Documentation for __setattr__

Solution 3

A simpler way to watch for an object's attribute change (which can also be a module-level variable or anything accessible with getattr) would be to leverage hunter library, a flexible code tracing toolkit. To detect state changes we need a predicate which can look like the following:

import traceback


class MutationWatcher:

    def __init__(self, target, attrs):
        self.target = target
        self.state = {k: getattr(target, k) for k in attrs}

    def __call__(self, event):
        result = False
        for k, v in self.state.items():
            current_value = getattr(self.target, k)
            if v != current_value:
                result = True
                self.state[k] = current_value
                print('Value of attribute {} has chaned from {!r} to {!r}'.format(
                    k, v, current_value))

        if result:
            traceback.print_stack(event.frame)

        return result

Then given a sample code:

class TargetThatChangesWeirdly:
    attr_name = 1


def some_nested_function_that_does_the_nasty_mutation(obj):
    obj.attr_name = 2


def some_public_api(obj):
    some_nested_function_that_does_the_nasty_mutation(obj)

We can instrument it with hunter like:

# or any other entry point that calls the public API of interest
if __name__ == '__main__':
    obj = TargetThatChangesWeirdly()

    import hunter
    watcher = MutationWatcher(obj, ['attr_name'])
    hunter.trace(watcher, stdlib=False, action=hunter.CodePrinter)

    some_public_api(obj)

Running the module produces:

Value of attribute attr_name has chaned from 1 to 2
  File "test.py", line 44, in <module>
    some_public_api(obj)
  File "test.py", line 10, in some_public_api
    some_nested_function_that_does_the_nasty_mutation(obj)
  File "test.py", line 6, in some_nested_function_that_does_the_nasty_mutation
    obj.attr_name = 2
                                 test.py:6     return        obj.attr_name = 2
                                               ...       return value: None

You can also use other actions that hunter supports. For instance, Debugger which breaks into pdb (debugger on an attribute change).

Solution 4

There's a very simple way to do this: use watchpoints.

Basically you only need to do

from watchpoints import watch
watch(your_object.attr)

That's it. Whenever the attribute is changed, it will print out the line that changed it and how it's changed. Super easy to use.

It also has more advanced features, for example, you can call pdb when the variable is changed, or use your own callback functions instead of print it to stdout.

View more solutions