Python memoising/deferred lookup property decorator
Solution 1
For all sorts of great utilities I'm using boltons.
As part of that library you have cachedproperty:
from boltons.cacheutils import cachedproperty
class Foo(object):
def __init__(self):
self.value = 4
@cachedproperty
def cached_prop(self):
self.value += 1
return self.value
f = Foo()
print(f.value) # initial value
print(f.cached_prop) # cached property is calculated
f.value = 1
print(f.cached_prop) # same value for the cached property - it isn't calculated again
print(f.value) # the backing value is different (it's essentially unrelated value)
Solution 2
Here is an example implementation of a lazy property decorator:
import functools
def lazyprop(fn):
attr_name = '_lazy_' + fn.__name__
@property
@functools.wraps(fn)
def _lazyprop(self):
if not hasattr(self, attr_name):
setattr(self, attr_name, fn(self))
return getattr(self, attr_name)
return _lazyprop
class Test(object):
@lazyprop
def a(self):
print 'generating "a"'
return range(5)
Interactive session:
>>> t = Test()
>>> t.__dict__
{}
>>> t.a
generating "a"
[0, 1, 2, 3, 4]
>>> t.__dict__
{'_lazy_a': [0, 1, 2, 3, 4]}
>>> t.a
[0, 1, 2, 3, 4]
Solution 3
I wrote this one for myself... To be used for true one-time calculated lazy properties. I like it because it avoids sticking extra attributes on objects, and once activated does not waste time checking for attribute presence, etc.:
import functools
class lazy_property(object):
'''
meant to be used for lazy evaluation of an object attribute.
property should represent non-mutable data, as it replaces itself.
'''
def __init__(self, fget):
self.fget = fget
# copy the getter function's docstring and other attributes
functools.update_wrapper(self, fget)
def __get__(self, obj, cls):
if obj is None:
return self
value = self.fget(obj)
setattr(obj, self.fget.__name__, value)
return value
class Test(object):
@lazy_property
def results(self):
calcs = 1 # Do a lot of calculation here
return calcs
Note: The lazy_property
class is a non-data descriptor, which means it is read-only. Adding a __set__
method would prevent it from working correctly.
Solution 4
property
is a class. A descriptor to be exact. Simply derive from it and implement the desired behavior.
class lazyproperty(property):
....
class testA(object):
....
a = lazyproperty('_a')
b = lazyproperty('_b')
Solution 5
Here's a callable that takes an optional timeout argument, in the __call__
you could also copy over the __name__
, __doc__
, __module__
from func's namespace:
import time
class Lazyproperty(object):
def __init__(self, timeout=None):
self.timeout = timeout
self._cache = {}
def __call__(self, func):
self.func = func
return self
def __get__(self, obj, objcls):
if obj not in self._cache or \
(self.timeout and time.time() - self._cache[key][1] > self.timeout):
self._cache[obj] = (self.func(obj), time.time())
return self._cache[obj]
ex:
class Foo(object):
@Lazyproperty(10)
def bar(self):
print('calculating')
return 'bar'
>>> x = Foo()
>>> print(x.bar)
calculating
bar
>>> print(x.bar)
bar
...(waiting 10 seconds)...
>>> print(x.bar)
calculating
bar
Related videos on Youtube
detly
Updated on July 08, 2022Comments
-
detly almost 2 years
Recently I've gone through an existing code base containing many classes where instance attributes reflect values stored in a database. I've refactored a lot of these attributes to have their database lookups be deferred, ie. not be initialised in the constructor but only upon first read. These attributes do not change over the lifetime of the instance, but they're a real bottleneck to calculate that first time and only really accessed for special cases. Hence they can also be cached after they've been retrieved from the database (this therefore fits the definition of memoisation where the input is simply "no input").
I find myself typing the following snippet of code over and over again for various attributes across various classes:
class testA(object): def __init__(self): self._a = None self._b = None @property def a(self): if self._a is None: # Calculate the attribute now self._a = 7 return self._a @property def b(self): #etc
Is there an existing decorator to do this already in Python that I'm simply unaware of? Or, is there a reasonably simple way to define a decorator that does this?
I'm working under Python 2.5, but 2.6 answers might still be interesting if they are significantly different.
Note
This question was asked before Python included a lot of ready-made decorators for this. I have updated it only to correct terminology.
-
Bamcclur almost 8 yearsI'm using Python 2.7, and I don't see anything about ready-made decorators for this. Can you provide a link to the ready-made decorators that are mentioned in the question?
-
detly almost 8 years@Bamcclur sorry, there used to be other comments detailing them, not sure why they got deleted. The only one I can find right now is a Python 3 one:
functools.lru_cache()
. -
Guy over 7 yearsNot sure there are built-ins (at least Python 2.7), but there's the Boltons library's cachedproperty
-
detly about 7 years@guyarad I didn't see this comment until now. That is a fantastic library! Post that as an answer so I can upvote it.
-
-
Mike Boers about 14 yearsCan someone recommend an appropriate name for the inner function? I'm so bad at naming things in the morning...
-
spenthil about 14 yearsI usually name the inner function the same as the outer function with a preceding underscore. So "_lazyprop" - follows the "internal use only" philosophy of pep 8.
-
detly about 14 yearsThis works great :) I don't know why it never occurred to me to use a decorator on a nested function like that, too.
-
Admin over 11 yearsgiven the non-data descriptor protocol, this one is much slower and less elegant than the answer below using
__get__
-
Mike Boers over 11 years@Ronny: I agree, although I believe this answer is more approachable. Either way, +1 on the other answer.
-
Paul Etherton almost 11 yearsThis took a little while to understand but is an absolutely stunning answer. I like how the function itself is replaced by the value it calculates.
-
André Caron over 10 years
-
divieira over 10 yearsThanks for noting that Werkzeug has werkzeug.utils.cached_property: werkzeug.pocoo.org/docs/utils/#werkzeug.utils.cached_property
-
Dave Butler over 10 yearsI found this method to be 7.6 times faster than the selected answer. (2.45 µs / 322 ns) See ipython notebook
-
letmaik about 10 yearsTip: Put a
@wraps(fn)
below@property
to not loose your doc strings etc. (wraps
comes fromfunctools
) -
Sorin Neacsu over 9 yearsHere's a more complete implementation, including setting/invalidating cached values, complete with unit tests: github.com/sorin/lazyprop
-
detly over 8 yearsI used the term "lazy" because in the original implementation, the member was computed/retrieved from a DB at the time of object initialisation, and I want to defer that computation until the property was actually used in a template. This seemed to me to match the definition of laziness. I agree that since my question already assumes a solution using
@property
, "lazy" doesn't make a lot of sense at that point. (I also thought of memoisation as a map of inputs to cached outputs, and since these properties have only one input, nothing, a map seemed like more complexity than necessary.) -
detly over 8 yearsNote that all of the decorators that people have suggested as "out of the box" solutions didn't exist when I asked this, either.
-
poindexter over 8 yearsI agree with Jason, this is a question about caching/memoization not lazy evaluation.
-
detly over 8 years@poindexter - Caching doesn't quite cover it; it does not distinguish looking the value up at object init time and caching it from looking the value up and caching it when the property is accessed (which is the key feature here). What should I call it? "Cache-after-first-use" decorator?
-
poindexter over 8 years@detly Memoize. You should call it Memoize. en.wikipedia.org/wiki/Memoization
-
detly about 8 years@poindexter Memoisation requires inputs. I suppose you could consider "no input" to be a special case of memoisation where there is only ever one input (nothing), but again, I think the term implies more than what is going on here to anyone asking the question.
-
detly about 8 years@poindexter Although now that I've said that I see the point. I've corrected the terminology I've used in the question. Thanks for the feedback. I will flag this answer for deletion though, since it's really a comment.
-
tribbloid over 7 yearsSorry did you make a typo on the indentation of 'return _lazyprop' line?
-
detly over 7 yearsNice one, does exactly what I needed... although Pyramid might be a big dependency for one decorator
:)
-
Engineero about 7 yearsI name the inner function
wrapper
, but I bet that's technically incorrect somehow :P -
detly about 7 yearsTurns out
lru_cache
is a bit useless for this on methods, because it just sees each object instance as an argument, leading to problems with hashability etc. -
Evan Borgstrom about 7 yearsEAFP instead of LYBL: Instead of checking if an attr exists, just assume it does and handle when it doesn't in an except block. For properties accessed very frequently in busy code the data heuristics are much better when you only have to getattr.
-
detly about 7 yearsI'm changing the accepted answer now (sorry @MikeBoers!) because after a few years, there's now a good library that does it that's widely available. I think it's wiser to point people to that.
-
scanny almost 6 yearsNB: this does not prevent assignment to
fget
the way@property
does. To ensure immutability/idempotence, you need to add a__set__()
method that raisesAttributeError('can\'t set attribute')
(or whatever exception/message suits you, but this is whatproperty
raises). This unfortunately comes with a performance impact of a fraction of a microsecond because__get__()
will be called on each access rather than pulling fget value from dict on second and subsequent access. Well worth it in my opinion to maintain immutability/idempotence, which is key for my use cases, but YMMV. -
Peter Wood almost 6 years@detly The decorator implementation is simple, and you could implement it yourself, no need for the
pyramid
dependency. -
Antti Haapala -- Слава Україні almost 6 yearsHence the link says "source linked" :D
-
Peter Wood almost 6 years@AnttiHaapala I noticed, but thought I'd highlight that it's simple to implement for those that don't follow the link.
-
fitz almost 4 years@Cyclone, Your answer is very enlightening, but i don't understand why it, inside the
__get__()
method, needif obj is None
, and when the obj can be None?