A python decorator
is a function (or class) that returns another function usually after applying
some transformation to it. Common examples of decorators are classmethod() and
staticmethod().
Decorators have a barrage of uses ranging from memoization, profiling, access
control to function timeouts. There is a collection of these and other decorator
code pieces here.
I have found myself mostly using decorators for retry and occasionally
timeouts for network centric operations. Normally I would have a single
decorator function that deals with particular exception and does several
retries before re-raising the exception.
Here is an example.
import time
from functools import wraps
from http.client import BadStatusLine
def if_http_errors_retry(func):
"""Decorator: retry calling function func in case of http.client errors.
Decorator will try to call, the function three times, with a ten seconds
delay between them. If the retries get maxed out, the decorator will raise
the http error.
args:
func (function): function to be decorated.
returns:
func (function): decorated function
"""
@wraps(func)
def wrapper(*args, **kwargs):
"""func wrapper"""
error = Exception
for _ in range(3):
try:
return func(*args, **kwargs)
except BadStatusLine as err:
error = err
time.sleep(10)
continue
raise error
return wrapper
Looking keenly at the above decorator, you will realise that it bears some
design flaws:
You can’t specify the amount of time delay: it’s hard coded
You can’t specifiy the number of retries: it’s hard coded
It’s hard to generalize the decorator for another exception without entirely
duplicating it
After noting how much my code stinks I decided to refactor it and deal with the
three issues above. The motivation for this mainly came when I saw this retry template.
def auto_retry(n=3, exc=Exception):
for i in range(n):
try:
yield None
return
except exc, err:
# perhaps log exception here
continue
raise # re-raise the exception we caught earlier
It’s not a decorator at all but it had all the three qualities that my decorator
lacked and it gave me a basis for me to start exploring.
It turns out refactoring the decorator was pretty easy.
import time
from functools import wraps
from http.client import BadStatusLine
def auto_retry(tries=3, exc=Exception, delay=5):
def deco(func):
@wraps(func)
def wrapper(*args, **kwargs):
for _ in range(tries):
try:
return func(*args, **kwargs)
except exc:
time.sleep(delay)
continue
raise exc
return wrapper
return deco
# decorating
@auto_retry(tries=3, exc=BadStatusLine, delay=5)
def network_call():
# some code using http.client
Voila, now I have a very flexible retry decorator that can be applied on any
type of exception, time delay and number of retries.
The key to achieving this is having a function (auto_retry) that returns a
decorator function (deco) which will in turn decorates a function (func).
Thanks to the power of closures the parameters passed to the high order
function (auto_retry) are also available to the nested functions and are
used for the control flow within them.