[python] Advanced Design Patterns in Python

source: http://pypix.com/python/advanced-data-structures/

Advanced Design Patterns in Python

  

 inShare22

The aim of this tutorial is to show off Advanced design structures in Python and the best way to use them. Depending on what you need from a data structure, whether it’s fast lookup, immutability, indexing, etc, you can choose the best data structure for the job and most of the time, you will be combining data structures together to get a logical and easy to understand data model. Python data structures are very intuitive from a syntax point of view and they offer a large choice of operations. This tutorial tries to put together the most common and useful information about each data structure and offer a guide on when it is best to use one structure or another.

Comprehensions

If you’ve used Python for very long, you’ve at least heard of list comprehensions. They’re a way to fit a for loop, an if statement, and an assignment all in one line. In other words, you can map and filter a list in one expression.

A list comprehension consists of the following parts:

• An Input Sequence.
• A Variable representing members of the input sequence.
• An Optional Predicate expression.
• An Output Expression producing elements of the output list from members of the Input Sequence that satisfy the predicate.

Say we need to get a list of all the integers, whose value is above zero in a sequence and then square them:

1 2 3 4 5 6 7 8 9

num = [1, 4, -5, 10, -7, 2, 3, -1] filtered_and_squared = []   for number in num:     if number > 0:         filtered_and_squared.append(number ** 2) print filtered_and_squared   # [1, 16, 100, 4, 9]

Pretty simple, right? But it took 4 lines, two degrees of nesting, and an append to do something completely trivial. You could reduce the size of the code with the filter, lambda and map function:

1 2 3 4 5

num = [1, 4, -5, 10, -7, 2, 3, -1] filtered_and_squared = map(lambda x: x ** 2, filter(lambda x: x > 0, num)) print filtered_and_squared   # [1, 16, 100, 4, 9]

The code is starting to expand in the horizontal direction now! Alas, what could we possibly do to simplify the code? List Comprehension to rescue.

1 2 3 4 5

num = [1, 4, -5, 10, -7, 2, 3, -1] filtered_and_squared = [ x**2 for x in num if x > 0] print filtered_and_squared   # [1, 16, 100, 4, 9]

• The iterator part iterates through each member x of the input sequence num.
• The predicate checks if the member is greater than zero.
• If the member is greater than zero then it is passed to the output expression, squared, to become a member of the output list.

The list comprehension is enclosed within a list so, it is immediately evident that a list is being produced. There is only one function call to type and no call to the cryptic lambda instead the list comprehension uses a conventional iterator, an expression and an if expression for the optional parameter.

There is a downside to list comprehensions: the entire list has to be stored in memory at once. This isn’t a problem for small lists like the ones in the above examples, or even of lists several orders of magnitude larger. But eventually this becomes pretty inefficient.

Generators to rescue this time. Generator expressions do not load the whole list into memory at once, but instead create a ‘generator object’ so only one list element has to be loaded at any time.

Generator expressions have the same syntax as list comprehensions, but with parentheses around the outside instead of brackets:

1 2 3 4 5 6 7 8 9 10

num = [1, 4, -5, 10, -7, 2, 3, -1] filtered_and_squared = ( x**2 for x in num if x > 0 ) print filtered_and_squared   # <generator object <genexpr> at 0x00583E18>   for item in filtered_and_squared:     print item   # 1, 16, 100 4,9

This is ever so slightly more efficient than using a list comprehension.

Lets change it to a more efficient code:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

num = [1, 4, -5, 10, -7, 2, 3, -1]   def square_generator(optional_parameter):     return (x ** 2 for x in num if x > optional_parameter)   print square_generator(0) # <generator object <genexpr> at 0x004E6418>   # Option I for k in square_generator(0):     print k # 1, 16, 100, 4, 9   # Option II g = list(square_generator(0)) print g # [1, 16, 100, 4, 9]

It’s probably good practice to use generator expressions unless there’s some reason not to, but you’re not going to see any real difference in efficiency unless the list is very large.

You can use zip() and dealing with two or more elements at a time:

1 2 3 4 5 6 7 8 9

alist = ['a1', 'a2', 'a3'] blist = ['1', '2', '3']   for a, b in zip(alist, blist):     print a, b   # a1 1 # a2 2 # a3 3

A two-level list comprehension using os.walk():

1 2 3 4 5 6 7 8

import os def tree(top):     for path, names, fnames in os.walk(top):         for fname in fnames:             yield os.path.join(path, fname)   for name in tree('C:\Users\XXX\Downloads\Test'):     print name

Decorators

Decorator provide a very useful method to add functionality to existing functions and classes. Sounds a bit like Aspect-Oriented Programming (AOP) in Java, doesn’t it? Except that it’s both much simpler and (as a result) much more powerful. For example, suppose you’d like to do something at the entry and exit points of a function (such as perform some kind of security, tracing, locking, etc. – all the standard arguments for AOP).

A decorator is a function that wraps another function: the main function is called and its return value is passed to the decorator. The decorator then returns a function that replaces the wrapped function as far as the rest of the program is concerned.

1 2 3 4 5 6 7 8 9 10

def timethis(func):     '''     Decorator that reports the execution time.     '''     pass   @timethis def countdown(n):     while n > 0:         n -= 1

The @ indicates the application of the decorator.

Now let’s go back and implement the first example in decorator section. Here, we’ll do the more typical thing and actually use the code in the decorated functions:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

import time from functools import wraps   def timethis(func):     '''     Decorator that reports the execution time.     '''     @wraps(func)     def wrapper(*args, **kwargs):         start = time.time()         result = func(*args, **kwargs)         end = time.time()         print(func.__name__, end-start)         return result     return wrapper   @timethis def countdown(n):     while n > 0:         n -= 1   countdown(100000)   # ('countdown', 0.006999969482421875)

Whenever you write code like this:

1 2	@timethis def countdown(n):

it’s the same as if you had performed these separate steps:

1 2 3

def countdown(n): ... countdown = timethis(countdown)

The code inside a decorator typically involves creating a new function that accepts any
arguments using *args and **kwargs, as shown with the wrapper() function in this
recipe. Inside this function, you place a call to the original input function and return its
result. However, you also place whatever extra code you want to add (e.g., timing). The
newly created function wrapper is returned as a result and takes the place of the original
function.

Lets look at another example

1 2 3

@decorator def function():     print("inside function")

When the compiler passes over this code, function() is compiled and the resulting function object is passed to the decorator code, which does something to produce a function-like object that is then substituted for the original function().

What does the decorator code look like? Well, most examples show this as a function, but I’ve found that it’s easier to start understanding decorators by using classes as decoration mechanisms instead of functions. In addition, it’s more powerful.

The only constraint upon the object returned by the decorator is that it can be used as a function – which basically means it must be callable. Thus, any classes we use as decorators must implement __call__.

What should the decorator do? Well, it can do anything but usually you expect the original function code to be used at some point. This is not required, however:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

class decorator(object):       def __init__(self, f):         print("inside decorator.__init__()")         f() # Prove that function definition has completed       def __call__(self):         print("inside decorator.__call__()")   @decorator def function():     print("inside function()")   print("Finished decorating function()")   function()   # inside decorator.__init__() # inside function() # Finished decorating function() # inside decorator.__call__()

Practical Example:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

def decorator(func):     def modify(*args, **kwargs):         variable = kwargs.pop('variable', None)         print variable         x,y=func(*args, **kwargs)         return x,y     return modify   @decorator def func(a,b):     print a**2,b**2     return a**2,b**2   func(a=4, b=5, variable="hi") func(a=4, b=5)   # hi # 16 25 # None # 16 25

ContextLib

The contextlib module contains utilities for working with context managers and the with statement. Normally, to write a context manager, you define a class with an __enter__() and __exit__() method, like this:

1 2 3 4 5 6 7 8 9 10 11

import time class demo:     def __init__(self, label):         self.label = label       def __enter__(self):         self.start = time.time()       def __exit__(self, exc_ty, exc_val, exc_tb):         end = time.time()         print('{}: {}'.format(self.label, end - self.start))

Complete Example:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

import time   class demo:     def __init__(self, label):         self.label = label       def __enter__(self):         self.start = time.time()       def __exit__(self, exc_ty, exc_val, exc_tb):         end = time.time()         print('{}: {}'.format(self.label, end - self.start))   with demo('counting'):     n = 10000000     while n > 0:         n -= 1   # counting: 1.36000013351

A context manager is enabled by the with statement, and the API involves two methods. The  __enter__() method is run when execution flow enters the code block inside the with. It returns an object to be used within the context. When execution flow leaves the with block, the __exit__() method of the context manager is called to clean up any resources being used.

Again writing the sample example using the @contextmanager decorator in the contextlibmodule.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

from contextlib import contextmanager import time   @contextmanager def demo(label):     start = time.time()     try:         yield     finally:         end = time.time()         print('{}: {}'.format(label, end - start))   with demo('counting'):     n = 10000000     while n > 0:         n -= 1   # counting: 1.32399988174

In the function, all of the code prior to the yield executes as the __enter__() method of a context manager. All of the code after the yield executes as the __exit__() method. If there was an exception, it is raised at the yield statement.

Descriptors

Descriptors determine how attribute of object are accessed. A descriptor is a way to customize what happens when you reference an attribute on a model.

The real trick to building a descriptor is defining at least one of the following three methods. Note that instance below returns to the object where the attribute was accessed, and owner is the class where the descriptor was assigned as an attribute.

• __get__(self, instance, owner) — This will be called when the attribute is retrieved (value = obj.attr), and whatever it returns is what will be given to the code that requested the attribute’s value.
• __set__(self, instance, value) — This gets called when a value is set to the attribute (obj.attr = ‘value’), and shouldn’t return anything at all.
• __delete__(self, instance) — This is called when the attribute is deleted from an object (del obj.attr).

LazyLoading Properties

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

import weakref   class lazyattribute(object):     def __init__(self, f):         self.data = weakref.WeakKeyDictionary()         self.f = f     def __get__(self, obj, cls):         if obj not in self.data:             self.data[obj] = self.f(obj)         return self.data[obj]   class Foo(object):     @lazyattribute     def bar(self):         print "Being lazy"         return 42   f = Foo()   print f.bar # Being lazy # 42   print f.bar # 42

Descriptors are a generalization of the concept of bound methods, which was central to the implementation of classic classes. In classic classes, when an instance attribute is not found in the instance dictionary, the search continues with the class dictionary, then the dictionaries of its base classes, and so on recursively. When the attribute is found in a class dictionary (as opposed to in the instance dictionary), the interpreter checks if the object found is a Python function object. If so, the returned value is not the object found, but a wrapper object that acts as a currying function. When the wrapper is called, it calls the original function object after inserting the instance in front of the argument list.“

As mentioned above, descriptors are assigned to classes, and the special methods are called automatically when the attribute is accessed, and the method used depends on what type of access is being performed.

MetaClasses

Metaclasses offer a powerful way to change how classes in Python behave.

A metaclass is defined as “the class of a class”. Any class whose instances are themselves classes, is a metaclass.

1 2 3 4 5 6 7 8 9 10

class demo(object):     pass   obj = demo()   print "Class of obj is {0}".format(obj.__class__) print "Class of obj is {0}".format(demo.__class__)   # Class of obj is <class '__main__.demo'> # Class of obj is <type 'type'>

We’ve created a class and an object of that class. Examining the __class__ of obj we saw that it’s demo. Next comes the interesting part. What is the class of demo? We can again examine it with __class__ and we see it’s type.

So type is the class of Python classes. In other words, while in the example above obj is a demo object, demo itself is a type object.

So, according to what we’ve seen above, this makes type a metaclass – in fact, the most commonly used metaclass in Python, since it’s the default metaclass of all classes.

Since a metaclass is the class of a class, it is used to construct classes (just as a class is used to construct objects). But wait a second, don’t we create classes with a standard class definition? Definitely, but what Python does under the hood is the following:

• When it sees a class definition, Python executes it to collect the attributes (including methods) into a dictionary.
• When the class definition is over, Python determines the metaclass of the class. Let’s call itMeta
• Eventually, Python executes Meta(name, bases, dct), where:
  • Meta is the metaclass, so this invocation is instantiating it.
  • name is the name of the newly created class
  • bases is a tuple of the class’s base classes
  • dct maps attribute names to objects, listing all of the class’s attributes

How do we determine the metaclass of a class? Simply stated, if either a class or one of its bases has a __metaclass__ attribute, it’s taken as the metaclass. Otherwise, type is the metaclass.

Patterns

“It’s easier to ask for forgiveness than permission (EFAP)”

One pythonic principle is “It’s easier to ask for forgiveness than permission (EFAP)”. Opposed to the approach to look before you leap, this principle states that you should first try an action and if it fails react appropriately. Python’ strong exception handling supports this principle and helps to develop robust and fault tolerant programs.

Singelton

Singeltons are objects of which only one instance is supposed to exist. Python provides several ways to implement singeltons.

Null Objects

Null objects can be used instead of the type None to avoid tests for None.

Observer

The observer pattern allows several objects to have access to the same data.

Constructor

Parameters of constructors are often assigned to instance variables. This pattern can replace a many lines of manual assignment with only one line of code.

Conclusion

Thanks for reading. Drop your comments for further discussion.

Download article as PDF

About the author

Written by Ajay Kumar N