Python Programming Tips #3

We'll start with a simple Rect class that uses integer coordinates, and see how much memory one million occupy. Then we'll start to reduce the memory consumption and do some code improvement along the way.

Here's the simple Rect class:

class Rect:

    def __init__(self, x1, y1, x2, y2):
        self.x1 = x1
        self.x2 = x2
        self.y1 = y1
        self.y2 = y2

When we created a million of these using a 64-bit Python 3 on a 64-bit operating system they occupied about 400KB.

Rather than try the normal __slots__ technique, we began by using a more subtle approach:

class Rect(tuple):

    __slots__ = ()

    def __new__(Class, x1, y1, x2, y2):
        return super().__new__(Class, (x1, y1, x2, y2))

This makes our Rect a tuple subclass. Using it reduces memory consumption by about 43% to about 227KB.

Now let's look at the classic __slots__ approach:

class Rect:

    __slots__ = ("x1", "x2", "y1", "y2")

    def __init__(self, x1, y1, x2, y2):
        self.x1 = x1
        self.x2 = x2
        self.y1 = y1
        self.y2 = y2

This reduces memory consumption by about 47% to around 212KB. This is what the textbooks teach, so a lot of people might be tempted to think this is as far as they can go (at least without dropping down to C).

Incidentally, there is one drawback of using __slots__: you can't add arbitrary extra data to Rect instances as you could if the class used a dict. For our use cases this hasn't mattered; and anyway, there's nothing to stop us adding more items to the __slots__ in the future.

But actually, we can do a lot better. We'll show two versions, the first rather long-winded but fairly easy to understand; the second, much more compact although a bit trickier.

class Rect:

    __slots__ = ("_data",)

    # We are not limited to using the same types; could mix any
    # fixed-width types we want. And, of course, we can add extra
    # items to the struct later if need be.
    Coords = struct.Struct("llll")

    def __init__(self, x1, y1, x2, y2):
        self._data = Rect.Coords.pack(x1, y1, x2, y2)

    @property
    def x1(self):
        return Rect.Coords.unpack(self._data)[0]

    @property
    def x2(self):
        return Rect.Coords.unpack(self._data)[1]

    @property
    def y1(self):
        return Rect.Coords.unpack(self._data)[2]

    @property
    def y2(self):
        return Rect.Coords.unpack(self._data)[3]

What we've done here is made every Rect store a single Python object (Rect._data), rather than four separate objects, thus potentially reducing the overhead by 75%. Of course we have to pay for this somehow, and we've paid by having a processing overhead in that when a coordinate is looked up we have to extract it. But with modern CPUs this probably doesn't matter if memory is at a premium.

Using this technique reduces memory consumption by 66%, taking it down to about 137KB compared with the 400KB the original class needed. And we could save even more memory by using a smaller integer size.

(Incidentally, throughout we've only shown read-only properties, there's no reason the examples couldn't be extended to have writable properties.)

Finally, here's a much shorter version that has the same performance characteristics, but is written using much less code.

def _make_unpacker(index):
    return lambda self: operator.itemgetter(index)(
        Rect.Coords.unpack(self._data))

class Rect:

    __slots__ = ("_data",)

    Coords = struct.Struct("llll")

    def __init__(self, x1, y1, x2, y2):
        self._data = Rect.Coords.pack(x1, y1, x2, y2)

    x1 = property(_make_unpacker(0))
    x2 = property(_make_unpacker(1))
    y1 = property(_make_unpacker(2))
    y2 = property(_make_unpacker(3))

Here, we've used a private helper function (_make_unpacker()) to create each property.

Does this matter in practice? We've certainly found it made a huge difference for an application that needed to create millions of items (not rectangles, but similar in principle).

For more see Python Programming Tips

Your Privacy • Copyright © 2006 Qtrac Ltd. All Rights Reserved.

Top