FAQ
My beta testers are complaining about excessive memory usage. It's a
wxPython app with several embedded mozilla activex controls and a local
web server.

Unfortunately, Python has some problems in this area. In particular,
since ubiquitous lists and dictionaries are dynamically resized as
needed, memory fragmentation seems inevitable.

Also, memory freed by python apparently is not returned to the OS
according to this article:
http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm

Are there any good general countermeasures to make an application use
less memory? Any good overview articles on this subject?

Thanks (and Happy Thanksgiving),
- Ken Seehart

Search Discussions

  • Skip at Nov 27, 2008 at 8:35 pm
    Ken> Unfortunately, Python has some problems in this area. In
    Ken> particular, since ubiquitous lists and dictionaries are dynamically
    Ken> resized as needed, memory fragmentation seems inevitable.

    That's not necessarily true. Also, I would say that Python has made
    tradeoffs in this area, not that it necessarily "has problems". There is a
    significant time/space tradeoff to be made. By hanging onto free lists and
    structuring its allocator to group objects of similar sizes together,
    allocation time of small objects (ints, floats, short strings, small dicts,
    tuples and lists, etc) is greatly improved. (Small object allocation
    dominates the dynamic memory allocation profile of Python.)

    Ken> Also, memory freed by python apparently is not returned to the OS
    Ken> according to this article:
    Ken> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm

    Correct. There is a hierarchy of memory allocators in any system. In Unix
    systems the system calls brk and sbrk are used to allocate and free large
    chunks of memory from the operating system. These are generally called by
    malloc, realloc and (sometimes) free. They are rarely, if ever, called
    directly by applications. Applications call malloc, free, realloc, etc.
    Python has its own allocator, obmalloc, which sits on top of malloc. Only
    brk and sbrk can actually free up memory pages (truly return them to the
    operating system). Some malloc implementations will do that (and then only
    if an entire page of memory is free), but many won't. You might find the
    large comments sprinkled throught obmalloc.c helpful:

    http://svn.python.org/view/python/trunk/Objects/obmalloc.c

    Note the big row of equal signs in the first comment block. Anything
    retained above that is memory charged to the application (size you see in
    top(1) for example). This is why the malloc library's behavior is
    important. Python's allocator can return arenas to the malloc library by
    calling free(), but there's no guarantee that the malloc library will.

    Ken> Are there any good general countermeasures to make an application
    Ken> use less memory? Any good overview articles on this subject?

    If your Python application is dealing with large lists of numbers you might
    investigate the array module or the external numpy module. In theory you
    could also try to find a different malloc library which has better policies
    about returning memory to the system.

    --
    Skip Montanaro - skip at pobox.com - http://smontanaro.dyndns.org/

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedNov 27, '08 at 7:50p
activeNov 27, '08 at 8:35p
posts2
users2
websitepython.org

2 users in discussion

Skip: 1 post Ken Seehart: 1 post

People

Translate

site design / logo © 2022 Grokbase