FAQ
When coding C I have often found static local variables useful for
doing once-only run-time initializations. For example:

int foo(int x, int y, int z) {

static int first_time = TRUE;
static Mongo *mongo;
if (first_time) {
mongo = heavy_lifting_at_runtime();
first_time = FALSE;
}

return frobnicate(mongo, x, y, z);
}

In this case, the static variable mongo is initialized only once
(at most).

What I like most about this is that it obviates the need for a
global variable to hold the persistent value (I avoid globals like
the plague, especially in Python). It also nicely encapsulates
the logic that determines whether initialization is required.

The best way I've found to achieve a similar effect in (procedural)
Python defines the function as a closure. For example, here's a
function that keeps track of (and prints out) how many times it
has been called:
def make_spam():
... counter = [0]
... def _():
... counter[0] += 1
... print counter[0]
... return _
...
spam = make_spam()
spam()
1
spam()
2
spam()
3

(Too bad that one can't stick the whole def inside parentheses and
call the function right there, like one can do with JavaScript.)

Another approach would be to stuff the static values in the function's
__dict__. This is less satisfactory than the closure approach
because the "pseudo-static" variable is accessible from outside
the function, but the code is arguably a little more straightforward,
and one does not end up with the now useless one-time closure-generating
function kicking around. Here's another version of the function
above:
def spam():
... d = spam.__dict__
... if not 's' in spam.__dict__:
... spam.s = 1
... print spam.s
... spam.s += 1
...
spam()
1
spam()
2
spam()
3

Besides the external accessibility issue, I don't like explictly
coding the name of the function within the function. Is there any
way to have the function access its own __dict__ without having to
explicitly code its name in its body? E.g., is there some generic
special variable that, within a function, refers to the function
object itself?

I'm sure that there are many other ways to skin this cat, especially
if one starts definining fancy callable classes and whatnot. But
is there a better *simple* way to achieve C-style static locals in
Python that does not require a lot of extra machinery?

TIA!

~K

Search Discussions

  • Chris Rebert at Apr 1, 2010 at 10:51 pm

    On Thu, Apr 1, 2010 at 3:34 PM, kj wrote:
    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. <snip>
    Another approach would be to stuff the static values in the function's
    __dict__. ?This is less satisfactory than the closure approach
    because the "pseudo-static" variable is accessible from outside
    the function, but the code is arguably a little more straightforward,
    and one does not end up with the now useless one-time closure-generating
    function kicking around. ?Here's another version of the function
    above:
    def spam():
    ... ? ? d = spam.__dict__
    ... ? ? if not 's' in spam.__dict__:
    ... ? ? ? ? spam.s = 1
    ... ? ? print spam.s
    ... ? ? spam.s += 1
    ...
    spam()
    1
    spam()
    2
    spam()
    3

    Besides the external accessibility issue, I don't like explictly
    coding the name of the function within the function. ?Is there any
    way to have the function access its own __dict__ without having to
    explicitly code its name in its body? ?E.g., is there some generic
    special variable that, within a function, refers to the function
    object itself?
    Nope. It's been proposed in that past
    (http://www.python.org/dev/peps/pep-3130/), but that proposal was
    rejected.
    I'm sure that there are many other ways to skin this cat, especially
    if one starts definining fancy callable classes and whatnot. ?But
    is there a better *simple* way to achieve C-style static locals in
    Python that does not require a lot of extra machinery?
    You can abuse the default argument value mechanism:

    def spam(s_cell=[1]):
    s = s_cell[0]
    print s
    s_cell[0] += 1

    It's a bit less ugly when the value itself is mutable, which isn't the
    case here with the integer.

    Personally, I hate such abuse with a passion; I think a global
    variable is clearest.

    Cheers,
    Chris
  • Steve Holden at Apr 1, 2010 at 11:10 pm

    Chris Rebert wrote:
    On Thu, Apr 1, 2010 at 3:34 PM, kj wrote:
    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. <snip>
    Another approach would be to stuff the static values in the function's
    __dict__. This is less satisfactory than the closure approach
    because the "pseudo-static" variable is accessible from outside
    the function, but the code is arguably a little more straightforward,
    and one does not end up with the now useless one-time closure-generating
    function kicking around. Here's another version of the function
    above:
    def spam():
    ... d = spam.__dict__
    ... if not 's' in spam.__dict__:
    ... spam.s = 1
    ... print spam.s
    ... spam.s += 1
    ...
    spam()
    1
    spam()
    2
    spam()
    3

    Besides the external accessibility issue, I don't like explictly
    coding the name of the function within the function. Is there any
    way to have the function access its own __dict__ without having to
    explicitly code its name in its body? E.g., is there some generic
    special variable that, within a function, refers to the function
    object itself?
    Nope. It's been proposed in that past
    (http://www.python.org/dev/peps/pep-3130/), but that proposal was
    rejected.
    I'm sure that there are many other ways to skin this cat, especially
    if one starts definining fancy callable classes and whatnot. But
    is there a better *simple* way to achieve C-style static locals in
    Python that does not require a lot of extra machinery?
    You can abuse the default argument value mechanism:

    def spam(s_cell=[1]):
    s = s_cell[0]
    print s
    s_cell[0] += 1

    It's a bit less ugly when the value itself is mutable, which isn't the
    case here with the integer.

    Personally, I hate such abuse with a passion; I think a global
    variable is clearest.
    But the real problem is that the OP is insisting on using purely
    procedural Python when the problem is screaming for an object-oriented
    answer.

    If the function were instead a method then the instance namespace would
    be the logical place to store the required data.

    regards
    Steve
    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/
    Holden Web LLC http://www.holdenweb.com/
    UPCOMING EVENTS: http://holdenweb.eventbrite.com/
  • Patrick Maupin at Apr 1, 2010 at 11:30 pm

    On Apr 1, 6:10?pm, Steve Holden wrote:
    Chris Rebert wrote:
    Personally, I hate such abuse with a passion; I think a global
    variable is clearest.
    But the real problem is that the OP is insisting on using purely
    procedural Python when the problem is screaming for an object-oriented
    answer.

    If the function were instead a method then the instance namespace would
    be the logical place to store the required data.
    In some situations I will use either the default parameter
    initialization Chris mentioned, or the closure mechanism that the OP
    presented, but only on code that I am optimizing for speed (local
    variable lookups, even in nested functions, being much faster than
    global or instance lookups). If it doesn't need to go fast, globals
    or instance variables are the way to go.

    Pat
  • Kj at Apr 2, 2010 at 4:08 pm
    In <mailman.1437.1270163476.23598.python-list at python.org> Steve Holden <steve at holdenweb.com> writes:
    But the real problem is that the OP is insisting on using purely
    procedural Python when the problem is screaming for an object-oriented
    answer.
    My initial reaction to this comment was something like "What? switch
    from procedural to OO just to be able to do some one-time initialization
    of function-private data???" But then, again, since Python allows
    easy mixing of both programming styles, I suppose one could refactor this:

    <procedural>
    def spam(x, y, z):
    try:
    mongo = spam.mongo
    except AttributeError:
    mongo = spam.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, mongo)

    ham = spam(3, 4, 5)
    </procedural>

    into this:

    <OO>
    class _Spam(object):
    @classmethod
    def _(cls, x, y, z):
    try:
    mongo = cls.mongo
    except AttributeError:
    mongo = cls.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, mongo)

    ham = _Spam._(1, 2, 3)
    </OO>


    Is this really more natural or more readable? Hmmm.

    In any case, the first solution does rely on the fact that functions
    are objects, and therefore can have attributes, so even the
    "procedural" version relies on Python's OO model.

    Other responses advocated for global variables. I avoid them in
    general, and doubly so in Python, because I find Python's shenanigans
    with globals mystifying (this business of becoming silently local
    if assigned to); it's one rare instance in which Python out-Perls
    Perl. And yes, I know that the language includes ways to deal with
    this (with the global keyword, etc.) but I find the whole scheme
    is so much "cutting against the grain".

    Thanks for all the replies. There are a lot of good ideas there.
    I'm particular, I'm thankful for the pointers to PEP 3130 (initial
    reaction: maybe I should learn Dutch) and to functools.wraps, and
    for the code snippets.

    ~K
  • Mel at Apr 2, 2010 at 5:35 pm

    kj wrote:

    In <mailman.1437.1270163476.23598.python-list at python.org> Steve Holden
    <steve at holdenweb.com> writes:

    But the real problem is that the OP is insisting on using purely
    procedural Python when the problem is screaming for an object-oriented
    answer.
    My initial reaction to this comment was something like "What? switch
    from procedural to OO just to be able to do some one-time initialization
    of function-private data???"
    Yeah, actually. If the subject had been "Python-style object attributes in
    C?" somebody might have suggested C static variables. An example I wrote
    lately

    volatile static int random_bit ()
    {
    static unsigned short lfsr = 0xACE1u; // seeded LFSR
    // taps: 16 14 13 11; characteristic polynomial: x^16 + x^14 + x^13 +
    x^11 + 1
    lfsr = (lfsr >> 1) ^ (-(lfsr & 1u) & 0xB400u);
    return lfsr & 1;
    } // random_bit

    (excuse is: this was written for cheap execution in an 8-bit processor.)

    This does OK -- but fails the instant I decide that my program needs more
    than one pseudo-random bit stream. Then I have the choice of writing
    several different random_bit functions, or extending random_bit to take a
    pointer to a seeded LFSR provided by the individual caller.

    Refactoring the Python function to a Python class, as you mention later,
    solves the static-access problem, but that solution is just as vulnerable to
    the need-more-than-just-the-one problem as my C function.

    Mel.
  • Duncan Booth at Apr 2, 2010 at 5:53 pm

    kj wrote:

    I suppose one could refactor this:

    <procedural>
    def spam(x, y, z):
    try:
    mongo = spam.mongo
    except AttributeError:
    mongo = spam.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, mongo)

    ham = spam(3, 4, 5)
    </procedural>

    into this:

    <OO>
    class _Spam(object):
    @classmethod
    def _(cls, x, y, z):
    try:
    mongo = cls.mongo
    except AttributeError:
    mongo = cls.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, mongo)

    ham = _Spam._(1, 2, 3)
    </OO>


    Is this really more natural or more readable? Hmmm.
    No, but that's because it is needlessly obfuscated. What's with the weird _
    method? Why use a class method? Why not just create an instance?

    class Spam(object):
    mongo = None
    def __call__(self, x, y, z):
    if self.mongo is None:
    self.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, self.mongo)
    spam = Spam()

    ham = spam(1, 2, 3)


    That's natural and readable.

    There's also another good reason why the class is better than the static
    variable: you can construct multiple different instances with different
    calls to 'heavy_lifting_at_runtime'. e.g. You could write a unit test where
    mongo is initialised to mock_heavy_lifting_at_runtime().
  • Kj at Apr 2, 2010 at 10:59 pm
    In <Xns9D4EC021DC8EAduncanbooth at 127.0.0.1> Duncan Booth <duncan.booth at invalid.invalid> writes:
    class Spam(object):
    mongo = None
    def __call__(self, x, y, z):
    if self.mongo is None:
    self.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, self.mongo)
    spam = Spam()
    ham = spam(1, 2, 3)
    I really like this. Thanks.
    That's natural and readable.
    From reading this thread, and the "(a==b) ? 'Yes' : 'No'" one, the
    inescapable conclusion is that "readability" (like beauty) is very
    much in the eye of the beholder, or, in this case, in the eye of
    Guido.

    ~K
  • Terry Reedy at Apr 3, 2010 at 1:46 am

    On 4/2/2010 6:59 PM, kj wrote:
    In<Xns9D4EC021DC8EAduncanbooth at 127.0.0.1> Duncan Booth<duncan.booth at invalid.invalid> writes:
    class Spam(object):
    mongo = None
    def __call__(self, x, y, z):
    if self.mongo is None:
    self.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, self.mongo)
    Unless one wants the intialization of mongo delayed in case spam is
    never called, it can go in __init__ instead.

    spam = Spam()
    ham = spam(1, 2, 3)
    I really like this. Thanks.
    That's natural and readable.
    From reading this thread, and the "(a==b) ? 'Yes' : 'No'" one, the
    inescapable conclusion is that "readability" (like beauty) is very
    much in the eye of the beholder, or, in this case, in the eye of
    Guido.

    ~K
  • Ethan Furman at Apr 3, 2010 at 2:42 am

    Terry Reedy wrote:
    In<Xns9D4EC021DC8EAduncanbooth at 127.0.0.1> Duncan
    Booth<duncan.booth at invalid.invalid> writes:
    class Spam(object):
    mongo = None
    def __call__(self, x, y, z):
    if self.mongo is None:
    self.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, self.mongo)

    Unless one wants the intialization of mongo delayed in case spam is
    never called, it can go in __init__ instead.
    As a matter of fact, I have an object that is usually not called during
    it's modules use, so I put in __getattr__. Sped the modules load time
    back up to pert near instantaneous. :)

    ~Ethan~
  • Stephen Hansen at Apr 3, 2010 at 3:11 am

    On 2010-04-02 19:42:29 -0700, Ethan Furman said:

    Terry Reedy wrote:
    In<Xns9D4EC021DC8EAduncanbooth at 127.0.0.1> Duncan
    Booth<duncan.booth at invalid.invalid> writes:
    class Spam(object):
    mongo = None
    def __call__(self, x, y, z):
    if self.mongo is None:
    self.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, self.mongo)

    Unless one wants the intialization of mongo delayed in case spam is
    never called, it can go in __init__ instead.
    As a matter of fact, I have an object that is usually not called during
    it's modules use, so I put in __getattr__. Sped the modules load time
    back up to pert near instantaneous. :)

    ~Ethan~
    I prefer:

    class Spam(object):
    def __init__(self):
    self._mondo = None

    def _get_mondo(self):
    if self._mondo is None:
    self._mondo = heavy_lifting_at_runtime()
    return self._mondo

    mondo = property(_get_mondo)

    def __call__(self, x, y, z):
    return frobnicate(x,y,z, self.mondo)

    I don't know if properties are really faster or slower then a
    __getattr__, but I find them a lot cleaner if I want to delay some
    calculation until needed like that.

    --
    --S

    ... p.s: change the ".invalid" to ".com" in email address to reply privately.
  • Patrick Maupin at Apr 3, 2010 at 3:24 am

    On Apr 2, 10:11?pm, Stephen Hansen wrote:
    I don't know if properties are really faster or slower then a
    __getattr__, but I find them a lot cleaner if I want to delay some
    calculation until needed like that.
    Well, the relative speed of properties vs. __getattr__ can become
    irrelevant in at least two ways:

    1) If the __getattr__ only calculates the value one time and then
    stuffs it into the instance dictionary, now you are really comparing
    the relative speed of properties vs. lookup of an attribute in the
    instance dict. If you're at all concerned about speed, I think there
    is a clear winner here.

    2) There is a single __getattr__ function, vs. one property for every
    attribute that needs a property. In cases where you can somehow
    easily compute the attribute names as well as the attribute values,
    __getattr__ can be a *lot* less code than defining dozens of
    properties.

    But you're absolutely right that, in many cases, property is the best
    way to go for readability (especially if the property is read-only and
    you're using a recent enough python to use decorators).

    Regards,
    Pat
  • Stephen Hansen at Apr 3, 2010 at 7:03 pm

    On 2010-04-02 20:24:46 -0700, Patrick Maupin said:
    On Apr 2, 10:11?pm, Stephen Hansen wrote:

    I don't know if properties are really faster or slower then a
    __getattr__, but I find them a lot cleaner if I want to delay some
    calculation until needed like that.
    Well, the relative speed of properties vs. __getattr__ can become
    irrelevant in at least two ways:

    1) If the __getattr__ only calculates the value one time and then
    stuffs it into the instance dictionary, now you are really comparing
    the relative speed of properties vs. lookup of an attribute in the
    instance dict. If you're at all concerned about speed, I think there
    is a clear winner here.
    I concede it would probably be notably faster, but there's a big
    difference between "at all concerned about speed" and "optimizing a
    profiled bottleneck".

    The speed difference between direct attribute lookup and properties may
    be notable, but that doesn't make a clear winner here. Now that I have
    (with either method) optimized the expensive value-calculation
    operation such that it only happens on-demand and once, I now have to
    weigh further optimization.

    Is the difference in speed between a standard attribute lookup and a
    property fetch worth losing the clarity the property brings over the
    __getattr__ solution, especially considering the __getattr__ creates a
    fuzzy 'sometimes this code is responsible, othertimes the dict is'
    situation that someone may down the road miss in maintenance?

    For me, usually not-- unless profiling pushes me to reconsider. But
    everyone makes these calls differently.
    2) There is a single __getattr__ function, vs. one property for every
    attribute that needs a property. In cases where you can somehow
    easily compute the attribute names as well as the attribute values,
    __getattr__ can be a *lot* less code than defining dozens of
    properties.
    I don't really mind a lot of properties, if they're simple. Then again,
    I often prefer regular ol' attributes where possible :) However, if I'm
    doing a dispatching sort of mechanism, or a situation where the "name"
    isn't something static, set in stone or pre-defined-- then certainly,
    __getattr__ is a fine solution. I don't mind it where its the clearest
    way to accomplish a goal.

    --
    --S

    ... p.s: change the ".invalid" to ".com" in email address to reply privately.
  • Ethan Furman at Apr 2, 2010 at 6:21 pm

    kj wrote:
    <OO>
    class _Spam(object):
    @classmethod
    def _(cls, x, y, z):
    try:
    mongo = cls.mongo
    except AttributeError:
    mongo = cls.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, mongo)

    ham = _Spam._(1, 2, 3)
    </OO>


    Is this really more natural or more readable? Hmmm.
    For this type of situation, my preference would be:

    class spam(object):
    def __call__(self, x, y, z):
    try:
    mongo = self.mongo
    except AttributeError:
    mongo = self.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, mongo)
    spam = spam()


    No extra objects, out-of-place underscores, etc.

    ~Ethan~
  • Patrick Maupin at Apr 2, 2010 at 6:39 pm

    On Apr 2, 1:21?pm, Ethan Furman wrote:
    For this type of situation, my preference would be:

    class spam(object):
    ? ? ?def __call__(self, x, y, z):
    ? ? ? ? ?try:
    ? ? ? ? ? ? ?mongo = self.mongo
    ? ? ? ? ?except AttributeError:
    ? ? ? ? ? ? ?mongo = self.mongo = heavy_lifting_at_runtime()
    ? ? ? ? ?return frobnicate(x, y, z, mongo)
    spam = spam()

    No extra objects, out-of-place underscores, etc.

    ~Ethan~
    Well, I'm not a big fan of unnecessary try/except, so I would at least
    change it to:

    class spam(object):
    def __getattr__(self, name):
    if name != 'mongo':
    raise AttributeError
    self.mongo = heavy_lifting_at_runtime()
    return self.mongo
    def __call__(self, x, y, z):
    return frobnicate(x, y, z, self.mongo)
    spam = spam()

    Regards,
    Pat
  • Ethan Furman at Apr 2, 2010 at 7:38 pm

    Patrick Maupin wrote:
    On Apr 2, 1:21 pm, Ethan Furman wrote:
    For this type of situation, my preference would be:

    class spam(object):
    def __call__(self, x, y, z):
    try:
    mongo = self.mongo
    except AttributeError:
    mongo = self.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, mongo)
    spam = spam()

    No extra objects, out-of-place underscores, etc.

    ~Ethan~
    Well, I'm not a big fan of unnecessary try/except, so I would at least
    change it to:

    class spam(object):
    def __getattr__(self, name):
    if name != 'mongo':
    raise AttributeError
    self.mongo = heavy_lifting_at_runtime()
    return self.mongo
    def __call__(self, x, y, z):
    return frobnicate(x, y, z, self.mongo)
    spam = spam()

    Regards,
    Pat

    Sounds like a personal preference issue, rather than a necessary /
    unnecessary issue -- after all, if you call that function a thousand
    times, only once is mongo not defined... clearly the exception. ;)

    ~Ethan~
  • Patrick Maupin at Apr 2, 2010 at 7:39 pm

    On Apr 2, 2:38?pm, Ethan Furman wrote:
    Patrick Maupin wrote:
    On Apr 2, 1:21 pm, Ethan Furman wrote:
    For this type of situation, my preference would be:
    class spam(object):
    ? ? ?def __call__(self, x, y, z):
    ? ? ? ? ?try:
    ? ? ? ? ? ? ?mongo = self.mongo
    ? ? ? ? ?except AttributeError:
    ? ? ? ? ? ? ?mongo = self.mongo = heavy_lifting_at_runtime()
    ? ? ? ? ?return frobnicate(x, y, z, mongo)
    spam = spam()
    No extra objects, out-of-place underscores, etc.
    ~Ethan~
    Well, I'm not a big fan of unnecessary try/except, so I would at least
    change it to:
    class spam(object):
    ? ? ?def __getattr__(self, name):
    ? ? ? ? ?if name != 'mongo':
    ? ? ? ? ? ? ?raise AttributeError
    ? ? ? ? ?self.mongo = heavy_lifting_at_runtime()
    ? ? ? ? ?return self.mongo
    ? ? ?def __call__(self, x, y, z):
    ? ? ? ? ?return frobnicate(x, y, z, self.mongo)
    spam = spam()
    Regards,
    Pat
    Sounds like a personal preference issue, rather than a necessary /
    unnecessary issue -- after all, if you call that function a thousand
    times, only once is mongo not defined... clearly the exception. ?;)

    ~Ethan~
    Well, I think the whole discussion has basically been about personal
    preference. OTOH, but if you call the function a few million times,
    you might find the cost of try/except to be something that you would
    rather not incur -- it might become a performance issue rather than a
    personal choice issue. On the other OTHER hand, if you call the
    function a few billion times, performance weighs more heavily in favor
    of the closure approach rather than the object approach, since local
    variable lookup is so much cheaper.

    Regards,
    Pat
  • Ethan Furman at Apr 2, 2010 at 8:33 pm
    Patrick Maupin wrote:

    [snippage]
    Well, I think the whole discussion has basically been about personal
    preference. OTOH, but if you call the function a few million times,
    you might find the cost of try/except to be something that you would
    rather not incur -- it might become a performance issue rather than a
    personal choice issue. On the other OTHER hand, if you call the
    function a few billion times, performance weighs more heavily in favor
    of the closure approach rather than the object approach, since local
    variable lookup is so much cheaper.
    Indeed. I was surprised to find your __getattr__ approach faster than
    the try/except approach (about 20% on my machine). I'll have to think
    about that for future situations like this.

    My main point, though, was using __call__, and not some weird _ method. ;)

    ~Ethan~
  • Patrick Maupin at Apr 2, 2010 at 8:42 pm

    On Apr 2, 3:33?pm, Ethan Furman wrote:
    My main point, though, was using __call__, and not some weird _ method. ?;)
    Yes, __call__ is good. In general, not naming things that don't need
    to be named is good (but if you have too many of them to keep track
    of, then, obviously, they need to be named, hence named tuples).

    But I didn't need to address that, since you already did :-)
  • Steven D'Aprano at Apr 2, 2010 at 11:57 pm

    On Fri, 02 Apr 2010 12:39:16 -0700, Patrick Maupin wrote:

    On Apr 2, 2:38?pm, Ethan Furman wrote:
    [...]
    Sounds like a personal preference issue, rather than a necessary /
    unnecessary issue -- after all, if you call that function a thousand
    times, only once is mongo not defined... clearly the exception. ?;)

    ~Ethan~
    Well, I think the whole discussion has basically been about personal
    preference. OTOH, but if you call the function a few million times, you
    might find the cost of try/except to be something that you would rather
    not incur -- it might become a performance issue rather than a personal
    choice issue.

    The cost of a try...except is *very* low -- about the same as a pass
    statement:
    from timeit import Timer
    t1 = Timer("pass", "")
    t2 = Timer("try:\n pass\nexcept Exception:\n pass", "")
    min(t2.repeat())/min(t1.repeat())
    1.9227982449955801


    Actually catching the exception, on the other hand, is quite expensive:
    t1 = Timer("len('')", "")
    t2 = Timer("try:\n len(0)\nexcept Exception:\n pass", "")
    min(t2.repeat())/min(t1.repeat())
    10.598482743564809


    The heuristic I use is, if I expect the try block to raise an exception
    more than about one time in ten, I change to an explicit test. In this
    case, since the exception should only be raised once, and then never
    again, I would use a try...except block.



    --
    Steven
  • Patrick Maupin at Apr 3, 2010 at 12:25 am

    On Apr 2, 6:57?pm, Steven D'Aprano <st... at REMOVE-THIS- cybersource.com.au> wrote:
    On Fri, 02 Apr 2010 12:39:16 -0700, Patrick Maupin wrote:
    On Apr 2, 2:38?pm, Ethan Furman wrote:
    [...]
    Sounds like a personal preference issue, rather than a necessary /
    unnecessary issue -- after all, if you call that function a thousand
    times, only once is mongo not defined... clearly the exception. ?;)
    ~Ethan~
    Well, I think the whole discussion has basically been about personal
    preference. ?OTOH, but if you call the function a few million times, you
    might find the cost of try/except to be something that you would rather
    not incur -- it might become a performance issue rather than a personal
    choice issue.
    The cost of a try...except is *very* low -- about the same as a pass
    statement:
    Agreed. In the example above, if frobnicate() is a null function, the
    try/except adds about 5% to execution time on my machine. If I were
    really worried about execution time, I would use a closure *for this
    particular example* as I mentioned elsewhere. However, the cost of
    the try/except is not zero, and when I have something I prefer looking
    at (the __getattr__ doesn't clutter up the main-line execution with
    conditionals for stuff that only gets used once at initialization)
    that is always known to be cheaper in execution, that's what I use. I
    suppose some people might not like looking at the __getattr__, but
    this is a memoization technique I use quite often, so I find it
    idiomatic.

    Regards,
    Pat
  • Ethan Furman at Apr 3, 2010 at 2:48 am

    Steven D'Aprano wrote:
    On Fri, 02 Apr 2010 12:39:16 -0700, Patrick Maupin wrote:

    On Apr 2, 2:38 pm, Ethan Furman wrote:
    [...]
    Sounds like a personal preference issue, rather than a necessary /
    unnecessary issue -- after all, if you call that function a thousand
    times, only once is mongo not defined... clearly the exception. ;)

    ~Ethan~
    Well, I think the whole discussion has basically been about personal
    preference. OTOH, but if you call the function a few million times, you
    might find the cost of try/except to be something that you would rather
    not incur -- it might become a performance issue rather than a personal
    choice issue.


    The cost of a try...except is *very* low -- about the same as a pass
    statement:

    from timeit import Timer
    t1 = Timer("pass", "")
    t2 = Timer("try:\n pass\nexcept Exception:\n pass", "")
    min(t2.repeat())/min(t1.repeat())
    1.9227982449955801


    Actually catching the exception, on the other hand, is quite expensive:

    t1 = Timer("len('')", "")
    t2 = Timer("try:\n len(0)\nexcept Exception:\n pass", "")
    min(t2.repeat())/min(t1.repeat())
    10.598482743564809


    The heuristic I use is, if I expect the try block to raise an exception
    more than about one time in ten, I change to an explicit test. In this
    case, since the exception should only be raised once, and then never
    again, I would use a try...except block.
    That was my reasoning as well, but when I timed it for one million runs
    (so 1 instantiation, 999,999 simple calls), the __getattr__ time was .5
    seconds, the try...execpt block was .6; at ten million it was 5 and 6.

    At those rates, personal preference takes over, at least for me.

    ~Ethan~
  • Steven D'Aprano at Apr 3, 2010 at 6:17 am

    On Fri, 02 Apr 2010 19:48:59 -0700, Ethan Furman wrote:

    The heuristic I use is, if I expect the try block to raise an exception
    more than about one time in ten, I change to an explicit test. In this
    case, since the exception should only be raised once, and then never
    again, I would use a try...except block.
    That was my reasoning as well, but when I timed it for one million runs
    (so 1 instantiation, 999,999 simple calls), the __getattr__ time was .5
    seconds, the try...execpt block was .6; at ten million it was 5 and 6.
    Care to share your timing code? Not that I don't trust your results, but
    timings are very sensitive to the exact thing you do, and I'd like to see
    what that is.




    --
    Steven
  • Ethan Furman at Apr 3, 2010 at 6:44 am

    Steven D'Aprano wrote:
    On Fri, 02 Apr 2010 19:48:59 -0700, Ethan Furman wrote:

    The heuristic I use is, if I expect the try block to raise an exception
    more than about one time in ten, I change to an explicit test. In this
    case, since the exception should only be raised once, and then never
    again, I would use a try...except block.
    That was my reasoning as well, but when I timed it for one million runs
    (so 1 instantiation, 999,999 simple calls), the __getattr__ time was .5
    seconds, the try...execpt block was .6; at ten million it was 5 and 6.

    Care to share your timing code? Not that I don't trust your results, but
    timings are very sensitive to the exact thing you do, and I'd like to see
    what that is.
    Happy to do so -- if I made a mistake I'd like to know about it and learn.

    It'll have to wait two days 'til I get back to work, though... I'll post
    it asap.

    ~Ethan~
  • Ethan Furman at Apr 5, 2010 at 11:50 pm

    Ethan Furman wrote:
    Steven D'Aprano wrote:
    On Fri, 02 Apr 2010 19:48:59 -0700, Ethan Furman wrote:

    The heuristic I use is, if I expect the try block to raise an exception
    more than about one time in ten, I change to an explicit test. In this
    case, since the exception should only be raised once, and then never
    again, I would use a try...except block.
    That was my reasoning as well, but when I timed it for one million runs
    (so 1 instantiation, 999,999 simple calls), the __getattr__ time was .5
    seconds, the try...execpt block was .6; at ten million it was 5 and 6.

    Care to share your timing code? Not that I don't trust your results,
    but timings are very sensitive to the exact thing you do, and I'd like
    to see what that is.
    Happy to do so -- if I made a mistake I'd like to know about it and learn.

    It'll have to wait two days 'til I get back to work, though... I'll post
    it asap.
    Well, so much for asap, but here's what I used (with one correction: in
    the 'if' code I had forgotten to actually reference the missing
    attribute, so the __getattr__ look up never happened; now the
    try...except block is /slightly/ faster, as opposed to 20% slower).


    class spam_except(object):
    def __call__(self, x, y, z):
    try:
    mongo = self.mongo
    except AttributeError:
    mongo = self.mongo = 1
    return

    class spam_if(object):
    def __getattr__(self, name):
    if name != 'mongo':
    raise AttributeError
    self.mongo = 1
    return self.mongo
    def __call__(self, x, y, z):
    self.mongo # didn't have this line before. d'oh!
    return


    --> timeit.Timer('spammer(1,2,3)','from spam import spam_except;
    spammer=spam_except()').timeit()
    0.65764130543749388

    --> timeit.Timer('spammer(1,2,3)','from spam import spam_if;
    spammer=spam_if()').timeit()
    0.66972877235545525

    ~Ethan~
  • Patrick Maupin at Apr 6, 2010 at 12:46 am
    On Apr 5, 6:50?pm, Ethan Furman wrote:

    (Posted some code with a timeit...)

    Well, I'm not going to debug this, but with the *original* thing you
    posted, and the thing I posted, with a call and everything (more
    realistic scenario), the exception version seems slower on my machine:

    #!/usr/bin/env python

    import timeit

    def frobnicate(a,b,c,d):
    pass

    def heavy_lifting_at_runtime():
    print 'heavy'

    class spam_except(object):
    def __call__(self, x, y, z):
    try:
    mongo = self.mongo
    except AttributeError:
    mongo = self.mongo = heavy_lifting_at_runtime()
    return frobnicate(x, y, z, mongo)
    se = spam_except()


    class spam_if(object):
    def __getattr__(self, name):
    if name != 'mongo':
    raise AttributeError
    self.mongo = heavy_lifting_at_runtime()
    return self.mongo
    def __call__(self, x, y, z):
    return frobnicate(x, y, z, self.mongo)
    si = spam_if()

    tse = timeit.Timer('se(1,2,3)', "from __main__ import se")
    tsi = timeit.Timer('si(1,2,3)', "from __main__ import si")

    for i in range(5):
    ve = tse.timeit(10000000)
    vi = tsi.timeit(10000000)
    print ve, vi, '%.1f' % ((ve-vi) / vi * 100)

    ------

    heavy
    heavy
    5.45695090294 5.10844397545 6.8
    5.43381404877 5.01345705986 8.4
    5.42474508286 5.02641201019 7.9
    5.40713405609 5.04178905487 7.2
    5.38063693047 4.96194696426 8.4

    The output indicates that the exception one is, on average, around
    7.5% slower.

    Regards,
    Pat
  • Steven D'Aprano at Apr 2, 2010 at 6:40 pm

    On Fri, 02 Apr 2010 16:08:42 +0000, kj wrote:

    Other responses advocated for global variables. I avoid them in
    general,
    In general this is wise, but remember that because Python globals are not
    globally global, but local to a single module, they're safer than globals
    in other languages. Still, it's better to avoid them when possible.

    and doubly so in Python, because I find Python's shenanigans
    with globals mystifying (this business of becoming silently local if
    assigned to);
    Globals don't become local when assigned to. You can shadow a global with
    a local of the same name, but the global remains untouched:
    myglobal = 42
    def test():
    ... myglobal = 0 # shadow the global with a new local
    ...
    test()
    myglobal
    42

    I find this behaviour perfectly natural, and desirable: it means I can
    assign to locals without worrying whether or not I'm about to stomp all
    over a global and destroy it. The alternative behaviour would be
    disastrous:
    def f(x): return x+1
    ...
    def test():
    ... f = 'spam'
    ...
    test()
    f(2) # this doesn't happen
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    TypeError: 'str' object is not callable



    --
    Steven
  • Terry Reedy at Apr 1, 2010 at 11:28 pm

    On 4/1/2010 6:34 PM, kj wrote:

    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. For example:

    int foo(int x, int y, int z) {

    static int first_time = TRUE;
    static Mongo *mongo;
    if (first_time) {
    mongo = heavy_lifting_at_runtime();
    first_time = FALSE;
    }

    return frobnicate(mongo, x, y, z);
    Global var or class or closure such as below (obviously untested ;=):

    make_foo()
    mongo = heavy_lifting_at_runtime();
    def _(x,y,z):
    return frobnicate(mongo, x, y, z)
    return _
    foo = make_foo
    del make_foo # to make sure it is *never* called again ;

    Now you only have foo with a hard-to-access private object and no
    first_time checks when you call it.

    Terry Jan Reedy
  • Steve Holden at Apr 1, 2010 at 11:48 pm

    Terry Reedy wrote:
    On 4/1/2010 6:34 PM, kj wrote:


    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. For example:

    int foo(int x, int y, int z) {

    static int first_time = TRUE;
    static Mongo *mongo;
    if (first_time) {
    mongo = heavy_lifting_at_runtime();
    first_time = FALSE;
    }

    return frobnicate(mongo, x, y, z);
    Global var or class or closure such as below (obviously untested ;=):

    make_foo()
    mongo = heavy_lifting_at_runtime();
    def _(x,y,z):
    return frobnicate(mongo, x, y, z)
    return _
    foo = make_foo
    I suspect you mean

    foo = make_foo()
    del make_foo # to make sure it is *never* called again ;

    Now you only have foo with a hard-to-access private object and no
    first_time checks when you call it.

    Terry Jan Reedy
    I don't think I'd ever want to use such an obscure technique in a
    program. You might want to consider using functools.wraps to make sure
    that the foo function looks right.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    See PyCon Talks from Atlanta 2010 http://pycon.blip.tv/
    Holden Web LLC http://www.holdenweb.com/
    UPCOMING EVENTS: http://holdenweb.eventbrite.com/
  • Albert van der Horst at Apr 15, 2010 at 2:38 pm
    In article <mailman.1441.1270165718.23598.python-list at python.org>,
    Steve Holden wrote:
    Terry Reedy wrote:
    On 4/1/2010 6:34 PM, kj wrote:


    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. For example:

    int foo(int x, int y, int z) {

    static int first_time = TRUE;
    static Mongo *mongo;
    if (first_time) {
    mongo = heavy_lifting_at_runtime();
    first_time = FALSE;
    }

    return frobnicate(mongo, x, y, z);
    Global var or class or closure such as below (obviously untested ;=):

    make_foo()
    mongo = heavy_lifting_at_runtime();
    def _(x,y,z):
    return frobnicate(mongo, x, y, z)
    return _
    foo = make_foo
    I suspect you mean

    foo = make_foo()
    del make_foo # to make sure it is *never* called again ;

    Now you only have foo with a hard-to-access private object and no
    first_time checks when you call it.

    Terry Jan Reedy
    I don't think I'd ever want to use such an obscure technique in a
    program. You might want to consider using functools.wraps to make sure
    that the foo function looks right.
    Imagine that heavy_lifting is only ever used here and uses 4 Gbyte of core.
    Suddenly deleting those function objects seems the right thing to do,
    instead of an obscure technique.
    (I'm not sure the Python compiler could take advantage of this,
    I know I could in my Forth compiler, under circumstances.)
    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Groetjes Albert

    --
    --
    Albert van der Horst, UTRECHT,THE NETHERLANDS
    Economic growth -- being exponential -- ultimately falters.
    albert at spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst
  • Alf P. Steinbach at Apr 2, 2010 at 1:37 am

    * kj:
    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. For example:

    int foo(int x, int y, int z) {

    static int first_time = TRUE;
    static Mongo *mongo;
    if (first_time) {
    mongo = heavy_lifting_at_runtime();
    first_time = FALSE;
    }

    return frobnicate(mongo, x, y, z);
    }

    In this case, the static variable mongo is initialized only once
    (at most).

    What I like most about this is that it obviates the need for a
    global variable to hold the persistent value (I avoid globals like
    the plague, especially in Python). It also nicely encapsulates
    the logic that determines whether initialization is required.
    In C++ you just write

    int foo( int x, int y, int z )
    {
    static Mongo* const mongo = heavy_lifting_at_runtime();
    return frobnicate( mongo, x, y, z );
    }


    The best way I've found to achieve a similar effect in (procedural)
    Python defines the function as a closure. For example, here's a
    function that keeps track of (and prints out) how many times it
    has been called:
    def make_spam():
    ... counter = [0]
    ... def _():
    ... counter[0] += 1
    ... print counter[0]
    ... return _
    ...
    spam = make_spam()
    spam()
    1
    spam()
    2
    spam()
    3

    (Too bad that one can't stick the whole def inside parentheses and
    call the function right there, like one can do with JavaScript.)
    Off the cuff, Py3:

    class Spam:
    def __init__( self ):
    self._counter = 0

    def __call__( self ):
    self._counter += 1
    print( counter )

    spam = Spam()
    spam()
    spam()
    spam()


    [snip]
    I'm sure that there are many other ways to skin this cat, especially
    if one starts definining fancy callable classes and whatnot.
    As I see it it's the closure that's fancy, and the class that's simple and direct.

    But
    is there a better *simple* way to achieve C-style static locals in
    Python that does not require a lot of extra machinery?
    If you often need this functionality you might consider a general decorator that
    supplies the function with a self argument, e.g. like this:


    <example>
    #Py3

    class Object: pass

    def static_initialization( init_func ):
    def self_aware( f ):
    def wrapped( *args, **kwargs ):
    return f( f, *args, **kwargs )
    init_func( f )
    return wrapped
    o = Object()
    o.body = self_aware
    return o


    # Example usage:

    @static_initialization
    def spam( self ):
    self.counter = 0

    @spam.body
    def spam( self ):
    self.counter += 1
    print( self.counter )

    spam()
    spam()
    spam()
    </example>


    But as mentioned, a class is (at least IMHO) simpler and more direct.



    Cheers & hth.,

    - Alf (department of disingenious solutions)
  • Paul Rubin at Apr 2, 2010 at 3:32 am

    kj <no.email at please.post> writes:
    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. For example:

    int foo(int x, int y, int z) {
    static int first_time = TRUE;
    static Mongo *mongo;
    if (first_time) { ...

    Here are some cheesy ways.

    1. Put an attribute onto the function name:

    def foo(x, y, z):
    if foo.first_time:
    foo.mongo = heavy_lifting_at_runtime()
    foo.first_time = False
    ...
    foo.first_time = True

    2. Use a mutable keyword parameter:

    def foo(x, y, z, wrapped_mongo=[]):
    if len(wrapped_mongo) == 0:
    wrapped_mongo.append(heavy_lifting_at_runtime())
    mongo = wrapped_mongo[0]
    ...

    3. Streamline the first method a little:

    def foo(x, y, z):
    if len(foo.wrapped_mongo == 0):
    foo.wrapped_mongo.append(heavy_lifting_at_runtime())
    mongo = foo.wrapped_mongo[0]
    ...
    foo.wrapped_mongo = []

    All of these of course don't give as good encapsulation as one might
    like.
  • Paul McGuire at Apr 2, 2010 at 5:28 pm

    On Apr 1, 5:34?pm, kj wrote:
    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. ?For example:
    Here is a decorator to make a function self-aware, giving it a "this"
    variable that points to itself, which you could then initialize from
    outside with static flags or values:

    from functools import wraps

    def self_aware(fn):
    @wraps(fn)
    def fn_(*args):
    return fn(*args)
    fn_.__globals__["this"] = fn_
    return fn_

    @self_aware
    def foo():
    this.counter += 1
    print this.counter

    foo.counter = 0

    foo()
    foo()
    foo()


    Prints:

    1
    2
    3

    -- Paul
  • Terry Reedy at Apr 3, 2010 at 1:48 am

    On 4/2/2010 1:28 PM, Paul McGuire wrote:
    On Apr 1, 5:34 pm, kjwrote:
    When coding C I have often found static local variables useful for
    doing once-only run-time initializations. For example:
    Here is a decorator to make a function self-aware, giving it a "this"
    variable that points to itself, which you could then initialize from
    outside with static flags or values:

    from functools import wraps

    def self_aware(fn):
    @wraps(fn)
    def fn_(*args):
    return fn(*args)
    fn_.__globals__["this"] = fn_
    return fn_
    In 3.1, at least, the wrapper is not needed.

    def self_aware(fn):
    fn.__globals__["this"] = fn
    return fn

    Acts the same
    @self_aware
    def foo():
    this.counter += 1
    print this.counter

    foo.counter = 0
    Explicit and separate initialization is a pain. This should be in a
    closure or class.
    foo()
    foo()
    foo()
    Prints:
    1
    2
    3
    However, either way, the __globals__ attribute *is* the globals dict,
    not a copy, so one has
    this
    <function foo at 0x00F5F5D0>

    Wrapping a second function would overwrite the global binding.

    Terry Jan Reedy
  • John Nagle at Apr 4, 2010 at 6:57 pm

    kj wrote:
    When coding C I have often found static local variables useful for
    doing once-only run-time initializations.
    If you want functions with state, use an object. That's what they're
    for. Don't muck with the internal representation of functions.

    John Nagle
  • Patrick Maupin at Apr 4, 2010 at 8:35 pm

    On Apr 4, 1:57?pm, John Nagle wrote:
    ? ? If you want functions with state, use an object. That's what they're
    for. ?Don't muck with the internal representation of functions.
    While "Don't muck with the internal representation of functions" is
    excellent advice over 99% of the time, it is also true that it is
    often possible, sometimes even encouraged, to have "functions with
    state."

    This is done without "mucking" and without explicitly declaring a
    class or a class instance. See, e.g. closures and generator
    functions.

    Regards,
    Pat
  • Lee Harr at Apr 5, 2010 at 9:04 pm

    Another approach would be to stuff the static values in the function's
    __dict__.
    That's how I did it when I wanted something similar.

    I created this decorator:


    def static(**kw):
    ??? '''
    ??? Used to create a decorator function that will add an
    ??? attribute to a function and initialize it.

    ???>>> @static(foo=5)
    ??? ... def bar():
    ??? ...???? print bar.foo
    ??? ...???? bar.foo += 1
    ??? ...
    ???>>> bar()
    ??? 5
    ???>>> bar()
    ??? 6
    ??? '''

    ??? def decorator(f):
    ??????? f.__dict__.update(kw)
    ??????? return f
    ??? return decorator


    _________________________________________________________________
    Hotmail: Trusted email with Microsoft?s powerful SPAM protection.
    https://signup.live.com/signup.aspx?id`969

Related Discussions

People

Translate

site design / logo © 2022 Grokbase