FAQ
Here's a proposed Q&A for the FAQ based on a couple recent
threads. Appropriate comments appreciated

X.Y: Why doesn't Python have macros like in Lisp or Scheme?

Before answering that, a clarification on what 'macro' means.
A Lisp macro is a way of modifying code when that code is first
defined. It can rearrange the structure of the code, and add and
remove parts of it. Unlike C's #define macro language, Lisp
macros understand the structure of the expression and the
context (or "closure") in which it is found.

Here's a simple example of what a macro might do for Python.
A complaint about Python is that constructors for simple data
types require typing the parameter names three times, as in

class Country:
def __init__(self, name, capitol):
self.name = name
self.capitol = capitol

Suppose you could create a new way to create functions, which
looked at the argument list and automatically set the corresponding
member. This would let you replace the above with something
more like

class Country:
def __init__(self, name, capitol) [arg_assign]:
pass

where the 'arg_assign' would use the function definition
and code to make a new function used in place of the
existing one and with the 'self.name = name' and
'self.capitol = capitol' code automatically added to
the front of the code.

This is a very powerful technique and especially nice for
high-performance code where the macro can optimize
code before it is actually evaluated. Macros work well in
Lisp or Scheme, where the code is the data, all written
pretty much as a parse tree.

The first problem with implementing a macro system in Python
is coming up with a readable syntax; that is, how it's "spelled."
The example shown above, using the [] after the function
definition, was taken from a proposed syntax for adding
function modifiers like 'staticmethod' and 'classmethod' to
Python.

The second problem is that Python's code blocks do not
store the original parse tree, so there's nothing to manipulate.
If you use 'dir()' or the inspect module you'll see some fields
like the list of argument names and the default value to use
for the fields. Nowhere will you find the original parse tree.
(The C implementation of Python does give access to the
Python byte code, but this is not part of the language
specification and Java implementation doesn't support it.)

These problems can be resolved, and Dylan shows that
an infix-based/Algol-like language can support macros
similar to those used in Lisp.

The deeper question is, should Python include macros?

People with a Lisp background might have a problem
understanding this viewpoint. Macros add flexibility, and
more flexibilty lets programmers be more productive; or
so the argument goes.

Those against including macros in Python point out that
Python has a different focus. It has a broad target audience.
Beginners can pick it up pretty easily yet advanced
developers also enjoy working in the language. New features,
like new builtin functions, additional core types, expanding
the standard libraries, list comprensions, metaclasses and
deriving from object, and macros, all make it harder to
learn how to use Python well and shifts the balance more
towards advanced programmers.

More importantly, the detractors -- including those
with plenty of experience using macros in Lisp -- argue that
macros cause dialects to form. Macros can modify other code to
make it fit the problem better, while functions only use other
code but make no modifications. This makes them very
powerful but means that understanding a section of code
requires also knowing about any macros which might use the
code. In an extreme case which wouldn't be used in real
projects, every * could be replaced with a +.

(As an aside, some proponents argue that macros and
functions are essentially the same. Alex Martelli made
an interesting observation about one difference: it's often
worthwhile to turn a set of code into a function even if it
is only called from one place, because it helps simplify
the code into smaller chucks which are easier to understand.
Macros, on the other hand, should almost never be used
unless they are used many times.)

With only one or a small group of people working together
on a project there is little problem. Macros help in developing
idioms specific to the problem and group. When groups
share code, they also share idioms, and anyone who has had
to deal with UK vs. US English knows the effect idioms have
in mututal understanding.

Those against macros say their benefits do not outweigh
the likelihood that the Python community will be broken up
into distinct subsets, where people from one group find it
hard to use code from another.

The proponents for macros say this has not been a problem
in Lisp. Macros are misused, but so are functions and classes.
The same community feedback which encourages people to
use proper coding styles also encourages people to use macros
correctly. Those against reassert that not only can it be a
problem but it's happened to them personally.

At this point the discussion usually breaks down, with the
Lispers pulling out the old canard about Python (and every
other language) being a half-complete, poor reimplementation
of Lisp, while Pythonistas trot out the expression "Lisp is like
a ball of mud. You can add any amount of mud to it and it
still looks like a ball of mud."

Those interested in actual language improvements then ask for
concrete examples of how macros might be used for Python.
So far, the examples given are ones which could equally well
be done in Python using metaclasses, properties, and other
existing mechanisms. For example, the arg_assign example
given above could, I think, be done with a modified__new__.

The problem at this point seems to be that the two languages
have different enough object models and accepted best
practices that it's hard for an intuition based on Lisp to
know immediately how to apply that intuition to Python.

If you want to encourage Python to have macros, you must
therefore propose a syntax for it and give solid examples of
what the result would look like when applied to real Python
code. Simply stating that macros are needed is not good
enough nor are examples showing how macros are useful
for Lisp code. (Besides, believe it or not but some of us
don't know enough Lisp to follow your examples ;)

In the meanwhile, if you want to experiment with manipulating
code trees, you can use decompyle to turn byte code back
into Python source and the compiler module to turn source into
an AST and thence into byte code.

Andrew Dalke
dalke at dalkescientific.com

Search Discussions

  • Michele Simionato at Aug 23, 2003 at 6:50 am
    "Andrew Dalke" <adalke at mindspring.com> wrote in message news:<Dsz1b.3409$Ej6.113 at newsread4.news.pas.earthlink.net>...
    Here's a proposed Q&A for the FAQ based on a couple recent
    threads. Appropriate comments appreciated

    X.Y: Why doesn't Python have macros like in Lisp or Scheme?
    This is all very interesting and I do agree that there shouuld be a
    FAQ about macros, nevertheless I don't see your message as suitable
    for a FAQ. It is ways too verbose. I mean, if one asks for a macro,
    s/he already knows what a macro is. The FAQ should be much
    shorter. What you wrote is more useful and interesting for
    Pythonistas than from Lispers.
    I think your document should be made available on the Web somewhere,
    but not in the FAQ. Of course the FAQ could and should give a pointer
    to it. Maybe lispers would be helped from a page describing how
    some of the things made in macros in Lisp can be made in Python.
    This page should be referred in the FAQ too. It is clear, however,
    that many things involving macros cannot just be done in Python
    (I mean, unless implementing a custom macro interpreter in Python,
    which actually I wrote few months ago as a proof of concept (google
    on c.l.py for simionato macros, first hit) and which I DO NOT use ;).

    Michele Simionato, Ph. D.
    MicheleSimionato at libero.it
    http://www.phyast.pitt.edu/~micheles
    --- Currently looking for a job ---
  • John J. Lee at Aug 23, 2003 at 4:10 pm

    mis6 at pitt.edu (Michele Simionato) writes:

    "Andrew Dalke" <adalke at mindspring.com> wrote in message news:<Dsz1b.3409$Ej6.113 at newsread4.news.pas.earthlink.net>...
    Here's a proposed Q&A for the FAQ based on a couple recent
    threads. Appropriate comments appreciated

    X.Y: Why doesn't Python have macros like in Lisp or Scheme?
    [...]
    I think your document should be made available on the Web somewhere,
    but not in the FAQ. Of course the FAQ could and should give a pointer
    to it.
    [...]

    +1, FWIW


    John
  • John J. Lee at Aug 23, 2003 at 4:20 pm

    "Andrew Dalke" <adalke at mindspring.com> writes:

    Here's a proposed Q&A for the FAQ based on a couple recent
    threads. Appropriate comments appreciated [...]
    Those against including macros in Python point out that
    Python has a different focus. It has a broad target audience.
    Beginners can pick it up pretty easily yet advanced
    developers also enjoy working in the language. New features,
    like new builtin functions, additional core types, expanding
    the standard libraries, list comprensions, metaclasses and
    deriving from object, and macros, all make it harder to
    learn how to use Python well and shifts the balance more
    towards advanced programmers.

    More importantly, the detractors -- including those
    with plenty of experience using macros in Lisp -- argue that
    macros cause dialects to form. Macros can modify other code to
    make it fit the problem better, while functions only use other
    code but make no modifications. This makes them very
    powerful but means that understanding a section of code
    requires also knowing about any macros which might use the
    code. In an extreme case which wouldn't be used in real
    projects, every * could be replaced with a +.
    [...]

    About the first paragraph here: I don't recall anyone arguing that,
    but maybe I forget. Anyway, my problem with it is that it reinforces
    the "Python is a scripting language for amateurs, not suitable for
    large scale / systems programming" attitude. I think Python
    demonstrates that there isn't much tension in reality between the
    'advanced programmers' and the beginners -- both seem to want a simple
    language. And as you say, the argument about dialects is more
    important anyway.

    Other than that, excellent stuff.


    John
  • Jacek Generowicz at Aug 24, 2003 at 12:42 am

    "Andrew Dalke" <adalke at mindspring.com> writes:

    Here's a proposed Q&A for the FAQ based on a couple recent threads.
    Appropriate comments appreciated
    Such an FAQ item is probably a good idea.

    However, I think that we should distil out much of the opinion
    (particularly the opinions about opinions of others :-) and try to
    base it on fact:

    - what are macros (C-like and Lisp-like),

    - what are the technical difficulties with introducing them into
    Python
    X.Y: Why doesn't Python have macros like in Lisp or Scheme?

    Before answering that, a clarification on what 'macro' means.
    A Lisp macro is a way of modifying code when that code is first
    defined. It can rearrange the structure of the code, and add and
    remove parts of it. Unlike C's #define macro language, Lisp
    macros understand the structure of the expression and the
    context (or "closure") in which it is found.
    A closure is a function which "remembers" variables from an enclosing
    lexical scope (also available in Python, since 2.1). Lisp macros
    merely replace themselves with some source code (their expansion)
    before the compiler sees them. I'm not sure what you mean by "macros
    ... understand the context".
    Here's a simple example of what a macro might do for Python.
    A complaint about Python is that constructors for simple data
    types require typing the parameter names three times, as in

    class Country:
    def __init__(self, name, capitol):
    self.name = name
    self.capitol = capitol
    [snip]

    Probably not the most exciting or edifying example, particularly
    given that metaclasses should be able to do it.

    I'm sure we'll be able to come up with a better one. I'll put some
    ideas forward at the end, to act as a starting point.
    Macros work well in Lisp or Scheme, where the code is the data, all
    written pretty much as a parse tree.
    This is very important, and is central to the problem of "how do we do
    it in Python?"
    The second problem is that Python's code blocks do not
    store the original parse tree, so there's nothing to manipulate. [...]
    Nowhere will you find the original parse tree.
    This is typically true in Lisp as well, particularly after you have
    compiled your code. (Common Lisp _must_ have a compiler, but may _in
    addition_ have an interpreter; the standard defines a function which
    returns the source code of a function, but it is allowed to say
    "sorry, you can't have it".)

    The point in Lisp is that you _write_ your code in the form of a parse
    tree, in Lisp lists, and you can pass your parse tree to a macro
    without having to re-format it, and have the macro easily manipulate
    it. It's not clear how you would do this in Python.

    But you seem to be implying that macros go off and find the parse
    trees of code that has already been compiled, and then change it?
    (Hint: they don't.)
    The deeper question is, should Python include macros?

    People with a Lisp background might have a problem understanding
    this viewpoint.
    For the record, I am not advocating the inclusion of macros in
    Python. (I do object to the suggestion that macros are responsible for
    some supposed "fragmentation" of Lisp.)
    More importantly, the detractors -- including those with plenty of
    experience using macros in Lisp -- argue that macros cause dialects
    to form.
    Could you please give me a reference to someone "with plenty of
    experience using macros in Lisp" arguing this ?

    I just don't believe it. (That's not to say that it's not true.)
    Macros can modify other code to make it fit the problem better,
    What do you mean by "other" code? Macros modify the code that is
    passed to them as an argument, transforming it before the compiler
    gets to see it. I get the impression that you believe that macros can
    somehow modify code from other parts of the program. They can't.

    Ultimately, macros just save you a lot of typing of source code. (And
    thereby save you a lot of bugs.) If you can't type it as soure code,
    then a macro can't do it.
    while functions only use other code but make no modifications.
    This only strengthens the above suspicion, but I'm not really sure
    what you mean, here.
    This makes them very powerful but means that understanding a section
    of code requires also knowing about any macros which might use the
    code.
    What do you mean by "macros using code" ?

    The macros are part of the code, just like functions are. To
    understand the code, you must understand what the macros and functions
    do.
    In an extreme case which wouldn't be used in real projects, every *
    could be replaced with a +.
    This almost completely convinces me that you are very confused about
    what macros can achieve.

    Either, you are suggesting that one might write a macro which replaces
    every * operator with a + operator, and then pass _the entire source
    code_ of a project to it (possible, but patently absurd); or you are
    suggesting that it is possible to write a macro "over here" which,
    somehow, surreptitiously modifies existing source code "over there".

    a) Macros cannot do the latter.

    b) You can achieve something very similar in Python, by re-binding
    attributes of __builtins__.

    Think of the point of macros in another way. The point is _not_ to
    take existing source code, and to change it. The point is to take
    something that is not (necessarily) valid source code, and turn it
    into valid source code, before the compiler gets to see it. Why?
    Because this allows you to create a shorthand notation, and use macros
    to expand it to real code. Think in terms of encoding design patterns.
    (As an aside, some proponents argue that macros and
    functions are essentially the same. Alex Martelli made
    an interesting observation about one difference: it's often
    worthwhile to turn a set of code into a function even if it
    is only called from one place, because it helps simplify
    the code into smaller chucks which are easier to understand.
    Macros, on the other hand, should almost never be used
    unless they are used many times.)
    I broadly agree. (I think :-)

    A function written for a single-location-call, does not really
    abstract anything, it just moves some code elsewhere to make the
    original location look less hairy. Rarely would you need to use a
    macro for this purpose, but in such cases there would be no objection
    to writing a single-invocation macro.

    Macros which truly abstract something, can be difficult to write and
    difficult to read; in such cases it is important that the cost of
    creating a robust macro and the cost incurred by others in trying to
    understanding it, be offset by the savings made by its _repeated_
    usage.

    But then, functions which provide an abstraction are also more
    difficult to write and understand than ones which merely "move code
    out of the way", and you'd have to think twice whether the abstraction
    is really useful, before deciding to pay the price for writing it, and
    making readers understand it.

    Even in this respect, there is no clear-cut distinction between
    functions and macros.
    With only one or a small group of people working together
    on a project there is little problem. Macros help in developing
    idioms specific to the problem and group. When groups
    share code, they also share idioms, and anyone who has had
    to deal with UK vs. US English knows the effect idioms have
    in mututal understanding.

    Those against macros say their benefits do not outweigh
    the likelihood that the Python community will be broken up
    into distinct subsets, where people from one group find it
    hard to use code from another.
    I believe that anyone reaching such a conclusion can only do so
    on the basis of a misunderstanding of what macros can do.

    People in the music software "group" will find it hard to use code
    from people writing software for bioinformatics ... with or without
    macros. This has nothing to do with macros.

    OK, I promised some examples of macros. Note that I have not shown a
    single line of Lisp code in these threads, because I agree that it is
    likely to be meaningless to most readers of this group. I'll try
    continue without having to resort to Lisp examples.

    ==== Example 1: a debugging aid ================================

    One litte macro I have for debugging purposes (let's call it "show"),
    allows me to print out the source code of a number of expressions,
    along with their values. At this point, note that Python's horribly
    irregular syntax <0.5 wink> already starts causing problems: should
    the macro look like a block of code, or should it look like a function
    call; in Lisp both look identical.

    The "block" version of the show macro invocation:

    show:
    sin(x)
    a
    a + b + c

    The "function" version of the show macro invocation:

    show(sin(x), a, a + b + c)

    In both cases, the action of the macro should be to replace itself
    with its expansion, _before_ the compiler gets to see the source code.
    The expansion should look like this:

    print "sin(x) =>", sin(x)
    print "a =>", a
    print "a + b + c =>", a + b + c

    Note the key points:

    1) The macro receives some data, and transforms it into valid Python
    source code

    2) This happens before compile time

    3) Nothing "outside" the macro call gets affected by the macro's
    action.

    Can this be implemented using metaclasses, or any other existing
    Python mechanism ?

    === Example 2: Alexander Schmolck's updating classes ===============

    Alexander recently expressed the desire to have all existing instances
    of a class be updated, when he changes the source of his class, and
    re-evaluates the definition.

    This might be achieved by doing something like the following:

    temp = MyClass
    class MyClass:
    blah
    temp.__dict__ = MyClass.__dict__

    I'm not so much interested in the fine Python details of exactly what
    needs to be modified (__dict__, __bases__, whatever); what is
    important is that at least 3 distinct steps need to be taken, and
    trying to bundle these three steps (this pattern) into a function
    is impossible, because you can't pass the text of a class definition
    body to a function. (You could pass it as a string, but then your IDE
    would treat it as a string and refuse to indent it for you, and
    writing classes this way would be very unnatural.)

    A macro might help as follows. Write a macro called updating_class,
    which you would use instead of the built-in class statement. This
    macro might work according to an algorithm like this:

    class_name = <the first token of what was passed in>
    the_body = <everything that was passed in except, the first token>
    expansion = "temp = %s" % class_name
    expansion += "class %s%s" % (class_name, the_body)
    expansion += "temp.__dict__ = %s.__dict__" % class_name
    return expansion

    Here I'm assuming that the return value of the macro replaces the
    macro call in the source code, before the compiler gets to see the
    source code (which is pretty much what happens in Lisp).

    So,

    updating_class foo:
    blah

    would turn into

    temp = foo
    class foo:
    blah
    temp.__dict__ = foo.__dict__

    Again, note that any code not appearing "inside" the macro call, is
    not affected by the macro in any way ... almost ...

    ... This second example also serves the purpose of demonstrating free
    variable capture (the other type of variable capture is called macro
    argument capture). Variable capture is what is (according to the
    Schemers) unhygenic about CL macros. What is it? Well, note that, if a
    variable called "temp" had already existed in our enclosing scope, the
    updating_class macro expansion would have clobbered it.

    Scheme gets around this by ensuring that "temp" lives inside a local
    scope of the macro. The CL philosophy respects the fact that sometimes
    you deliberately _want_ to capture some variables. CL allows you to
    protect yourself against variable capture by using the "gensym"
    function; a function which creates a symbol (think of it as an
    identifier), which is guaranteed not to have existed before.


    This has turned out rather long, but I hope that it demystifies these
    scary exotic macros for at least one or two people on the list.
  • Lulu of the Lotus-Eaters at Aug 24, 2003 at 3:33 am

    Jacek Generowicz <jacek.generowicz at cern.ch> wrote previously:
    Could you please give me a reference to someone "with plenty of
    experience using macros in Lisp" arguing this ?
    At least Laura Creighton and Alex Martelli stand out as examples. I'm
    not sure either of them wants to be named specifically in this FAQ, just
    to attract a lot of argumentative email.

    Yours, Lulu...

    --
    Keeping medicines from the bloodstreams of the sick; food from the bellies
    of the hungry; books from the hands of the uneducated; technology from the
    underdeveloped; and putting advocates of freedom in prisons. Intellectual
    property is to the 21st century what the slave trade was to the 16th.
  • Doug Tolton at Aug 24, 2003 at 7:15 am
    Jacek Generowicz <jacek.generowicz at cern.ch> wrote in message news:<tyfbruf998w.fsf at lxplus014.cern.ch>...
    Macros can modify other code to make it fit the problem better,
    What do you mean by "other" code? Macros modify the code that is
    passed to them as an argument, transforming it before the compiler
    gets to see it. I get the impression that you believe that macros can
    somehow modify code from other parts of the program. They can't.

    Ultimately, macros just save you a lot of typing of source code. (And
    thereby save you a lot of bugs.) If you can't type it as soure code,
    then a macro can't do it.
    while functions only use other code but make no modifications.
    This only strengthens the above suspicion, but I'm not really sure
    what you mean, here.
    This makes them very powerful but means that understanding a section
    of code requires also knowing about any macros which might use the
    code.
    What do you mean by "macros using code" ?

    The macros are part of the code, just like functions are. To
    understand the code, you must understand what the macros and functions
    do.
    In an extreme case which wouldn't be used in real projects, every *
    could be replaced with a +.
    This almost completely convinces me that you are very confused about
    what macros can achieve.

    Either, you are suggesting that one might write a macro which replaces
    every * operator with a + operator, and then pass _the entire source
    code_ of a project to it (possible, but patently absurd); or you are
    suggesting that it is possible to write a macro "over here" which,
    somehow, surreptitiously modifies existing source code "over there".

    a) Macros cannot do the latter.

    b) You can achieve something very similar in Python, by re-binding
    attributes of __builtins__.

    Think of the point of macros in another way. The point is _not_ to
    take existing source code, and to change it. The point is to take
    something that is not (necessarily) valid source code, and turn it
    into valid source code, before the compiler gets to see it. Why?
    Because this allows you to create a shorthand notation, and use macros
    to expand it to real code. Think in terms of encoding design patterns.
    I think Jacek has made an extremely good point here, and I want to
    follow up on it to determine if everyone has a clear understanding of
    what macros are and are not. Following I have outlined a series of
    questions an answers to attempt to illuminate the issue. Please
    suggest any clarifications or distinctions that need to be made. Also
    note that the examples presented are purposefully trivial and could in
    many instances be accomplished other ways, however I am presenting
    these examples in order to provide a meaningful context to proceed
    from.

    Question: What are Macros?

    Answer: Macros are an advanced templating system that allow dynamic
    evaluation of code during run time. Essentially Macros allow you to
    pass in some variables that are then expanded into valid Python syntax
    which is then executed.

    Question: Can't I accomplish this same functionality via string
    parsing and eval?

    Answer: Eval will accept any valid Python expression and execute it.
    This is however regarded by many as a security hole. Macros allow
    this type of expressiveness while limiting many of the security risks.
    With a Macro you can more specifically control what types of actions
    are permissible.

    Question: I have some existing modules, if I import a macro from
    someone else, could the macro make my existing code perform in
    unexpected ways?

    Answer: No. A macro doesn't modify existing code, in order to invoke
    a macro it must be called just like a function call. So a Macro
    couldn't alter the functionality of existing code any more than a
    function could. The only way to accomplish such a feat would be to
    manually open each file and parse through them and re-evaluate them.
    This type of functionality is not what macros are about any more than
    a function call would be.

    Question: How would Macros work in practice?

    Answer: Here is a trivial example macro to start with:

    def HelloFunc(message):
    print "%s\n" % (message)

    defmacro uselessmacro(message):
    macroexpansion = "HelloFunc(%s)" % (message)

    defmacro callfunc(funcname, message):
    macroexpansion = """
    calllog = open("/var/log/functions.log", "a")
    calllog.write("%(funcname)s was called.\n")
    calllog.flush()
    calllog.close()
    %(funcname)s(%(message)s)
    """ % locals()

    if __name__ == "__main__":
    uselessmacro("Hello World")
    callfunc("HelloFunc", "Hello World")

    # Note, Jacek's method of using indentation is an interesting
    alternative to calling them as a
    # function.

    # Second note, this textarea box does not have a compiler, so please
    forgive any syntax errors :)

    uselessmacro simply passes it's arguments through to HelloFunc
    undisturbed
    callfunc logs all calls to a function before evaluating the function

    You could clearly accomplish this functionality using other means,
    however I have used such a trivial example in order to illustrate
    several points:

    1) Macros once they are defined would most likely be invoked the same
    way any other function call would be or represented as a block of
    code.

    2) Macros taken in isolation would not allow any strange changes to
    the language to be made (eg changing all *'s to +'s, or changing the
    behavior of existing classes - metaclasses, builtins and magic methods
    are required for this)

    3) Understanding a macro's behavior is typically easy for someone
    familiar with Python's syntax and constructs. In otherwords a Macro
    allows you to encapsulate commonly used idioms into a cleaner syntax,
    but would not allow you to alter the behavior of the language itself.

    My examples are barely touching the surface of what is possible with
    macros, but I wanted to start with some simple examples in order to
    give everyone some solid examples of what Macros are, and what Macros
    are not.
  • Jacek Generowicz at Aug 24, 2003 at 9:31 am

    dtolton at yahoo.com (Doug Tolton) writes:

    Also note that the examples presented are purposefully trivial and
    could in many instances be accomplished other ways,
    I hoped that the examples that I presented satisfied the requirements
    that

    a) They be simple enough to understand

    b) not be (at least easily) implementable using existing Python
    features

    c) Not be completely useless

    Did I fail ? :-)
    Essentially Macros allow you to pass in some variables that are then
    expanded into valid Python syntax which is then executed.
    Well, not just variables, data in general, some of which might be
    variables, some of which might be variable names (identifiers) and
    some of which might be (peudo-) code.
    Question: Can't I accomplish this same functionality via string
    parsing and eval?

    Answer: Eval will accept any valid Python expression and execute it.
    This is however regarded by many as a security hole. Macros allow
    this type of expressiveness while limiting many of the security risks.
    With a Macro you can more specifically control what types of actions
    are permissible.
    I don't think that this is relevant (or true, for that matter). Lisp
    macros do not expand arbitrary data that a user passes in, they appear
    as literals in your source code. The security issues are the same as
    those surrounding "os.system('rm -rf /')" appearing literally in your
    source code.

    What is relevant, is the pain involved in doing it via strings. The
    (pseudo) code that you pass to a Lisp macro, not only looks exactly
    like Lisp code, but it is automatically tokenized and parsed for you,
    and very easily manipulatable.
    # Note, Jacek's method of using indentation is an interesting
    alternative to calling them as a
    # function.
    It's probably worth pointing out, again, how important Lisp's
    uniformity of syntax is, for its macro system. Generally speaking, a
    macro accepts as arguments some mixture of data, identifiers and
    expressions (let's forget about the existence of Python statements for
    now, they complicate the situation even more). How do you pass in a
    mixture of data and code?

    Data is usually presented in a tuple:

    foo(datum1, datum2, datum3)

    Expressions are usually presented on separate lines:

    def foo(...):
    expression1
    expression2
    expression3

    Let's try to think of what a Python with-open-file macro call would
    look like.

    In CL, with-open-file takes

    1) a list containing an identifier (a symbol), a pathname (let's call
    it a string) with some optional keywords.

    2) arbitrary code

    It then binds the identifer to a stream associated with the file, and
    expands the source code inside an exception handling environment.

    So, the expansion of a (simplistic) Python with_open_file might look
    like this:

    outfile = file("foo/bar", "w")
    try:
    print >> outfile, foo(...)
    for i in range(10):
    print >> outfile, bar(...)
    except:
    perform the necessary closing and cleaning up

    But what should the corresponding call to with_open_file look like?

    We have to pass in the identifier "outfile" (this is where it is very
    handy to have symbols as a first-class data type, as in Lisp), and the
    file name "foo/bar" ... but then we also have to pass in the chunk of
    code

    print >> outfile, foo(...)
    for i in range(10):
    print >> outfile, bar(i)

    In Lisp it would look something like this:

    (with-open-file (outfile "foo/bar" :direction :output)
    (format outfile "~a" (foo ...))
    (dotimes (i 10)
    (format outfile "~&~a" (bar i))))

    Now, I refuse to contemplate passing the source code in a string, in
    the Python version.

    Maybe we have to accept the distinction between data-like data, and
    code-like data, having the macro accept both a tuple of arguments, and
    blocks of code, so the call might look thus:

    with_open_file("outfile", "foo/bar", "w"):
    code:
    print >> outfile, foo(...)
    for i in range(10):
    print >> outfile, bar(...)

    (If you need to pass in more than one code block, each one would go in
    a seperate "code:" block.)

    What could the definition look like?

    defmacro with_open_file(stream, filename, mode)(block1)
    expand(stream) = file(filename, mode)
    try:
    expand(block1)
    except:
    perform the necessary closing and clearing up

    (expand would be a function which gets replaced by its argument, at
    macro expansion time.)

    Hmm, just about bearable, I guess, but I doubt that it generalizes
    very well.

    Note that we didn't transform the source code argument, just pasted it
    in directly. If we wanted to manipulate it first, we'd be in for
    serious suffering.
    You could clearly accomplish this functionality using other means,
    I think it is best to steer clear of such examples, as experience
    shows that some people will not be able to resist the temptation to
    conclude (and suggest) that there is no point to macros, and the
    discussion gets derailed. We've seen enough of "you could do this
    quite easily without macros, but ...", and not enough of "This would
    be soooo painful without macros ..." (I won't say "impossible", as
    there is always Turing completeness.)

    What was wrong with my original examples? I'd like to know, so that I
    can replace them with better ones.
  • John J. Lee at Aug 24, 2003 at 1:04 pm

    Jacek Generowicz <jacek.generowicz at cern.ch> writes:

    dtolton at yahoo.com (Doug Tolton) writes: [...]
    I hoped that the examples that I presented satisfied the requirements
    that

    a) They be simple enough to understand

    b) not be (at least easily) implementable using existing Python
    features

    c) Not be completely useless

    Did I fail ? :-)
    Yes. :-)


    John
  • Doug Tolton at Aug 24, 2003 at 7:36 pm
    Jacek Generowicz <jacek.generowicz at cern.ch> wrote in message news:<tyfhe47e705.fsf at lxplus085.cern.ch>...
    dtolton at yahoo.com (Doug Tolton) writes:
    Also note that the examples presented are purposefully trivial and
    could in many instances be accomplished other ways,
    I hoped that the examples that I presented satisfied the requirements
    that

    a) They be simple enough to understand

    b) not be (at least easily) implementable using existing Python
    features

    c) Not be completely useless

    Did I fail ? :-)
    No I don't think you failed, I just wanted to nail down that specific
    point. The reason I purposefully used trivial examples was to
    illustrate the point the Macros don't somehow magically alter existing
    source code, rather they operate on source code that is passed into
    them. I actually liked your examples, I was just hoping to come at
    them from a slightly different example.
    Essentially Macros allow you to pass in some variables that are then
    expanded into valid Python syntax which is then executed.
    Well, not just variables, data in general, some of which might be
    variables, some of which might be variable names (identifiers) and
    some of which might be (peudo-) code.
    Question: Can't I accomplish this same functionality via string
    parsing and eval?

    Answer: Eval will accept any valid Python expression and execute it.
    This is however regarded by many as a security hole. Macros allow
    this type of expressiveness while limiting many of the security risks.
    With a Macro you can more specifically control what types of actions
    are permissible.
    I don't think that this is relevant (or true, for that matter). Lisp
    macros do not expand arbitrary data that a user passes in, they appear
    as literals in your source code. The security issues are the same as
    those surrounding "os.system('rm -rf /')" appearing literally in your
    source code.

    What is relevant, is the pain involved in doing it via strings. The
    (pseudo) code that you pass to a Lisp macro, not only looks exactly
    like Lisp code, but it is automatically tokenized and parsed for you,
    and very easily manipulatable.
    Again, I wanted to focus the discussion for a moment to help alleviate
    people's fear of Macro's, as they seem to be based on
    mis-understanding what they are about. Please forgive my over
    simplification.
    # Note, Jacek's method of using indentation is an interesting
    alternative to calling them as a
    # function.
    It's probably worth pointing out, again, how important Lisp's
    uniformity of syntax is, for its macro system. Generally speaking, a
    macro accepts as arguments some mixture of data, identifiers and
    expressions (let's forget about the existence of Python statements for
    now, they complicate the situation even more). How do you pass in a
    mixture of data and code?

    Data is usually presented in a tuple:

    foo(datum1, datum2, datum3)

    Expressions are usually presented on separate lines:

    def foo(...):
    expression1
    expression2
    expression3

    Let's try to think of what a Python with-open-file macro call would
    look like.

    In CL, with-open-file takes

    1) a list containing an identifier (a symbol), a pathname (let's call
    it a string) with some optional keywords.

    2) arbitrary code

    It then binds the identifer to a stream associated with the file, and
    expands the source code inside an exception handling environment.

    So, the expansion of a (simplistic) Python with_open_file might look
    like this:

    outfile = file("foo/bar", "w")
    try:
    print >> outfile, foo(...)
    for i in range(10):
    print >> outfile, bar(...)
    except:
    perform the necessary closing and cleaning up

    But what should the corresponding call to with_open_file look like?

    We have to pass in the identifier "outfile" (this is where it is very
    handy to have symbols as a first-class data type, as in Lisp), and the
    file name "foo/bar" ... but then we also have to pass in the chunk of
    code

    print >> outfile, foo(...)
    for i in range(10):
    print >> outfile, bar(i)

    In Lisp it would look something like this:

    (with-open-file (outfile "foo/bar" :direction :output)
    (format outfile "~a" (foo ...))
    (dotimes (i 10)
    (format outfile "~&~a" (bar i))))

    Now, I refuse to contemplate passing the source code in a string, in
    the Python version.

    Maybe we have to accept the distinction between data-like data, and
    code-like data, having the macro accept both a tuple of arguments, and
    blocks of code, so the call might look thus:

    with_open_file("outfile", "foo/bar", "w"):
    code:
    print >> outfile, foo(...)
    for i in range(10):
    print >> outfile, bar(...)

    (If you need to pass in more than one code block, each one would go in
    a seperate "code:" block.)

    What could the definition look like?

    defmacro with_open_file(stream, filename, mode)(block1)
    expand(stream) = file(filename, mode)
    try:
    expand(block1)
    except:
    perform the necessary closing and clearing up

    (expand would be a function which gets replaced by its argument, at
    macro expansion time.)

    Hmm, just about bearable, I guess, but I doubt that it generalizes
    very well.

    Note that we didn't transform the source code argument, just pasted it
    in directly. If we wanted to manipulate it first, we'd be in for
    serious suffering.
    You could clearly accomplish this functionality using other means,
    I think it is best to steer clear of such examples, as experience
    shows that some people will not be able to resist the temptation to
    conclude (and suggest) that there is no point to macros, and the
    discussion gets derailed. We've seen enough of "you could do this
    quite easily without macros, but ...", and not enough of "This would
    be soooo painful without macros ..." (I won't say "impossible", as
    there is always Turing completeness.)

    What was wrong with my original examples? I'd like to know, so that I
    can replace them with better ones.
    I agree some people have been derailed by that concept, but I'd hoped
    that making it painfully clear that this wasn't the only way to do it,
    nor was it necessarily a representative sample of Macros, that maybe
    some people would see conceptually what Macros are in an extremely
    simple way. The Hello World of Macros.

    Again, I thought your examples were fine, I just wanted to simplify
    them to help the discussion along. If you hate them that much, we can
    forget I ever posted them. :-p

    Doug Tolton
  • Jacek Generowicz at Aug 25, 2003 at 7:30 am

    dtolton at yahoo.com (Doug Tolton) writes:

    Again, I wanted to focus the discussion for a moment to help alleviate
    people's fear of Macro's, as they seem to be based on
    mis-understanding what they are about. Please forgive my over
    simplification.
    I was merely pointing out that I think that the security issue is
    completely orthogonal.
    If you hate them that much, we can forget I ever posted them. :-p
    But Google won't :-)
  • Terry Reedy at Aug 24, 2003 at 7:46 am
    "Jacek Generowicz" <jacek.generowicz at cern.ch> wrote in message
    news:tyfbruf998w.fsf at lxplus014.cern.ch...
    This has turned out rather long, but I hope that it demystifies these
    scary exotic macros for at least one or two people on the list.
    Definitely. The sort of concrete example I asked for. Thank you.

    Terry
  • Michele Simionato at Aug 24, 2003 at 9:03 am
    Jacek Generowicz <jacek.generowicz at cern.ch> wrote in message news:<tyfbruf998w.fsf at lxplus014.cern.ch>...
    ==== Example 1: a debugging aid ===============================>
    One litte macro I have for debugging purposes (let's call it "show"),
    allows me to print out the source code of a number of expressions,
    along with their values. At this point, note that Python's horribly
    irregular syntax <0.5 wink> already starts causing problems: should
    the macro look like a block of code, or should it look like a function
    call; in Lisp both look identical.

    The "block" version of the show macro invocation:

    show:
    sin(x)
    a
    a + b + c

    The "function" version of the show macro invocation:

    show(sin(x), a, a + b + c)

    In both cases, the action of the macro should be to replace itself
    with its expansion, _before_ the compiler gets to see the source code.
    The expansion should look like this:

    print "sin(x) =>", sin(x)
    print "a =>", a
    print "a + b + c =>", a + b + c

    Note the key points:

    1) The macro receives some data, and transforms it into valid Python
    source code

    2) This happens before compile time

    3) Nothing "outside" the macro call gets affected by the macro's
    action.

    Can this be implemented using metaclasses, or any other existing
    Python mechanism ?
    Yes.

    http://groups.google.it/groups?hl=it&lr=&ie=UTF-8&threadm"59b0e2.0306040808.1d737a54%40posting.google.com&rnum=2&prev=/groups%3Fhl%3Dit%26lr%3D%26ie%3DISO-8859-1%26q%3Dsimionato%2Bmacros%26meta%3Dgroup%253Dcomp.lang.python.*

    At the end it uses metaclasses+exec+a simple argument parser based on
    tokenize. EXPERIMENTAL code, which I would not trust except as proof
    of concept.
    === Example 2: Alexander Schmolck's updating classes ==============
    I think this also can be done with metaclasses (somebody already
    posted a recipe in some other thread, Diederich?)

    Michele Simionato, Ph. D.
    MicheleSimionato at libero.it
    http://www.phyast.pitt.edu/~micheles
    --- Currently looking for a job ---
  • Jacek Generowicz at Aug 24, 2003 at 9:44 am

    Unfortunately I can't easily access this at the moment. I'll try to
    have a look later.
    At the end it uses metaclasses+exec+a simple argument parser based on
    tokenize. EXPERIMENTAL code, which I would not trust except as proof
    of concept.
    If it uses exec, then it is faking up macros.

    Lisp macros are little else besides offering language support for the
    use of exec for this kind of thing.

    If people start resorting to this sort of trick themselves, and
    re-inventing it over and over again, then you might as well design it
    properly, once and for all, and call it a macro system.

    I submit that Python already has macros ... they just need a bit of
    streamlining :-)
  • John J. Lee at Aug 24, 2003 at 1:11 pm

    Jacek Generowicz <jacek.generowicz at cern.ch> writes:

    mis6 at pitt.edu (Michele Simionato) writes:
    Can this be implemented using metaclasses, or any other existing
    Python mechanism ?
    Yes.
    [...]
    At the end it uses metaclasses+exec+a simple argument parser based on
    tokenize. EXPERIMENTAL code, which I would not trust except as proof
    of concept.
    If it uses exec, then it is faking up macros.

    Lisp macros are little else besides offering language support for the
    use of exec for this kind of thing.

    If people start resorting to this sort of trick themselves, and
    re-inventing it over and over again, then you might as well design it
    properly, once and for all, and call it a macro system.
    [...]

    Well, as has been repeatedly argued, that's just the point: people
    *don't* do that over and over again. Who uses these 'faked-up
    macros'? A few people who like messing around with metaclasses for
    fun (note Michele's warning about 'EXPERIMENTAL' and 'proof of
    concept'). If it's made easier, relatively sane people might do it
    too, and then we'll all suffer ;-)

    I'm sure you're capable of understanding this position, but the
    argument has started going round in circles.


    John
  • Jacek Generowicz at Aug 24, 2003 at 2:43 pm

    jjl at pobox.com (John J. Lee) writes:

    I'm sure you're capable of understanding this position, but the
    argument has started going round in circles.
    As I've said repeatedly:

    - I do _not_ necessarily advocate the inclusion of macros in Python.

    - I merely wish to dispel the myth that macros make code less
    readable[*]: they _ease_ understanding of the code.


    [*] And they don't fragment languages either.
  • John J. Lee at Aug 25, 2003 at 12:38 pm

    Jacek Generowicz <jacek.generowicz at cern.ch> writes:

    jjl at pobox.com (John J. Lee) writes:
    I'm sure you're capable of understanding this position, but the
    argument has started going round in circles.
    As I've said repeatedly:

    - I do _not_ necessarily advocate the inclusion of macros in Python.
    But you do advocate them in other languages, right? And that was a
    significant part of the argument.

    - I merely wish to dispel the myth that macros make code less
    readable[*]: they _ease_ understanding of the code.


    [*] And they don't fragment languages either.

    John
  • Jacek Generowicz at Aug 25, 2003 at 1:39 pm

    jjl at pobox.com (John J. Lee) writes:

    Jacek Generowicz <jacek.generowicz at cern.ch> writes:
    - I do _not_ necessarily advocate the inclusion of macros in Python.
    But you do advocate them in other languages, right? And that was a
    significant part of the argument.
    You've lost me.

    What I advocate is that, in the process of constructing an FAQ or
    article about macros in Python, one should try to exclude material
    which sounds like this:

    Q: What is static typing, and why isn't it included in Python?

    A: Static typing is the requirement that the programmer write his
    code while standing up. Some people believe that this causes
    infertility.

    Or, to make a closer analogy:

    Q: What are functions, and why aren't they included in Assembler ?

    A: Functions are a very powerful means of creating
    abstarctions. Some people believe that functions make a program
    more difficult to understand. Functions also cause languages to
    split, as evidenced by the existence of many different languages
    which support functions. If any programmer on your project writes
    a function, random parts of the code _you_ wrote will stop working;
    shortly thereafter frogs will rain from the sky for forty days.

    Put another way, I advocate avoiding FUD.

    To this end, I have tried to provide information which would help
    readers of this list, who are unfamiliar with Lisp-style macros, to
    understand what Lisp-style macros are.

    If I am involved in an argument, then it is an argument about what
    Lisp-style macros _are_ and what they can or can't do ... completely
    independently of any language in which they may or may not be
    available, now or in the future.

    I offer no opinion on whether Lisp-style macros are a good idea. I
    would like people to form their own opinions on the matter, but I
    would like those opinions to be based on fact rather than on FUD.

    Now, what is it that _you_ are arguing ?
  • Michele Simionato at Aug 25, 2003 at 7:37 am
    jjl at pobox.com (John J. Lee) wrote in message news:<87znhzi4k4.fsf at pobox.com>...
    Jacek Generowicz <jacek.generowicz at cern.ch> writes:
    Well, as has been repeatedly argued, that's just the point: people
    *don't* do that over and over again. Who uses these 'faked-up
    macros'? A few people who like messing around with metaclasses for
    fun (note Michele's warning about 'EXPERIMENTAL' and 'proof of
    concept'). If it's made easier, relatively sane people might do it
    too, and then we'll all suffer ;-)
    I wholeheartly agree. I DO NOT use macros, I wrote that module as
    a reply to a thread asking about macros. I wanted to see by myself
    how difficult it was to implement them. The answer is: EXTREMELY
    difficult, if you want a RELIABLE implementation. My implementation
    is very poor man with no error checking at all, working only in simple
    cases, but still it is not trivial. I am not convinced of the advantages
    of having macros in Python, whereas I am convinced of the disadvantages.
    So, I am for the status quo.

    Michele Simionato, Ph. D.
    MicheleSimionato at libero.it
    http://www.phyast.pitt.edu/~micheles
    --- Currently looking for a job ---
  • John J. Lee at Aug 24, 2003 at 1:02 pm

    Jacek Generowicz <jacek.generowicz at cern.ch> writes:

    "Andrew Dalke" <adalke at mindspring.com> writes:
    Here's a proposed Q&A for the FAQ based on a couple recent threads.
    Appropriate comments appreciated
    Such an FAQ item is probably a good idea.

    However, I think that we should distil out much of the opinion
    (particularly the opinions about opinions of others :-) and try to
    base it on fact:
    I disagree. People reading a FAQ do want to know the opinion of
    language users, and more importantly, of Guido, because that's what
    explains why the language is as it is, and determines what will happen
    in the future. That's fact too, just fact about people's opinions.

    - what are macros (C-like and Lisp-like),

    - what are the technical difficulties with introducing them into
    Python
    Arguably, these are both secondary issues given the fact that both
    Guido (I presume?) and most Python users don't want macros. Obviously
    some background should be in there explaining what macros are and how
    they might apply to Python, but Andrew did just that.


    [...snip useful technical criticism of Andrew's FAQ...]
    Could you please give me a reference to someone "with plenty of
    experience using macros in Lisp" arguing this ?
    Maybe he was referring to Laura Creighton? He posted a quote and a
    link to another of her posts.


    [...snip more technical points...]
    But then, functions which provide an abstraction are also more
    difficult to write and understand than ones which merely "move code
    out of the way", and you'd have to think twice whether the abstraction
    is really useful, before deciding to pay the price for writing it, and
    making readers understand it.
    I'm amazed that you say this. I think most programmers would disagree
    with you here (certainly I do). Sure, abstract functions require some
    work to understand the concept involved, but the important way in
    which one function is easier or harder to understand than another has
    to do with how much detail you need to know, not how hard the concepts
    are. Needing to know details causes bugs. So does needing to know
    concepts (though less often), but you can't get rid of the concepts,
    whereas you *can* hide the details. And that's what abstraction is:
    hiding details.

    (and there's no need to answer here that you think macros are a good
    way of hiding detail: we've gathered that ;-)

    [...]
    People in the music software "group" will find it hard to use code
    from people writing software for bioinformatics ... with or without
    macros. This has nothing to do with macros.
    But people writing code using one bioinformatics library will not find
    it hard to read code using another. That might not be the case if
    macros were built-in.

    OK, I promised some examples of macros. Note that I have not shown a
    single line of Lisp code in these threads, because I agree that it is
    likely to be meaningless to most readers of this group. I'll try
    continue without having to resort to Lisp examples.

    ==== Example 1: a debugging aid ================================

    One litte macro I have for debugging purposes (let's call it "show"),
    allows me to print out the source code of a number of expressions,
    along with their values. At this point, note that Python's horribly [...]
    The "function" version of the show macro invocation:

    show(sin(x), a, a + b + c)
    [...is equivalent to...]
    print "sin(x) =>", sin(x)
    print "a =>", a
    print "a + b + c =>", a + b + c [...]
    Can this be implemented using metaclasses, or any other existing
    Python mechanism ?
    Yes, using Lisp ;-P -- you can do this in Python with an emacs macro.

    === Example 2: Alexander Schmolck's updating classes ===============

    Alexander recently expressed the desire to have all existing instances
    of a class be updated, when he changes the source of his class, and
    re-evaluates the definition.
    [...]

    There was a thread recently about how the problem behind this might be
    solved in Python. I started it (a search for "DOM" or "4DOM" or
    "hierarchy" and my email address should turn it up).

    No, wait a minute -- instances, not classes?? Sounds like an abuse of
    the language to me. What *is* the underlying problem to be solved
    here?


    John
  • Jacek Generowicz at Aug 24, 2003 at 2:34 pm

    jjl at pobox.com (John J. Lee) writes:

    Jacek Generowicz <jacek.generowicz at cern.ch> writes:
    However, I think that we should distil out much of the opinion
    (particularly the opinions about opinions of others :-) and try to
    base it on fact:
    I disagree. People reading a FAQ do want to know the opinion of
    language users, and more importantly, of Guido, because that's what
    explains why the language is as it is, and determines what will happen
    in the future. That's fact too, just fact about people's opinions.
    I agree. However, my feeling was that the original contained too much
    uneducated opinion. I was trying to suggest that we first establish
    the facts, and then include opinions based on those facts, rather than
    ones based on FUD. Particularly, we should banish the myth that macros
    fragment languages, or make code less unedrstandable. (Quite the
    reverse is true.)
    [...] the fact that both Guido (I presume?) and most Python users
    don't want macros.
    <uk7cpjssf.fsf at boost-consulting.com>
    Could you please give me a reference to someone "with plenty of
    experience using macros in Lisp" arguing this ?
    Maybe he was referring to Laura Creighton? He posted a quote and a
    link to another of her posts.
    The impression I get is that Laura was writing macros for kicks, not
    as part of a serious development effort. Of course you could use
    macros to write your own language much more easily than if you had to
    do it from scratch, but what you would be doing would still be
    wrtiting your own, separate, language.

    My point is that macros could be immensely useful in making
    constructive domain-specific extensions to the language, while _at the
    same time_ not fragmenting the Python community.
    [...snip more technical points...]
    But then, functions which provide an abstraction are also more
    difficult to write and understand than ones which merely "move code
    out of the way", and you'd have to think twice whether the abstraction
    is really useful, before deciding to pay the price for writing it, and
    making readers understand it.
    I'm amazed that you say this. I think most programmers would disagree
    with you here (certainly I do). Sure, abstract functions require some
    work to understand the concept involved, but the important way in
    which one function is easier or harder to understand than another has
    to do with how much detail you need to know, not how hard the concepts
    are. Needing to know details causes bugs. So does needing to know
    concepts (though less often), but you can't get rid of the concepts,
    whereas you *can* hide the details. And that's what abstraction is:
    hiding details.
    I agree that abstractions usually hide details. I disagree with the
    claim that hiding details is the essence of abstraction. To me, the
    essence of abstraction is noticing a similarity between certain
    "things", understandig how they differ, and encoding the whole set of
    them in one implementation, with an obvious degree of freedom which
    selects a specific "thing" from the whole family.

    If I have a function which is 400 lines long, and I choose to make it
    clearer by cutting and pasting bits of its body into smaller
    functions, and calling those functions from the original, I haven't
    abstracted anything; I have merely split it into more digestible
    chunks.
    (and there's no need to answer here that you think macros are a good
    way of hiding detail: we've gathered that ;-)
    But remember that I don't necessarily think they are a good idea in
    Python.
    [...]
    People in the music software "group" will find it hard to use code
    from people writing software for bioinformatics ... with or without
    macros. This has nothing to do with macros.
    But people writing code using one bioinformatics library will not find
    it hard to read code using another. That might not be the case if
    macros were built-in.
    It might also be the case even in the absence of macros.

    Why do so many people here think that macros can perform some
    obfuscating magic, of which other language constructs such as
    functions or classes are not capable ?
    Can this be implemented using metaclasses, or any other existing
    Python mechanism ?
    Yes, using Lisp ;-P -- you can do this in Python with an emacs macro.
    :-)
    === Example 2: Alexander Schmolck's updating classes ===============

    Alexander recently expressed the desire to have all existing instances
    of a class be updated, when he changes the source of his class, and
    re-evaluates the definition.
    [...]

    There was a thread recently about how the problem behind this might be
    solved in Python. I started it (a search for "DOM" or "4DOM" or
    "hierarchy" and my email address should turn it up).

    No, wait a minute -- instances, not classes?? Sounds like an abuse of
    the language to me. What *is* the underlying problem to be solved
    here?
    It started in the Ruby thread: in Ruby classes are "open".

    Say you're developing a class (foo), make a gazillion instances of it,
    and play around with them as you are continuing to develop your
    program. Suddenly you realize "Darn, I really need a fubar method!"

    In Ruby you do something like:

    class foo
    def fubar ...

    end
    end

    All your existing gazillion instances of foo have just acquired a
    fubar method, and you can happily continue your exploration with all
    the state you built up still intact.

    The first thing you might try to do in Python is to add the fubar
    method to the class source code, and re-evaluate it ... only to find
    that your existing instances know nothing about the new method ... so
    you have to go through the hassle of re-creating a gazillion of them,
    before you can continue with what you were doing.

    (Note, this is advocated as an interactive development aid, _not_ as a
    production code technique.)
  • John J. Lee at Aug 24, 2003 at 6:47 pm

    Jacek Generowicz <jacek.generowicz at cern.ch> writes:

    jjl at pobox.com (John J. Lee) writes: [...]
    I agree. However, my feeling was that the original contained too much
    uneducated opinion. I was trying to suggest that we first establish
    the facts, and then include opinions based on those facts, rather than
    ones based on FUD. Particularly, we should banish the myth that macros
    fragment languages, or make code less unedrstandable. (Quite the
    reverse is true.)
    It would be nice to make a definitive ruling :-), but it seems clear
    most people here disagree with that. If Andrew's FAQ answer is going
    to be linked to, no reason why other people can't supply an
    alternative answer and have that linked to from the Python FAQ too.


    [...]
    I agree that abstractions usually hide details. I disagree with the
    claim that hiding details is the essence of abstraction. To me, the
    essence of abstraction is noticing a similarity between certain
    "things", understandig how they differ, and encoding the whole set of
    them in one implementation, with an obvious degree of freedom which
    selects a specific "thing" from the whole family.
    Or, abstracting a bit: hiding details <wink>.

    If I have a function which is 400 lines long, and I choose to make it
    clearer by cutting and pasting bits of its body into smaller
    functions, and calling those functions from the original, I haven't
    abstracted anything; I have merely split it into more digestible
    chunks.
    We're probably in violent agreement.


    John
  • Jacek Generowicz at Aug 24, 2003 at 9:03 pm

    jjl at pobox.com (John J. Lee) writes:

    Jacek Generowicz <jacek.generowicz at cern.ch> writes:
    jjl at pobox.com (John J. Lee) writes: [...]
    Particularly, we should banish the myth that macros fragment
    languages, or make code less unedrstandable. (Quite the reverse is
    true.)
    it seems clear most people here disagree with that.
    It seems clear to me, that most of the people who disagree with it,
    disagree with it on the basis of a misunderstanding of what it is that
    macros actually do.

    I am keen to avoid a whole FAQ item, or a whole web page to be built
    around a bogus starting point.
  • Christos TZOTZIOY Georgiou at Aug 25, 2003 at 7:50 am
    On 24 Aug 2003 16:34:08 +0200, rumours say that Jacek Generowicz
    <jacek.generowicz at cern.ch> might have written:

    [See at the end for an interactive session transcript -- a simple copy -
    paste]
    No, wait a minute -- instances, not classes?? Sounds like an abuse of
    the language to me. What *is* the underlying problem to be solved
    here?
    It started in the Ruby thread: in Ruby classes are "open".

    Say you're developing a class (foo), make a gazillion instances of it,
    and play around with them as you are continuing to develop your
    program. Suddenly you realize "Darn, I really need a fubar method!"

    In Ruby you do something like:

    class foo
    def fubar ...

    end
    end

    All your existing gazillion instances of foo have just acquired a
    fubar method, and you can happily continue your exploration with all
    the state you built up still intact.

    The first thing you might try to do in Python is to add the fubar
    method to the class source code, and re-evaluate it ... only to find
    that your existing instances know nothing about the new method ... so
    you have to go through the hassle of re-creating a gazillion of them,
    before you can continue with what you were doing.
    ##### code starts here #####
    class A(object):
    pass
    a= A()
    a.test()
    Traceback (most recent call last):
    File "<pyshell#21>", line 1, in ?
    a.test()
    AttributeError: 'A' object has no attribute 'test'
    def test(self):
    print id(self)

    A.test = test
    a.test()
    13111376

    ##### code ends here #####
    --
    TZOTZIOY, I speak England very best,
    Microsoft Security Alert: the Matrix began as open source.
  • Jacek Generowicz at Aug 25, 2003 at 11:01 am

    Christos "TZOTZIOY" Georgiou <tzot at sil-tec.gr> writes:

    On 24 Aug 2003 16:34:08 +0200, rumours say that Jacek Generowicz
    <jacek.generowicz at cern.ch> might have written:
    The first thing you might try to do in Python is to add the fubar
    method to the class source code, and re-evaluate it ... only to find
    that your existing instances know nothing about the new method ... so
    you have to go through the hassle of re-creating a gazillion of them,
    before you can continue with what you were doing.
    ##### code starts here #####
    class A(object):
    pass
    a= A()
    a.test()
    Traceback (most recent call last):
    File "<pyshell#21>", line 1, in ?
    a.test()
    AttributeError: 'A' object has no attribute 'test'
    def test(self):
    print id(self)

    A.test = test
    a.test()
    13111376

    ##### code ends here #####
    No, you are missing the point. The point is that you want your source
    code to be in sync with your program state, and you want your source
    code to look natural. By "natural" I mean that you want your source
    code to look like this:

    class A(object):
    def test(self):
    print id(self)

    rather than like this:

    class A(object):
    pass

    def test(self):
    print id(self)

    A.test = test

    If you still don't understand the motivation, please read the original
    thread; there's no point in repeating the arguments all over again in
    this thread, which is about something completely different.
  • Hannu Kankaanpää at Aug 25, 2003 at 8:52 pm
    Jacek Generowicz <jacek.generowicz at cern.ch> wrote in message news:<tyfbruf998w.fsf at lxplus014.cern.ch>...
    ==== Example 1: a debugging aid ================================
    === Example 2: Alexander Schmolck's updating classes ===============
    Here's another example of a real life situation
    where I thought it'd be good to have macros in Python.
    I was writing a GUI system and I had to define lots of
    code twice, both for the x-axis and y-axis. Such
    as


    if y > self.y - bs and y < self.y + bs + self.height:
    return (abs(x - self.x) < bs,
    abs(x - (self.x + self.width)) < bs)
    else:
    return False, False


    and then


    if x > self.x - bs and x < self.x + bs + self.width:
    return (abs(y - self.y) < bs,
    abs(y - (self.y + self.height)) < bs)
    else:
    return False, False


    Obviously this was quite unsatisfactory. I ended up
    putting the axis code in a separate class so I could
    use them interchangeably. I.e. If I passed
    func(self.y, self.x)
    and then
    func(self.x, self.y)

    I would get the same effect on both axises. But this
    would've been an excellent place for macros IMO (unless
    there's a more clever solution as a whole). Using macros
    that combine both function call and "code block" syntax,
    I could've written a simple function like this:

    defBoth getSize(self):
    return self.size

    And it would've been expanded to

    def getWidth(self):
    return self.width

    def getHeight(self):
    return self.height

    The macro would've had to change all "size" to either
    "width" or "height", and also change "pos" to either "x"
    or "y" and so on.

    This way, I could've got no duplicated code but
    also a more intuitive interface than I currently have
    (to get width, one needs to type obj.x.getSize() instead
    of obj.getWidth()). And it's obvious this kind of "defBoth"
    wouldn't be added as a language level construct -- Thus
    macros are the only good solution.
  • Jacek Generowicz at Aug 26, 2003 at 7:09 am

    hanzspam at yahoo.com.au (Hannu Kankaanp??) writes:


    if y > self.y - bs and y < self.y + bs + self.height:
    return (abs(x - self.x) < bs,
    abs(x - (self.x + self.width)) < bs)
    else:
    return False, False


    and then


    if x > self.x - bs and x < self.x + bs + self.width:
    return (abs(y - self.y) < bs,
    abs(y - (self.y + self.height)) < bs)
    else:
    return False, False

    Obviously this was quite unsatisfactory. I ended up
    putting the axis code in a separate class
    Do you mean "function" rather than "class" ?
    so I could
    use them interchangeably. I.e. If I passed
    func(self.y, self.x)
    and then
    func(self.x, self.y)

    I would get the same effect on both axises. But this
    would've been an excellent place for macros IMO
    I don't see what you gain by using a macro, wrt to using a function in
    _this_ case. As no Python macro system exists, I cannot form an
    opinion of what the two approaches would look like in Python, but in
    Lisp the calls to the macro would look just like the calls to the
    function, I think. The only difference would be in the definition of
    the macro/function, and the macro would be no simpler, so you wouldn't
    really gain anything.
    Using macros
    that combine both function call and "code block" syntax,
    I could've written a simple function like this:

    defBoth getSize(self):
    return self.size

    And it would've been expanded to

    def getWidth(self):
    return self.width

    def getHeight(self):
    return self.height

    The macro would've had to change all "size" to either
    "width" or "height", and also change "pos" to either "x"
    or "y" and so on.
    So you would not only replace whole symbols, but even fragments of
    symbols (getSize -> getHeight), and thus macically/implicitly create
    new symbols. Many people consider this bad practice. The objection
    being that readers of the code come across a symbol in the source, go
    off and search for its definition, and tricks like this mean that they
    end up wasting their time.

    The defstruct macro does this sort of thing; (defstruct foo ... )
    would "magically" define a constructor called "make-foo" (and other
    stuff). IIRC, the experience with defstruct led the designers of CLOS
    to explicitly avoid doing this in CLOS' defclass macro (but I'm far
    from the world's leading authority on Lisp history).

    Incidentally, in this case, having a string based code representation
    would make your job much easier than the structured representation
    which Lisp uses. In a string you'd merely do a search and replace; in
    an s-expression you would have to recursively search for all symbols
    in all sub-expressions, and then do the search and replace within the
    name of each symbol you find.
    This way, I could've got no duplicated code but
    also a more intuitive interface than I currently have
    (to get width, one needs to type obj.x.getSize() instead
    of obj.getWidth()). And it's obvious this kind of "defBoth"
    wouldn't be added as a language level construct -- Thus
    macros are the only good solution.
    Cue metaclass solution ... :-)

    (I suspect that even with metaclasses, you wouldn't be able to avoid
    eval ... and that amounts to "informally" writing a macro)
  • Hannu Kankaanpää at Aug 26, 2003 at 2:42 pm
    Jacek Generowicz <jacek.generowicz at cern.ch> wrote in message news:<tyf1xv8ao8q.fsf at pcepsft001.cern.ch>...
    hanzspam at yahoo.com.au (Hannu Kankaanp??) writes:
    Obviously this was quite unsatisfactory. I ended up
    putting the axis code in a separate class
    Do you mean "function" rather than "class" ?
    Actually, I did mean class. Normally I'd have

    class Widget:
    def __init__(self):
    self.x = 0
    self.y = 0
    self.width = 0
    self.height = 0

    def getWidth(self):
    return self.width # normally they wouldn't be this empty!

    def getHeight(self):
    return self.height

    But by wrapping it inside a new class, I could get rid of
    the duplication (partly, at least):

    class Widget:
    def __init__(self):
    self.x = Widget.Axis(0, 0)
    self.y = Widget.Axis(0, 0)

    class Axis:
    def __init__(self, pos, size)
    self.pos = pos
    self.size = size

    def getSize(self): # this is only once here now
    return self.size

    While this tiny example doesn't show decrease in code
    size, it shows that I have a common definition for all
    "Axis"-specific code inside the appropriate Axis class.
    Rest of the Widget methods would be in the widget class.
    Thus self.x.getSize() instead of self.getWidth().
    so I could
    use them interchangeably. I.e. If I passed
    func(self.y, self.x)
    and then
    func(self.x, self.y)

    I would get the same effect on both axises. But this
    would've been an excellent place for macros IMO
    I don't see what you gain by using a macro, wrt to using a function in
    _this_ case.
    Ok, it was a bad example. I hope the code above shows
    a bit more clearly what I wanted. Anyway, without the code
    in the axis-class, I would've had to often say

    self.x = func(self.x, self.y, self.width, self.height)
    self.y = func(self.y, self.x, self.height, self.width)

    Instead of

    func(self.x, self.y)
    func(self.y, self.x)

    Which could modify the axis-specific stuff within the
    func()tion. (self.x is no longer a non-modifiable number,
    but a modifiable class)
    So you would not only replace whole symbols, but even fragments of
    symbols (getSize -> getHeight), and thus macically/implicitly create
    new symbols. Many people consider this bad practice.
    Well, I don't, really. Like any macro that could do something
    weird, it just needs to be properly understood by anyone who
    wishes to read the code.
    Incidentally, in this case, having a string based code representation
    would make your job much easier than the structured representation
    which Lisp uses. In a string you'd merely do a search and replace; in
    an s-expression you would have to recursively search for all symbols
    in all sub-expressions, and then do the search and replace within the
    name of each symbol you find.
    Well, such a recursive search isn't a problem - With help from
    a macro ;)
    This way, I could've got no duplicated code but
    also a more intuitive interface than I currently have
    (to get width, one needs to type obj.x.getSize() instead
    of obj.getWidth()). And it's obvious this kind of "defBoth"
    wouldn't be added as a language level construct -- Thus
    macros are the only good solution.
    Cue metaclass solution ... :-)
    How could metaclasses help? I'm quite inexperienced with them.
    Anyway, if I take the eval route, I might as well do

    defBoth('''getSize():
    return size''')

    ,retreive appropriate locals() from the stack and modify it
    to include the new functions. I'd rather not, though :-)
  • Miki Tebeka at Aug 24, 2003 at 8:58 am
    Hello Andrew,

    Great work!
    For a good reading about macros and why/how to use them you can point
    to Paul Graham's "On Lisp" which is freely available online at
    http://www.paulgraham.com/onlisptext.html

    He has great examples over there. My personal favorites are the
    anaphoric macros (aif ...)

    HTH.
    Miki
  • Jacek Generowicz at Aug 24, 2003 at 9:46 am

    tebeka at cs.bgu.ac.il (Miki Tebeka) writes:

    My personal favorites are the anaphoric macros (aif ...)
    Though, you do realize that these are buggy, in terms of their
    interaction with the package system :-)
  • Eltronic at Aug 24, 2003 at 7:12 pm

    On 24 Aug 2003 16:43:42 +0200 Jacek Generowicz <jacek.generowicz at cern.ch> writes:
    As I've said repeatedly: true
    - I do _not_ necessarily advocate the inclusion of macros in
    Python.
    grain of salt?
    - I merely wish to dispel the myth that macros make code less
    readable[*]: they _ease_ understanding of the code.
    maybe with 'a few' simple macros. the fear, if any, is that imported
    files depending on more macros will eventually be too great to fit your
    brain. its a decision many agree with and has been amply fleshed out with
    all sides represented.
    we would all use them, debug them, curse them often.
    macros containing macros will be possible.
    with latent bugs and security holes now multiplied.
    they can solve nothing useful speedwise since they wont replace code
    inline. with lazy evaluation of code this is less of a concern.
    [*] And they don't fragment languages either.
    I'm betting you usually get the last word.

    macros [will/might/beyond all doubt can] fragment the language into
    harder to read pieces. not the origional argument, but a concern.
    if everyone shoots for 100% of their personal truth the FAQ will be
    unreadable.

    keep it to at most few short paragraphs please.


    e
    please forward all spam to "not my real email"<help at ftc.gov>

    ________________________________________________________________
    The best thing to hit the internet in years - Juno SpeedBand!
    Surf the web up to FIVE TIMES FASTER!
    Only $14.95/ month - visit www.juno.com to sign up today!

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedAug 23, '03 at 1:30a
activeAug 26, '03 at 2:42p
posts31
users11
websitepython.org

People

Translate

site design / logo © 2022 Grokbase