FAQ

[Python] #line in python

Ross Boylan
Feb 19, 2012 at 12:54 am
The ast module shows that elements of the syntax tree have line and
column numbers. Would it be sensible to attempt to revise them to
achieve effects like the #line directive in C?

Context: Using noweb, a literate programming tool, which from a source
file foo.nw produces foo.py. The lines in the two files may be in
completely different sequenes. For debugging, it is useful to receive
error reports that refer to the original line number in foo.nw.

I am not sure how such rewriting would interact with debugger commands
that set a breakpoint at a file and line number. I'm also not sure it
would change the reported line numbers of errors.

The lack of a file name could be problematic if multiple sources
contributed to the same .py file, but that is an unlikely scenario.

As an extension or alternate, could there be a decorator like
@source_line(lineno, filename)
for classes and methods that could do the conversion on the fly? I
don't know if there's a way to go from the function (or class) object
the decorator receives to the AST.

Comments?

Ross Boylan
reply

Search Discussions

3 responses

  • Duncan Booth at Feb 20, 2012 at 11:27 am

    Ross Boylan wrote:

    As an extension or alternate, could there be a decorator like
    @source_line(lineno, filename)
    for classes and methods that could do the conversion on the fly? I
    don't know if there's a way to go from the function (or class) object
    the decorator receives to the AST.
    No [easy] way to go from bytecodes back to AST, but I see no reason why you
    can't create a new code object with your filename and line numbers and then
    create a new function using your modified code object.

    If you don't have a 1:1 correspondence of lines then you'll need to pick
    out all the existing line numbers from the code object co_lnotab and modify
    them: see dis.py findlinestarts() for how to do this.

    Classes would be harder: the decorator doesn't run until after the class
    body has executed, so you can't change the line numbers that way until it's
    too late. The only thing I can think would be to put all of the generated
    code inside a function and fix up that function with a decorator that scans
    the bytecode to find all contained classes and fix them up.

    Or you could generate a .pyc file and then fix up line numbers in the whole
    file: see
    http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html for
    some code that shows you what's in a .pyc
  • Ross Boylan at Feb 20, 2012 at 3:08 pm
    Duncan Booth wrote
    ________________________________________________________________________
    Ross Boylan wrote:
    As an extension or alternate, could there be a decorator like
    @source_line(lineno, filename)
    for classes and methods that could do the conversion on the fly? I
    don't know if there's a way to go from the function (or class) object
    the decorator receives to the AST.
    No [easy] way to go from bytecodes back to AST, but I see no reason why you
    can't create a new code object with your filename and line numbers and then
    create a new function using your modified code object.
    Could you elaborate? I don't understand what you are suggesting.
    If you don't have a 1:1 correspondence of lines then you'll need to pick
    out all the existing line numbers from the code object co_lnotab and modify
    them: see dis.py findlinestarts() for how to do this.

    Classes would be harder: the decorator doesn't run until after the class
    body has executed, so you can't change the line numbers that way until it's
    too late. The only thing I can think would be to put all of the generated
    code inside a function and fix up that function with a decorator that scans
    the bytecode to find all contained classes and fix them up.

    Or you could generate a .pyc file and then fix up line numbers in the whole
    file: see
    http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html for
    some code that shows you what's in a .pyc
    My latest concept is to produce code that rewrites itself. Suppose the
    naive file would be
    ----------- mycode.py (naive) --------------------------------
    class SomeClass:
    "class comment"

    def some_function(self, bar):
    pass
    --------- end -------------------------------

    Protect that code by putting an "if 0:" in front of it and indenting
    each line one space. Than prepend a bit of code to do the rewriting,
    and add indicators of the original line numbers.

    ----------- mycode.py (after wrapping) -------------
    from detangle import detangle
    detangle("mycode.py", "mycode.nw")
    if 0:
    # original code goes here
    class SomeClass:
    "class comment"
    #and when line numbering changes
    #line 35
    def some_function(self, bar):
    pass
    ------------- end -------------------
    I would write detangle so that it scans through the file in which it
    appears (named in the first argument), rewriting so that it appears to
    come from the original file (mycode.nw) given in the second argument.

    The scanning would look for the "if 0:" in the file. At that point it
    would accumulate code by reading lines and stripping the leading space.
    If it found a #line directive it would remember it and then remove it
    from the string it was accumulating. Finally, detangle would would
    pass the string of code to ast.compile, catching any syntax errors and
    rewriting the file and line number (I might rewrite columns too with an
    extension) and then rethrowing them.

    If compilation succeeded detangle could rewrite the AST and then exec
    it.

    Ross
  • Duncan Booth at Feb 20, 2012 at 6:54 pm

    Ross Boylan wrote:

    No [easy] way to go from bytecodes back to AST, but I see no reason
    why you can't create a new code object with your filename and line
    numbers and then create a new function using your modified code
    object.
    Could you elaborate? I don't understand what you are suggesting.
    This (written for Python 3.x, needs some attribute names changing for
    Python 2.x:

    import functools
    def source_line(lineno, filename = None):
    def decorator(f):
    c = f.__code__
    code = type(c)(
    c.co_argcount, c.co_kwonlyargcount, c.co_nlocals,
    c.co_stacksize, c.co_flags, c.co_code,
    c.co_consts, c.co_names, c.co_varnames,
    c.co_filename if filename is None else filename,
    c.co_name,
    lineno,
    c.co_lnotab, c.co_freevars, c.co_cellvars)
    f.__code__ = code
    return f
    return decorator


    @source_line(43, 'foo.bar')
    def foo():
    """This is foo"""
    for i in range(10):
    bar()

    @source_line(665, 'evil.txt')
    def bar():
    raise RuntimeError("oops")

    if __name__=='__main__':
    import traceback
    try:
    foo()
    except RuntimeError as ex:
    traceback.print_exc()

    When you run it the output looks something like this (if you create a
    file evil.txt with 667 lines):

    Traceback (most recent call last):
    File "C:\temp\foo.py", line 30, in <module>
    foo()
    File "foo.bar", line 47, in foo
    File "evil.txt", line 667, in bar
    Evil line 667
    RuntimeError: oops

Related Discussions

Discussion Navigation
viewthread | post

2 users in discussion

Ross Boylan: 2 posts Duncan Booth: 2 posts