FAQ
Hello all,

In a nutshell, what I'd like to do is develop a utility that would print out
every line in a python program as the program executes (I may even just
want to print line numbers). I can see hints of how to do this in pdb (and
more directly in the underlying bdb class) and also in the code module.

So,

1) Should I try and inherit a class and override some methods or should I
just pull out the relevant code and hand spin a small interpreter.

2) Anyone have a 20 liner solution for this?

Thanks.

Regards,
Mark

Search Discussions

  • Peter Hansen at Sep 19, 2002 at 3:14 am

    Mark wrote:
    Hello all,

    In a nutshell, what I'd like to do is develop a utility that would print out
    every line in a python program as the program executes (I may even just
    want to print line numbers). I can see hints of how to do this in pdb (and
    more directly in the underlying bdb class) and also in the code module.

    So,

    1) Should I try and inherit a class and override some methods or should I
    just pull out the relevant code and hand spin a small interpreter.

    2) Anyone have a 20 liner solution for this?
    Sorry, I couldn't find a solution that was larger than six lines:
    def trace(frame, event, arg):
    ... if event == 'line':
    ... print '%s: %s' % (frame.f_code.co_filename, frame.f_lineno)
    ... return trace
    ...
    import sys
    sys.settrace(trace)
    Well, if you want to print the actual line of source you could probably
    stretch it out to eight or ten lines... although I think you need
    only one more line for the simplest approach. ;-)

    -Peter
  • Mark at Sep 19, 2002 at 7:46 pm

    Peter Hansen wrote:
    Sorry, I couldn't find a solution that was larger than six lines:
    def trace(frame, event, arg):
    ... if event == 'line':
    ... print '%s: %s' % (frame.f_code.co_filename, frame.f_lineno)
    ... return trace
    ...
    import sys
    sys.settrace(trace)
    -Peter
    Hey,

    This is exactly what I needed. Now, can you explain to me how I hook in a
    "local" trace function to take over once I execute the function I'm
    interested in tracing? I'd like to use the "event" argument to take note
    of calls that are 1 level deep (ie in my foo function, any calls made) and
    turn off tracing of these until they return back to the level of foo.

    I presume I could implement this with a "if event == 'call':" condition, but
    I can't seem to get a global trace function to hook up to a local trace
    function ... it does one or two things right, then goes crazy.

    Regards,
    Mark
  • Peter Hansen at Sep 19, 2002 at 11:40 pm

    Mark wrote:
    Peter Hansen wrote:
    Sorry, I couldn't find a solution that was larger than six lines:
    def trace(frame, event, arg):
    ... if event == 'line':
    ... print '%s: %s' % (frame.f_code.co_filename, frame.f_lineno)
    ... return trace
    ...
    import sys
    sys.settrace(trace)
    This is exactly what I needed. Now, can you explain to me how I hook in a
    "local" trace function to take over once I execute the function I'm
    interested in tracing? I'd like to use the "event" argument to take note
    of calls that are 1 level deep (ie in my foo function, any calls made) and
    turn off tracing of these until they return back to the level of foo.
    I guess you'd need a global to track the level, or implement the trace
    "method" as an object with a __call__ method so you can track state as
    well as receive the tracing calls.

    As for local tracing... the above function actually does that already.
    As the docs note, the global function (installed by sys.settrace)
    returns a reference to a local trace function when event is "call".
    I'm just using the same function for both purposes as a cheat. :)
    I presume I could implement this with a "if event == 'call':" condition, but
    I can't seem to get a global trace function to hook up to a local trace
    function ... it does one or two things right, then goes crazy.
    Post sample code if you're having trouble... but I recommend playing
    a lot at the interactive prompt before doing so. You learn lots more
    that way.

    -Peter
  • Delaney, Timothy at Sep 19, 2002 at 3:25 am

    From: Peter Hansen [mailto:peter at engcorp.com]

    Mark wrote:
    Hello all,

    In a nutshell, what I'd like to do is develop a utility
    that would print out
    every line in a python program as the program executes (I
    may even just
    want to print line numbers). I can see hints of how to do
    this in pdb (and
    more directly in the underlying bdb class) and also in the
    code module.
    So,

    1) Should I try and inherit a class and override some
    methods or should I
    just pull out the relevant code and hand spin a small interpreter.

    2) Anyone have a 20 liner solution for this?
    Sorry, I couldn't find a solution that was larger than six lines:
    def trace(frame, event, arg):
    ... if event == 'line':
    ... print '%s: %s' % (frame.f_code.co_filename, frame.f_lineno)
    ... return trace
    ...
    import sys
    sys.settrace(trace)
    Having been writing a fairly comprehensive code and path coverage tool
    (incomplete - currently working on determining what *is* an executable line
    ... think I need to work with the AST), I'd like to point out a couple of
    things with the above:

    1. frame.f_code.co_filename will return the file that the .pyc was compiled
    from. Particularly in the case of standard libraries, this will often not be
    the path to the file in *your* filesystem. There is a way to get the
    corresponding file in your filesystem (basically, determine the module the
    frame comes from, then get the module filename using inspect).

    2. The above will be *slow*. Very very slow. For a short program, this may
    not matter. For a long program, it will (I'm talking order of magnitude
    slower).

    The approach I have taken is to simply stored the line numbers and file info
    in a list, then dump it out at the end. Well, that's not *precisely* what I
    do, but that's the gist of it. I actually do lots of caching of file info,
    compression of the data, etc ... but you don't want to see that code ;)
  • Mark at Sep 19, 2002 at 4:16 am
    Okay, here's my first stab:

    from bdb import *

    class Mdb(Bdb):
    def user_call(self, frame, args):
    name = frame.f_code.co_name
    if not name: name = '???'
    def user_line(self, frame):
    import linecache
    name = frame.f_code.co_name
    if not name: name = '???'
    fn = self.canonic(frame.f_code.co_filename)
    line = linecache.getline(fn, frame.f_lineno)
    print '+++', frame.f_lineno, name, ':', line.strip()
    def user_return(self, frame, retval):
    None
    def user_exception(self, frame, exc_stuff):
    self.set_continue()

    This is copied out of bdb.py and modified from the Tdb class.

    To use it, you simply:

    import mdb # contains definition of Mdb
    import foo # contains function to trace
    m = mdb.Mdb()
    m.run('foo.main()') # main() is the method I'd like to trace

    and out pops a list like (things are a hair cleaner if the input is done
    from a file instead of interactively):

    +++ <line number> <code on that line>

    My next wish is to NOT have it step into functions .... this isn't a real
    biggie but it would help eliminate some (soon to be) volumious output.

    Regards,
    Mark
  • Inyeol Lee at Sep 19, 2002 at 4:37 pm
    I was writing similar tools, and found strange things related to f_lineno.
    For multi-line statements, python 2.2.1 sometimes sets f_lineno to the
    start line, and sometimes to the current line. This is not related to -O
    optimization.
    I reported this to SF bug 607092, but this case is now closed with
    "won't fix" status. Anyway, there seems to be an extensive change to
    f_lineno in 2.3, so I'm just waiting for it.

    --Inyeol
  • Peter Hansen at Sep 19, 2002 at 11:20 am

    Delaney, Timothy wrote:
    From: Peter Hansen [mailto:peter at engcorp.com]
    def trace(frame, event, arg):
    ... if event == 'line':
    ... print '%s: %s' % (frame.f_code.co_filename, frame.f_lineno)
    ... return trace
    ...
    import sys
    sys.settrace(trace)
    1. frame.f_code.co_filename will return the file that the .pyc was compiled
    from. Particularly in the case of standard libraries, this will often not be
    the path to the file in *your* filesystem. There is a way to get the
    corresponding file in your filesystem (basically, determine the module the
    frame comes from, then get the module filename using inspect).
    True... presumably either the OP is not concerned about standard
    library modules anyway, or he would check the archives and add
    a simple way of doing this. YAGNI...
    2. The above will be *slow*. Very very slow. For a short program, this may
    not matter. For a long program, it will (I'm talking order of magnitude
    slower).
    Hmmm... I didn't see any implication that speed was even remotely
    a concern for the OP. After all, he did ask to *print* each line
    as it executed. What could possibly be slower than that?
    (I doubt he would care to deal with the volume of information
    generated when using this with a large program anyway... needs
    a way to turn it on and off, at least.)

    My standard comments about Python speed and Knuth's comments about
    premature optimization apply...

    I'm looking forward to seeing your tool though, Timothy. :)

    -Peter
  • Mark at Sep 19, 2002 at 7:43 pm

    Peter Hansen wrote:
    True... presumably either the OP is not concerned about standard
    library modules anyway, or he would check the archives and add
    a simple way of doing this. YAGNI...
    Correct, I'm currently trying to figure out how to ignore them (ie have it
    step "over" instead of "into" standard library calls (in particular, the
    linecache.getline(foo, x) call).

    Also, I had quite a time figuring out the damn name for what I was trying to
    do. My first thought was implementing a REP loop of my own. Then I
    started looking at the debugging code. Finally, I was put on the correct
    trail by looking at "tracing".
    Hmmm... I didn't see any implication that speed was even remotely
    a concern for the OP. After all, he did ask to *print* each line
    as it executed. What could possibly be slower than that?
    (I doubt he would care to deal with the volume of information
    generated when using this with a large program anyway... needs
    a way to turn it on and off, at least.)
    Yup, for now I simply need to look at traces. Speed is not important (1.
    this is a prototype and 2. speed may never be important). Further, I'm
    only looking at single function (aside from library function calls)
    algorithms for the moment.

    Thanks for the input.

    Regards,
    Mark
  • Delaney, Timothy at Sep 19, 2002 at 11:47 pm

    From: Peter Hansen [mailto:peter at engcorp.com]

    Hmmm... I didn't see any implication that speed was even remotely
    a concern for the OP. After all, he did ask to *print* each line
    as it executed. What could possibly be slower than that?
    (I doubt he would care to deal with the volume of information
    generated when using this with a large program anyway... needs
    a way to turn it on and off, at least.)
    Very true - hence my qualification. The speed comes into effect when tracing
    a long-running process - for example, while using the "standard" two
    coverage tools (trace.py and coverage.py) performance was unacceptably slow
    for coverage of some unit tests here (well, once I'd hacked them to actually
    allow unittest to determine what was a test case ...).
    My standard comments about Python speed and Knuth's comments about
    premature optimization apply...
    Oh yeah. Unfortunately, tracing becomes a real bottleneck if done wrongly -
    I've achieved two orders of magnitude improvement since I first started on
    this tool - both in performance and memory use.
    I'm looking forward to seeing your tool though, Timothy. :)
    I'd love it to eventually become part of the core - I *expect* I'll be able
    to release it (but of course, it's up to my employer ...). It's doing
    everything I want now except for determining exactly which zero-count lines
    can be safely ignored (comments, multiline expressions). Oh - and I haven't
    got an actual "trace as you go" mode (which is the topic of this thread ;)
    but that will be easy to add. And I haven't got any path coverage analysis
    yet - the data is captured, but no analysis is performed. And it currently
    does a relatively primitive text-only coverage dump. And ...

    :)

    Tim Delaney
  • Peter Hansen at Sep 20, 2002 at 12:08 am

    Delaney, Timothy wrote:
    From: Peter Hansen [mailto:peter at engcorp.com]
    I'm looking forward to seeing your tool though, Timothy. :)
    I'd love it to eventually become part of the core - I *expect* I'll be able
    to release it (but of course, it's up to my employer ...). It's doing
    everything I want now except for determining exactly which zero-count lines
    can be safely ignored (comments, multiline expressions). Oh - and I haven't
    got an actual "trace as you go" mode (which is the topic of this thread ;)
    but that will be easy to add. And I haven't got any path coverage analysis
    yet - the data is captured, but no analysis is performed. And it currently
    does a relatively primitive text-only coverage dump. And ...
    Heh. :-)

    I'm a commercial Python user as well, so I can imagine the kinds of
    things your employer think about whether to release or not. We have
    some items which we simply wouldn't consider releasing, but there
    are a number of tools which we hope to release at some point. They
    have similar limitations to yours (well, okay, they're actually much
    more primitive even than that :-) yet with a little more work they
    would be useful to others.

    More helpfully in convincing my own employer that we should release
    back to the community is pointing out how with our limited resources
    these tools will always remain primitive and simplistic. If we can
    do a little more work *and* release them, we stand a chance of seeing
    them improved by others, and taking advantage of the improvements
    ourselves. See, it's easy to convince those conservative, selfish
    minds that they can profit by giving things away. :-)

    Well, if you can't release it (I guess I just assume from all your
    mentions of it that it was going to be open), we'll probably be
    building some of the same types of things ourselves. It would
    be a real shame if we both (and others) went to all that effort
    in parallel without sharing. :(

    -Peter
  • Delaney, Timothy at Sep 20, 2002 at 12:47 am

    From: Peter Hansen [mailto:peter at engcorp.com]

    I'm a commercial Python user as well, so I can imagine the kinds of
    things your employer think about whether to release or not. We have
    As I said, I *expect* to be able to release (there's no commercial reason
    not to, and we have done so with such things in the past).

    Tim Delaney
  • Holger krekel at Sep 20, 2002 at 1:11 am

    Delaney, Timothy wrote:
    From: Peter Hansen [mailto:peter at engcorp.com]

    I'm a commercial Python user as well, so I can imagine the kinds of
    things your employer think about whether to release or not. We have
    As I said, I *expect* to be able to release (there's no commercial reason
    not to, and we have done so with such things in the past).
    Maybe we are turning circles [1] but why don't you do it, then?

    Maybe you need to invest some time in obfuscating your
    source code and you don't want to do it?

    Why do you *expect* others to solve this problem for you?

    holger




    [1] Sounds like a circular argument which might get GCed, soon :-)
  • Delaney, Timothy at Sep 20, 2002 at 1:18 am

    From: holger krekel [mailto:pyth at devel.trillke.net]

    Delaney, Timothy wrote:
    From: Peter Hansen [mailto:peter at engcorp.com]
    As I said, I *expect* to be able to release (there's no
    commercial reason
    not to, and we have done so with such things in the past).
    Maybe we are turning circles [1] but why don't you do it, then?

    Maybe you need to invest some time in obfuscating your
    source code and you don't want to do it?

    Why do you *expect* others to solve this problem for you?
    Holger, what the hell are you talking about?

    I'm not releasing yet because:

    1. The tool is not feature-complete.

    2. The code is not in a fit state for release - needs more documentation,
    etc - it's in a state of high flux, and it's a personal (but work-related)
    project so I've been a bit lax ;)

    3. I have not even approached my employer as to whether I am allowed to
    release it (but I expect that when I do so I will be given permission).

    As I stated, I would like this to get into the core library eventually - how
    does that in any way suggest that I want to:

    1. Obfuscate the code

    or

    2. Expect others to do the work for me

    ???

    Tim Delaney
  • Holger krekel at Sep 20, 2002 at 10:02 pm

    Delaney, Timothy wrote:
    From: holger krekel [mailto:pyth at devel.trillke.net]

    Delaney, Timothy wrote:
    From: Peter Hansen [mailto:peter at engcorp.com]
    As I said, I *expect* to be able to release (there's no
    commercial reason
    not to, and we have done so with such things in the past).
    Maybe we are turning circles [1] but why don't you do it, then?

    Maybe you need to invest some time in obfuscating your
    source code and you don't want to do it?

    Why do you *expect* others to solve this problem for you?
    Holger, what the hell are you talking about?
    Oh, i am very sorry. I completly mixed up discussions
    and for some reason posted to the complete wrong thread :-(
    I thought this still was the thread on how to protect a
    commercial release of a python program ....

    It was late in the night, i had a sloppy network connection
    and all that. so please take my excuse,

    holger
  • Richie Hindle at Sep 20, 2002 at 2:31 pm
    Tim,
    Having been writing a fairly comprehensive code and path coverage tool
    (incomplete - currently working on determining what *is* an executable line
    ... think I need to work with the AST)
    I've done this recently as well, and I'm now on my third iteration of
    the work-out-which-lines-are-executable code (!)

    The first version used the AST, but there are several holes in that -
    'else' lines, for instance, are identified by the parser as executable
    lines, but the trace function doesn't always get called for them. I
    then wrote a system that found all the SET_LINENO instructions, and
    that gave perfect results (as long as you weren't using -O) but then I
    found out that SET_LINENO is going away in Python 2.3...

    My current system (which I think works 8-) uses co_lnotab, which is a
    delightful data structure that describes the relationship between
    bytecodes and line numbers. Here's the relevant piece of code, which
    builds in 'lineNumbers' a list of the execuable line number in the
    code object 'code':

    # Derive the line numbers from co_lnotab; that's a list of pairs of
    # increments, one for the bytecode address and one for the line
    # number (see compile.c in the Python sources).
    lineNumbers = []
    lnotab = code.co_lnotab
    previousAddress = -1
    previousLineNo = -1
    lineNo = code.co_firstlineno
    byteCodeAddress = 0
    for i in range( 0, len( lnotab ), 2 ):
    byteCodeAddress = byteCodeAddress + ord( lnotab[ i ] )
    lineNo = lineNo + ord( lnotab[ i+1 ] )

    # When the compiler wants to increment the line number by more
    # than 255 in one go, it increments it by 255 as many times as it
    # needs and then the remainder. Fair enough, but there's a bug
    # whereby the byte code address gets incremented along
    # with the *first* 255, not the last. Hence you get a line that
    # claims to have code on it when in fact it doesn't. We detect
    # that here.
    if byteCodeAddress != previousAddress and not \
    ( lineNo - previousLineNo == 255 and
    len( lnotab ) > i+2 and
    ord( lnotab[ i+2 ] ) == 0 ):
    lineNumbers.append( lineNo )
    previousAddress = byteCodeAddress
    previousLineNo = lineNo

    Hope that helps. The coverage tool that this belongs to is finished
    but unreleased (mostly because of lack of thorough testing, but I'm
    using it myself with no problems) - if you'd like a copy, drop me an
    email.

    --
    Richie Hindle
    richie at entrian.com
  • Bruce Dawson at Sep 22, 2002 at 5:19 am
    On a tangential note...

    While doing some Python optimizing (for our about to ship Python based
    game) I tweaked the HAP Python debugger to count lines of Python code
    being executed. Then I displayed lines-of-code per frame in the game
    title bar. That turns out to be a very useful coarse indicator.

    Then I added the expression HeDbg.GetLineCount()[1] to the watch window.
    Since the GetLineCount() feature returns total lines executed and lines
    executed since the last call, and since the watch expression is
    evaluated each time control returns to the debugger, this simple
    expression tells me how many lines of code execute each time I step or go.

    So, if I step over a trivial looking function and the watch window says
    700 then I investigate to find out whether 700 lines of Python executed
    for the function is reasonable.

    For profiling of a game I found this feature set worked really well.

    I don't think we've released an updated version of HAP that has this
    feature, but we certainly will - lots of other updates also.

    Mark wrote:
    Hello all,

    In a nutshell, what I'd like to do is develop a utility that would print out
    every line in a python program as the program executes (I may even just
    want to print line numbers). I can see hints of how to do this in pdb (and
    more directly in the underlying bdb class) and also in the code module.

    So,

    1) Should I try and inherit a class and override some methods or should I
    just pull out the relevant code and hand spin a small interpreter.

    2) Anyone have a 20 liner solution for this?

    Thanks.

    Regards,
    Mark
  • Delaney, Timothy at Sep 22, 2002 at 11:51 pm

    From: richie at entrian.com [mailto:richie at entrian.com]

    The first version used the AST, but there are several holes in that -
    'else' lines, for instance, are identified by the parser as executable
    lines, but the trace function doesn't always get called for them.
    Something I would have had to find out myself ... thanks :)
    found out that SET_LINENO is going away in Python 2.3...
    The reason I didn't do the same thing ...
    My current system (which I think works 8-) uses co_lnotab, which is a
    delightful data structure that describes the relationship between
    bytecodes and line numbers. Here's the relevant piece of code, which
    builds in 'lineNumbers' a list of the execuable line number in the
    code object 'code':
    Thank you. This should help immensely.
    Hope that helps. The coverage tool that this belongs to is finished
    but unreleased (mostly because of lack of thorough testing, but I'm
    using it myself with no problems) - if you'd like a copy, drop me an
    email.
    Please do - it will be interesting to see how our approaches differ. Don't
    bother about tidying it up unless it's a real mess.

    I'd better get onto asking my employer if I can pass this code onto other
    people I guess :)

    Tim Delaney

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedSep 19, '02 at 2:54a
activeSep 22, '02 at 11:51p
posts18
users7
websitepython.org

People

Translate

site design / logo © 2022 Grokbase