FAQ
I'm trying to make a graphical editor and browser for Pickled files. One
aspect I'm not sure about is how to detect multiple references to the
same data.

For instance, say I had the Pickled data:
a=[1,2,3]
b=[a,4,5]
c=[b,6,7]
d=[a,b,c]

The idea is to allow the user to browse this data and indicate
references. In this case, if 'a' was selected, the browser should show
that 'b', 'c', and 'd' contain a reference to 'a'. Inversely, if 'c'
were selected, it should indicate that it's first element just isn't a
list, but a reference to a list defined elsewhere, namely 'b'.

Naturally, I could just recursively parse all the data comparing every
element to every previously listed object, but is there a less obtrusive
method? Python figures out when to delete objects based on the remaining
references to an object. Is there a way to access this information to
automatically lookup these references? Any help is greatly appreciated.

Search Discussions

  • Anthony Baxter at Aug 6, 2004 at 5:46 am

    On Fri, 06 Aug 2004 05:39:51 GMT, Chris S. wrote:
    Naturally, I could just recursively parse all the data comparing every
    element to every previously listed object, but is there a less obtrusive
    method? Python figures out when to delete objects based on the remaining
    references to an object. Is there a way to access this information to
    automatically lookup these references? Any help is greatly appreciated.
    Use the "id()" of the objects?
  • Chris S. at Aug 6, 2004 at 8:50 am

    Anthony Baxter wrote:
    On Fri, 06 Aug 2004 05:39:51 GMT, Chris S. wrote:

    Naturally, I could just recursively parse all the data comparing every
    element to every previously listed object, but is there a less obtrusive
    method? Python figures out when to delete objects based on the remaining
    references to an object. Is there a way to access this information to
    automatically lookup these references? Any help is greatly appreciated.

    Use the "id()" of the objects?
    Well, I was thinking of a function that takes an object's id and returns
    the ids of all the objects that reference it. Given that Python keeps
    this information internally (I believe), my question is how do I access
    his data and what would be the easiest way to code this function?
  • Gandalf at Aug 6, 2004 at 4:45 pm

    I'm trying to make a graphical editor and browser for Pickled files.
    One aspect I'm not sure about is how to detect multiple references to
    the same data. ...
    For instance, say I had the Pickled data:
    a=[1,2,3]
    b=[a,4,5]
    c=[b,6,7]
    d=[a,b,c]

    The idea is to allow the user to browse this data and indicate
    references. In this case, if 'a' was selected, the browser should show
    that 'b', 'c', and 'd' contain a reference to 'a'. Inversely, if 'c'
    were selected, it should indicate that it's first element just isn't a
    list, but a reference to a list defined elsewhere, namely 'b'.
    Naturally, I could just recursively parse all the data comparing every
    element to every previously listed object, but is there a less
    obtrusive
    method? Python figures out when to delete objects based on the
    remaining
    references to an object. Is there a way to access this information to
    automatically lookup these references? Any help is greatly appreciated.

    Use the "id()" of the objects?

    Well, I was thinking of a function that takes an object's id and
    returns the ids of all the objects that reference it. Given that
    Python keeps this information internally (I believe), my question is
    how do I access his data and what would be the easiest way to code
    this function?
    Yes, Python figures out when to delete an object, based on its reference
    count. But Python only counts the references. It keeps track of the
    number of references for the object. It does not know where the
    reference is located in memory. The short answer is that you cannot get
    this information effectively.

    Long answer follows.

    Question: There can be many objects in the structure without a name
    (e.g. not referenced from the value part of a dictionary).
    How do you want to display them? As an example:

    class A:
    pass

    a = A() # THE object
    b = [a,a,a,a]
    del a

    # At this point, THE object does not have a named reference at all but
    it is referenced four times. Do you really want to identify the objects
    with their memory address? The id function does this to you (Anthony
    Baxter said the same.) But this will be a useless pickle browser since
    nobody will notice memory address duplications.

    Another big problem here. In Python, the only things you can have are an
    object references. You cannot name the object itself. There are no
    'object variables', only 'reference variables'. For example:

    class A:
    pass

    a = A() # THE object we are talking about
    b = [a]
    c = [b]

    Here is the question: does the object b "references" a? The answer is
    that the question is wrong because 'a' is a reference itself - a
    reference can only reference one object at a time. Good questions are:

    - does 'a' and 'b' reference to the same object? (No)
    - does 'b[0]' and 'a' reference to the same object? (Yes)

    You may think that a reference of 'a' is contained in 'b'. But this is
    not a general assumption. For example:

    a = A()
    b = A()
    a.o = b
    b.o = a

    Now which is contained in which?

    Sorry for the long e-mail.

    G
  • Chris S. at Aug 7, 2004 at 1:38 am

    Gandalf wrote:

    Yes, Python figures out when to delete an object, based on its reference
    count. But Python only counts the references. It keeps track of the
    number of references for the object. It does not know where the
    reference is located in memory. The short answer is that you cannot get
    this information effectively.
    The gc module has methods that list an object's referrers and referents.
    Although you're correct in that we're unable to access the total number
    of references since duplicate references are not counted. For instance

    import gc
    a=[1,2,3]
    b=[a,a,a,a]
    print gc.get_referrers(a)

    would only print out one instance of b, even though it references 'a'
    four times. However, this only seems to apply for direct references, so
    the total number of references can be determined by scanning all
    referrers and counting the duplicate ids.
  • Michael Hudson at Aug 6, 2004 at 11:28 am

    "Chris S." <chrisks at NOSPAMudel.edu> writes:

    I'm trying to make a graphical editor and browser for Pickled
    files. One aspect I'm not sure about is how to detect multiple
    references to the same data.

    For instance, say I had the Pickled data:
    a=[1,2,3]
    b=[a,4,5]
    c=[b,6,7]
    d=[a,b,c]

    The idea is to allow the user to browse this data and indicate
    references. In this case, if 'a' was selected, the browser should show
    that 'b', 'c', and 'd' contain a reference to 'a'. Inversely, if 'c'
    were selected, it should indicate that it's first element just isn't a
    list, but a reference to a list defined elsewhere, namely 'b'.
    I would recommend getting initmately familiar with the format of
    pickles (something I can't claim to be). The information must be in
    there somewhere.

    Actually, maybe you could write a custom unpickler that keeps track of
    this kind of information. Something else I haven't tried :-) But,
    seriously, I think the knowledge that these objects came out of a
    particular pickle is going to be a powerful advantage for this sort of
    thing, and you should use it for everything it's worth.
    Naturally, I could just recursively parse all the data comparing every
    element to every previously listed object, but is there a less
    obtrusive method? Python figures out when to delete objects based on
    the remaining references to an object. Is there a way to access this
    information to automatically lookup these references? Any help is
    greatly appreciated.
    Well, there's gc.get_referrers() and gc.get_referrents(), but I think
    they are probably overkill (see above).

    Cheers,
    mwh

    --
    (ps: don't feed the lawyers: they just lose their fear of humans)
    -- Peter Wood, comp.lang.lisp
  • Chris S. at Aug 6, 2004 at 4:40 pm

    Michael Hudson wrote:

    Naturally, I could just recursively parse all the data comparing every
    element to every previously listed object, but is there a less
    obtrusive method? Python figures out when to delete objects based on
    the remaining references to an object. Is there a way to access this
    information to automatically lookup these references? Any help is
    greatly appreciated.

    Well, there's gc.get_referrers() and gc.get_referrents(), but I think
    they are probably overkill (see above).
    Thanks, these are exactly what I need. Pickle is just the file format.
    The data are Python constructs, so I shouldn't have to bother with the
    format at all, since I'm simply converting the Pickled file into generic
    Python data. Plus, writing my own Pickle parser would be hugely
    complicated since Pickle is essentially its own small programming language.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedAug 6, '04 at 5:39a
activeAug 7, '04 at 1:38a
posts7
users4
websitepython.org

People

Translate

site design / logo © 2022 Grokbase