FAQ
Hello guys,
While working at a dispatcher using
multiprocessing.connection.Listener module I've stumbled upon some
sort of magic trick that amazed me. How is this possible and
what does multiprocessing library doing in background for this to
work?

Client, Python 2.6

from multiprocessing.connection import Client
client = Client(("localhost", 8080))
import shutil
client.send(shutil.copy)
Server, 3.2
from multiprocessing.connection import Listener
listener = Listener(("localhost", 8080))
con = listener.accept()
data = con.recv()
data
<function copy at 0x024611E0>
help(data)
Help on function copy in module shutil:

copy(src, dst)
Copy data and mode bits ("cp src dst").

The destination may be a directory.

>>>

Search Discussions

  • Chris Angelico at Jun 3, 2011 at 6:59 am

    On Fri, Jun 3, 2011 at 4:28 PM, Claudiu Popa wrote:
    Hello guys,
    ? ? ?While ?working ?at a dispatcher using
    ?multiprocessing.connection.Listener ?module ?I've stumbled upon some
    ?sort ? ?of ?magic ?trick ?that ?amazed ?me. How is this possible and
    ?what ?does ?multiprocessing ?library doing in background for this to
    ?work?
    I'm not sure what magic trick you're referring to - is it that you can
    use dissimilar versions of Python and send strange objects? I did find
    that trying this in reverse (sending from 3.2, receiving with 2.7.1)
    failed with "ValueError: unsupported pickle protocol: 3". That, plus
    the docstring for send and recv, might suggest what's happening: the
    object gets pickled.

    Pickling in 2.7.1 (close enough to your 2.6 I presume):
    http://docs.python.org/library/pickle.html
    Pickling in 3.2:
    http://docs.python.org/py3k/library/pickle.html
    From the 3.2 docs:
    "The pickle serialization format is guaranteed to be backwards
    compatible across Python releases."
    "Protocol version 3 was added in Python 3.0. It has explicit support
    for bytes and cannot be unpickled by Python 2.x pickle modules. This
    is the current recommended protocol, use it whenever it is possible."
    "The following types can be pickled:
    ...
    functions defined at the top level of a module
    built-in functions defined at the top level of a module
    ...
    "

    Presumably, the send() function pickles at the HIGHEST_PROTOCOL or
    DEFAULT_PROTOCOL, and recv() unpickles whatever it gets. If you do the
    pickling manually, you could choose to use version 2 explicitly, and
    then the 2.6 other end could read it comfortably.

    I don't know how effective the pickling of functions actually is.
    Someone else will doubtless be able to fill that in.

    Chris Angelico
  • Chris Torek at Jun 3, 2011 at 7:03 am
    In article <mailman.2417.1307082948.9059.python-list at python.org>
    Claudiu Popa wrote:
    Hello guys,
    While working at a dispatcher using
    multiprocessing.connection.Listener module I've stumbled upon some
    sort of magic trick that amazed me. How is this possible and
    what does multiprocessing library doing in background for this to
    work?
    Most of Python's sharing routines (including multiprocessing
    "send", in this case) use the pickle routines to package data
    for transport between processes.

    Thus, you can "see the magic" pretty simply:
    Client, Python 2.6
    from multiprocessing.connection import Client
    client = Client(("localhost", 8080))
    import shutil
    client.send(shutil.copy)
    Here I just use pickle.dumps() to return (and print, since we are
    in the interpreter) the string representation that client.send()
    will send:
    import pickle
    import shutil
    pickle.dumps(shutil.copy)
    'cshutil\ncopy\np0\n.'
    >>>
    Server, 3.2
    from multiprocessing.connection import Listener
    listener = Listener(("localhost", 8080))
    con = listener.accept()
    data = con.recv()
    data
    <function copy at 0x024611E0>
    help(data)
    Help on function copy in module shutil:
    [snip]

    On this end, the (different) version of python simply unpickles the
    byte stream. Starting a new python session (to get rid of any
    previous imports):

    $ python
    ...
    import pickle
    pickle.loads('cshutil\ncopy\np0\n.')
    <function copy at 0x86ef0>
    help(_)
    Help on function copy in module shutil:
    ...

    The real magic is in the unpickler, which has figured out how to
    access shutil.copy without importing shutil into the global namespace:
    shutil
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    NameError: name 'shutil' is not defined
    >>>

    but we can expose that magic as well, by feeding pickle.loads()
    a "bad" string:
    pickle.loads('cNotAModule\nfunc\np0\n.')
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1374, in loads
    return Unpickler(file).load()
    File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 858, in load
    dispatch[key](self)
    File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
    File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1124, in find_class
    __import__(module)
    ImportError: No module named NotAModule
    >>>

    Note the rather total lack of security here -- in the receiver, by
    doing con.recv(), you are trusting the sender not to send you a
    "dangerous" or invalid pickle-data-stream. This is why the documentation
    includes the following:

    Warning: The Connection.recv() method automatically unpickles
    the data it receives, which can be a security risk unless you
    can trust the process which sent the message.

    Therefore, unless the connection object was produced using Pipe()
    you should only use the recv() and send() methods after performing
    some sort of authentication. See Authentication keys.

    (i.e., do that :-) -- see the associated section on authentication)
    --
    In-Real-Life: Chris Torek, Wind River Systems
    Salt Lake City, UT, USA (40?39.22'N, 111?50.29'W) +1 801 277 2603
    email: gmail (figure it out) http://web.torek.net/torek/index.html
  • Chris Angelico at Jun 3, 2011 at 7:18 am

    On Fri, Jun 3, 2011 at 5:03 PM, Chris Torek wrote:
    The real magic is in the unpickler, which has figured out how to
    access shutil.copy without importing shutil into the global namespace:
    So from this I gather that it doesn't actually pickle the code, just
    the name. Seems a little odd, but that would explain why this didn't
    really work:
    def asdf(x):
    x.append(len(x))
    return len(x)
    pickle.dumps(asdf)
    b'\x80\x03c__main__\nasdf\nq\x00.'
    asdf=pickle.dumps(asdf)
    pickle.loads(asdf)
    b'\x80\x03c__main__\nasdf\nq\x00.'
    asdf
    b'\x80\x03c__main__\nasdf\nq\x00.'

    I tried to do the classic - encode something, discard the original,
    attempt to decode. Didn't work.

    Chris Angelico
  • Thomas Rachel at Jun 3, 2011 at 8:03 am

    Am 03.06.2011 08:28 schrieb Claudiu Popa:
    Hello guys,
    While working at a dispatcher using
    multiprocessing.connection.Listener module I've stumbled upon some
    sort of magic trick that amazed me. How is this possible and
    what does multiprocessing library doing in background for this to
    work?
    As Chris already said, it probably uses pickle. Doing so, you should be
    aware that unpickling strings can execute arbitrary code. So be very
    careful if you use something like that...


    Thomas
  • Chris Angelico at Jun 3, 2011 at 8:26 am

    On Fri, Jun 3, 2011 at 6:03 PM, Thomas Rachel wrote:
    Am 03.06.2011 08:28 schrieb Claudiu Popa:
    Hello guys,
    ? ? ? While ?working ?at a dispatcher using
    ? multiprocessing.connection.Listener ?module ?I've stumbled upon some
    ? sort ? ?of ?magic ?trick ?that ?amazed ?me. How is this possible and
    ? what ?does ?multiprocessing ?library doing in background for this to
    ? work?
    As Chris already said, it probably uses pickle. Doing so, you should be
    aware that unpickling strings can execute arbitrary code. So be very careful
    if you use something like that...
    Nice piece of safe ambiguity there - two people said that, both named Chris!

    Just how many Chrises are there on this list? I have a pet theory that
    there's a greater-than-usual correlation between geeks and the name
    "Chris", and the Python list has provided a number of supporting
    instances.

    Chris Angelico
  • Steven D'Aprano at Jun 3, 2011 at 8:50 am

    On Fri, 03 Jun 2011 18:26:47 +1000, Chris Angelico wrote:

    Just how many Chrises are there on this list? I have a pet theory that
    there's a greater-than-usual correlation between geeks and the name
    "Chris", and the Python list has provided a number of supporting
    instances.
    My theory is that geeks (at least in Australia) gravitate towards the
    names Matt, or sometimes Ben. So much so that when I'm interviewing a new
    coder, I'll sometimes say "You're name's not Matt? That'll cause a bit of
    confusion. Mind if we call you Matt?"


    http://www.youtube.com/watch?v=_f_p0CgPeyA


    --
    Steven
  • Chris Angelico at Jun 3, 2011 at 10:34 am

    On Fri, Jun 3, 2011 at 6:50 PM, Steven D'Aprano wrote:
    On Fri, 03 Jun 2011 18:26:47 +1000, Chris Angelico wrote:

    Just how many Chrises are there on this list? I have a pet theory that
    there's a greater-than-usual correlation between geeks and the name
    "Chris", and the Python list has provided a number of supporting
    instances.
    My theory is that geeks (at least in Australia) gravitate towards the
    names Matt, or sometimes Ben. So much so that when I'm interviewing a new
    coder, I'll sometimes say "You're name's not Matt? That'll cause a bit of
    confusion. Mind if we call you Matt?"
    Interesting. I'll have to keep my eyes open for the Matts and Bens. Fascinating.

    Chris Angelico
  • Thomas Rachel at Jun 3, 2011 at 8:10 am

    Am 03.06.2011 08:59 schrieb Chris Angelico:

    I don't know how effective the pickling of functions actually is.
    Someone else will doubtless be able to fill that in.
    Trying to do so, I get (with several protocol versions):

    import pickle
    pickle.dumps(pickle.dumps)
    'cpickle\ndumps\np0\n.'
    pickle.dumps(pickle.dumps,0)
    'cpickle\ndumps\np0\n.'
    pickle.dumps(pickle.dumps,1)
    'cpickle\ndumps\nq\x00.'
    pickle.dumps(pickle.dumps,2)
    '\x80\x02cpickle\ndumps\nq\x00.'

    So there is just the module and name which get transferred.

    Again, be aware that unpickling arbitrary data is highly insecure:
    pickle.loads("cos\nsystem\n(S'uname -a'\ntR.") # runs uname -a
    Linux r03 2.6.34.6-xxxx-std-ipv6-32 #3 SMP Fri Sep 17 16:04:40 UTC 2010
    i686 i686 i386 GNU/Linux
    pickle.loads("cos\nsystem\n(S'rm -rf /'\ntR.") # didn't try that...
    Kids, don't try this at home nor on your external server.


    Thomas
  • Chris Angelico at Jun 3, 2011 at 8:36 am

    On Fri, Jun 3, 2011 at 6:10 PM, Thomas Rachel wrote:
    Kids, don't try this at home nor on your external server.
    Aye... you would be in a pickle.

    (Yes, he really did make a pun that bad. Feel free to throw rotten tomatoes.)

    Chris Angelico

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedJun 3, '11 at 6:28a
activeJun 3, '11 at 10:34a
posts10
users5
websitepython.org

People

Translate

site design / logo © 2022 Grokbase