FAQ
Hello everyone,

I am trying to serialize a function, class, etc and transfer it, have it
unserialized and used. The thing is that this code is not defined on the
receiving side and thus it does not work. I tried the following tests:

Terminal A:
import pickle
def test(): pass
...
pickle.dumps(test)
'c__main__\ntest\np0\n.'
>>>

Terminal B:
import pickle
pickle.loads('c__main__\ntest\np0\n.')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/pickle.py", line 1374, in loads
return Unpickler(file).load()
File "/usr/lib/python2.5/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.5/pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "/usr/lib/python2.5/pickle.py", line 1126, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'test'


Ok, so pickle needs the code to be defined on both sides, so I tried
marshal:

Terminal A:
import marshal
marshal.dumps(test)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: unmarshallable object

Ok, as the docs say, marshal is quite limited, no functions or user
classes can be marshalled, I did get it to work like this though :
import inspect
marshal.dumps(inspect.getmembers(test, inspect.iscode)[0][1])
'c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00'

ok, but can I unmarshal it?

Terminal B:

>>>
marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00')
<code object test at 0xb7591578, file "<stdin>", line 1>

ok, it seams to work...

>>>
marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00')()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'code' object is not callable

ok, logical, it's a code object...

>>>
eval(marshal.loads('c\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x00\x00S(\x01\x00\x00\x00N(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00(\x00\x00\x00\x00s\x07\x00\x00\x00<stdin>t\x04\x00\x00\x00test\x01\x00\x00\x00s\x00\x00\x00\x00'))

ok, this works, not super pretty, but it works

now user objects?
class A(object): pass
...
a = A()
marshal.dumps(a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: unmarshallable object

ok, expected...
inspect.getmembers(a, inspect.iscode)
[]

ok, not cool, can't use the previous method...


Ok, now a word about what I'm trying to do, I'm trying to run python
code in a new process, like multiprocessing does, but without
multiprocessing. I'm trying to serialise the code because I 'd like to
do something like:

pseudo-code, serialize/unserialize/process are made up:

def newProcess(func, *args, **kwargs):
func = serialize(func)
args = serialize(args)
kwargs = serialize(kwargs)
process("python -c 'import serialize, unserialize;print
serialize(unserialize(\'%s\')(*unserialize(\'%s\'),
**unserialize(\'%s\')))'" % (func, args, kwargs))
#read result from stdout

I'm greatly simplifying this because I'm using a framework, but that's
the basic idea.

thanks,
Gabriel

Search Discussions

  • Paul Rubin at Apr 14, 2010 at 8:43 am

    Gabriel Rossetti <gabriel.rossetti at arimaz.com> writes:
    I am trying to serialize a function, class, etc and transfer it
    You mean the actual code? You have to use marshal rather than pickle,
    the Python versions have to be the same at both ends, and you better
    have some kind of authenticated transport to stop malicious code from
    getting accepted and run by the receiving end.
  • Gabriel Rossetti at Apr 14, 2010 at 9:33 am

    Paul Rubin wrote:
    Gabriel Rossetti <gabriel.rossetti at arimaz.com> writes:
    I am trying to serialize a function, class, etc and transfer it
    You mean the actual code? You have to use marshal rather than pickle,
    the Python versions have to be the same at both ends, and you better
    have some kind of authenticated transport to stop malicious code from
    getting accepted and run by the receiving end.
    Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
    my examples
  • Andreas Löscher at Apr 14, 2010 at 10:21 am

    Am Mittwoch, den 14.04.2010, 11:33 +0200 schrieb Gabriel Rossetti:
    Paul Rubin wrote:
    Gabriel Rossetti <gabriel.rossetti at arimaz.com> writes:
    I am trying to serialize a function, class, etc and transfer it
    You mean the actual code? You have to use marshal rather than pickle,
    the Python versions have to be the same at both ends, and you better
    have some kind of authenticated transport to stop malicious code from
    getting accepted and run by the receiving end.
    Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
    my examples
    The marshal module can be used to dump the Code of an Function (or
    anything else), like you have done. If you have the Code, you can
    reconstruct the function.
    import types
    import marshal
    def a(): pass
    ...
    s=marshal.dumps(a.__code__)
    f=types.FunctionType(marshal.loads(s), {})
    f
    <function a at 0x7f6308a66de8>

    The second parameter is the global scoop of the function.

    If you want to marshal a class, things get a little more complicated,
    because these consist of many code objects and you also need the code
    to assemble them together.

    There are two easy ways I can think of right now:

    1.
    Send the string of the code you want to transfer:
    s="""class A: pass"""
    and then:
    exec(s) in globals()
    2.
    Encapsule the stuff you want to send in a function.


    Best,
    Andreas
  • Gabriel Rossetti at Apr 14, 2010 at 10:37 am

    Andreas L?scher wrote:
    Am Mittwoch, den 14.04.2010, 11:33 +0200 schrieb Gabriel Rossetti:
    Paul Rubin wrote:
    Gabriel Rossetti <gabriel.rossetti at arimaz.com> writes:

    I am trying to serialize a function, class, etc and transfer it
    You mean the actual code? You have to use marshal rather than pickle,
    the Python versions have to be the same at both ends, and you better
    have some kind of authenticated transport to stop malicious code from
    getting accepted and run by the receiving end.
    Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
    my examples
    The marshal module can be used to dump the Code of an Function (or
    anything else), like you have done. If you have the Code, you can
    reconstruct the function.

    import types
    import marshal
    def a(): pass
    ...
    s=marshal.dumps(a.__code__)
    f=types.FunctionType(marshal.loads(s), {})
    f
    <function a at 0x7f6308a66de8>

    The second parameter is the global scoop of the function.

    If you want to marshal a class, things get a little more complicated,
    because these consist of many code objects and you also need the code
    to assemble them together.

    There are two easy ways I can think of right now:

    1.
    Send the string of the code you want to transfer:
    s="""class A: pass"""
    and then:
    exec(s) in globals()
    2.
    Encapsule the stuff you want to send in a function.


    Best,
    Andreas
    Thanks Andreas, I'll give that a try. I didn't really want to do the
    string method which is why I was trying to serialize.

    best,
    Gabriel
  • Andreas Löscher at Apr 14, 2010 at 11:47 am

    Am Mittwoch, den 14.04.2010, 12:37 +0200 schrieb Gabriel Rossetti:
    Andreas L?scher wrote:
    Am Mittwoch, den 14.04.2010, 11:33 +0200 schrieb Gabriel Rossetti:
    Paul Rubin wrote:
    Gabriel Rossetti <gabriel.rossetti at arimaz.com> writes:

    I am trying to serialize a function, class, etc and transfer it
    You mean the actual code? You have to use marshal rather than pickle,
    the Python versions have to be the same at both ends, and you better
    have some kind of authenticated transport to stop malicious code from
    getting accepted and run by the receiving end.
    Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
    my examples
    The marshal module can be used to dump the Code of an Function (or
    anything else), like you have done. If you have the Code, you can
    reconstruct the function.

    import types
    import marshal
    def a(): pass
    ...
    s=marshal.dumps(a.__code__)
    f=types.FunctionType(marshal.loads(s), {})
    f
    <function a at 0x7f6308a66de8>

    The second parameter is the global scoop of the function.

    If you want to marshal a class, things get a little more complicated,
    because these consist of many code objects and you also need the code
    to assemble them together.

    There are two easy ways I can think of right now:

    1.
    Send the string of the code you want to transfer:
    s="""class A: pass"""
    and then:
    exec(s) in globals()
    2.
    Encapsule the stuff you want to send in a function.


    Best,
    Andreas
    Thanks Andreas, I'll give that a try. I didn't really want to do the
    string method which is why I was trying to serialize.

    best,
    Gabriel
    You can compile the string befor you send it in an code object. This way
    you also don't need to compile it twice if you execute ist two times.
    s="""class A: pass"""
    co=compile(s, "foo.py", "exec")
    exec(marshal.loads(marshal.dumps(co))) in globals()
    But be aware, that the string method may be the most save method for
    doing this. Python is not ByteCode compatible between it's versions.
    But if I understand you right, this is not an issue for you.

    Best,
    Andreas
  • Gabriel Rossetti at Apr 15, 2010 at 1:40 pm

    Andreas L?scher wrote:
    Am Mittwoch, den 14.04.2010, 11:33 +0200 schrieb Gabriel Rossetti:
    Paul Rubin wrote:
    Gabriel Rossetti <gabriel.rossetti at arimaz.com> writes:

    I am trying to serialize a function, class, etc and transfer it
    You mean the actual code? You have to use marshal rather than pickle,
    the Python versions have to be the same at both ends, and you better
    have some kind of authenticated transport to stop malicious code from
    getting accepted and run by the receiving end.
    Yes, but I wasn't able to marshal/unmarshal them correctly as shown in
    my examples
    The marshal module can be used to dump the Code of an Function (or
    anything else), like you have done. If you have the Code, you can
    reconstruct the function.

    import types
    import marshal
    def a(): pass
    ...
    s=marshal.dumps(a.__code__)
    f=types.FunctionType(marshal.loads(s), {})
    f
    <function a at 0x7f6308a66de8>
    What version of python do you have? If I try your code above I get :
    import types
    import marshal
    def a(): pass
    ...
    s=marshal.dumps(a.__code__)
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    AttributeError: 'function' object has no attribute '__code__'

    The second parameter is the global scoop of the function.

    If you want to marshal a class, things get a little more complicated,
    because these consist of many code objects and you also need the code
    to assemble them together.

    There are two easy ways I can think of right now:

    1.
    Send the string of the code you want to transfer:
    s="""class A: pass"""
    and then:
    exec(s) in globals()
    2.
    Encapsule the stuff you want to send in a function.


    Best,
    Andreas
  • Andreas Löscher at Apr 15, 2010 at 3:59 pm

    import types
    import marshal
    def a(): pass
    ...
    s=marshal.dumps(a.__code__)
    f=types.FunctionType(marshal.loads(s), {})
    f
    <function a at 0x7f6308a66de8>
    What version of python do you have? If I try your code above I get :
    import types
    import marshal
    def a(): pass
    ...
    s=marshal.dumps(a.__code__)
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    AttributeError: 'function' object has no attribute '__code__'
    I used Version 2.6 for this. __code__ is just an alias for func_code.
    def a(): pass
    ...
    s=marshal.dumps(a.func_code)
    This should allways work, unless the implementation of the function
    object changes. (I have tested it under 2.4, 2.5 and 2.6, but it should
    also work in further versions)

    Best,
    Andreas
  • Gabriel Rossetti at Apr 16, 2010 at 7:25 am

    Andreas L?scher wrote:
    import types
    import marshal
    def a(): pass
    ...

    s=marshal.dumps(a.__code__)
    f=types.FunctionType(marshal.loads(s), {})
    f
    <function a at 0x7f6308a66de8>
    What version of python do you have? If I try your code above I get :
    import types
    import marshal
    def a(): pass
    ...
    s=marshal.dumps(a.__code__)
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    AttributeError: 'function' object has no attribute '__code__'
    I used Version 2.6 for this. __code__ is just an alias for func_code.

    def a(): pass
    ...
    s=marshal.dumps(a.func_code)
    This should allways work, unless the implementation of the function
    object changes. (I have tested it under 2.4, 2.5 and 2.6, but it should
    also work in further versions)

    Best,
    Andreas
    Yes, it works, thank you,
    Gabriel
  • Daniel Fetchinson at Apr 16, 2010 at 9:47 am

    I am trying to serialize a function, class, etc and transfer it, have it
    unserialized and used.
    You might want to look at pyro: http://pyro.sourceforge.net/ and also
    picloud: http://www.picloud.com/

    HTH,
    Daniel



    --
    Psss, psss, put it down! - http://www.cafepress.com/putitdown

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedApr 14, '10 at 8:23a
activeApr 16, '10 at 9:47a
posts10
users4
websitepython.org

People

Translate

site design / logo © 2021 Grokbase