FAQ
Hello,


Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
the PEP 3154 implementation - pickle protocol 4.


I think it would be good to get the PEP and the implementation accepted
for 3.4. As far as I can say, this has been a low-controvery proposal,
and it brings fairly obvious improvements to the table (which table?).
I still need some kind of BDFL or BDFL delegate to do that, though --
unless I am allowed to mark my own PEP accepted :-)


(I've asked Tim, specifically, for comments, since he contributed a lot
to previous versions of the pickle protocol.)


The PEP is at http://www.python.org/dev/peps/pep-3154/ (should be
rebuilt soon by the server, I'd say)


Alexandre's implementation is tracked at
http://bugs.python.org/issue17810


Regards


Antoine.

Search Discussions

  • Eric Snow at Nov 16, 2013 at 6:44 pm

    On Sat, Nov 16, 2013 at 11:15 AM, Antoine Pitrou wrote:
    Hello,

    Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
    the PEP 3154 implementation - pickle protocol 4.

    I think it would be good to get the PEP and the implementation accepted
    for 3.4.

    +1


    Once the core portion of the PEP 451 implementation is done, I plan on
    tweaking pickle to take advantage of module.__spec__ (where
    applicable). It would be nice to have the PEP 3154 implementation in
    place before I do anything in that space.


    -eric
  • Nick Coghlan at Nov 17, 2013 at 1:41 am

    On 17 Nov 2013 04:45, "Eric Snow" wrote:
    On Sat, Nov 16, 2013 at 11:15 AM, Antoine Pitrou wrote:

    Hello,

    Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
    the PEP 3154 implementation - pickle protocol 4.

    I think it would be good to get the PEP and the implementation accepted
    for 3.4.
    +1

    Once the core portion of the PEP 451 implementation is done, I plan on
    tweaking pickle to take advantage of module.__spec__ (where
    applicable). It would be nice to have the PEP 3154 implementation in
    place before I do anything in that space.

    +1 from me, too.


    Cheers,
    Nick.

    -eric
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131117/d28aa35d/attachment.html>
  • Guido van Rossum at Nov 17, 2013 at 2:27 am

    On Sat, Nov 16, 2013 at 10:15 AM, Antoine Pitrou wrote:


    Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
    the PEP 3154 implementation - pickle protocol 4.

    I think it would be good to get the PEP and the implementation accepted
    for 3.4. As far as I can say, this has been a low-controvery proposal,
    and it brings fairly obvious improvements to the table (which table?).
    I still need some kind of BDFL or BDFL delegate to do that, though --
    unless I am allowed to mark my own PEP accepted :-)

    (I've asked Tim, specifically, for comments, since he contributed a lot
    to previous versions of the pickle protocol.)

    The PEP is at http://www.python.org/dev/peps/pep-3154/ (should be
    rebuilt soon by the server, I'd say)

    Alexandre's implementation is tracked at
    http://bugs.python.org/issue17810

    Assuming Tim doesn't object (hi Tim!) I think this PEP is fine to accept --
    all the ideas sound good, and I agree with moving to support 8-byte sizes
    and framing. I haven't looked at the implementation but I trust you as a
    reviewer and the beta release process.


    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131116/e5c3f3b9/attachment.html>
  • Tim Peters at Nov 18, 2013 at 5:53 am
    [Antoine Pitrou]
    Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
    the PEP 3154 implementation - pickle protocol 4.

    I think it would be good to get the PEP and the implementation accepted
    for 3.4. As far as I can say, this has been a low-controvery proposal,
    and it brings fairly obvious improvements to the table (which table?).
    I still need some kind of BDFL or BDFL delegate to do that, though --
    unless I am allowed to mark my own PEP accepted :-)

    Try it and see whether anyone complains ;-)


    I like it. I didn't review the code, but the PEP addresses real
    issues, and the solutions look good on paper ;-)


    One thing I wonder about: I don't know that non-seekable pickle
    streams are important use cases, but am willing to be told that they
    are. In which case, framing is a great idea.


    But I wonder why it isn't done with a new framing opcode instead (say,
    FRAME followed by 8-byte count). I suppose that would be like the
    "prefetch" idea, except that framing opcodes would be mandatory
    (instead of optional) in proto 4. Why I initially like that:


    - Uniform decoding loop ("the next thing" _always_ starts with an opcode).


    - Some easy sanity checking due to the tiny redundancy (if the byte
    immediately following the current frame is not a FRAME opcode, the
    pickle is corrupt; and also corrupt if a FRAME opcode is encountered
    _inside_ the current frame).


    When slinging 8-byte counts, _some_ sanity-checking seems like a good idea ;-)
  • Serhiy Storchaka at Nov 18, 2013 at 11:28 am

    18.11.13 07:53, Tim Peters ???????(??):
    - Some easy sanity checking due to the tiny redundancy (if the byte
    immediately following the current frame is not a FRAME opcode, the
    pickle is corrupt; and also corrupt if a FRAME opcode is encountered
    _inside_ the current frame).

    For efficient unpickling a FRAME opcode followed by 8-byte count should
    be *last* thing in a frame (unless it is a last frame).
  • Guido van Rossum at Nov 18, 2013 at 3:48 pm

    On Mon, Nov 18, 2013 at 3:28 AM, Serhiy Storchaka wrote:


    18.11.13 07:53, Tim Peters ???????(??):

    - Some easy sanity checking due to the tiny redundancy (if the byte
    immediately following the current frame is not a FRAME opcode, the
    pickle is corrupt; and also corrupt if a FRAME opcode is encountered
    _inside_ the current frame).
    For efficient unpickling a FRAME opcode followed by 8-byte count should be
    *last* thing in a frame (unless it is a last frame).

    I don't understand that.


    Clearly the framing is the weakest point of the PEP (== elicits the most
    bikeshedding). I am also unsure about the value of framing when pickles are
    written to strings.


    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131118/2e1d4688/attachment.html>
  • Antoine Pitrou at Nov 18, 2013 at 4:10 pm

    On Mon, 18 Nov 2013 07:48:27 -0800 Guido van Rossum wrote:
    On Mon, Nov 18, 2013 at 3:28 AM, Serhiy Storchaka wrote:

    18.11.13 07:53, Tim Peters ???????(??):

    - Some easy sanity checking due to the tiny redundancy (if the byte
    immediately following the current frame is not a FRAME opcode, the
    pickle is corrupt; and also corrupt if a FRAME opcode is encountered
    _inside_ the current frame).
    For efficient unpickling a FRAME opcode followed by 8-byte count should be
    *last* thing in a frame (unless it is a last frame).
    I don't understand that.

    Clearly the framing is the weakest point of the PEP (== elicits the most
    bikeshedding). I am also unsure about the value of framing when pickles are
    written to strings.

    It hasn't much value in that case, but the cost is also small (8 bytes
    every 64KB, roughly).


    Regards


    Antoine.
  • Guido van Rossum at Nov 18, 2013 at 6:07 pm

    On Mon, Nov 18, 2013 at 8:10 AM, Antoine Pitrou wrote:


    On Mon, 18 Nov 2013 07:48:27 -0800
    Guido van Rossum wrote:
    On Mon, Nov 18, 2013 at 3:28 AM, Serhiy Storchaka <storchaka at gmail.com
    wrote:
    18.11.13 07:53, Tim Peters ???????(??):

    - Some easy sanity checking due to the tiny redundancy (if the byte
    immediately following the current frame is not a FRAME opcode, the
    pickle is corrupt; and also corrupt if a FRAME opcode is encountered
    _inside_ the current frame).
    For efficient unpickling a FRAME opcode followed by 8-byte count
    should be
    *last* thing in a frame (unless it is a last frame).
    I don't understand that.

    Clearly the framing is the weakest point of the PEP (== elicits the most
    bikeshedding). I am also unsure about the value of framing when pickles are
    written to strings.
    It hasn't much value in that case, but the cost is also small (8 bytes
    every 64KB, roughly).

    That's small if your pickle is large, but for small pickles it can add up.
    Still, not enough to reject the PEP. Just get Tim to agree with you, or
    switch to Tim's proposal.


    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131118/7e9765a6/attachment.html>
  • Tim Peters at Nov 18, 2013 at 10:02 pm
    [Guido]
    Clearly the framing is the weakest point of the PEP (== elicits the most
    bikeshedding). I am also unsure about the value of framing when pickles are
    written to strings.

    [Antoine]
    It hasn't much value in that case,

    It has _no_ value in that case, yes? It doesn't appear to have _much_
    value in the case of a seekable stream, either - the implementation
    has always been free to read ahead then. The real value appears to be
    in cases of non-seekable streams.



    but the cost is also small (8 bytes every 64KB, roughly).
    That's small if your pickle is large, but for small pickles it can add up.

    Which is annoying. It was already annoying when the PROTO opcode was
    introduced, and the size of small pickles increased by 2 bytes. That
    added up too :-(



    Still, not enough to reject the PEP. Just get Tim to agree with you,
    or switch to Tim's proposal.

    I just asked a question ;-) If a mandatory proto 4 FRAME opcode were
    added, it would just increase the bloat for small pickles (up from the
    currently proposed 8 bytes of additional overhead to 9).
  • Antoine Pitrou at Nov 18, 2013 at 10:08 pm

    On Mon, 18 Nov 2013 16:02:31 -0600 Tim Peters wrote:
    [Guido]
    Clearly the framing is the weakest point of the PEP (== elicits the most
    bikeshedding). I am also unsure about the value of framing when pickles are
    written to strings.
    [Antoine]
    It hasn't much value in that case,
    It has _no_ value in that case, yes? It doesn't appear to have _much_
    value in the case of a seekable stream, either - the implementation
    has always been free to read ahead then. The real value appears to be
    in cases of non-seekable streams.

    but the cost is also small (8 bytes every 64KB, roughly).
    That's small if your pickle is large, but for small pickles it can add up.
    Which is annoying. It was already annoying when the PROTO opcode was
    introduced, and the size of small pickles increased by 2 bytes. That
    added up too :-(

    Are very small pickles that size-sensitive? I have the impression that
    if 8 bytes vs. e.g. 15 bytes makes a difference for your application,
    you'd be better off with a hand-made format.


    Regards


    Antoine.
  • Tim Peters at Nov 18, 2013 at 10:25 pm
    [Tim]
    ...
    It was already annoying when the PROTO opcode was introduced,
    and the size of small pickles increased by 2 bytes. That
    added up too :-(

    [Antoine]
    Are very small pickles that size-sensitive? I have the impression that
    if 8 bytes vs. e.g. 15 bytes makes a difference for your application,
    you'd be better off with a hand-made format.

    The difference between 8 and 15 is, e.g., nearly doubling the amount
    of network traffic (for apps that use pickles across processes or
    machines).


    A general approach has no way to guess how it will be used. For
    example, `multiprocessing` uses pickles extensively for inter-process
    communication of Python data. Some users try broadcasting giant
    arrays across processes, while others broadcast oceans of tiny
    integers (like indices into giant arrays inherited via fork()).


    Since pickle intends to be "the" Python serialization format, it
    really should try to be friendly for all plausible uses.
  • Antoine Pitrou at Nov 18, 2013 at 10:39 pm

    On Mon, 18 Nov 2013 16:25:07 -0600 Tim Peters wrote:

    [Antoine]
    Are very small pickles that size-sensitive? I have the impression that
    if 8 bytes vs. e.g. 15 bytes makes a difference for your application,
    you'd be better off with a hand-made format.
    The difference between 8 and 15 is, e.g., nearly doubling the amount
    of network traffic (for apps that use pickles across processes or
    machines).

    A general approach has no way to guess how it will be used. For
    example, `multiprocessing` uses pickles extensively for inter-process
    communication of Python data. Some users try broadcasting giant
    arrays across processes, while others broadcast oceans of tiny
    integers (like indices into giant arrays inherited via fork()).

    Well, sending oceans of tiny integers will also incur many system calls
    and additional synchronization costs, since sending data on a
    multiprocessing Queue has to acquire a semaphore. So it generally
    sounds like a bad idea, IMHO.


    That said, I agree with:
    Since pickle intends to be "the" Python serialization format, it
    really should try to be friendly for all plausible uses.

    I simply don't think adding a fixed 8-byte overhead is actually
    annoying. It's less than the PyObject overhead in 64-bit mode...


    Regards


    Antoine.
  • Tim Peters at Nov 18, 2013 at 10:54 pm
    [Antoine]
    Well, sending oceans of tiny integers will also incur many system calls
    and additional synchronization costs, since sending data on a
    multiprocessing Queue has to acquire a semaphore. So it generally
    sounds like a bad idea, IMHO.

    That said, I agree with:
    Since pickle intends to be "the" Python serialization format, it
    really should try to be friendly for all plausible uses.
    I simply don't think adding a fixed 8-byte overhead is actually
    annoying. It's less than the PyObject overhead in 64-bit mode...

    A long-running process can legitimately put billions of items on work
    queues, far more than could ever fit in RAM simultaneously. Comparing
    this to PyObject overhead makes no sense to me. Neither does the line
    of argument "there are several kinds of overheads, so making this
    overhead worse too doesn't matter".


    When possible, we should strive not to add overheads that don't repay
    their costs. For small pickles, an 8-byte size field doesn't appear
    to buy anything. But I appreciate that it costs implementation effort
    to avoid producing it in these cases.
  • Nick Coghlan at Nov 18, 2013 at 11:51 pm

    On 19 Nov 2013 09:33, "Tim Peters" wrote:
    [Antoine]
    Well, sending oceans of tiny integers will also incur many system calls
    and additional synchronization costs, since sending data on a
    multiprocessing Queue has to acquire a semaphore. So it generally
    sounds like a bad idea, IMHO.

    That said, I agree with:
    Since pickle intends to be "the" Python serialization format, it
    really should try to be friendly for all plausible uses.
    I simply don't think adding a fixed 8-byte overhead is actually
    annoying. It's less than the PyObject overhead in 64-bit mode...
    A long-running process can legitimately put billions of items on work
    queues, far more than could ever fit in RAM simultaneously. Comparing
    this to PyObject overhead makes no sense to me. Neither does the line
    of argument "there are several kinds of overheads, so making this
    overhead worse too doesn't matter".

    When possible, we should strive not to add overheads that don't repay
    their costs. For small pickles, an 8-byte size field doesn't appear
    to buy anything. But I appreciate that it costs implementation effort
    to avoid producing it in these cases.

    Given that cases where the overhead matters can drop back to proto 3 if
    absolutely necessary, perhaps it's reasonable to just run with the current
    proposal?


    Cheers,
    Nick.

    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131119/8557b500/attachment.html>
  • Richard Oudkerk at Nov 18, 2013 at 11:31 pm

    On 18/11/2013 10:25pm, Tim Peters wrote:
    The difference between 8 and 15 is, e.g., nearly doubling the amount
    of network traffic (for apps that use pickles across processes or
    machines).

    I tried using multiprocessing.Pipe() and send_bytes()/recv_bytes() to
    send messages between processes:


        8 bytes messages -- 525,000 msgs/sec
        15 bytes messages -- 556,000 msgs/sec


    So the size of small messages does not seem to make much difference.


    (This was in a Linux VM with Python 2.7. Python 3.3 is much slower
    because it is not implemented in C, but gives similarly close results.)


    --
    Richard
  • Tim Peters at Nov 19, 2013 at 12:55 am
    [Richard Oudkerk]
    I tried using multiprocessing.Pipe() and send_bytes()/recv_bytes() to send
    messages between processes:

    8 bytes messages -- 525,000 msgs/sec
    15 bytes messages -- 556,000 msgs/sec

    So the size of small messages does not seem to make much difference.

    To the contrary, the lesson is clear: to speed up multiprocessing,
    the larger the messages the faster it goes ;-)
  • Richard Oudkerk at Nov 19, 2013 at 1:09 am

    On 19/11/2013 12:55am, Tim Peters wrote:
    [Richard Oudkerk]
    I tried using multiprocessing.Pipe() and send_bytes()/recv_bytes() to send
    messages between processes:

    8 bytes messages -- 525,000 msgs/sec
    15 bytes messages -- 556,000 msgs/sec

    So the size of small messages does not seem to make much difference.
    To the contrary, the lesson is clear: to speed up multiprocessing,
    the larger the messages the faster it goes ;-)

    Ah, yes. It was probably round the other way.


    --
    Richard
  • Larry Hastings at Nov 26, 2013 at 10:39 pm

    On 11/18/2013 07:48 AM, Guido van Rossum wrote:
    Clearly the framing is the weakest point of the PEP (== elicits the
    most bikeshedding).

    Indeed--it is still ongoing:


         http://bugs.python.org/issue19780






    //arry/
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131126/7705e197/attachment.html>
  • Antoine Pitrou at Nov 18, 2013 at 4:14 pm

    On Sun, 17 Nov 2013 23:53:09 -0600 Tim Peters wrote:

    But I wonder why it isn't done with a new framing opcode instead (say,
    FRAME followed by 8-byte count). I suppose that would be like the
    "prefetch" idea, except that framing opcodes would be mandatory
    (instead of optional) in proto 4. Why I initially like that:

    - Uniform decoding loop ("the next thing" _always_ starts with an opcode).

    But it's not actually uniform. A frame isn't a normal opcode, it's a
    large section of bytes that contains potentially many opcodes.


    The framing layer is really below the opcode layer, it makes also sense
    to implement it like that.


    (I also tried to implement Serhiy's PREFETCH idea, but it didn't bring
    any actual simplification)

    When slinging 8-byte counts, _some_ sanity-checking seems like a good idea ;-)

    I don't know. It's not much worse (for denial of service opportunities)
    than a 4-byte count, which already exists in earlier protocols.


    Regards


    Antoine.
  • Antoine Pitrou at Nov 18, 2013 at 6:41 pm

    On Mon, 18 Nov 2013 17:14:21 +0100 Antoine Pitrou wrote:
    On Sun, 17 Nov 2013 23:53:09 -0600
    Tim Peters wrote:
    But I wonder why it isn't done with a new framing opcode instead (say,
    FRAME followed by 8-byte count). I suppose that would be like the
    "prefetch" idea, except that framing opcodes would be mandatory
    (instead of optional) in proto 4. Why I initially like that:

    - Uniform decoding loop ("the next thing" _always_ starts with an opcode).
    But it's not actually uniform. A frame isn't a normal opcode, it's a
    large section of bytes that contains potentially many opcodes.

    The framing layer is really below the opcode layer, it makes also sense
    to implement it like that.

    (I also tried to implement Serhiy's PREFETCH idea, but it didn't bring
    any actual simplification)
    When slinging 8-byte counts, _some_ sanity-checking seems like a good idea ;-)
    I don't know. It's not much worse (for denial of service opportunities)
    than a 4-byte count, which already exists in earlier protocols.

    Actually, now that I think of it, it's even better. A 2**63 bytes
    allocation is guaranteed to fail, since most 64-bit CPUs have a smaller
    address space than that (for example, x86-64 CPUs seem to have a 48
    bits virtual address space).


    On the other hand, a 2**31 bytes allocation may very well succeed, eat
    almost all the RAM and/or slow down the machine by swapping out.


    Regards


    Antoine.
  • Tim Peters at Nov 18, 2013 at 10:18 pm
    [Tim]
    But I wonder why it isn't done with a new framing opcode instead (say,
    FRAME followed by 8-byte count). I suppose that would be like the
    "prefetch" idea, except that framing opcodes would be mandatory
    (instead of optional) in proto 4. Why I initially like that:

    - Uniform decoding loop ("the next thing" _always_ starts with an opcode).
    But it's not actually uniform. A frame isn't a normal opcode, it's a
    large section of bytes that contains potentially many opcodes.

    The framing layer is really below the opcode layer, it makes also sense
    to implement it like that.

    That makes sense to me.



    (I also tried to implement Serhiy's PREFETCH idea, but it didn't bring
    any actual simplification)

    But it has a different kind of advantage: PREFETCH was optional. As
    Guido said, it's annoying to bloat the size of small pickles (which
    may, although individually small, occur in great numbers) by 8 bytes
    each. There's really no point to framing small chunks of data, right?


    Which leads to another idea: after the PROTO opcode, there is, or is
    not, an optional PREFETCH opcde with an 8-byte argument. If the
    PREFETCH opcode exists, then it gives the number of bytes up to and
    including the pickle's STOP opcode. So there's exactly 0 or 1
    PREFETCH opcodes per pickle.


    Is there an advantage to spraying multiple 8-byte "frame counts"
    throughout a pickle stream? 8 bytes is surely enough to specify the
    size of any single pickle for half a generation ;-) to come.

    When slinging 8-byte counts, _some_ sanity-checking seems like a good idea ;-)
    I don't know. It's not much worse (for denial of service opportunities)
    than a 4-byte count, which already exists in earlier protocols.

    I'm not thinking of DOS at all, just general sanity as data objects
    get larger & larger. Pickles have almost no internal checks now. But
    I've seen my share of corrupted pickles! About the only thing that
    catches them early is hitting a byte that isn't a legitimate pickle
    opcode. That _used_ to be a much stronger check than it is now,
    because the 8-bit opcode space was sparsely populated at first. But,
    over time, more and more opcodes get added, so the chance of mistaking
    a garbage byte for a legit opcode has increased correspondingly.


    A PREFETCH opcode with a "bytes until STOP" makes for a decent bad ;-)
    sanity check too ;-)
  • Antoine Pitrou at Nov 18, 2013 at 10:31 pm

    On Mon, 18 Nov 2013 16:18:21 -0600 Tim Peters wrote:

    But it has a different kind of advantage: PREFETCH was optional. As
    Guido said, it's annoying to bloat the size of small pickles (which
    may, although individually small, occur in great numbers) by 8 bytes
    each. There's really no point to framing small chunks of data, right?

    You can't know how much space the pickle will take until the pickling
    ends, though, which makes it difficult to decide whether you want to
    emit a PREFETCH opcode or not.

    Which leads to another idea: after the PROTO opcode, there is, or is
    not, an optional PREFETCH opcde with an 8-byte argument. If the
    PREFETCH opcode exists, then it gives the number of bytes up to and
    including the pickle's STOP opcode. So there's exactly 0 or 1
    PREFETCH opcodes per pickle.

    Is there an advantage to spraying multiple 8-byte "frame counts"
    throughout a pickle stream?

    Well, yes: much better memory usage for large pickles.
    Some people use pickles to store huge data, which was the motivation to
    add the 8-byte-size opcodes after all.

    I'm not thinking of DOS at all, just general sanity as data objects
    get larger & larger. Pickles have almost no internal checks now. But
    I've seen my share of corrupted pickles!

    IMO, any validation should use a dedicated CRC-like scheme, rather than
    relying on the fact that correct pickles are statistically unlikely :-)


    (or you can implicitly rely on TCP or UDP checksums, or you disk
    subsystem's own sanity checks)


    Regards


    Antoine.
  • Tim Peters at Nov 18, 2013 at 10:44 pm
    [Tim]
    But it has a different kind of advantage: PREFETCH was optional. As
    Guido said, it's annoying to bloat the size of small pickles (which
    may, although individually small, occur in great numbers) by 8 bytes
    each. There's really no point to framing small chunks of data, right?

    [Antoine]
    You can't know how much space the pickle will take until the pickling
    ends, though, which makes it difficult to decide whether you want to
    emit a PREFETCH opcode or not.

    Ah, of course. Presumably the outgoing pickle stream is first stored
    in some memory buffer, right? If pickling completes before the buffer
    is first flushed, then you know exactly how large the entire pickle
    is. If "it's small" (say, < 100 bytes), don't write out the PREFETCH
    part. Else do.



    Which leads to another idea: after the PROTO opcode, there is, or is
    not, an optional PREFETCH opcde with an 8-byte argument. If the
    PREFETCH opcode exists, then it gives the number of bytes up to and
    including the pickle's STOP opcode. So there's exactly 0 or 1
    PREFETCH opcodes per pickle.

    Is there an advantage to spraying multiple 8-byte "frame counts"
    throughout a pickle stream?
    Well, yes: much better memory usage for large pickles.
    Some people use pickles to store huge data, which was the motivation to
    add the 8-byte-size opcodes after all.

    We'd have the same advantage _if_ it were feasible to know the entire
    size up front. I understand now that it's not feasible.

    ...
    IMO, any validation should use a dedicated CRC-like scheme, rather than
    relying on the fact that correct pickles are statistically unlikely :-)

    OK! A new CRC opcode every two bytes ;-)

    (or you can implicitly rely on TCP or UDP checksums, or you disk
    subsystem's own sanity checks)

    Of course we've been doing that all along. Yet I've still seen my
    share of corrupted pickles anyway. Don't underestimate the
    determination of users to screw up everything possible in every way
    possible ;-)
  • Nick Coghlan at Nov 18, 2013 at 11:02 pm

    On 19 Nov 2013 08:52, "Tim Peters" wrote:
    [Tim]
    But it has a different kind of advantage: PREFETCH was optional. As
    Guido said, it's annoying to bloat the size of small pickles (which
    may, although individually small, occur in great numbers) by 8 bytes
    each. There's really no point to framing small chunks of data, right?
    [Antoine]
    You can't know how much space the pickle will take until the pickling
    ends, though, which makes it difficult to decide whether you want to
    emit a PREFETCH opcode or not.
    Ah, of course. Presumably the outgoing pickle stream is first stored
    in some memory buffer, right? If pickling completes before the buffer
    is first flushed, then you know exactly how large the entire pickle
    is. If "it's small" (say, < 100 bytes), don't write out the PREFETCH
    part. Else do.

    Which leads to another idea: after the PROTO opcode, there is, or is
    not, an optional PREFETCH opcde with an 8-byte argument. If the
    PREFETCH opcode exists, then it gives the number of bytes up to and
    including the pickle's STOP opcode. So there's exactly 0 or 1
    PREFETCH opcodes per pickle.

    Is there an advantage to spraying multiple 8-byte "frame counts"
    throughout a pickle stream?
    Well, yes: much better memory usage for large pickles.
    Some people use pickles to store huge data, which was the motivation to
    add the 8-byte-size opcodes after all.
    We'd have the same advantage _if_ it were feasible to know the entire
    size up front. I understand now that it's not feasible.

    This may be a dumb suggestion (since I don't know the pickle infrastructure
    at all), but could there be a short section of unframed data before
    switching to framing mode?


    For example, emit up to 256 bytes unframed in all pickles, then start
    emitting appropriate FRAME opcodes if the pickle continues on?


    Cheers,
    Nick.

    ...
    IMO, any validation should use a dedicated CRC-like scheme, rather than
    relying on the fact that correct pickles are statistically unlikely :-)
    OK! A new CRC opcode every two bytes ;-)
    (or you can implicitly rely on TCP or UDP checksums, or you disk
    subsystem's own sanity checks)
    Of course we've been doing that all along. Yet I've still seen my
    share of corrupted pickles anyway. Don't underestimate the
    determination of users to screw up everything possible in every way
    possible ;-)
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131119/f7d09fd4/attachment.html>
  • Antoine Pitrou at Nov 18, 2013 at 11:10 pm
    Ok, how about merging the two sub-threads :-)


    On Mon, 18 Nov 2013 16:44:59 -0600
    Tim Peters wrote:
    [Antoine]
    You can't know how much space the pickle will take until the pickling
    ends, though, which makes it difficult to decide whether you want to
    emit a PREFETCH opcode or not.
    Ah, of course. Presumably the outgoing pickle stream is first stored
    in some memory buffer, right? If pickling completes before the buffer
    is first flushed, then you know exactly how large the entire pickle
    is. If "it's small" (say, < 100 bytes), don't write out the PREFETCH
    part. Else do.

    That's true. We could also have a SMALLPREFETCH opcode with a one-byte
    length to still get the benefits of prefetching.

    Well, yes: much better memory usage for large pickles.
    Some people use pickles to store huge data, which was the motivation to
    add the 8-byte-size opcodes after all.
    We'd have the same advantage _if_ it were feasible to know the entire
    size up front. I understand now that it's not feasible.

    AFAICT, it would only be possible by doing two-pass pickling, which
    would also slow it down massively.

    A long-running process can legitimately put billions of items on work
    queues, far more than could ever fit in RAM simultaneously. Comparing
    this to PyObject overhead makes no sense to me. Neither does the line
    of argument "there are several kinds of overheads, so making this
    overhead worse too doesn't matter".

    Well, it's a question of cost / benefit: does it make sense to optimize
    something that will be dwarfed by other factors in real world
    situations?

    When possible, we should strive not to add overheads that don't repay
    their costs. For small pickles, an 8-byte size field doesn't appear
    to buy anything. But I appreciate that it costs implementation effort
    to avoid producing it in these cases.

    I share the concern, although I still don't think the "ocean of tiny
    pickles" is a reasonable use case :-)


    That said, assuming you think this is important (do you?), we're left
    with the following constraints:
    - it would be nice to have this PEP in 3.4
    - 3.4 beta1 and feature freeze is in approximately one week
    - switching to the PREFETCH scheme requires some non-trivial work on the
       current patch, work done by either Alexandre or me (but I already
       have pathlib (PEP 428) on my plate, so it'll have to be Alexandre) -
       unless you want to do it, of course?


    What do you think?


    Regards


    Antoine.
  • Tim Peters at Nov 19, 2013 at 1:17 am
    [Antoine]
    ...
    Well, it's a question of cost / benefit: does it make sense to optimize
    something that will be dwarfed by other factors in real world
    situations?

    For most of my career, a megabyte of RAM was an unthinkable luxury.
    Now I'm running on an OS that needs a gigabyte of RAM just to boot.
    So I'm old enough to think from the opposite end: "does it make sense
    to bloat something that's already working fine?". It's the "death of
    a thousand cuts" (well, this thing doesn't matter _that_ much - and
    neither does that, nor those others over there ...) that leads to a
    GiB-swallowing OS and a once-every-4-years crusade to reduce Python's
    startup time. For examples ;-)

    ...
    That said, assuming you think this is important (do you?),

    Honestly, it's more important to me "in theory" to oppose unnecessary
    bloat (of all kinds). But, over time, it's attitude that shapes
    results, so I'm not apologetic about that.

    we're left with the following constraints:
    - it would be nice to have this PEP in 3.4
    - 3.4 beta1 and feature freeze is in approximately one week
    - switching to the PREFETCH scheme requires some non-trivial work on the
    current patch, work done by either Alexandre or me (but I already
    have pathlib (PEP 428) on my plate, so it'll have to be Alexandre) -
    unless you want to do it, of course?

    What do you think?

    I wouldn't hold up PEP acceptance for this. It won't be a disaster in
    any case, just - at worse - another little ratchet in the "needless
    bloat" direction. And the PEP has more things that are pure wins.


    If it's possible to squeeze in the variable-length encoding, that
    would be great. If I were you, though, I'd check the patch in as-is,
    just in case. I can't tell whether I'll have time to work on it (have
    other things going now outside my control).
  • Serhiy Storchaka at Nov 19, 2013 at 7:52 am

    19.11.13 01:10, Antoine Pitrou ???????(??):
    - switching to the PREFETCH scheme requires some non-trivial work on the
    current patch, work done by either Alexandre or me (but I already
    have pathlib (PEP 428) on my plate, so it'll have to be Alexandre) -
    unless you want to do it, of course?

    I had implemented optimized PREFETCH scheme months ago (the code was
    smaller than with the framing layer). I need a day or two to update the
    code to the current Alexandre's patch.
  • Antoine Pitrou at Nov 18, 2013 at 11:57 pm

    On Mon, 18 Nov 2013 16:44:59 -0600 Tim Peters wrote:
    [Tim]
    But it has a different kind of advantage: PREFETCH was optional. As
    Guido said, it's annoying to bloat the size of small pickles (which
    may, although individually small, occur in great numbers) by 8 bytes
    each. There's really no point to framing small chunks of data, right?
    [Antoine]
    You can't know how much space the pickle will take until the pickling
    ends, though, which makes it difficult to decide whether you want to
    emit a PREFETCH opcode or not.
    Ah, of course. Presumably the outgoing pickle stream is first stored
    in some memory buffer, right? If pickling completes before the buffer
    is first flushed, then you know exactly how large the entire pickle
    is. If "it's small" (say, < 100 bytes), don't write out the PREFETCH
    part. Else do.

    Yet another possibility: keep framing but use a variable-length
    encoding for the frame size:


    - first byte: bits 7-5: N (= frame size bytes length - 1)
    - first byte: bits 4-0: first 5 bits of frame size
    - remaning N bytes: remaining bits of frame size


    With this scheme, very small pickles have a one byte overhead; small
    ones a two byte overhead; and the max frame size is 2**61 rather than
    2**64, which should still be sufficient.


    And the frame size is read using either one or two read() calls, which
    is efficient.


    Regards


    Antoine.
  • Nick Coghlan at Nov 19, 2013 at 12:30 am

    On 19 November 2013 09:57, Antoine Pitrou wrote:
    On Mon, 18 Nov 2013 16:44:59 -0600
    Tim Peters wrote:
    [Tim]
    But it has a different kind of advantage: PREFETCH was optional. As
    Guido said, it's annoying to bloat the size of small pickles (which
    may, although individually small, occur in great numbers) by 8 bytes
    each. There's really no point to framing small chunks of data, right?
    [Antoine]
    You can't know how much space the pickle will take until the pickling
    ends, though, which makes it difficult to decide whether you want to
    emit a PREFETCH opcode or not.
    Ah, of course. Presumably the outgoing pickle stream is first stored
    in some memory buffer, right? If pickling completes before the buffer
    is first flushed, then you know exactly how large the entire pickle
    is. If "it's small" (say, < 100 bytes), don't write out the PREFETCH
    part. Else do.
    Yet another possibility: keep framing but use a variable-length
    encoding for the frame size:

    - first byte: bits 7-5: N (= frame size bytes length - 1)
    - first byte: bits 4-0: first 5 bits of frame size
    - remaning N bytes: remaining bits of frame size

    With this scheme, very small pickles have a one byte overhead; small
    ones a two byte overhead; and the max frame size is 2**61 rather than
    2**64, which should still be sufficient.

    And the frame size is read using either one or two read() calls, which
    is efficient.

    And it's only a minimal change from the current patch. Sounds good to me.


    Cheers,
    Nick.


    --
    Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
  • Guido van Rossum at Nov 19, 2013 at 12:48 am

    On Mon, Nov 18, 2013 at 4:30 PM, Nick Coghlan wrote:

    On 19 November 2013 09:57, Antoine Pitrou wrote:
    On Mon, 18 Nov 2013 16:44:59 -0600
    Tim Peters wrote:
    [Tim]
    But it has a different kind of advantage: PREFETCH was optional. As
    Guido said, it's annoying to bloat the size of small pickles (which
    may, although individually small, occur in great numbers) by 8 bytes
    each. There's really no point to framing small chunks of data,
    right?
    [Antoine]
    You can't know how much space the pickle will take until the pickling
    ends, though, which makes it difficult to decide whether you want to
    emit a PREFETCH opcode or not.
    Ah, of course. Presumably the outgoing pickle stream is first stored
    in some memory buffer, right? If pickling completes before the buffer
    is first flushed, then you know exactly how large the entire pickle
    is. If "it's small" (say, < 100 bytes), don't write out the PREFETCH
    part. Else do.
    Yet another possibility: keep framing but use a variable-length
    encoding for the frame size:

    - first byte: bits 7-5: N (= frame size bytes length - 1)
    - first byte: bits 4-0: first 5 bits of frame size
    - remaning N bytes: remaining bits of frame size

    With this scheme, very small pickles have a one byte overhead; small
    ones a two byte overhead; and the max frame size is 2**61 rather than
    2**64, which should still be sufficient.

    And the frame size is read using either one or two read() calls, which
    is efficient.
    And it's only a minimal change from the current patch. Sounds good to me.

    Food for thought: maybe we should have variable-encoding lengths for all
    opcodes, rather than the current cumbersome scheme?


    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131118/113e6bcb/attachment.html>
  • Tim Peters at Nov 19, 2013 at 12:53 am
    [Guido]
    Food for thought: maybe we should have variable-encoding lengths for all
    opcodes, rather than the current cumbersome scheme?

    Yes, but not for protocol 4 - time's running out fast for that. When
    we "only" had the XXX1, XXX2, and XXX4 opcodes, it was kinda silly,
    but after adding XXX8 flavors to all of those it's become unbearable.
  • Xiscu at Nov 19, 2013 at 6:36 pm

    Food for thought: maybe we should have variable-encoding lengths for
    all opcodes, rather than the current cumbersome scheme?
    Funny, it sounds like UTF-8 :-)
  • Francis at Nov 19, 2013 at 6:39 pm

    Food for thought: maybe we should have variable-encoding lengths for
    all opcodes, rather than the current cumbersome scheme?
    Funny, it sounds like UTF-8 :-)
  • Antoine Pitrou at Nov 19, 2013 at 6:51 pm

    On Mon, 18 Nov 2013 16:48:05 -0800 Guido van Rossum wrote:

    Food for thought: maybe we should have variable-encoding lengths for all
    opcodes, rather than the current cumbersome scheme?

    Well, it's not that cumbersome... If you look at CPU encodings, they
    also tend to have different opcodes for different immediate lengths.


    In your case, I'd say it mostly leads to a bit of code duplication. But
    the opcode space is far from exhausted right now :)


    Regards


    Antoine.
  • Guido van Rossum at Nov 19, 2013 at 6:52 pm
    So why is framing different?




    On Tue, Nov 19, 2013 at 10:51 AM, Antoine Pitrou wrote:

    On Mon, 18 Nov 2013 16:48:05 -0800
    Guido van Rossum wrote:
    Food for thought: maybe we should have variable-encoding lengths for all
    opcodes, rather than the current cumbersome scheme?
    Well, it's not that cumbersome... If you look at CPU encodings, they
    also tend to have different opcodes for different immediate lengths.

    In your case, I'd say it mostly leads to a bit of code duplication. But
    the opcode space is far from exhausted right now :)

    Regards

    Antoine.





    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131119/8c1de9a6/attachment.html>
  • Antoine Pitrou at Nov 19, 2013 at 6:57 pm

    On Tue, 19 Nov 2013 10:52:58 -0800 Guido van Rossum wrote:
    So why is framing different?

    Because it doesn't use opcodes, so it can't use different opcodes to
    differentiate between different frame size widths :-)


    Regards


    Antoine.
  • Guido van Rossum at Nov 19, 2013 at 7:05 pm
    So using an opcode for framing is out? (Sorry, I've lost track of the
    back-and-forth.)




    On Tue, Nov 19, 2013 at 10:57 AM, Antoine Pitrou wrote:

    On Tue, 19 Nov 2013 10:52:58 -0800
    Guido van Rossum wrote:
    So why is framing different?
    Because it doesn't use opcodes, so it can't use different opcodes to
    differentiate between different frame size widths :-)

    Regards

    Antoine.





    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131119/98d5e11e/attachment.html>
  • Antoine Pitrou at Nov 19, 2013 at 7:11 pm

    On Tue, 19 Nov 2013 11:05:45 -0800 Guido van Rossum wrote:


    So using an opcode for framing is out? (Sorry, I've lost track of the
    back-and-forth.)

    It doesn't seem to bring anything, and it makes the overhead worse for
    tiny pickles (since it will be two bytes at least, instead of one byte
    with the current variable length encoding proposal).


    If overhead doesn't matter, I'm fine with keeping a simple 8-bytes
    frame size :-)


    Regards


    Antoine.
  • Guido van Rossum at Nov 19, 2013 at 7:26 pm
    Well, both fixed 8-byte framing and variable-size framing it introduces a
    new way of representing numbers in the stream, which means that everyone
    parsing and generating pickles must be able to support both styles. (But
    fixed is easier since the XXX8 opcodes use the same format.)


    I'm thinking of how you correctly read a pickle from a non-buffering pipe
    with the minimum number of read() calls without ever reading beyond the end
    of a valid pickle. (That's a requirement, right?)


    If you know it's protocol 4:
         with fixed framing:
             read 10 bytes, that's the magic word plus first frame; then you can
    start buffering
         with variable framing:
             read 3 bytes, then depending on the 3rd byte read some more to find
    the frame size; then you can start buffering
         with mandatory frame opcode:
             pretty much the same
         with optional frame opcode:
             pretty much the same (the 3rd byte must be a valid opcode, even if
    it isn't a frame opcode)


    if you don't know the protocol number:
         read the first byte, then read the second byte (or not if it's not
    explicitly versioned), then you know the protocol and can do the rest as
    above




    On Tue, Nov 19, 2013 at 11:11 AM, Antoine Pitrou wrote:

    On Tue, 19 Nov 2013 11:05:45 -0800
    Guido van Rossum wrote:
    So using an opcode for framing is out? (Sorry, I've lost track of the
    back-and-forth.)
    It doesn't seem to bring anything, and it makes the overhead worse for
    tiny pickles (since it will be two bytes at least, instead of one byte
    with the current variable length encoding proposal).

    If overhead doesn't matter, I'm fine with keeping a simple 8-bytes
    frame size :-)

    Regards

    Antoine.





    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131119/fe342929/attachment.html>
  • Tim Peters at Nov 19, 2013 at 7:22 pm
    [Guido]
    So using an opcode for framing is out? (Sorry, I've lost track of the
    back-and-forth.)

    It was never in ;-) I'd *prefer* one, but not enough to try to block
    the PEP. As is, framing is done at a "lower level" than opcode
    decoding. I fear this is brittle, for all the usual "explicit is
    better than implicit" kinds of reasons. The only way now to know that
    you're looking at a frame size is to keep a running count of bytes
    processed and realize you've reached a byte offset where a frame size
    "is expected".


    With an opcode, framing could also be optional (whenever desired),
    because frame sizes would be _explicitly_ marked in the byte stream
    Then the framing overhead for small pickles could drop to 0 bytes
    (instead of the current 8, or 1 thru 9 under various other schemes).


    Ideal would be an explicit framing opcode combined with
    variable-length size encoding. That would never require more bytes
    than the current scheme, and 'almost always" require fewer. But even
    I don't think it's of much value to chop a few bytes off every 64KB of
    pickle ;-)
  • Antoine Pitrou at Nov 19, 2013 at 7:59 pm

    On Tue, 19 Nov 2013 13:22:52 -0600 Tim Peters wrote:
    [Guido]
    So using an opcode for framing is out? (Sorry, I've lost track of the
    back-and-forth.)
    It was never in ;-) I'd *prefer* one, but not enough to try to block
    the PEP. As is, framing is done at a "lower level" than opcode
    decoding. I fear this is brittle, for all the usual "explicit is
    better than implicit" kinds of reasons. The only way now to know that
    you're looking at a frame size is to keep a running count of bytes
    processed and realize you've reached a byte offset where a frame size
    "is expected".

    That's integrated to the built-in buffering. It's not really an
    additional constraint: the frame sizes simply dictate how buffering
    happens in practice. The main point of framing is to *simplify* the
    buffering logic (of course, the old buffering logic is still there for
    protocols <= 3, unfortunately).


    Note some drawbacks of frame opcodes:
    - the decoder has to sanity check the frame opcodes (what if a frame
       opcode is encountered when already inside a frame?)
    - a pickle-mutating function such as pickletools.optimize() may naively
       ignore the frame opcodes while rearranging the pickle stream, only to
       emit a new pickle with invalid frame sizes


    Regards


    Antoine.
  • Martin v. Löwis at Nov 19, 2013 at 8:25 pm

    Am 19.11.13 20:59, schrieb Antoine Pitrou:
    That's integrated to the built-in buffering. It's not really an
    additional constraint: the frame sizes simply dictate how buffering
    happens in practice. The main point of framing is to *simplify* the
    buffering logic (of course, the old buffering logic is still there for
    protocols <= 3, unfortunately).

    I wonder why this needs to be part of the pickle protocol at all,
    if it really is "below" the opcodes. Anybody desiring framing could
    just implement a framing version of the io.BufferedReader, which
    could be used on top of a socket connection (say) to allow fetching
    larger blocks from the network stack. This would then be transparent
    to the pickle implementation; the framing reader would, of course,
    provide the peek() operation to allow the unpickler to continue to use
    buffering.


    Such a framing BufferedReader might even be included in the standard
    library.


    Regards,
    Martin
  • Antoine Pitrou at Nov 19, 2013 at 8:28 pm

    On Tue, 19 Nov 2013 21:25:34 +0100 "Martin v. L?wis" wrote:
    Am 19.11.13 20:59, schrieb Antoine Pitrou:
    That's integrated to the built-in buffering. It's not really an
    additional constraint: the frame sizes simply dictate how buffering
    happens in practice. The main point of framing is to *simplify* the
    buffering logic (of course, the old buffering logic is still there for
    protocols <= 3, unfortunately).
    I wonder why this needs to be part of the pickle protocol at all,
    if it really is "below" the opcodes. Anybody desiring framing could
    just implement a framing version of the io.BufferedReader, which
    could be used on top of a socket connection (say) to allow fetching
    larger blocks from the network stack. This would then be transparent
    to the pickle implementation; the framing reader would, of course,
    provide the peek() operation to allow the unpickler to continue to use
    buffering.

    Such a framing BufferedReader might even be included in the standard
    library.

    Well, unless you propose a patch before Saturday, I will happily ignore
    your proposal.


    Regards


    Antoine.
  • Martin v. Löwis at Nov 19, 2013 at 10:05 pm

    Am 19.11.13 21:28, schrieb Antoine Pitrou:


    Well, unless you propose a patch before Saturday, I will happily ignore
    your proposal.

    See


    http://bugs.python.org/file32709/framing.diff


    Regards,
    Martin
  • Antoine Pitrou at Nov 19, 2013 at 10:50 pm

    On Tue, 19 Nov 2013 23:05:07 +0100 "Martin v. L?wis" wrote:
    Am 19.11.13 21:28, schrieb Antoine Pitrou:
    Well, unless you propose a patch before Saturday, I will happily ignore
    your proposal.
    See

    http://bugs.python.org/file32709/framing.diff

    Ok, thanks. So now that I look at the patch I see the following problems
    with this idea:


    - "pickle + framing" becomes a different protocol than "pickle" alone,
       which means we lose the benefit of protocol autodetection. It's as
       though pickle.load() required you to give the protocol number,
       instead of inferring it from the pickle bytestream.


    - it is less efficient than framing built inside pickle, since it
       adds separate buffers and memory copies (while the point of framing
       is to make buffering more efficient).


    Your idea is morally similar to saying "we don't need to optimize the
    size of pickles, since you can gzip them anyway". However, the fact
    that the _pickle module currently goes to lengths to try to optimize
    buffering, implies to me that it's reasonable to also improve the
    pickle protocol so as to optimize buffering.


    Regards


    Antoine.
  • Martin v. Löwis at Nov 19, 2013 at 11:56 pm

    Am 19.11.13 23:50, schrieb Antoine Pitrou:
    Ok, thanks. So now that I look at the patch I see the following problems
    with this idea:

    - "pickle + framing" becomes a different protocol than "pickle" alone,
    which means we lose the benefit of protocol autodetection. It's as
    though pickle.load() required you to give the protocol number,
    instead of inferring it from the pickle bytestream.

    Not necessarily. Framing becomes a different protocol, yes. But
    autodetection would still be possible (it actually is possible in
    my proposed definition).

    - it is less efficient than framing built inside pickle, since it
    adds separate buffers and memory copies (while the point of framing
    is to make buffering more efficient).

    Correct. However, if the intent is to reduce the number of system
    calls, then this is still achieved.

    Your idea is morally similar to saying "we don't need to optimize the
    size of pickles, since you can gzip them anyway".

    Not really. In the case of gzip, it might be that the size reduction
    of properly saving bytes in pickle might be even larger. Here,
    the wire representation, and the number of system calls is actually
    (nearly) identical.

    However, the fact
    that the _pickle module currently goes to lengths to try to optimize
    buffering, implies to me that it's reasonable to also improve the
    pickle protocol so as to optimize buffering.

    AFAICT, the real driving force is the desire to not read-ahead
    more than the pickle is long. This is what complicates the code.
    The easiest (and most space-efficient) solution to that problem
    would be to prefix the entire pickle with a data size field
    (possibly in a variable-length representation), i.e. to make a
    single frame.


    If that was done, I would guess that Tim's concerns about brittleness
    would go away (as you couldn't have a length field in the middle of
    data). IMO, the PEP has nearly the same flaw as the HTTP chunked
    transfer, which also puts length fields in the middle of the payload
    (except that HTTP makes it worse by making them optional).


    Of course, a single length field has other drawbacks, such as having
    to pickle everything before sending out the first bytes.


    Regards,
    Martin
  • Antoine Pitrou at Nov 20, 2013 at 12:10 am

    On Wed, 20 Nov 2013 00:56:13 +0100 "Martin v. L?wis" wrote:
    AFAICT, the real driving force is the desire to not read-ahead
    more than the pickle is long. This is what complicates the code.
    The easiest (and most space-efficient) solution to that problem
    would be to prefix the entire pickle with a data size field
    (possibly in a variable-length representation), i.e. to make a
    single frame.

    Pickling then becomes very problematic: you have to keep the entire
    pickle in memory until the end, when you finally can write the size at
    the beginning of the pickle.

    If that was done, I would guess that Tim's concerns about brittleness
    would go away (as you couldn't have a length field in the middle of
    data). IMO, the PEP has nearly the same flaw as the HTTP chunked
    transfer, which also puts length fields in the middle of the payload
    (except that HTTP makes it worse by making them optional).

    Tim's concern is easily addressed with a FRAME opcode, without
    changing the overall scheme (as he lately proposed).


    Regards


    Antoine.
  • Tim Peters at Nov 20, 2013 at 5:18 am
    [Martin v. L?wis]
    ...
    AFAICT, the real driving force is the desire to not read-ahead
    more than the pickle is long. This is what complicates the code.
    The easiest (and most space-efficient) solution to that problem
    would be to prefix the entire pickle with a data size field
    (possibly in a variable-length representation), i.e. to make a
    single frame.

    In a bout of giddy optimism, I suggested that earlier in the thread.
    It would be sweet :-)



    If that was done, I would guess that Tim's concerns about brittleness
    would go away (as you couldn't have a length field in the middle of
    data). IMO, the PEP has nearly the same flaw as the HTTP chunked
    transfer, which also puts length fields in the middle of the payload
    (except that HTTP makes it worse by making them optional).

    Of course, a single length field has other drawbacks, such as having
    to pickle everything before sending out the first bytes.

    And that's the killer. Pickle strings are generally produced
    incrementally, in smallish pieces. But that may go on for very many
    iterations, and there's no way to guess the final size in advance. I
    only see three ways to do it:


    1. Hope the whole string fits in RAM.
    2. Pickle twice, the first time just to get the final size (& throw
    the pickle pieces away on the first pass while summing their sizes).
    3. Flush the pickle string to disk periodically, then after it's done
    read it up and copy it to the intended output stream.


    All of those really suck :-(


    BTW, I'm not a web guy: in what way is HTTP chunked transfer mode
    viewed as being flawed? Everything I ever read about it seemed to
    think it was A Good Idea.
  • Martin v. Löwis at Nov 20, 2013 at 3:44 pm

    Am 20.11.13 06:18, schrieb Tim Peters:
    BTW, I'm not a web guy: in what way is HTTP chunked transfer mode
    viewed as being flawed? Everything I ever read about it seemed to
    think it was A Good Idea.

    It just didn't work for some time, see e.g.


    http://bugs.python.org/issue1486335
    http://bugs.python.org/issue1966
    http://bugs.python.org/issue1312980
    http://bugs.python.org/issue3761


    It's not that the protocol was underspecified - just the implementation
    was "brittle" (if I understand that word correctly). And I believe (and
    agree with you) that the cause for this "difficult to implement"
    property is that the framing is in putting framing "in the middle"
    of the stack (i.e. not really *below* pickle itself, but into pickle
    but below the opcodes - just like http chunked transfer is "in" http,
    but below the content encoding).


    Regards,
    Martin
  • Guido van Rossum at Nov 20, 2013 at 4:07 pm
    A problem with chunked IIRC is that the frame headers are variable-length
    (a CRLF followed by a hex number followed by some optional gunk followed by
    CRLF) so you have to drop back into one-byte-at-a-time to read it. (Well, I
    suppose you could read 5 bytes, which is the minimum: CR, LF, X, CR, LF,
    and when the second CR isn't among these, you have a lower bound for how
    much more to read, although at that point you better load up on coffee
    before writing the rest of the code. :-)


    Some good things about it:


    - Explicit final frame (byte count zero), so no need to rely on the data to
    know the end.


    - The redundancy in the format (start and end with CRLF, hex numbers) makes
    it more likely that framing errors (e.g. due to an incorrect counting or
    some layer collapsing CRLF into LF) are detected.




    On Wed, Nov 20, 2013 at 7:44 AM, "Martin v. L?wis" wrote:

    Am 20.11.13 06:18, schrieb Tim Peters:
    BTW, I'm not a web guy: in what way is HTTP chunked transfer mode
    viewed as being flawed? Everything I ever read about it seemed to
    think it was A Good Idea.
    It just didn't work for some time, see e.g.

    http://bugs.python.org/issue1486335
    http://bugs.python.org/issue1966
    http://bugs.python.org/issue1312980
    http://bugs.python.org/issue3761

    It's not that the protocol was underspecified - just the implementation
    was "brittle" (if I understand that word correctly). And I believe (and
    agree with you) that the cause for this "difficult to implement"
    property is that the framing is in putting framing "in the middle"
    of the stack (i.e. not really *below* pickle itself, but into pickle
    but below the opcodes - just like http chunked transfer is "in" http,
    but below the content encoding).

    Regards,
    Martin

    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/guido%40python.org





    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20131120/fc32fb7b/attachment.html>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-dev @
categoriespython
postedNov 16, '13 at 6:15p
activeNov 26, '13 at 10:39p
posts70
users12
websitepython.org

People

Translate

site design / logo © 2018 Grokbase