FAQ
When i wanted to send an .iso file of 4GB length, i had traceback:
"OverflowError: requested number of bytes is more than a Python string
can hold"

Sockets are being used in every network app, i.e: p2p progs (like
BitTorrent), and exchanged data is often bigger than 4GB. So why i've
had that Traceback? How many number of bytes Python string can hold?

Search Discussions

  • Steven D'Aprano at Apr 13, 2009 at 1:45 am

    On Mon, 13 Apr 2009 00:21:34 +0200, Ryniek90 wrote:

    When i wanted to send an .iso file of 4GB length, i had traceback:
    "OverflowError: requested number of bytes is more than a Python string
    can hold"

    Sockets are being used in every network app, i.e: p2p progs (like
    BitTorrent), and exchanged data is often bigger than 4GB.
    But they don't transfer the entire file as ONE packet. Split your data
    into smaller packets. I don't know what a good size for each packet would
    be, but if I were doing this, I'd probably start with 4096 or 8192
    *bytes*.

    http://www.amk.ca/python/howto/sockets/

    So why i've
    had that Traceback? How many number of bytes Python string can hold?
    The documentation doesn't seem to specify a maximum string length:

    http://docs.python.org/library/stdtypes.html

    which suggests to me that it will be implementation dependent. However,
    I'd guess that the current CPython implementation will have a hard limit
    of 2**32 bytes (4GB), and a soft limit on the amount of memory that you
    have. You're trying to create a single, continuous block of memory 4GB in
    size! Unless you've got *at least* 4GB of RAM, this is impossible even in
    principle, and in practice you need more than that to allow for the
    overhead of the operating system, Python, and any other applications you
    have running.



    --
    Steven
  • Jean-Paul Calderone at Apr 13, 2009 at 1:55 am

    On 13 Apr 2009 01:45:56 GMT, Steven D'Aprano wrote:
    On Mon, 13 Apr 2009 00:21:34 +0200, Ryniek90 wrote:

    When i wanted to send an .iso file of 4GB length, i had traceback:
    "OverflowError: requested number of bytes is more than a Python string
    can hold"

    Sockets are being used in every network app, i.e: p2p progs (like
    BitTorrent), and exchanged data is often bigger than 4GB.
    But they don't transfer the entire file as ONE packet. Split your data
    into smaller packets. I don't know what a good size for each packet would
    be, but if I were doing this, I'd probably start with 4096 or 8192
    *bytes*.
    Everything in that paragraph is true, but it's rather misleading. The
    size of the packets you send is *not* determined by the length of the
    string you pass to socket.send. Packet size in a TCP connection is
    determined by various things beyond the control of an application using
    the BSD socket API.

    Jean-Paul
  • Benjamin Peterson at Apr 13, 2009 at 1:58 am

    Steven D'Aprano <steve <at> REMOVE-THIS-cybersource.com.au> writes:

    which suggests to me that it will be implementation dependent
    The length of sequences is constrained by sys.maxsize
    (and no, you can't change it).
  • Gabriel Genellina at Apr 13, 2009 at 6:15 am

    En Sun, 12 Apr 2009 19:28:43 -0300, <//phr.cx at nospam.invalid>> escribi?:
    Ryniek90 <ryniek90 at gmail.com> writes:
    When i wanted to send an .iso file of 4GB length, i had traceback:
    "OverflowError: requested number of bytes is more than a Python string
    can hold"
    You're not supposed to put the 4GB all in one string. Open the
    socket and send smaller packets through it.
    And instead of reinventing the wheel again, use the shutil module to do
    exactly that.

    --
    Gabriel Genellina

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedApr 12, '09 at 10:21p
activeApr 13, '09 at 6:15a
posts5
users5
websitepython.org

People

Translate

site design / logo © 2022 Grokbase