FAQ
Hi All,


This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation.


Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email).


Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP CPU with 18 cores, hyper-threading off, turbo off).


Below is a summary of the performance we saw on the "grand unified python benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 3 rigorous runs of the following benchmarks. In each rigorous run, a benchmark is run 100 times with and without the computed goto patch. Below we show the average performance boost for the 3 rigorous runs.


Python 2.7.10 (original) vs Computed Goto performance
Benchmark


Delta (rigorous run #1) %


Delta (rigorous run #2) %


Delta (rigorous run #3) %


Avg. Delta %


iterative_count


24.48


24.36


23.78


24.2


unpack_sequence


19.06


18.47


19.48


19.0


slowspitfire


14.36


13.41


16.65


14.8


threaded_count


15.85


13.43


13.93


14.4


pystone


10.68


11.67


11.08


11.1


nbody


10.25


8.93


9.28


9.5


go


7.96


8.76


7.69


8.1


pybench


6.3


6.8


7.2


6.8


spectral_norm


5.49


9.37


4.62


6.5


float


6.09


6.2


6.96


6.4


richards


6.19


6.41


6.42


6.3


slowunpickle


6.37


8.78


3.55


6.2


json_dump_v2


1.96


12.53


3.57


6.0


call_simple


6.37


5.91


3.92


5.4


chaos


4.57


5.34


3.85


4.6


call_method_slots


2.63


3.27


7.71


4.5


telco


5.18


1.83


6.47


4.5


simple_logging


3.48


1.57


7.4


4.2


call_method


2.61


5.4


3.88


4.0


chameleon


2.03


6.26


3.2


3.8


fannkuch


3.89


3.19


4.39


3.8


silent_logging


4.33


3.07


3.39


3.6


slowpickle


5.72


-1.12


6.06


3.6


2to3


2.99


3.6


3.45


3.3


etree_iterparse


3.41


2.51


3


3.0


regex_compile


3.44


2.48


2.84


2.9


mako_v2


2.14


1.29


5.22


2.9


meteor_contest


2.01


2.2


3.88


2.7


django


6.68


-1.23


2.56


2.7


formatted_logging


1.97


5.82


-0.11


2.6


hexiom2


2.83


2.1


2.55


2.5


django_v2


1.93


2.53


2.92


2.5


etree_generate


2.38


2.13


2.51


2.3


mako


-0.3


9.66


-3.11


2.1


bzr_startup


0.35


1.97


3


1.8


etree_process


1.84


1.01


1.9


1.6


spambayes


1.76


0.76


0.48


1.0


regex_v8


1.96


-0.66


1.63


1.0


html5lib


0.83


0.72


0.97


0.8


normal_startup


1.41


0.39


0.24


0.7


startup_nosite


1.2


0.41


0.42


0.7


etree_parse


0.24


0.9


0.79


0.6


json_load


1.38


0.56


-0.25


0.6


pidigits


0.45


0.33


0.28


0.4


hg_startup


0.32


2.07


-1.41


0.3


rietveld


0.05


0.91


-0.43


0.2


tornado_http


2.34


-0.92


-1.27


0.1


call_method_unknown


0.72


1.26


-1.85


0.0


raytrace


-0.35


-0.75


0.94


-0.1


regex_effbot


1.97


-1.18


-2.57


-0.6


fastunpickle


-1.65


0.5


-0.88


-0.7


nqueens


-2.24


-1.53


-0.81


-1.5


fastpickle


-0.74


1.98


-6.26


-1.7






Thanks,
Vamsi


------------------------------------------------------------------------------------------------------------------------------------------------------------
From: gvanrossum at gmail.com[mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum
Sent: Tuesday, May 19, 2015 1:59 PM
To: Cohn, Robert S
Cc: R. David Murray (r.david.murray at murrayandwalker.com<mailto:r.david.murray@murrayandwalker.com>)
Subject: Re: meeting at PyCon


Hi Robert and David,
I just skimmed that thread. There were a lot of noises about backporting it to 2.7 but the final message on the topic, by Antoine, claimed it was too late for 2.7. However, that was before we had announced the EOL extension of 2.7 till 2020, and perhaps we were also in denial about 3.x uptake vs. 2.x. So I think it's definitively worth bringing this up. I would start with a post on python-dev linking to the source code for your patch, and adding a message to the original tracker issue too (without reopening it though -- just so the people who were on the bug will be pinged about it).
Because of backwards compatibility with previous 2.7.x releases, it's very important that the patch not break anything -- in particular this means you can't add opcodes or change their specification. You will also undoubtedly be asked to test this on a variety of platforms 32-bit and 64-bit that people care about. But I'm sure you're expecting all that. :-)


You might also check with Benjamin Peterson, who is the 2.7 release manager. I think he just announced 2.7.10, so it's too late for that, but I assume we'll keep doing 2.7.x releases until 2020.
Good luck,


--Guido


PS. I am assuming you are contributing this under a PSF-accepted license, e.g. Apache 2.0, otherwise it's an automatic nogo.


On Tue, May 19, 2015 at 9:33 AM, Cohn, Robert S <robert.s.cohn at intel.comwrote:
Hi Guido,




When we met for lunch at pycon, I asked if performance related patches would be ok for python 2.x. My understanding is that you thought it was possible if it did not create a maintainability problem. We have an example now, a 2.7 patch for computed goto based on the implementation in python 3 http://bugs.python.org/issue4753 It increases performance by up to 10% across a wide range of workloads.






As I mentioned at lunch, we hired David Murray's company, and he is guiding intel through the development process for cpython. David and I thought it would be good to run this by you before raising the issue on python-dev. Do you have a specific concern about this patch or a more general concern about performance patches to 2.7? Thanks.






Robert
--------


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/d348f039/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cgoto_py2710_hg_final.patch
Type: application/octet-stream
Size: 49186 bytes
Desc: cgoto_py2710_hg_final.patch
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/d348f039/attachment-0001.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Instructions_cgoto_patch.txt
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/d348f039/attachment-0001.txt>

Search Discussions

  • Nick Coghlan at May 28, 2015 at 1:31 am

    On 28 May 2015 at 10:17, Parasa, Srinivas Vamsi wrote:


    Hi All,



    This is Vamsi from Server Scripting Languages Optimization team at Intel
    Corporation.



    Would like to submit a request to enable the computed goto based dispatch
    in Python 2.x (which happens to be enabled by default in Python 3 given its
    performance benefits on a wide range of workloads). We talked about this
    patch with Guido and he encouraged us to submit a request on Python-dev
    (email conversation with Guido shown at the bottom of this email).

    +1 from me, for basically the same reasons Guido gives: Python 2.7 is going
    to be with us for a long time, and this particular change shouldn't have
    any externally visible impacts at either an ABI or API level.


    Cheers,
    Nick.


    --
    Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/75ab0af0/attachment.html>
  • Raymond Hettinger at May 28, 2015 at 2:27 am

    On May 27, 2015, at 6:31 PM, Nick Coghlan wrote:

    On 28 May 2015 at 10:17, Parasa, Srinivas Vamsi wrote:
    Hi All,



    This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation.



    Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email).


    +1 from me, for basically the same reasons Guido gives: Python 2.7 is going to be with us for a long time, and this particular change shouldn't have any externally visible impacts at either an ABI or API level.

    +1 from me a well. We probably should have done this long ago.




    Raymond Hettinger
  • Terry Reedy at May 28, 2015 at 5:01 am

    On 5/27/2015 9:31 PM, Nick Coghlan wrote:


    +1 from me, for basically the same reasons Guido gives: Python 2.7 is
    going to be with us for a long time, and this particular change
    shouldn't have any externally visible impacts at either an ABI or API level.

    Immediately after a release, giving the patch plenty of time to be be
    tested, seems like a good time.


    --
    Terry Jan Reedy
  • Gregory P. Smith at May 28, 2015 at 5:46 am
    Why now? We intentionally decided not to do this for 2.7 in the past
    because it was too late for the release cutoff.


    Has anyone benchmarked compiling python in profile-opt mode vs having the
    computed goto patch? I'd *expect* the benefits to be the roughly the same.
    Has this been compared to that? (Anyone not compiling their Python
    interpreter in profile-opt mode is doing themselves a major disservice.)


    Does it shows noteworthy improvements even when used with a profile-opt
    build using a PROFILE_TASK of regrtest.py with the same test exclusion list
    as the debian python2.7 package?


    Could you please rerun benchmarks including the profile-opt build with and
    without the patch for comparsion.


    -gps


    PS I recommend attaching the up to date patch against 2.7.10 to issue4753.
    That is where anyone will go looking for it, not buried in a mailing list
    archive.


    On Wed, May 27, 2015 at 10:01 PM Terry Reedy wrote:

    On 5/27/2015 9:31 PM, Nick Coghlan wrote:

    +1 from me, for basically the same reasons Guido gives: Python 2.7 is
    going to be with us for a long time, and this particular change
    shouldn't have any externally visible impacts at either an ABI or API
    level.

    Immediately after a release, giving the patch plenty of time to be be
    tested, seems like a good time.

    --
    Terry Jan Reedy

    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/97dbc8d0/attachment-0001.html>
  • M.-A. Lemburg at May 28, 2015 at 8:12 am

    On 28.05.2015 02:17, Parasa, Srinivas Vamsi wrote:
    Hi All,

    This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation.

    Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email).

    +1.


    It's been successful for Python 3, doesn't change semantics and
    increases performance.


    --
    Marc-Andre Lemburg
    eGenix.com


    Professional Python Services directly from the Source (#1, May 28 2015)
    Python Projects, Coaching and Consulting ... http://www.egenix.com/
    mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/
    mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
    ________________________________________________________________________


    ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::


        eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
         D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
                Registered at Amtsgericht Duesseldorf: HRB 46611
                    http://www.egenix.com/company/contact/
  • Berker Peksağ at May 28, 2015 at 8:54 am

    On Thu, May 28, 2015 at 3:17 AM, Parasa, Srinivas Vamsi wrote:
    Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon ? Haswell EP CPU with 18 cores, hyper-threading off, turbo off).

    Hi Vamsi,


    Thank you for your work and your detailed email.


    I'm -1 on the idea because:


    * Performance improvements are not bug fixes
    * The patch doesn't make the migration process from Python 2 to Python 3 easier
    * In long term, it would be nice to work on making Python 3 better:
    See http://bugs.python.org/issue11549 for an example task.


    --Berker
  • Nick Coghlan at May 28, 2015 at 9:47 am

    On 28 May 2015 at 18:54, Berker Peksa? wrote:
    On Thu, May 28, 2015 at 3:17 AM, Parasa, Srinivas Vamsi
    wrote:
    Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon ? Haswell EP CPU with 18 cores, hyper-threading off, turbo off).
    Hi Vamsi,

    Thank you for your work and your detailed email.

    I'm -1 on the idea because:

    * Performance improvements are not bug fixes

    The "nothing except backwards compatible bug fixes in maintenance
    releases" rule was adopted for the *benefit of Python users*. When a
    new feature release can be reliably expected every 18-24 months, it
    makes sense to err heavily on the side of minimising risks to
    stability when it comes to making judgement calls on whether or not a
    change is appropriate to a maintenance release or not.


    Python 2.7 is an unusual case, as even though there *are* newer
    feature releases available, the barriers to migration are much higher
    than they would otherwise be. Each progressive 3.x release has brought
    those barriers down a bit, and 3.5 and the static type checking work
    being done through mypy et al will bring them down even further, but
    version migration work is still work where end users don't see any
    immediate practical benefit - they only see the benefit *after*
    they're able to drop Python 2 compatibility, and can start using
    Python 3 only features like matrix multiplication and the async/await
    syntax.


    *Adding* features to Python 2.7 is quite rightly still controversial,
    as they add complexity to the compatibility matrix for testing
    purposes. Code that runs correctly with the PEP 466 and 476 changes to
    SSL handling, may fail on earlier versions.


    Internal performance improvements, by contrast, don't hurt end users
    at all beyond the stability risks, and in this case, the request to
    make the change is being accompanied by the offer to assist with
    ongoing maintenance (including engaging an experienced core developer
    to help coach Intel contributors through the contribution process).


    So when folks ask "What changed?" in relation to this request, what
    changed is the fact that it isn't expected to be a one off
    contribution, but rather part of a broader effort focused on improving
    the performance of both Python 2 and Python 3, including contributions
    to ongoing maintenance activities.

    * The patch doesn't make the migration process from Python 2 to Python 3 easier
    * In long term, it would be nice to work on making Python 3 better:

    Indeed, but we also need help living up to the "Python 2.7 will be
    supported to 2020" commitment. Python 2.7 maintenance is *not* a
    particularly exciting task, and it's only going to get less
    interesting as Python 3 adoption climbs, so we're going to need paid
    contributors to start filling the gap as volunteers (quite reasonably)
    move on to the more inherently rewarding task of working to move
    Python 3 forward.


    That's going to be a negotiation process - companies don't typically
    contribute paid development time to open source projects out of the
    kindness of their hearts, they do it either because they're using the
    project themselves, because of deals they've made with individual
    contributors around how they spend their time, or because it helps
    them influence the direction of upstream development in ways that help
    them and their customers. (And sometimes it's a mix of all 3 of those
    factors)


    Regards,
    Nick.


    --
    Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
  • Nick Coghlan at May 28, 2015 at 10:01 am

    On 28 May 2015 at 19:47, Nick Coghlan wrote:
    That's going to be a negotiation process - companies don't typically
    contribute paid development time to open source projects out of the
    kindness of their hearts, they do it either because they're using the
    project themselves, because of deals they've made with individual
    contributors around how they spend their time, or because it helps
    them influence the direction of upstream development in ways that help
    them and their customers. (And sometimes it's a mix of all 3 of those
    factors)

    And to be completely transparent about this: this probably won't be
    the last of these kinds of discussions we're likely to see.


    Various folks (including me) have been negotiating to have their
    employers fund paid CPython contribution positions and as we coach
    colleagues that take up these roles through the core team's
    contribution processes, one of the consequences will be that we will
    sometimes advocate for acceptance of changes that we would have
    historically rejected as imposing too high a maintenance burden for an
    all-volunteer development team to be expected to deal with.


    Regards,
    Nick.


    --
    Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
  • Julian Taylor at May 28, 2015 at 11:30 am
    won't this need python compiled with gcc 5.1 to have any effect? Which
    compiler version was used for the benchmark?
    the issue that negated most computed goto improvements
    (https://gcc.gnu.org/bugzilla/show_bug.cgi?id9284) was only closed
    very recently (r212172, 9f4ec746affbde1)
  • Maciej Fijalkowski at May 28, 2015 at 11:55 am

    I'm -1 on the idea because:

    * Performance improvements are not bug fixes
    * The patch doesn't make the migration process from Python 2 to Python 3 easier

    And this is why people have been porting Python applications to Go.
    Maybe addressing Python performance and making Python (2 or 3) a
    better language/platform would mitigate that.


    Cheers,
    fijal
  • Nick Coghlan at May 28, 2015 at 2:08 pm

    On 28 May 2015 at 21:55, Maciej Fijalkowski wrote:
    I'm -1 on the idea because:

    * Performance improvements are not bug fixes
    * The patch doesn't make the migration process from Python 2 to Python 3 easier
    And this is why people have been porting Python applications to Go.

    For folks hitting the kinds of scalability problems that Go is
    designed to help with, a few percentage points here and there in
    CPython performance aren't going to make a big difference - they'll
    need the kinds of speed multipliers that PyPy can offer.


    Given that Go can't run Python C extensions any more than PyPy can,
    and involves a rewrite in a different programming language to boot,
    we'd do well to ponder what Go currently offers that PyPy doesn't. If
    we ignore the fashion-driven aspect of "Google wrote it, so it must be
    cool" (which we can't do anything about), and if we ignore the
    multi-vendor commercial support question (which tends to significantly
    lag community adoption for true community driven projects like PyPy),
    then one of the big keys in my view is the easy redistributability of
    Go binaries.


    For Linux based network services (and even Windows these days), Docker
    containers offer a potentially compelling way of bundling the PyPy
    runtime with Python applications, and Docker, Inc already maintain a
    set of PyPy base images at https://registry.hub.docker.com/_/pypy/


    Docker's image layering model then means that applications sharing a
    PyPy runtime shouldn't need to download the interpreter runtime itself
    more than once.


    As a result, I personally suspect that better documenting and
    promoting the CPython->Docker+PyPy migration option is likely to offer
    a more effective alternative to CPython->Go migrations than the more
    modest performance improvements we can make to the CPython runtime
    itself. (I still think the latter are a good idea, though - there's no
    point leaving Python 2.7 slower than it needs to be given the offer of
    assistance in maintaining it)


    Regards,
    Nick.


    --
    Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
  • Donald Stufft at May 28, 2015 at 2:37 pm

    On May 28, 2015 at 10:10:03 AM, Nick Coghlan (ncoghlan at gmail.com) wrote:
    On 28 May 2015 at 21:55, Maciej Fijalkowski wrote:
    I'm -1 on the idea because:

    * Performance improvements are not bug fixes
    * The patch doesn't make the migration process from Python 2 to Python 3 easier
    And this is why people have been porting Python applications to Go.
    For folks hitting the kinds of scalability problems that Go is
    designed to help with, a few percentage points here and there in
    CPython performance aren't going to make a big difference - they'll
    need the kinds of speed multipliers that PyPy can offer.

    Given that Go can't run Python C extensions any more than PyPy can,
    and involves a rewrite in a different programming language to boot,
    we'd do well to ponder what Go currently offers that PyPy doesn't. If
    we ignore the fashion-driven aspect of "Google wrote it, so it must be
    cool" (which we can't do anything about), and if we ignore the
    multi-vendor commercial support question (which tends to significantly
    lag community adoption for true community driven projects like PyPy),
    then one of the big keys in my view is the easy redistributability of
    Go binaries.

    For Linux based network services (and even Windows these days), Docker
    containers offer a potentially compelling way of bundling the PyPy
    runtime with Python applications, and Docker, Inc already maintain a
    set of PyPy base images at https://registry.hub.docker.com/_/pypy/

    Docker's image layering model then means that applications sharing a
    PyPy runtime shouldn't need to download the interpreter runtime itself
    more than once.

    As a result, I personally suspect that better documenting and
    promoting the CPython->Docker+PyPy migration option is likely to offer
    a more effective alternative to CPython->Go migrations than the more
    modest performance improvements we can make to the CPython runtime
    itself. (I still think the latter are a good idea, though - there's no
    point leaving Python 2.7 slower than it needs to be given the offer of
    assistance in maintaining it)

    Regards,
    Nick.

    --
    Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io



    I think docker is a pretty crummy answer to Go?s static binaries. What I would
    love is for Python to get:


    * The ability to import .so modules via zipzimport (ideally without a temporary
    ? directory, but that might require newer APIs from libc and such).
    * The ability to create a ?static? Python that links everything it needs into
    ? the binary to do a zipimport of everything else (including the stdlib).
    * The ability to execute a zipfile that has been concat onto the end of the
    ? Python binary.


    I think that if we get all of that, we could easily create a single file executable
    with real, native support from Python by simply compiling Python in that static
    mode and then appending a zip file containing the standard library and any other
    distributions we need to the end of it.


    We?d probably want some more quality of life improvements around accessing resources
    from within that zip file as well, but that can be done as a library easier than
    the above three things can.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
  • Paul Moore at May 28, 2015 at 2:52 pm

    On 28 May 2015 at 15:37, Donald Stufft wrote:
    I think docker is a pretty crummy answer to Go?s static binaries. What I would
    love is for Python to get:

    * The ability to import .so modules via zipzimport (ideally without a temporary
    directory, but that might require newer APIs from libc and such).
    * The ability to create a ?static? Python that links everything it needs into
    the binary to do a zipimport of everything else (including the stdlib).
    * The ability to execute a zipfile that has been concat onto the end of the
    Python binary.

    I think that if we get all of that, we could easily create a single file executable
    with real, native support from Python by simply compiling Python in that static
    mode and then appending a zip file containing the standard library and any other
    distributions we need to the end of it.

    We?d probably want some more quality of life improvements around accessing resources
    from within that zip file as well, but that can be done as a library easier than
    the above three things can.

    +1. The new embeddable Python distribution for Windows is a great step
    forward for this. It's not single-file, but it's easy to produce a
    single-directory self-contained application with it. I don't know if
    there's anything equivalent for Linux/OSX - maybe it's something we
    should look at for them as well (although the whole "static binaries"
    concept seems to be fairly frowned on in the Unix world, from what
    I've seen).


    Paul
  • Nick Coghlan at May 28, 2015 at 10:38 pm

    On 29 May 2015 00:52, "Paul Moore" wrote:
    +1. The new embeddable Python distribution for Windows is a great step
    forward for this. It's not single-file, but it's easy to produce a
    single-directory self-contained application with it. I don't know if
    there's anything equivalent for Linux/OSX - maybe it's something we
    should look at for them as well (although the whole "static binaries"
    concept seems to be fairly frowned on in the Unix world, from what
    I've seen).

    Correct - in the absence of the capacity to rebuild and redeploy the world
    at the click of a button, widespread deployment of static binaries poses an
    appallingly high security risk. It isn't an accident that Linux container
    orchestration is co-evolving with Linux container formats.


    Those efforts are mostly focused on network services & GUI applications,
    though. For portable console applications, Go is still one of the nicest
    options currently available, as the relatively limited ability to
    interoperate with the system provided C/C++ libraries makes it much harder
    to create dependencies between the app and the platform. It's similar to
    Java in that respect, but without the dependency on a large language
    runtime like the JVM.


    In that vein, it might be interesting to see what could be done with
    MicroPython in terms of providing a lightweight portable Python runtime
    without CPython's extensive integration with the underlying OS.


    Cheers,
    Nick.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150529/39986e3b/attachment.html>
  • Paul Sokolovsky at May 28, 2015 at 11:01 pm
    Hello,


    On Fri, 29 May 2015 08:38:44 +1000
    Nick Coghlan wrote:


    []

    In that vein, it might be interesting to see what could be done with
    MicroPython in terms of providing a lightweight portable Python
    runtime without CPython's extensive integration with the underlying
    OS.

    Thanks for mentioning that. That's definitely what I have on my mind,
    actually I wanted to do long ago a Lua-esque hack of being able to cat
    together an interpreter and a script, so resulting executable would
    just run the script. What stopped me is that it would be Lua-esque-ly
    useless, as how much useful one can do with a bare script without
    dependencies?


    And MicroPython definitely has some puzzle pieces for a generic
    solution, but so far not a complete picture:


    1. There're frozen modules support, but they're that - modules,
    packages not supported for freezing so far.


    2. Then frozen modules require recompiling, and that's not real-world
    scalable.


    3. Standard library is already "distributed" (vs CPython's monolithic),
    but half of modules are dummy so far.




    That said, making a demo of self-contained webapp server in 350-400K is
    definitely on my TODO list (package support for frozen modules is the
    only blocker for that).



    Cheers,
    Nick.





    --
    Best regards,
      Paul mailto:pmiscml at gmail.com
  • Eric Snow at May 29, 2015 at 1:47 am

    On Thu, May 28, 2015 at 5:01 PM, Paul Sokolovsky wrote:
    That said, making a demo of self-contained webapp server in 350-400K is
    definitely on my TODO list (package support for frozen modules is the
    only blocker for that).

    It may be worth taking this over to import-sig at python.org for more discussion.


    -eric
  • Xavier Combelle at May 31, 2015 at 9:14 am

    +1. The new embeddable Python distribution for Windows is a great step
    forward for this. It's not single-file, but it's easy to produce a
    single-directory self-contained application with it. I don't know if
    there's anything equivalent for Linux/OSX - maybe it's something we
    should look at for them as well (although the whole "static binaries"
    concept seems to be fairly frowned on in the Unix world, from what
    I've seen).

    Just curious What is "the new embeddable Python distribution for Windows" ?
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150531/4cc39e4d/attachment.html>
  • Paul Moore at May 31, 2015 at 10:41 am

    On 31 May 2015 at 10:14, Xavier Combelle wrote:
    +1. The new embeddable Python distribution for Windows is a great step
    forward for this. It's not single-file, but it's easy to produce a
    single-directory self-contained application with it. I don't know if
    there's anything equivalent for Linux/OSX - maybe it's something we
    should look at for them as well (although the whole "static binaries"
    concept seems to be fairly frowned on in the Unix world, from what
    I've seen).
    Just curious What is "the new embeddable Python distribution for Windows" ?

    Python 3.5 ships a zipfile which contains a self-contained Python
    installation, intended for embedding. The idea is that you unzip it
    into your application directory, and use it from within your
    application (either via the embedding API, or using the included
    python.exe/pythonw.exe). It doesn't use the registry, or any global
    resources, so it's independent of any installed python that might be
    present.


    Paul
  • Paul Moore at May 31, 2015 at 10:47 am

    On 31 May 2015 at 11:41, Paul Moore wrote:
    On 31 May 2015 at 10:14, Xavier Combelle wrote:
    +1. The new embeddable Python distribution for Windows is a great step
    forward for this. It's not single-file, but it's easy to produce a
    single-directory self-contained application with it. I don't know if
    there's anything equivalent for Linux/OSX - maybe it's something we
    should look at for them as well (although the whole "static binaries"
    concept seems to be fairly frowned on in the Unix world, from what
    I've seen).
    Just curious What is "the new embeddable Python distribution for Windows" ?
    Python 3.5 ships a zipfile which contains a self-contained Python
    installation, intended for embedding. The idea is that you unzip it
    into your application directory, and use it from within your
    application (either via the embedding API, or using the included
    python.exe/pythonw.exe). It doesn't use the registry, or any global
    resources, so it's independent of any installed python that might be
    present.

    By the way, IMO the new embeddable distribution is a pretty big deal
    on Windows. To make sure that it doesn't end up unnoticed, can I
    suggest we include a prominent "What's New" entry for it, and a
    section in "Python Setup and Usage" under "Using Python on Windows"
    for it?


    I'd hate to find that 3 or 4 versions from now, we're still trying to
    remind people that they can use the embeddable distribution, in the
    same way that executable zipfiles ended up an almost unknown feature
    for ages.


    Paul
  • Alexander Walters at May 31, 2015 at 10:59 am
    A better course of action would be to deprecate the non-portable
    version. Other than setting the PATH envvar, why do we need to continue
    even touching the system on install? It is highly annoying for those of
    us that maintain several installs of python on a single windows system,
    and it really should stop.


    The only use I can think of for ever touching the registry in the first
    place is to tell distutils installers where python is. I can tell you
    right now, that design choice is a bug. There are some mighty hacks you
    have to go through to correct that behavior when you happen to be using
    a virtualenv.


    (We are calling it 'embedable', but the rest of the world would call it
    'portable', as in, runable from a usb stick)

    On 5/31/2015 06:47, Paul Moore wrote:
    On 31 May 2015 at 11:41, Paul Moore wrote:
    On 31 May 2015 at 10:14, Xavier Combelle wrote:
    +1. The new embeddable Python distribution for Windows is a great step
    forward for this. It's not single-file, but it's easy to produce a
    single-directory self-contained application with it. I don't know if
    there's anything equivalent for Linux/OSX - maybe it's something we
    should look at for them as well (although the whole "static binaries"
    concept seems to be fairly frowned on in the Unix world, from what
    I've seen).
    Just curious What is "the new embeddable Python distribution for Windows" ?
    Python 3.5 ships a zipfile which contains a self-contained Python
    installation, intended for embedding. The idea is that you unzip it
    into your application directory, and use it from within your
    application (either via the embedding API, or using the included
    python.exe/pythonw.exe). It doesn't use the registry, or any global
    resources, so it's independent of any installed python that might be
    present.
    By the way, IMO the new embeddable distribution is a pretty big deal
    on Windows. To make sure that it doesn't end up unnoticed, can I
    suggest we include a prominent "What's New" entry for it, and a
    section in "Python Setup and Usage" under "Using Python on Windows"
    for it?

    I'd hate to find that 3 or 4 versions from now, we're still trying to
    remind people that they can use the embeddable distribution, in the
    same way that executable zipfiles ended up an almost unknown feature
    for ages.

    Paul
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com
  • Terry Reedy at May 31, 2015 at 2:44 pm

    On 5/31/2015 6:59 AM, Alexander Walters wrote:
    A better course of action would be to deprecate the non-portable
    version. Other than setting the PATH envvar, why do we need to continue
    even touching the system on install? It is highly annoying for those of
    us that maintain several installs of python on a single windows system,
    and it really should stop.

    Some people want the right-click context menu entries -- Run (also
    double click) and Edit with Idle, which should be Edit with Idle x.y.

    The only use I can think of for ever touching the registry in the first
    place is to tell distutils installers where python is. I can tell you
    right now, that design choice is a bug. There are some mighty hacks you
    have to go through to correct that behavior when you happen to be using
    a virtualenv.

    (We are calling it 'embedable', but the rest of the world would call it
    'portable', as in, runable from a usb stick)

    --
    Terry Jan Reedy
  • Nick Coghlan at May 31, 2015 at 3:16 pm

    On 1 June 2015 at 00:44, Terry Reedy wrote:
    On 5/31/2015 6:59 AM, Alexander Walters wrote:

    A better course of action would be to deprecate the non-portable
    version. Other than setting the PATH envvar, why do we need to continue
    even touching the system on install? It is highly annoying for those of
    us that maintain several installs of python on a single windows system,
    and it really should stop.

    Some people want the right-click context menu entries -- Run (also double
    click) and Edit with Idle, which should be Edit with Idle x.y.

    And system administrators responsible for deploying and maintaining
    Standard Operating Environments want the MSI integration. In that
    regard, the default behaviour of the python.org installers is the
    rough equivalent of the system Python on Linux distributions (with the
    added complexity of needing to deal with the Windows registry).


    Portable installations are often good for developers, but they come at
    the cost of failing to integrate properly with the underlying
    operating system.


    Cheers,
    Nick.


    --
    Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
  • Steve Dower at May 31, 2015 at 6:32 pm
    "We are calling it 'embedable', but the rest of the world would call it
    'portable', as in, runable from a usb stick"


    I called it embeddable because it's not intended for direct use and is not complete. There's no test suite, no documentation, no tkinter (pending high demand), no pip, no site-packages, and no folder structure. It really is meant to be a component in another application that provides the rest of the layout for its own needs. (I probably ought to blog about it so there's at least one detailed example of what it's for...)


    A nice side-effect is that you can make a regular per-user install portable by adding a pyvenv.cfg with "applocal = True", which disables regular path resolution (and also ignores PYTHONPATH, which is a feature or a bug, depending on your point of view). This only works on Windows right now, but could probably be ported from getpathp.c into getpath.c easily.


    Cheers,
    Steve


    Top-posted from my Windows Phone
    ________________________________
    From: Alexander Walters<mailto:tritium-list@sdamon.com>
    Sent: ?5/?31/?2015 6:39
    To: python-dev at python.org<mailto:python-dev@python.org>
    Subject: Re: [Python-Dev] Computed Goto dispatch for Python 2


    A better course of action would be to deprecate the non-portable
    version. Other than setting the PATH envvar, why do we need to continue
    even touching the system on install? It is highly annoying for those of
    us that maintain several installs of python on a single windows system,
    and it really should stop.


    The only use I can think of for ever touching the registry in the first
    place is to tell distutils installers where python is. I can tell you
    right now, that design choice is a bug. There are some mighty hacks you
    have to go through to correct that behavior when you happen to be using
    a virtualenv.


    (We are calling it 'embedable', but the rest of the world would call it
    'portable', as in, runable from a usb stick)

    On 5/31/2015 06:47, Paul Moore wrote:
    On 31 May 2015 at 11:41, Paul Moore wrote:
    On 31 May 2015 at 10:14, Xavier Combelle wrote:
    +1. The new embeddable Python distribution for Windows is a great step
    forward for this. It's not single-file, but it's easy to produce a
    single-directory self-contained application with it. I don't know if
    there's anything equivalent for Linux/OSX - maybe it's something we
    should look at for them as well (although the whole "static binaries"
    concept seems to be fairly frowned on in the Unix world, from what
    I've seen).
    Just curious What is "the new embeddable Python distribution for Windows" ?
    Python 3.5 ships a zipfile which contains a self-contained Python
    installation, intended for embedding. The idea is that you unzip it
    into your application directory, and use it from within your
    application (either via the embedding API, or using the included
    python.exe/pythonw.exe). It doesn't use the registry, or any global
    resources, so it's independent of any installed python that might be
    present.
    By the way, IMO the new embeddable distribution is a pretty big deal
    on Windows. To make sure that it doesn't end up unnoticed, can I
    suggest we include a prominent "What's New" entry for it, and a
    section in "Python Setup and Usage" under "Using Python on Windows"
    for it?

    I'd hate to find that 3 or 4 versions from now, we're still trying to
    remind people that they can use the embeddable distribution, in the
    same way that executable zipfiles ended up an almost unknown feature
    for ages.

    Paul
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com

    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150531/40036676/attachment.html>
  • Steve Dower at May 28, 2015 at 2:55 pm

    Donald Stufft wrote:
    I think docker is a pretty crummy answer to Go?s static binaries. What I would
    love is for Python to get:

    * The ability to import .so modules via zipzimport (ideally without a temporary
    directory, but that might require newer APIs from libc and such).
    * The ability to create a ?static? Python that links everything it needs into
    the binary to do a zipimport of everything else (including the stdlib).
    * The ability to execute a zipfile that has been concat onto the end of the
    Python binary.

    I think that if we get all of that, we could easily create a single file
    executable with real, native support from Python by simply compiling Python in
    that static mode and then appending a zip file containing the standard library
    and any other distributions we need to the end of it.

    And it would look like a 20MB+ file just for a simple 1KB Python script...


    For Windows at least, I'd prefer to have some app-style installer generation (e.g. http://pynsist.readthedocs.org/en/latest/) which, combined with the embeddable Python distro (new for 3.5.0b1 in case anyone missed it), can simply extract everything into an install directory and run it from there. None of the items on the list above are needed for or would help with this.


    (Some other Windows-specific advantages of the latter - installers get special compatibility treatment when the OS does stuff to break them, modifying official Python binaries breaks the signatures, signed executables are fully scanned before running (slow if the file is big), IT departments know how to deal with installers and users know how to deal with installed binaries, and probably more.)


    Alright everyone, back on topic now unless you want to rename the thread :)


    Cheers,
    Steve
  • Donald Stufft at May 28, 2015 at 3:01 pm

    On May 28, 2015 at 10:55:08 AM, Steve Dower (steve.dower at microsoft.com) wrote:
    Donald Stufft wrote:
    I think docker is a pretty crummy answer to Go?s static binaries. What I would
    love is for Python to get:

    * The ability to import .so modules via zipzimport (ideally without a temporary
    directory, but that might require newer APIs from libc and such).
    * The ability to create a ?static? Python that links everything it needs into
    the binary to do a zipimport of everything else (including the stdlib).
    * The ability to execute a zipfile that has been concat onto the end of the
    Python binary.

    I think that if we get all of that, we could easily create a single file
    executable with real, native support from Python by simply compiling Python in
    that static mode and then appending a zip file containing the standard library
    and any other distributions we need to the end of it.
    And it would look like a 20MB+ file just for a simple 1KB Python script...

    For Windows at least, I'd prefer to have some app-style installer generation (e.g. http://pynsist.readthedocs.org/en/latest/)
    which, combined with the embeddable Python distro (new for 3.5.0b1 in case anyone missed
    it), can simply extract everything into an install directory and run it from there. None
    of the items on the list above are needed for or would help with this.

    (Some other Windows-specific advantages of the latter - installers get special compatibility
    treatment when the OS does stuff to break them, modifying official Python binaries breaks
    the signatures, signed executables are fully scanned before running (slow if the file
    is big), IT departments know how to deal with installers and users know how to deal with
    installed binaries, and probably more.)

    Alright everyone, back on topic now unless you want to rename the thread :)

    Cheers,
    Steve

    ?

    Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python
    script + whatever other pieces you need. It would be entirely possible to only
    include the parts of the standard library you actually need. There's no rule
    that if your single file executable doesn't use xmlrpc that you have to include
    xmlrpc just for purities sake.


    This isn't something that can't be done today using something like PyInstaller,
    it's just super janky and finnicky because it's being hacked in after the fact
    by PyInstaller and because it's not an officially supported thing a lot of
    projects simply don't support it. A CLI I worked on that uses PyInstaller is
    5MB. It's certainly a trade off but it's not nearly as big of a trade off as
    you say.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
  • Terry Reedy at May 28, 2015 at 6:08 pm

    On 5/28/2015 10:55 AM, Steve Dower wrote:


    And it would look like a 20MB+ file just for a simple 1KB Python
    script...

    For Windows at least, I'd prefer to have some app-style installer
    generation (e.g. http://pynsist.readthedocs.org/en/latest/) which,
    combined with the embeddable Python distro (new for 3.5.0b1 in case
    anyone missed it), can simply extract everything into an install
    directory and run it from there. None of the items on the list above
    are needed for or would help with this.

    What I wish, of course, is that Windows just came with Python3, the way
    that DOS came with BASIC, so people could publish and trade Python
    programs the way we once did with BASIC programs. Then a simple 1KB
    Python script would just take an extra 1KB on disk. To me, the removal
    of a simple, builtin programming language for everyone was the biggest
    Windows mistake.


    Failing that, maybe PSF & edu-sig could somehow encourage universities
    that requires students to have a computer to recommend or even require
    that Python be loaded so students could at least assume that others
    students have Python already loaded. Python is the BASIC of the 21 century.


    Somehow, trying to make it easier to have 50 duplicate copies of Python
    on a system seems the wrong direction to go.


    --
    Terry Jan Reedy
  • Donald Stufft at May 28, 2015 at 6:33 pm

    On May 28, 2015 at 2:11:02 PM, Terry Reedy (tjreedy at udel.edu) wrote:
    On 5/28/2015 10:55 AM, Steve Dower wrote:

    And it would look like a 20MB+ file just for a simple 1KB Python
    script...

    For Windows at least, I'd prefer to have some app-style installer
    generation (e.g. http://pynsist.readthedocs.org/en/latest/) which,
    combined with the embeddable Python distro (new for 3.5.0b1 in case
    anyone missed it), can simply extract everything into an install
    directory and run it from there. None of the items on the list above
    are needed for or would help with this.
    What I wish, of course, is that Windows just came with Python3, the way
    that DOS came with BASIC, so people could publish and trade Python
    programs the way we once did with BASIC programs. Then a simple 1KB
    Python script would just take an extra 1KB on disk. To me, the removal
    of a simple, builtin programming language for everyone was the biggest
    Windows mistake.

    Failing that, maybe PSF & edu-sig could somehow encourage universities
    that requires students to have a computer to recommend or even require
    that Python be loaded so students could at least assume that others
    students have Python already loaded. Python is the BASIC of the 21 century.

    Somehow, trying to make it easier to have 50 duplicate copies of Python
    on a system seems the wrong direction to go.

    --
    Terry Jan Reedy

    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

    Honestly, I?m on an OS that *does* ship Python (OS X) and part of me hopes
    that they stop shipping it. It?s very rare that someone ships Python as
    part of their OS without modifying it in some way, and those modifications
    almost always cause pain to some set of users (and since I work on pip, they
    tend to come to us with the weirdo problems). Case in point: Python on OS X
    adds some preinstalled software, but they put this pre-installed software before
    site-packages in sys.path, so pip can?t upgrade those pre-installed software
    packages at all.?


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
  • Sturla Molden at May 28, 2015 at 7:25 pm

    Donald Stufft wrote:


    Honestly, I?m on an OS that *does* ship Python (OS X) and part of me hopes
    that they stop shipping it. It?s very rare that someone ships Python as
    part of their OS without modifying it in some way, and those modifications
    almost always cause pain to some set of users (and since I work on pip, they
    tend to come to us with the weirdo problems). Case in point: Python on OS X
    adds some preinstalled software, but they put this pre-installed software before
    site-packages in sys.path, so pip can?t upgrade those pre-installed software
    packages at all.

    Many Unix tools need Python, so Mac OS X (like Linux distros and FreeBSD)
    will always need a system Python. Yes, it would be great if could be called
    spython or something else than python. But the main problem is that it is
    used by end-users as well, not just the operating system.


    Anyone who use Python on OSX should install their own Python. The system
    Python should be left alone as it is.


    If the system Python needs updating, it is the responsibility of Apple to
    distribute the upgrade. Nobody should attempt to use pip to update the
    system Python. Who knows what side-effects it might have. Preferably pip
    should have a check for it and bluntly refuse to do it.


    Sturla
  • Chris Barker at May 28, 2015 at 7:37 pm

    On Thu, May 28, 2015 at 12:25 PM, Sturla Molden wrote:


    Many Unix tools need Python, so Mac OS X (like Linux distros and FreeBSD)
    will always need a system Python. Yes, it would be great if could be called
    spython or something else than python. But the main problem is that it is
    used by end-users as well, not just the operating system.

    I think it's great for it to be used by end users as a system library /
    utility. i.e. like you would a the system libc -- so if you can write a
    little python script that only uses the stdlib -- you can simply deliver
    that script.


    But if you want to go an install a bunch of extra non-standard packages
    (or, for heaven's sake, want a version with bug fixes!), they you really
    are better off installing a new python you can control.


    The system
    Python should be left alone as it is.

    absolutely!


    By the way, py2app will build an application bundle that depends on the
    system python, indeed, that's all it will do if you run it with the system
    python, as Apple has added some non-redistributable bits in there. But
    things get kin dof confusing if you want to rely on non-system packages...


    -Chris






    --


    Christopher Barker, Ph.D.
    Oceanographer


    Emergency Response Division
    NOAA/NOS/OR&R (206) 526-6959 voice
    7600 Sand Point Way NE (206) 526-6329 fax
    Seattle, WA 98115 (206) 526-6317 main reception


    Chris.Barker at noaa.gov
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/71b50e13/attachment.html>
  • Sturla Molden at May 28, 2015 at 8:48 pm

    On 28/05/15 21:37, Chris Barker wrote:


    I think it's great for it to be used by end users as a system library /
    utility. i.e. like you would a the system libc -- so if you can write a
    little python script that only uses the stdlib -- you can simply deliver
    that script.

    No it is not, because someone will be 'clever' and try to upgrade it
    with pip or install packages into it.


    Sturla
  • Ronald Oussoren at May 29, 2015 at 9:37 pm

    Op 28 mei 2015 om 21:37 heeft Chris Barker <chris.barker@noaa.gov> het volgende geschreven:

    On Thu, May 28, 2015 at 12:25 PM, Sturla Molden wrote:

    The system
    Python should be left alone as it is.
    absolutely!

    By the way, py2app will build an application bundle that depends on the system python, indeed, that's all it will do if you run it with the system python, as Apple has added some non-redistributable bits in there.

    That's not quite the reason. It's more that I don't want to guess whether or not it is valid to bundle binaries from a system location. Furthermore bundling files from a base install of the OS is pretty useless, especially when those binaries won't run on earlier releases anyway due to the compilation options used.


    Ronald
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150529/f9065302/attachment.html>
  • Barry Warsaw at May 28, 2015 at 4:13 pm
    Go seems to be popular where I work. It is replacing Python in a number of
    places, although Python (and especially Python 3) is still a very important
    part of our language toolbox.


    There are several reasons why Go is gaining popularity. Single-file
    executables is definitely a reason; it makes deployment very easy, even if it
    increases the maintenance burden (e.g. without shared libraries, you have
    multiple copies of things so when a security fix is required for one of those
    things you have to recompile the world).


    Start up times and memory footprint are also factors. Probably not much to be
    done about the latter, but perhaps PEP 432 can lead to improvements in the
    former. (Hey Nick, I'm guessing you'll want to bump that one back to 3.6.)


    Certainly better support for multi-cores comes up a lot. It should be a SMoE
    to just get rid of the GIL once and for all <wink>.


    One thing I've seen more than once is that new development happens in Python
    until the problem is understood, then the code is ported to Go. Python's
    short path from idea to working code, along with its ability to quickly morph
    as requirements and understanding changes, its batteries included philosophy,
    and its "fits-your-brain" consistency are its biggest strengths!


    On May 28, 2015, at 10:37 AM, Donald Stufft wrote:

    I think docker is a pretty crummy answer to Go?s static binaries. What I would
    love is for Python to get:

    * The ability to import .so modules via zipzimport (ideally without a
    temporary ? directory, but that might require newer APIs from libc and such).

    +1 - Thomas Wouters mentioned at the language summit some work being done on
    glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
    files directly from a zip. Not sure what the status is of that, but it would
    be a great addition.

    * The ability to create a ?static? Python that links everything it needs into
    the binary to do a zipimport of everything else (including the stdlib).

    +1

    *The ability to execute a zipfile that has been concat onto the end of the ?
    Python binary.

    +1

    I think that if we get all of that, we could easily create a single file
    executable with real, native support from Python by simply compiling Python
    in that static mode and then appending a zip file containing the standard
    library and any other distributions we need to the end of it.

    We?d probably want some more quality of life improvements around accessing
    resources from within that zip file as well, but that can be done as a
    library easier than the above three things can.

    E.g. you really should be using the pkg_resources APIs for loading resources
    from your packages, otherwise you're gonna have problems with zip
    executables. We've talked before about adopting some of these APIs into
    Python's stdlib. pkgutil is a start, and the higher level APIs from
    pkg_resources should probably go there.


    Cheers,
    -Barry
  • Guido van Rossum at May 28, 2015 at 4:47 pm
    Single-file binaries are indeed important. (Though in most cases they don't
    have to be totally stand-alone -- they can depend on a system python and
    its stdlib. At least in typical datacenter setups.) Have people looked at
    PEX (a format developed by Twitter) or Pants (which seems to be an
    open-source tool that can build PEX files)?


    ?ukasz has told me that at Facebook they have a similar system that they
    are now using to deploy Python 3 binaries.


    --
    --Guido van Rossum (python.org/~guido)
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/c8a36f13/attachment.html>
  • Paul Moore at May 28, 2015 at 5:15 pm

    On 28 May 2015 at 17:47, Guido van Rossum wrote:
    Single-file binaries are indeed important. (Though in most cases they don't
    have to be totally stand-alone -- they can depend on a system python and its
    stdlib. At least in typical datacenter setups.) Have people looked at PEX (a
    format developed by Twitter) or Pants (which seems to be an open-source tool
    that can build PEX files)?

    There appear to be problems with pex on Windows. I've reported a
    couple of bugs, which appear to have been fixed (although I don't know
    the timeframe, I lost interest before a fix was implemented). But
    there are new ones I just found on a quick test.


    I'd like pex to work, it looks like a nice tool. But I don't want to
    be their Windows support resource, and it seems like they may not
    currently have anyone else :-)


    Paul


    (I'll probably go and lie down now and stop banging the cross-platform
    drum for a while :-))
  • Paul Moore at May 28, 2015 at 5:01 pm

    On 28 May 2015 at 17:13, Barry Warsaw wrote:
    On May 28, 2015, at 10:37 AM, Donald Stufft wrote:

    I think docker is a pretty crummy answer to Go?s static binaries. What I would
    love is for Python to get:

    * The ability to import .so modules via zipzimport (ideally without a
    temporary directory, but that might require newer APIs from libc and such).
    +1 - Thomas Wouters mentioned at the language summit some work being done on
    glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
    files directly from a zip. Not sure what the status is of that, but it would
    be a great addition.

    +1 but it needs to be cross-platform - py2exe has something similar
    for Windows, which we should expect to use if the glibc solution is
    Unix-only.

    * The ability to create a ?static? Python that links everything it needs into
    the binary to do a zipimport of everything else (including the stdlib).
    +1

    +0 - I suppose it would be nice, but how important is this really? If
    all we do is end up with a single massive file instead of a directory,
    then I don't see we've gained much.


    Smallish C programs tend to hit the 10-100k executable size. Offhand I
    don't know how big a typical go "single file executable" is. If I
    could bundle up my Python script (something simple, let's say a "Hello
    world" for argument's sake) and get a 10-20k single-file executable,
    this would be worthwhile. If the resulting exe was 5MB (which is what
    today's solutions that bundle the Python DLL and full zipped stdlib
    tend to weigh in at) then I'm not interested.

    *The ability to execute a zipfile that has been concat onto the end of the
    Python binary.
    +1

    +1. This would be immensely useful to tack on the front of a pyz
    archive. Even just the existing (39K on Windows) python.exe which
    relies on the Python installation would be great. Basically it could
    replace the setuptools/distlib "script wrapper" executables with
    something that doesn't need to run 2 processes just to run a script.

    We?d probably want some more quality of life improvements around accessing
    resources from within that zip file as well, but that can be done as a
    library easier than the above three things can.
    E.g. you really should be using the pkg_resources APIs for loading resources
    from your packages, otherwise you're gonna have problems with zip
    executables. We've talked before about adopting some of these APIs into
    Python's stdlib. pkgutil is a start, and the higher level APIs from
    pkg_resources should probably go there.

    +1. We need the APIs available in the stdlib, then 3rd party libraries
    have a solution when users complain "your library isn't zipimport
    safe". (Of course "well, we have to support Python 2.7/3.4" remains a
    response for some time yet :-() Then we can start working on the
    culture change where library authors start expecting their code to be
    deployed in single-file "run-from-zip" applications.


    Paul
  • Brett Cannon at May 28, 2015 at 5:25 pm
    On Thu, May 28, 2015, 12:14 Barry Warsaw wrote:


    Go seems to be popular where I work. It is replacing Python in a number of
    places, although Python (and especially Python 3) is still a very important
    part of our language toolbox.


    There are several reasons why Go is gaining popularity. Single-file
    executables is definitely a reason; it makes deployment very easy, even if
    it
    increases the maintenance burden (e.g. without shared libraries, you have
    multiple copies of things so when a security fix is required for one of
    those
    things you have to recompile the world).


    Start up times and memory footprint are also factors. Probably not much to
    be
    done about the latter, but perhaps PEP 432 can lead to improvements in the
    former. (Hey Nick, I'm guessing you'll want to bump that one back to 3.6.)


    Certainly better support for multi-cores comes up a lot. It should be a
    SMoE
    to just get rid of the GIL once and for all <wink>.


    One thing I've seen more than once is that new development happens in Python
    until the problem is understood, then the code is ported to Go. Python's
    short path from idea to working code, along with its ability to quickly
    morph
    as requirements and understanding changes, its batteries included
    philosophy,
    and its "fits-your-brain" consistency are its biggest strengths!


    On May 28, 2015, at 10:37 AM, Donald Stufft wrote:

    I think docker is a pretty crummy answer to Go?s static binaries. What I would
    love is for Python to get:

    * The ability to import .so modules via zipzimport (ideally without a
    temporary directory, but that might require newer APIs from libc and
    such).


    +1 - Thomas Wouters mentioned at the language summit some work being done on
    glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
    files directly from a zip. Not sure what the status is of that, but it
    would
    be a great addition.

    * The ability to create a ?static? Python that links everything it needs into
    the binary to do a zipimport of everything else (including the stdlib).

    +1

    *The ability to execute a zipfile that has been concat onto the end of the
    Python binary.

    +1

    I think that if we get all of that, we could easily create a single file
    executable with real, native support from Python by simply compiling Python
    in that static mode and then appending a zip file containing the standard
    library and any other distributions we need to the end of it.

    We?d probably want some more quality of life improvements around accessing
    resources from within that zip file as well, but that can be done as a
    library easier than the above three things can.

    E.g. you really should be using the pkg_resources APIs for loading resources
    from your packages, otherwise you're gonna have problems with zip
    executables. We've talked before about adopting some of these APIs into
    Python's stdlib. pkgutil is a start, and the higher level APIs from
    pkg_resources should probably go there.


      Donald Stuff proposed importlib.resources a little while back to handle
    the storage-agnostic api dor reading data and I have been thinking about it
    for years. I plan to make it happen in Python 3.6.


    -brett




    Cheers,
    -Barry


    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/brett%40python.org
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/31967f52/attachment.html>
  • Matthias Klose at May 28, 2015 at 9:48 pm

    On 05/28/2015 06:13 PM, Barry Warsaw wrote:
    Go seems to be popular where I work. It is replacing Python in a number of
    places, although Python (and especially Python 3) is still a very important
    part of our language toolbox.

    There are several reasons why Go is gaining popularity. Single-file
    executables is definitely a reason; it makes deployment very easy, even if it
    increases the maintenance burden (e.g. without shared libraries, you have
    multiple copies of things so when a security fix is required for one of those
    things you have to recompile the world).

    And the very same place where you are working is investing in getting shared
    libraries working for Go. Single binaries may be popular for distributing end
    user applications, but definitely not for distributing a core OS or a SDK.
    Sorry, you didn't yet arrive in distro land ...


    Matthias
  • Donald Stufft at May 28, 2015 at 10:04 pm

    On May 28, 2015 at 5:50:32 PM, Matthias Klose (doko at ubuntu.com) wrote:
    On 05/28/2015 06:13 PM, Barry Warsaw wrote:
    Go seems to be popular where I work. It is replacing Python in a number of
    places, although Python (and especially Python 3) is still a very important
    part of our language toolbox.

    There are several reasons why Go is gaining popularity. Single-file
    executables is definitely a reason; it makes deployment very easy, even if it
    increases the maintenance burden (e.g. without shared libraries, you have
    multiple copies of things so when a security fix is required for one of those
    things you have to recompile the world).
    And the very same place where you are working is investing in getting shared
    libraries working for Go. Single binaries may be popular for distributing end
    user applications, but definitely not for distributing a core OS or a SDK.
    Sorry, you didn't yet arrive in distro land ...

    I don?t think anyone is claiming that single file should be the *only* way, just
    that for a sizable set of people it is a very attractive way.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
  • Paul Sokolovsky at May 28, 2015 at 10:08 pm
    Hello,


    On Thu, 28 May 2015 23:48:59 +0200
    Matthias Klose wrote:


    []

    And the very same place where you are working is investing in getting
    shared libraries working for Go. Single binaries may be popular for
    distributing end user applications, but definitely not for
    distributing a core OS or a SDK. Sorry, you didn't yet arrive in
    distro land ...

    Of course it did. Like, Ubuntu 14.04LTS ships Go 1.2. No, it starts
    with the fact that when you don't have Go installed and type "go", it
    suggests to install gccgo, which just segfaults on running. Then you
    figure out that you need to install "golang", and that's 1.2, and a lot
    of things simply don't work with that version, like "go get" reports
    that a package not found, while it perfectly exists. So, let Go stay
    what it is - a corporate toy lingo for press-releases. That's until
    Google has thought that it generated enough buzz and it's time to shut
    it down like their numerous other projects. (Isn't Go old already and
    "everyone" uses Rust?)




    --
    Best regards,
      Paul mailto:pmiscml at gmail.com
  • Ian Cordasco at May 28, 2015 at 10:18 pm

    On Thu, May 28, 2015 at 5:08 PM, Paul Sokolovsky wrote:
    Hello,

    On Thu, 28 May 2015 23:48:59 +0200
    Matthias Klose wrote:

    []
    And the very same place where you are working is investing in getting
    shared libraries working for Go. Single binaries may be popular for
    distributing end user applications, but definitely not for
    distributing a core OS or a SDK. Sorry, you didn't yet arrive in
    distro land ...
    Of course it did. Like, Ubuntu 14.04LTS ships Go 1.2. No, it starts
    with the fact that when you don't have Go installed and type "go", it
    suggests to install gccgo, which just segfaults on running. Then you
    figure out that you need to install "golang", and that's 1.2, and a lot
    of things simply don't work with that version, like "go get" reports
    that a package not found, while it perfectly exists. So, let Go stay
    what it is - a corporate toy lingo for press-releases. That's until
    Google has thought that it generated enough buzz and it's time to shut
    it down like their numerous other projects. (Isn't Go old already and
    "everyone" uses Rust?)


    --
    Best regards,
    Paul mailto:pmiscml at gmail.com
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe: https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com

    Note that as much as I love Rust, it still isn't the replacement for
    Go. It doesn't have a stable ABI so if you distribute a binary and
    that person has a different version of Rust 1.x installed, it won't be
    guaranteed to work (and, at this point, probably won't work anyway).
    Go is just more popular because it's been around longer and it (as far
    as a single developer is concerned) gets rid of the dependency mess.
    That's why developers like it.
  • Barry Warsaw at May 28, 2015 at 11:01 pm
    On May 28, 2015, at 11:48 PM, Matthias Klose wrote:

    And the very same place where you are working is investing in getting shared
    libraries working for Go. Single binaries may be popular for distributing end
    user applications, but definitely not for distributing a core OS or a SDK.

    Yep, I mentioned this in an earlier message (probably buried in the
    centithread avalanche). Both static and dynamic linking have valid use
    cases.


    Cheers,
    -Barry
  • Nick Coghlan at May 28, 2015 at 11:38 pm

    On 29 May 2015 2:16 am, "Barry Warsaw" wrote:
    Go seems to be popular where I work. It is replacing Python in a number of
    places, although Python (and especially Python 3) is still a very important
    part of our language toolbox.

    There are several reasons why Go is gaining popularity. Single-file
    executables is definitely a reason; it makes deployment very easy, even if it
    increases the maintenance burden (e.g. without shared libraries, you have
    multiple copies of things so when a security fix is required for one of those
    things you have to recompile the world).

    Start up times and memory footprint are also factors. Probably not much to be
    done about the latter, but perhaps PEP 432 can lead to improvements in the
    former. (Hey Nick, I'm guessing you'll want to bump that one back to
    3.6.)


    Yep. I got the feature branch mostly working again just after PyCon (since
    several folks expressed interest in helping with it), and thanks to Eric
    Snow, the biggest blocker to further progress (splitting the import system
    initialisation into two distinct phases) has already been addressed for 3.5
    (that's not merged into the feature branch in my sandbox repo yet, though).


    PEP 432 itself isn't likely to change startup time for the full interpreter
    runtime very much (as it's mostly about rearranging how we call the
    existing setup steps, rather than changing the steps themselves), but
    having more of the C API available earlier in the bootstrapping cycle will
    hopefully lay a foundation for future improvements.


    The intent is also to make embedding *much* easier, and have it be trivial
    to skip initialising any subsystems that a given application doesn't need.

    Certainly better support for multi-cores comes up a lot. It should be a SMoE
    to just get rid of the GIL once and for all <wink>.

    Eric's been looking into this as well, and we think there's a plausible
    path forward based on changing the way subinterpreters work such that the
    GIL can be changed to a read/write lock, and each subinterpreter gets its
    own Local Interpreter Lock.


    Expect to hear more on that front before too long :)

    One thing I've seen more than once is that new development happens in Python
    until the problem is understood, then the code is ported to Go. Python's
    short path from idea to working code, along with its ability to quickly morph
    as requirements and understanding changes, its batteries included
    philosophy,
    and its "fits-your-brain" consistency are its biggest strengths!

    Right, Go is displacing C/C++ in that regard (moreso than Python itself),
    and now that Rust has hit 1.0, I expect we'll see it becoming another
    contender for this task. Rust's big advantage over Go in that regard is
    being compatible with the C/C++ ecosystem, including Python's cffi.


    Cheers,
    Nick.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150529/fa60ca57/attachment-0001.html>
  • Donald Stufft at May 28, 2015 at 11:48 pm

    On May 28, 2015 at 7:40:26 PM, Nick Coghlan (ncoghlan at gmail.com) wrote:

    One thing I've seen more than once is that new development happens in Python
    until the problem is understood, then the code is ported to Go. Python's
    short path from idea to working code, along with its ability
    to quickly morph
    as requirements and understanding changes, its batteries
    included philosophy,
    and its "fits-your-brain" consistency are its biggest strengths!

    Right, Go is displacing C/C++ in that regard (moreso than Python
    itself), and now that Rust has hit 1.0, I expect we'll see it becoming
    another contender for this task. Rust's big advantage over Go
    in that regard is being compatible with the C/C++ ecosystem,
    including Python's cffi.

    I?m not sure if I?m reading this right or not, but just to be clear, I?ve
    seen a number of people express the sentiment that they are switching from
    Python to Go and that the deployment story is one of the reasons. It?s not
    just people switching from C/C++.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
  • Nick Coghlan at May 29, 2015 at 6:58 am

    On 29 May 2015 9:48 am, "Donald Stufft" wrote:

    On May 28, 2015 at 7:40:26 PM, Nick Coghlan (ncoghlan at gmail.com) wrote:

    One thing I've seen more than once is that new development happens in Python
    until the problem is understood, then the code is ported to Go. Python's
    short path from idea to working code, along with its ability
    to quickly morph
    as requirements and understanding changes, its batteries
    included philosophy,
    and its "fits-your-brain" consistency are its biggest strengths!

    Right, Go is displacing C/C++ in that regard (moreso than Python
    itself), and now that Rust has hit 1.0, I expect we'll see it becoming
    another contender for this task. Rust's big advantage over Go
    in that regard is being compatible with the C/C++ ecosystem,
    including Python's cffi.
    I?m not sure if I?m reading this right or not, but just to be clear, I?ve
    seen a number of people express the sentiment that they are switching from
    Python to Go and that the deployment story is one of the reasons. It?s not
    just people switching from C/C++.

    C and C++ used to be the main "second version" languages used to create
    statically linked standalone binaries after an initial prototype in Python.


    Folks that learned Python first understandably weren't keen on that idea,
    so they tended to either use Cython (or its predecessor, Pyrex), or else
    not bother doing it at all until first Go and now Rust came along (for
    reasons unknown to me, D appears to have never gained any popularity
    outside the ACM crowd).


    If I seem blase about Go, that's the main reason why - the benefits it
    offers aren't novel from the point of view of C/C++ programmers, they're
    just now available without having to put up with arcane syntax, manual
    memory management, an unmaintainable threading model, relatively poor
    support for text manipulation, etc, etc.


    There's no shortage of software needing to be written, so powerful new
    additions to our collective toolkit like Go are advancements to be
    celebrated and learned from, rather than feared.


    Cheers,
    Nick.

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150529/c3579acb/attachment.html>
  • Donald Stufft at May 29, 2015 at 12:50 pm

    On May 29, 2015 at 2:58:28 AM, Nick Coghlan (ncoghlan at gmail.com) wrote:
    On 29 May 2015 9:48 am, "Donald Stufft" wrote:


    On May 28, 2015 at 7:40:26 PM, Nick Coghlan (ncoghlan at gmail.com) wrote:

    One thing I've seen more than once is that new development happens in Python
    until the problem is understood, then the code is ported to Go. Python's
    short path from idea to working code, along with its ability
    to quickly morph
    as requirements and understanding changes, its batteries
    included philosophy,
    and its "fits-your-brain" consistency are its biggest strengths!

    Right, Go is displacing C/C++ in that regard (moreso than Python
    itself), and now that Rust has hit 1.0, I expect we'll see it becoming
    another contender for this task. Rust's big advantage over Go
    in that regard is being compatible with the C/C++ ecosystem,
    including Python's cffi.
    I?m not sure if I?m reading this right or not, but just to be clear, I?ve
    seen a number of people express the sentiment that they are switching from
    Python to Go and that the deployment story is one of the reasons. It?s not
    just people switching from C/C++.
    C and C++ used to be the main "second version" languages used to create
    statically linked standalone binaries after an initial prototype in Python.

    Folks that learned Python first understandably weren't keen on that idea,
    so they tended to either use Cython (or its predecessor, Pyrex), or else
    not bother doing it at all until first Go and now Rust came along (for
    reasons unknown to me, D appears to have never gained any popularity
    outside the ACM crowd).

    If I seem blase about Go, that's the main reason why - the benefits it
    offers aren't novel from the point of view of C/C++ programmers, they're
    just now available without having to put up with arcane syntax, manual
    memory management, an unmaintainable threading model, relatively poor
    support for text manipulation, etc, etc.

    I don't think Go is going to "kill" Python or anything, but I do think that
    not taking a look at other languages and why people are picking them over
    Python is important, otherwise we will end up dying (and would deserve to)
    because we'd be like the big company that didn't bother to keep up with the
    times and just assumed we'd be big forever. I talk to a lot of people about
    the distribution story of Python applications, what works and what doesn't.
    A very large majority of the people who have used both Go and Python in a
    serious capacity have indicated that they've at least considered writing new
    things in Go instead of Python due to the fact that distributing it is much
    easier and a not insignificant number of them have in fact started to switch
    to using Go in situations where they are trying to distribute things to
    disparate boxes.


    This might be something that people could have done before with C/C++ but with
    a nicer language behind it... but that's kind of the point? You don't need to
    be stuck with a terrible language to get a nice single file executable anymore,
    you can get that and use a good language at the same time which makes it a lot
    more compelling to a lot more people than having to be stuck with C.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
  • André Freitas at May 29, 2015 at 1:41 pm
    Speaking about distribution I believe Pip is the simplest way of
    distributing. I have used some freezing tools in the past such cxfreeze but
    with more complex projects they start being hard to manage. Now instead of
    saying people to goto an url, download and put in the path I just say: pip
    install <project>


    Unfortunately, this approach only works well with products built for
    developers.


    A Sex, 29/05/2015, 13:50, Donald Stufft <donald@stufft.io> escreveu:

    On May 29, 2015 at 2:58:28 AM, Nick Coghlan (ncoghlan at gmail.com) wrote:
    On 29 May 2015 9:48 am, "Donald Stufft" wrote:



    On May 28, 2015 at 7:40:26 PM, Nick Coghlan (ncoghlan at gmail.com)
    wrote:
    One thing I've seen more than once is that new development happens in Python
    until the problem is understood, then the code is ported to Go. Python's
    short path from idea to working code, along with its ability
    to quickly morph
    as requirements and understanding changes, its batteries
    included philosophy,
    and its "fits-your-brain" consistency are its biggest strengths!

    Right, Go is displacing C/C++ in that regard (moreso than Python
    itself), and now that Rust has hit 1.0, I expect we'll see it
    becoming
    another contender for this task. Rust's big advantage over Go
    in that regard is being compatible with the C/C++ ecosystem,
    including Python's cffi.
    I?m not sure if I?m reading this right or not, but just to be clear,
    I?ve
    seen a number of people express the sentiment that they are switching
    from
    Python to Go and that the deployment story is one of the reasons. It?s
    not
    just people switching from C/C++.
    C and C++ used to be the main "second version" languages used to create
    statically linked standalone binaries after an initial prototype in Python.
    Folks that learned Python first understandably weren't keen on that idea,
    so they tended to either use Cython (or its predecessor, Pyrex), or else
    not bother doing it at all until first Go and now Rust came along (for
    reasons unknown to me, D appears to have never gained any popularity
    outside the ACM crowd).

    If I seem blase about Go, that's the main reason why - the benefits it
    offers aren't novel from the point of view of C/C++ programmers, they're
    just now available without having to put up with arcane syntax, manual
    memory management, an unmaintainable threading model, relatively poor
    support for text manipulation, etc, etc.
    I don't think Go is going to "kill" Python or anything, but I do think that
    not taking a look at other languages and why people are picking them over
    Python is important, otherwise we will end up dying (and would deserve to)
    because we'd be like the big company that didn't bother to keep up with the
    times and just assumed we'd be big forever. I talk to a lot of people about
    the distribution story of Python applications, what works and what doesn't.
    A very large majority of the people who have used both Go and Python in a
    serious capacity have indicated that they've at least considered writing
    new
    things in Go instead of Python due to the fact that distributing it is much
    easier and a not insignificant number of them have in fact started to
    switch
    to using Go in situations where they are trying to distribute things to
    disparate boxes.

    This might be something that people could have done before with C/C++ but
    with
    a nicer language behind it... but that's kind of the point? You don't need
    to
    be stuck with a terrible language to get a nice single file executable
    anymore,
    you can get that and use a good language at the same time which makes it a
    lot
    more compelling to a lot more people than having to be stuck with C.

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/p.andrefreitas%40gmail.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150529/66e3de14/attachment.html>
  • Nick Coghlan at May 29, 2015 at 11:08 pm

    On 29 May 2015 22:50, "Donald Stufft" wrote:
    This might be something that people could have done before with C/C++ but with
    a nicer language behind it... but that's kind of the point? You don't need to
    be stuck with a terrible language to get a nice single file executable anymore,
    you can get that and use a good language at the same time which makes it a lot
    more compelling to a lot more people than having to be stuck with C.

    Right, but the only things you can really write in Go are network services
    and console applications - once you start looking at curses & GUI
    applications on the end user side, you're back to the same kind of
    distribution complexity as C/C++ (where you have to choose between external
    dependency management or very large downloads), and once you start looking
    at the infrastructure side, Docker, Rocket & Kubernetes are bringing this
    kind of easy deployability to network services written in arbitrary
    languages.


    Hence my comment about MicroPython: the easiest way to make an interpreter
    runtime that's lighter than CPython is to have it *do less*.


    Communicating with embedded MicroPython instances via cffi could even
    potentially offer a way for both CPython and PyPy to split work across
    multiple cores without having to fundamentally redesign their main
    interpreters.


    Cheers,
    Nick.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150530/93121c46/attachment.html>
  • Thomas Wouters at May 29, 2015 at 2:04 am

    On Thu, May 28, 2015 at 6:13 PM, Barry Warsaw wrote:


    Go seems to be popular where I work. It is replacing Python in a number of
    places, although Python (and especially Python 3) is still a very important
    part of our language toolbox.

    There are several reasons why Go is gaining popularity. Single-file
    executables is definitely a reason; it makes deployment very easy, even if
    it
    increases the maintenance burden (e.g. without shared libraries, you have
    multiple copies of things so when a security fix is required for one of
    those
    things you have to recompile the world).

    Start up times and memory footprint are also factors. Probably not much
    to be
    done about the latter, but perhaps PEP 432 can lead to improvements in the
    former. (Hey Nick, I'm guessing you'll want to bump that one back to 3.6.)

    Certainly better support for multi-cores comes up a lot. It should be a
    SMoE
    to just get rid of the GIL once and for all <wink>.

    One thing I've seen more than once is that new development happens in
    Python
    until the problem is understood, then the code is ported to Go. Python's
    short path from idea to working code, along with its ability to quickly
    morph
    as requirements and understanding changes, its batteries included
    philosophy,
    and its "fits-your-brain" consistency are its biggest strengths!
    On May 28, 2015, at 10:37 AM, Donald Stufft wrote:

    I think docker is a pretty crummy answer to Go?s static binaries. What I would
    love is for Python to get:

    * The ability to import .so modules via zipzimport (ideally without a
    temporary directory, but that might require newer APIs from libc and
    such).

    +1 - Thomas Wouters mentioned at the language summit some work being done
    on
    glibc to add dlopen_from_memory() (sp?) which would allow for loading .so
    files directly from a zip. Not sure what the status is of that, but it
    would
    be a great addition.

    It's dlopen_with_offset:
    https://sourceware.org/bugzilla/show_bug.cgi?id767. There's also a patch
    that's sort-of dlopen_from_memory (dlopen_ehdr), but it requires a lot of
    manual setup to map the right bits to the right places; dlopen_with_offset
    is a lot simpler.


    Building a Python application into a single file doesn't require
    dlopen_with_offset, *iff* you build everything from source. It's not easy
    to do this -- Python's setup.py and third-party's uses of distutils don't
    allow this -- but it's mostly possible using the old Modules/Setup file.
    (Or you could do what we routinely do at Google with third-party packages
    and re-implement the build in your own build system ;P)





    * The ability to create a ?static? Python that links everything it needs into
    the binary to do a zipimport of everything else (including the stdlib).
    This is possible (with some jumping through hoops) using Modules/Setup and
    some post-processing of the standard library. It would be a lot easier if
    we got rid of distutils for building Python (or for everything) -- or made
    it output a Modules/Setup-like file :) (For those who don't remember,
    Modules/Setup was the file we used to build stdlib extension modules before
    we had distutils, and it's parsed and incorporated into the regular
    Makefile. It can build both static and dynamic extension modules.)



    +1
    *The ability to execute a zipfile that has been concat onto the end of the
    Python binary.

    This is already possible, just not with the regular 'python' binary. All it
    takes is fifty lines of C or so, a tiny application that embeds Python,
    sets sys.path[0] to argv[0], and uses the runpy module to execute something
    from the ZIP file. There are some issues with this approach (like what
    sys.executable should be :) but they're mostly cosmetic



    +1
    I think that if we get all of that, we could easily create a single file
    executable with real, native support from Python by simply compiling Python
    in that static mode and then appending a zip file containing the standard
    library and any other distributions we need to the end of it.

    We?d probably want some more quality of life improvements around accessing
    resources from within that zip file as well, but that can be done as a
    library easier than the above three things can.
    E.g. you really should be using the pkg_resources APIs for loading
    resources
    from your packages, otherwise you're gonna have problems with zip
    executables. We've talked before about adopting some of these APIs into
    Python's stdlib. pkgutil is a start, and the higher level APIs from
    pkg_resources should probably go there.

    Cheers,
    -Barry

    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/thomas%40python.org





    --
    Thomas Wouters <thomas@python.org>


    Hi! I'm an email virus! Think twice before sending your email to help me
    spread!
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150529/1f9afe42/attachment-0001.html>
  • Raymond Hettinger at May 28, 2015 at 12:46 pm

    On May 28, 2015, at 1:54 AM, Berker Peksa? wrote:

    * Performance improvements are not bug fixes

    Practicality beats purity here.
    Recognize that a huge number of Python users will remain in the Python2.7 world
    for some time. We have a responsibility to the bulk of our users (my estimate is
    that adoption rate for Python 3 is under 2%). The computed goto patch makes
    substantial performance improvements. It is callous to deny the improvement
    to 2.7 users.



    * The patch doesn't make the migration process from Python 2 to Python 3 easier

    Sorry, that is a red-herring (an orthogonal issue).
    If you care about 2-to-3 migration, then start
    opposing proposals for API changes that increase
    the semantic difference between 2 and 3.






    Raymond
  • Brett Cannon at May 28, 2015 at 2:02 pm
    On Thu, May 28, 2015 at 8:47 AM Raymond Hettinger wrote:

    On May 28, 2015, at 1:54 AM, Berker Peksa? wrote:

    * Performance improvements are not bug fixes
    Practicality beats purity here.

    Recognize that a huge number of Python users will remain in the Python2.7
    world
    for some time. We have a responsibility to the bulk of our users (my
    estimate is
    that adoption rate for Python 3 is under 2%).



    Where does that number come from? I have not seen numbers less than 5% for
    the overall community adoption (and all of them are extremely rough and all
    skewed towards Python 2 for various technical reasons).



    The computed goto patch makes
    substantial performance improvements. It is callous to deny the
    improvement
    to 2.7 users.

    But you could argue that "Special cases aren't special enough to break the
    rules" and that's what we are proposing here by claiming Python 2.7 is a
    special case and thus we should accept a patch that is not a one-liner
    change to gain some performance in a bugfix release.



    * The patch doesn't make the migration process from Python 2 to Python 3
    easier

    Sorry, that is a red-herring (an orthogonal issue).
    If you care about 2-to-3 migration, then start
    opposing proposals for API changes that increase
    the semantic difference between 2 and 3.

    That's misdirection for Berker's point that the proposal at hand does not
    help with getting to people to Python 3 even though what is proposed is an
    enhancement and not a bugfix (since we are not fixing a performance
    regression). I had someone on Twitter earlier this month point out that
    Python 3 was slower than Python 2 on some benchmark and that was a reason
    they weren't switching. Doing this is not going to help make that case
    (although I think arguing about performance between 2 and 3 is misleading
    when I've seen other workloads win out in Python 3).


    I'm -0 on this because I would like to stick to our policy of no
    enhancements in a bugfix release, but in the end it's Benjamin's call as
    2.7 RM as to whether this is appropriate for 2.7.11.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150528/0f432d9a/attachment.html>

Related Discussions

People

Translate

site design / logo © 2017 Grokbase