FAQ
One nice feature of Java (sorry!) is the ability to ship an application
as a single JAR file - containing all code, resources, etc, etc. Is
there any equivalent feature available in Python? The obvious place
where it would be is in Gordon McMillan's Installer package, with its
Python Archive format, but I don't think (from looking at the docs) that
does quite what I'm after.

I have no requirement to include a Python interpreter, or C extensions,
in the archive. The code is all pure Python. But, I do want to be able
to include support files (text files, image files for use in wxPython
toolbars, graphics for display on screen, sound files, etc etc) which
can be picked up from the same archive.

I'm looking for something which works a bit like Java's
ClassLoader.getResourceAsStream() (along with the necessary stuff to
allow the stream to then be *used* as an icon/image/whatever).

It seems like the sort of thing which should be possible with Python,
but I can't find it. Did I miss something?

Thanks in advance,
Paul.

Search Discussions

  • Robin Becker at Feb 14, 2002 at 5:56 pm
    In article <2upn6uoqotqejiji69bg8kebfacdu9oamb at 4ax.com>, Paul Moore <paul.moore at atosorigin.com>
    writes
    One nice feature of Java (sorry!) is the ability to ship an application
    as a single JAR file - containing all code, resources, etc, etc. Is
    there any equivalent feature available in Python? The obvious place
    where it would be is in Gordon McMillan's Installer package, with its
    Python Archive format, but I don't think (from looking at the docs) that
    does quite what I'm after.

    I have no requirement to include a Python interpreter, or C extensions,
    in the archive. The code is all pure Python. But, I do want to be able
    to include support files (text files, image files for use in wxPython
    toolbars, graphics for display on screen, sound files, etc etc) which
    can be picked up from the same archive.

    I'm looking for something which works a bit like Java's
    ClassLoader.getResourceAsStream() (along with the necessary stuff to
    allow the stream to then be *used* as an icon/image/whatever).

    It seems like the sort of thing which should be possible with Python,
    but I can't find it. Did I miss something?

    Thanks in advance,
    Paul.
    well his .pyz is a compressed archive form and I have been using it to compress the entire
    python library. All that's required is a special importer to allow imports from such an archive
    to work. I believe that the archive is allowed to contain data files as well so provided you
    alter the open/read you can at least consider doing that.

    The startup code I use is below. Assuming that zlib is available means
    that the modules archive_rt & imputils don't need to be specially obtained
    with a linked version of libz.

    In our demo we do this just after initialising
    python and before importing site. If a full python is already available
    you'd only need to do this in a stub python file to get access to all of your
    modules etc.

    ###################################################
    def _rl_readPYZ(fn):
    import zlib, imp, marshal, sys, struct
    MAGIC = 'PYZ\\0'
    TOCPOS = 8
    pymagic = imp.get_magic()
    lib = open(fn,'rb')
    if lib.read(len(MAGIC)) != MAGIC:
    raise RuntimeError, fn+' is not a valid ZArchive file'
    if lib.read(len(pymagic)) != pymagic:
    raise RuntimeError, fn+' has version mismatch to dll'
    lib.seek(TOCPOS)
    (offset,) = struct.unpack('=i', lib.read(4))
    lib.seek(offset)
    toc = marshal.load(lib)
    for name in ('imputil','archive_rt'):
    (ispkg, pos, lngth) = toc.get(name, (0, None, 0))
    if pos is None: raise ValueError, 'Module '+name+' not found'
    lib.seek(pos)
    code = marshal.loads(zlib.decompress(lib.read(lngth)))
    if ispkg: values['__path__'] = [fqname]
    is_module = isinstance(code, type(sys))
    module = is_module and code or imp.new_module(name)
    module.__dict__.update({'__file__':code.co_filename,'__iname__':name})
    sys.modules[name] = module
    if not is_module: exec code in module.__dict__
    module = sys.modules[name]
    module.__name__ = name
    import archive_rt
    sys.importers.append(archive_rt.ZlibArchive(fn, 0))
    _rl_readPYZ(r'reportlab.pyz')
    del _rl_readPYZ
    --
    Robin Becker
  • Paul Moore at Feb 14, 2002 at 9:34 pm

    On Thu, 14 Feb 2002 17:56:11 +0000, Robin Becker wrote:

    well his .pyz is a compressed archive form and I have been using it to compress the entire
    python library. All that's required is a special importer to allow imports from such an archive
    to work. I believe that the archive is allowed to contain data files as well so provided you
    alter the open/read you can at least consider doing that.
    Hmm. Looks more messy than I'd hoped. Needs a bootstrap module, and then
    extra hacking to handle open/read. And I guess the fact that I'm doing
    this manually means that there's not much chance of something like a
    wxPython toolbar "just working" with icons stored in the archive.

    I guess it's not worth the hassle.

    The reason I liked the Java approach, BTW, is that a complete
    application comes in a single file, which you can just double click on
    and it works. Nothing to unpack, put it anywhere you like, run off a
    floppy/CD/zip disk, no dependencies (other than needing Java installed).
    But it's just a convenience thing - not enough to endure writing all my
    code in Java :-)

    Nice to keep an eye on what the competition can do, though...

    Thanks for the explanation,
    Paul.
  • Chris Liechti at Feb 14, 2002 at 10:34 pm
    Paul Moore <gustav at morpheus.demon.co.uk> wrote in
    news:efao6us1tt3ut2hp6aor19p7qfqad3k4t1 at 4ax.com:
    ...
    Hmm. Looks more messy than I'd hoped. Needs a bootstrap module, and
    then extra hacking to handle open/read. And I guess the fact that
    I'm doing this manually means that there's not much chance of
    something like a wxPython toolbar "just working" with icons stored
    in the archive.
    well you could pack the stuff in a zipfile and retreive the data with
    the zipfile module. you get the data as string but you can wrap that
    again in a cStringIO object to get a file-like object.

    but then wxWindows Bitmap.LoadFile wants real file...
    but: """wxBitmapFromXPMData(listOfStrings) Create a bitmap from a
    Python list of strings whose contents are XPM data""" so you would be
    limited to XPM images.

    py2exe uses a zip as storage and adds a loader in front of it a la
    self-extract zip. those files can be opened with e.g with WinZip maybe
    you could add files this way too (never tried myself)?

    as described above getting data from a file-like object to an
    extestion in c isn't that painless. of course you could allways create
    a temp file (os.tmpfile()), copy the stuff from the zip to that file
    and use it (Bitmap.LoadFile(temp)) etc.

    I guess it's not worth the hassle.
    the following could be worth a try:
    -make a py2exe dist
    -add data files

    the running programm does:
    -extract data to a temp file for extensions that need a file
    (i.e. for Bitmap.LoadFile())
    -store thedata in a cStringIO object for pure python stuff
    (config files, readme.txt for an help dialog etc.)
    The reason I liked the Java approach, BTW, is that a complete
    application comes in a single file, which you can just double click
    on and it works. Nothing to unpack, put it anywhere you like, run
    off a floppy/CD/zip disk, no dependencies (other than needing Java
    installed). But it's just a convenience thing - not enough to endure
    writing all my code in Java :-)
    yes its easy when you have a jar but i don't understand why "java"
    just doesn't cut off ".class" so that you could launch those files by
    doubleklick too....
    Nice to keep an eye on what the competition can do, though...
    <sarcasm>
    yeah, they finaly made it to include it a select like interface for
    e.g. sockets "to write high performace/scalable servers" and regex'es
    are now included in the 1.4 release thats out these days, wow. again
    ten years and they *might* catch up to python...
    </sarcasm> ;-)

    chris

    --
    Chris <cliechti at gmx.net>
  • Paul Moore at Feb 15, 2002 at 10:05 pm

    On 14 Feb 2002 23:34:12 +0100, Chris Liechti wrote:
    Paul Moore <gustav at morpheus.demon.co.uk> wrote in
    news:efao6us1tt3ut2hp6aor19p7qfqad3k4t1 at 4ax.com:
    ...
    Hmm. Looks more messy than I'd hoped. Needs a bootstrap module, and
    then extra hacking to handle open/read. And I guess the fact that
    I'm doing this manually means that there's not much chance of
    something like a wxPython toolbar "just working" with icons stored
    in the archive.
    well you could pack the stuff in a zipfile and retreive the data with
    the zipfile module. you get the data as string but you can wrap that
    again in a cStringIO object to get a file-like object.
    I looked a bit closer at this. Interesting. Using the zipfile module
    makes it trivial to get the raw data from an entry in a zipfile. And as
    you say, I can use cStringIO to get a Python file object from this.

    For text files, that's all I need really.

    For sound files (on Windows) I can use the winsound module, with the
    SND_MEMORY option to play from a memory image.
    but then wxWindows Bitmap.LoadFile wants real file...
    but: """wxBitmapFromXPMData(listOfStrings) Create a bitmap from a
    Python list of strings whose contents are XPM data""" so you would be
    limited to XPM images.
    Looks like it's wxPython that has the problem...

    Annoyingly, wxWindows has a wxImage constructor, wxImage(wxInputStream)
    to load from a stream, but wxPython *doesn't* seem to wrap this
    constructor. OK, so there would need to be a conversion from a Python
    file object to a wxInputStream (or something).

    But some way of getting a wxImage from in-memory data (of arbitrary
    format) is really all I'm after - and there ought to be a way to get it.
    I've copied this to the wxPython list. Is there some way I've missed -
    or could a way be added?

    Thanks,
    Paul Moore
  • Mike C. Fletcher at Feb 15, 2002 at 10:27 pm
    wxImages can be loaded from strings. See the wxPython Wiki entry on
    working with images (particularly the sections on PIL and Numeric):

    http://wiki.wxpython.org/index.cgi/WorkingWithImages

    You can probably use PIL to get GIF/JPEG/Whatever formatted data into
    the RAW format for use by wxImage

    HTH,
    Mike

    Paul Moore wrote:
    On 14 Feb 2002 23:34:12 +0100, Chris Liechti wrote:
    [comments on loading various file types from zip files]
    ...
    but then wxWindows Bitmap.LoadFile wants real file...
    but: """wxBitmapFromXPMData(listOfStrings) Create a bitmap from a
    Python list of strings whose contents are XPM data""" so you would be
    limited to XPM images.
    Looks like it's wxPython that has the problem...

    Annoyingly, wxWindows has a wxImage constructor, wxImage(wxInputStream)
    to load from a stream, but wxPython *doesn't* seem to wrap this
    constructor. OK, so there would need to be a conversion from a Python
    file object to a wxInputStream (or something).

    But some way of getting a wxImage from in-memory data (of arbitrary
    format) is really all I'm after - and there ought to be a way to get it.
    I've copied this to the wxPython list. Is there some way I've missed -
    or could a way be added?

    Thanks,
    Paul Moore
    _______________________________________
    Mike C. Fletcher
    http://members.rogers.com/mcfletch/
  • Paul Moore at Feb 16, 2002 at 12:52 pm

    On Fri, 15 Feb 2002 17:27:47 -0500, "Mike C. Fletcher" wrote:

    wxImages can be loaded from strings. See the wxPython Wiki entry on
    working with images (particularly the sections on PIL and Numeric):

    http://wiki.wxpython.org/index.cgi/WorkingWithImages

    You can probably use PIL to get GIF/JPEG/Whatever formatted data into
    the RAW format for use by wxImage
    and Chris Liechti wrote:
    ok just tried this: [...]
    you can use wxPython to convert the images to strings, save them to a
    zip using the zipfile module. the image sizes have to be saved
    separately. i've choosen to stuff them into the filename.
    Thanks for both of these pointers, guys. Looks like just what I need.

    I'll go and work through all the suggestions I have and come up with
    something suitable. I'll report back on what I manage to produce.

    Paul.
  • Paul Moore at Feb 20, 2002 at 9:59 pm

    On Sat, 16 Feb 2002 12:52:55 +0000, Paul Moore wrote:

    I'll go and work through all the suggestions I have and come up with
    something suitable. I'll report back on what I manage to produce.
    I've had a play with the various options, and mulled the whole thing
    over a bit. The conclusion I've come to is that it's certainly possible
    to do the sort of thing I'm after, but it's probably too messy (at the
    moment) to make it worthwhile.

    One thing I discovered is PEP 273, which proposes to allow zipfiles on
    sys.path. This is proposed for Python 2.3, and would make zipping up
    .py[c] files a much more reasonable option - until this gets
    implemented, the necessary import hacks to load modules from a zip file
    are possible, but messy. I'll wait for the official implementation...

    Getting "resources" out of a zipfile is relatively simple. The zipfile
    module lets you get at the data, and StringIO lets you wrap it as a
    file-like object. There are still a few issues, though...

    1. The data is read in "binary" (with \r\n line terminators on Windows).
    StringIO wraps this "raw", which could be a problem if you are getting a
    textfile resource, and need to read in text mode. The simplest answer is
    to do a data = data.replace("\r","") before wrapping the data.

    2. All the data gets read into memory, which could be an issue for big
    chunks of data. Translating text files as in (1) above creates a second
    copy - more memory used.

    Looking at specific types of data

    Text data - No real problem. Read as a string, or wrap as a StringIO.

    Sound - Not a problem on Windows, as the winsound module takes in-memory
    data as an option.

    Images (PIL) - Not a problem, as PIL constructors can take a file-like
    object as well as a filename.

    Images (wxPython) - A bit messy, the simplest way is to go via PIL.
    (This leads to yet more in-memory copying, so it may be a problem for
    big images...) The next version of wxPython is likely to contain
    constructors which take file-like objects, so things will be simpler
    then...

    One obvious point is that you have to code specifically for reading from
    a zipfile rather than from the filesystem. This is a nuisance, unless
    you write a wrapper class which lets you choose filesystem vs zipfile at
    runtime.

    Fredrik Lundh's Squeeze package lets you pack up Python code and data
    files in a single archive, and read datafiles from the archive at
    runtime. It's almost exactly what I'm after. It uses an internal data
    format rather than standard zipfiles - a reasonable decision, but not
    quite what I was aiming for.

    The basic feeling I get is that there are loads of ways of doing this
    sort of thing. None of them are too hard, but on the other hand, there
    isn't one "off the shelf" standard way of doing this, so it would be
    necessary to code what you wanted into your application (or more
    sensibly, as a reusable component which you use in your application). In
    most cases, I suspect it's not worth the bother for any specific
    application.

    I could see some value in a PEP to propose a standard way of doing this
    sort of thing - probably as a library module, maybe with some sort of
    core support. But I don't know that anyone's likely to feel strongly
    enough to push it. (I don't think I am...)

    Thanks for all the suggestions and comments, it was an interesting
    exercise...

    Paul.
  • Thomas Heller at Feb 21, 2002 at 10:08 am
    Another interesting technology are Jean-Claude Wippler's
    'Scripted Documents'. The main link seems to be

    http://www.equi4.com/scripdoc/

    but you should also google for others.

    Thomas
  • Paul Moore at Feb 22, 2002 at 9:50 pm

    On Thu, 21 Feb 2002 11:08:39 +0100, "Thomas Heller" wrote:
    Another interesting technology are Jean-Claude Wippler's
    'Scripted Documents'. The main link seems to be

    http://www.equi4.com/scripdoc/

    but you should also google for others.
    That's very interesting. Thanks for the pointer.

    Paul.
  • Jacob Smullyan at Feb 22, 2002 at 11:49 pm
    Along similar lines, I'm currently developing a product archive format
    for SkunkWeb which uses Gordon McMillan's iu.py. I first attempted to
    use imputil.py and ran into serious problems -- iu.py really saved the
    day. I recommend it; either it should become part of the core, or
    PythonLabs should bite the bullet and rewrite the import mechanism
    with a much higher degree of generality, so that weird import hooks
    won't be necessary any longer.


    --
    Jacob Smullyan | smulloni at smullyan.org
    http://www.smullyan.org/smulloni/
  • Chris Liechti at Feb 16, 2002 at 12:46 am
    "Mike C. Fletcher" <mcfletch at geocities.com> wrote in
    news:mailman.1013812227.6703.python-list at python.org:
    wxImages can be loaded from strings. See the wxPython Wiki entry on
    working with images (particularly the sections on PIL and Numeric):
    ok just tried this:
    ------------
    import zipfile
    from wxPython.wx import *
    wxInitAllImageHandlers() #init wx stuff
    #convert file
    bmp = wxImage('img.png', wxBITMAP_TYPE_PNG)
    raw = bmp.GetData()
    zipf = zipfile.ZipFile('hello.zip','w')
    info = zipfile.ZipInfo('hello.%d.%d' % (bmp.GetWidth(), bmp.GetHeight
    ()))
    info.compress_type=zipfile.ZIP_DEFLATED
    zipf.writestr(info, raw)
    del zipf

    #use it
    zipf = zipfile.ZipFile('hello.zip','r')
    entry = zipf.namelist()[0]
    name,w,h = entry.split('.')
    raw = zipf.read(entry)
    img = wxEmptyImage(int(w), int(h))
    img.SetData(raw)
    #img = img.Rotate(45, (10,10)) #just for fun
    img.SaveFile("hello.png", wxBITMAP_TYPE_PNG)
    --------

    you can use wxPython to convert the images to strings, save them to a
    zip using the zipfile module. the image sizes have to be saved
    separately. i've choosen to stuff them into the filename.

    storing uncompressed images in a zip isn't that bad, as e.g png or gif
    also uses some similar compression algos. (in my test the zip was
    smaller than the original png)

    --
    Chris <cliechti at gmx.net>
  • Jeff Shannon at Feb 14, 2002 at 7:58 pm

    Paul Moore wrote:

    I have no requirement to include a Python interpreter, or C extensions,
    in the archive. The code is all pure Python. But, I do want to be able
    to include support files (text files, image files for use in wxPython
    toolbars, graphics for display on screen, sound files, etc etc) which
    can be picked up from the same archive.
    If you require this being in a single archive file only for transfer
    purposes, you can use distutils (standard in 2.1+, not sure about 2.0).
    Distutils will create a single zipfile/.tgz/rpm that can be distributed
    easily. To install the package, it's only necessary to open the archive and
    type 'python setup.py install'. (It may also be able to create a
    single-file self-installing .exe on Windows.) Note that once you install
    the package, it is no longer a single archive file, but is fully extracted
    to its individual component files, so this may or may not meet your needs.

    Jeff Shannon
    Technician/Programmer
    Credit International
  • Delaney, Timothy at Feb 14, 2002 at 11:39 pm

    Hmm. Looks more messy than I'd hoped. Needs a bootstrap
    module, and then
    extra hacking to handle open/read. And I guess the fact that I'm doing
    this manually means that there's not much chance of something like a
    wxPython toolbar "just working" with icons stored in the archive.

    I guess it's not worth the hassle.

    The reason I liked the Java approach, BTW, is that a complete
    application comes in a single file, which you can just double click on
    and it works. Nothing to unpack, put it anywhere you like, run off a
    floppy/CD/zip disk, no dependencies (other than needing Java
    installed).
    But it's just a convenience thing - not enough to endure
    writing all my
    code in Java :-)
    http://www.jython.org/

    Python implemented in Java. Has a standard system (jythonc) for compiling to
    java classes and packaging them up as a jar file which does everything you
    want.

    In most cases you don't need to produce any java code at all - just code in
    Python. It makes it *really* easy to use Swing too - much easier than in
    Java.

    Tim Delaney
  • Thomas Heller at Feb 15, 2002 at 5:03 pm
    "Paul Moore" <paul.moore at atosorigin.com> wrote in message news:2upn6uoqotqejiji69bg8kebfacdu9oamb at 4ax.com...
    One nice feature of Java (sorry!) is the ability to ship an application
    as a single JAR file - containing all code, resources, etc, etc. Is
    there any equivalent feature available in Python? The obvious place
    where it would be is in Gordon McMillan's Installer package, with its
    Python Archive format, but I don't think (from looking at the docs) that
    does quite what I'm after.

    I have no requirement to include a Python interpreter, or C extensions,
    in the archive. The code is all pure Python. But, I do want to be able
    to include support files (text files, image files for use in wxPython
    toolbars, graphics for display on screen, sound files, etc etc) which
    can be picked up from the same archive.

    I'm looking for something which works a bit like Java's
    ClassLoader.getResourceAsStream() (along with the necessary stuff to
    allow the stream to then be *used* as an icon/image/whatever).

    It seems like the sort of thing which should be possible with Python,
    but I can't find it. Did I miss something?
    Have you looked into /F's squeeze? Should exactly do what you want,
    from the description...

    Thomas
  • Paul Moore at Feb 16, 2002 at 12:44 pm

    On Fri, 15 Feb 2002 18:03:13 +0100, "Thomas Heller" wrote:
    Have you looked into /F's squeeze? Should exactly do what you want,
    from the description...
    Looks pretty close... I'll have to look into it a little. Thanks for the
    pointer.

    Paul.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-list @
categoriespython
postedFeb 14, '02 at 4:56p
activeFeb 22, '02 at 11:49p
posts16
users9
websitepython.org

People

Translate

site design / logo © 2022 Grokbase