FAQ
I should start off by saying I don't plan to pursue this idea, but I wanted
to write it down for posterity and in case anyone else has thought about
this.


That being said, the idea of macros and other source-transforming things
done to Python code has come up a few times on python-ideas as of late. Now
experimenting with this sort of thing using a custom loader is not hard,
and thanks to importlib.abc.ResourceLoader.source_to_code()
<https://docs.python.org/3/library/importlib.html#importlib.abc.InspectLoader.source_to_code>
it's fairly easy to do (by design; I tried to initially structure
importlib's APIs to making alternative storage backends easy as well as
alternative syntax stuff like Quixote from back in the day).


But one thing I realized is that while finders and loaders are necessary
for alternative code storage mechanisms, they are not the right abstraction
for tweaking code semantics. Really all you need is a function that takes
in source code and spits out a code object to use with exec() (hence
ResourceLoader.source_to_code() even existing). It somewhat sucks that
people who just want to tweak code semantics have to define a loader
subclass and instantiate a new finder when all that is mostly stuff that
doesn't concern them. It also sucks that they would have to do that for
every storage type, e.g. local files and zip files.


Now I don't have a solid solution to propose for this niche use case. It
makes me want to have some kind of way to register compiler functions, but
that would be limiting if it went source -> code object. AST -> AST would
allow for chaining much like Victor has proposed in the past, but it also
means that people who want a transpiler to go source -> source are left
out. And then there is the whole thing of how to get the loaders to know of
these transpilers/transformers/compilers as adding more global state to sys
feels dirty (maybe an attribute on finders that they can draw from if they
so choose?), but maybe it isn't that big of a deal as long as they are just
callables and people realize they must be re-entrant.


As I said, I don't plan to work on this, but I wanted to get my ideas
written down in case someone else cared.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20150528/4b0161ff/attachment.html>

Search Discussions

  • Eric Snow at May 28, 2015 at 10:18 pm

    On Thu, May 28, 2015 at 9:11 AM, Brett Cannon wrote:
    I should start off by saying I don't plan to pursue this idea, but I wanted
    to write it down for posterity and in case anyone else has thought about
    this.

    That being said, the idea of macros and other source-transforming things
    done to Python code has come up a few times on python-ideas as of late. Now
    experimenting with this sort of thing using a custom loader is not hard, and
    thanks to importlib.abc.ResourceLoader.source_to_code() it's fairly easy to
    do (by design; I tried to initially structure importlib's APIs to making
    alternative storage backends easy as well as alternative syntax stuff like
    Quixote from back in the day).

    But one thing I realized is that while finders and loaders are necessary for
    alternative code storage mechanisms, they are not the right abstraction for
    tweaking code semantics.

    Agreed.

    Really all you need is a function that takes in
    source code and spits out a code object to use with exec() (hence
    ResourceLoader.source_to_code() even existing). It somewhat sucks that
    people who just want to tweak code semantics have to define a loader
    subclass and instantiate a new finder when all that is mostly stuff that
    doesn't concern them. It also sucks that they would have to do that for
    every storage type, e.g. local files and zip files.

    Yep.

    Now I don't have a solid solution to propose for this niche use case. It
    makes me want to have some kind of way to register compiler functions, but

    I had the same thought.

    that would be limiting if it went source -> code object. AST -> AST would
    allow for chaining much like Victor has proposed in the past, but it also
    means that people who want a transpiler to go source -> source are left out.

    Yeah, it feels like there's an encapsulation there around the various
    pieces of compilation. Furthermore, I'd expect such an abstraction to
    consider the needs of alternate Python implementations as well.

    And then there is the whole thing of how to get the loaders to know of these
    transpilers/transformers/compilers
    as adding more global state to sys feels
    dirty (maybe an attribute on finders that they can draw from if they so
    choose?), but maybe it isn't that big of a deal as long as they are just
    callables and people realize they must be re-entrant.

    This is where something like ImportSystem (nee ImportEngine) would
    help. We'd just have sys.importsystem and add state there as
    appropriate without further cluttering up the sys module.


    FWIW, I've considered a number of minor additions similar to what
    you're talking about for niche needs where it would still be nice to
    have a convenient API because of the overhead of writing and managing
    a finder/loader.


    Perhaps it's just a matter of providing helper decorators along the
    lines of contextlib.contextmanager, which convert your simple function
    into the necessary format at register it in the correct place (e.g.
    finder+loader -> sys.path/sys.metapath).

    As I said, I don't plan to work on this, but I wanted to get my ideas
    written down in case someone else cared.

    :)


    -eric

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupimport-sig @
categoriespython
postedMay 28, '15 at 3:11p
activeMay 28, '15 at 10:18p
posts2
users2
websitepython.org

2 users in discussion

Brett Cannon: 1 post Eric Snow: 1 post

People

Translate

site design / logo © 2018 Grokbase