FAQ

[Cython] PEP 3135 -- New Super

Vitja Makarov
Jul 5, 2011 at 6:21 am
I was thinking about implementing new super() with no arguments.

The problem is where to store __class__, I see two options here:

1. Add func_class member to CyFunction, this way __class__ will be
private and not visible for inner functions:
2. Put it into closure

Actually, I like the first one.
And I don't think that __class__ should be use somewhere outside super()

--
vitja.
reply

Search Discussions

30 responses

  • Stefan Behnel at Jul 5, 2011 at 7:00 am

    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    Please do :)

    If you start working on it, please assign the ticket to you:

    http://trac.cython.org/cython_trac/ticket/696

    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super() is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this may
    make a difference.

    OTOH, not all methods have a closure, so creating one just to store the
    "__class__" field is very wasteful, in terms of both runtime and memory
    overhead. A lot more wasteful than paying 8 bytes of memory for each
    function, with no additional time overhead.

    And I don't think that __class__ should be use somewhere outside super()
    Agreed. CPython simply uses a compile time heuristic ("is there a function
    call to something global named 'super'?") when creating this field, so it's
    strictly reserved for this use case.

    BTW, I like the irony in the fact that CPython essentially gives Cython
    semantics to the "super" builtin here, by (partially) evaluating it at
    compile time.

    Stefan
  • Vitja Makarov at Jul 5, 2011 at 7:17 am

    2011/7/5 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    Please do :)

    If you start working on it, please assign the ticket to you:

    http://trac.cython.org/cython_trac/ticket/696
    Ok, I'll do this if I start.
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    OTOH, not all methods have a closure, so creating one just to store the
    "__class__" field is very wasteful, in terms of both runtime and memory
    overhead. A lot more wasteful than paying 8 bytes of memory for each
    function, with no additional time overhead.
    Going this way it only requires to initialize closure:

    closure.func_class = class


    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    This also could help to solve default args problem.

    And I don't think that __class__ should be use somewhere outside super()
    Agreed. CPython simply uses a compile time heuristic ("is there a function
    call to something global named 'super'?") when creating this field, so it's
    strictly reserved for this use case.

    BTW, I like the irony in the fact that CPython essentially gives Cython
    semantics to the "super" builtin here, by (partially) evaluating it at
    compile time.
    Yeah, I think Cython super with no args should be a little bit faster
    then classic one.

    --
    vitja.
  • Stefan Behnel at Jul 5, 2011 at 8:04 am

    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class
    Remember that no-args super() can only be used in functions that are
    literally written inside of a class body, so we actually know at compile
    time which functions need this field. We can thus do better than a generic
    loop over all fields. We even have the function object pointers directly
    available in the module init function where we create the class body.

    BTW, we also need a way to make this work for cdef classes. No idea how
    different that would be.

    OTOH, not all methods have a closure, so creating one just to store the
    "__class__" field is very wasteful, in terms of both runtime and memory
    overhead. A lot more wasteful than paying 8 bytes of memory for each
    function, with no additional time overhead.
    Going this way it only requires to initialize closure:
    Yes, and that's costly.

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?

    This also could help to solve default args problem.
    And potentially other problems, too. Think of heap allocated modules, for
    example.

    http://trac.cython.org/cython_trac/ticket/173
    http://trac.cython.org/cython_trac/ticket/218

    Seeing this, I'm all for using a field in CyFunction.

    And I don't think that __class__ should be use somewhere outside super()
    Agreed. CPython simply uses a compile time heuristic ("is there a function
    call to something global named 'super'?") when creating this field, so it's
    strictly reserved for this use case.

    BTW, I like the irony in the fact that CPython essentially gives Cython
    semantics to the "super" builtin here, by (partially) evaluating it at
    compile time.
    Yeah, I think Cython super with no args should be a little bit faster
    then classic one.
    I think speed isn't really all that important here. Calling Python methods
    is costly enough anyway.

    IMO, the main reason for the heuristic is to prevent class objects from
    being kept alive by their methods, except for the single case where super()
    is used. Keeping a class alive just because one of its methods is still
    used somewhere can be very costly, depending on the content of the class
    dict. It also creates a reference cycle, which is another costly thing in
    CPython as it requires a GC run over the whole class dict to get cleaned up.

    The situation for modules is at least slightly different, as modules do not
    tend to get unloaded, so there's always a reference to them from
    sys.modules. If that ever gets removed, it's really ok if it takes time to
    clean things up.

    Stefan
  • Vitja Makarov at Jul 5, 2011 at 8:37 am

    2011/7/5 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    ? ?if isinstance(value, CyFunction) and value.func_class is WantClass:
    ? ? ? ?value.func_class = class
    Remember that no-args super() can only be used in functions that are
    literally written inside of a class body, so we actually know at compile
    time which functions need this field. We can thus do better than a generic
    loop over all fields. We even have the function object pointers directly
    available in the module init function where we create the class body.
    Yes, but now cyfunction references are lost as they were in temps.
    BTW, we also need a way to make this work for cdef classes. No idea how
    different that would be.
    I think super() for cdef classes should be much easy to implement as
    we already know cname for a class at compile time. That would be
    simple transform.
    OTOH, not all methods have a closure, so creating one just to store the
    "__class__" field is very wasteful, in terms of both runtime and memory
    overhead. A lot more wasteful than paying 8 bytes of memory for each
    function, with no additional time overhead.
    Going this way it only requires to initialize closure:
    Yes, and that's costly.

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    Right. When we pass CyFunction closure should be moved into its members.
    This also could help to solve default args problem.
    And potentially other problems, too. Think of heap allocated modules, for
    example.

    http://trac.cython.org/cython_trac/ticket/173
    http://trac.cython.org/cython_trac/ticket/218

    Seeing this, I'm all for using a field in CyFunction.

    Yeah, I think Cython super with no args should be a little bit faster
    then classic one.
    I think speed isn't really all that important here. Calling Python methods
    is costly enough anyway.

    IMO, the main reason for the heuristic is to prevent class objects from
    being kept alive by their methods, except for the single case where super()
    is used. Keeping a class alive just because one of its methods is still used
    somewhere can be very costly, depending on the content of the class dict. It
    also creates a reference cycle, which is another costly thing in CPython as
    it requires a GC run over the whole class dict to get cleaned up.

    The situation for modules is at least slightly different, as modules do not
    tend to get unloaded, so there's always a reference to them from
    sys.modules. If that ever gets removed, it's really ok if it takes time to
    clean things up.
    Yes, but I hope that classes are rarely deleted.
    And I'm afraid that we can't avoid cycles when implementing __class__
    for pure-python classes.

    --
    vitja.
  • Stefan Behnel at Jul 5, 2011 at 8:49 am

    Vitja Makarov, 05.07.2011 10:37:
    2011/7/5 Stefan Behnel:
    IMO, the main reason for the heuristic is to prevent class objects from
    being kept alive by their methods, except for the single case where super()
    is used. Keeping a class alive just because one of its methods is still used
    somewhere can be very costly, depending on the content of the class dict. It
    also creates a reference cycle, which is another costly thing in CPython as
    it requires a GC run over the whole class dict to get cleaned up.

    The situation for modules is at least slightly different, as modules do not
    tend to get unloaded, so there's always a reference to them from
    sys.modules. If that ever gets removed, it's really ok if it takes time to
    clean things up.
    Yes, but I hope that classes are rarely deleted.
    And I'm afraid that we can't avoid cycles when implementing __class__
    for pure-python classes.
    As I said, it's fine if super() is used, but not otherwise. Basically,
    no-args super() is just a shortcut for "super(TheClass, self)", with the
    difference that the long form looks up the name at runtime, whereas the
    short form keeps a reference to the type at module setup time. So, both
    need the class at runtime, but have different characteristics otherwise.

    (but if the code is not using super(), there is no need to have the methods
    own a reference to their class)

    Stefan
  • Stefan Behnel at Jul 6, 2011 at 12:36 am

    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows the
    class it was defined in, which doesn't help here.

    Stefan
  • Vitja Makarov at Jul 6, 2011 at 7:05 am

    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__, <firstarg>)"
    So we only have func_class (that is inside cyfunction object) and
    first function argument of method to super()

    I tried to implement super() for cdef classes and it worked. It works
    very simple:

    cdef class Foo:
    def meth(self):
    super().meth() # is transformed into super(Foo, self)

    --
    vitja.
  • Stefan Behnel at Jul 6, 2011 at 7:22 am

    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass 'self' when
    we pass the CyFunction object as the first argument?

    Stefan
  • Vitja Makarov at Jul 6, 2011 at 8:01 am

    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass 'self' when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.

    --
    vitja.
  • Vitja Makarov at Jul 7, 2011 at 3:09 pm

    2011/7/6 Vitja Makarov <vitja.makarov at gmail.com>:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass 'self' when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    I've add support for new super() for cdef classes here:
    https://github.com/vitek/cython/compare/_new_super

    Pure python classes would require my _bindings branch.


    --
    vitja.
  • Mark florisson at Jul 7, 2011 at 7:47 pm

    On 7 July 2011 17:09, Vitja Makarov wrote:
    2011/7/6 Vitja Makarov <vitja.makarov at gmail.com>:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass 'self' when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    I've add support for new super() for cdef classes here:
    https://github.com/vitek/cython/compare/_new_super

    Pure python classes would require my _bindings branch.
    I also need that branch to continue my work on fused types. The naming
    suggests I can't pull from it and start building on top of it? Did you
    have any plans to review it Stefan? Otherwise I won't mind giving it a
    go.
    --
    vitja.
    _______________________________________________
    cython-devel mailing list
    cython-devel at python.org
    http://mail.python.org/mailman/listinfo/cython-devel
  • Mark florisson at Jul 7, 2011 at 7:44 pm

    On 6 July 2011 10:01, Vitja Makarov wrote:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass 'self' when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    Would it be easier to make scope objects attributes of functions? Then
    you could still utilize PyCFunction_Call without needing to check the
    argument flags and such right?
    --
    vitja.
    _______________________________________________
    cython-devel mailing list
    cython-devel at python.org
    http://mail.python.org/mailman/listinfo/cython-devel
  • Vitja Makarov at Jul 7, 2011 at 8:15 pm
    2011/7/7 mark florisson <markflorisson88 at gmail.com>
    On 6 July 2011 10:01, Vitja Makarov wrote:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only
    when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions
    than
    specifically methods that use a no-argument call to super(), so
    this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows
    the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass 'self'
    when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    Would it be easier to make scope objects attributes of functions? Then
    you could still utilize PyCFunction_Call without needing to check the
    argument flags and such right?
    Sure, scope object is already functions attribute now it's stored inside
    self.
    But, instancemethods desc_get builds new args tuple with self as first
    element.
    We can implement cython version of instance method and change method
    signature a little bit.



    --
    vitja.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110708/96bc6ac2/attachment.html>
  • Mark florisson at Jul 7, 2011 at 8:30 pm

    On 7 July 2011 22:15, Vitja Makarov wrote:

    2011/7/7 mark florisson <markflorisson88 at gmail.com>
    On 6 July 2011 10:01, Vitja Makarov wrote:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options here:

    1. Add func_class member to CyFunction, this way __class__ will
    be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only
    when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions
    than
    specifically methods that use a no-argument call to super(), so
    this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would
    accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self'
    object
    there, which the CyFunction doesn't know. If anything, it only knows
    the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass 'self'
    when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    Would it be easier to make scope objects attributes of functions? Then
    you could still utilize PyCFunction_Call without needing to check the
    argument flags and such right?
    Sure, scope object is already functions attribute now it's stored inside
    self.
    But, instancemethods desc_get builds new args tuple with self as first
    element.
    We can implement cython version of instance method and change method
    signature a little bit.
    Right, and you pass it in as the first argument to the C function.
    However, if you want to change the signature, you have to override
    PyCFunction_Call, which, if I'm not mistaken, means you will have to
    interpret the flags from the PyMethodDef and call the C function in
    the right way (e.g. with or without the args tuple and kwargs) and
    check if it's being called correctly. If you leave it as an attribute
    of the function, you can pass in the function and access the scope
    object from the function object in the closure C function. So I think
    changing the signature might be a bit harder?
    --
    vitja.


    _______________________________________________
    cython-devel mailing list
    cython-devel at python.org
    http://mail.python.org/mailman/listinfo/cython-devel
  • Vitja Makarov at Jul 7, 2011 at 8:39 pm
    2011/7/8 mark florisson <markflorisson88 at gmail.com>
    On 7 July 2011 22:15, Vitja Makarov wrote:


    2011/7/7 mark florisson <markflorisson88 at gmail.com>
    On 6 July 2011 10:01, Vitja Makarov wrote:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no
    arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options
    here:
    1. Add func_class member to CyFunction, this way __class__ will
    be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only
    when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions
    than
    specifically methods that use a no-argument call to super(), so
    this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is
    WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would
    accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self'
    object
    there, which the CyFunction doesn't know. If anything, it only
    knows
    the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass
    'self'
    when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    Would it be easier to make scope objects attributes of functions? Then
    you could still utilize PyCFunction_Call without needing to check the
    argument flags and such right?
    Sure, scope object is already functions attribute now it's stored inside
    self.
    But, instancemethods desc_get builds new args tuple with self as first
    element.
    We can implement cython version of instance method and change method
    signature a little bit.
    Right, and you pass it in as the first argument to the C function.
    However, if you want to change the signature, you have to override
    PyCFunction_Call, which, if I'm not mistaken, means you will have to
    interpret the flags from the PyMethodDef and call the C function in
    the right way (e.g. with or without the args tuple and kwargs) and
    check if it's being called correctly. If you leave it as an attribute
    of the function, you can pass in the function and access the scope
    object from the function object in the closure C function. So I think
    changing the signature might be a bit harder?
    Yes, but how to handle instantmethods self arg?
    We can't store it inside function object as it is different for each
    instance.

    btw I've already moved pycyfunction_call into cython..

    --
    vitja.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110708/a88aa5b0/attachment.html>
  • Mark florisson at Jul 7, 2011 at 8:50 pm

    On 7 July 2011 22:39, Vitja Makarov wrote:

    2011/7/8 mark florisson <markflorisson88 at gmail.com>
    On 7 July 2011 22:15, Vitja Makarov wrote:


    2011/7/7 mark florisson <markflorisson88 at gmail.com>
    On 6 July 2011 10:01, Vitja Makarov wrote:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no
    arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options
    here:

    1. Add func_class member to CyFunction, this way __class__
    will
    be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only
    when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions
    than
    specifically methods that use a no-argument call to super(), so
    this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is
    WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would
    accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self'
    object
    there, which the CyFunction doesn't know. If anything, it only
    knows
    the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass
    'self'
    when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call
    we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    Would it be easier to make scope objects attributes of functions? Then
    you could still utilize PyCFunction_Call without needing to check the
    argument flags and such right?
    Sure, scope object is already functions attribute now it's stored inside
    self.
    But, instancemethods desc_get builds new args tuple with self as first
    element.
    We can implement cython version of instance method and change method
    signature a little bit.
    Right, and you pass it in as the first argument to the C function.
    However, if you want to change the signature, you have to override
    PyCFunction_Call, which, if I'm not mistaken, means you will have to
    interpret the flags from the PyMethodDef and call the C function in
    the right way (e.g. with or without the args tuple and kwargs) and
    check if it's being called correctly. If you leave it as an attribute
    of the function, you can pass in the function and access the scope
    object from the function object in the closure C function. So I think
    changing the signature might be a bit harder?
    Yes, but how to handle instantmethods self arg?
    We can't store it inside function object as it is different for each
    instance.
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Anyway, the important question was whether I could pull from your
    branch or whether I should wait for the merge.
    btw I've already moved pycyfunction_call into cython..
    --
    vitja.


    _______________________________________________
    cython-devel mailing list
    cython-devel at python.org
    http://mail.python.org/mailman/listinfo/cython-devel
  • Vitja Makarov at Jul 7, 2011 at 9:13 pm
    2011/7/8 mark florisson <markflorisson88 at gmail.com>
    On 7 July 2011 22:39, Vitja Makarov wrote:


    2011/7/8 mark florisson <markflorisson88 at gmail.com>
    On 7 July 2011 22:15, Vitja Makarov wrote:


    2011/7/7 mark florisson <markflorisson88 at gmail.com>
    On 6 July 2011 10:01, Vitja Makarov wrote:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no
    arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options
    here:

    1. Add func_class member to CyFunction, this way __class__
    will
    be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field
    only
    when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more
    functions
    than
    specifically methods that use a no-argument call to super(),
    so
    this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for
    CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is
    WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would
    accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self'
    object
    there, which the CyFunction doesn't know. If anything, it only
    knows
    the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass
    'self'
    when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call
    we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    Would it be easier to make scope objects attributes of functions?
    Then
    you could still utilize PyCFunction_Call without needing to check the
    argument flags and such right?
    Sure, scope object is already functions attribute now it's stored
    inside
    self.
    But, instancemethods desc_get builds new args tuple with self as first
    element.
    We can implement cython version of instance method and change method
    signature a little bit.
    Right, and you pass it in as the first argument to the C function.
    However, if you want to change the signature, you have to override
    PyCFunction_Call, which, if I'm not mistaken, means you will have to
    interpret the flags from the PyMethodDef and call the C function in
    the right way (e.g. with or without the args tuple and kwargs) and
    check if it's being called correctly. If you leave it as an attribute
    of the function, you can pass in the function and access the scope
    object from the function object in the closure C function. So I think
    changing the signature might be a bit harder?
    Yes, but how to handle instantmethods self arg?
    We can't store it inside function object as it is different for each
    instance.
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Yes, that would work) But I think CyFunction is too heavy for methods.
    I've just realized that descr_get is called each time you access method,
    each time you call it and so on.


    Anyway, the important question was whether I could pull from your
    branch or whether I should wait for the merge.
    It's better not to use my branch this way as it's marked with underscore and
    sometimes I do forced pushes.
    I hope it will be merged soon.
    There is one more thing left CPython uses pool for frequently used objects
    to avoid unnecessary memory allocations.
    Probably we should implement something like that too.

    You are going to subclass cyfunction and then override its tp_call method,
    right?

    --
    vitja.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110708/946413f3/attachment-0001.html>
  • Mark florisson at Jul 7, 2011 at 9:23 pm

    On 7 July 2011 23:13, Vitja Makarov wrote:

    2011/7/8 mark florisson <markflorisson88 at gmail.com>
    On 7 July 2011 22:39, Vitja Makarov wrote:


    2011/7/8 mark florisson <markflorisson88 at gmail.com>
    On 7 July 2011 22:15, Vitja Makarov wrote:


    2011/7/7 mark florisson <markflorisson88 at gmail.com>
    On 6 July 2011 10:01, Vitja Makarov wrote:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no
    arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options
    here:

    1. Add func_class member to CyFunction, this way __class__
    will
    be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field
    only
    when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more
    functions
    than
    specifically methods that use a no-argument call to super(),
    so
    this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for
    CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is
    WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would
    accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self'
    object
    there, which the CyFunction doesn't know. If anything, it only
    knows
    the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to:
    super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass
    'self'
    when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self
    is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses
    PyCFunction_Call
    we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    Would it be easier to make scope objects attributes of functions?
    Then
    you could still utilize PyCFunction_Call without needing to check
    the
    argument flags and such right?
    Sure, scope object is already functions attribute now it's stored
    inside
    self.
    But, instancemethods desc_get builds new args tuple with self as
    first
    element.
    We can implement cython version of instance method and change method
    signature a little bit.
    Right, and you pass it in as the first argument to the C function.
    However, if you want to change the signature, you have to override
    PyCFunction_Call, which, if I'm not mistaken, means you will have to
    interpret the flags from the PyMethodDef and call the C function in
    the right way (e.g. with or without the args tuple and kwargs) and
    check if it's being called correctly. If you leave it as an attribute
    of the function, you can pass in the function and access the scope
    object from the function object in the closure C function. So I think
    changing the signature might be a bit harder?
    Yes, but how to handle instantmethods self arg?
    We can't store it inside function object as it is different for each
    instance.
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Yes, that would work) But I think CyFunction is too heavy for methods.
    I thought the python-like behaviour of them was appealing enough to
    consider defaulting to it? Otherwise you'd only have to use them for
    super(), __class__ and closures.
    I've just realized that descr_get is called each time you access method,
    each time you call it and so on.
    Anyway, the important question was whether I could pull from your
    branch or whether I should wait for the merge.
    It's better not to use my branch this way as it's marked with underscore and
    sometimes I do forced pushes.
    I hope it will be merged soon.
    There is one more thing left CPython uses pool for?frequently used objects
    to avoid?unnecessary memory allocations.
    Probably we should implement something like that too.

    You are going to subclass cyfunction and then override its tp_call method,
    right?
    Yes, exactly. The thing is that I only need to use it in extension
    methods, where the self for the method is passed in as m_self instead
    of in the args tuple. But I don't think a signature change will
    inconvenience that, so I think it's fine :)
    --
    vitja.


    _______________________________________________
    cython-devel mailing list
    cython-devel at python.org
    http://mail.python.org/mailman/listinfo/cython-devel
  • Robert Bradshaw at Jul 21, 2011 at 9:21 am

    On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote:

    2011/7/8 mark florisson <markflorisson88 at gmail.com>
    On 7 July 2011 22:39, Vitja Makarov wrote:


    2011/7/8 mark florisson <markflorisson88 at gmail.com>
    On 7 July 2011 22:15, Vitja Makarov wrote:


    2011/7/7 mark florisson <markflorisson88 at gmail.com>
    On 6 July 2011 10:01, Vitja Makarov wrote:
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 06.07.2011 09:05:
    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no
    arguments.
    http://trac.cython.org/cython_trac/ticket/696
    The problem is where to store __class__, I see two options
    here:

    1. Add func_class member to CyFunction, this way __class__
    will
    be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field
    only
    when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more
    functions
    than
    specifically methods that use a no-argument call to super(),
    so
    this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for
    CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is
    WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would
    accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self'
    object
    there, which the CyFunction doesn't know. If anything, it only
    knows
    the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to:
    super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass
    'self'
    when
    we pass the CyFunction object as the first argument?

    Oh, ok. Now we pass closure or nothing in self. So method's self
    is
    passed via tuple.
    Instancemethod do this for us. Now CyFucntion uses
    PyCFunction_Call
    we
    can override this and change signature of cyfunction to:

    PyObject func(CyFunction *func, PyObject *self, PyObject *args,
    PyObject *kwargs);

    This way we should implement new instancemethod type.
    Would it be easier to make scope objects attributes of functions?
    Then
    you could still utilize PyCFunction_Call without needing to check
    the
    argument flags and such right?
    Sure, scope object is already functions attribute now it's stored
    inside
    self.
    But, instancemethods desc_get builds new args tuple with self as
    first
    element.
    We can implement cython version of instance method and change method
    signature a little bit.
    Right, and you pass it in as the first argument to the C function.
    However, if you want to change the signature, you have to override
    PyCFunction_Call, which, if I'm not mistaken, means you will have to
    interpret the flags from the PyMethodDef and call the C function in
    the right way (e.g. with or without the args tuple and kwargs) and
    check if it's being called correctly. If you leave it as an attribute
    of the function, you can pass in the function and access the scope
    object from the function object in the closure C function. So I think
    changing the signature might be a bit harder?
    Yes, but how to handle instantmethods self arg?
    We can't store it inside function object as it is different for each
    instance.
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Yes, that would work) But I think CyFunction is too heavy for methods.
    I've just realized that descr_get is called each time you access method,
    each time you call it and so on.
    True, but this is also the case for CPython methods and method
    descriptors. There's also an inefficiency in calling bound methods in
    that a new tuple needs to be created with the bound parameter as the
    first component. Perhaps we could do away with that as well with
    specialized unpacking code. It would simultaneously be a step forward
    and backwards in terms of compatibility, but we could do away with
    ever generating PyCFunctions, methods, and method descriptors. Perhaps
    a CEP is in order to nail down the behavior of the CyFunction object.

    - Robert
  • Stefan Behnel at Jul 21, 2011 at 4:29 pm

    Robert Bradshaw, 21.07.2011 11:21:
    On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote:
    2011/7/8 mark florisson
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Yes, that would work) But I think CyFunction is too heavy for methods.
    I've just realized that descr_get is called each time you access method,
    each time you call it and so on.
    True, but this is also the case for CPython methods and method
    descriptors. There's also an inefficiency in calling bound methods in
    that a new tuple needs to be created with the bound parameter as the
    first component. Perhaps we could do away with that as well with
    specialized unpacking code. It would simultaneously be a step forward
    and backwards in terms of compatibility, but we could do away with
    ever generating PyCFunctions, methods, and method descriptors.
    That sounds like a really cool optimisation path. We could get rid of one
    indirection by replacing tp_call (originally PyCFunction_Call) directly by
    the real function entry point, i.e. the Python function wrapper (assuming
    the DefNode refactoring has taken place by then).

    It would also (somewhat) simplify the argument unpacking code, as that
    would no longer have to care about the METH_O and METH_NOARGS special
    cases. They'd be handled naturally inside of the Python call wrapper.

    Perhaps
    a CEP is in order to nail down the behavior of the CyFunction object.
    +1, totally helps in figuring out the details.

    Stefan
  • Vitja Makarov at Jul 21, 2011 at 6:57 pm

    2011/7/21 Stefan Behnel <stefan_ml at behnel.de>:
    Robert Bradshaw, 21.07.2011 11:21:
    On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote:

    2011/7/8 mark florisson
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Yes, that would work) But I think CyFunction is too heavy for methods.
    I've just realized that descr_get is called each time you access method,
    each time you call it and so on.
    True, but this is also the case for CPython methods and method
    descriptors. There's also an inefficiency in calling bound methods in
    that a new tuple needs to be created with the bound parameter as the
    first component. Perhaps we could do away with that as well with
    specialized unpacking code. It would simultaneously be a step forward
    and backwards in terms of compatibility, but we could do away with
    ever generating PyCFunctions, methods, and method descriptors.
    Yeah, I was thinking about that too.
    That sounds like a really cool optimisation path. We could get rid of one
    indirection by replacing tp_call (originally PyCFunction_Call) directly by
    the real function entry point, i.e. the Python function wrapper (assuming
    the DefNode refactoring has taken place by then).
    +1 I think DefNode refactoring should be done first (Did we froget about that?)
    It would also (somewhat) simplify the argument unpacking code, as that would
    no longer have to care about the METH_O and METH_NOARGS special cases.
    They'd be handled naturally inside of the Python call wrapper.

    Perhaps
    a CEP is in order to nail down the behavior of the CyFunction object.
    +1, totally helps in figuring out the details.
    CEP for refactored DefNode would be a good idea too.

    As I rember the idea is to turn def node into cdef node with wrapper
    call function just like cpdef buf without C-level access.
    So here we have to decide how to how to match python function's
    signature and cdef one.


    --
    vitja.
  • Stefan Behnel at Jul 22, 2011 at 5:58 am

    Vitja Makarov, 21.07.2011 20:57:
    2011/7/21 Stefan Behnel<stefan_ml at behnel.de>:
    Robert Bradshaw, 21.07.2011 11:21:
    On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote:

    2011/7/8 mark florisson
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Yes, that would work) But I think CyFunction is too heavy for methods.
    I've just realized that descr_get is called each time you access method,
    each time you call it and so on.
    True, but this is also the case for CPython methods and method
    descriptors. There's also an inefficiency in calling bound methods in
    that a new tuple needs to be created with the bound parameter as the
    first component. Perhaps we could do away with that as well with
    specialized unpacking code. It would simultaneously be a step forward
    and backwards in terms of compatibility, but we could do away with
    ever generating PyCFunctions, methods, and method descriptors.
    Yeah, I was thinking about that too.
    That sounds like a really cool optimisation path. We could get rid of one
    indirection by replacing tp_call (originally PyCFunction_Call) directly by
    the real function entry point, i.e. the Python function wrapper (assuming
    the DefNode refactoring has taken place by then).
    +1 I think DefNode refactoring should be done first (Did we froget about that?)
    It would also (somewhat) simplify the argument unpacking code, as that would
    no longer have to care about the METH_O and METH_NOARGS special cases.
    They'd be handled naturally inside of the Python call wrapper.

    Perhaps
    a CEP is in order to nail down the behavior of the CyFunction object.
    +1, totally helps in figuring out the details.
    CEP for refactored DefNode would be a good idea too.
    Well, there's still the part I wrote for the generator CEP:

    http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring

    As I rember the idea is to turn def node into cdef node with wrapper
    call function just like cpdef buf without C-level access.
    So here we have to decide how to how to match python function's
    signature and cdef one.
    I think that's mostly trivial - the argument unpacking code already does
    the mapping for us.

    Stefan
  • Vitja Makarov at Jul 22, 2011 at 6:13 am

    2011/7/22 Stefan Behnel <stefan_ml at behnel.de>:
    Vitja Makarov, 21.07.2011 20:57:
    2011/7/21 Stefan Behnel<stefan_ml at behnel.de>:
    Robert Bradshaw, 21.07.2011 11:21:
    On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote:

    2011/7/8 mark florisson
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Yes, that would work) But I think CyFunction is too heavy for methods.
    I've just realized that descr_get is called each time you access
    method,
    each time you call it and so on.
    True, but this is also the case for CPython methods and method
    descriptors. There's also an inefficiency in calling bound methods in
    that a new tuple needs to be created with the bound parameter as the
    first component. Perhaps we could do away with that as well with
    specialized unpacking code. It would simultaneously be a step forward
    and backwards in terms of compatibility, but we could do away with
    ever generating PyCFunctions, methods, and method descriptors.
    Yeah, I was thinking about that too.
    That sounds like a really cool optimisation path. We could get rid of one
    indirection by replacing tp_call (originally PyCFunction_Call) directly
    by
    the real function entry point, i.e. the Python function wrapper (assuming
    the DefNode refactoring has taken place by then).
    +1 I think DefNode refactoring should be done first (Did we froget about
    that?)
    It would also (somewhat) simplify the argument unpacking code, as that
    would
    no longer have to care about the METH_O and METH_NOARGS special cases.
    They'd be handled naturally inside of the Python call wrapper.

    Perhaps
    a CEP is in order to nail down the behavior of the CyFunction object.
    +1, totally helps in figuring out the details.
    CEP for refactored DefNode would be a good idea too.
    Well, there's still the part I wrote for the generator CEP:

    http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring

    As I rember the idea is to turn def node into cdef node with wrapper
    call function just like cpdef buf without C-level access.
    So here we have to decide how to how to match python function's
    signature and cdef one.
    I think that's mostly trivial - the argument unpacking code already does the
    mapping for us.
    So the wrapper could pass all the args (including defaults) to
    underlaying C-function.
    That solves the problem.

    --
    vitja.
  • Vitja Makarov at Jul 22, 2011 at 7:00 am

    2011/7/21 Stefan Behnel <stefan_ml at behnel.de>:
    Robert Bradshaw, 21.07.2011 11:21:
    On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote:

    2011/7/8 mark florisson
    In descr_get you create and return a new CyFunction with a __self__
    set (note, not m_self) and in __call__ you put __self__ in the args
    tuple and the function as m_self, and then call the CyFunction using
    PyCFunction_Call. But copying and modifying PyCFunction_Call works
    just as well I suppose :)
    Yes, that would work) But I think CyFunction is too heavy for methods.
    I've just realized that descr_get is called each time you access method,
    each time you call it and so on.
    True, but this is also the case for CPython methods and method
    descriptors. There's also an inefficiency in calling bound methods in
    that a new tuple needs to be created with the bound parameter as the
    first component. Perhaps we could do away with that as well with
    specialized unpacking code. It would simultaneously be a step forward
    and backwards in terms of compatibility, but we could do away with
    ever generating PyCFunctions, methods, and method descriptors.
    That sounds like a really cool optimisation path. We could get rid of one
    indirection by replacing tp_call (originally PyCFunction_Call) directly by
    the real function entry point, i.e. the Python function wrapper (assuming
    the DefNode refactoring has taken place by then).
    To replace tp_call with direct entry we have to create new type for
    each function,
    it's better to make tp_call call indirectly our wrapper.


    --
    vitja.
  • Vitja Makarov at Jul 12, 2011 at 7:46 am
    2011/7/6 Stefan Behnel <stefan_ml at behnel.de>
    Vitja Makarov, 06.07.2011 09:05:

    2011/7/6 Stefan Behnel<stefan_ml at behnel.de>:
    Stefan Behnel, 05.07.2011 10:04:
    Vitja Makarov, 05.07.2011 09:17:
    2011/7/5 Stefan Behnel:
    Vitja Makarov, 05.07.2011 08:21:
    I was thinking about implementing new super() with no arguments.
    http://trac.cython.org/cython_**trac/ticket/696<http://trac.cython.org/cython_trac/ticket/696>

    The problem is where to store __class__, I see two options here:
    1. Add func_class member to CyFunction, this way __class__ will be
    private and not visible for inner functions:
    2. Put it into closure
    The second option has the advantage of requiring the field only when
    super()
    is used, whereas the first impacts all functions.

    I would expect that programs commonly have a lot more functions than
    specifically methods that use a no-argument call to super(), so this
    may
    make a difference.
    So, now classes are created the following way:

    class_dict = {}
    class_dict.foo = foo_func
    class = CreateClass(class_dict)

    So after class is created I should check its dict for CyFunction
    members (maybe only ones that actually require __class__)
    and set __class__:

    for value in class.__dict__.itervalues():
    if isinstance(value, CyFunction) and value.func_class is WantClass:
    value.func_class = class

    Btw, first way requires cyfunction signature change, it would accept
    cyfunction object as first argument.
    We currently pass the binding (i.e. owning) object, right?
    So, how would this work for methods? We need to pass the 'self' object
    there, which the CyFunction doesn't know. If anything, it only knows the
    class it was defined in, which doesn't help here.
    From PEP: "super() is equivalent to: super(__class__,<firstarg>)"
    I wasn't speaking of super(). What I meant, was: how do we pass 'self' when
    we pass the CyFunction object as the first argument?

    About cdef classes: probably it's better to
    transform super().method(...) into direct form, e.g. BaseClass.method(self,
    ...)
    That would be more cdef-like than actually calling super.

    Btw, super() should still be used if no method call pattern is matched.

    --
    vitja.
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110712/3584af9a/attachment.html>
  • Stefan Behnel at Jul 12, 2011 at 8:16 am

    Vitja Makarov, 12.07.2011 09:46:
    About cdef classes: probably it's better to
    transform super().method(...) into direct form, e.g. BaseClass.method(self,
    ...)
    Except when it doesn't work. ;)

    A
    / \
    B C
    \ /
    D

    The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's
    implementation will be called twice, as it was already called by B.

    http://www.python.org/download/releases/2.2/descrintro/#mro

    Stefan
  • Vitja Makarov at Jul 12, 2011 at 9:46 am
    2011/7/12 Stefan Behnel <stefan_ml at behnel.de>
    Vitja Makarov, 12.07.2011 09:46:
    About cdef classes: probably it's better to
    transform super().method(...) into direct form, e.g. BaseClass.method(self,
    ...)
    Except when it doesn't work. ;)

    ? A
    ?/ \
    ?B ?C
    ?\ /
    ? D

    The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's implementation will be called twice, as it was already called by B.

    http://www.python.org/download/releases/2.2/descrintro/#mro
    According to your link mro in the example is DBCA ;)

    Is that a problem for cdef classes? Cdef class have only one base, isn't it?

    --
    vitja.
  • Mark florisson at Jul 12, 2011 at 10:11 am

    On 12 July 2011 11:46, Vitja Makarov wrote:
    2011/7/12 Stefan Behnel <stefan_ml at behnel.de>
    Vitja Makarov, 12.07.2011 09:46:
    About cdef classes: probably it's better to
    transform super().method(...) into direct form, e.g. BaseClass.method(self,
    ...)
    Except when it doesn't work. ;)

    ? A
    ?/ \
    ?B ?C
    ?\ /
    ? D

    The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's implementation will be called twice, as it was already called by B.

    http://www.python.org/download/releases/2.2/descrintro/#mro
    According to your link mro in the example is DBCA ;)

    Is that a problem for cdef classes? Cdef class have only one base, isn't it?
    That is irrelevant, if B calls A's method then C's method will be
    skipped. If it would use super() it would call C's method, and C's
    method would call A's method.
    --
    vitja.
    _______________________________________________
    cython-devel mailing list
    cython-devel at python.org
    http://mail.python.org/mailman/listinfo/cython-devel
  • Stefan Behnel at Jul 12, 2011 at 10:42 am

    mark florisson, 12.07.2011 12:11:
    On 12 July 2011 11:46, Vitja Makarov wrote:
    2011/7/12 Stefan Behnel
    Vitja Makarov, 12.07.2011 09:46:
    About cdef classes: probably it's better to
    transform super().method(...) into direct form, e.g. BaseClass.method(self,
    ...)
    Except when it doesn't work. ;)

    A
    / \
    B C
    \ /
    D

    The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's implementation will be called twice, as it was already called by B.

    http://www.python.org/download/releases/2.2/descrintro/#mro
    According to your link mro in the example is DBCA ;)
    Totally proves my point. ;)

    Is that a problem for cdef classes? Cdef class have only one base, isn't it?
    That is irrelevant, if B calls A's method then C's method will be
    skipped. If it would use super() it would call C's method, and C's
    method would call A's method.
    Exactly. So, calling directly into the base classes method implementation
    (i.e. issuing a direct C function call) makes a suitable optimisation for
    "final" classes, but not for the general case.

    Stefan
  • Vitja Makarov at Jul 12, 2011 at 10:45 am

    2011/7/12 Stefan Behnel <stefan_ml at behnel.de>:
    mark florisson, 12.07.2011 12:11:
    On 12 July 2011 11:46, Vitja Makarov wrote:

    2011/7/12 Stefan Behnel
    Vitja Makarov, 12.07.2011 09:46:
    About cdef classes: probably it's better to
    transform super().method(...) into direct form, e.g.
    BaseClass.method(self,
    ...)
    Except when it doesn't work. ;)

    ? A
    ?/ \
    ?B ?C
    ?\ /
    ? D

    The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's
    implementation will be called twice, as it was already called by B.

    http://www.python.org/download/releases/2.2/descrintro/#mro
    According to your link mro in the example is DBCA ;)
    Totally proves my point. ;)

    Is that a problem for cdef classes? Cdef class have only one base, isn't
    it?
    That is irrelevant, if B calls A's method then C's method will be
    skipped. If it would use super() it would call C's method, and C's
    method would call A's method.
    Exactly. So, calling directly into the base classes method implementation
    (i.e. issuing a direct C function call) makes a suitable optimisation for
    "final" classes, but not for the general case.
    Ohh, I got it.

    --
    vitja.

Related Discussions

Discussion Navigation
viewthread | post