FAQ
Hi,


With the help of CentOS project I am happy to announce an automated
system [1] to test patches from bugs.python.org. This can be fully automated
to test the patches whenever someone uploads a patch in the roundup, but
for now it accepts IRC commands on #python-dev channel. I worked on a
docker based prototype during sprints in PyCon.


How to use it?
---------------


1. Join #python-dev on irc.freenode.net.
2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
3. They will issue a simple command. #add: YOUR_NICK_NAME
4. You can then test by issuing the following command in the channel:


     #test: BUGNUMBER
     like #test: 21271


This will do the following:
Start a new job on ci.centos.org, announce it on the channel, and
announce the result also.


I will be working on a minimal lint for patches, and include it
the workflow.


The current steps can be found at [2]. Each build is happening on a
fresh system.


Limitations
-----------


1. It excepts one single patch to contain all the required changes
(instead of a series of patches).
2. It runs only on x86_64 architecture, CentOS7 based systems.






[1] https://ci.centos.org/job/cPython-build-patch/
[2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh


Kushal
--
Fedora Cloud Engineer
CPython Core Developer
Director @ Python Software Foundation
http://kushaldas.in

Search Discussions

  • Brett Cannon at May 19, 2015 at 3:25 pm
    In the airport but I wanted to say thanks for this!


    On Tue, May 19, 2015, 11:03 Kushal Das wrote:

    Hi,

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully
    automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271

    This will do the following:
    Start a new job on ci.centos.org, announce it on the channel, and
    announce the result also.

    I will be working on a minimal lint for patches, and include it
    the workflow.

    The current steps can be found at [2]. Each build is happening on a
    fresh system.

    Limitations
    -----------

    1. It excepts one single patch to contain all the required changes
    (instead of a series of patches).
    2. It runs only on x86_64 architecture, CentOS7 based systems.



    [1] https://ci.centos.org/job/cPython-build-patch/
    [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh

    Kushal
    --
    Fedora Cloud Engineer
    CPython Core Developer
    Director @ Python Software Foundation
    http://kushaldas.in
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/brett%40python.org
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150519/6845a7a5/attachment.html>
  • Berker Peksağ at May 19, 2015 at 3:48 pm

    On Tue, May 19, 2015 at 6:02 PM, Kushal Das wrote:
    Hi,

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271

    This will do the following:
    Start a new job on ci.centos.org, announce it on the channel, and
    announce the result also.

    Hi Kushal,


    Looks great, thanks! :)


    Two comments:


    * It would be good to have a pypatcher repository at hg.python.org (at
    least a mirror), so we can work on it together without dealing with
    "add me to the repo" messages on GitHub.
    * Do you have a roadmap or a TODO list? For example, I think
    downloading a tarball of the default branch every time (or is it
    cached?) would be a little bit slow. Do you have a plan to make the
    workflow Mercurial based (e.g. "hg pull -u, hg imp --no-c
    issueXXXX.diff, compile" instead of "wget tarball, extract it, apply
    patch, compile")?

    I will be working on a minimal lint for patches, and include it
    the workflow.

    Could you give more details about the linter? Can we use
    Tools/scripts/patchcheck.py?


    Thanks!


    --Berker
  • Kushal Das at May 19, 2015 at 4:53 pm

    On 19/05/15, Berker Peksa? wrote:
    On Tue, May 19, 2015 at 6:02 PM, Kushal Das wrote:
    Hi,

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271

    This will do the following:
    Start a new job on ci.centos.org, announce it on the channel, and
    announce the result also.
    Hi Kushal,

    Looks great, thanks! :)

    Two comments:

    * It would be good to have a pypatcher repository at hg.python.org (at
    least a mirror), so we can work on it together without dealing with
    "add me to the repo" messages on GitHub.

    We can surely do this. I started with github as generally most people
    are already there. Do you know what is the procedure for creating a new
    repo in hg.python.org?



    * Do you have a roadmap or a TODO list? For example, I think
    downloading a tarball of the default branch every time (or is it
    cached?) would be a little bit slow. Do you have a plan to make the
    workflow Mercurial based (e.g. "hg pull -u, hg imp --no-c
    issueXXXX.diff, compile" instead of "wget tarball, extract it, apply
    patch, compile")?
    I will have to work on the TODO list, I will post it on the repo itself.
    The downloading tarball currently takes around 15-16 seconds, which I
    found fast enough to start with. I personally always use standard patch
    command, that is why I chose this approach instead of hg. We can always
    improve the workflow :)



    I will be working on a minimal lint for patches, and include it
    the workflow.
    Could you give more details about the linter? Can we use
    Tools/scripts/patchcheck.py?
    For this I really never thought much. We should discuss more on this to
    find out what all we can do.


    Kushal
    --
    Fedora Cloud Engineer
    CPython Core Developer
    Director @ Python Software Foundation
    http://kushaldas.in
  • Wes Turner at May 19, 2015 at 6:22 pm
    Cool! Thanks!


    BuildBot integration?


    * http://docs.buildbot.net/latest/full.html#source-stamps
    * http://docs.buildbot.net/latest/full.html#enabling-the-irc-bot
    * http://docs.buildbot.net/latest/full.html#choosing-a-change-source


    https://github.com/audreyr/cookiecutter-pypackage (requirements.txt,
    setup.py, Makefile)


        pip install cookiecutter
        cookiecutter gh:audreyr/cookiecutter-pypackage




    On Tue, May 19, 2015 at 10:02 AM, Kushal Das wrote:

    Hi,

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully
    automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271

    This will do the following:
    Start a new job on ci.centos.org, announce it on the channel, and
    announce the result also.

    I will be working on a minimal lint for patches, and include it
    the workflow.

    The current steps can be found at [2]. Each build is happening on a
    fresh system.

    Limitations
    -----------

    1. It excepts one single patch to contain all the required changes
    (instead of a series of patches).
    2. It runs only on x86_64 architecture, CentOS7 based systems.



    [1] https://ci.centos.org/job/cPython-build-patch/
    [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh

    Kushal
    --
    Fedora Cloud Engineer
    CPython Core Developer
    Director @ Python Software Foundation
    http://kushaldas.in
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150519/1fe92b3e/attachment.html>
  • Wes Turner at May 19, 2015 at 6:25 pm
    http://docs.buildbot.net/latest/search.html?q=docker ...


    Here's a BuildBot Dockerfile:
    http://docs.buildbot.net/latest/manual/cfg-buildslaves-docker.html#image-creation


    On Tue, May 19, 2015 at 1:22 PM, Wes Turner wrote:

    Cool! Thanks!

    BuildBot integration?

    * http://docs.buildbot.net/latest/full.html#source-stamps
    * http://docs.buildbot.net/latest/full.html#enabling-the-irc-bot
    * http://docs.buildbot.net/latest/full.html#choosing-a-change-source

    https://github.com/audreyr/cookiecutter-pypackage (requirements.txt,
    setup.py, Makefile)

    pip install cookiecutter
    cookiecutter gh:audreyr/cookiecutter-pypackage

    On Tue, May 19, 2015 at 10:02 AM, Kushal Das wrote:

    Hi,

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully
    automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271

    This will do the following:
    Start a new job on ci.centos.org, announce it on the channel, and
    announce the result also.

    I will be working on a minimal lint for patches, and include it
    the workflow.

    The current steps can be found at [2]. Each build is happening on a
    fresh system.

    Limitations
    -----------

    1. It excepts one single patch to contain all the required changes
    (instead of a series of patches).
    2. It runs only on x86_64 architecture, CentOS7 based systems.



    [1] https://ci.centos.org/job/cPython-build-patch/
    [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh

    Kushal
    --
    Fedora Cloud Engineer
    CPython Core Developer
    Director @ Python Software Foundation
    http://kushaldas.in
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150519/6a0a5fc3/attachment-0001.html>
  • Wes Turner at May 19, 2015 at 6:27 pm
    Here's a GerritChangeSource for BuildBot events:
    http://docs.buildbot.net/latest/full.html#chsrc-GerritChangeSource


    Great idea, thanks again!


    On Tue, May 19, 2015 at 1:25 PM, Wes Turner wrote:

    http://docs.buildbot.net/latest/search.html?q=docker ...

    Here's a BuildBot Dockerfile:
    http://docs.buildbot.net/latest/manual/cfg-buildslaves-docker.html#image-creation
    On Tue, May 19, 2015 at 1:22 PM, Wes Turner wrote:

    Cool! Thanks!

    BuildBot integration?

    * http://docs.buildbot.net/latest/full.html#source-stamps
    * http://docs.buildbot.net/latest/full.html#enabling-the-irc-bot
    * http://docs.buildbot.net/latest/full.html#choosing-a-change-source

    https://github.com/audreyr/cookiecutter-pypackage (requirements.txt,
    setup.py, Makefile)

    pip install cookiecutter
    cookiecutter gh:audreyr/cookiecutter-pypackage

    On Tue, May 19, 2015 at 10:02 AM, Kushal Das wrote:

    Hi,

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully
    automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271

    This will do the following:
    Start a new job on ci.centos.org, announce it on the channel, and
    announce the result also.

    I will be working on a minimal lint for patches, and include it
    the workflow.

    The current steps can be found at [2]. Each build is happening on a
    fresh system.

    Limitations
    -----------

    1. It excepts one single patch to contain all the required changes
    (instead of a series of patches).
    2. It runs only on x86_64 architecture, CentOS7 based systems.



    [1] https://ci.centos.org/job/cPython-build-patch/
    [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh

    Kushal
    --
    Fedora Cloud Engineer
    CPython Core Developer
    Director @ Python Software Foundation
    http://kushaldas.in
    _______________________________________________
    Python-Dev mailing list
    Python-Dev at python.org
    https://mail.python.org/mailman/listinfo/python-dev
    Unsubscribe:
    https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/python-dev/attachments/20150519/593211ee/attachment-0001.html>
  • Terry Reedy at May 19, 2015 at 8:37 pm

    On 5/19/2015 11:02 AM, Kushal Das wrote:
    Hi,

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271

    What if there are multiple patches on the issue? Pick the latest?
    This is not correct if someone follows up a patch with a 2.7 backport,
    or if there are competing patches.

    This will do the following:
    Start a new job on ci.centos.org, announce it on the channel, and
    announce the result also.

    I will be working on a minimal lint for patches, and include it
    the workflow.

    The current steps can be found at [2]. Each build is happening on a
    fresh system.

    Limitations
    -----------

    1. It excepts one single patch to contain all the required changes
    (instead of a series of patches).




    2. It runs only on x86_64 architecture, CentOS7 based systems.



    [1] https://ci.centos.org/job/cPython-build-patch/
    [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh

    Kushal



    --
    Terry Jan Reedy
  • Anatoly techtonik at May 20, 2015 at 7:44 am

    On Tue, May 19, 2015 at 11:37 PM, Terry Reedy wrote:
    On 5/19/2015 11:02 AM, Kushal Das wrote:

    Hi,

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully
    automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271

    What if there are multiple patches on the issue? Pick the latest?
    This is not correct if someone follows up a patch with a 2.7 backport, or if
    there are competing patches.

    Here is the code that checks how much outstanding patches a certain
    module has by downloading all patches from open issues, parsing them
    and comparing the paths. It is possible to reuse the parser to check paths
    in patch against paths present in certain Python versions, or add different
    heuristics.


    https://bitbucket.org/techtonik/python-stdlib


    All this is pure Python and should work cross-platform too.




    This was intended to add status for bugs.python.org, but the work on
    Roundup had stalled due to uncertainty and despair on how to handle
    utf-8 (internal to Roundup) vs unicode (internal to Jinja2) in this issue:
    http://issues.roundup-tracker.org/issue2550811
    The root of the problem is that Python 2.7 uses 'ascii' and not 'utf-8'
    internally, so Jinja2 engine fails with 'ascii' ordinal not in range
    somewhere in the way. Need an expert advice how to handle that, because
    my brain power is not enough to process it.
  • Terry Reedy at May 20, 2015 at 6:03 pm

    On 5/20/2015 3:44 AM, anatoly techtonik wrote:


    This was intended to add status for bugs.python.org, but the work on
    Roundup had stalled due to uncertainty and despair on how to handle
    utf-8 (internal to Roundup) vs unicode (internal to Jinja2) in this issue:
    http://issues.roundup-tracker.org/issue2550811
    The root of the problem is that Python 2.7 uses 'ascii' and not 'utf-8'
    internally, so Jinja2 engine fails with 'ascii' ordinal not in range
    somewhere in the way. Need an expert advice how to handle that, because
    my brain power is not enough to process it.

    In my view, the root of the problem is using Python 2 and working with
    encoded bytes. The fix is to upgrade to Python 3.3+, with an improved
    text model (unicode str class), and work with text. Follow the standard
    process: decode encoded bytes to text when received from browsers, work
    internally with text, and encode output just before sending to browsers.


    --
    Terry Jan Reedy
  • Olemis Lang at May 20, 2015 at 8:42 pm

    On 5/19/15, Terry Reedy wrote:
    On 5/19/2015 11:02 AM, Kushal Das wrote:
    Hi,

    Hi !


    I'm not very familiar with python-dev development workflows .
    Nonetheless I just wanted to mention something that proved to be
    useful for me in the past .

    With the help of CentOS project I am happy to announce an automated
    system [1] to test patches from bugs.python.org. This can be fully
    automated
    to test the patches whenever someone uploads a patch in the roundup, but
    for now it accepts IRC commands on #python-dev channel. I worked on a
    docker based prototype during sprints in PyCon.

    How to use it?
    ---------------

    1. Join #python-dev on irc.freenode.net.
    2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer
    3. They will issue a simple command. #add: YOUR_NICK_NAME
    4. You can then test by issuing the following command in the channel:

    #test: BUGNUMBER
    like #test: 21271
    What if there are multiple patches on the issue? Pick the latest?
    This is not correct if someone follows up a patch with a 2.7 backport,
    or if there are competing patches.
    [...]


    It is a fact that running automated tests for patches is a really
    useful feature . Nevertheless , IMHO for this to succeed at large
    scale there is a need to manage the content of patches themselves ,
    the base version they were built upon , as well as their order should
    they be stacked . My suggestion for you therefore is to use Hg patch
    repositories [1]_ as the starting point for your patch CI system .
    Some of the benefits I could mention :


       - triggering (patch) builds on commit via web hooks
       - CI infrastructure needed turns out to be very
         similar to the one setup for the main project
       - Commands to work on patch queue repositories are easy to learn
       - The possibility of editing series file is also useful for ignoring
         some patches without removing their contents .
       - halt if patch cannot be applied upon latest version
         * ... but still be able to see it in action by checking out the
    right version
           of the code base used to build it in first place .
       - try the patch against different versions of the code base as it evolves
       - fuzzy refresh
       - version control for patches
       - multiple branches
         * which may be bound to tickets in many ways e.g. naming conventions
         * ... particularly useful for competing patches .


    There are a few black spots too . Patch repositories deal with the
    diff of a diff , hence some operations applied upon patches (e.g.
    merging) might be quite messy , Most of the time this is no big deal
    though .


    The following are repositories I used while developing Apache
    Bloodhound , during incubation and after it became a TLP . I'm
    including them to illustrate branching and naming conventions (I used)
    to keep track of tickets .


    https://bitbucket.org/olemis/bloodhound-incubator-mq/
    https://bitbucket.org/olemis/bloodhound-mq


    HTH , since this way the workflow would be tightly integrated with
    Mercurial , as highlighted by Berker Peksa? in previous messages .


    .. [1] http://mercurial.selenic.com/wiki/MqTutorial


    --
    Regards,


    Olemis - @olemislc


    Apache? Bloodhound contributor
    http://issues.apache.org/bloodhound
    http://blood-hound.net


    Blog ES: http://simelo-es.blogspot.com/
    Blog EN: http://simelo-en.blogspot.com/


    Featured article:

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppython-dev @
categoriespython
postedMay 19, '15 at 3:02p
activeMay 20, '15 at 8:42p
posts11
users7
websitepython.org

People

Translate

site design / logo © 2017 Grokbase