FAQ
I was talking to the Travis CI folks and they mentioned they have the
possibility of using openstack builders now (or in the near future). We talked
a little bit about the possibility of getting PyCA it's own dedicated set of
builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.


Currently the entirety of the PyPA organization on Github gets 10 concurrent
builders (typically this number is 5) across all repositories. Assuming Travis
CI is able to do it (they'd need to check with their ops team, and there would
be some cost associated with things on their side as well) we could essentially
control how much concurrency we want by just throwing more Rackspace VMs at
our builds.


Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace
Compute1-4 which is similar we'd be looking at the ability to run ~27 of those
VMs full time on the "standard" 2k/month free cloud offering that Rackspace
offers OSS projects. However we'd actually be able to get more than that
probably because Travis starts up the VMs on demand so anytime there aren't
tests to run we won't have any VMs running.


Independent of this, they are also working on the ability to run custom docker
images (which means custom Linux OS images) that we can preinstall different
software into.


Is this something we'd want to explore? Assuming that the Travis CI ops team
and such are OK with it, it could essentially let us scale up our concurrency
on Travis to whatever amount of dollars of Rackspace cloud we want to throw at
it.


---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150328/b268caec/attachment.sig>

Search Discussions

  • Alex Gaynor at Mar 28, 2015 at 10:58 pm
    Just to be clear, can we do *both* the custom docker and the Rackspace
    thing?


    This sounds like it'd be a big win: we could centralize all of our Linux
    builders onto that, they'd manage it, we'd get all the concurrency we need
    because the system would have VMs shut down most of the time to save money,
    and it'd even be easy to have them all feeding into coveralls!


    Alex


    On Sat, Mar 28, 2015 at 6:56 PM, Donald Stufft wrote:

    I was talking to the Travis CI folks and they mentioned they have the
    possibility of using openstack builders now (or in the near future). We
    talked
    a little bit about the possibility of getting PyCA it's own dedicated set
    of
    builders in Travis CI hosted inside of Rackspace on one of our cloud
    accounts.

    Currently the entirety of the PyPA organization on Github gets 10
    concurrent
    builders (typically this number is 5) across all repositories. Assuming
    Travis
    CI is able to do it (they'd need to check with their ops team, and there
    would
    be some cost associated with things on their side as well) we could
    essentially
    control how much concurrency we want by just throwing more Rackspace VMs at
    our builds.

    Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the
    Rackspace
    Compute1-4 which is similar we'd be looking at the ability to run ~27 of
    those
    VMs full time on the "standard" 2k/month free cloud offering that Rackspace
    offers OSS projects. However we'd actually be able to get more than that
    probably because Travis starts up the VMs on demand so anytime there aren't
    tests to run we won't have any VMs running.

    Independent of this, they are also working on the ability to run custom
    docker
    images (which means custom Linux OS images) that we can preinstall
    different
    software into.

    Is this something we'd want to explore? Assuming that the Travis CI ops
    team
    and such are OK with it, it could essentially let us scale up our
    concurrency
    on Travis to whatever amount of dollars of Rackspace cloud we want to
    throw at
    it.

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev



    --
    "I disapprove of what you say, but I will defend to the death your right to
    say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint: 125F 5C67 DFE9 4084
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150328/cc68ad09/attachment.html>
  • Donald Stufft at Mar 28, 2015 at 11:06 pm
    Yes, we can run custom docker images *and* have dedicated builders running inside Rackspace.



    On Mar 28, 2015, at 6:58 PM, Alex Gaynor wrote:

    Just to be clear, can we do *both* the custom docker and the Rackspace thing?

    This sounds like it'd be a big win: we could centralize all of our Linux builders onto that, they'd manage it, we'd get all the concurrency we need because the system would have VMs shut down most of the time to save money, and it'd even be easy to have them all feeding into coveralls!

    Alex

    On Sat, Mar 28, 2015 at 6:56 PM, Donald Stufft <donald at stufft.io wrote:
    I was talking to the Travis CI folks and they mentioned they have the
    possibility of using openstack builders now (or in the near future). We talked
    a little bit about the possibility of getting PyCA it's own dedicated set of
    builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.

    Currently the entirety of the PyPA organization on Github gets 10 concurrent
    builders (typically this number is 5) across all repositories. Assuming Travis
    CI is able to do it (they'd need to check with their ops team, and there would
    be some cost associated with things on their side as well) we could essentially
    control how much concurrency we want by just throwing more Rackspace VMs at
    our builds.

    Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace
    Compute1-4 which is similar we'd be looking at the ability to run ~27 of those
    VMs full time on the "standard" 2k/month free cloud offering that Rackspace
    offers OSS projects. However we'd actually be able to get more than that
    probably because Travis starts up the VMs on demand so anytime there aren't
    tests to run we won't have any VMs running.

    Independent of this, they are also working on the ability to run custom docker
    images (which means custom Linux OS images) that we can preinstall different
    software into.

    Is this something we'd want to explore? Assuming that the Travis CI ops team
    and such are OK with it, it could essentially let us scale up our concurrency
    on Travis to whatever amount of dollars of Rackspace cloud we want to throw at
    it.

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org <mailto:cryptography-dev@python.org>
    https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>




    --
    "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint: 125F 5C67 DFE9 4084
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150328/e96cb517/attachment-0001.html>
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: signature.asc
    Type: application/pgp-signature
    Size: 801 bytes
    Desc: Message signed with OpenPGP using GPGMail
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150328/e96cb517/attachment-0001.sig>
  • Paul Kehrer at Mar 29, 2015 at 4:53 pm
    (Sorry for the exposition that's about to happen, just wanted to write it all down first)


    We use two different CI systems right now:


    - Travis, which provides OS X 10.9 builds as well as linux builds (against Ubuntu 12.04 with two different versions of OpenSSL).
    - Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie, Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL, Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X 10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server 2012 (64-bit Pythons).


    Travis provides us some (large) advantages over Jenkins, including not having to manage our own fleet of builders, far better (albeit less flexible) configuration through .travis.yml, and a difficult to quantify "pleasantness" to the entire experience.


    So, what problems do we run into with Travis?


    Coverage
    ------------
    At this time we use coveralls (which is now commenting up to 9 times on every PR for reasons that pass human understanding) to gain a view of our combined coverage. Unfortunately this system ties us to reporting coverage exclusively from Travis. This means code paths (like windows only code) cannot have coverage tracked.


    Speed/Reliability
    ----------------------
    We have limited (albeit very high at the moment) concurrency available from Travis. On a typical workday it can be 1-3 hours before CI completes, which significantly impairs our ability to quickly review/merge code.??We have had occasions in the past where we've had Travis jobs waiting in the queue for over 8 hours due to reliability problems within the Travis infrastructure.


    Flexibility
    ------------
    Travis provides only OS X 10.9 and Ubuntu 12.04.






    The openstack builder possibility solves much of the speed issue, and custom docker images would negate the need for our jenkins linux builders (of which there are currently 7). I am, however, concerned about coverage. Without being able to run more than OS X 10.9 and (potentially) various userlands of Linux via docker images we're still significantly handicapped. It is unreasonable to expect to be able to run arbitrary OSes so we'll always run a small jenkins instance for things like alternate OS X versions, FreeBSD, etc, but the lack of Windows support is painful.


    We have recently discussed moving entirely off Travis (https://github.com/pyca/cryptography/issues/1729), but if the above comes to fruition and there's a good way to combine coverage from multiple sources (and someone wants to own getting it deployed that isn't me) then I'd be happy to stay with Travis and only use jenkins for the edge cases.


    -Paul


    On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io) wrote:


    I was talking to the Travis CI folks and they mentioned they have the
    possibility of using openstack builders now (or in the near future). We talked
    a little bit about the possibility of getting PyCA it's own dedicated set of
    builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.


    Currently the entirety of the PyPA organization on Github gets 10 concurrent
    builders (typically this number is 5) across all repositories. Assuming Travis
    CI is able to do it (they'd need to check with their ops team, and there would
    be some cost associated with things on their side as well) we could essentially
    control how much concurrency we want by just throwing more Rackspace VMs at
    our builds.


    Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace
    Compute1-4 which is similar we'd be looking at the ability to run ~27 of those
    VMs full time on the "standard" 2k/month free cloud offering that Rackspace
    offers OSS projects. However we'd actually be able to get more than that
    probably because Travis starts up the VMs on demand so anytime there aren't
    tests to run we won't have any VMs running.


    Independent of this, they are also working on the ability to run custom docker
    images (which means custom Linux OS images) that we can preinstall different
    software into.


    Is this something we'd want to explore? Assuming that the Travis CI ops team
    and such are OK with it, it could essentially let us scale up our concurrency
    on Travis to whatever amount of dollars of Rackspace cloud we want to throw at
    it.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/9f28558d/attachment-0001.html>
  • Alex Gaynor at Mar 29, 2015 at 4:55 pm
    Paul, it seems like coveralls is kind of orthagonal to this porposal,
    besides the fact that in the short term moving more stuff under Travis
    would mean coveralls included more stuff?


    Alex


    On Sun, Mar 29, 2015 at 12:53 PM, Paul Kehrer wrote:

    (Sorry for the exposition that's about to happen, just wanted to write it
    all down first)

    We use two different CI systems right now:

    - Travis, which provides OS X 10.9 builds as well as linux builds (against
    Ubuntu 12.04 with two different versions of OpenSSL).
    - Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie,
    Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL,
    Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X
    10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server
    2012 (64-bit Pythons).

    Travis provides us some (large) advantages over Jenkins, including not
    having to manage our own fleet of builders, far better (albeit less
    flexible) configuration through .travis.yml, and a difficult to quantify
    "pleasantness" to the entire experience.

    So, what problems do we run into with Travis?

    Coverage
    ------------
    At this time we use coveralls (which is now commenting up to 9 times on
    every PR for reasons that pass human understanding) to gain a view of our
    combined coverage. Unfortunately this system ties us to reporting coverage
    exclusively from Travis. This means code paths (like windows only code)
    cannot have coverage tracked.

    Speed/Reliability
    ----------------------
    We have limited (albeit very high at the moment) concurrency available
    from Travis. On a typical workday it can be 1-3 hours before CI completes,
    which significantly impairs our ability to quickly review/merge code. We
    have had occasions in the past where we've had Travis jobs waiting in the
    queue for over 8 hours due to reliability problems within the Travis
    infrastructure.

    Flexibility
    ------------
    Travis provides only OS X 10.9 and Ubuntu 12.04.



    The openstack builder possibility solves much of the speed issue, and
    custom docker images would negate the need for our jenkins linux builders
    (of which there are currently 7). I am, however, concerned about coverage.
    Without being able to run more than OS X 10.9 and (potentially) various
    userlands of Linux via docker images we're still significantly handicapped.
    It is unreasonable to expect to be able to run arbitrary OSes so we'll
    always run a small jenkins instance for things like alternate OS X
    versions, FreeBSD, etc, but the lack of Windows support is painful.

    We have recently discussed moving entirely off Travis (
    https://github.com/pyca/cryptography/issues/1729), but if the above comes
    to fruition and there's a good way to combine coverage from multiple
    sources (and someone wants to own getting it deployed that isn't me) then
    I'd be happy to stay with Travis and only use jenkins for the edge cases.

    -Paul

    On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io) wrote:

    I was talking to the Travis CI folks and they mentioned they have the
    possibility of using openstack builders now (or in the near future). We
    talked
    a little bit about the possibility of getting PyCA it's own dedicated set
    of
    builders in Travis CI hosted inside of Rackspace on one of our cloud
    accounts.

    Currently the entirety of the PyPA organization on Github gets 10
    concurrent
    builders (typically this number is 5) across all repositories. Assuming
    Travis
    CI is able to do it (they'd need to check with their ops team, and there
    would
    be some cost associated with things on their side as well) we could
    essentially
    control how much concurrency we want by just throwing more Rackspace VMs
    at
    our builds.

    Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the
    Rackspace
    Compute1-4 which is similar we'd be looking at the ability to run ~27 of
    those
    VMs full time on the "standard" 2k/month free cloud offering that
    Rackspace
    offers OSS projects. However we'd actually be able to get more than that
    probably because Travis starts up the VMs on demand so anytime there
    aren't
    tests to run we won't have any VMs running.

    Independent of this, they are also working on the ability to run custom
    docker
    images (which means custom Linux OS images) that we can preinstall
    different
    software into.

    Is this something we'd want to explore? Assuming that the Travis CI ops
    team
    and such are OK with it, it could essentially let us scale up our
    concurrency
    on Travis to whatever amount of dollars of Rackspace cloud we want to
    throw at
    it.

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

    ------------------------------
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev



    --
    "I disapprove of what you say, but I will defend to the death your right to
    say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint: 125F 5C67 DFE9 4084
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/227df108/attachment.html>
  • Paul Kehrer at Mar 29, 2015 at 4:59 pm
    Travis is (potentially) offering to undertake significant work on our behalf so I believe we need to have an answer to our long term coverage questions before we commit to using this solution. Being faster and having custom docker images doesn't grant us a workable coverage solution so we're going to be discussing moving entirely to jenkins again at some point in the future if we can't resolve that issue.


    -Paul
    On March 29, 2015 at 11:55:55 AM, Alex Gaynor (alex.gaynor at gmail.com) wrote:


    Paul, it seems like coveralls is kind of orthagonal to this porposal, besides the fact that in the short term moving more stuff under Travis would mean coveralls included more stuff?


    Alex


    On Sun, Mar 29, 2015 at 12:53 PM, Paul Kehrer wrote:
    (Sorry for the exposition that's about to happen, just wanted to write it all down first)


    We use two different CI systems right now:


    - Travis, which provides OS X 10.9 builds as well as linux builds (against Ubuntu 12.04 with two different versions of OpenSSL).
    - Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie, Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL, Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X 10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server 2012 (64-bit Pythons).


    Travis provides us some (large) advantages over Jenkins, including not having to manage our own fleet of builders, far better (albeit less flexible) configuration through .travis.yml, and a difficult to quantify "pleasantness" to the entire experience.


    So, what problems do we run into with Travis?


    Coverage
    ------------
    At this time we use coveralls (which is now commenting up to 9 times on every PR for reasons that pass human understanding) to gain a view of our combined coverage. Unfortunately this system ties us to reporting coverage exclusively from Travis. This means code paths (like windows only code) cannot have coverage tracked.


    Speed/Reliability
    ----------------------
    We have limited (albeit very high at the moment) concurrency available from Travis. On a typical workday it can be 1-3 hours before CI completes, which significantly impairs our ability to quickly review/merge code.??We have had occasions in the past where we've had Travis jobs waiting in the queue for over 8 hours due to reliability problems within the Travis infrastructure.


    Flexibility
    ------------
    Travis provides only OS X 10.9 and Ubuntu 12.04.






    The openstack builder possibility solves much of the speed issue, and custom docker images would negate the need for our jenkins linux builders (of which there are currently 7). I am, however, concerned about coverage. Without being able to run more than OS X 10.9 and (potentially) various userlands of Linux via docker images we're still significantly handicapped. It is unreasonable to expect to be able to run arbitrary OSes so we'll always run a small jenkins instance for things like alternate OS X versions, FreeBSD, etc, but the lack of Windows support is painful.


    We have recently discussed moving entirely off Travis (https://github.com/pyca/cryptography/issues/1729), but if the above comes to fruition and there's a good way to combine coverage from multiple sources (and someone wants to own getting it deployed that isn't me) then I'd be happy to stay with Travis and only use jenkins for the edge cases.


    -Paul


    On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io) wrote:


    I was talking to the Travis CI folks and they mentioned they have the
    possibility of using openstack builders now (or in the near future). We talked
    a little bit about the possibility of getting PyCA it's own dedicated set of
    builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.


    Currently the entirety of the PyPA organization on Github gets 10 concurrent
    builders (typically this number is 5) across all repositories. Assuming Travis
    CI is able to do it (they'd need to check with their ops team, and there would
    be some cost associated with things on their side as well) we could essentially
    control how much concurrency we want by just throwing more Rackspace VMs at
    our builds.


    Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace
    Compute1-4 which is similar we'd be looking at the ability to run ~27 of those
    VMs full time on the "standard" 2k/month free cloud offering that Rackspace
    offers OSS projects. However we'd actually be able to get more than that
    probably because Travis starts up the VMs on demand so anytime there aren't
    tests to run we won't have any VMs running.


    Independent of this, they are also working on the ability to run custom docker
    images (which means custom Linux OS images) that we can preinstall different
    software into.


    Is this something we'd want to explore? Assuming that the Travis CI ops team
    and such are OK with it, it could essentially let us scale up our concurrency
    on Travis to whatever amount of dollars of Rackspace cloud we want to throw at
    it.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev








    --
    "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint:?125F 5C67 DFE9 4084
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/790f5fe1/attachment-0001.html>
  • Alex Gaynor at Mar 29, 2015 at 5:02 pm
    Sorry, I'm not following, it seems like you're assuming that "thing that
    replaces coveralls" wouldn't be able to work with both Travis and Jenkins?


    Alex


    On Sun, Mar 29, 2015 at 12:59 PM, Paul Kehrer wrote:

    Travis is (potentially) offering to undertake significant work on our
    behalf so I believe we need to have an answer to our long term coverage
    questions before we commit to using this solution. Being faster and having
    custom docker images doesn't grant us a workable coverage solution so we're
    going to be discussing moving entirely to jenkins again at some point in
    the future if we can't resolve that issue.

    -Paul

    On March 29, 2015 at 11:55:55 AM, Alex Gaynor (alex.gaynor at gmail.com)
    wrote:

    Paul, it seems like coveralls is kind of orthagonal to this porposal,
    besides the fact that in the short term moving more stuff under Travis
    would mean coveralls included more stuff?

    Alex
    On Sun, Mar 29, 2015 at 12:53 PM, Paul Kehrer wrote:

    (Sorry for the exposition that's about to happen, just wanted to write
    it all down first)

    We use two different CI systems right now:

    - Travis, which provides OS X 10.9 builds as well as linux builds
    (against Ubuntu 12.04 with two different versions of OpenSSL).
    - Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie,
    Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL,
    Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X
    10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server
    2012 (64-bit Pythons).

    Travis provides us some (large) advantages over Jenkins, including not
    having to manage our own fleet of builders, far better (albeit less
    flexible) configuration through .travis.yml, and a difficult to quantify
    "pleasantness" to the entire experience.

    So, what problems do we run into with Travis?

    Coverage
    ------------
    At this time we use coveralls (which is now commenting up to 9 times on
    every PR for reasons that pass human understanding) to gain a view of our
    combined coverage. Unfortunately this system ties us to reporting coverage
    exclusively from Travis. This means code paths (like windows only code)
    cannot have coverage tracked.

    Speed/Reliability
    ----------------------
    We have limited (albeit very high at the moment) concurrency available
    from Travis. On a typical workday it can be 1-3 hours before CI completes,
    which significantly impairs our ability to quickly review/merge code. We
    have had occasions in the past where we've had Travis jobs waiting in the
    queue for over 8 hours due to reliability problems within the Travis
    infrastructure.

    Flexibility
    ------------
    Travis provides only OS X 10.9 and Ubuntu 12.04.



    The openstack builder possibility solves much of the speed issue, and
    custom docker images would negate the need for our jenkins linux builders
    (of which there are currently 7). I am, however, concerned about coverage.
    Without being able to run more than OS X 10.9 and (potentially) various
    userlands of Linux via docker images we're still significantly handicapped.
    It is unreasonable to expect to be able to run arbitrary OSes so we'll
    always run a small jenkins instance for things like alternate OS X
    versions, FreeBSD, etc, but the lack of Windows support is painful.

    We have recently discussed moving entirely off Travis (
    https://github.com/pyca/cryptography/issues/1729), but if the above
    comes to fruition and there's a good way to combine coverage from multiple
    sources (and someone wants to own getting it deployed that isn't me) then
    I'd be happy to stay with Travis and only use jenkins for the edge cases.

    -Paul

    On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io) wrote:

    I was talking to the Travis CI folks and they mentioned they have the
    possibility of using openstack builders now (or in the near future). We
    talked
    a little bit about the possibility of getting PyCA it's own dedicated set
    of
    builders in Travis CI hosted inside of Rackspace on one of our cloud
    accounts.

    Currently the entirety of the PyPA organization on Github gets 10
    concurrent
    builders (typically this number is 5) across all repositories. Assuming
    Travis
    CI is able to do it (they'd need to check with their ops team, and there
    would
    be some cost associated with things on their side as well) we could
    essentially
    control how much concurrency we want by just throwing more Rackspace VMs
    at
    our builds.

    Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the
    Rackspace
    Compute1-4 which is similar we'd be looking at the ability to run ~27 of
    those
    VMs full time on the "standard" 2k/month free cloud offering that
    Rackspace
    offers OSS projects. However we'd actually be able to get more than that
    probably because Travis starts up the VMs on demand so anytime there
    aren't
    tests to run we won't have any VMs running.

    Independent of this, they are also working on the ability to run custom
    docker
    images (which means custom Linux OS images) that we can preinstall
    different
    software into.

    Is this something we'd want to explore? Assuming that the Travis CI ops
    team
    and such are OK with it, it could essentially let us scale up our
    concurrency
    on Travis to whatever amount of dollars of Rackspace cloud we want to
    throw at
    it.

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

    ------------------------------
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev

    --
    "I disapprove of what you say, but I will defend to the death your right
    to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint: 125F 5C67 DFE9 4084
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev



    --
    "I disapprove of what you say, but I will defend to the death your right to
    say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint: 125F 5C67 DFE9 4084
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/f43dd743/attachment.html>
  • Donald Stufft at Mar 29, 2015 at 7:51 pm
    One option for coveralls replacement is having each Jenkins job export the .coverage file as an artifact, and then have another job which consumes all of the .coverage files from those jobs and combines them. That would be Jenkins specific if we went with that solution.



    On Mar 29, 2015, at 1:02 PM, Alex Gaynor wrote:

    Sorry, I'm not following, it seems like you're assuming that "thing that replaces coveralls" wouldn't be able to work with both Travis and Jenkins?

    Alex

    On Sun, Mar 29, 2015 at 12:59 PM, Paul Kehrer <paul.l.kehrer at gmail.com wrote:
    Travis is (potentially) offering to undertake significant work on our behalf so I believe we need to have an answer to our long term coverage questions before we commit to using this solution. Being faster and having custom docker images doesn't grant us a workable coverage solution so we're going to be discussing moving entirely to jenkins again at some point in the future if we can't resolve that issue.

    -Paul
    On March 29, 2015 at 11:55:55 AM, Alex Gaynor (alex.gaynor at gmail.com wrote:

    Paul, it seems like coveralls is kind of orthagonal to this porposal, besides the fact that in the short term moving more stuff under Travis would mean coveralls included more stuff?

    Alex

    On Sun, Mar 29, 2015 at 12:53 PM, Paul Kehrer <paul.l.kehrer at gmail.com wrote:
    (Sorry for the exposition that's about to happen, just wanted to write it all down first)

    We use two different CI systems right now:

    - Travis, which provides OS X 10.9 builds as well as linux builds (against Ubuntu 12.04 with two different versions of OpenSSL).
    - Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie, Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL, Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X 10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server 2012 (64-bit Pythons).

    Travis provides us some (large) advantages over Jenkins, including not having to manage our own fleet of builders, far better (albeit less flexible) configuration through .travis.yml, and a difficult to quantify "pleasantness" to the entire experience.

    So, what problems do we run into with Travis?

    Coverage
    ------------
    At this time we use coveralls (which is now commenting up to 9 times on every PR for reasons that pass human understanding) to gain a view of our combined coverage. Unfortunately this system ties us to reporting coverage exclusively from Travis. This means code paths (like windows only code) cannot have coverage tracked.

    Speed/Reliability
    ----------------------
    We have limited (albeit very high at the moment) concurrency available from Travis. On a typical workday it can be 1-3 hours before CI completes, which significantly impairs our ability to quickly review/merge code. We have had occasions in the past where we've had Travis jobs waiting in the queue for over 8 hours due to reliability problems within the Travis infrastructure.

    Flexibility
    ------------
    Travis provides only OS X 10.9 and Ubuntu 12.04.



    The openstack builder possibility solves much of the speed issue, and custom docker images would negate the need for our jenkins linux builders (of which there are currently 7). I am, however, concerned about coverage. Without being able to run more than OS X 10.9 and (potentially) various userlands of Linux via docker images we're still significantly handicapped. It is unreasonable to expect to be able to run arbitrary OSes so we'll always run a small jenkins instance for things like alternate OS X versions, FreeBSD, etc, but the lack of Windows support is painful.

    We have recently discussed moving entirely off Travis (https://github.com/pyca/cryptography/issues/1729 <https://github.com/pyca/cryptography/issues/1729>), but if the above comes to fruition and there's a good way to combine coverage from multiple sources (and someone wants to own getting it deployed that isn't me) then I'd be happy to stay with Travis and only use jenkins for the edge cases.

    -Paul
    On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io wrote:

    I was talking to the Travis CI folks and they mentioned they have the
    possibility of using openstack builders now (or in the near future). We talked
    a little bit about the possibility of getting PyCA it's own dedicated set of
    builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.

    Currently the entirety of the PyPA organization on Github gets 10 concurrent
    builders (typically this number is 5) across all repositories. Assuming Travis
    CI is able to do it (they'd need to check with their ops team, and there would
    be some cost associated with things on their side as well) we could essentially
    control how much concurrency we want by just throwing more Rackspace VMs at
    our builds.

    Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace
    Compute1-4 which is similar we'd be looking at the ability to run ~27 of those
    VMs full time on the "standard" 2k/month free cloud offering that Rackspace
    offers OSS projects. However we'd actually be able to get more than that
    probably because Travis starts up the VMs on demand so anytime there aren't
    tests to run we won't have any VMs running.

    Independent of this, they are also working on the ability to run custom docker
    images (which means custom Linux OS images) that we can preinstall different
    software into.

    Is this something we'd want to explore? Assuming that the Travis CI ops team
    and such are OK with it, it could essentially let us scale up our concurrency
    on Travis to whatever amount of dollars of Rackspace cloud we want to throw at
    it.

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org <mailto:cryptography-dev@python.org>
    https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org <mailto:cryptography-dev@python.org>
    https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>




    --
    "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint: 125F 5C67 DFE9 4084
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org <mailto:cryptography-dev@python.org>
    https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org <mailto:cryptography-dev@python.org>
    https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev>




    --
    "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint: 125F 5C67 DFE9 4084
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev

    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/6deccf92/attachment-0001.html>
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: signature.asc
    Type: application/pgp-signature
    Size: 801 bytes
    Desc: Message signed with OpenPGP using GPGMail
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/6deccf92/attachment-0001.sig>
  • Paul Kehrer at Mar 29, 2015 at 8:30 pm
    (we had some conversation in IRC regarding this, but here is a summary of my remarks)


    I would feel terrible if we tell the Travis people we want them to do all this work, then ultimately decide we need to move off Travis because doing combined coverage there is simple. If we do go down this path then in the future we'll need to deploy a coverage service that can aggregate data from both Travis and jenkins. We've talked about that in the past (and Donald wrote https://github.com/dstufft/converge) but nobody has been willing to step up and own that to date. :( Maybe we can get some volunteers for this effort? Anybody out there want to help?


    -Paul


    On March 29, 2015 at 12:02:29 PM, Alex Gaynor (alex.gaynor at gmail.com) wrote:


    Sorry, I'm not following, it seems like you're assuming that "thing that replaces coveralls" wouldn't be able to work with both Travis and Jenkins?


    Alex


    On Sun, Mar 29, 2015 at 12:59 PM, Paul Kehrer wrote:
    Travis is (potentially) offering to undertake significant work on our behalf so I believe we need to have an answer to our long term coverage questions before we commit to using this solution. Being faster and having custom docker images doesn't grant us a workable coverage solution so we're going to be discussing moving entirely to jenkins again at some point in the future if we can't resolve that issue.


    -Paul


    On March 29, 2015 at 11:55:55 AM, Alex Gaynor (alex.gaynor at gmail.com) wrote:


    Paul, it seems like coveralls is kind of orthagonal to this porposal, besides the fact that in the short term moving more stuff under Travis would mean coveralls included more stuff?


    Alex


    On Sun, Mar 29, 2015 at 12:53 PM, Paul Kehrer wrote:
    (Sorry for the exposition that's about to happen, just wanted to write it all down first)


    We use two different CI systems right now:


    - Travis, which provides OS X 10.9 builds as well as linux builds (against Ubuntu 12.04 with two different versions of OpenSSL).
    - Jenkins, which provides CentOS 5, CentOS 6.4, CentOS 7, Debian Jessie, Debian Wheezy, FreeBSD 10.1, Ubuntu linking cryptography against LibreSSL, Ubuntu linking cryptography against OpenSSL 1.0.2{letter}, OS X 10.7, OS X 10.8, OS X 10.10, Windows Server 2008 (32-bit Pythons), and Windows Server 2012 (64-bit Pythons).


    Travis provides us some (large) advantages over Jenkins, including not having to manage our own fleet of builders, far better (albeit less flexible) configuration through .travis.yml, and a difficult to quantify "pleasantness" to the entire experience.


    So, what problems do we run into with Travis?


    Coverage
    ------------
    At this time we use coveralls (which is now commenting up to 9 times on every PR for reasons that pass human understanding) to gain a view of our combined coverage. Unfortunately this system ties us to reporting coverage exclusively from Travis. This means code paths (like windows only code) cannot have coverage tracked.


    Speed/Reliability
    ----------------------
    We have limited (albeit very high at the moment) concurrency available from Travis. On a typical workday it can be 1-3 hours before CI completes, which significantly impairs our ability to quickly review/merge code.??We have had occasions in the past where we've had Travis jobs waiting in the queue for over 8 hours due to reliability problems within the Travis infrastructure.


    Flexibility
    ------------
    Travis provides only OS X 10.9 and Ubuntu 12.04.






    The openstack builder possibility solves much of the speed issue, and custom docker images would negate the need for our jenkins linux builders (of which there are currently 7). I am, however, concerned about coverage. Without being able to run more than OS X 10.9 and (potentially) various userlands of Linux via docker images we're still significantly handicapped. It is unreasonable to expect to be able to run arbitrary OSes so we'll always run a small jenkins instance for things like alternate OS X versions, FreeBSD, etc, but the lack of Windows support is painful.


    We have recently discussed moving entirely off Travis (https://github.com/pyca/cryptography/issues/1729), but if the above comes to fruition and there's a good way to combine coverage from multiple sources (and someone wants to own getting it deployed that isn't me) then I'd be happy to stay with Travis and only use jenkins for the edge cases.


    -Paul


    On March 28, 2015 at 5:56:31 PM, Donald Stufft (donald at stufft.io) wrote:


    I was talking to the Travis CI folks and they mentioned they have the
    possibility of using openstack builders now (or in the near future). We talked
    a little bit about the possibility of getting PyCA it's own dedicated set of
    builders in Travis CI hosted inside of Rackspace on one of our cloud accounts.


    Currently the entirety of the PyPA organization on Github gets 10 concurrent
    builders (typically this number is 5) across all repositories. Assuming Travis
    CI is able to do it (they'd need to check with their ops team, and there would
    be some cost associated with things on their side as well) we could essentially
    control how much concurrency we want by just throwing more Rackspace VMs at
    our builds.


    Josh said they use a 2 vCPU machine w/ 4GB of ram, so looking at the Rackspace
    Compute1-4 which is similar we'd be looking at the ability to run ~27 of those
    VMs full time on the "standard" 2k/month free cloud offering that Rackspace
    offers OSS projects. However we'd actually be able to get more than that
    probably because Travis starts up the VMs on demand so anytime there aren't
    tests to run we won't have any VMs running.


    Independent of this, they are also working on the ability to run custom docker
    images (which means custom Linux OS images) that we can preinstall different
    software into.


    Is this something we'd want to explore? Assuming that the Travis CI ops team
    and such are OK with it, it could essentially let us scale up our concurrency
    on Travis to whatever amount of dollars of Rackspace cloud we want to throw at
    it.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev








    --
    "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint:?125F 5C67 DFE9 4084
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev


    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev








    --
    "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
    "The people's good is the highest law." -- Cicero
    GPG Key fingerprint:?125F 5C67 DFE9 4084
    _______________________________________________
    Cryptography-dev mailing list
    Cryptography-dev at python.org
    https://mail.python.org/mailman/listinfo/cryptography-dev
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/ed3ed080/attachment.html>
  • Donald Stufft at Mar 29, 2015 at 7:50 pm

    On Mar 29, 2015, at 12:53 PM, Paul Kehrer wrote:

    It is unreasonable to expect to be able to run arbitrary OSes so we'll always run a small jenkins instance for things like alternate OS X versions, FreeBSD, etc, but the lack of Windows support is painful.



    For whatever it?s worth, I introduced Travis CI to a Microsoft employee who wanted to help them get Windows support something like ~2 weeks ago or so. Windows support is on their roadmap. FreeBSD support and the like isn?t? (though if they are just spinning up Openstack Cloud images and ssh?ing into them and running commands, I wonder if it?s possible to use custom openstack images per matrix item. If it is FreeBSD would probably just work depending on what Travis expects to exist on the target host. It would probably involve us needing to manage our own images if it even did work, and I have no idea if that capability is possible or even likely.


    ---
    Donald Stufft
    PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/1e3a94e0/attachment.html>
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: signature.asc
    Type: application/pgp-signature
    Size: 801 bytes
    Desc: Message signed with OpenPGP using GPGMail
    URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20150329/1e3a94e0/attachment.sig>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcryptography-dev @
categoriespython
postedMar 28, '15 at 10:56p
activeMar 29, '15 at 8:30p
posts10
users3
websitepython.org

People

Translate

site design / logo © 2017 Grokbase