Hi!
As many of us know, we have a setup on Travis CI which automatically
runs unit tests for our active branches. This is very convenient for
evaluating pull request and status of the branches and keeping us from
making changes that break something without us noticing. However,
there's a problem with this setup, and the problem is that CI build is
never kept in the green for more than a a few days, the "normal" state
of it is always failing. I think this situation is not normal and,
frankly, embarrassing for the project, and propose to institute a policy
to change it.
The start would be recognizing that the goal is to keep the CI tests
green. Once they are green, and somebody commits a change that makes
them fail, we have the following options (note I'm not advocating for
now any of them, just enumerating them) to fix it:
1. Revert the change immediately, and ask the submitter to fix the pull
(pulls can be tested too on CI too) and re-submit it when it's green.
2. Wait for the change developer for a reasonable short time to fix the
tests, if that does not happen, revert the change.
3. Put the failing tests into XFAIL until they are fixed by somebody.
4. Have somebody - e.g. RM for the branch - to modify the failing test
to reflect the new results.
These options, of course, can be used in combination and in
case-per-case basis, but I think we need some explicit attention to the
topic. I welcome everybody to propose suggestions about other options or
ways we could fix this situation going forward and would like to hear
the ideas about how we could make Travis CI tests useful and passing
successfully, thus ensuring better quality in the project. After the
initial discussion, I plan to write an RFC summarizing the proposals and
have it voted on and, hopefully, accepted, so we'd have official policy
on it, but fir now I'd also like to raise awareness of the issue and
solicit ideas and participation of the members of the project on this topic.
Thanks,
