FAQ
Hi all,
flaky tests are often really hard to make not flaky, so I've got the idea
to just retry failed tests several times automatically and only then report
the failure.
I was not the only one of course: there's nice initiative in face of
https://github.com/box/flaky
The big downside of that plugin's approach, is that only 'call' phase is
retried, while 'setup' is not re-executed and so all function-scoped
fixtures stay as is from previous try.
This is not acceptable if you use fixtures intensively of course.
I created an issue where I have a snippet how it could work
https://github.com/box/flaky/issues/53
But then realized that it doesn't work properly still: when it's the only
test collected which we're going to retry, all fixtures, including
session-scoped are being resetup as well!
Then looked more and couldn't find any good solutions to actually very
simple question: how do i rerun the test in the same test run?
Any help is appreciated. I think making that plugin work or even making it
available in the core would be a great improvement.
Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20150809/a8ae15fc/attachment.html>

Search Discussions

  • Eduardo Schettino at Aug 9, 2015 at 6:14 am
    Hi,


    I will just throw out another idea that I had about handling flaky tests.


    I think it would be nice to mark a test as "flaky" and than treat it
    differently if it fails.


    For example, once a test is marked as flaky you could choose to ignore
    flaky test failures and maybe report them
    as an "expect failure" or some other custom status. Or still consider it as
    failure if run on a "strict mode".


    regards,
       Eduardo


    On Sun, Aug 9, 2015 at 8:54 AM, Anatoly Bubenkov wrote:

    Hi all,
    flaky tests are often really hard to make not flaky, so I've got the idea
    to just retry failed tests several times automatically and only then report
    the failure.
    I was not the only one of course: there's nice initiative in face of
    https://github.com/box/flaky
    The big downside of that plugin's approach, is that only 'call' phase is
    retried, while 'setup' is not re-executed and so all function-scoped
    fixtures stay as is from previous try.
    This is not acceptable if you use fixtures intensively of course.
    I created an issue where I have a snippet how it could work
    https://github.com/box/flaky/issues/53
    But then realized that it doesn't work properly still: when it's the only
    test collected which we're going to retry, all fixtures, including
    session-scoped are being resetup as well!
    Then looked more and couldn't find any good solutions to actually very
    simple question: how do i rerun the test in the same test run?
    Any help is appreciated. I think making that plugin work or even making it
    available in the core would be a great improvement.
    Thanks!

    _______________________________________________
    pytest-dev mailing list
    pytest-dev at python.org
    https://mail.python.org/mailman/listinfo/pytest-dev
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20150809/d5a076b7/attachment.html>
  • Florian Bruhin at Aug 9, 2015 at 8:57 am

    * Eduardo Schettino [2015-08-09 14:14:09 +0800]:
    I will just throw out another idea that I had about handling flaky tests.

    I think it would be nice to mark a test as "flaky" and than treat it
    differently if it fails.

    For example, once a test is marked as flaky you could choose to ignore
    flaky test failures and maybe report them
    as an "expect failure" or some other custom status. Or still consider it as
    failure if run on a "strict mode".

    Also see this issue which proposes a separate marker for flaky tests:


    https://github.com/pytest-dev/pytest/issues/814


    Florian


    --
    http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
        GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
              I love long mails! | http://email.is-not-s.ms/
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: not available
    Type: application/pgp-signature
    Size: 819 bytes
    Desc: not available
    URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20150809/5c50ad98/attachment.sig>
  • Florian Bruhin at Aug 9, 2015 at 8:56 am

    * Anatoly Bubenkov [2015-08-09 00:54:20 +0000]:
    The big downside of that plugin's approach, is that only 'call' phase is
    retried, while 'setup' is not re-executed and so all function-scoped
    fixtures stay as is from previous try.
    This is not acceptable if you use fixtures intensively of course.
    I created an issue where I have a snippet how it could work
    https://github.com/box/flaky/issues/53
    But then realized that it doesn't work properly still: when it's the only
    test collected which we're going to retry, all fixtures, including
    session-scoped are being resetup as well!
    Then looked more and couldn't find any good solutions to actually very
    simple question: how do i rerun the test in the same test run?
    Any help is appreciated. I think making that plugin work or even making it
    available in the core would be a great improvement.
    Thanks!

    This sounds like another possible usecase for this issue:


    https://github.com/pytest-dev/pytest/issues/916


    Mind adding a comment there explaining your usecase?


    Florian


    --
    http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
        GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
              I love long mails! | http://email.is-not-s.ms/
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: not available
    Type: application/pgp-signature
    Size: 819 bytes
    Desc: not available
    URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20150809/056d0ea8/attachment.sig>

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
grouppytest-dev @
categoriespython
postedAug 9, '15 at 12:54a
activeAug 9, '15 at 8:57a
posts4
users3
websitepython.org

People

Translate

site design / logo © 2018 Grokbase