Short version for those who dont want to read the story below: After
upgrading spree, what is your pre deployment testing procedure to ensure
the upgrade hasn't created some unwanted side effects / bugs?

Hi Guys, last week one of our customers spotted a bug in their spree app.
We searched github and found a fix had been committed to the latest version
so we just needed to upgrade spree to acquire it. Our customer was eager
for the fix to be deployed so we tested that the fix was working, had a
general sweep over the app to see everything else was ok and rushed a
deployment out of the door. This was our mistake as our customer then
encountered a collection of more serious bugs in production; creating
duplicate payments and stopping payments being made. We have fixed these
bugs and it is all working now, however, we don't want to be in this
situation again, if we can help it. We have talked about how we can improve
our post upgrade testing procedure, but I thought I would open the
discussion to see how everyone else does it, hoping you have better ideas
than us.

Thanks,
Rob

--
You received this message because you are subscribed to the Google Groups "Spree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spree-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Search Discussions

  • Nate Lowrie at Feb 27, 2013 at 1:45 pm
    Rob,

    I don't upgrade real lightly. Even just upgrading to the latest commit on
    a particular branch has issues. Here's my general testing procedure:

    1. Run the upgrade for the gem you want to upgrade.
    2. Run the spree tests and the test for all installed extensions and
    make sure they pass.
    3. I have a list of critical user activities like creating an order that
    we absolutely must test by hand to verify nothing is broken.
    1. If something is broken, we put a test in for it and fix the issue.
    4. We don't generally hit all parts of the application so occasionally
    we might have a styling issue or a minor bug that needs to be worked out.
    It takes too much time to do a manual run through minor items.

    The tests we have in place actually catch most all of the issues we
    encounter.

    Regards,

    Nate
    On Wednesday, February 27, 2013 4:38:47 AM UTC-5, Robert Oles wrote:

    Short version for those who dont want to read the story below: After
    upgrading spree, what is your pre deployment testing procedure to ensure
    the upgrade hasn't created some unwanted side effects / bugs?

    Hi Guys, last week one of our customers spotted a bug in their spree app.
    We searched github and found a fix had been committed to the latest version
    so we just needed to upgrade spree to acquire it. Our customer was eager
    for the fix to be deployed so we tested that the fix was working, had a
    general sweep over the app to see everything else was ok and rushed a
    deployment out of the door. This was our mistake as our customer then
    encountered a collection of more serious bugs in production; creating
    duplicate payments and stopping payments being made. We have fixed these
    bugs and it is all working now, however, we don't want to be in this
    situation again, if we can help it. We have talked about how we can improve
    our post upgrade testing procedure, but I thought I would open the
    discussion to see how everyone else does it, hoping you have better ideas
    than us.

    Thanks,
    Rob
    --
    You received this message because you are subscribed to the Google Groups "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to spree-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Calvin Yu at Feb 27, 2013 at 3:50 pm
    We always test the checkout flow manually on upgrade, but the thing that's
    helped me the most is that I spent a good amount of time upfront setting
    request specs with capybara that tests out the core flow (adding product to
    cart, checkout, etc). This has been really effective for identifying
    issues with upgrades.

    On Wed, Feb 27, 2013 at 8:45 AM, Nate Lowrie wrote:

    Rob,

    I don't upgrade real lightly. Even just upgrading to the latest commit on
    a particular branch has issues. Here's my general testing procedure:

    1. Run the upgrade for the gem you want to upgrade.
    2. Run the spree tests and the test for all installed extensions and
    make sure they pass.
    3. I have a list of critical user activities like creating an order
    that we absolutely must test by hand to verify nothing is broken.
    1. If something is broken, we put a test in for it and fix the
    issue.
    4. We don't generally hit all parts of the application so occasionally
    we might have a styling issue or a minor bug that needs to be worked out.
    It takes too much time to do a manual run through minor items.

    The tests we have in place actually catch most all of the issues we
    encounter.

    Regards,

    Nate
    On Wednesday, February 27, 2013 4:38:47 AM UTC-5, Robert Oles wrote:

    Short version for those who dont want to read the story below: After
    upgrading spree, what is your pre deployment testing procedure to ensure
    the upgrade hasn't created some unwanted side effects / bugs?

    Hi Guys, last week one of our customers spotted a bug in their spree app.
    We searched github and found a fix had been committed to the latest version
    so we just needed to upgrade spree to acquire it. Our customer was eager
    for the fix to be deployed so we tested that the fix was working, had a
    general sweep over the app to see everything else was ok and rushed a
    deployment out of the door. This was our mistake as our customer then
    encountered a collection of more serious bugs in production; creating
    duplicate payments and stopping payments being made. We have fixed these
    bugs and it is all working now, however, we don't want to be in this
    situation again, if we can help it. We have talked about how we can improve
    our post upgrade testing procedure, but I thought I would open the
    discussion to see how everyone else does it, hoping you have better ideas
    than us.

    Thanks,
    Rob
    --
    You received this message because you are subscribed to the Google Groups
    "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to spree-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to spree-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Robert Oles at Mar 1, 2013 at 7:37 am
    I like the idea of a checklist to follow, I need to get that in place.

    As far as capybara is concerned, are those tests rolled in to spree, or are
    they something you have written specifically for your app. If they are
    specific, do you think there is scope to add something similar to spree?
    On Wednesday, February 27, 2013 3:49:58 PM UTC, Calvin wrote:

    We always test the checkout flow manually on upgrade, but the thing that's
    helped me the most is that I spent a good amount of time upfront setting
    request specs with capybara that tests out the core flow (adding product to
    cart, checkout, etc). This has been really effective for identifying
    issues with upgrades.


    On Wed, Feb 27, 2013 at 8:45 AM, Nate Lowrie <na...@finelineautomation.com<javascript:>
    wrote:
    Rob,

    I don't upgrade real lightly. Even just upgrading to the latest commit
    on a particular branch has issues. Here's my general testing procedure:

    1. Run the upgrade for the gem you want to upgrade.
    2. Run the spree tests and the test for all installed extensions and
    make sure they pass.
    3. I have a list of critical user activities like creating an order
    that we absolutely must test by hand to verify nothing is broken.
    1. If something is broken, we put a test in for it and fix the
    issue.
    4. We don't generally hit all parts of the application so
    occasionally we might have a styling issue or a minor bug that needs to be
    worked out. It takes too much time to do a manual run through minor items.

    The tests we have in place actually catch most all of the issues we
    encounter.

    Regards,

    Nate
    On Wednesday, February 27, 2013 4:38:47 AM UTC-5, Robert Oles wrote:

    Short version for those who dont want to read the story below: After
    upgrading spree, what is your pre deployment testing procedure to ensure
    the upgrade hasn't created some unwanted side effects / bugs?

    Hi Guys, last week one of our customers spotted a bug in their spree
    app. We searched github and found a fix had been committed to the latest
    version so we just needed to upgrade spree to acquire it. Our customer was
    eager for the fix to be deployed so we tested that the fix was working, had
    a general sweep over the app to see everything else was ok and rushed a
    deployment out of the door. This was our mistake as our customer then
    encountered a collection of more serious bugs in production; creating
    duplicate payments and stopping payments being made. We have fixed these
    bugs and it is all working now, however, we don't want to be in this
    situation again, if we can help it. We have talked about how we can improve
    our post upgrade testing procedure, but I thought I would open the
    discussion to see how everyone else does it, hoping you have better ideas
    than us.

    Thanks,
    Rob
    --
    You received this message because you are subscribed to the Google Groups
    "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to spree-user+...@googlegroups.com <javascript:>.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to spree-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Calvin Yu at Mar 1, 2013 at 12:24 pm
    They are specific to our app based on the the customizations we made on top
    of spree. I'm not sure what you have in mind, but none of my spec
    themselves are ones that should make it into spree b/c of the
    customizations. Spree could possibly make this easier through
    documentation, more helper code, and better test factories, but still a
    large portion of the responsibility will be on our end.

    On Fri, Mar 1, 2013 at 2:37 AM, Robert Oles wrote:

    I like the idea of a checklist to follow, I need to get that in place.

    As far as capybara is concerned, are those tests rolled in to spree, or
    are they something you have written specifically for your app. If they are
    specific, do you think there is scope to add something similar to spree?

    On Wednesday, February 27, 2013 3:49:58 PM UTC, Calvin wrote:

    We always test the checkout flow manually on upgrade, but the thing
    that's helped me the most is that I spent a good amount of time upfront
    setting request specs with capybara that tests out the core flow (adding
    product to cart, checkout, etc). This has been really effective for
    identifying issues with upgrades.


    On Wed, Feb 27, 2013 at 8:45 AM, Nate Lowrie <
    na...@finelineautomation.com> wrote:
    Rob,

    I don't upgrade real lightly. Even just upgrading to the latest commit
    on a particular branch has issues. Here's my general testing procedure:

    1. Run the upgrade for the gem you want to upgrade.
    2. Run the spree tests and the test for all installed extensions and
    make sure they pass.
    3. I have a list of critical user activities like creating an order
    that we absolutely must test by hand to verify nothing is broken.
    1. If something is broken, we put a test in for it and fix the
    issue.
    4. We don't generally hit all parts of the application so
    occasionally we might have a styling issue or a minor bug that needs to be
    worked out. It takes too much time to do a manual run through minor items.

    The tests we have in place actually catch most all of the issues we
    encounter.

    Regards,

    Nate
    On Wednesday, February 27, 2013 4:38:47 AM UTC-5, Robert Oles wrote:

    Short version for those who dont want to read the story below: After
    upgrading spree, what is your pre deployment testing procedure to ensure
    the upgrade hasn't created some unwanted side effects / bugs?

    Hi Guys, last week one of our customers spotted a bug in their spree
    app. We searched github and found a fix had been committed to the latest
    version so we just needed to upgrade spree to acquire it. Our customer was
    eager for the fix to be deployed so we tested that the fix was working, had
    a general sweep over the app to see everything else was ok and rushed a
    deployment out of the door. This was our mistake as our customer then
    encountered a collection of more serious bugs in production; creating
    duplicate payments and stopping payments being made. We have fixed these
    bugs and it is all working now, however, we don't want to be in this
    situation again, if we can help it. We have talked about how we can improve
    our post upgrade testing procedure, but I thought I would open the
    discussion to see how everyone else does it, hoping you have better ideas
    than us.

    Thanks,
    Rob
    --
    You received this message because you are subscribed to the Google
    Groups "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to spree-user+...@**googlegroups.com.

    For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
    .

    --
    You received this message because you are subscribed to the Google Groups
    "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to spree-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to spree-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.
  • Michael Sevestre at Mar 1, 2013 at 2:00 pm
    I second Nate on that. We also pulled the tests out of spree core in our
    apps to ensure that everything works as expected
    Cheers,
    Michael
    On Fri, Mar 1, 2013 at 2:45 PM, Nate Lowrie wrote:

    Robert,

    What we've done is pulled the request tests out of spree core and into our
    app. We then modified and added tests for our small amount of
    customization. It took 1/2 a day to do and has been well worth it. Couple
    that with your checklist for manual testing and you should be pretty good.

    When we deploy, we always re-run our manual check operations to verify the
    deploy went as planned.

    Regards,

    Nate

    On Friday, March 1, 2013 2:37:30 AM UTC-5, Robert Oles wrote:

    I like the idea of a checklist to follow, I need to get that in place.

    As far as capybara is concerned, are those tests rolled in to spree, or
    are they something you have written specifically for your app. If they are
    specific, do you think there is scope to add something similar to spree?
    On Wednesday, February 27, 2013 3:49:58 PM UTC, Calvin wrote:

    We always test the checkout flow manually on upgrade, but the thing
    that's helped me the most is that I spent a good amount of time upfront
    setting request specs with capybara that tests out the core flow (adding
    product to cart, checkout, etc). This has been really effective for
    identifying issues with upgrades.


    On Wed, Feb 27, 2013 at 8:45 AM, Nate Lowrie <
    na...@finelineautomation.com> wrote:
    Rob,

    I don't upgrade real lightly. Even just upgrading to the latest commit
    on a particular branch has issues. Here's my general testing procedure:

    1. Run the upgrade for the gem you want to upgrade.
    2. Run the spree tests and the test for all installed extensions
    and make sure they pass.
    3. I have a list of critical user activities like creating an order
    that we absolutely must test by hand to verify nothing is broken.
    1. If something is broken, we put a test in for it and fix the
    issue.
    4. We don't generally hit all parts of the application so
    occasionally we might have a styling issue or a minor bug that needs to be
    worked out. It takes too much time to do a manual run through minor items.

    The tests we have in place actually catch most all of the issues we
    encounter.

    Regards,

    Nate
    On Wednesday, February 27, 2013 4:38:47 AM UTC-5, Robert Oles wrote:

    Short version for those who dont want to read the story below: After
    upgrading spree, what is your pre deployment testing procedure to ensure
    the upgrade hasn't created some unwanted side effects / bugs?

    Hi Guys, last week one of our customers spotted a bug in their spree
    app. We searched github and found a fix had been committed to the latest
    version so we just needed to upgrade spree to acquire it. Our customer was
    eager for the fix to be deployed so we tested that the fix was working, had
    a general sweep over the app to see everything else was ok and rushed a
    deployment out of the door. This was our mistake as our customer then
    encountered a collection of more serious bugs in production; creating
    duplicate payments and stopping payments being made. We have fixed these
    bugs and it is all working now, however, we don't want to be in this
    situation again, if we can help it. We have talked about how we can improve
    our post upgrade testing procedure, but I thought I would open the
    discussion to see how everyone else does it, hoping you have better ideas
    than us.

    Thanks,
    Rob
    --
    You received this message because you are subscribed to the Google
    Groups "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send
    an email to spree-user+...@googlegroups.**com.
    For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
    .

    --
    You received this message because you are subscribed to the Google Groups
    "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to spree-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

    --
    You received this message because you are subscribed to the Google Groups "Spree" group.
    To unsubscribe from this group and stop receiving emails from it, send an email to spree-user+unsubscribe@googlegroups.com.
    For more options, visit https://groups.google.com/groups/opt_out.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupspree-user @
categoriesrubyonrails
postedFeb 27, '13 at 10:15a
activeMar 1, '13 at 2:00p
posts6
users4
websitespreecommerce.com
irc#RubyOnRails

People

Translate

site design / logo © 2022 Grokbase