QA/Execution/Web Testing/Continuous Deployment: Difference between revisions

QA/Execution/Web Testing/Continuous Deployment (view source)

Revision as of 01:40, 13 July 2011

344 bytes added , 13 July 2011

→‎Risks & Plans & Tools

Rbillings

Confirmed users

1,504

edits

@@ Line 36: / Line 36: @@
 ==Risks & Plans & Tools==
-* Less critical tests do not need to be covered.
+* Risk: Less critical tests will not be covered with automation.
 ** If the feature breaks it won't cause as much of a problem. It is deemed acceptable to to wait an hour for a fix. Examples include ability to answer a question.
 * Risk: More bugs with less automation, why not automate more rather than less?
 ** Time. Creating more tests in Selenium is guaranteeing with unit tests have already verified, and it slows the process down. Right now there is a lot of duplicate effort. The team will discuss what is not currently covered by unit cases.
-* One of the major risks with continuous deployment is the change to no manual QA for each release.  The risk of releasing with bugs from new features or regressions is increased.
-** The plan
+* Risk: One of the major risks with continuous deployment is the change to no manual QA for each release.  The risk of releasing with bugs from new features or regressions is increased.
+** New feature testing is already being implemented with flags, and there should be no new issues regarding the ability to release features to designated users.
 ** Community involvement will be a key component to alerting the team to any issues which appear in production. We are currently developing plans to encourage user SUMO feedback. Examples include adding a feedback button on each page, similar to Get Satisfaction. There is a plan to bring in UX people to discuss. This will provide an active feedback system to complement the passive monitoring tool feedback.
+** There is a general increased risk with less manual testing. The compensation for that is more data measurement, faster fixes, and more time for QA to work in other areas.
-* Cannot test notifications
+* Risk: Cannot test notifications
-** We can guarantee is sent, not that it is recv'd- no answer at the moment
+** We can guarantee a notification is sent, not that it is ever received. Currently there is no answer to this problem, but it is being worked on.
 * Risk: How do you trust use of feature flags in production?
 ** We already use Waffle for Django in production. This tool is easy and flexible. It is specific to individual permissions or features. There is always a potential risk that a new permission or feature flag will touch an existing feature in an unexpected way, but up until this point that has not happened and is always taken into consideration.
-means of detection and mean time to resolve
+* Risk: Releases may have hidden bugs or regressions which affect users.
-* Risk that releases may have hidden bugs or regressions which affect users.
 ** Using Graphite as a tool. This tool will monitor usage of many common SUMO functions. It is already in production and is collecting user data. It is currently monitored closely during releases. Any significant data spikes or dips after a release will quickly indicate if there is an issue.
 ** Using StatsD as a tool. Another tool to monitor site usage by users, which may indicate issues if they exist.
-* Risk: Quality is going down, how do we change direction
+* Risk: Quality is going down, how do we change direction?
 ** We are already releasing more quickly, which allows for fixes to go out faster. If we get to a point where we are not satisfied with the quality we will weigh that against the benefits of the speed of fixes. Ultimately we do not want to go back to a slower release cycle. If continuous deployment is not working well enough, it is possible to scale back to releasing 5x or 3x a week.
-* Risk: Quality will go down over time as QA is less involved with the daily release process
+* Risk: Quality will go down over time as QA is less involved with the daily release process.
 ** There is no endpoint for looking at quality, goal is to give the best product as fast as possible. Monitoring releases will provide quality data statistics to refer to.
-* Risk: Continuous deployment risks * Continuous deployment will evolve to no longer need IT during releases. The risk is that deployment may not go well and IT may not be available to assist with a fix.
+* Risk: Continuous deployment will evolve to no longer need IT during releases. The risk is that deployment may not go well and IT may not be available to assist with a fix.
 ** The worst case scenario is deploying incorrectly. A lot of work is being done to automate the deployment process with IT. This can happen now, but currently IT is present during each release. In the future we will no longer be able to roll back releases. Patches can be added, or fixes- but only by adding code not by undoing a database change. Can redeploy within an hour, which is comparable to what exists now. This change will happen after IT is confident in the new process.
-* Risk: We don't have a current baseline for quality
+* Risk: We don't have a current baseline for quality.
 ** We are gathering data all the time with new services. It is really difficult to measure the current level of regression and new bugs which are sent out in releases. The entire team is keeping an eye on quality in order to keep the levels up. In order to maintain current levels of quality we will keep an eye on regressions and feedback.
 ==Assessment==