Confirmed users
1,867
edits
Websites/Affiliates/Test Plan: Difference between revisions
Jump to navigation
Jump to search
→Risks, plans, and tools
| Line 27: | Line 27: | ||
===Risks, plans, and tools=== | ===Risks, plans, and tools=== | ||
1. Risk: Less critical tests will not be covered with automation. | |||
*If the feature breaks it won't cause as much of a problem. It is deemed acceptable to to wait an hour for a fix. | *If the feature breaks it won't cause as much of a problem. It is deemed acceptable to to wait an hour for a fix. | ||
2. Risk: More bugs with less automation, why not automate more rather than less? | |||
*Time. Creating more tests in Selenium is guaranteeing with unit tests have already verified, and it slows the process down. Right now there is a lot of duplicate effort. The team will discuss what is not currently covered by unit cases. | *Time. Creating more tests in Selenium is guaranteeing with unit tests have already verified, and it slows the process down. Right now there is a lot of duplicate effort. The team will discuss what is not currently covered by unit cases. | ||
3. Risk: Releases may have hidden bugs or regressions which affect users. | |||
*Using Graphite as a tool. This tool will monitor usage of many common Affiliates functions. Any significant data spikes or dips after a release will quickly indicate if there is an issue. | *Using Graphite as a tool. This tool will monitor usage of many common Affiliates functions. Any significant data spikes or dips after a release will quickly indicate if there is an issue. | ||
*Using StatsD as a tool. Another tool to monitor site usage by users, which may indicate issues if they exist. | *Using StatsD as a tool. Another tool to monitor site usage by users, which may indicate issues if they exist. | ||
4. Risk: Quality is going down, how do we change direction? | |||
*We are already releasing very quickly, which allows for fixes to go out faster. If we get to a point where we are not satisfied with the quality we will weigh that against the benefits of the speed of fixes. | *We are already releasing very quickly, which allows for fixes to go out faster. If we get to a point where we are not satisfied with the quality we will weigh that against the benefits of the speed of fixes. | ||
5. Risk: Quality will go down over time as QA is less involved with the release process. | |||
*There is no endpoint for looking at quality, goal is to give the best product as fast as possible. Monitoring releases will provide quality data statistics to refer to. | *There is no endpoint for looking at quality, goal is to give the best product as fast as possible. Monitoring releases will provide quality data statistics to refer to. | ||
6. Risk: We don't have a current baseline for quality. | |||
*We are gathering data all the time with new services. It is really difficult to measure the current level of regression and new bugs which are sent out in releases. The entire team is keeping an eye on quality in order to keep the levels up. In order to maintain current levels of quality we will keep an eye on regressions and feedback. | *We are gathering data all the time with new services. It is really difficult to measure the current level of regression and new bugs which are sent out in releases. The entire team is keeping an eye on quality in order to keep the levels up. In order to maintain current levels of quality we will keep an eye on regressions and feedback. | ||