Confirmed users
1,867
edits
Websites/Affiliates/Test Plan: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
* | =Test Plan= | ||
==Analysis== | |||
===Currently in use=== | |||
*dev: | |||
**nose | |||
**unittest | |||
**webtrends | |||
**other metrics collected manually on a weekly basis | |||
*qa | |||
**pytest | |||
**unittestzero | |||
**Selenium | |||
**Manual | |||
***fuzzing | |||
***Netsparker | |||
===Planned for Q1=== | |||
*dev | |||
**QUnit | |||
**Statsd/Graphite | |||
*qa | |||
**Automated fuzzing | |||
**Garmr | |||
===Not planned=== | |||
*load testing | |||
Risks, plans, and tools | |||
Risk: Less critical tests will not be covered with automation. | |||
If the feature breaks it won't cause as much of a problem. It is deemed acceptable to to wait an hour for a fix. | |||
Risk: More bugs with less automation, why not automate more rather than less? | |||
Time. Creating more tests in Selenium is guaranteeing with unit tests have already verified, and it slows the process down. Right now there is a lot of duplicate effort. The team will discuss what is not currently covered by unit cases. | |||
Risk: Releases may have hidden bugs or regressions which affect users. | |||
Using Graphite as a tool. This tool will monitor usage of many common Affiliates functions. Any significant data spikes or dips after a release will quickly indicate if there is an issue. | |||
Using StatsD as a tool. Another tool to monitor site usage by users, which may indicate issues if they exist. | |||
Risk: Quality is going down, how do we change direction? | |||
We are already releasing very quickly, which allows for fixes to go out faster. If we get to a point where we are not satisfied with the quality we will weigh that against the benefits of the speed of fixes. | |||
Risk: Quality will go down over time as QA is less involved with the release process. | |||
There is no endpoint for looking at quality, goal is to give the best product as fast as possible. Monitoring releases will provide quality data statistics to refer to. | |||
Risk: We don't have a current baseline for quality. | |||
We are gathering data all the time with new services. It is really difficult to measure the current level of regression and new bugs which are sent out in releases. The entire team is keeping an eye on quality in order to keep the levels up. In order to maintain current levels of quality we will keep an eye on regressions and feedback. | |||
Risk areas: | |||
1. Registration | |||
1. Banner links work - CDN | |||
1. Password recovery | |||
1. BrowserID | |||
2. Making Banners | |||
2. Profile editing | |||
2. My banners | |||
3. Email newsletter registration | |||
3. Leaderboard | |||
3. Calendar | |||