User:Mwobensmith/Security Bug Regression
Problem: We can't verify all fixed security bugs.
Reasons:
- We don't have time.
- We don't have builds that we need.
- Currently no ASan builds more than one month old
- No ASan builds for things we currently only build as release builds (release, beta, ESR)
Current scenario:
- We verify some bugs - the most critical and/or testable - and not the rest.
Best case scenario:
- Verify all fixed security bugs, both on broken and fixed builds, and ensure test case lands in a test suite.
Worst case scenario:
- No verification of anything.
Some possible solutions:
- Get more ASan builds created for branches where we previously haven't had them.
- Get ASan builds on a more frequent basis to determine regression range. (?)
- Get a bigger repository that can hold more than one month of ASan builds (even for just the ASan builds we currently support).
- Create an automated or streamlined way to land bug files into test suites, thus reducing the need for manual regression.
- As above, create a tool/system like JSBugMon for ASan bugs.
- Strict enforcement of test cases for security bugs.
- Do what Google does. (?)
Other issues:
- QA verifies broken build for a given bug, yet current automated testing for JS bugs doesn't - only fixed build. How would this work in a fully automated system?
- Related issue of old JS bugs in Bugzilla whose tests have never landed in test suites - up to 2500 of them.
- Any automated solutions need to be runnable by others, can integrate/scale as needed. Avoid specialized solutions.
- We are trying to solve issues with bug regression, but there may be related problems of tracking and finding regression range as well.