Security/Firefox/Security Bug Life Cycle: Difference between revisions

m
edit suggestions from Thyla
(Added pending sec-approval query)
m (edit suggestions from Thyla)
Line 1: Line 1:
= A Bug is Born =
= A Bug is Born =


Reports of security vulnerabilities come from many different sources. Many are directly filed as security bugs by various groups:
Reports of security vulnerabilities come from many different sources. Many are directly filed as security bugs by various groups including:
* Our security teams (e.g. fuzzing, static analysis, security reviews and audits)
* Our security teams (via fuzzing, static analysis, security reviews and audits)
* External security researchers (including bounty hunters)
* External security researchers (including bounty hunters)
* Engineers developing, reviewing, or testing notice vulnerabilities as they work on non-security bugs
* Engineers developing, reviewing, or testing may notice vulnerabilities as they work on non-security bugs
* QA and others looking at raw crashes on Socorro
* QA and others looking at raw crashes on Socorro
* Users noticing something that worries them
* Users noticing something that worries them




Some issues are found outside the Mozilla community. The security team or other Mozilla community members file bugs for these issues when they come to our attention:
Some issues are found outside of the Mozilla community. The security team or other Mozilla community members file bugs for these issues when they come to our attention via:
* Concerns or incidents mailed to security@mozilla.org
* Concerns or incidents mailed to security@mozilla.org
* Blogs and social media of known security researchers
* Blogs and social media of known security researchers
Line 21: Line 21:
== Incoming ==
== Incoming ==


The main goal at this stage is to get security bugs rated appropriately and into the purview of the engineers who manage that area of code. Only a limited number of people can see these security bugs so we need to ensure that includes the right people. For each bug:
The main goal at this stage is to get security bugs rated appropriately and into the purview of the engineers who manage the relevant areas of code. Only a limited number of people can see security bugs by default so we need to ensure that each bug is in the right security group and CC additional people as necessary. When triaging, consider the following for each bug:
* Is the bug well formed and reproducible?
* Is the bug well formed and reproducible?
** If it is make sure it’s NEW rather than UNCONFIRMED.
** If it is make sure it’s NEW rather than UNCONFIRMED.
** If not, “needinfo?” the reporter until it is or the bug is closed (potentially as INCOMPLETE or WORKSFORME).
** If not, “needinfo?” the reporter until it is or the bug is closed (potentially as INCOMPLETE or WORKSFORME).
* Is it in the right Product and Component?
* Is it in the right Product and Component?
* Is it in the right security group for the component (''especially'' if it’s in the “Core” product)? See [https://securitywiki.mozilla.org/Client_Security_Teams#Development_Teams Security teams and components]
* Is it in the right security group for the component (''especially'' if it’s in the “Core” product)? See [https://securitywiki.mozilla.org/Client_Security_Teams#Development_Teams Security teams and components] for a mapping between security groups and components.
* Are appropriate developers CC’d so they can see the bug and needinfo'd so they are aware of it?
* Are the appropriate developers CC’d so they can see the bug and needinfo'd so they are aware of it?
* If you can't select an appropriate [https://wiki.mozilla.org/Security_Severity_Ratings security severity rating] needinfo? someone who can (either a senior security team member or a senior engineer in the appropriate component)
* If you can't select an appropriate [https://wiki.mozilla.org/Security_Severity_Ratings security severity rating], needinfo? someone for help. This is typically either a senior security team member or a senior engineer in that area of the code.




Line 42: Line 42:
* Set the appropriate version status flags to “affected”
* Set the appropriate version status flags to “affected”
* Set the version tracking flags to “+”
* Set the version tracking flags to “+”
* Assign to an appropriate owner. If there’s no better person use the Triage Owner
* Assign to an appropriate owner (if there’s no better person use the Triage Owner)




Line 51: Line 51:
== Administrivia ==
== Administrivia ==


Once a fix lands the security group on that bug should be changed to the “Release-track” group (core-security-release) so QA folks can see and verify the bugs.<br>
Once a fix lands the security group on that bug should be changed to the “Release-track” group (core-security-release) so that QA can see and verify the bugs.
 
 
[https://bugzilla.mozilla.org/buglist.cgi?quicksearch=FIX%20group%3Acore-security%20-group%3Acore-security-release '''Fixed security bugs that need to be moved to "release track"''']
[https://bugzilla.mozilla.org/buglist.cgi?quicksearch=FIX%20group%3Acore-security%20-group%3Acore-security-release '''Fixed security bugs that need to be moved to "release track"''']


Line 64: Line 66:
== Fixing Vulnerabilities ==
== Fixing Vulnerabilities ==


Severe security bugs need to be fixed with deliberate speed. Some external reporters even have a 60-day deadline before they report the issue publicly.
Severe security bugs need to be fixed with deliberate speed to protect our users. In addition, some external reporters have a disclosure deadline such as 60 or 90 days before they report the issue publicly.
* Within three days the assignee of the bug should comment on its status, to acknowledge receipt of the bug and to give a generalized ETA of a patch for planning. Even if the ETA is “can’t look at it until after bugs X,Y, and Z” even that much is helpful for planning and, if necessary, finding a different assignee.
* Within three days the assignee of the bug should comment on its status, to acknowledge receipt of the bug and to give a generalized ETA of a patch for planning. Even if the ETA is “can’t look at it until after bugs X,Y, and Z” even that much is helpful for planning and, if necessary, finding a different assignee.
* Sec-critical bugs are highest priority and should be fixed within two weeks. If that can’t be accomplished because of other priorities check with the security team and your manager to resolve the conflict.
* Sec-critical bugs are highest priority and should be fixed within two weeks. If that can’t be accomplished because of other priorities check with the security team and your manager to resolve the conflict.
* Sec-high bugs should be fixed within a few weeks: 6 weeks maximum is a good goal. 60 days is a common disclosure deadline, and in addition to writing the patch we have to account for time spent on QA and the release process as a whole.
* Sec-high bugs should be fixed within a few weeks: 6 weeks maximum is a good goal. 60 days is a common disclosure deadline, and in addition to writing the patch, we have to account for time spent on QA and the release process as a whole.




Line 77: Line 79:
== Landing Fixes ==
== Landing Fixes ==


We know people watch our check-ins and we don’t want to 0-day ourselves by landing obvious fixes and test cases that demonstrate how to trigger the vulnerability. The [https://wiki.mozilla.org/Security/Bug_Approval_Process '''Security Bug Approval Process'''] is designed to prevent that. Part of the approval process is evaluating what bugs need to be pushed to beta and which are risky and need to ride the trains, and whether the patch is needed on supported ESR branches.
We know people watch our check-ins and we don’t want to 0-day ourselves by landing obvious fixes and test cases that demonstrate how to trigger the vulnerability. The [https://wiki.mozilla.org/Security/Bug_Approval_Process '''Security Bug Approval Process'''] is designed to prevent that. Part of the approval process is evaluating what bugs need to be pushed to Beta and which are risky and need to ride the trains, and whether or not the patch is needed on supported ESR branches.


[https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20sec-approval%3F '''Pending sec-approval requests''']
[https://bugzilla.mozilla.org/buglist.cgi?quicksearch=ALL%20sec-approval%3F '''Pending sec-approval requests''']
Line 83: Line 85:


== Verifying Fixes ==
== Verifying Fixes ==
It's generally important to have bug fixes tested by fresh eyes, who might catch problems with incorrect assumptions made by the original fix. This is especially important for security fixes because we announce these fixes in our advisories: if the fix doesn't work we have put people at risk. Verification is especially important when we uplift/back-port patches to the Beta or, worse, the ESR branches since they're more likely to suffer from subtle dependencies on code changes from normal trunk development that weren't back-ported to those branches.
It's generally important to have bug fixes tested by fresh eyes who might catch problems with incorrect assumptions made by the original fix. This is especially important for security fixes because we announce these fixes in our advisories: if the fix doesn't work we put people at risk. Verification is especially important when we uplift/back-port patches to the Beta or, worse, the ESR branches since they're more likely to suffer from subtle dependencies on code changes from normal trunk development that weren't back-ported to those branches.


The QA team's process for verifying security bugs for release is described in the [https://securitywiki.mozilla.org/PostCritSmash '''“Post CritSmash”'''] document.
The QA team's process for verifying security bugs for release is described in the [https://securitywiki.mozilla.org/PostCritSmash '''“Post CritSmash”'''] document.
Line 90: Line 92:
== ESR ==  
== ESR ==  
We have committed to supporting Extended Support Release (ESR) branches for roughly a year each with a two release overlap between ESR branches. “Support” primarily means security fixes. Security bugs labeled sec-critical or sec-high are automatic candidates for back-porting. Some less-severe security bugs are also included after evaluating their impact, risk, and visibility. See the  
We have committed to supporting Extended Support Release (ESR) branches for roughly a year each with a two release overlap between ESR branches. “Support” primarily means security fixes. Security bugs labeled sec-critical or sec-high are automatic candidates for back-porting. Some less-severe security bugs are also included after evaluating their impact, risk, and visibility. See the  
[https://wiki.mozilla.org/Release_Management/ESR_Landing_Process '''ESR landing process'''] page where there are additional triage queries.
[https://wiki.mozilla.org/Release_Management/ESR_Landing_Process '''ESR landing process'''] page for additional release-management triage queries.




== Security Advisories ==  
== Security Advisories ==  
The fixed bugs that had been present in a shipped release need to have a CVE assigned and be written up in our release advisories. Security fixes for recent regressions that only affected Nightly or Beta don’t need an advisory. [link to Al's doc]
The fixed bugs that had been present in a shipped release need to have a CVE assigned and to be written up in our release advisories. Security fixes for recent regressions that only affected Nightly or Beta don’t need an advisory. [link to Al's doc]


For historical write-ups see our  
For historical write-ups see our  
Line 102: Line 104:
== The Pit of Despair ==
== The Pit of Despair ==


Sometimes we can't make much progress on finding and fixing a security bug, especially if we don't have a reliable way to reproduce it. This is a particular problem with crashes filed from crash-stats evidence. They are real bugs, they may even happen fairly often, and the crash stacks show memory corruption that is likely exploitable if it can be triggered reliably. These are worth filing and treating as security vulnerabilities because we do manage to fix a significant number of them when we investigate, but sadly many others are so generic and often crashing far after the actual cause of corruption that we can't make progress. These bugs are given the keyword "stalled" and removed from active work. There are sometimes ways to make further progress (e.g. diagnostic asserts might be added to narrow down theories about what is going wrong). If there's no longer any hope or ideas for further progress, many times these are eventually going to have to be closed as INCOMPLETE.
Sometimes we can't make much progress on finding and fixing a security bug, especially if we don't have a reliable way to reproduce it. This is a particular problem with crashes filed from crash-stats reports. They are real bugs, they may even happen fairly often, and the crash stacks show memory corruption that is likely exploitable if it can be triggered reliably. These should be filed and treated as security vulnerabilities because we do manage to fix a significant number of them when we investigate. However, many others are generic and the crash is detected so far after the actual cause of corruption that we can't make progress. These bugs are given the keyword "stalled" and removed from active work. There are sometimes ways to make further progress (e.g. diagnostic asserts might be added to narrow down theories about what is going wrong), but once all ideas are exhausted and there is no longer any hope for further progress many of these bugs are eventually going to have to be closed as INCOMPLETE.




Line 109: Line 111:
= Triage tools =
= Triage tools =


The [https://addons.mozilla.org/en-US/firefox/addon/open-selected-links/ Open Selected Links] extension can be helpful for opening multiple bugs at once from a buglist during triage. It also has a "View link source" context menu item that can be useful for inspecting testcases and what-not.
The [https://addons.mozilla.org/en-US/firefox/addon/open-selected-links/ Open Selected Links] extension can be helpful for opening multiple bugs at once from a buglist during triage. It also has a "View link source" context menu item that can be useful for inspecting testcases.


The "Bug Age" bookmarklet can be run on any buglist for basic age stats. Find it at [https://gist.github.com/dveditz/47b1558ec29e0651b9ac4bab4d02538c this gist].
The "Bug Age" bookmarklet can be run on any buglist for basic age stats. Find it at [https://gist.github.com/dveditz/47b1558ec29e0651b9ac4bab4d02538c this gist].
canmove, Confirmed users
640

edits