From MozillaWiki
Jump to: navigation, search

Firefox Operations Security

Firefox Operations Security protects the core infrastructure Mozilla relies on to build, ship and run Firefox.



Email us at

To report a security issue on a given site, use the bug bounty form as explained here.

To tell us about a new service create a New Service issue.


1. Improve operational security of the core infrastructure

1.1 Implement Test Driven Security (TDS) in CI/CD

Security tests should be part of the continuous integration (CI) and continuous delivery (CD) pipelines.

  • CI integration should be part of the code commit/review process, either in an existing CI (travis-ci, circleci, taskcluster) or in a security dedicated one. CI tests should include static code analysis and recommendations, docker containers testing and dependency checks (vulnerability management).
  • CD integration should be done at Jenkins' level, when stage environments are built and promoted.All services are regularly rebuilt by Jenkins. CD tests should include application vulnerability scanning (ZAProxy) and infrastructure access control tests (security groups, IAM permissions, ...).

TDS should output directly in the build pipeline at first, and allow dev & ops to control levels that block integration & delivery. In a second phase, TDS outputs should be aggregated into a central security tracking platform.

1.2 Make use of the logging pipeline to detect fraud and anomalies

Heka, ElasticSearch and Kafka are powerful tools on top of which we can plug various pattern detection mechanisms to identify known bad actors, or unusual behavior. Fraud detection is a highly requested feature that devs don’t want to rebuild every time. Fraud detection should operate autonomously for each service, taking into account business rules set by the developers and the security team.

1.3 Improve user management and authentication

We should make better use of LDAP to add and remove employees from various third party services and admin panels.

  • Admin panels should rely on Mozilla's Identity Management platform provided by IT
  • Third-party services (datadog, pagerduty, aws) should have automated user management (userplex).

secops need to facilitate integration with Mozilla's IAM with standard libraries and tools.

1.4 Harden the infrastructure

All services and tools that are part of the standard infrastructure should undergo security hardening. Hardening rules should be testable in the CD pipeline (see TDS above) to prevent security regressions. Some examples:

  • SSH should enforce MFA authentication
  • Disabled users should be removed from all systems, particularly bastion hosts
  • AWS permissions must prevent services from compromising each other
  • Secrets must be provisioned encrypted
  • ...

2. Increase security maturity

2.1 Help new projects identify threats and controls (RRA, threat models,...)

Risk assessment and threat modeling help people think through failure scenarios they wouldn’t evaluate otherwise. RRAs often leads to architectural changes that are best identified early. Each new project must undergo a 30/60min RRA with one of the member of secops to assess the security posture of the project.

2.2 Implement baseline services security standards

Content Security Policy (CSP), HSTS, HPKP, data signature and encryption, input validation, XSS and SQLi protection are part of techniques developers should care about when building new services. secops defines services security standards that devs can implement and tests in TDS.

2.3 Communicate security effectively throughout the organization

Teams need a channel to ask security questions, discuss concerns and share techniques. secops must organize information flow and broadcast to developers, ops and managers. This includes general security best practices, analyzis and actions to take on CVE vulnerabilities, response and communication on incidents.

2.4 Use Mozilla’s bug bounty program

The bug bounty program is a fantastic tool: for a small amount of money, we reward people worldwide for helping us improve our security posture. Most security issues identified in our services come from the bug bounty program. We must ensure that all services are part of the bug bounty program and that triaging is performed regularly. As much as possible, we must assist developers in fixing security issues that are reported through bug bounties.

3. Build core security services

3.1 Sign data that changes the configuration of user agents

We iterate fast, and eventually someone, either us or a partner, is bound to make a mistake and open a door that could put our users at risk. Signing the data we send to our users helps cover that risk. Digital signature for Firefox is a complex topic - not every project can implement it independently - so secops must provide the tooling and services to facilitate signing ([autograph](

3.2 Monitor our ecosystem for external threats

There are many scenarios in which our users can be at risk because of the fraudulent or careless behavior of a third party. A bad certificate authority could issue a certificate that impersonates us. A careless partner could leak addon signing keys. A web startup could get hacked and leak web push endpoints. We should implement the tools needed to identify fraudulent behavior outside of our organization that impact us, so we can react in a timely manner and protect Firefox users.

3.3 Partner with external firms to monitor our security

We can’t do everything ourselves. External security firms can help us keep an eye on and audit our services. Some of their work may be redundant with current efforts, such as automated security testing, but would help cover the interim. We should evaluate various vendors and partner with the ones that have the best support of our technologies.

Security Checklist

The checklist below is in MARKDOWN format to be copy/pasted into Github issues.

Risk Management
* [ ] The service must have performed a Rapid Risk Assessment and have a Risk Record bug
* [ ] The service must be registered via a [New Service issue](


* [ ] Access and application logs must be archived for a minimum of 90 days
* [ ] Use [Modern]( or [Intermediate]( TLS
* [ ] Set HSTS to 31536000 (1 year)
  * `strict-transport-security: max-age=31536000`
  * [ ] If the service is not hosted under ``, it must be manually added to [Firefox's preloaded pins]( This only applies to production services, not short-lived experiments.
* If service has an admin panels, it must:
  * [ ] only be available behind Mozilla VPN (which provides MFA)
  * [ ] require Auth0 authentication

* [ ] Ensure your code repository is configured and located appropriately:
  * [ ] Application built internally should be hosted in trusted GitHub organizations (mozilla, mozilla-services, mozilla-bteam, mozilla-conduit, mozilla-mobile, taskcluster). Sometimes we build and deploy applications we don't fully control. In those cases, the Dockerfile that builds the application container should be hosted in its own repository in a trusted organization.
  * [ ] Secure your repository by implementing [Mozilla's GitHub security standard](
* [ ] Sign all release tags, and ideally commits as well
  * Developers should [configure git to sign all tags]( and upload their PGP fingerprint to
  * The signature verification will eventually become a requirement to shipping a release to staging & prod: the tag being deployed in the pipeline must have a matching tag in git signed by a project owner. This control is designed to reduce the risk of a 3rd party GitHub integration from compromising our source code.
* [ ] enable security scanning of 3rd-party libraries and dependencies
  * For node.js, use [`npm audit`]( with [audit-filter]( to review and handle exceptions (see example in [speech-proxy](
  * For Python, enable pyup security updates:
    * Add a pyup config to your repo (example config:
    * Enable branch protection for master and other development branches. Make sure the approved-mozilla-pyup-configuration team *CANNOT* push to those branches.
    * From the "add a team" dropdown for your repo /settings page 
      * Add the "Approved Mozilla PyUp Configuration" team for your github org (e.g. for [mozilla]( and [mozilla-services](
      * Grant it write permission so it can make pull requests
    * notify to enable the integration in pyup
* [ ] Keep 3rd-party libraries up to date (in addition to the security updates)
  * For NodeJS applications, use [dependabot](, [renovate](, or [GreenKeeper](
  * For Python, use ``pip list --outdated`` or []( or pyup outdated checks
  * For Rust, use `cargo update` and [cargo upgrade]( when changing versions
* [ ] Integrate static code analysis in CI, and avoid merging code with issues
    * Javascript applications should use ESLint with the [Mozilla ruleset](
    * Python applications should use [Bandit](
    * Go applications should use the [Go Meta Linter](
    * Use whitelisting mechanisms in these tools to deal with false positives

Dual Sign Off
* [ ] Services that push data to Firefox clients must require a dual sign off on every change, implemented in their admin panels
  * This mechanism must be reviewed and approved by the Firefox Operations Security team before being enabled in production

* [ ] Publish detailed logs in [mozlog]( format (**APP-MOZLOG**)
  * Business logic must be logged with app specific codes (see [FxA](
  * Access control failures must be logged at WARN level

Web Applications
* [ ] Must have a CSP with
  * [ ] a report-uri pointing to the service's own `/__cspreport__` endpoint
  * [ ] web API responses should return `default-src 'none'; frame-ancestors 'none'; base-uri 'none'; report-uri /__cspreport__` to disallowing all content rendering, framing, and report violations
  * [ ] if default-src is not `none`, frame-src, and object-src should be `none` or only allow specific origins
  * [ ] no use of unsafe-inline or unsafe-eval in script-src, style-src, and img-src
* [ ] Third-party javascript must be pinned to specific versions using [Subresource Integrity (SRI)](
* [ ] Web APIs must set a non-HTML content-type on all responses, including 300s, 400s and 500s
* [ ] Set the Secure and HTTPOnly flags on [Cookies](, and use sensible Expiration
* [ ] Make sure your application gets an A+ on the [Mozilla Observatory](
* [ ] Verify your application doesn't have any failures on the [Security Baseline](
  * Contact secops@ or ping 'psiinon' on github to document exceptions to the baseline, mark csrf exempt forms, etc.
* [ ] Web APIs should export an OpenAPI (Swagger) to facilitate automated vulnerability tests

Security Features
* [ ] Authentication of end-users should be via FxA. Authentication of Mozillians should be via Auth0/SSO. Any exceptions must be approved by the security team.
* [ ] Session Management should be via existing and well regarded frameworks. In all cases you should contact the security team for a design and implementation review
  * Store session keys server side (typically in a db) so that they can be revoked immediately.
  * Session keys must be changed on login to prevent session fixation attacks.
  * Session cookies must have HttpOnly and Secure flags set and the SameSite attribute set to 'strict' or 'lax' (which allows external regular links to login).
  * For more information about potential pitfalls see the [OWASP Session Management Cheat Sheet](
* [ ] Form that change state should use anti CSRF tokens. Anti CSRF tokens can be dropped for internal sites using SameSite session cookies where we are sure all users will be on Firefox 60+. Forms that do not change state (e.g. search forms) should use the 'data-no-csrf' form attribute.
* [ ] Access Control should be via existing and well regarded frameworks. If you really do need to roll your own then contact the security team for a design and implementation review.
* [ ] If you are building a core Firefox service, consider adding it to the list of restricted domains in the preference `extensions.webextensions.restrictedDomains`. This will prevent a malicious extension from being able to steal sensitive information from it, see [bug 1415644](

* [ ] All SQL queries must be parameterized, not concatenated
* [ ] Applications must use accounts with limited GRANTS when connecting to databases
  * In particular, applications **must not use admin or owner accounts**, to decrease the impact of a sql injection vulnerability.

Common issues
* [ ] User data must be [escaped for the right context]( prior to reflecting it
  * When inserting user generated html into an html context:
    * Python applications should use [Bleach](
    * Javascript applications should use [DOMPurify](
* [ ] Apply sensible limits to user inputs, see [input validation](
  * POST body size should be small (<500kB) unless explicitly needed
* [ ] When allowing users to upload or generate content, make sure to host that content on a separate domain (eg., etc.). This will prevent malicious content from having access to storage and cookies from the origin.
  * Also use this technique to host rich content you can't protect with a CSP, such as metrics reports, wiki pages, etc.
* [ ] When managing permissions, make sure access controls are enforced server-side
* [ ] If an authenticated user accesses protected resource, make sure the pages with those resource arent cached and served up to unauthenticated users (like via a CDN).
* [ ] If handling cryptographic keys, must have a mechanism to handle quarterly key rotations
  * Keys used to sign sessions don't need a rotation mechanism if destroying all sessions is acceptable in case of emergency.
* [ ] Do not proxy requests from users without strong limitations and filtering (see [Pocket UserData vulnerability]( Don't proxy requests to [link local, loopback, or private networks]( or DNS that resolves to addresses in those ranges (i.e.,,,,,,
* [ ] Do not use `target="_blank"` in external links unless you also use `rel="noopener noreferrer"` (to prevent [Reverse Tabnabbing](

The Firefox Operations Security logo is derived from this work by Synth Agency, and published under Creative Commons Attribution-NonCommercial 4.0 International Public License.