CA/Lessons Learned: Difference between revisions

From MozillaWiki
< CA
Jump to navigation Jump to search
m (Minor)
(→‎Policy and Practice Failures: Organized with headings)
 
(9 intermediate revisions by the same user not shown)
Line 3: Line 3:
This wiki page is a work in progress, and we invite suggestions from the Mozilla community on how it can be improved.
This wiki page is a work in progress, and we invite suggestions from the Mozilla community on how it can be improved.


== Certificate Misissuance ==
{| class="wikitable"
{| class="wikitable"
! colspan="3" style="text-align:left;" | Incorrect Certificate Profiles and Misconfigured Certificates
! colspan="3" style="text-align:left;" | Incorrect Certificate Profiles and Misconfigured Certificates
Line 18: Line 19:
(see below - certificates issued with profiles not adhering to requirements, certificates with incorrect Subject attribute order, incorrect key usages, etc.)
(see below - certificates issued with profiles not adhering to requirements, certificates with incorrect Subject attribute order, incorrect key usages, etc.)
|  
|  
| Use standardized templates that have been validated against CABF and Mozilla requirements; automate the profile validation process
| Use standardized templates that have been validated against CABF and Mozilla requirements; automate the profile validation process; archive or eliminate any "special", outdated, or rarely-used profiles
|-
|-
| Certificates containing "https" or "ldap" instead of "http" URLs
| Certificates containing "https" or "ldap" instead of "http" URLs
Line 99: Line 100:
| Summary search string: [https://bugzilla.mozilla.org/buglist.cgi?short_desc=CAA&query_format=advanced&product=CA%20Program&resolution=FIXED&component=CA%20Certificate%20Compliance&short_desc_type=substring CAA] Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1951415 1951415]
| Summary search string: [https://bugzilla.mozilla.org/buglist.cgi?short_desc=CAA&query_format=advanced&product=CA%20Program&resolution=FIXED&component=CA%20Certificate%20Compliance&short_desc_type=substring CAA] Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1951415 1951415]
| Check all domains to be contained in the certificate, automate and do not bypass CAA record checks; review and clearly understand CAA flags; keep CAA verification logic up to date; document and automate CAA validator configurations; communicate CAA checking requirements clearly to developers; train staff who configure and perform CAA checking; run CAA checks immediately before certificate issuance (to avoid TTL issues with CAA records)
| Check all domains to be contained in the certificate, automate and do not bypass CAA record checks; review and clearly understand CAA flags; keep CAA verification logic up to date; document and automate CAA validator configurations; communicate CAA checking requirements clearly to developers; train staff who configure and perform CAA checking; run CAA checks immediately before certificate issuance (to avoid TTL issues with CAA records)
|-
! colspan="3" style="text-align:left;" | S/MIME Certificate Misissuance
|-
| Email Validation Issues
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1942130 1942130], [https://bugzilla.mozilla.org/show_bug.cgi?id=1949755 1949755], [https://bugzilla.mozilla.org/show_bug.cgi?id=1920659 1920659]
| Require pre-issuance validation check to confirm that all required validations have been performed, consolidate validation logic, conduct QA/peer review of all processes, enforce correct domain validation reuse periods, avoid manual bypasses
|-
| Email Address Issues
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1906467 1906467], [https://bugzilla.mozilla.org/show_bug.cgi?id=1906470 1906470], [https://bugzilla.mozilla.org/show_bug.cgi?id=1910195 1910195], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914020 1914020]
| Streamline/normalize handling of case-sensitivity in email addresses, enforce IA5STRING compliance in SAN fields, disallow email addresses in CN unless also present in subjectAltName, enable pre-issuance linting (PKILINT) of subject fields,
|-
| Certificate Profile Issues
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1936906 1936906], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914023 1914023], [https://bugzilla.mozilla.org/show_bug.cgi?id=1929189 1929189]
| Deprecate old certificate profiles, create good documentation that supports the newer certificate profiles found in the S/MIME Bseline Requirements, perform pre-issuance linting (PKILINT)
|-
| OrgID, Country Code and Jurisdiction Issues
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1944815 1944815], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914999 1914999], [https://bugzilla.mozilla.org/show_bug.cgi?id=1917571 1917571], [https://bugzilla.mozilla.org/show_bug.cgi?id=1927506 1927506]
| Implement restrictions to prevent mismatched/invalid NTR + country code and OrgID + Country combinations, implement enhanced pre-issuance linting to catch OrgID/country alignment issues, develop training materials and require training to prevent common country code and jurisdiction misunderstandings
|-
|-
! colspan="3" style="text-align:left;" | Other Causes of Certificate Misissuance
! colspan="3" style="text-align:left;" | Other Causes of Certificate Misissuance
Line 109: Line 128:
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1909948 1909948]
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1909948 1909948]
| Review code to ensure that it only allows reuse of validation data within allowed timeframes,  
| Review code to ensure that it only allows reuse of validation data within allowed timeframes,  
|-
|
|
|
|-
|-
| Issuance to Compromised Keys
| Issuance to Compromised Keys
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1927532 1927532], [https://bugzilla.mozilla.org/show_bug.cgi?id=1931683 1931683], [https://bugzilla.mozilla.org/show_bug.cgi?id=1927384 1927384], [https://bugzilla.mozilla.org/show_bug.cgi?id=1931515 1931515]
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1927532 1927532], [https://bugzilla.mozilla.org/show_bug.cgi?id=1931683 1931683], [https://bugzilla.mozilla.org/show_bug.cgi?id=1927384 1927384], [https://bugzilla.mozilla.org/show_bug.cgi?id=1931515 1931515]
| When keyCompromise is the revocation reason, then block the key from reuse, disallow reissuance using the same CSR after revocation, maintain a hash-indexed registry of compromised public keys, screen certificate requests for disallowed key re-use, implement tooling to automatically check public keys at issuance time against lists of compromised keys, identify existing certificates with compromised keys
| When keyCompromise is the revocation reason, then block the key from reuse, disallow reissuance using the same CSR after revocation, maintain a hash-indexed registry of compromised public keys, screen certificate requests for disallowed key re-use, implement tooling to automatically check public keys at issuance time against lists of compromised keys, identify existing certificates with compromised keys
|}
== Revocation-related Issues ==
{| class="wikitable"
! colspan="3" style="text-align:left;" | Failure to Revoke and Revocation Delays
|-
|-
! colspan="3" style="text-align:left;" | Failure to Revoke and Revocation Delays
! style="width: 30%;" | Compliance Issue
! style="width: 20%;" | Bug References
! style="width: 50%;" | Corrective Measures
|-
|-
| Delayed Revocations
| Delayed Revocations
Line 124: Line 146:
| Reduced mis-issuance (see rows above), including pre-issuance linting, thorough review of certificate profiles, and improved validation; detailed changes to policies and procedures and develop and implement new tools to significantly accelerate the revocation process, including improved incident response procedures, adopt new guidelines that explicitly state that revocation delays are not allowed, even for exceptional circumstances, implement checklists and streamline approval processes, provide clearer subscriber communications; revise incident response processes to address mass revocation events; and automation and technological improvements to the infrastructure, including monitoring, auditing, and alerting, for faster detection and response to incidents requiring revocation and to quickly identify lapses in compliance with such policies and procedures
| Reduced mis-issuance (see rows above), including pre-issuance linting, thorough review of certificate profiles, and improved validation; detailed changes to policies and procedures and develop and implement new tools to significantly accelerate the revocation process, including improved incident response procedures, adopt new guidelines that explicitly state that revocation delays are not allowed, even for exceptional circumstances, implement checklists and streamline approval processes, provide clearer subscriber communications; revise incident response processes to address mass revocation events; and automation and technological improvements to the infrastructure, including monitoring, auditing, and alerting, for faster detection and response to incidents requiring revocation and to quickly identify lapses in compliance with such policies and procedures
|-
|-
! colspan="3" style="text-align:left;" | CRL and OCSP Failures
|-
| Unavailable CRLs and OCSP Service Outages, including expired domains, misconfigured alerting and syncing/uploading, and problems with error handling
| [https://bugzilla.mozilla.org/buglist.cgi?component=CA%20Certificate%20Compliance&status_whiteboard=crl%20ocsp&product=CA%20Program&short_desc_type=allwordssubstr&query_format=advanced&status_whiteboard_type=anywordssubstr&resolution=---&resolution=FIXED&short_desc=avail Whiteboard: crl OR ocsp  Summary search string: avail] 
OCSP Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1954580 1954580], [https://bugzilla.mozilla.org/show_bug.cgi?id=1957140 1957140], [https://bugzilla.mozilla.org/show_bug.cgi?id=1964866 1964866], [https://bugzilla.mozilla.org/show_bug.cgi?id=1908128 1908128], [https://bugzilla.mozilla.org/show_bug.cgi?id=1917459 1917459], [https://bugzilla.mozilla.org/show_bug.cgi?id=1931636 1931636], [https://bugzilla.mozilla.org/show_bug.cgi?id=1933353 1933353], [https://bugzilla.mozilla.org/show_bug.cgi?id=1946927 1946927]
| Deploy robust and high-availability solutions on redundant systems; publish to CDNs; increase frequency of publication and distribution; implement continuous monitoring and alerts, including for domain registration renewal--but do not rely on alerting alone; make sure that configurations and changes are carefully performed; check for any performance issues or failures following any changes, including after the application of server OS updates; clearly document procedures and processes; monitor on https://sslmate.com/labs/crl_watch/ and https://sslmate.com/labs/ocsp_watch/; make sure CRLs have the correct distinguished names and match byte-wise what is in certificates
|-
| Incorrect or Missing Revocation Reason Codes
|  Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1913310 1913310], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914365 1914365], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914383 1914383], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914419 1914419], [https://bugzilla.mozilla.org/show_bug.cgi?id=1931886 1931886], [https://bugzilla.mozilla.org/show_bug.cgi?id=1907949 1907949]
| Configure system to reject non-standard codes; perform unit testing on improved processes; provide a clearer user interface for revocation; train users on revocation reason codes; implement CRL linting
|-
| Incorrect OCSP Responses
| [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&short_desc=response&product=CA%20Program&resolution=FIXED&component=CA%20Certificate%20Compliance&short_desc_type=allwordssubstr&status_whiteboard=ocsp&query_format=advanced Whiteboard: ocsp  Summary search string: response]
| Update CA software; perform QA testing; monitor performance of internal systems; regularly check https://sslmate.com/labs/ocsp_watch/; ensure that OCSP responses are provided for pre-certificates
|-
| Expired or Invalid CRLs
| [https://bugzilla.mozilla.org/buglist.cgi?component=CA%20Certificate%20Compliance&status_whiteboard=crl&product=CA%20Program&status_whiteboard_type=allwordssubstr&resolution=FIXED&short_desc=expired%20invalid&short_desc_type=anywordssubstr&query_format=advanced Whiteboard: crl  Summary search string:expired]
| Implement automated CRL management; validate CRL profiles against CABF and root program requirements
|-
| Mismatch Between CA SubjectDN and CRL Issuer SubjectDN
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1888371 1888371]
| Implement consistency checks and ensure that CRL issuer matches CA subject byte-for-byte; monitor on https://sslmate.com/labs/crl_watch/
|-
| Other Systemic Problems with CRLs (two CAs with same CDP, CRL Not DER-Encoded, early CRL Removal)
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1949203 1949203], [https://bugzilla.mozilla.org/show_bug.cgi?id=1943379 1943379], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914893 1914893], [https://bugzilla.mozilla.org/show_bug.cgi?id=1938167 1938167], [https://bugzilla.mozilla.org/show_bug.cgi?id=1954861 1954861]
| Automate; configure/force DER formatting as default; carefully apply vendor patches/updates; implement tests and monitoring
|}
== Disclosure/Reporting Failures ==
{| class="wikitable"
! colspan="3" style="text-align:left;" | Disclosure/Reporting Failures
! colspan="3" style="text-align:left;" | Disclosure/Reporting Failures
|-
! style="width: 30%;" | Compliance Issue
! style="width: 20%;" | Bug References
! style="width: 50%;" | Corrective Measures
|-
|-
| Delayed, Incomplete, or Failed Disclosure of Intermediate CA Certificates in the CCADB
| Delayed, Incomplete, or Failed Disclosure of Intermediate CA Certificates in the CCADB
| [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard=disclosure&component=CA%20Certificate%20Compliance&status_whiteboard_type=allwordssubstr&query_format=advanced&short_desc=intermediate&product=CA%20Program&short_desc_type=allwordssubstr&resolution=FIXED Whiteboard: disclosure Summary search string: intermediate], Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1965559 1965559], [https://bugzilla.mozilla.org/show_bug.cgi?id=1921596 1921596]
| [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard=disclosure&component=CA%20Certificate%20Compliance&status_whiteboard_type=allwordssubstr&query_format=advanced&short_desc=intermediate&product=CA%20Program&short_desc_type=allwordssubstr&resolution=FIXED Whiteboard: disclosure Summary search string: intermediate], Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1965559 1965559], [https://bugzilla.mozilla.org/show_bug.cgi?id=1921596 1921596]
| Ensure coverage; provide training on CCADB tasks, especially on staff turnover; use automated tools for timely disclosure; include CCADB disclosure in key ceremony procedures; cross-reference internal databases with information in the CCADB; conduct regular audits; monitor the activity of external intermediate/subordinate CAs
| Ensure coverage; provide training on CCADB tasks, especially whenever there is staff turnover; use automated tools for timely disclosure; include CCADB disclosure in key ceremony procedures; cross-reference internal databases with information in the CCADB; conduct regular audits; monitor the activity of external intermediate/subordinate CAs
|-
|-
| Failed Disclosure of CA Owner Information  
| Failed Disclosure of CA Owner Information  
Line 149: Line 206:
| See "Delayed Responses" below
| See "Delayed Responses" below
| See "Delayed Responses" below
| See "Delayed Responses" below
|-
| Failed Incident Reporting Procedures (delays in providing 7-day status updates for compliance incidents)
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1955799 1955799], [https://bugzilla.mozilla.org/show_bug.cgi?id=1937210 1937210], [https://bugzilla.mozilla.org/show_bug.cgi?id=1957499 1957499], [https://bugzilla.mozilla.org/show_bug.cgi?id=1957474 1957474]
| CA management should prioritize the incident-reporting function within the organization, establish a certificate incident handling playbook and incident ticketing and tracking system, make sure that response deadlines are accurately calculated, invest more human resources in effort (more than a single person), require and document that incident response personnel have received regular and mandatory training, subscribe to Bugzilla CA Certificate Compliance component and check Bugzilla at least twice a week, develop an automated workflow with push notifications
|-
|-
| Failure to Respond to CA Survey
| Failure to Respond to CA Survey
| Summary search string: [https://bugzilla.mozilla.org/buglist.cgi?short_desc=survey&query_format=advanced&product=CA%20Program&resolution=FIXED&component=CA%20Certificate%20Compliance&short_desc_type=substring Survey]
| Summary search string: [https://bugzilla.mozilla.org/buglist.cgi?short_desc=survey&query_format=advanced&product=CA%20Program&resolution=FIXED&component=CA%20Certificate%20Compliance&short_desc_type=substring Survey]
| Make sure emails are received; keep CCADB updated with communication group email addresses; provide training; document procedures; prioritize responses to root programs; set deadlines in calendaring systems
| Make sure emails are received; keep CCADB updated with communication group email addresses; provide training; document procedures; prioritize responses to root programs; set deadlines in calendaring systems
|}
== Policy and Practice Failures ==
{| class="wikitable"
! colspan="3" style="text-align:left;" | CP and CPS Related Failures
|-
|-
! colspan="3" style="text-align:left;" | CRL and OCSP Failures
! style="width: 30%;" | Compliance Issue
|-
! style="width: 20%;" | Bug References
| Unavailable CRLs and OCSP Service Outages, including expired domains, misconfigured alerting and syncing/uploading, and problems with error handling
! style="width: 50%;" | Corrective Measures
| [https://bugzilla.mozilla.org/buglist.cgi?component=CA%20Certificate%20Compliance&status_whiteboard=crl%20ocsp&product=CA%20Program&short_desc_type=allwordssubstr&query_format=advanced&status_whiteboard_type=anywordssubstr&resolution=---&resolution=FIXED&short_desc=avail Whiteboard: crl OR ocsp  Summary search string: avail]
| Deploy high-availability solutions on redundant systems; publish to CDNs; increase frequency of publication and distribution; implement continuous monitoring and alerts, including for domain registration renewal; make sure that configurations and changes are carefully performed; check performance following any changes, including after the application of server OS updates; clearly document procedures and processes; monitor on https://sslmate.com/labs/crl_watch/ and https://sslmate.com/labs/ocsp_watch/; make sure CRLs have the correct distinguished names and match byte-wise what is in certificates
|-
| Incorrect or Missing Revocation Reason Codes
|  Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1913310 1913310], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914365 1914365], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914383 1914383], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914419 1914419], [https://bugzilla.mozilla.org/show_bug.cgi?id=1931886 1931886], [https://bugzilla.mozilla.org/show_bug.cgi?id=1907949 1907949]
| Configure system to reject non-standard codes; perform unit testing on improved processes; provide a clearer user interface for revocation; train users on revocation reason codes; implement CRL linting
|-
| Incorrect OCSP Responses
| [https://bugzilla.mozilla.org/buglist.cgi?status_whiteboard_type=allwordssubstr&short_desc=response&product=CA%20Program&resolution=FIXED&component=CA%20Certificate%20Compliance&short_desc_type=allwordssubstr&status_whiteboard=ocsp&query_format=advanced Whiteboard: ocsp  Summary search string: response]
| Update CA software; perform QA testing; monitor performance of internal systems; regularly check https://sslmate.com/labs/ocsp_watch/; ensure that OCSP responses are provided for pre-certificates
|-
| Expired or Invalid CRLs
| [https://bugzilla.mozilla.org/buglist.cgi?component=CA%20Certificate%20Compliance&status_whiteboard=crl&product=CA%20Program&status_whiteboard_type=allwordssubstr&resolution=FIXED&short_desc=expired%20invalid&short_desc_type=anywordssubstr&query_format=advanced Whiteboard: crl  Summary search string:expired]
| Implement automated CRL management; validate CRL profiles against CABF and root program requirements
|-
| Mismatch Between CA SubjectDN and CRL Issuer SubjectDN
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1888371 1888371]
| Implement consistency checks and ensure that CRL issuer matches CA subject byte-for-byte; monitor on https://sslmate.com/labs/crl_watch/
|-
| Other Systemic Problems with CRLs (two CAs with same CDP, CRL Not DER-Encoded, early CRL Removal)
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1949203 1949203], [https://bugzilla.mozilla.org/show_bug.cgi?id=1943379 1943379], [https://bugzilla.mozilla.org/show_bug.cgi?id=1914893 1914893], [https://bugzilla.mozilla.org/show_bug.cgi?id=1938167 1938167], [https://bugzilla.mozilla.org/show_bug.cgi?id=1954861 1954861]
| Automate; configure/force DER formatting as default; carefully apply vendor patches/updates; implement tests and monitoring
|-
! colspan="3" style="text-align:left;" | Policy and Practice Failures
|-
|-
| Failure to Publish Annual CP or CPS Updates
| Failure to Publish Annual CP or CPS Updates
Line 196: Line 229:
| [https://bugzilla.mozilla.org/buglist.cgi?component=CA%20Certificate%20Compliance&resolution=---&resolution=FIXED&query_format=advanced&status_whiteboard_type=allwordssubstr&short_desc_type=allwordssubstr&short_desc=cps%20&status_whiteboard=policy&product=CA%20Program Whiteboard: policy  Summary search string:cps]
| [https://bugzilla.mozilla.org/buglist.cgi?component=CA%20Certificate%20Compliance&resolution=---&resolution=FIXED&query_format=advanced&status_whiteboard_type=allwordssubstr&short_desc_type=allwordssubstr&short_desc=cps%20&status_whiteboard=policy&product=CA%20Program Whiteboard: policy  Summary search string:cps]
| Update CP/CPS to address CABF and root program changes to requirements (e.g. domain validation methods); implement validation processes for CP/CPS updates; conduct regular CP/CPS reviews that also checks operational behaviors (e.g. CRL issuance frequency); double-check certificate profiles published in CP/CPS; conduct peer reviews before CP/CPS publication; include CP/CPS review in operational change processes (whenever a proposed code change will introduce or change a feature)
| Update CP/CPS to address CABF and root program changes to requirements (e.g. domain validation methods); implement validation processes for CP/CPS updates; conduct regular CP/CPS reviews that also checks operational behaviors (e.g. CRL issuance frequency); double-check certificate profiles published in CP/CPS; conduct peer reviews before CP/CPS publication; include CP/CPS review in operational change processes (whenever a proposed code change will introduce or change a feature)
|-
| Practice Misalignment
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1962829 1962829], [https://bugzilla.mozilla.org/show_bug.cgi?id=1962830 1962830],
| Ensure that CPS reflects operational reality (verify CPS against implementation); sync change control with policy documentation updates; include both policy and technical staff in change reviews
|-
! colspan="3" style="text-align:left;" | Roles and Responsibilities
|-
| Gaps in responsibility and lack of role clarity
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1955721 1955721], [https://bugzilla.mozilla.org/show_bug.cgi?id=1963629 1963629]
| Always define compliance ownership in writing; assign backup personnel, and implement formal handoff procedures
|-
| Lack of formalized procedures
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1959721 1959721], [https://bugzilla.mozilla.org/show_bug.cgi?id=1935393 1935393]
| Implement shared, checklist-driven tracking for compliance tasks; cross-check with policy references; stay updated via working groups; automate alerts for awareness of compliance deadlines
|-
! colspan="3" style="text-align:left;" | Monitoring and Alerting
|-
| Failure to properly configure monitoring and alerting
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1962426 1962426], [https://bugzilla.mozilla.org/show_bug.cgi?id=1962809 1962809], [https://bugzilla.mozilla.org/show_bug.cgi?id=1947207 1947207]
| Monitor test and production infrastructure and configurations; validate monitoring configurations before putting into production; categorize and route alerts by severity; avoid single-channel alert flooding; automate test certificate renewal
|-
! colspan="3" style="text-align:left;" | Timely CPR Response and Incident Reporting
|-
|-
| Delayed Responses to Certificate Problem Reports (CPRs)
| Delayed Responses to Certificate Problem Reports (CPRs)
| [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=FIXED&component=CA%20Certificate%20Compliance&short_desc_type=allwordssubstr&short_desc=cpr&product=CA%20Program&query_format=advanced&status_whiteboard=policy&status_whiteboard_type=allwordssubstr Whiteboard label:policy-failure  Summary search string:cpr], Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1959733 1959733], [https://bugzilla.mozilla.org/show_bug.cgi?id=1963629 1963629], [https://bugzilla.mozilla.org/show_bug.cgi?id=1907568 1907568], [https://bugzilla.mozilla.org/show_bug.cgi?id=1927675 1927675], [https://bugzilla.mozilla.org/show_bug.cgi?id=1942241 1942241], [https://bugzilla.mozilla.org/show_bug.cgi?id=1942879 1942879], [https://bugzilla.mozilla.org/show_bug.cgi?id=1943528 1943528]
| [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&resolution=FIXED&component=CA%20Certificate%20Compliance&short_desc_type=allwordssubstr&short_desc=cpr&product=CA%20Program&query_format=advanced&status_whiteboard=policy&status_whiteboard_type=allwordssubstr Whiteboard label:policy-failure  Summary search string:cpr], Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1959733 1959733], [https://bugzilla.mozilla.org/show_bug.cgi?id=1963629 1963629], [https://bugzilla.mozilla.org/show_bug.cgi?id=1907568 1907568], [https://bugzilla.mozilla.org/show_bug.cgi?id=1927675 1927675], [https://bugzilla.mozilla.org/show_bug.cgi?id=1942241 1942241], [https://bugzilla.mozilla.org/show_bug.cgi?id=1942879 1942879], [https://bugzilla.mozilla.org/show_bug.cgi?id=1943528 1943528]
| Establish clear response time policies, immediately update CPR contact information in both the CPS and the CCADB, provide clear instructions, use email distribution lists, designate a sufficient number of responsible individuals to answer CPRs, allow large email attachments and white-list certain file types (.zip, .crt, .cer, .xz, .tar, .pfx, .p7b, .p7c, .p12, .der, etc.), check email filters/spam folders/server logs for intercepted emails, use a web form submission process, implement an automated ticketing system, develop an automated workflow with push notifications, programatically ensure 24-hour responses to CPRs, assign 24x7 on-duty responsibility, test the CPR process quarterly
| Establish clear response time policies, immediately update CPR contact information in both the CPS and the CCADB, provide clear instructions, use email distribution lists, designate a sufficient number of responsible individuals to answer CPRs, allow large email attachments and white-list certain file types (.zip, .crt, .cer, .xz, .tar, .pfx, .p7b, .p7c, .p12, .der, etc.), check email filters/spam folders/server logs for intercepted emails, use a web form submission process, implement an automated ticketing system, develop an automated workflow with push notifications, programmatically ensure 24-hour responses to CPRs, assign 24x7 on-duty responsibility, test the CPR process quarterly
|-
|-
| Failed Incident Reporting Procedures (delays in providing 7-day status updates for compliance incidents)
| Bug #s [https://bugzilla.mozilla.org/show_bug.cgi?id=1955799 1955799], [https://bugzilla.mozilla.org/show_bug.cgi?id=1937210 1937210], [https://bugzilla.mozilla.org/show_bug.cgi?id=1957499 1957499], [https://bugzilla.mozilla.org/show_bug.cgi?id=1957474 1957474]
| CA management should prioritize the incident-reporting function within the organization, establish a certificate incident handling playbook and incident ticketing and tracking system, make sure that response deadlines are accurately calculated, invest more human resources in effort (more than a single person), require and document that incident response personnel have received regular and mandatory training, subscribe to Bugzilla CA Certificate Compliance component and check Bugzilla at least twice a week, develop an automated workflow with push notifications
|}
== Audit Issues ==
{| class="wikitable"
! colspan="3" style="text-align:left;" | Audit Issues, Delays, and Failures
! colspan="3" style="text-align:left;" | Audit Issues, Delays, and Failures
|-
! style="width: 30%;" | Compliance Issue
! style="width: 20%;" | Bug References
! style="width: 50%;" | Corrective Measures
|-
|-
| Delayed Audit Statements
| Delayed Audit Statements
Line 218: Line 284:
| Whiteboard label: [https://bugzilla.mozilla.org/buglist.cgi?component=CA%20Certificate%20Compliance&query_format=advanced&resolution=---&resolution=FIXED&product=CA%20Program&status_whiteboard=auditor&status_whiteboard_type=allwordssubstr auditor-compliance]
| Whiteboard label: [https://bugzilla.mozilla.org/buglist.cgi?component=CA%20Certificate%20Compliance&query_format=advanced&resolution=---&resolution=FIXED&product=CA%20Program&status_whiteboard=auditor&status_whiteboard_type=allwordssubstr auditor-compliance]
| Ensure auditors are qualified and certified; review the Mozilla wikipage on [[CA/Audit_Statements#Auditor_Qualifications|Auditor Qualifications]]
| Ensure auditors are qualified and certified; review the Mozilla wikipage on [[CA/Audit_Statements#Auditor_Qualifications|Auditor Qualifications]]
|}
== Other Matters ==
{| class="wikitable"
! colspan="3" style="text-align:left;" | Test Certificates
|-
|-
! colspan="3" style="text-align:left;" | Test Certificates
! style="width: 30%;" | Compliance Issue
! style="width: 20%;" | Bug References
! style="width: 50%;" | Corrective Measures
|-
|-
| Test Website Certificates
| Test Website Certificates
| Summary search string: [https://bugzilla.mozilla.org/buglist.cgi?short_desc_type=allwordssubstr&short_desc=test%20website&product=CA%20Program&component=CA%20Certificate%20Compliance&resolution=---&resolution=FIXED&query_format=advanced test website]
| Summary search string: [https://bugzilla.mozilla.org/buglist.cgi?short_desc_type=allwordssubstr&short_desc=test%20website&product=CA%20Program&component=CA%20Certificate%20Compliance&resolution=---&resolution=FIXED&query_format=advanced test website]
| Implement certificate management tools; and calendaring for replacing the certificates used on test websites; regularly check for certificate expiration for “valid” and “revoked” certificates; provide training
| Implement certificate management tools; and calendaring and automation for replacing the certificates used on test websites; regularly check for certificate expiration for “valid” and “revoked” certificates; provide training
|-
|-
| Issuance of Test Certificates
| Issuance of Test Certificates
| Summary search string: [https://bugzilla.mozilla.org/buglist.cgi?product=CA%20Program&resolution=---&resolution=FIXED&short_desc=test%20certificate&query_format=advanced&short_desc_type=allwordssubstr&component=CA%20Certificate%20Compliance test certificate]
| Summary search string: [https://bugzilla.mozilla.org/buglist.cgi?product=CA%20Program&resolution=---&resolution=FIXED&short_desc=test%20certificate&query_format=advanced&short_desc_type=allwordssubstr&component=CA%20Certificate%20Compliance test certificate]
| Use PKI hierarchies that are not publicly trusted or complete all validation and other pre-issuance steps for the test certificate
| Use PKI hierarchies that are not publicly trusted or complete all validation and other pre-issuance steps for the test certificate
|-
! colspan="3" style="text-align:left;" | Service Outage
|-
| DNS Delegation Mistake
| Bug # [https://bugzilla.mozilla.org/show_bug.cgi?id=1958645 1958645]
| Verify DNS changes against authoritative values before submitting to registrars; never treat changes to production- and high-level zones as routine and instead implement risk-based change controls; ensure monitoring includes redundant external sources that alert persistently (do not rely on internal monitoring only — test from external perspectives); use tools that automatically diff old vs. new NS records to avoid human error
|-
| Power failure
| Bug # [https://bugzilla.mozilla.org/show_bug.cgi?id=1945409 1945409]
| Design recovery infrastructure to avoid single points of failure; maintain offline or cloud-stored copies of critical system configurations and restore data; ensure that all key services can gracefully restart under degraded conditions; perform disaster recovery drills that simulate rare but impactful failure scenarios; implement error-specific training for operations teams; validate the integrity of switch backups and recovery plans regularly; continuously update and improve escalation procedures and runbooks based on new learnings
|-
|-
! colspan="3" style="text-align:left;" | Internal Security Issues
! colspan="3" style="text-align:left;" | Internal Security Issues

Latest revision as of 21:16, 31 May 2025

Since 2014, over 1,000 incidents involving Certification Authorities (CAs) have been recorded. This page aims to collect, categorize, and analyze common compliance issues, including their underlying causes and the corrective measures that CA operators have implemented. A review of these incidents has uncovered a variety of recurring problems, ranging from certificate misissuance to reporting issues. The table below provides a high-level, categorized overview of these compliance issues and sets forth a few remediation actions that CA operators can implement to address each issue. By learning from these past mistakes and adopting these recommended practices, CA operators can enhance their compliance posture and ensure the integrity and reliability of the certificates they issue.

This wiki page is a work in progress, and we invite suggestions from the Mozilla community on how it can be improved.

Certificate Misissuance

Incorrect Certificate Profiles and Misconfigured Certificates
Compliance Issue Bug References Corrective Measures
General Issues of Non-Compliance

(e.g. certificates that do not comply with CA/B Forum requirements or Mozilla Policy)

Closely monitor changes in requirements; conduct regular audits and reviews; provide training; implement automated compliance tools
Certificate Profile Errors

(see below - certificates issued with profiles not adhering to requirements, certificates with incorrect Subject attribute order, incorrect key usages, etc.)

Use standardized templates that have been validated against CABF and Mozilla requirements; automate the profile validation process; archive or eliminate any "special", outdated, or rarely-used profiles
Certificates containing "https" or "ldap" instead of "http" URLs Bug #s 1963456, 1906690, 1916489, 1922906, 1924385 Review and compare new requirements with existing systems, code, configurations, and certificate profiles to ensure compliance; scan certificate profile configurations to ensure that the URLs for OCSP, CRL, AIAs, etc. indicate http and not https or ldap
Incorrect Certificate Policy Identifiers Bug #s 1963663, 1921597, 1921598 Ensure proper interpretation of CA/Browser Forum requirements concerning CABF Reserved Certificate Policy Identifiers (CP OIDs); accurately incorporate CP-OID requirements in certificate profiles; update CA generation procedures and certificate profiles; add compliance check prior to certificate issuance; implement an automatic linter to check conformity of intermediate CA certificates
Duplicate Serial Numbers Bug #s 1636140, 1677737, 1907667 Unique serial number generation; database checks; eliminate the potential that certificate orders remain in the issuance queue when re-starting or re-configuring CA systems; generate the final certificate immediately upon receipt of the SCTs
Insufficient Serial Number Entropy Numerous bugs Check entropy with pre-issuance linting; specify more entropy than is required; follow cryptographic best practices; keep CA software up to date; test CA software for compliance with requirements; provide developers with training on the proper calculation of entropy
Improper Key Usage Bug #s 1756122, 1647468, 1667448, 1703528 Pre-issuance linting; check keyUsage configuration in certificate profiles using automated tools; review section 7 of the Baseline Requirements; implement dual control for certificate template changes
Invalid CN/SAN Entries Bug #s 1687139, 1705187, 1716123, 1462423, 1897346 Pre-issuance linting; implement automated checks for CN and SAN matching; conduct code review and system testing
Invalid Certificate Extensions/Non-Standard Extensions Bug #s 1899466, 1876565, 1498463, 1524451 Implement strict validation processes to detect and reject non-standard extensions; stay updated on revisions to requirements; implement pre-issuance linting
Invalid OrganizationIdentifier Bug #s 1897538, 1898986, 1769240, 1900492 Write detailed specifications; conduct code review; improve training and internal communications; improve linting; update validation scheme logic; replace manual processes with automation
Overly Long Certificate Lifetimes/Validity Periods Bug #s 1826713, 1774418, 1676352 Keep certificate profile management system updated; review certificate profiles on system startup; implement pre-issuance linting; set maximum validity periods to much less than that allowed by the requirements; don’t give credits for early certificate renewals
Use of Deprecated or Incorrect Algorithms Bug #s 1648472, 1793441, 1664328 Stay up-to-date with approved algorithms listed in requirements; conduct detailed certificate profile checks, and use automation where feasible; update system logic so that it selects the correct algorithm; implement pre-issuance linting
Wildcard Mis-issuance Bug #s 1446121, 1528263, 1782391, 1731939 Block wildcards in EV certificates; ensure proper syntax and ASN.1 encoding per RFC 5280; implement pre-issuance linting
Other Certificate Profile Errors Bug #s 1946921 Conduct compliance review of certificate profile selection
Incorrect Certificate Subject Details
Invalid Organization Information Bug #s 1680083, 1674886, 1838371, 1535735, 1662382, 1705647, 1746421, 1813989, 1815527, 1828105, 1826235 Implement stringent validation and pre-issuance linting; ensure that any abbreviations used are correct; cross-check with multiple authoritative databases; sanitize internal lookup databases; ensure correct domain name registrant's organization name is placed in the certificate; prevent placement of organization information in DV certificates; test system changes; automate lookups and reduce human error; do not rely on CSRs for organization information
Incorrect Address Fields streetAddress,

locality names,

stateOrProvinceName,

postalCode,

country

Automate address validation and reduce human involvement that leads to typos, etc.; cross-check with multiple authoritative databases; prevent placement of organization information in DV certificates; sanitize internal lookup databases; do not rely on CSRs for organization information; implement tools that verify locality, state, and country combinations; do not allow the pass-through of default/filler data into the certificate; use the correct abbreviations for geographic locations
cabfOrganizationIdentifier, registrationReference and jurisdictionCountryName Issues Bug #s 1915883, 1921254 Conduct QA testing, use up-to-date lints,
Insufficient Domain Validation
Inadequate Domain Validation Summary contains "domain validation method" or "unregistered", Bug #s 1961406, 1917896, 1910322 Perform compliance review of validation logic to ensure proper implementation of approved methods; enforce correct use of approved methods; eliminate potential for human error; regularly update CP and CPS with allowed methods; use proper Random Values; do not use look-ups to external resources that are subject to attack; perform domain lookups using DNSSEC (and not outdated WHOIS information)
CAA Checking Failures and CAA-based Misissuances Summary search string: CAA Bug #s 1951415 Check all domains to be contained in the certificate, automate and do not bypass CAA record checks; review and clearly understand CAA flags; keep CAA verification logic up to date; document and automate CAA validator configurations; communicate CAA checking requirements clearly to developers; train staff who configure and perform CAA checking; run CAA checks immediately before certificate issuance (to avoid TTL issues with CAA records)
S/MIME Certificate Misissuance
Email Validation Issues Bug #s 1942130, 1949755, 1920659 Require pre-issuance validation check to confirm that all required validations have been performed, consolidate validation logic, conduct QA/peer review of all processes, enforce correct domain validation reuse periods, avoid manual bypasses
Email Address Issues Bug #s 1906467, 1906470, 1910195, 1914020 Streamline/normalize handling of case-sensitivity in email addresses, enforce IA5STRING compliance in SAN fields, disallow email addresses in CN unless also present in subjectAltName, enable pre-issuance linting (PKILINT) of subject fields,
Certificate Profile Issues Bug #s 1936906, 1914023, 1929189 Deprecate old certificate profiles, create good documentation that supports the newer certificate profiles found in the S/MIME Bseline Requirements, perform pre-issuance linting (PKILINT)
OrgID, Country Code and Jurisdiction Issues Bug #s 1944815, 1914999, 1917571, 1927506 Implement restrictions to prevent mismatched/invalid NTR + country code and OrgID + Country combinations, implement enhanced pre-issuance linting to catch OrgID/country alignment issues, develop training materials and require training to prevent common country code and jurisdiction misunderstandings
Other Causes of Certificate Misissuance
Failed Pre-Certificate/SCT Processes Bug #s 1952591, 1922844, 1949131 Ensure proper timestamps on SCTs, implement tests to verify SCTs, track pre-certificates as if they were final certificates, revoke pre-certificates that have not completed the full issuance process,
Reuse of Outdated Validation Data Bug #s 1909948 Review code to ensure that it only allows reuse of validation data within allowed timeframes,
Issuance to Compromised Keys Bug #s 1927532, 1931683, 1927384, 1931515 When keyCompromise is the revocation reason, then block the key from reuse, disallow reissuance using the same CSR after revocation, maintain a hash-indexed registry of compromised public keys, screen certificate requests for disallowed key re-use, implement tooling to automatically check public keys at issuance time against lists of compromised keys, identify existing certificates with compromised keys

Revocation-related Issues

Failure to Revoke and Revocation Delays
Compliance Issue Bug References Corrective Measures
Delayed Revocations Whiteboard search: leaf Reduced mis-issuance (see rows above), including pre-issuance linting, thorough review of certificate profiles, and improved validation; detailed changes to policies and procedures and develop and implement new tools to significantly accelerate the revocation process, including improved incident response procedures, adopt new guidelines that explicitly state that revocation delays are not allowed, even for exceptional circumstances, implement checklists and streamline approval processes, provide clearer subscriber communications; revise incident response processes to address mass revocation events; and automation and technological improvements to the infrastructure, including monitoring, auditing, and alerting, for faster detection and response to incidents requiring revocation and to quickly identify lapses in compliance with such policies and procedures
CRL and OCSP Failures
Unavailable CRLs and OCSP Service Outages, including expired domains, misconfigured alerting and syncing/uploading, and problems with error handling Whiteboard: crl OR ocsp Summary search string: avail

OCSP Bug #s 1954580, 1957140, 1964866, 1908128, 1917459, 1931636, 1933353, 1946927

Deploy robust and high-availability solutions on redundant systems; publish to CDNs; increase frequency of publication and distribution; implement continuous monitoring and alerts, including for domain registration renewal--but do not rely on alerting alone; make sure that configurations and changes are carefully performed; check for any performance issues or failures following any changes, including after the application of server OS updates; clearly document procedures and processes; monitor on https://sslmate.com/labs/crl_watch/ and https://sslmate.com/labs/ocsp_watch/; make sure CRLs have the correct distinguished names and match byte-wise what is in certificates
Incorrect or Missing Revocation Reason Codes Bug #s 1913310, 1914365, 1914383, 1914419, 1931886, 1907949 Configure system to reject non-standard codes; perform unit testing on improved processes; provide a clearer user interface for revocation; train users on revocation reason codes; implement CRL linting
Incorrect OCSP Responses Whiteboard: ocsp Summary search string: response Update CA software; perform QA testing; monitor performance of internal systems; regularly check https://sslmate.com/labs/ocsp_watch/; ensure that OCSP responses are provided for pre-certificates
Expired or Invalid CRLs Whiteboard: crl Summary search string:expired Implement automated CRL management; validate CRL profiles against CABF and root program requirements
Mismatch Between CA SubjectDN and CRL Issuer SubjectDN Bug #s 1888371 Implement consistency checks and ensure that CRL issuer matches CA subject byte-for-byte; monitor on https://sslmate.com/labs/crl_watch/
Other Systemic Problems with CRLs (two CAs with same CDP, CRL Not DER-Encoded, early CRL Removal) Bug #s 1949203, 1943379, 1914893, 1938167, 1954861 Automate; configure/force DER formatting as default; carefully apply vendor patches/updates; implement tests and monitoring

Disclosure/Reporting Failures

Disclosure/Reporting Failures
Compliance Issue Bug References Corrective Measures
Delayed, Incomplete, or Failed Disclosure of Intermediate CA Certificates in the CCADB Whiteboard: disclosure Summary search string: intermediate, Bug #s 1965559, 1921596 Ensure coverage; provide training on CCADB tasks, especially whenever there is staff turnover; use automated tools for timely disclosure; include CCADB disclosure in key ceremony procedures; cross-reference internal databases with information in the CCADB; conduct regular audits; monitor the activity of external intermediate/subordinate CAs
Failed Disclosure of CA Owner Information Bug #s 1924492 Review CCADB contact information monthly, ensure that at least two individuals are registered as Points of Contact in the CCADB,
Errors Posting Non-Audit Documents in the CCADB (CP, CPS, Annual Self Assessment, etc.) Bug #s 1956681 (Entrust), 1925106, 1942651, 1948600 Coordinate monthly with individuals/teams that have policy approval and publication authority, policy update procedures should require that the CCADB is updated every time there is a change in a non-audit document, ensure that document effective dates are less than 365 days from the previous document's effective date, ,
Failed Disclosure of CRL Distribution in the CCADB Bug #s 1964167 The CA creation process flow and checklist should include step to update the CCADB's "Pertaining to Certificates Issued by this CA" and the field "Full CRL Issued By This CA" or "JSON Array of Partitioned CRLs", regularly review the CCADB CA Task List and check for entries listed under "Root Certificates with missing Full CRL" and "Intermediate Certificates with missing Full CRL"
Failed Disclosure of CA Revocation in the CCADB Bug #s 1966006 The CA revocation process flow and checklist should include step to update the CA's "Revocation Information for This Certificate" in the CCADB
Failed Certificate Problem Report (CPR) Response See "Delayed Responses" below See "Delayed Responses" below
Failure to Respond to CA Survey Summary search string: Survey Make sure emails are received; keep CCADB updated with communication group email addresses; provide training; document procedures; prioritize responses to root programs; set deadlines in calendaring systems

Policy and Practice Failures

CP and CPS Related Failures
Compliance Issue Bug References Corrective Measures
Failure to Publish Annual CP or CPS Updates Bug #s 1565494, 1769222

Summary search string: annual cps update

Schedule regular updates; involve stakeholders in review process; ensure adequate staffing
Mistakes and erroneous information in CP or CPS Whiteboard: policy Summary search string:cps Update CP/CPS to address CABF and root program changes to requirements (e.g. domain validation methods); implement validation processes for CP/CPS updates; conduct regular CP/CPS reviews that also checks operational behaviors (e.g. CRL issuance frequency); double-check certificate profiles published in CP/CPS; conduct peer reviews before CP/CPS publication; include CP/CPS review in operational change processes (whenever a proposed code change will introduce or change a feature)
Practice Misalignment Bug #s 1962829, 1962830, Ensure that CPS reflects operational reality (verify CPS against implementation); sync change control with policy documentation updates; include both policy and technical staff in change reviews
Roles and Responsibilities
Gaps in responsibility and lack of role clarity Bug #s 1955721, 1963629 Always define compliance ownership in writing; assign backup personnel, and implement formal handoff procedures
Lack of formalized procedures Bug #s 1959721, 1935393 Implement shared, checklist-driven tracking for compliance tasks; cross-check with policy references; stay updated via working groups; automate alerts for awareness of compliance deadlines
Monitoring and Alerting
Failure to properly configure monitoring and alerting Bug #s 1962426, 1962809, 1947207 Monitor test and production infrastructure and configurations; validate monitoring configurations before putting into production; categorize and route alerts by severity; avoid single-channel alert flooding; automate test certificate renewal
Timely CPR Response and Incident Reporting
Delayed Responses to Certificate Problem Reports (CPRs) Whiteboard label:policy-failure Summary search string:cpr, Bug #s 1959733, 1963629, 1907568, 1927675, 1942241, 1942879, 1943528 Establish clear response time policies, immediately update CPR contact information in both the CPS and the CCADB, provide clear instructions, use email distribution lists, designate a sufficient number of responsible individuals to answer CPRs, allow large email attachments and white-list certain file types (.zip, .crt, .cer, .xz, .tar, .pfx, .p7b, .p7c, .p12, .der, etc.), check email filters/spam folders/server logs for intercepted emails, use a web form submission process, implement an automated ticketing system, develop an automated workflow with push notifications, programmatically ensure 24-hour responses to CPRs, assign 24x7 on-duty responsibility, test the CPR process quarterly
Failed Incident Reporting Procedures (delays in providing 7-day status updates for compliance incidents) Bug #s 1955799, 1937210, 1957499, 1957474 CA management should prioritize the incident-reporting function within the organization, establish a certificate incident handling playbook and incident ticketing and tracking system, make sure that response deadlines are accurately calculated, invest more human resources in effort (more than a single person), require and document that incident response personnel have received regular and mandatory training, subscribe to Bugzilla CA Certificate Compliance component and check Bugzilla at least twice a week, develop an automated workflow with push notifications

Audit Issues

Audit Issues, Delays, and Failures
Compliance Issue Bug References Corrective Measures
Delayed Audit Statements Whiteboard label: audit-delay Implement strict audit scheduling; use calendaring, monitoring and alerting; prioritize operational audit obligations over system enhancements; enter into engagement letters with auditors well ahead of the planned audit dates; develop contingency plans to address potential delays, disruptions, or auditor unavailability; coordinate with third parties, such as externally-operated CAs, to eliminate unanticipated dependencies; gather documentation in advance of audit, including all SHA256 hashes for CA certificates; follow up with auditor on expected delivery of audit letter; run preliminary audit letters through the CCADB's ALV process in advance to detect inconsistencies early
Audit Letter Validation Failures Summary search string: ALV Follow all guidance on the CCADB website, including https://www.ccadb.org/cas/alv
Missing CAs in Audit Letters Whiteboard label:audit-failure Summary search string:intermediate Conduct thorough and comprehensive reviews of audit scope and coverage to ensure contiguous audit coverage beginning with the auditor's key generation audit report and annual period-of-time audits; include all intermediate and cross-certified CAs “capable of issuing” the particular kind of end entity certificate covered by the audit (TLS Capable, S/MIME Capable, etc.); use the the CCADB's All CA Certificates CSV list to identify all CA certificates that are “capable” of such issuance; communicate audit letter requirements with your auditor well in advance
Auditor Qualifications Whiteboard label: auditor-compliance Ensure auditors are qualified and certified; review the Mozilla wikipage on Auditor Qualifications

Other Matters

Test Certificates
Compliance Issue Bug References Corrective Measures
Test Website Certificates Summary search string: test website Implement certificate management tools; and calendaring and automation for replacing the certificates used on test websites; regularly check for certificate expiration for “valid” and “revoked” certificates; provide training
Issuance of Test Certificates Summary search string: test certificate Use PKI hierarchies that are not publicly trusted or complete all validation and other pre-issuance steps for the test certificate
Service Outage
DNS Delegation Mistake Bug # 1958645 Verify DNS changes against authoritative values before submitting to registrars; never treat changes to production- and high-level zones as routine and instead implement risk-based change controls; ensure monitoring includes redundant external sources that alert persistently (do not rely on internal monitoring only — test from external perspectives); use tools that automatically diff old vs. new NS records to avoid human error
Power failure Bug # 1945409 Design recovery infrastructure to avoid single points of failure; maintain offline or cloud-stored copies of critical system configurations and restore data; ensure that all key services can gracefully restart under degraded conditions; perform disaster recovery drills that simulate rare but impactful failure scenarios; implement error-specific training for operations teams; validate the integrity of switch backups and recovery plans regularly; continuously update and improve escalation procedures and runbooks based on new learnings
Internal Security Issues
Logging Issues Summary search string: log Implement robust log management systems; log to a separate log server; leverage “big data” solutions; freeze operations if logging is not working; monitor logs in real-time with solutions that provide alerting and data analysis; regularly audit system logs
Improper Access Control Summary search string: access Establish and follow strict access control policies; conduct regular reviews of access control lists
Retrieved from "https://wiki.mozilla.org/index.php?title=CA/Lessons_Learned&oldid=1254266"