CA/Entrust Issues
This page lists recent (March-May 2024) bugs involving Entrust. The list of issues might not be comprehensive, and it will be updated by Mozilla as more information becomes available, but please do not edit this page yourself. If you have proposed changes, post them to the Mozilla dev-security-policy list or email them to certificates@mozilla.org.
Summary of Entrust Incidents - March-May 2024
The incidents listed below in this section A are related to Bug #1883843.
1. EV TLS Certificate cPSuri missing -
https://bugzilla.mozilla.org/show_bug.cgi?id=1883843
After an alert from Ryan Dickson, Entrust discovered that all of its EV certificates issued since the implementation of changes due to Ballot SC-62v2 (approximately 26,668 certificates) were missing the CPS URI, in violation of the EV Guidelines. Entrust said this was due to discrepancies/misinterpretations between the CA/Browser Forum’s TLS Baseline Requirements and the Extended Validation Guidelines. Entrust initially chose to not stop issuing the EV certificates and did not begin the process of revoking the certificates. They argued that: the absence of the CPS URI in the EV certificates was due to ambiguities in CA/B Forum requirements; the absence of the CPS URI had no security impact; and halting and revoking the certificates would negatively impact customers and the broader WebPKI ecosystem. Entrust then also proposed a ballot to adjust the EV Guidelines to not require the CPS URI. It also argued that its efforts were better spent focusing on improving automation and handling of certificates, rather than on revocation and reissuance.
Entrust’s decisions met with substantial backlash from the Mozilla community and other stakeholders, who argued that Entrust's failure to comply with established rules and guidelines and its reluctance to take immediate corrective actions were unacceptable. They emphasized the need for adherence to standards and prompt remediation and revocation as per standard incident response protocols.
Ultimately, Entrust acknowledged the necessity to stop issuance and to revoke the affected certificates in response to the community’s concerns. It admitted that its initial response did not align with community expectations, and it committed to improving its incident handling, communication, and operational procedures. Entrust also highlighted its efforts to enhance system processes for detecting and addressing errors more effectively and to improve automation and resilience among its subscribers for better management of similar incidents in the future. It has recognized the need to address guideline discrepancies more responsibly and now views the experience as a learning opportunity to improve its approach to certificate issuance and incident management.
Issues: Misinterpretation of Requirements; Policy/Procedure Failure; Certificate Mis-issuance; Early Detection; Incident Response; Incident Handling; Delayed Revocation; Poor Communication
2. Failed to provide a preliminary incident report according to TLS BR 4.9.5 -
https://bugzilla.mozilla.org/show_bug.cgi?id=1890123
Related to bug #1883843, directly above. Entrust failed to provide a preliminary incident report as required by the TLS Baseline Requirements, section 4.9.5, after receiving a certificate problem report (CPR). (Entrust confirmed the issue in just under 26 hours and posted its preliminary report about 43 hours after the initial notification.) The delay was attributed to a lack of specific procedural steps and reminders in the CPR flow and a gap in Entrust's incident handling procedures. To prevent similar issues, Entrust is improving its incident management procedures, by June 30, 2024, and will be incorporating specific steps and reminders to ensure timely reporting of future incidents.
Issues: Incident Handling; Incident Response
3. CPR was not responded to in 24 hours -
https://bugzilla.mozilla.org/show_bug.cgi?id=1885754
This is related to bug #1883843 (discussed above). Entrust did not respond to a Certificate Problem Report (CPR) for its AffirmTrust PKI hierarchy within 24 hours. Entrust said that it was already aware of the issue due to a previous incident report concerning the related issue for the Entrust PKI hierarchy, but that the CPR reporter was unaware of the receipt of and action on that CPR. Root cause was that a Technical Support Specialist did not recognize CPR as such, leading to the delayed response. CPR reporting processes needed to be improved. Action items included re-training for Technical Support, as well as implementation of automation for flagging and responding to CPRs and creation of a CPR form for reporters to complete.
Issues: Incident Handling; Incident Response
4. Delayed revocation of EV TLS certificates with missing cPSuri -
https://bugzilla.mozilla.org/show_bug.cgi?id=1886532
All affected certificates should have been revoked within 5 days. Entrust asserted that delayed revocation was due to confusion about TLS EV profile changes and various customer-related factors, as discussed above in bug #1883843. Weekly updates are being provided. However, as of May 1, 2024, approximately 14,736 certificates of the 26,668 affected certificates had been revoked (55.26%).
Issues: Delayed Revocation
5. EV Certificate missing Issuer’s EV Policy OID -
https://bugzilla.mozilla.org/show_bug.cgi?id=1888714
Entrust issued 1,963 EV TLS certificates September 11-22, 2023, without including an EV TLS CP OID. Root Causes were the misinterpretation of the EV Guidelines and the TLS BRs and a failure to recognize the overriding requirements of the EV Guidelines. (A misinterpretation of standards led to non-compliant certificates, and linting failed to detect the issue.) Entrust also failed to provide its list of affected certificates or its incident report by a promised date, and did not give an explanation for that delay.
As remediation, since April 11, 2024, Entrust has used pkilint as a post-issuance linter to detect similar issues. (Mis-issued certificates are a subset of the certificates disclosed and being revoked under bug #1883843. Status of revocation is listed in bug #1886532.)
Issues: Misinterpretation of Requirements; Policy/Procedure Failure; Certificate Mis-issuance; Incident Handling; Incident Response
6. Delay in Updating CPS -
https://bugzilla.mozilla.org/show_bug.cgi?id=1887753
There was an oversight during an update related to Entrust’s EV TLS certificates, see bug #1883843. The incident involved the improper removal and later reinstatement of the Entrust EV TLS Certificate Policy (CP) Object Identifier (OID) without updating the CPS accordingly.
Entrust removed the EV TLS CP OID from certificates on September 11, 2023, to align with the Baseline Requirements, which caused some browsers not to display these certificates as EV certificates. It then added the CP OID back into the certificates on September 22, 2023, but the CPS was not immediately updated to reflect this change. The CPS was finally updated on March 22, 2024, until then, certificates were technically compliant with TLS Baseline Requirements, but not with Entrust's own CPS.
The root cause was that procedures for changing certificate profiles did not include a step to update the CPS concurrently. Entrust learned that there needs to be a mechanism to flag changes affecting compliance documentation to ensure it is updated alongside any policy changes. Action is being taken to review and update release and emergency release processes to better synchronize with compliance documentation. (This incident underscores the importance of accurate documentation and the need for procedures that ensure CPS updates are consistent with other changes in certificate policies and profiles.)
Issues: Misinterpretation of Requirements; Policy/Procedure
7. Failure to revoke EV TLS certificates issued before CPS update -
https://bugzilla.mozilla.org/show_bug.cgi?id=1890685
This incident covers a subset of certificates mentioned above in bug #1887753 that were issued between March 18-21, 2024. The certificates did not comply with the Entrust CPS published at the time, although they conformed to the actual intended certificate profile and EV Guidelines. Entrust’s decision not to revoke contrasts with actions taken for certificates issued prior to March 18, 2024, which were revoked and reissued due to a different issue (bug #1883843) that necessitated a profile change.
Issues: Delayed Revocation
B. Certificates without serverAuth EKU and Delayed Revocation
1. clientAuth TLS Certificates without serverAuth EKU -
https://bugzilla.mozilla.org/show_bug.cgi?id=1886467
Entrust issued 1,176 TLS certificates with id-kp-clientAuth EKU, but not id-kp-serverAuth. Root cause was a system that allowed customers to configure the EKU. One “lesson learned” was that Entrust needs to review user-configurable options and requirements more thoroughly. It was noted that linters did not detect the issue, highlighting the need for improved automated detection tools. Action items included fixing EKU checking in lint tools and reviewing all certificate profiles for potential errors.
Issues: Certificate Mis-issuance; Implementation/Configuration Error
2. Delayed revocation of clientAuth TLS Certificates without serverAuth EKU -
https://bugzilla.mozilla.org/show_bug.cgi?id=1887705
Delayed revocation bug filed regarding bug #1886467, above. Weekly updates are being provided. Entrust said that the delayed revocation was due mainly to the high workload already as a result of incident described in bug #1883843 (discussed above), certificate profile changes, customer workload, and system lockdowns for end-of-quarter / holiday issues. As of May 1, 2024, 310 certificates of 1,176 affected certificates had been revoked (26.36%).
Issues: Delayed Revocation
C. Policy-Procedure Failure: CPS
1. CPS typographical (text placement) error -
https://bugzilla.mozilla.org/show_bug.cgi?id=1890896
From March 22-26, 2024, Entrust issued 6,008 OV TLS certificates under a CPS requirement related to the policyQualifier in the OV certificate profile. The error was corrected in a subsequent CPS. (The certificates were compliant with the CA/B Forum’s TLS BRs and Entrust's OV profile.) Root-cause factors included generic naming of certificate profiles and concurrent updates to the CPS during incidents requiring rapid certificate replacement and revocation. Remediation actions included errata to the affected CPS, informing subscribers of CPS error, reviewing CPS update procedure, and renaming certificate profile for clarity. (Community members questioned Entrust’s decision to not revoke the certificates, and Entrust responded regarding its position on such matters--Entrust maintains that the issue was addressed by adding an erratum in the CPS and informing its subscribers and that it can handle mass revocation and reissuance.)
Issues: Policy/Procedure Failure
2. Delayed incident report - CPS typographical (text placement) error (Closed) -
https://bugzilla.mozilla.org/show_bug.cgi?id=1890901
This is related to bug #1890896, above. Entrust delayed in filing an incident report (an initial report should be filed within 72 hours of the CA being made aware of an incident). Entrust explained that priorities of other incidents delayed the reporting of the incident in bug #1890896.
Issues: Incident Handling; Incident Response
3. Failure to revoke OV TLS - CPS typographical (text placement) error -
https://bugzilla.mozilla.org/show_bug.cgi?id=1890898
This is related to bug #1890896 above. The error and correction involved the CPS, not the certificates themselves, and re-issuing would result in similar certificates with new issuance dates. The error was discovered and corrected on March 26 with the posting of CPS version 3.20. However, community members have raised concerns about Entrust’s commitment to compliance with Baseline Requirements, its assertion of exceptional conditions, and the deviation from revocation timelines set forth in the Baseline Requirements.
Issues: Delayed Revocation
4. Not updating Problem Reporting Mechanism fields in CCADB -
https://bugzilla.mozilla.org/show_bug.cgi?id=1894111
Entrust's entry in the Problem Reporting Mechanism field of the Common CA Database (CCADB) was inconsistent with its CPS and outdated information needed to be updated. Entrust updated this information in the CCADB.
Issues: Policy/Procedure Failure
D. OCSP and CRL Issues
1. OCSP response signed with SHA-1 -
https://bugzilla.mozilla.org/show_bug.cgi?id=1879602
On February 3, 2024, Entrust became aware that the SHA-1 algorithm was being used when signing OCSP responses. (This was because the default setting for online OCSP responders was to sign OCSP responses using the same algorithm as the CA certificate, which was SHA-1.) By February 6, a fix was applied to the production system so that OCSP signatures would be SHA-256. Entrust has enhanced its OCSP monitoring system to verify correct signing algorithms and incorporated OCSP Watch into its daily monitoring routines. This incident underlines the importance of diligent monitoring and updating of cryptographic practices to adhere to industry standards.
Issues: Implementation/Configuration Error
2. CRL non-conformance with the TLS BRs (Closed) -
https://bugzilla.mozilla.org/show_bug.cgi?id=1889217
Two CRLs listed in the CCADB did not meet the TLS BRs and RFC 5280 because they contained the "revoked certificates" field, despite having no revoked certificates. The issue was detected using pkilint. Entrust updated CRL generation software and all affected CRLs were reissued. The root cause was identified as the use of a non-compliant function (CreateCRL) in its Go Crypto/x509 library. As remediation, Entrust is to conduct regular review and validation of dependencies, including forked packages, deploy linting tools and automated tests for compliance, implement continuous monitoring and review practices, such as CRL Watch and OCSP Watch. Also, Entrust to develop a plan for periodic review of forked libraries and their updates and to enhance processes for tracking and analyzing changes in requirements.
Issues: Implementation/Configuration Error
E. Issues in Recent History
1. Invalid data in State/Province Field -
https://bugzilla.mozilla.org/show_bug.cgi?id=1658792
It was initially discovered that Entrust had issued 395 OV SSL certificates to a large international organization with “NA” for the state/province information. Entrust worked on a drop-down list to prevent the error. Certificate revocation would not occur within established timeframes, so Bug #1658794 for delayed revocation was opened.
Issues: Certificate Mis-issuance
2. Late Revocation for Invalid State/Province Issue -
https://bugzilla.mozilla.org/show_bug.cgi?id=1658794
This is the delayed revocation bug related to Bug #1658792, above. Entrust said that when educating large institutions about rapid revocation, factors include who owns a certificate, where it is deployed, and the type of system or application that requires the certificate. It also said that it was advocating automation with such institutions to help speed up certificate replacement and to minimize human error.
Issues: Delayed Revocation
3. EV TLS Certificate incorrect jurisdiction -
https://bugzilla.mozilla.org/show_bug.cgi?id=1802916
Entrust mis-issued 322 EV certificates with the wrong state and locality jurisdiction fields due to complex data entry processes. Entrust implemented a different automated dropdown system for jurisdiction selection. Certificate revocation would not occur within established timeframes, so Bug #1804753 for delayed revocation was opened.
Issues: Certificate Mis-issuance
4. Delayed Revocation for EV TLS Certificate incorrect jurisdiction -
https://bugzilla.mozilla.org/show_bug.cgi?id=1804753
This is the delayed revocation bug related to Bug #1802916, above. Entrust listed 8 Subscribers who were pushing back on immediate certificate revocation and the reasons given (e.g. extensions granted due to end-of-year freezes). Entrust committed to “continue to develop and extend methods for automatic certificate renewal.”
Issues: Delayed Revocation
5. Jurisdiction Locality Wrong in EV Certificate -
https://bugzilla.mozilla.org/show_bug.cgi?id=1867130
Two EV TLS Certificates were mis-issued due to human error in the Jurisdiction Locality field. (The incident revealed 340 additional accounts needing similar updates.) Although not expressed in the bug, it appears that certificate revocation was delayed as well. Entrust said it would enhance its linting processes to include possibly using an external service to validate locality data against verified country data.
Issues: Certificate Mis-issuance; Delayed Revocation
6. SHA-256 hash algorithm used with ECC P-384 key -
https://bugzilla.mozilla.org/show_bug.cgi?id=1648472
A Mozilla policy was adopted to require hashing with SHA-384 for an ECC P-384 key. Existing CAs using SHA-256 were not re-configured when Mozilla adopted this policy. This incident revealed a serious gap in taking new requirements and implementing them. Ryan Sleevi noted that linting was just a safety net and not a systemic solution. Entrust was also criticized for the lack of detail in its incident report and its decision to not revoke the certificates.
Entrust committed to improving its monitoring and implementation of policy changes to prevent similar incidents. Ryan set forth a number of proactive systemic corrections that Entrust needed to take, rather than taking a reactive stance on matters of non-compliance. Entrust committed to rigorous review of certificate profiles, browser policy revisions, and industry developments. As a final comment, Ryan said, “My big concern is, going forward, we see incident reports from Entrust take a more systemic, holistic response, like Comment #16, to try and cover the scenarios, and to provide sufficient detail about the situation and its failures to understand how those relate. The goal isn't to make CAs wear proverbial sackcloth, it's to try and make sure we're understanding how things go wrong, so that we can effectively collaborate on identifying solutions to avoid that going forward.”
Issues: Certificate Mis-issuance; Policy/Procedure Failure; Incident Response; Incident Handling
7. Late Revocation due to SHA-256 hash algorithm -
https://bugzilla.mozilla.org/show_bug.cgi?id=1651481
This bug is related to Bug #1648472. Entrust issued TLS certificates using ECC P-384 keys hashed with SHA-256, contrary to Mozilla policy requiring SHA-384 for hashing. Entrust’s initial decision was to allow certificates to expire naturally without revocation, but this was revised with a decision to revoke all affected certificates. Entrust committed to: filing incident report within one business day for future incidents, filing late revocation incident reports within the required 24 hours or 5 days, as applicable, and advising Subscribers about revocation within 24 hours or 5 days, or provide an explanation if they are unable to meet such timeframes. Entrust was told it needed to align its revocation procedures more closely with the Baseline Requirements and Mozilla’s policy, especially in providing a detailed rationale for any delays in revocation on a per-subscriber basis and ensuring timely revocation in line with the Baseline Requirements.
Issues: Delayed Revocation
