CA/Symantec Issues

From MozillaWiki
< CA
Jump to: navigation, search

Following the investigation documented below, a consensus proposal was reached among multiple browser makers for a graduated distrust of Symantec roots. The dates in that proposal and how they apply to Mozilla's Root Program and Firefox are as follows:

  • December 1st, 2017: Symantec to implement "Managed CA" proposal
  • January 2018 (Firefox 58): Notices in the Developer Console will warn about Symantec certificates issued before 2016-06-01, to encourage site owners to migrate their TLS certs.
  • May 2018 (Firefox 60): Websites will show an untrusted connection error if they have a TLS cert issued before 2016-06-01 that chains up to a Symantec root.
  • October 2018 (Firefox 63): Removal/distrust of Symantec roots, with caveats described below.

Note: Mozilla's planned release content and schedules are subject to change.

This page lists all confirmed issues involving the CA "Symantec". It may be further updated by Mozilla as more information becomes available. Please do not edit this page yourself; if you have proposed changes, email Wayne. Information here is correct to the best of Mozilla's knowledge and belief. Symantec has also confirmed the accuracy of the information, although errors transcribing their statements into this page remain Mozilla's.

The issues are in broadly chronological order by end date.


Issue B: Issuance of 1024-bit Certificate Expiring After Deadline (Dec 2013 - Jan 2014)

Symantec issued a cert to one of its customers that did not comply with at least one provision of both the CA/Browser Forum Baseline Requirements and Mozilla policy. It was a 1024-bit cert which expired after the end of 2013. Symantec believed this was the only technical way to ensure continuity of service for the customer concerned.

This cert was issued directly from the root. Symantec's write-up points out that issuance from the root is permitted by BRs version 1.1.6, the version in force at the time, if 5 conditions are met, and they say they were met.

This cert was backdated, but that is not a BR or Mozilla policy violation, as long as it was not done to evade a technical control.

The cert had a short serial number. Entropy in the serial number is a SHOULD in BRs 1.1.6. 20 bits of entropy is a MUST in the Mozilla policy (version 2.2), but it doesn't say it has to be in the serial number - it could be that they randomised the notBefore time. I am told Microsoft removed the allowance for doing entropy in the Date field on 11th November 2013, so this was a violation of their policies. Symantec say that they got a verbal exception from Microsoft.

Symantec did not request permission to issue in advance, they disclosed the issuance at least a month after it had happened, and the replacement certificate (unlike the original) asserted a "BR Compliant" OID.

The incident is recorded in bug 966350.

Symantec Response

Symantec self-reported this issuance to Mozilla via Bugzilla at the end of January 2014. Even though a question was asked about the BR Compliance OID that the new cert asserted, they did not comment on the bug beyond the initial report. Recently, Symantec have produced a longer write-up of the incident.

Further Comments and Conclusion

Given that we did not query it at the time, we must accept that the BR criteria for direct issuance from the root were met. The Mozilla policy does not include the exceptions directly, but does reference BRs chapter 12, and so could be said to include the exceptions by reference.

The issuance of a 1024-bit cert expiring after the deadline was both a BR and a Mozilla policy violation. Symantec say: "we did not engage the broader browser community due to the time pressure around the holiday." This seems like a weak excuse.

The inclusion of a BR-compliant OID in a non-BR cert was disappointing, but can be accepted as an oversight.

Issue C: Unauthorized EV Issuance by RAs (January 2014 - February 2015)

Symantec have stated that their four RA partners (CrossCert, CertSuperior, Certisign and Certisur) had the technical ability to issue EV certificates despite not having EV audits for their operations. Symantec does not directly state so, but it is assumed that, given the lack of audits and the infrequency of occurrence (normal process was for them to pass EV request information on to Symantec) that the RAs were not authorized to make such issuances. However, three of those organizations did so on a number of occasions. Looking at the dates, this appears to have happened in two successive audit periods.

Symantec Response

Symantec says that they revalidated the EV information used to issue the certs once they discovered the issuance, and that the validations met the EV authentication guidelines. However, it seems that they did not implement technical measures to prevent further occurrences. Symantec discovered all this before the recent investigation, but did not disclose these events to Mozilla as misissuances at the time.

Further Comments and Conclusion

EV issuance is a trusted privilege; Symantec should not have made it possible for organizations without the appropriate audits to issue EV, and they should have stopped them after it happened.

Issue D: Test Certificate Misissuance (April 2009 - September 2015)

Between 2009 and 2015, Symantec issued a large number of test certificates in their publicly trusted hierarchies. These contained domains that Symantec did not own or control, and for which domain validation was not performed. Some of these domains were unregistered, and others were owned by other organizations. Issuing test certificates for unregistered domains was not a BR violation before 3rd April 2014 (ballot 112), because they counted as Internal Server Names, but they continued the practice even after that date. The registered domains used included those belonging to Google and Opera Software. Given the numbers involved, this sort of test certificate issuance appears to have been common practice at Symantec. Some of the test certificates (including one for left Symantec's network because they were logged in CT. (Symantec claim that no certificates left their network; however, it's not clear how this can be true, and clarification is being sought.) However, Symantec personnel would have had access to the public and private keys of the certs.

Some details of this incident are recorded in bug 1214321.

Symantec Response

It took quite a while for the full scale of the problem to emerge, and Symantec had to publish numerous updates to their "Final Report". Symantec released a number of documents relating to this - Final Report, Final Report v2, Final Report v3, list of certs containing owned domains, list of certs containing unregistered domains, further update. They also issued a blog post, "A Tough Day As Leaders", which is no longer on their website but has been archived. It indicates that 3 employees were fired as a result of the misissuance. They posted another page with "complete investigation results".

In their report, Symantec wrote:

We have identified three root causes underlying the mis-issuance of these test certificates. First, we continued to issue internal test certificates to unregistered domains after the April 2014 change in the Baseline Requirements that removed authorization to do so. The overwhelming majority of the mis-issued test certificates fall into this first category. Second, certain Symantec Quality Assurance (QA) personnel had systems access, including the ability to use certain legacy tools, which enabled them to request a limited number of test certificates that were issued without review by authentication personnel. Third, authentication personnel did not consistently follow all verification steps when they received test certificate requests from their Symantec colleagues, or requested test certificates themselves.

Symantec took a number of remediation steps, as outlined in the report.

Further Comments and Conclusion

Symantec later revealed that there was a further small (6 certificate) incident of this type in March 2016, "involving approved domains in a test account".

Issue E: Domain Validation Vulnerability (October 2015)

In October 2015, it was discovered that Symantec's DV certificate products (e.g. RapidSSL, QuickSSL) had a flaw when parsing email address data from WHOIS, in that they did not parse + and = characters correctly. This meant that, in cases where the domain owner had used these characters, the address as parsed by Symantec was not the one the domain owner had inserted, and if other email addresses of a particular form could be registered at that domain, there was a possibility that an attacker could get a cert for the domain.

Symantec Response

Symantec fixed the issue within a week, audited their logs to make sure the flaw had not been abused, and issued a security advisory. They have said they have no further comment.

Further Comments and Conclusion

The set of circumstances which would have allowed this issue to be exploited (+ or = character in WHOIS, domain where arbitrary email address registration by 3rd parties is possible, necessary email address still available to register) are relatively rare, and Symantec fixed the issue quickly and performed appropriate remediation. Software has bugs. I do not consider this issue to be particularly severe.

Issue F: Symantec Audit Issues 2015 (December 2014 - November 2015)

Symantec's 2015 audit reports can be found in their legal repository. Symantec's standard audit period is from December 1st to November 31st.

The 2015 Baseline Requirements audits for Symantec's GeoTrust roots and their Symantec and Thawte roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the following violations of the Baseline Requirements or Network Security Guidelines:

  1. Issuance of Internal Server Names past the deadline date
  2. Test certificates issued for domains Symantec did not own or control (see above)
  3. No audit report, or invalid audit report, obtained for 3 of 5 external partner sub-CAs (GeoTrust only)
  4. Failure to maintain physical security records for an appropriate period of time
  5. Unauthorized employees with access to certificate issuance capability
  6. Failure to review application and system logs

The 2015 WebTrust for CAs audits for Symantec's Verisign and own-brand roots, their Thawte roots and their GeoTrust roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the following violations:

  1. Background checks not renewed for trusted personnel
  2. Unauthorized employees with access to certificate issuance capability
  3. Failure to maintain physical security records for an appropriate period of time (GeoTrust only)
  4. Test certificates issued for domains Symantec did not own or control (see above)

Of these, only the 'background checks' issue is not a repeat of an issue raised in the BR audits.

The 2015 Extended Validation audits for Symantec's Verisign and own-brand roots, their Thawte roots and their GeoTrust roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the 'test certificates' and the 'physical security records' issues which are noted above.

Symantec Response

Each of the documents contains, in a following table, Symantec's comments on the qualifications and what they have done or are doing to remedy them. They have said that they have no further comment.

Further Comments and Conclusion

Mozilla did not object to these qualifications in Symantec's audits at the time the audit documentation was submitted to us. Because of this, it is not reasonable for us to take action based on the mere existence of these qualifications. They are listed here because they are one part of the general picture of Symantec's compliance or otherwise with the BRs.

Issue H: SHA-1 Issuance After Deadline (January 2016)

Symantec issued five SHA-1 certificates after the SHA-1 issuance deadline. These were certificates requested ("enrolled") in 2015 but issued in 2016. Therefore, they passed through the point in the process where the presence of SHA-1 was checked for before the check was enabled.

Symantec Response

Symantec self-reported this issue to the CAB Forum public mailing list, and recently stated that they have no further comment.

Further Comments and Conclusion

Symantec were not the only CA to have similar issues with accidental SHA-1 issuance. This issue is listed here because it was not the only time it happened (see below).

Issue J: SHA-1 Issuance After Deadline, Again (February 2016)

Symantec issued a SHA-1 CT precertificate in February 2016. They then seemed to claim that misissuance of a pre-certificate is not misissuance although they now acknowledge that it was. (The CT RFC states that issuance of a pre-certificate is considered equivalent to issuance of the certificate, and so Mozilla considers that pre-certificate misissuance is misissuance.)

Symantec Response

Their explanation for the incident was as follows:

Following discussions with the customer who initiated this order, we have identified a technical deficiency in our system that allowed for hash algorithm modifications by a subset of customers to existing enrollments in limited circumstances, and only when pending administrator review prior to issuance. We released a patch today to add this case to our system-wide SHA1 blocking mechanisms. In addition, as an added precaution, we are evaluating an update to actively change any SHA1 orders encountered in our system to SHA256.

They have also produced a more detailed explanation of the events, including details of the remediation steps taken.

Further Comments and Conclusion

It is concerning that their first experience with SHA-1 misissuance did not cause them to analyse their systems and find this potential problem, or to put in place SHA-1 blocks in enough places to catch this.

Issue L: Cross-Signing the US Federal Bridge (2009 - July 2016)

The US Government has an extremely complicated PKI called the Federal PKI. It has applied for inclusion in the Mozilla root store but that application seemed unlikely ever to be successful due to the difficulty of bringing the entire FPKI in line with Mozilla's policies. During the period in question, it had a number of non-audited subordinate CAs.

Since 2009, Symantec has regularly had a valid cross-sign for one or both of "Federal Bridge CA" and "Federal Bridge CA 2013", which are both part of the FPKI, thereby making (as far as I can tell) all certificates in the FPKI be publicly trusted, and technically making Symantec responsible to Mozilla for all certificates issued in the FPKI, including any BR violations. The intermediate CA certificate(s) concerned were not disclosed in the CCADB, as Mozilla practice at the time required. This was reported in m.d.s.policy.

Symantec is not the only CA to have done this; IdenTrust also did it on multiple occasions from 2011-01-14 onwards. I don't believe there are any unexpired unrevoked (by OneCRL) links between the FPKI and the Mozilla trust store any more, via any CA.

Symantec appear not to have been aware that this would lead to certificates from the FPKI being trusted in browsers until around the time it was drawn to their attention by Eric Mill in February 2016.

Symantec Response

After this was drawn to their attention, Symantec did not revoke the cross-sign certificate under discussion, instead allowing it to expire. (By contrast, Identrust revoked their similar cross-signature in mid-late February, a week or so after being notified of the issue by Mozilla.)

Symantec claim that the problem is with browsers not processing certificate policy extensions which are used within the FPKI. When they realised the problem, they negotiated with the FPKI to allow the relevant cross-cert to expire rather than renewing it.

Further Comments and Conclusions

Symantec state that they were aware of the issues with this cross-signing since 2014 and were investigating whether it was actually required. It would seem it took two years to figure that out, and at no time was Mozilla informed about the fact that Symantec was enabling public trust for the entire Federal PKI.

While technical controls within the FPKI may have prevented it, Symantec had no way of knowing or controlling whether the FPKI was issuing EV certificates using Symantec's OID.

Issue N: Premature Manual Signing Using SHA-1 (July 2016)

Symantec applied to the root programs for leave to issue SHA-1 certificates for a customer. As part of that process, they planned to prepare tbsCertificates ("To Be Signed Certificates" - all the data except for the signature) for CAB Forum review. However, they accidentally issued real certificates - i.e. they signed the tbsCertificate data using SHA-1. This was supposedly a manual process but it seems like the relevant instructions were either deficient (in that they specified too many steps) or not followed.

Symantec Response

Symantec self-reported this issue to Mozilla. Their remediation was to revoke all of those certificates, and issue new ones after permission was granted.

Symantec state:

This matter represents the first time any CA attempted to follow the exception process which was developed over the course of weeks, beginning at the Bilbao CABF face-to-face meeting in May 2016, and with the input of our partners. Google initially proposed this exception process, which was later modified following input from other CABF members. Our internal process did not clearly specify to our PKI Operations team to stop at the point of TBS creation, which subsequently resulted in the creation of signed certificates instead of TBS Certificates. Importantly, our audit process promptly identified the error, and Symantec never released the certificates. We also promptly improved our internal process.

Further Comments and Conclusion

Is is true that this is a new process, and that stopping the issuance process at the point of creating TBSCertificates is unusual. While perhaps embarrassing, the risk here seems minimal given that near-identical certificates were issued and distributed a week later.

Issue P: UniCredit Sub CA Failing To Follow BRs (March - October 2016)

Symantec issued an unconstrained sub-CA to a company called UniCredit as part of their GeoRoot program (see also Issue V). This company persistently issued certificates which were BR-noncompliant (e.g. missing SANs, missing OCSP URIs). During this time, they did not have the appropriate audits and Symantec were aware of this. Symantec claimed, in response to the CA Communication of March 2016, that they had disclosed to Mozilla all their intermediate CAs along with audits, but this was not true in the case of UniCredit. The UniCredit CPS was also BR-noncompliant, in that it said that SANs were optional.

This incident is recorded in bug 1261919.

Symantec Response

Symantec's initial response was to get UniCredit to put in place controls to fix the violations found, and to review and replace any affected certificates. However, they continued to be without an audit. Symantec eventually asked them for one, and when they were unable to produce (presumably, pass) one, ordered them to stop issuing. However they continued, in violation of that agreement. Symantec then finally revoked their intermediate.

Symantec admit that the omission of UniCredit from their CA Communication response was an oversight. They further state:

We worked with UniCredit over a long period of time to enforce their compliance with audit requirements. In July 2016, we received an assessment that did not meet WebTrust audit standards. We then took action, helping UniCredit transition to a managed PKI solution for their certificate needs that did not require an audit. In parallel, we notified them of termination of their subordinate CA.

Because our customers are our top priority, we attempted to minimize business disruption while they transitioned by permitting UniCredit to operate only its CRL service until November 30, 2016, at which point we would revoke the UniCredit subordinate CA. In October 2016, UniCredit issued one new certificate in violation of the terms of that transition plan. Following that, Symantec promptly revoked the UniCredit subordinate CA on October 18, 2016.

Further Comments and Conclusion

There is doubt as to whether Symantec's indulgence of UniCredit's continued incompetence is within the letter and/or spirit of the BRs and root policy. Firstly, there is the long timeline over which UniCredit was out of compliance. But even when Symantec took action, Baseline Requirements 1.4.4, Section, item 8 states that Symantec should have revoked the intermediate immediately upon removing their right to issue, unless Symantec took over the CRL and OCSP duties, which they did not.

This section of the BRs is presumably there to prevent an incompetent or untrustworthy ex-sub-CA owner from continuing to add risk to the WebPKI once their incompetence has been acknowledged.

Issue Q: Symantec Audit Issues 2016 (December 2015 - November 2016)

The Baseline Requirements section 8.6 says that CAs SHOULD provide audits within 90 days of the end of the audit period; this SHOULD was not followed by Symantec for both the 2014/15 and 2015/16 audit cycles. However, Symantec is not the only CA which regularly supplies its audits late.

Symantec's 2016 audit reports can be found in their legal repository. Symantec's standard audit period is from December 1st to November 31st. However, for 2016, they have split the audits into two roughly six-month periods, and had separate audit opinions issued for each.

As detailed in their covering letter, the audits for the second period, June 16th to November 30th 2016, are mostly unqualified. The BR audits have a total of five qualifications, two of which relate to previously disclosed incidents which are not of concern, and three other qualifications which seem to be only of minor concern.

The audits for the first period contain many or all of the same issues as the 2015 audits (Issue F). One can surmise that Symantec chose to split the audits in order that they would not have a qualified audit covering the entirety of 2016. This raised questions about how long the issues which led to these qualifications persisted.

None of the audits contain any qualification related to the audit status of the GeoRoot or RA program partners.

Symantec Response

Each of the documents contains, in a following table, Symantec's comments on the qualifications and what they have done or are doing to remedy them.

Symantec recently stated: "In our 2014-2015 audits, certain issues were identified that we promptly took action on, including addressing the test certificate incident. We continued these efforts until the Point in Time audit was conducted."

When asked about the amount of time taken to remediate, Symantec's response stated that the issues were discovered late in 2015 or early in 2016, and some took several months to remediate if you include the need for revalidations and HR team reorganizations, which is why they persisted into the 2016 audit period. Specifically:

  • Test certs to registered domains: Discovered and fixed in Sep 2015, but "while inappropriate use of registered domains for testing stopped during the course of our 2014-2015 audits, we did not complete the ID and revocation of all certificates until Mar 2016, and so the finding remained in our first-half Dec 1, 2015-Jun 15, 2016 audits."
  • Test certs to un-registered domains: Discovered and fixed in October 2015, but there was a "discovery of additional instance involving approved domains in a test account resulting in 6 additional issued certificates in Approximately Mar 2016."
  • Unauthorized employees with access to certificate issuance capability: discovered in September 2015, last problem of this type remediated in June 2016 after extensive security review.
  • Failure to maintain physical security records for 7 years: discovered and fixed in January 2016.
  • Failure to review application and system logs: discovered and fixed in January 2016.
  • The failure to refresh background checks every 5 years: discovered in February 2016, fixed in June 2016 (required an internal reorganization).

In regard to the lack of qualifications related to GeoRoot and the RA program, Symantec said: "We acknowledge there were deficiencies in audits for both the GeoRoot and RA programs. The plan for the GeoRoot deficiencies was communicated in the cover letter accompanying our Point in Time audit (see issue V) and for the RA program, in the cover letter to browsers with our 2015-2016 audits."

Additional Comments and Conclusion

It is noteworthy that even if third party audits are qualified, that does not lead to a qualification on Symantec's main audit.

STRUCK: Issue R: Insecure Issuance API (2013 or earlier - November 2016)

According to a report, it is alleged that for several years Symantec operated an issuance API which was insufficiently secure, such that URL parameter substitution attacks would allow one customer to view, reissue, revoke and otherwise control certificates (including non-server certs) belonging to another customer. It is further alleged that, when made aware of these issues, Symantec took a very long time to fix them, and they may not have been fully fixed at the time Symantec terminated its RA program entirely.

Symantec Response

Symantec commented to the press:

"We have looked into Chris Byrne’s research claim and could not recreate the problem. We would welcome the proof of concept from the original research in 2015 as well as the most recent research. In addition, we are unaware of any real-world scenario of harm or evidence of the problem. However, we can confirm that no private keys were accessed, as that is not technically feasible.

In addition, Tarah from Symantec has posted a detailed comment which suggests that the issue is or was substantially less serious than the initial write-up made it sound. They have also made additional comment in response to this document.

Further Comments and Conclusion

At the moment, there is no compelling evidence that Symantec's account of events is incorrect. If their account of events is correct then I don't see a problem here. For better or worse, the sending of emails with somewhat privileged access URLs in them is common practice in this and other industries.

Issue T: CrossCert Misissuances (January 2010 - January 2017)

For several years, Symantec operated an RA (Registration Authority) program. The companies in this program had independent authority to issue certificates under Symantec intermediates, and those certificates were not specifically marked to say that a particular RA had issued them instead of Symantec directly. This by itself is not against any existing policy.

However, In January 2017, it was discovered that a small number of certificates had been misissued using Symantec intermediates by an RA called CrossCert in Korea. The investigation of these matters led to increasing further discoveries of misissuances (totalling 127 confirmed cases) and potential misissuances by CrossCert, together with poor CA practice by them and other companies in Symantec's RA network. This issue deals with the certificate misissuances; the following issue deals with the audit-related problems uncovered.

Types of misissuance by CrossCert:

  • Use of unvalidated domain names not owned by either Symantec or CrossCert
  • Typos in domain names
  • Bogus ST, L, O and OU fields: numbers, "test" or similar
  • Violation of CPS (use of non-KR country code)

Some of these misissuance were caused by employees of CrossCert overriding compliance flags in Symantec's issuance system. Symantec had no process in place to review the logs of overridden flags. For some of the certs, they contained domains neither Symantec nor CrossCert own or control, and CrossCert did not complete the appropriate domain validations for them.

Lastly, it later emerged that for a number of Certs, CrossCert has incompletely logged records of their telephone validations and so it could not be confirmed that the validation work had been performed correctly.

This incident is recorded in bug 1334377.

Symantec Response

Symantec made a number of comments on this issue - 0, 1, 2, 3, 4.

The Baseline Requirements, in section item 9, state that the CA SHALL revoke a certificate if "The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement". However, Symantec did not revoke all the certificates.

Instead, Symantec decided to shut down the RA program entirely and re-assess every certificate issued under it. Symantec committed to revalidating all of the CrossCert-issued certificates (10,000+) and any of the 20,000+ certificates issued by their other RAs if deficient validation was discovered. However, the determination of deficient validation (in cases other than CrossCert) was made based on the RAs own logs of activity, which may themselves be suspect given some of the audit deficiencies found at these RAs.

Symantec has produced a further, long account of the situation. It confirms that, while there were multiple systems designed to teach the RAs what to do, the only system which checked that they were actually doing it (and which Symantec paid attention to) were the WebTrust audits. They disagree that full revocation of all certs is a proportionate response.

Further Comments and Conclusion

Symantec's reaction to the discovery of these problems was, to mind, swift and comprehensive - the entire RA program was shut down in fairly short order.

Their case is that WebTrust audit monitoring should have been sufficient, but that they were let down by their auditor, who failed to notice some of the problems, or in other cases it just so happened that the issues were either a long time ago or too recent to be caught by audit. This case is undermined by Issue W - CrossCert and many of the other RA partners had significantly deficient audits and/or audits with serious qualifications.

I conclude that the reason that the situation at CrossCert was not replicated elsewhere was a matter of luck, and that Symantec's monitoring regime for its RA partners - who had full powers of issuance in the WebPKI - was not sufficiently robust.

Issue V: GeoRoot Program Audit Issues (2013 or earlier - January 2017)

Symantec runs a program called GeoRoot, where intermediate CAs have been created for the sole use and independent operation by specific customers at premises under their control. Some of these customers appear to have had a history of poor compliance with the BRs and other audit requirements.

Over multiple years (2013-12-01 to 2014-11-30, 2014-12-01 to 2015-11-30), Symantec's "GeoTrust" audits were qualified to say that they did not have proper audit information for some of these GeoRoot customers. This information was in their management assertions, and repeated in the audit findings. So the poor audit situation was ongoing and known.

There are five customers mentioned in the audits, and Symantec say they are Intel, Aetna, UniCredit, Google and Apple. They also say:

Separately, Symantec operates two subordinate CAs solely for NTT DoCoMo in an enterprise PKI application. These subordinate CAs had been considered part of the "GeoRoot" program as well, and we had therefore excluded them (similar to the above externally operated ones) from the list of Symantec CAs in our audits. After reviewing our approach, our compliance team determined that they should be included going forward. As such, for the 2016-2017 Period in Time, these subordinate CAs are included in the GeoTrust WebTrust for CA and BR audits.

Symantec Response

  • Symantec state: "Intel's subordinate CA, which expired in 2016, was not subject to audits either contractually or by previous agreements with both Mozilla and Microsoft given its limited use." This is true; the CA is mothballed and got out of storage only once a quarter to issue a CRL. Symantec agreed privately with Mozilla that audits were not necessary for a CA in such a state.
  • Symantec state: "Symantec provided the letter quoted below to Google, Mozilla, Microsoft, and Apple when we shared the Point in Time Audits on September 6, 2016 to specifically address the GeoRoot audit status and remediation plan. That cover letter outlined the plan to wind down the Aetna and UniCredit subordinate CAs." This is true, although Aetna and UniCredit are not mentioned by name in the letter.

Symantec have also stated that, as of April 21st, 2017, the "Intel, Aetna, and Unicredit CAs have all expired or been revoked." This leaves Google and Apple as the only participants in the GeoRoot program. They also say that: "We agree that getting audits for Aetna and Unicredit took too long. After many discussions, requests, and delays, they finally produced the reports that they did. This experience informed our decision to transition them to alternative solutions."

On the subject of NTT DoCoMo, Symantec write: "All authentication performed for the NTT CA’s has been completed by Symantec personnel. The authentication applications used have historically been fully within the scope of our WTCA/WTBR audits. The infrastructure on which this specific enterprise PKI application and those CAs operate was not covered under audit until these CAs were added to the 2015-2016 audits."

Further Comments and Conclusion

It seems that the NTT DoCoMo infrastructure did fall through the cracks audit-wise until 2015-2016.

Given the power which those organizations held, Symantec did not pursue Aetna and UniCredit for proper audits and appropriate compliance with sufficient alacrity (on UniCredit, see Issue P). However, to a degree, Symantec did keep Mozilla somewhat informed of what was going on, and Mozilla made no comment. It is our understanding that at least one other root program was applying more pressure to remediate this situation than Mozilla was.

Issue W: RA Program Audit Issues (2013 or earlier - January 2017)

Symantec's RA program had four participating companies - CrossCert, Certisign, Certsuperior, and Certisur.

Certsuperior's audit is particularly dreadful:

  • There was no legible CPS;
  • inadequately-trained personnel were doing issuance;
  • the network was an insecure mess; and
  • non-trusted staff had access to issuance.

Prior to 2016, Certsuperior was providing WebTrust audits from an unlicensed auditor.

CrossCert's audit does not list or cover the full number of Symantec roots under which they had issuance capability. Symantec did not notice this mismatch until their recent investigations, when they discovered that CrossCert had the scope of the audit reduced for cost reasons.

Certisur's audit is only a WebTrust for CAs audits - they do not appear to have a Baseline Requirements audit. According to Symantec's records, this has been true ever since BR audits became required. Mozilla policy requires "CA operations and issuance of certificates to be used for SSL-enabled servers" to conform to the Baseline Requirements. As Symantec has stated that audit was their only mechanism for monitoring their RAs, they can have had no assurance that RAs without a BR audit were actually following the BRs.

Certisign's audits appear to be in order, although their 2016 audit is past due.

Symantec Response

Symantec required the issues at CertSuperior to be fixed and a 90-day action plan was executed to fix them. However, until they decided to shut down the RA program, no certificates issued during the period of suspect operations were checked to see if the poor practice had caused misissuance. They have requested that their next audit include both WebTrust for CAs and WebTrust Baseline.

Symantec appears to have taken no action to deal with that fact that Certisur did not have a BR audit until recently, when they have requested that their next audit include both WebTrust for CAs and WebTrust Baseline.

Symantec did not notice that CrossCert's audits did not cover all the relevant roots until they did the RA investigation in early 2017.

Further Comments and Conclusion

Despite the clear warning signs shown on the Certsuperior audit, Symantec did not put in place any monitoring of their RAs, other than audit, to check that they were correctly performing the tasks delegated to them under the BRs. (There were some - overridable - technical checks on certificate issuance.)

Symantec's compliance department did not notice many or any of these audit scope problems until 2016 - in particular those from Certisur and CrossCert.

Symantec have asserted that their oversight of these RA partners was primarily based on reviewing audits. However, the audits had a number of irregularities of various kinds, including being of the wrong type and so not covering the BRs at all, or not covering all issuance, which were not noted until recently. This claim of oversight therefore rings fairly hollow.

STRUCK: Issue X: Incomplete RA Program Remediation (February - March 2017)

At the time Symantec shut down their RA program, they had four RAs - CrossCert, Certisign, Certsuperior, and Certisur. Symantec committed to revalidating all certificates issued by those RAs. Independent of the rightness or otherwise of this course of action, it should have been applied consistently. However, the program seems to have previously had additional RAs, and Symantec has as yet taken no action to revalidate the certificates that they issued, despite some still being valid.

Those RAs include at least E-Sign, for which we have found audits from March - June 2014, April - July 2015 and July - September 2016. This party does not appear to have an unbroken series of audits; it is unclear what that means.

Another possible RA is MSC TrustGate, which appears to have been operating as a sub-CA based on the scope of their audit. However this audit, like those for some other Symantec RAs, is only to the WebTrust for CAs criteria and does not cover the Baseline Requirements. The WebTrust audit criteria require that such a CA has a BR audit.

Symantec Response

Symantec state:

The only Symantec RAs capable of authorizing and issuing publicly trusted SSL/TLS certificates are: CrossCert, Certisign, Certsuperior and Certisur. Symantec continues to maintain a partner program for non-TLS certificates. E-Sign SA and MSC Trustgate are amongst these partners.

Symantec agree that the audits are not clear about what specific CAs are used for, but assert that these organizations are not in a position to issue SSL/TLS certificates.

Further Comments and Conclusion

In the absence of evidence that these organizations have the ability to issue SSL/TLS certificates, we must accept Symantec's assertion.

Issue Y: Under-audited or Unaudited Unconstrained Intermediates (December 2015 - April 2017)

Two intermediate CAs, which are subordinates of or cross-certified by VeriSign Universal Root Certification Authority (which is EV-enabled), have audit and control problems:

Both intermediates are disclosed in Salesforce, and both have 15 or so also-disclosed sub-CAs which seem to be specific to particular companies. The audit associated with both of them in Salesforce is this one from May 2016, i.e. from Symantec's 2015 set of audits (i.e. the set before the current one), but that audit document does not list the intermediate CAs that it covers. Symantec has produced a more recent audit but not yet updated Salesforce. This one does list the intermediate CAs covered. However, like that from the previous year, this is a WebTrust for CAs audit, and does not include a BR audit.

These intermediates appear to be related to the US Federal Bridge PKI (see Issue L) As far as we can tell, they are unconstrained, unrevoked and fully capable of issuing server authentication certificates which are trusted by Mozilla browsers. Mozilla policy is based on capability, not intent - if a sub-CA is capable of issuing SSL certs we trust, it must be appropriately constrained or audited. These intermediates have deficient audits and, as far as we can tell, sub-CAs of them are effectively controlled by entities without any audits at all. Specifically:

  • The CP/CPS does not state adherence to the Baseline Requirements.
  • The audit is only a WebTrust for CAs audit, not a BR audit.
  • A number of sub-CAs seem excluded from even the scope of that audit, as they are not listed in it: 1, 2, 3, 4, 5, 6, 7, 8.
  • The CP/CPS has a profile which includes issuing certificates with dNSName and iPAddress SANs, and Symantec states that Windows domain controller certs are within scope for the program. Such certs are fully TLS server certificates.
  • A Symantec statement suggests that customers of their NF SSP program can perform RA duties for the issuance of certs for Windows domain controllers and, according to the audit report, those RA activities are outside the scope of the audit entirely.

Symantec Response

Symantec say that the subCAs under these intermediates are under Symantec's control and, in all cases but one, are technically prevented from issuing for TLS server authentication. In one case, a customer does have the ability to issue TLS server auth certificates. The subCAs concerned ("CSC Device CA – G2" and "CSRA FBCA C3 Device CA"), like all the others, do not have BR audits.

They say that their technical controls, which apply to all these subCAs, prevent EV issuance.

Nevertheless, they plan to privatise this bit of the PKI by revoking trust in the certificates which link it to the WebPKI on May 24th, 2017.

Further Comments and Conclusions

Symantec say that these hierarchies were not BR audited based on the system controls they have in place and the intended usage. But Mozilla policy is based on capability, not intent. And in one case, both capability and intent led to the issuance of TLS server certs, and yet no audit was obtained.