Update (25th August 2023 22:50 GMT)

This change has been rolled back by Microsoft. Certificates that were previously revoked are now showing as valid across multiple systems. It can be seen here https://ccadb.my.salesforce-sites.com/microsoft/IncludedCACertificateReportForMSFT that the G5 Root Certificate (SHA1 4EB6D578499B1CCF5F581EAD56BE3D9B6744A5E5) is now showing as ‘Not Before’ rather than ‘Disabled’.

See Microsoft’s definition of ‘Not Before’ here https://learn.microsoft.com/en-us/security/trusted-root/deprecation

This change should be reflected quickly in customer environments, however if customers have a need to manually force this change the command ‘certutil -verifyctl -f authrootwu‘ can be run.

At this time we do not have any official information from Microsoft and are unsure if this change was intentional in the first instance, or rolled back due to ecosystem impact.

Background

On 23rd of August 2023 Airlock Digital received reports from customers seeing a sudden increase in file executions that had ‘Invalid Certificate Chains’.

Airlock Digital investigated these reports and found that all occurrences of this certificate status chained trust up to the Verisign Class 3 Public Primary Certification Authority – G5 Root Certificate (serial: 18dad19e267de8bb4a2158cdcc6b3b4a). Over the coming hours, it was identified that many internet connected Windows 10 & 11 computers within the Airlock Digital environment also began reporting files chained to this root as having ‘Invalid Certificate Chains’.

History

This particular root certificate is responsible for chained trust in a significant number of issued digital certificates globally between 2006 – 2018. As PCWorld reports “According to a Netcraft survey from 2015, Symantec is responsible for about one in every three SSL certificates used on the web, making it the largest commercial certificate issuer in the world.” Source: https://www.pcworld.com/article/406198/to-punish-symantec-google-may-distrust-a-third-of-the-webs-ssl-certificates.html

Its downfall however came when browser vendors revoked trust in Verisign’s root certificates in 2018, due to repeated occurrences where Symantec (who owned Verisign at the time) improperly issued SSL certificates from this and other roots to customers and its re-sellers.

As a result of this browser revocation, it effectively pulled the pin on Symantec’s certificate business. Around this time, the business was acquired by DigiCert.

Revocation Event

Airlock Digital worked throughout the 23rd of August to determine why this revocation was occurring on Microsoft Windows systems. There was little information publicly and no word from Microsoft or DigiCert. The only other information that could be found online, was a discussion by the great folks over at /r/sysadmin stating they were having trouble loading Quickbooks due to an invalid certificate. Analysis of the QuickBooks binary in question showed that it chained to the same Verisign root. Kudos to Intuit (vendor of Quickbooks) for having an application which validates Digital Certificates of its libraries before launching them.

QuickBooks – Application with Revoked Certificate
by u/WhyPartyPizza in sysadmin

This revocation can be easily recreated on any Windows 10 or 11 machine by downloading the PEM certificate from the following site https://crt.sh/?id=93&opt=ocsp and viewing the Certificate in Windows Explorer. In Airlock Digital’s testing, unsupported versions of Windows such as Windows 8 and below are unaffected.

At this point, the question we were unable to answer was how did this certificate become revoked? Its serial was not listed on any online Certificate Revocation Lists (CRL’s) and the Online Certificate Status Protocol (OSCP) still lists the certificate as valid. So where is this coming from?

The only possibility left was Microsoft, as Windows was indicating that the certificate had a ‘disable’ flag applied as per Microsoft’s ‘Deprecation Definitions’ https://learn.microsoft.com/en-us/security/trusted-root/deprecation this explains why this did not show on any publicly available lists.

At the time of writing there is no notice from Microsoft on their trusted root certificate program of this change: https://learn.microsoft.com/en-us/security/trusted-root/release-notes

Confirmation

On the 24th of August, DigiCert responded to an Airlock Digital support request confirming the revocation by Microsoft:

“Yes, we have received a few reports in regard to the same issue.

After escalating it to our engineering team and they have confirmed with Microsoft that these legacy root certificates were supposed to be distrusted back to 2019 – 2021.

And for VeriSign Class 3 Public Primary Certification Authority – G5 was supposed to be distrusted by Microsoft on May 21st 2019.

Here is the DigiCert link for your reference: https://knowledge.digicert.com/alerts/symantec_root_distrust.html

However, Microsoft remains trusting this Root certificate until Aug 23rd 2023. And distrusted this root certificate yesterday.

So that our customers who are currently using this root certificate and others listed on that page have experienced the same situation.”

Airlock Digital has reached out to Microsoft Security (secure[at]microsoft[dot]com) for confirmation of this change and will update this blog post with more information if it becomes available.

Thoughts

There are many lenses through which this change can be viewed. When it comes to SSL certificates this change entirely makes sense, however for the purposes of code signing this is problematic, as previously signed files can’t simply have the certificate updated on them in the same way websites can. This is likely why Microsoft trusted this root certificate for much longer in Microsoft Windows than browser vendors did.

I’m also surprised at how little noise this change made even within the information security community. It speaks volumes at how little digital signatures are used to proactively enforce the integrity of software at load time within the majority of the ecosystem (again kudos to Intuit).

Microsoft moved quickly on this one without public notification, was there current threat activity associated with this certificate requiring a whole of ecosystem response? or was this just something Microsoft considered to be a non-event? we can only speculate.

Airlock Digital Customer Recommendations

Over the coming months Airlock Digital customers may notice an elevated occurrence of files reporting ‘(invalid certificate chains)’ for a subset of software that was digitally signed between 2006 – 2017.

It is recommended that if customers have a need to continue running these files, that in the short term that affected ‘(invalid certificate chain)’ publishers are be trusted within policy to minimise business impact. This trust can then be converted to hash based trust over time, by using Airlock Digital’s ‘trusted logging’ features on these specific publishers (a Knowledge Base article on this process will be available shortly). The good news here is that modern version of applications will be signed with newer certificates and these occurrences will be minimised in the long term.

Additionally, many Airlock Digital customers with existing agents may not immediately notice impact from this issue, due to the certificate caching behavior of the Airlock Digital client to improve runtime performance. Customers can independently validate a files certificate status by downloading Microsoft Sigcheck from https://live.sysinternals.com and running ‘sigcheck -a -i [path to file]’ and viewing the resulting chain.