Incident report: Failure to verify authenticity for some partner requests

------=_NextPart_000_029A_01D38A1E.A31E99E0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

 

Hi everyone, 
 
There was a bug in our OEM integration that led to a lapse in the
verification of authenticity of some OV certificate requests coming in
through the reseller/partner system.
 
As you know, BR 3.2.5 requires CAs to verify the authenticity of a request
for an OV certificate through a Reliable Method of Communication (RMOC).
Email can be a RMOC, but in these cases, the email address was a constructed
email address as in BR 3.2.2.4.4.  Despite the fact that these addresses are
standardized in RFC 2142 or elsewhere, we do not believe this meets the
standard of "verified using a source other than the Applicant
Representative."
 
The issue was discovered by TBS Internet on Dec 30, 2018. Apologies for the
delay in reporting this. Because of the holidays, it took longer than we
wanted to collect the data we needed.  We patched the system to prevent
continued use of constructed emails for authenticity verification early, but
getting the number of impacted orgs took a bit more time. We are using the
lessons learned to implement changes that will benefit overall user security
as we migrate the legacy Symantec practices and systems to DigiCert.   
 
Here's the incident report:
 
1.    How your CA first became aware of the problem (e.g. via a problem
report submitted to your Problem Reporting Mechanism, via a discussion in
mozilla.dev.security.policy, or via a Bugzilla bug), and the date. 
 
Email from JP at TBS about the issue on Dec 30, 2017.  
 
2.    A timeline of the actions your CA took in response. 
 
A. Dec 30, 2017 - Received report that indirect accounts did not require a
third-party source for authenticity checks. Constructed emails bled from the
domain verification approval list to the authenticity approval list. 
B. Dec 30, 2017 - Investigation began. Shut off email verification of
authenticity.
C. Jan 3, 2017 - Call with JP to investigate what he was seeing and
confirmed that all indirect accounts were potentially impacted.
D. Jan 3, 2017 - Fixed issue where constructed emails were showing as a
permitted address for authenticity verification.
E. Jan 5, 2017 - Invalidated all indirect order's authenticity checks.
Started calling on verified numbers to confirm authenticity for impacted
accounts. 
F. Jan 6, 2017 - Narrowed scope to only identify customers impacted (where
the validation staff used a constructed email rather than a verified
number).
G. Jan 10, 2017 - This disclosure.
 
Ongoing: 
H. Reverification of all impacted accounts
I. Training of verification staff on permitted authenticity verification
 
3.    Confirmation that your CA has stopped issuing TLS/SSL certificates
with the problem. 
 
Confirmed. Email verification of authenticity remains disabled until we can
ensure additional safeguards.
 
4.    A summary of the problematic certificates. For each problem: number of
certs, and the date the first and last certs with that problem were issued. 
 
There are 3,437 orgs impacted, with a total of 5,067 certificates.  The
certificates were issued between December 1st and December 30th.
 
5.    The complete certificate data for the problematic certificates. The
recommended way to provide this is to ensure each certificate is logged to
CT and then list the fingerprints or crt.sh IDs, either in the report or as
an attached spreadsheet, with one list per distinct problem. 
 
Will add to CT once we grab it all.  I will provide a list of affected
certificates in a separate email (it's big, so it was getting this post
moderated).
 
6.    Explanation about how and why the mistakes were made or bugs
introduced, and how they avoided detection until now. 
 
In truth, it comes down to a short timeframe to implement the
Symantec-DigiCert system integration and properly train everyone we hired.
We are implementing lessons learned to correct this and improve security
overall as we migrate legacy Symantec practices and systems to DigiCert. In
this case, there are mitigating controls.  For example, these are mostly
existing Symantec certs that are migrating to the DigiCert backend. The
verification by Symantec previously means that the number of potentially
problematic certs is pretty low. There's also a mitigating factor that we
did not use method 1 to confirm domain control. In each case, someone from
the approved constructed emails had to sign off on the certificate before
issuance.  This is limited to OV certificates, meaning EV certificates were
not impacted. Despite the mitigating factors, we believe this is a
compliance issue, even though we believe the security risk is minimal.
 
7.    List of steps your CA is taking to resolve the situation and ensure
such issuance will not be repeated in the future, accompanied with a
timeline of when your CA expects to accomplish these things. 
 
A. We clarified in the system what is required for an authenticity check. 
B. We removed email verification for authenticity checks until appropriate
new safeguards can be added.
C. We are re-validating authenticity for all potentially problematic
certificates and will revoke any that fail to validate (none have failed so
far)
D. We improved training for all validation specialists on the rules for
authenticity checking.
E. We are undergoing quarterly audits on the OEM system to ensure all checks
are in place (per the root transfer requirements from Google).
 
-Tim

 


------=_NextPart_000_029A_01D38A1E.A31E99E0
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCD0sw
ggO3MIICn6ADAgECAhAM5+DlF9hG/o/lYPwb8DA5MA0GCSqGSIb3DQEBBQUAMGUxCzAJBgNVBAYT
AlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5jb20xJDAi
BgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0wNjExMTAwMDAwMDBaFw0zMTEx
MTAwMDAwMDBaMGUxCzAJBgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsT
EHd3dy5kaWdpY2VydC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK0OFc7kQ4BcsYfzt2D5cRKlrtwmlIiq9M71
IDkoWGAM+IDaqRWVMmE8tbEohIqK3J8KDIMXeo+QrIrneVNcMYQq9g+YMjZ2zN7dPKii72r7IfJS
Yd+fINcf4rHZ/hhk0hJbX/lYGDW8R82hNvlrf9SwOD7BG8OMM9nYLxj+KA+zp4PWw25EwGE1lhb+
WZyLdm3X8aJLDSv/C3LanmDQjpA1xnhVhyChz+VtCshJfDGYM2wi6YfQMlqiuhOCEe05F52ZOnKh
5vqk2dUXMXWuhX0irj8BRob2KHnIsdrkVxfEfhwOsLSSplazvbKX7aqn8LfFqD+VFtD/oZbrCF8Y
d08CAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFEXr
oq/0ksuCMS1Ri6enIZ3zbcgPMB8GA1UdIwQYMBaAFEXroq/0ksuCMS1Ri6enIZ3zbcgPMA0GCSqG
SIb3DQEBBQUAA4IBAQCiDrzf4u3w43JzemSUv/dyZtgy5EJ1Yq6H6/LV2d5Ws5/MzhQouQ2XYFwS
TFjk0z2DSUVYlzVpGqhH6lbGeasS2GeBhN9/CTyU5rgmLCC9PbMoifdf/yLil4Qf6WXvh+DfwWdJ
s13rsgkq6ybteL59PyvztyY1bV+JAbZJW58BBZurPSXBzLZ/wvFvhsb6ZGjrgS2U60K3+owe3WLx
vlBnt2y98/Efaww2BxZ/N3ypW2168RJGYIPXJwS+S86XvsNnKmgR34DnDDNmvxMNFG7zfx9jEB76
jRslbWyPpbdhAbHSoyahEHGdreLD+cOZUbcrBwjOLuZQsqf6CkUvovDyMIIFOjCCBCKgAwIBAgIQ
Di7WjgxCjxTrYbReNHesEzANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJVUzEVMBMGA1UEChMM
RGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cuZGlnaWNlcnQuY29tMSQwIgYDVQQDExtEaWdpQ2Vy
dCBTSEEyIEFzc3VyZWQgSUQgQ0EwHhcNMTcxMTI4MDAwMDAwWhcNMjIwMjI1MTIwMDAwWjBWMQsw
CQYDVQQGEwJVUzENMAsGA1UECBMEVXRhaDENMAsGA1UEBxMETGVoaTERMA8GA1UEChMIRGlnaUNl
cnQxFjAUBgNVBAMTDVRpbSBIb2xsZWJlZWswggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
AQDKUTIS9F3d7CfkCjsf4my28pYoZJDkEAiXVqGP4jzbFkszUQNfW3PYpFUo1GnKQykl/tM0qnzw
05bfVLo1+ce0e9fyAwYfulr+HaAVCPqx+PZw9CDY6c0NYd7Fc7S0scONxKekNF4q1mUucfGuGapW
sEsyix0CuR0NMuJ4I+w8qMn9MzjzI7bvduG+uVLmZIi0p6D8+2R5BOQFy0tVeQ/aLfS91fG1DTYF
YkPF+a/6JlFxzywPzCth8KW2Po4w8JqQWtam/ADKrgMaOnEJs9csefTW/FWRDeGQk5t3rnyS19FP
QfpyPPau4ChB5xokfRcg3VEwqfOoIIexjUhZY5X9AgMBAAGjggHzMIIB7zAfBgNVHSMEGDAWgBTn
AiOAAE/Y17yUC9k/dDlJMjyKeTAdBgNVHQ4EFgQUjqBhf3GcBV6YGYSmp2iS4Wi/3N4wDAYDVR0T
AQH/BAIwADAlBgNVHREEHjAcgRp0aW0uaG9sbGViZWVrQGRpZ2ljZXJ0LmNvbTAOBgNVHQ8BAf8E
BAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMEMGA1UdIAQ8MDowOAYKYIZIAYb9
bAQBAjAqMCgGCCsGAQUFBwIBFhxodHRwczovL3d3dy5kaWdpY2VydC5jb20vQ1BTMIGIBgNVHR8E
gYAwfjA9oDugOYY3aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkFzc3VyZWRJ
RENBLWcyLmNybDA9oDugOYY3aHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0U0hBMkFz
c3VyZWRJRENBLWcyLmNybDB5BggrBgEFBQcBAQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcwAoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0Rp
Z2lDZXJ0U0hBMkFzc3VyZWRJRENBLmNydDANBgkqhkiG9w0BAQsFAAOCAQEAmOLw9+cVMHn8tJ0k
76baCfFZwkvfvxSAlCXo+Fcsv55/og0V065Rpb4HvVTi0e0qKCMbBxc71NWxhMvKJHt+sfSmVatX
mAOPNDRvtVvJBkcd0bvzMut/r3npQqs1wezHLtAq+MlQZDjgiJB+DkNblnnphzEQSp7q/4K9oMoP
KViRxBv+/kseA8GOfhHU6EVmeu9xQrBqexH1DPUrUSGpNGDyvtUaU+bBy8Kz2hQfOu6f/73wLqUx
e583C9y2Gqn1xCB77yPxXqRSLLRC6FbrToJbKiFYQJ4znZZyhPYJHL0SOpWyXfVKp4PEO54A/xr5
oVyPhEQhOtasoIRCLtHZrzCCBk4wggU2oAMCAQICEASueWBmZpAaucV/pmxb3M0wDQYJKoZIhvcN
AQELBQAwZTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3
LmRpZ2ljZXJ0LmNvbTEkMCIGA1UEAxMbRGlnaUNlcnQgQXNzdXJlZCBJRCBSb290IENBMB4XDTEz
MTEwNTEyMDAwMFoXDTI4MTEwNTEyMDAwMFowZTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lD
ZXJ0IEluYzEZMBcGA1UECxMQd3d3LmRpZ2ljZXJ0LmNvbTEkMCIGA1UEAxMbRGlnaUNlcnQgU0hB
MiBBc3N1cmVkIElEIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3PgRIz9qte/A
J3kbLQWHohBDMd8O1BUbT3ekIs4+jHDwvgeO3ScqvAEdtiwKyt1pWB9B7WoFH9pjeFkeIiwr+Lp+
yTU7VvEffEJ+JbAjGcZFONc9RPkgfGCuHLBaGAS+jzv3qfCUmqYMY0m2QRdTQDK9T+ZQelAfJUXo
8Ymvzf9e/1Dz8BcR/73FifW9YrnY+45FBIVtmc3FSE39JqsCNkXqNtdfauIagkEK3OnZ9ZEXjsYh
rTg8E+Yef2ac1U3ZRtr2z1KnfTskw7TBUTXGm+vU737kewPhRL16CzfgT8uCig1xGOSm4IksG/Oy
czzBsJKeGH29q33FfQihLMKfcwIDAQABo4IC+DCCAvQwEgYDVR0TAQH/BAgwBgEB/wIBADAOBgNV
HQ8BAf8EBAMCAYYwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdp
Y2VydC5jb20wgYEGA1UdHwR6MHgwOqA4oDaGNGh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9EaWdp
Q2VydEFzc3VyZWRJRFJvb3RDQS5jcmwwOqA4oDaGNGh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9E
aWdpQ2VydEFzc3VyZWRJRFJvb3RDQS5jcmwwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwME
MIIBswYDVR0gBIIBqjCCAaYwggGiBgpghkgBhv1sAAIEMIIBkjAoBggrBgEFBQcCARYcaHR0cHM6
Ly93d3cuZGlnaWNlcnQuY29tL0NQUzCCAWQGCCsGAQUFBwICMIIBVh6CAVIAQQBuAHkAIAB1AHMA
ZQAgAG8AZgAgAHQAaABpAHMAIABDAGUAcgB0AGkAZgBpAGMAYQB0AGUAIABjAG8AbgBzAHQAaQB0
AHUAdABlAHMAIABhAGMAYwBlAHAAdABhAG4AYwBlACAAbwBmACAAdABoAGUAIABEAGkAZwBpAEMA
ZQByAHQAIABDAFAALwBDAFAAUwAgAGEAbgBkACAAdABoAGUAIABSAGUAbAB5AGkAbgBnACAAUABh
AHIAdAB5ACAAQQBnAHIAZQBlAG0AZQBuAHQAIAB3AGgAaQBjAGgAIABsAGkAbQBpAHQAIABsAGkA
YQBiAGkAbABpAHQAeQAgAGEAbgBkACAAYQByAGUAIABpAG4AYwBvAHIAcABvAHIAYQB0AGUAZAAg
AGgAZQByAGUAaQBuACAAYgB5ACAAcgBlAGYAZQByAGUAbgBjAGUALjAdBgNVHQ4EFgQU5wIjgABP
2Ne8lAvZP3Q5STI8inkwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZIhvcN
AQELBQADggEBAE7UiSe5/R2Hd34PKAWQ8QovyTs+vZOckMav+pFRhzJUa+jKwXFRXJmOtfrgYhmZ
pgeafBMn2+UCooQS2RX2CkRXxDSPbXMfOtagAT3e44LkRWuy6yX9gF4dOZC+W0L2zpFg4/mgVgxI
EM4zaHvNk6vwastPWA+5e10bBIGepyLiV0kn7pKTCL5pCFMCOi5dyBn0UIBOAtmwXZG0k4f5lpaB
VUCOZu2C2LsoX+1MYe0GWCgZUxFEvEcgKbIEbNiJVJk7ddtneCweknjGVT1YEhEybr1DDE0023vG
QtvsvqubYUwGkuOO3yEqUFcEwGCiNdUknmY3CUnP1fhls+DibsIxggO/MIIDuwIBATB5MGUxCzAJ
BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
b20xJDAiBgNVBAMTG0RpZ2lDZXJ0IFNIQTIgQXNzdXJlZCBJRCBDQQIQDi7WjgxCjxTrYbReNHes
EzANBglghkgBZQMEAgEFAKCCAhcwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0B
CQUxDxcNMTgwMTEwMjEyMzUxWjAvBgkqhkiG9w0BCQQxIgQgVUyHHsXCpo7YytgOOD3s7EBo7530
b8TO6Edo2fcdLa8wgYgGCSsGAQQBgjcQBDF7MHkwZTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDERp
Z2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3LmRpZ2ljZXJ0LmNvbTEkMCIGA1UEAxMbRGlnaUNlcnQg
U0hBMiBBc3N1cmVkIElEIENBAhAOLtaODEKPFOthtF40d6wTMIGKBgsqhkiG9w0BCRACCzF7oHkw
ZTELMAkGA1UEBhMCVVMxFTATBgNVBAoTDERpZ2lDZXJ0IEluYzEZMBcGA1UECxMQd3d3LmRpZ2lj
ZXJ0LmNvbTEkMCIGA1UEAxMbRGlnaUNlcnQgU0hBMiBBc3N1cmVkIElEIENBAhAOLtaODEKPFOth
tF40d6wTMIGTBgkqhkiG9w0BCQ8xgYUwgYIwCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBFjAKBggq
hkiG9w0DBzALBglghkgBZQMEAQIwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAsGCWCG
SAFlAwQCATALBglghkgBZQMEAgMwCwYJYIZIAWUDBAICMAcGBSsOAwIaMA0GCSqGSIb3DQEBAQUA
BIIBALOBR2tncZn/SzleUK2Vu2hlHOTe0fyPuKsHdX2jkBmwKLLDLsP9ouOLJ819c1sR7B84BkQY
wMNb7sK9rQtb8ToTqynb3uNgoehJJw0hYzj2K3jgZtVjS8M+EbrLzjVDLDb4RtL+/trPdkDcJlGM
YyKluqEmSMQC/+cC3xAZLmZjTZs6uXOr3fwZuC02sJT3wPSUytc5mrS4/LO7+Dch4pIEr7RkBTZw
R1X7fyRs7noiCCyQMJ4fjj06aoueQ5N1G5STeHmrlXTGenoY4VSu3UW/wBZvBTaa9YdHlOVSDMUC
GoqEySyt5RlTnp0JVA/MuVs0/mf78Z9O6WLOeJGv6YUAAAAAAAA=

------=_NextPart_000_029A_01D38A1E.A31E99E0--
0
Tim
1/10/2018 9:24:02 PM
mozilla.dev.security.policy 1327 articles. 2 followers. Post Follow

2 Replies
38 Views

Similar Articles

[PageSpeed] 41

On Wednesday, January 10, 2018 at 4:24:54 PM UTC-5, Tim Hollebeek wrote:
=20
> As you know, BR 3.2.5 requires CAs to verify the authenticity of a reques=
t
> for an OV certificate through a Reliable Method of Communication (RMOC).
> Email can be a RMOC, but in these cases, the email address was a construc=
ted
> email address as in BR 3.2.2.4.4.  Despite the fact that these addresses =
are
> standardized in RFC 2142 or elsewhere, we do not believe this meets the
> standard of "verified using a source other than the Applicant
> Representative."

I agree. The intention for the constructed email from BR 3.2.2.4.4 was supp=
osed to be to "confirm the Applicant's control over	the FQDN" and not to pe=
rform the BR 3.2.5 requirement "to verify the authenticity of the Applicant=
 Representative=E2=80=99s certificate request."

I also don't think a CA should use the information from 3.2.2.4.2 (Email, F=
ax, SMS, or Postal Mail) or 3.2.2.4.3 (phone number) to get the BR 3.2.5 au=
thorization. The issue is the CA may end up using the same data to perform =
both 3.2.2.4 and 3.2.5 and will not mitigate the risk that the attacker con=
trols the WHOIS data.

It would be more secure if the CA used two separate methods of communicatio=
n for 3.2.2.4 and 3.2.5.
0
Bruce
1/12/2018 9:58:29 PM
I updated the bugzilla thread (https://bugzilla.mozilla.org/show_bug.cgi?id=
=3D1429639). We ended up revoking 35 certs where we couldn't complete the a=
uthenticity check. I don't think these were actually issued to the wrong or=
ganization. Most of them are foreign, which means getting them on the phone=
 when they already had their cert was pretty difficult.
0
Amus
6/2/2018 6:01:30 AM
Reply: