Telia CA - incorrect OID value

Indident reports:

ERROR IN DV OID VALUE (deviation 4)

How Telia became aware:
Telia got preliminary CA audit report on 25th June 2018. One of its BR devi=
ations was a finding that "17 Telia DV certificates had incorrectly same OI=
D value that was used for Telia OV certificates."=20

Timeline of actions:
On the same day Telia fixed the OID value into DV profile so that error won=
't happen again. Telia's opinion is that the incorrect OID value has no imp=
act on any common system but anyway Telia's plan is to revoke all incorrect=
 certificates ASAP and latest at September 2018. Customers need to replace =
their original incorrect certificate with a new certificate provided by Tel=
ia. Telia will update this bug until all incorrect certificates are revoked=
..

Summary and details of problematic certificates:
About ~300 of Telia DV certificates for a single pilot DV Customer included=
 OV OID 2.23.140.1.2.2 instead of DV OID 2.23.140.1.2.1. All incorrect ones=
 were enrolled between 20-Mar-2018 and 25-Jun-2018. All are logged to CT an=
d can be found using given dates and issuer "Telia Domain Validation SSL CA=
 v1". Certificates are also available in Telia CA database.

Explanation about how and why the mistakes were made or bugs introduced, an=
d how they avoided detection until now:
Telia CA started to enroll DV SSL certificates in March 2018. Previously al=
l Telia's SSL certificates were OV SSL certificates. The new certificate ty=
pe was basically sub-type of Telia OV certificate but with fewer subject fi=
elds. Its profile was copied from OV and then modified, tested and piloted =
but still there was this error in the OID value that was undetected because=
 it won't have any effect anywhere and was commonly used by Telia before.

Steps to fix:
1. fix the DV profile; DONE 25-Jun-2018, no errors occurred after that
2. reproduce all incorrect certificates any provide those to Customer; ONGO=
ING, planned to finnish 30-Sep-2018
3. revoke all incorrect ones; ONGOING, planned to finnish 30-Sep-2018
4. Telia CA decided to improve its testing process to avoid similar errors =
in the future; DONE 6-Jul-2018
0
pekka
8/3/2018 8:51:24 AM
mozilla.dev.security.policy 1337 articles. 2 followers. Post Follow

3 Replies
36 Views

Similar Articles

[PageSpeed] 10

Am Freitag, 3. August 2018 10:51:34 UTC+2 schrieb pekka.la...@teliasonera.c=
om:
> Steps to fix:
> 1. fix the DV profile; DONE 25-Jun-2018, no errors occurred after that
> 2. reproduce all incorrect certificates any provide those to Customer; ON=
GOING, planned to finnish 30-Sep-2018
> 3. revoke all incorrect ones; ONGOING, planned to finnish 30-Sep-2018
> 4. Telia CA decided to improve its testing process to avoid similar error=
s in the future; DONE 6-Jul-2018

Why will the detecting/revocation process take 2 months? I think this is an=
 extremely long time period for misissued certificates (I personally think =
a wrong OID is a misissurance). Also, shouldn't it be quite easy to find al=
l wrong certificates with something like an script or query, or does it req=
uire a lot of human intervention?
0
Lewis
8/5/2018 10:07:40 PM
Telia got a serious lesson with this incident that should not have happened=
.. Important detail also to know is that certificates were not issued to wro=
ng entities and issuing new certificates with wrong OID field was prevented=
 immediately.

1) Telia has a development process with multiple steps when doing a change =
to SSL process. Some steps of the process include creating test certificate=
s in test and pre-production systems with documented change plan and a revi=
ew. Unfortunately test certificates were using test OID values so that the =
problem couldn=E2=80=99t be detected at test side. Telia has analysed reaso=
ns that caused this error. The main reason was not adequately implemented t=
esting. Test process didn't include certificate comparison correctly agains=
t so-called model certificate. Telia has model certificates for each certif=
icate type that are used in comparison when any certificate profile changes=
.. This time there wasn't DV model certificate at all (except in test system=
 with test OID) because DV was a completely new certificate type for Telia.=
 OV model certificate (that had OV OID value) was used instead by the revie=
wers. Telia should have created a DV model certificate at first. In model c=
ertificate creation there are several eye pairs including senior developers=
 when accepting a new one. As a resolution Telia has now enhanced processes=
 so that it is mandatory to create model certificate when completely new ce=
rtificate type is created.
2) We have concluded that the main reason for this problem was not a lack o=
f training but the incomplete test process and documentation. CA audits hav=
e annually evaluated Telia training. Recommendations about improvements hav=
e been documented into our internal audit reports if necessary. Recommendat=
ions (or issues) from CA auditors are always added to Telia Security Plan t=
o improve Telia CA process continuously. Persons involved in the review hav=
e got many different types of training that vary from general security to d=
eeper CA software related trainings. E.g. recently Feisty Duck =E2=80=93 Th=
e Best TLS training in the World and several trainings from our CA Vendor.
a) CA software vendor trainings have been held quarterly.
b) Vendor keep the materials up to date and we update our own training mate=
rials annually or when needed
c) CA audits have annually evaluated Telia internal trainings to Registrati=
on officers

3) When this problem was discovered the issue was immediately escalated acc=
ording to Telia Incident process. One of the main steps of Telia incident p=
rocess is to evaluate the effect of the incident. This time the evaluation =
result was that no immediate risk was caused as OID was correct OV OID, all=
 certificates with wrong OID fields were issued to known Telia hosted servi=
ce customers, even though the issue itself was confirmed serious. Telia pre=
vented the issue to repeat by fixing the profile on the same day. As none o=
f these certificates were issued to wrong entities Telia decided to do the =
replacement/revoking in the organized way that overall harm would be minima=
l by replacing these certificates with the customers.


Telia will revoke all incorrect certificates. Most of them have already bee=
n revoked. Today ~100 is left to be revoked in co-operation with the Custom=
er.
0
pekka
8/8/2018 10:34:43 AM
Thanks! I think this is more in line with the goal of these discussions -
trying to learn, share, and disseminate best practices.

Here, the best practice is that, prior to any configuration, the CA should
determine what the 'model' certificate should look like. This model
certificate is, in effect, the technical equivalent of their certificate
profile (e.g. 7.1 of a CA's CP/CPS) - indeed, it might even make sense for
CAs to include their 'model certificates' as appendices in their CP/CPS,
which helps ensure that the CP/CPS is updated whenever the profile is
updated, but also ensures there's a technically verifiable examination of
the profile.

Going further, it might make sense for CAs to share their model
certificates in advance, for community review and evaluation - although
we're not quite there yet, it could potentially help identify or mitigate
issues before hand, as well as help the CA ensure it's considered
everything in the profile.

Similarly, examining these model certificates through linting is another
thing to consider, and to compare these against the test certificates'
linted results. One thing to consider with model certificates is every
configurable option you allow (e.g. key size) can create different model
certificates, so as a testing procedure, you'll want to make sure you have
model certificates for every configurable option, as well as consider test
certificates for various permutations. For example, lets say you're
introducing a new subject attribute to a certificate - as part of
developing your model certificate, and your test certificate, you'll likely
want to examine the various constraints on that field (e.g. length of
field, acceptable characters), and run tests to make sure they produce the
correct and expected results. Consider situations like "all whitespace" -
does it do the expected thing (which could be to omit the field and allow
issuance, or prevent issuance, etc).

As far as training goes, it does sound like there is an opportunity for
routine training regarding changes to the BRs (and relevant RFCs), to make
sure the team constructing and reviewing profiles know what is and is not
acceptable. While it's good to examine the policies for RAs, looking more
holistically, you want to make sure the team tasked with creating and
reviewing these models is given adequate support and training to know - and
critically evaluate - what is or isn't permitted.

On Wed, Aug 8, 2018 at 6:34 AM, pekka.lahtiharju--- via dev-security-policy
<dev-security-policy@lists.mozilla.org> wrote:

> Telia got a serious lesson with this incident that should not have
> happened. Important detail also to know is that certificates were not
> issued to wrong entities and issuing new certificates with wrong OID fiel=
d
> was prevented immediately.
>
> 1) Telia has a development process with multiple steps when doing a chang=
e
> to SSL process. Some steps of the process include creating test
> certificates in test and pre-production systems with documented change pl=
an
> and a review. Unfortunately test certificates were using test OID values =
so
> that the problem couldn=E2=80=99t be detected at test side. Telia has ana=
lysed
> reasons that caused this error. The main reason was not adequately
> implemented testing. Test process didn't include certificate comparison
> correctly against so-called model certificate. Telia has model certificat=
es
> for each certificate type that are used in comparison when any certificat=
e
> profile changes. This time there wasn't DV model certificate at all (exce=
pt
> in test system with test OID) because DV was a completely new certificate
> type for Telia. OV model certificate (that had OV OID value) was used
> instead by the reviewers. Telia should have created a DV model certificat=
e
> at first. In model certificate creation there are several eye pairs
> including senior developers when accepting a new one. As a resolution Tel=
ia
> has now enhanced processes so that it is mandatory to create model
> certificate when completely new certificate type is created.
> 2) We have concluded that the main reason for this problem was not a lack
> of training but the incomplete test process and documentation. CA audits
> have annually evaluated Telia training. Recommendations about improvement=
s
> have been documented into our internal audit reports if necessary.
> Recommendations (or issues) from CA auditors are always added to Telia
> Security Plan to improve Telia CA process continuously. Persons involved =
in
> the review have got many different types of training that vary from gener=
al
> security to deeper CA software related trainings. E.g. recently Feisty Du=
ck
> =E2=80=93 The Best TLS training in the World and several trainings from o=
ur CA
> Vendor.
> a) CA software vendor trainings have been held quarterly.
> b) Vendor keep the materials up to date and we update our own training
> materials annually or when needed
> c) CA audits have annually evaluated Telia internal trainings to
> Registration officers
>
> 3) When this problem was discovered the issue was immediately escalated
> according to Telia Incident process. One of the main steps of Telia
> incident process is to evaluate the effect of the incident. This time the
> evaluation result was that no immediate risk was caused as OID was correc=
t
> OV OID, all certificates with wrong OID fields were issued to known Telia
> hosted service customers, even though the issue itself was confirmed
> serious. Telia prevented the issue to repeat by fixing the profile on the
> same day. As none of these certificates were issued to wrong entities Tel=
ia
> decided to do the replacement/revoking in the organized way that overall
> harm would be minimal by replacing these certificates with the customers.
>
>
> Telia will revoke all incorrect certificates. Most of them have already
> been revoked. Today ~100 is left to be revoked in co-operation with the
> Customer.
> _______________________________________________
> dev-security-policy mailing list
> dev-security-policy@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>
0
Ryan
8/8/2018 3:23:26 PM
Reply: