The sec-approval process makes users safer

Those of you longer in the tooth may remember Firefox was successfully exploited in Pwn2own 2012...and we didn't have to lift a finger to fix it.  We already had -- in the Firefox release shipping days later.  🤦

https://bugzilla.mozilla.org/show_bug.cgi?id=735104 (pwn2own bug)
https://bugzilla.mozilla.org/show_bug.cgi?id=720511 (cover bug, discussion only of a spec-compliance issue)
https://bugzilla.mozilla.org/show_bug.cgi?id=720079 (sec bug noting the sec issue)

We later discussed whether the exploit had been "achieved" by reading our public commits.  https://bugzilla.mozilla.org/show_bug.cgi?id=735104#c2  The fruit of this discussion was our security approval process, where security patches land only after approval, in relative lockstep close to release, with incriminating tests/comments removed.  This is also where sec-approval comment hoop-jumping began.

sec-approval is a pain.  Is it really worth it?

The recent Apple iOS WebKit/JSC exploits conclusively answer "yes".  The Project Zero post https://googleprojectzero.blogspot.com/2019/08/jsc-exploits.html discusses the possibility that some exploits were acquired from published, pre-release security fixes and testcases:

> For many of the exploits it is unclear whether they were originally exploited as 0day or as 1day after a fix had already shipped. It is also unknown how the attackers obtained knowledge of the vulnerabilities in the first place. Generally they could have discovered the vulnerabilities themselves or used public exploits released after a fix had shipped.
>
> Furthermore, at least for WebKit, it is often possible to extract details of a vulnerability from the public source code repository before the fix has been shipped to users.
>
> CVE-2019-8518 https://bugs.chromium.org/p/project-zero/issues/detail?id=1775 can be used to highlight this problem (as can many other recent vulnerabilities). The vulnerability was publicly fixed in WebKit HEAD on Feb 9 2019 with commit <https://github.com/WebKit/webkit/commit/4a23c92e6883b230a437bcc09f94422d7df8756c>. This commit contains a testcase that triggers the issue and causes an out-of-bounds access into a JSArray - a scenario that is usually easy to exploit. However, the fix only shipped to users with the release of iOS 12.2 on March 25 2019, roughly one and a half months after details about the vulnerability were public.
>
> An attacker in possession of a working exploit for an older WebKit vulnerability would likely only need a few days to replace the underlying vulnerability and thus gain the capability to exploit up-to-date devices without the need to find new vulnerabilities themselves. It is likely that this happened for at least some of the following exploits. 

(paragraph breaks added)  Incredibly, saying, "It is likely that [attackers used public commits/testcases to create exploits]" *soft-pedals* it.  The first exploit the post discusses includes this text and image:

> [T]he exploit trigger is almost exactly the same as in the bug report and the regression test file in the WebKit repository. This can be seen in the following two images, the left one showing the testcase published in the WebKit code repository as part of the bugfix and the right showing the part of the in-the-wild exploit code that triggered the bug.
> https://1.bp.blogspot.com/-PEZlVLEefs0/XWg4BdDSxkI/AAAAAAAANUs/ELjHWgzHOZIRKSTV45E-moRivJKrAWIkACLcBGAs/s1600/JSC%2BDIFF.png (alt text: "This image shows a line-by-line, side-by-side comparison between the vulnerability test case from the webkit tracker on the left, and the vulnerability trigger code used in the exploit on the right. They are very similar, with matching variable names and code structure. The only difference is that one value which had the fixed value 1.45 in the test case has been parameterised out to a variable named ow in the exploit, and the trigger has been wrapped inside a function definition which takes ow as an argument.")

The *only* difference between testcase and exploit is s/1.45/ow/.  There's no plausible way that similarity (including variable names) is coincidental.  Attackers (the Chinese government, it seems) copied the testcase into a function, then changed the one bit of the test controlling how a crash happens to make it usefully push-button.

So if you ever questioned whether all that sec-approval nonsense is really necessary...well, it is.  Approvals, syncing up with release timelines, getting rejected because too much risk, it's all pretty obnoxious.  But it's better we're careful for the users who aren't being exploited for one, and it's better than the week WebKit/JSC are probably having now for two.

Jeff
0
Jeff
9/10/2019 12:58:20 AM
mozilla.dev.platform 6557 articles. 0 followers. Post Follow

9 Replies
9 Views

Similar Articles

[PageSpeed] 39

Hi Jeff,

thank you for bringing this up!

Halvar Flake (also formerly of P0) argues here that committing a patch
is not very different from committing the test case:
<http://addxorrol.blogspot.com/2019/08/rashomon-of-disclosure.html>
Which is not something I'm willing to believe in totality.
I think our sec-approval process is useful because it buys us some time.
But it's important to care about timing once a patch is committed. The
fact whether a bug is a sec bug is observable after all.





Am 10.09.19 um 02:58 schrieb Jeff Walden:
> Those of you longer in the tooth may remember Firefox was successfully exploited in Pwn2own 2012...and we didn't have to lift a finger to fix it.  We already had -- in the Firefox release shipping days later.  🤦
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=735104 (pwn2own bug)
> https://bugzilla.mozilla.org/show_bug.cgi?id=720511 (cover bug, discussion only of a spec-compliance issue)
> https://bugzilla.mozilla.org/show_bug.cgi?id=720079 (sec bug noting the sec issue)
> 
> We later discussed whether the exploit had been "achieved" by reading our public commits.  https://bugzilla.mozilla.org/show_bug.cgi?id=735104#c2  The fruit of this discussion was our security approval process, where security patches land only after approval, in relative lockstep close to release, with incriminating tests/comments removed.  This is also where sec-approval comment hoop-jumping began.
> 
> sec-approval is a pain.  Is it really worth it?
> 
> The recent Apple iOS WebKit/JSC exploits conclusively answer "yes".  The Project Zero post https://googleprojectzero.blogspot.com/2019/08/jsc-exploits.html discusses the possibility that some exploits were acquired from published, pre-release security fixes and testcases:
> 
>> For many of the exploits it is unclear whether they were originally exploited as 0day or as 1day after a fix had already shipped. It is also unknown how the attackers obtained knowledge of the vulnerabilities in the first place. Generally they could have discovered the vulnerabilities themselves or used public exploits released after a fix had shipped.
>>
>> Furthermore, at least for WebKit, it is often possible to extract details of a vulnerability from the public source code repository before the fix has been shipped to users.
>>
>> CVE-2019-8518 https://bugs.chromium.org/p/project-zero/issues/detail?id=1775 can be used to highlight this problem (as can many other recent vulnerabilities). The vulnerability was publicly fixed in WebKit HEAD on Feb 9 2019 with commit <https://github.com/WebKit/webkit/commit/4a23c92e6883b230a437bcc09f94422d7df8756c>. This commit contains a testcase that triggers the issue and causes an out-of-bounds access into a JSArray - a scenario that is usually easy to exploit. However, the fix only shipped to users with the release of iOS 12.2 on March 25 2019, roughly one and a half months after details about the vulnerability were public.
>>
>> An attacker in possession of a working exploit for an older WebKit vulnerability would likely only need a few days to replace the underlying vulnerability and thus gain the capability to exploit up-to-date devices without the need to find new vulnerabilities themselves. It is likely that this happened for at least some of the following exploits. 
> 
> (paragraph breaks added)  Incredibly, saying, "It is likely that [attackers used public commits/testcases to create exploits]" *soft-pedals* it.  The first exploit the post discusses includes this text and image:
> 
>> [T]he exploit trigger is almost exactly the same as in the bug report and the regression test file in the WebKit repository. This can be seen in the following two images, the left one showing the testcase published in the WebKit code repository as part of the bugfix and the right showing the part of the in-the-wild exploit code that triggered the bug.
>> https://1.bp.blogspot.com/-PEZlVLEefs0/XWg4BdDSxkI/AAAAAAAANUs/ELjHWgzHOZIRKSTV45E-moRivJKrAWIkACLcBGAs/s1600/JSC%2BDIFF.png (alt text: "This image shows a line-by-line, side-by-side comparison between the vulnerability test case from the webkit tracker on the left, and the vulnerability trigger code used in the exploit on the right. They are very similar, with matching variable names and code structure. The only difference is that one value which had the fixed value 1.45 in the test case has been parameterised out to a variable named ow in the exploit, and the trigger has been wrapped inside a function definition which takes ow as an argument.")
> 
> The *only* difference between testcase and exploit is s/1.45/ow/.  There's no plausible way that similarity (including variable names) is coincidental.  Attackers (the Chinese government, it seems) copied the testcase into a function, then changed the one bit of the test controlling how a crash happens to make it usefully push-button.
> 
> So if you ever questioned whether all that sec-approval nonsense is really necessary...well, it is.  Approvals, syncing up with release timelines, getting rejected because too much risk, it's all pretty obnoxious.  But it's better we're careful for the users who aren't being exploited for one, and it's better than the week WebKit/JSC are probably having now for two.
> 
> Jeff
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 
0
Frederik
9/10/2019 7:53:58 AM
On Mon, Sep 9, 2019 at 6:01 PM Jeff Walden <jwalden@mit.edu> wrote:

> Those of you longer in the tooth may remember Firefox was successfully
> exploited in Pwn2own 2012...and we didn't have to lift a finger to fix it=
..
> We already had -- in the Firefox release shipping days later.  =F0=9F=A4=
=A6
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=3D735104 (pwn2own bug)
> https://bugzilla.mozilla.org/show_bug.cgi?id=3D720511 (cover bug,
> discussion only of a spec-compliance issue)
> https://bugzilla.mozilla.org/show_bug.cgi?id=3D720079 (sec bug noting the
> sec issue)
>
> We later discussed whether the exploit had been "achieved" by reading our
> public commits.  https://bugzilla.mozilla.org/show_bug.cgi?id=3D735104#c2
> The fruit of this discussion was our security approval process, where
> security patches land only after approval, in relative lockstep close to
> release, with incriminating tests/comments removed.  This is also where
> sec-approval comment hoop-jumping began.


How often do we go back and land those tests and comments after the fix has
been in the release builds for a suitable amount of time?
0
Dave
9/10/2019 2:55:37 PM
On 9/10/19 7:55 AM, Dave Townsend wrote:
> How often do we go back and land those tests and comments after the fix has
> been in the release builds for a suitable amount of time?

I always land my tests...at some point.  I don't know  if everyone else adequately remembers to do so.  We don't formally or systematically track this.

I've argued for years that we should have something like a private repository for security fixes to allow us to actively test security patches (including their tests) without publicly exposing them.  (The resulting builds would be undistributable, but they could be tested.)  But it'd take a ton of infra work to have private repository/treeherder/etc. only accessible behind LDAP or something, so it's easier said than done.

Personally, I just keep tests in bookmarks that I continually rebase.  Eventually I'll conclude a bookmark is safely landable (where "landable" depends on how likely tests are to bitrot, how long the security issue was in place, the earliest release that received a backport, &c.) and then land it.  And because each test is staring at me every time I |hg wip|, I won't lose track of them forever.

Jeff
0
Jeff
9/10/2019 4:03:38 PM
On Tue, Sep 10, 2019 at 4:55 PM Dave Townsend <dtownsend@mozilla.com> wrote=
:

> On Mon, Sep 9, 2019 at 6:01 PM Jeff Walden <jwalden@mit.edu> wrote:
>
> > Those of you longer in the tooth may remember Firefox was successfully
> > exploited in Pwn2own 2012...and we didn't have to lift a finger to fix
> it.
> > We already had -- in the Firefox release shipping days later.  =F0=9F=
=A4=A6
> >
> > https://bugzilla.mozilla.org/show_bug.cgi?id=3D735104 (pwn2own bug)
> > https://bugzilla.mozilla.org/show_bug.cgi?id=3D720511 (cover bug,
> > discussion only of a spec-compliance issue)
> > https://bugzilla.mozilla.org/show_bug.cgi?id=3D720079 (sec bug noting t=
he
> > sec issue)
> >
> > We later discussed whether the exploit had been "achieved" by reading o=
ur
> > public commits.  https://bugzilla.mozilla.org/show_bug.cgi?id=3D735104#=
c2
> > The fruit of this discussion was our security approval process, where
> > security patches land only after approval, in relative lockstep close t=
o
> > release, with incriminating tests/comments removed.  This is also where
> > sec-approval comment hoop-jumping began.
>
>
> How often do we go back and land those tests and comments after the fix h=
as
> been in the release builds for a suitable amount of time?
>

In theory, you should set the in-testsuite? flag when you land without the
test, and then when the bug is opened, land the test, but in practice I
don't think anybody makes sure that happens. I feel like I've seen one or
two cases over the years where we fixed some sec issue, no test landed,
then much later we regressed it.

_______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
0
Andrew
9/10/2019 4:10:50 PM
Seems like it ought to be straightforward to do something to cause
in-testsuite? flags to send mail occasionally, or show up on some
dashboard, or...

Dan

Am Di., 10. Sept. 2019 um 09:11 Uhr schrieb Andrew McCreight
<amccreight@mozilla.com>:
>
> On Tue, Sep 10, 2019 at 4:55 PM Dave Townsend <dtownsend@mozilla.com> wro=
te:
>
> > On Mon, Sep 9, 2019 at 6:01 PM Jeff Walden <jwalden@mit.edu> wrote:
> >
> > > Those of you longer in the tooth may remember Firefox was successfull=
y
> > > exploited in Pwn2own 2012...and we didn't have to lift a finger to fi=
x
> > it.
> > > We already had -- in the Firefox release shipping days later.  =F0=9F=
=A4=A6
> > >
> > > https://bugzilla.mozilla.org/show_bug.cgi?id=3D735104 (pwn2own bug)
> > > https://bugzilla.mozilla.org/show_bug.cgi?id=3D720511 (cover bug,
> > > discussion only of a spec-compliance issue)
> > > https://bugzilla.mozilla.org/show_bug.cgi?id=3D720079 (sec bug noting=
 the
> > > sec issue)
> > >
> > > We later discussed whether the exploit had been "achieved" by reading=
 our
> > > public commits.  https://bugzilla.mozilla.org/show_bug.cgi?id=3D73510=
4#c2
> > > The fruit of this discussion was our security approval process, where
> > > security patches land only after approval, in relative lockstep close=
 to
> > > release, with incriminating tests/comments removed.  This is also whe=
re
> > > sec-approval comment hoop-jumping began.
> >
> >
> > How often do we go back and land those tests and comments after the fix=
 has
> > been in the release builds for a suitable amount of time?
> >
>
> In theory, you should set the in-testsuite? flag when you land without th=
e
> test, and then when the bug is opened, land the test, but in practice I
> don't think anybody makes sure that happens. I feel like I've seen one or
> two cases over the years where we fixed some sec issue, no test landed,
> then much later we regressed it.
>
> _______________________________________________
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
0
Dan
9/10/2019 4:16:32 PM
On 9/10/19 12:16 PM, Dan Mosedale wrote:
> Seems like it ought to be straightforward to do something to cause
> in-testsuite? flags to send mail occasionally, or show up on some
> dashboard, or...

Could be worth it.

I just checked, and there are currently 826 bugs that have 
"in-testsuite?" set on them where I am the flag requester.  Clearly that 
list has not been seeing much love...

-Boris
0
Boris
9/10/2019 4:30:16 PM
On 9/10/19 12:30 PM, Boris Zbarsky wrote:
> I just checked, and there are currently 826 bugs that have 
> "in-testsuite?" set on them where I am the flag requester.

And overall there seem to be ~7300 bugs that have that flag set.

-Boris
0
Boris
9/10/2019 4:31:58 PM
On Tue, Sep 10, 2019 at 9:35 AM Boris Zbarsky <bzbarsky@mit.edu> wrote:

> On 9/10/19 12:30 PM, Boris Zbarsky wrote:
> > I just checked, and there are currently 826 bugs that have
> > "in-testsuite?" set on them where I am the flag requester.
>
> And overall there seem to be ~7300 bugs that have that flag set.
>

Other groups must be using that flag also: e.g. non-security bug 1190 from
1998 has that flag. "Only" 718 fixed bugs with a sec- keyword have that
flag.

-Dan Veditz
0
Daniel
9/10/2019 7:53:07 PM
On 9/10/19 3:53 PM, Daniel Veditz wrote:
> Other groups must be using that flag also

Sure, it just means "this needs a test checked in".

> "Only" 718 fixed bugs with a sec- keyword have that flag.

A mere trifle!  ;)

-Boris
0
Boris
9/10/2019 8:46:45 PM
Reply: