MinGW Target on TaskCluster

As part of our work with Tor, we=E2=80=99ve been working on getting a MinGW=
-based
build of Windows into TaskCluster. Tor is currently using ESR releases, and
every ESR they have to go through a large amount of work to get the build
working under MinGW again; by continually building (and testing) that build
we=E2=80=99ll be able to cut weeks to months of effort for them each ESR re=
lease.
(Not breaking the MinGW build is also a necessity if they were ever to move
off ESRs.)

Unlike our normal Windows builds with msvc or clang-cl, this is a
cross-compile on Linux for Windows, as well as the first time in a long
time we=E2=80=99ve built for Windows with gcc. Building with gcc for Window=
s has
exposed a number of new warnings
<https://bugzilla.mozilla.org/show_bug.cgi?id=3D1394433> to clean up, as we=
ll
as some C++ spec incompatibilities that MSVC (and clang-cl) accept. All of
the breaks have been resolved; and a lot of the warnings and similar are
either resolved or in-progress.

Effective this weekend, MinGW is on Treeherder, with the single build
target win32-mingw32 (debug). Its toolchain builds are under TMW (with a
few under TL because they are generic Linux packages.)

The one-click loaner system works for MinGW, and we=E2=80=99re going to all=
ow this
to bake at Tier 2 for a while. Your help in keeping the build green is
greatly appreciated, and if you hit a MinGW bug you can=E2=80=99t decipher,=
 I=E2=80=99ll be
happy to help, just send me an email or a needinfo. Some of the most common
things that cause errors for MinGW:


 - #include <Windows.h> instead of #include <windows.h> and similar

 - _uuidof() instead of IID_* ref
<https://stackoverflow.com/questions/18606546/how-to-use-uuidof-in-mingw>

 - Casting nullptr to bool or int

 - Using the (in)correct assembly code

 - Breaking --disable-webrtc, --disable-sandbox, or --disable-accessibility

 - MinGW lagging behind on defining new constants Microsoft has defined

There are a few things left outstanding
<https://bugzilla.mozilla.org/showdependencytree.cgi?id=3D1330608&hide_reso=
lved=3D1>
for the MinGW build, including webrtc
<https://bugzilla.mozilla.org/show_bug.cgi?id=3D1393901>, stylo
<https://bugzilla.mozilla.org/show_bug.cgi?id=3D1390583> (which may be the
most difficult in this list), the sandbox
<https://bugzilla.mozilla.org/show_bug.cgi?id=3D1230910>,
--enable-accessibility
<https://sourceforge.net/p/mingw-w64/bugs/648/>, cleaning
up warnings <https://bugzilla.mozilla.org/show_bug.cgi?id=3D1394433>, and r=
unning
tests <https://bugzilla.mozilla.org/show_bug.cgi?id=3D1395047>. In the
future, there is hope (by me at least) to throw all of this work away.
Compiling on Linux for Windows with clang (or clang-cl) would probably be a
significant improvement for Tor=E2=80=99s builds, and would enable them to =
take
advantage of CFI and SEH, as well as simplifying the number of platforms
Mozilla supports. Chrome has been pushing towards this
<https://bugs.chromium.org/p/chromium/issues/detail?id=3D495204>, so we=E2=
=80=99re
keeping an eye on their progress.

Finally, I want to thank everyone who brought this long process to this
point: everyone who wrote patches, explained things (sometimes over and
over again), and reviewed. That=E2=80=99s including but not limited to: bok=
lm,
dmajor, froydnj, glandium, georg, jacek, jrmuizel, and ted; plus the dozen
or so more people who have reviewed my patches all over the codebase.


-tom
0
Tom
10/9/2017 5:31:37 AM
mozilla.dev.platform 6367 articles. 0 followers. Post Follow

7 Replies
55 Views

Similar Articles

[PageSpeed] 47

Am 09.10.2017 um 07:31 schrieb Tom Ritter:
> As part of our work with Tor, we’ve been working on getting a MinGW-based
> build of Windows into TaskCluster. 

A maybe too obvious question from the side lines: Why is the Tor browser
cross-compiled and not using MSVC?

Philipp
0
Philipp
10/9/2017 3:31:52 PM
On Mon, Oct 9, 2017 at 10:31 AM, Philipp Wagner <news@philipp-wagner.com>
wrote:

> Am 09.10.2017 um 07:31 schrieb Tom Ritter:
> > As part of our work with Tor, we=E2=80=99ve been working on getting a M=
inGW-based
> > build of Windows into TaskCluster.
>
> A maybe too obvious question from the side lines: Why is the Tor browser
> cross-compiled and not using MSVC?
>
>
Building on Linux allows Tor Browser (including its entire toolchain and
dependencies) to be built deterministically and thus reproducibly using an
entirely open source toolchain. (There are a few small exceptions but
they're quite small.)

Reproducible builds significantly reduce the trust inherent in an
organization's build infrastructure and when recreated independently ensure
that nothing unexpected was inserted into the final executable. Having the
entire toolchain open source ensures that anyone who wants to inspect the
code or reason about its behavior can do so. (And as I've learned in the
past year Tor actually has a good amount of anonymous contributors reading
the source code of its toolchain and reporting things.)

You can read more about the philosophy behind this movement at
https://blog.torproject.org/deterministic-builds-part-one-cyberwar-and-glob=
al-compromise
https://reproducible-builds.org/
CCleaner is a good example of attackers backdooring compiled software.

The next step, past reproducible builds, is Binary Transparency, which
ensures that before an update is applied, it is publicly known, so
attackers cannot surreptitiously subvert the update mechanism.  Tor is
exploring some avenues there. FLAME is a good example of attacking the
update mechanism.

I would describe Mozilla as 'curious' about reproducible builds (see
https://bugzilla.mozilla.org/show_bug.cgi?id=3D885777); but we are actively
working on Binary Transparency (see
https://bugzilla.mozilla.org/show_bug.cgi?id=3D1341395).

-tom
0
Tom
10/9/2017 4:14:26 PM
I'd like to follow up on this old thread to discuss what we can do about
improving the mingw developer experience for people doing Windows-centric
stuff.

I've had several patches backed out over the past month because of mingw
header files differing in case from the headers distributed by the official
Windows SDK. Frankly, it's a waste of developer time to be dealing with
this. This is particularly frustrating in the case of autoland, where we
can't just push our own follow-up patches.

Those of us who are trying to do Windows-specific stuff are effectively
stuck simultaneously coding to two similar, yet still different, SDKs,
being built across two distinct operating systems that differ in case
sensitivity.

In some cases, discrepancies between the SDKs are inevitable: Perhaps mingw
hasn't kept up with the official Windows SDK and some headers/APIs are
missing. We just have to take those as they come.

But when it comes to the header case mismatch issue, we could automate
this. Would it be possible for us to write some code to automatically
detect these case discrepancies and automatically land corrections? Where
would be the right place for something like that to live?

Those of us locally building on Windows already have enough annoyances with
the build as-is, so I don't consider "you should be building locally on
both Windows/MSVC and Linux/mingw simultaneously" to be a reasonable
option. Nor do I consider always pushing mingw jobs to try to be ideal: it
delays the landing of patches and potentially wastes build machine time,
compared to an automated lint.

Furthermore, can we clarify whether mingw is currently a tier 1 or tier 2
platform? Earlier in this thread, mingw was denoted as tier 2 (and it still
shows up as tier 2 on treeherder), yet sheriffs are backing out patches
when mingw bustage occurs. Tor is important, so of course we want to make
our best effort to ensure that mingw isn't broken, but this "tier 2 but
really tier 1" state is not helpful.

I apologize for being curt, but this is becoming rather frustrating.

Aaron


On Mon, Oct 9, 2017 at 10:14 AM, Tom Ritter <tom@mozilla.com> wrote:

> On Mon, Oct 9, 2017 at 10:31 AM, Philipp Wagner <news@philipp-wagner.com>
> wrote:
>
> > Am 09.10.2017 um 07:31 schrieb Tom Ritter:
> > > As part of our work with Tor, we=E2=80=99ve been working on getting a
> MinGW-based
> > > build of Windows into TaskCluster.
> >
> > A maybe too obvious question from the side lines: Why is the Tor browse=
r
> > cross-compiled and not using MSVC?
> >
> >
> Building on Linux allows Tor Browser (including its entire toolchain and
> dependencies) to be built deterministically and thus reproducibly using a=
n
> entirely open source toolchain. (There are a few small exceptions but
> they're quite small.)
>
> Reproducible builds significantly reduce the trust inherent in an
> organization's build infrastructure and when recreated independently ensu=
re
> that nothing unexpected was inserted into the final executable. Having th=
e
> entire toolchain open source ensures that anyone who wants to inspect the
> code or reason about its behavior can do so. (And as I've learned in the
> past year Tor actually has a good amount of anonymous contributors readin=
g
> the source code of its toolchain and reporting things.)
>
> You can read more about the philosophy behind this movement at
> https://blog.torproject.org/deterministic-builds-part-one-
> cyberwar-and-global-compromise
> https://reproducible-builds.org/
> CCleaner is a good example of attackers backdooring compiled software.
>
> The next step, past reproducible builds, is Binary Transparency, which
> ensures that before an update is applied, it is publicly known, so
> attackers cannot surreptitiously subvert the update mechanism.  Tor is
> exploring some avenues there. FLAME is a good example of attacking the
> update mechanism.
>
> I would describe Mozilla as 'curious' about reproducible builds (see
> https://bugzilla.mozilla.org/show_bug.cgi?id=3D885777); but we are active=
ly
> working on Binary Transparency (see
> https://bugzilla.mozilla.org/show_bug.cgi?id=3D1341395).
>
> -tom
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
0
Aaron
2/6/2018 11:36:50 PM
Yes, let's fix it.

I'm very grateful to you and everyone else who has kept the MinGW
build running; and I'd like to keep you... well as happy as you can be
while supporting an esoteric build target that's inconvenient for you.

With regards to backouts, when sheriffs have asked, my response has
been that if something is directly attributable to a patch and it
doesn't look like it's a MinGW problem (like the recent missing
dwrite_3.h issue) - I'd appreciate a backout. This is mostly
selfishness, the times there was no backout and MinGW starts
perma-failing it's been up to me to diagnose the issue and fix it;
only once have MinGW patches been needed. If that needs to change,
alright.


As far as what we can do to make it easier. I think/hope MinGW isn't a
strain on TC; there are no tests associated, it even skips the normal
check-tests that run for other build jobs; and it runs on a small
Linux box so it should be the cheapest option for us. It could build
faster though; sccache doesn't work for it and I haven't been able to
figure out why (Bug 1411212).

> But when it comes to the header case mismatch issue, we could automate th=
is.
> Would it be possible for us to write some code to automatically detect th=
ese case
> discrepancies and automatically land corrections? Where would be the righ=
t place
> for something like that to live?

I'd be happy to build something that does this. I don't know what's
possible or where it should live though. It's probably easy for me to
add it to the lint job, wherever that is.  I might be able to
integrate it into the new MozReview Static Analysis thing. I don't
know if it's possible, or how, to automatically land corrections* but
maybe?



Also, not that this is a solution for today but as I mentioned in the
first email - my hope is that some of this becomes easier with a move
to clang. It won't address everything, but I hope at least some of the
MinGW pain is lessened by a mingw-clang option. I could see us
patching our mingw-clang build to work with case insensitive includes
for example; and the __try/__except issues we have would go away also.
There's a few paths forward on this front. One using a mingw-clang,
one using clang-cl on Linux. The work on the first one is happening in
Bug 1390583.

-tom

* and I am nervous about giving some new system the ability to
automatically land code

On Tue, Feb 6, 2018 at 5:36 PM, Aaron Klotz <aklotz@mozilla.com> wrote:
> I'd like to follow up on this old thread to discuss what we can do about
> improving the mingw developer experience for people doing Windows-centric
> stuff.
>
> I've had several patches backed out over the past month because of mingw
> header files differing in case from the headers distributed by the offici=
al
> Windows SDK. Frankly, it's a waste of developer time to be dealing with
> this. This is particularly frustrating in the case of autoland, where we
> can't just push our own follow-up patches.
>
> Those of us who are trying to do Windows-specific stuff are effectively
> stuck simultaneously coding to two similar, yet still different, SDKs, be=
ing
> built across two distinct operating systems that differ in case sensitivi=
ty.
>
> In some cases, discrepancies between the SDKs are inevitable: Perhaps min=
gw
> hasn't kept up with the official Windows SDK and some headers/APIs are
> missing. We just have to take those as they come.
>
> But when it comes to the header case mismatch issue, we could automate th=
is.
> Would it be possible for us to write some code to automatically detect th=
ese
> case discrepancies and automatically land corrections? Where would be the
> right place for something like that to live?
>
> Those of us locally building on Windows already have enough annoyances wi=
th
> the build as-is, so I don't consider "you should be building locally on b=
oth
> Windows/MSVC and Linux/mingw simultaneously" to be a reasonable option. N=
or
> do I consider always pushing mingw jobs to try to be ideal: it delays the
> landing of patches and potentially wastes build machine time, compared to=
 an
> automated lint.
>
> Furthermore, can we clarify whether mingw is currently a tier 1 or tier 2
> platform? Earlier in this thread, mingw was denoted as tier 2 (and it sti=
ll
> shows up as tier 2 on treeherder), yet sheriffs are backing out patches w=
hen
> mingw bustage occurs. Tor is important, so of course we want to make our
> best effort to ensure that mingw isn't broken, but this "tier 2 but reall=
y
> tier 1" state is not helpful.
>
> I apologize for being curt, but this is becoming rather frustrating.
>
> Aaron
>
>
> On Mon, Oct 9, 2017 at 10:14 AM, Tom Ritter <tom@mozilla.com> wrote:
>>
>> On Mon, Oct 9, 2017 at 10:31 AM, Philipp Wagner <news@philipp-wagner.com=
>
>> wrote:
>>
>> > Am 09.10.2017 um 07:31 schrieb Tom Ritter:
>> > > As part of our work with Tor, we=E2=80=99ve been working on getting =
a
>> > > MinGW-based
>> > > build of Windows into TaskCluster.
>> >
>> > A maybe too obvious question from the side lines: Why is the Tor brows=
er
>> > cross-compiled and not using MSVC?
>> >
>> >
>> Building on Linux allows Tor Browser (including its entire toolchain and
>> dependencies) to be built deterministically and thus reproducibly using =
an
>> entirely open source toolchain. (There are a few small exceptions but
>> they're quite small.)
>>
>> Reproducible builds significantly reduce the trust inherent in an
>> organization's build infrastructure and when recreated independently
>> ensure
>> that nothing unexpected was inserted into the final executable. Having t=
he
>> entire toolchain open source ensures that anyone who wants to inspect th=
e
>> code or reason about its behavior can do so. (And as I've learned in the
>> past year Tor actually has a good amount of anonymous contributors readi=
ng
>> the source code of its toolchain and reporting things.)
>>
>> You can read more about the philosophy behind this movement at
>>
>> https://blog.torproject.org/deterministic-builds-part-one-cyberwar-and-g=
lobal-compromise
>> https://reproducible-builds.org/
>> CCleaner is a good example of attackers backdooring compiled software.
>>
>> The next step, past reproducible builds, is Binary Transparency, which
>> ensures that before an update is applied, it is publicly known, so
>> attackers cannot surreptitiously subvert the update mechanism.  Tor is
>> exploring some avenues there. FLAME is a good example of attacking the
>> update mechanism.
>>
>> I would describe Mozilla as 'curious' about reproducible builds (see
>> https://bugzilla.mozilla.org/show_bug.cgi?id=3D885777); but we are activ=
ely
>> working on Binary Transparency (see
>> https://bugzilla.mozilla.org/show_bug.cgi?id=3D1341395).
>>
>> -tom
>> _______________________________________________
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>
>
0
Tom
2/7/2018 2:27:04 AM
On 2/6/2018 6:36 PM, Aaron Klotz wrote:
> Furthermore, can we clarify whether mingw is currently a tier 1 or tier 2
> platform? Earlier in this thread, mingw was denoted as tier 2 (and it still
> shows up as tier 2 on treeherder), yet sheriffs are backing out patches
> when mingw bustage occurs. Tor is important, so of course we want to make
> our best effort to ensure that mingw isn't broken, but this "tier 2 but
> really tier 1" state is not helpful.

Tier 2 job bustage shouldn't lead to an automatic backout under normal 
circumstances. Normal practice would be to file a bug for the bustage 
with a NI directed to the author of the patch that caused it. If you've 
been backed out without any such notice, feel free to raise your 
concerns directly with either myself or Sebastian.

Note that the sheriffing team does reserve the right to backout a patch 
for Tier 2 bustage if it's not addressed in a reasonable time frame 
(Tier 2 != unowned), but that would typically only happen in the 
situation where a bug Was filed and saw no further activity afterwards.

-Ryan
0
Ryan
2/7/2018 2:43:05 AM
On 2/6/18 6:27 PM, Tom Ritter wrote:
>> But when it comes to the header case mismatch issue, we could automate this.
>> Would it be possible for us to write some code to automatically detect these case
>> discrepancies and automatically land corrections? Where would be the right place
>> for something like that to live?
> I'd be happy to build something that does this. I don't know what's
> possible or where it should live though. It's probably easy for me to
> add it to the lint job, wherever that is.  I might be able to
> integrate it into the new MozReview Static Analysis thing. I don't
> know if it's possible, or how, to automatically land corrections* but
> maybe?


This came up in a less pressing situation recently within the js/src 
tree, where we were thinking that the nicest way for this to work would 
be to have some limited pre-push lint checks. In our case, it could be 
easily restricted to a particular portion of the tree. I don't know if 
it would be fast enough to not add friction if it had to check any 
source file anywhere in the tree, but it seems like the nicest developer 
experience -- you wouldn't be bothered by it until you tried to land, 
and then an opt-out pre-push hook would tell you that you're going to 
break something. There has been talk of hooking up mozlint in this way, 
so that might be the place to look if you're thinking of setting it up. 
I'm not sure what the story there is for a git workflow, but at least 
for mercurial it would seem to make sense to have the hooks installed 
via mach bootstrap / mach mercurial-setup.

The new phabricator workflow may have a place for this sort of thing as 
well?


0
Steve
2/7/2018 5:16:13 PM
On Tue, Feb 6, 2018 at 5:36 PM, Aaron Klotz <aklotz@mozilla.com> wrote:
> I'd like to follow up on this old thread to discuss what we can do about
> improving the mingw developer experience for people doing Windows-centric
> stuff.


To follow up on this, in
https://bugzilla.mozilla.org/show_bug.cgi?id=1444167 I landed a lint
job that will look for #includes that match one of MinGW's headers but
isn't all-lowercase.

If you're adding third-party code that (for whatever reason) doesn't
break the MinGW build but still has uppercase includes and the lint
test is failing, you can add file or directory exceptions here:
https://hg.mozilla.org/mozilla-central/file/tip/tools/lint/mingw-capitalization.yml

-tom
0
Tom
3/14/2018 1:18:18 PM
Reply: