Firefox and clang-cl

Hi everyone,

As you may have heard by now, Chromium has started to switch their Windows
builds to use clang-cl instead of MSVC [1].  This has improved their
Speedometer v2 benchmark score on x86 (but not on x86-64) by about 30%
according to AWFY [2]. Over the past few days, several people have reached
out to me to ask about how close we are to switching to clang-cl on
Windows, I think due to seeing this improvement on their side.  I thought
I'd write up a summary of the current state of affairs to my knowledge in
the hopes that many people would find it interesting.

First things first, please note that just because Chromium has seen this
improvement shouldn't make us automatically expect a similar improvement in
Speedometer v2 scores in Firefox when built with clang-cl.  Both Chromium
and Firefox are large enough codebases that we shouldn't assume any such
results to transfer from one side to the other.  Also, you should note that
the Chrome have also seen some regressions in some other benchmarks as a
result of this change, and those regressions are currently being
investigated.  Those who are curious can see the dependency list of this
bug [3].

About our current status with clang-cl, right now Firefox builds with
clang-cl.  We use these builds for the purpose of static analysis using our
custom clang plugin (similarly to Mac and Linux).  These builds are stood
up on TreeHerder under "Windows 2012 opt" and "Windows 2012 x64 opt" marked
as "S" jobs.

That being said, we still have a lot of work ahead of us before we can get
to a point where we can consider switching to clang-cl for the builds that
we ship to our users.  The below is a rough list of things we need to look
into before we can consider doing so.

* Keeping up with the LLVM trunk.
Any serious attempt for us to switch from MSVC to clang-cl will involve
fixing bugs on the LLVM side in addition to on the Firefox side (as the
long history of the work done so far [4] demonstrates.)  Right now, the
current LLVM version we use on Windows [5] was last updated in February and
is outdated.  glandium recently tried building with LLVM trunk and there is
a regression on the LLVM side causing the build to fail.  :-(  But in the
periods of time when we have been actively working on the clang-cl port, we
have tried to follow the LLVM trunk as closely as possible in order to
reduce the amount of work involved in cherry-picking the fixes we need.

* Ensuring the correctness of the resulting build.
clang-cl implements Microsoft's ABI and attempts to produce object files
that are compatible with those produced by MSVC.  As such, even though we
already build and ship our code with clang, it is possible that we still
have bugs lurking either on our side or on the LLVM side that we need to
find and fix (not to speak of all of the Windows specific code we have
which hasn't been exercised in a shipping environment with clang.)  The
first step here would be to stand up all of our tests on the clang-cl
builds and making them green.  Ensuring things like crash rates being
similar to MSVC builds, etc. would be the next steps.

* Ensuring the performance of the resulting build.
The MSVC builds that we ship are compiled using the PGO compiler.  In order
to perform a fair comparison with clang-cl, we should probably try to get
PGO builds with clang-cl to work [6].  There is no theoretical reason why
this can't work, but this isn't something that we currently support.
Failing that, there is another option which is using LTO [7] but that
requires us to also port Firefox to link with lld [8] as well.  Another
open question is how to compare the performance.  The obvious answers would
be to run our Talos benchmarks, and AWFY benchmarks against the two
builds.  Whether that would be enough is an open question.

* Ensuring debuggability of the builds.
I haven't paid much close attention to the recent LLVM developments for
CodeView debug info support, but clang-cl has some support for -Z7 and -Zi
flags [9].  We need to ensure that the generated debug info works well for
our stackwalking needs (both locally using a debugger/programatically and
on the server side for crash-stats) and the generated builds are usefully
debuggable on Windows.

* (If there are other potential details I'm not thinking of right now,
please feel free to mention it here.)


Last but not least, you may ask yourself why would we want to spend this
much effort to switch to clang-cl on Windows?  I believe this is an
important long term shift that is beneficial for us.  First and foremost,
clang is a vibrant open source compiler, and being able to use open source
toolchains on our most important platforms is really important for us in
terms of being able to contribute to the compiler where needed (anyone
remember the issues we had a few years back with regards to MSVC PGO
compiler hitting the maximum address space limit on Win32 when linking
Firefox?).

But more importantly, clang supports many exciting features that MSVC
lacks.  For example, clang usually implements newer language features
faster than MSVC (sometimes by years), and our usage of MSVC holds us back
in terms of the adoption of such features.  Also, it has features such as
various sanitizers, some of which [10] we may want to consider turning on
by default in the builds that we ship to our users.

I hope this is helpful.

Cheers,
Ehsan


[1]
https://groups.google.com/a/chromium.org/forum/#!msg/chromium-dev/Y3OEIKkdlu0/TCcT1SvwAwAJ
[2]
https://arewefastyet.com/#machine=37&view=single&suite=speedometer-misc&subtest=score&start=1501023532&end=1501883073
[3] https://bugs.chromium.org/p/chromium/issues/detail?id=82385
[4] https://bugzilla.mozilla.org/show_bug.cgi?id=winclang
[5] See llvm_revision in
https://searchfox.org/mozilla-central/source/build/build-clang/clang-win32.json
and
https://searchfox.org/mozilla-central/source/build/build-clang/clang-win64.json
[6] While it is possible to compare the MSVC PGO build's performance to
clang non-PGO build's performance, that is really an apples vs. oranges
comparison since the types of optimizations done by the compiler would be
quite different.
[7] http://llvm.org/docs/LinkTimeOptimization.html
[8] https://lld.llvm.org/
[9] https://msdn.microsoft.com/en-us/library/958x11bc.aspx
[10] For example, ubsan <
https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html>, CPI <
https://clang.llvm.org/docs/SafeStack.html> and CFI <
https://clang.llvm.org/docs/ControlFlowIntegrity.html>.

-- 
Ehsan
0
Ehsan
8/13/2017 1:40:24 AM
mozilla.dev.platform 5872 articles. 0 followers. Post Follow

4 Replies
7 Views

Similar Articles

[PageSpeed] 44

Hi,

we (Chromium) are also happy to answer questions if there's interest. We've looked at most of these issues in some detail.

In bullet points:
* Correctness: You might have some UB here and there but I wouldn't expect this to be a big problem.
* Performance: We switched from msvc+pgo to clang without pgo and got comparable perf. We did have to use an order file (/order: flag to link.exe) to get comparable startup perf.
*Debuggability: Basically works, see blockers of https://crbug.com/636111 for in-progress work. link.exe can produce pdbs with clang's codeview debug info. -Z7 and -Zi are aliased to each other in clang-cl, we don't do mspdbsrv)

You can find us on #chromium on freenode, or over email.

Nico
0
thakis
8/14/2017 5:12:43 PM
On 08/14/2017 01:12 PM, thakis@chromium.org wrote:
> Hi,
>
> we (Chromium) are also happy to answer questions if there's interest. We've looked at most of these issues in some detail.
Thanks Nico, much appreciated!

(For the record, we have already gotten a lot of help from the Google 
compiler folks with the LLVM side fixes for the bugs that we discovered 
while porting Firefox to build with clang-cl.  Getting to where we are 
now would certainly not be possible without that help! I wish I had done 
a better job at compiling a full list of all of those contributions over 
the years...)
> In bullet points:
> * Correctness: You might have some UB here and there but I wouldn't expect this to be a big problem.
Yes, we have indeed found and fixed bugs in our Windows specific code 
where MSVC has been too lenient in accepting bad C++ code, e.g. 
https://bugzilla.mozilla.org/show_bug.cgi?id=1251226.

> * Performance: We switched from msvc+pgo to clang without pgo and got comparable perf. We did have to use an order file (/order: flag to link.exe) to get comparable startup perf.
That is very interesting!  This is one of the aspects that we have been 
worried about a lot.  We should probably also think about using /order 
as well.

Does Chromium plan to switch to use clang with PGO on Windows by any chance?
> *Debuggability: Basically works, see blockers of https://crbug.com/636111 for in-progress work. link.exe can produce pdbs with clang's codeview debug info.
Wow, it looks like things have improved quite a bit on this front since 
the last time I looked at this closely.  Really impressive work!
> -Z7 and -Zi are aliased to each other in clang-cl, we don't do mspdbsrv)
I think this should be sufficient for Firefox's needs as well.

Cheers,
Ehsan
> You can find us on #chromium on freenode, or over email.
>
> Nico
> _______________________________________________
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

0
Ehsan
8/14/2017 8:36:25 PM
On Mon, Aug 14, 2017 at 1:36 PM, Ehsan Akhgari <ehsan.akhgari@gmail.com> wrote:
> Does Chromium plan to switch to use clang with PGO on Windows by any chance?

Yes, we want to do LTO+PGO builds eventually. (In particular, we'd
like to use ThinLTO for more manageable build times.) That requires
switching to using the lld linker, which in turn is requires adding
support for writing pdb files, an area that's been making lots of
progress lately.

Cheers,
Hans
0
Hans
8/14/2017 8:46:29 PM
On Mon, Aug 14, 2017, at 04:36 PM, Ehsan Akhgari wrote:
> > * Performance: We switched from msvc+pgo to clang without pgo and got comparable perf. We did have to use an order file (/order: flag to link.exe) to get comparable startup perf.
> That is very interesting!  This is one of the aspects that we have been 
> worried about a lot.  We should probably also think about using /order 
> as well.

It seems plausible that we could use our existing PGO build steps to
capture the proper ordering and then re-link using that as input to
/order. We already instrument the order of access of omni.ja entries
during that step and use that to produce an optimized omni.ja in the
second build pass.

> > *Debuggability: Basically works, see blockers of https://crbug.com/636111 for in-progress work. link.exe can produce pdbs with clang's codeview debug info.
> Wow, it looks like things have improved quite a bit on this front since 
> the last time I looked at this closely.  Really impressive work!
> > -Z7 and -Zi are aliased to each other in clang-cl, we don't do mspdbsrv)
> I think this should be sufficient for Firefox's needs as well.

We already build all of our non-PGO Windows builds with -Z7 for
compatibility with sccache anyway:
https://dxr.mozilla.org/mozilla-central/rev/b95b1638db48fc3d450b95b98da6bcd2f9326d2f/build/mozconfig.cache#137


-Ted
0
Ted
8/15/2017 4:30:52 PM
Reply: