Help make the build faster by porting make rules to the misc tier

We added a capability to the Firefox build system that will ultimately 
lead to speeding up the build and we need your help to realize its 
potential.

Background
==========

The Firefox build executes as a pipeline of stages called tiers. The 
tiers are currently export -> compile -> misc -> libs -> tools. The 
build system iterates the tiers and builds directories in those tiers.

The export, compile, and misc tiers are mostly concurrent. If you 
execute with |make -j32| or more, they should saturate your cores. If 
you have access to a modern, multi-core machine, you realize the 
benefits of that machine.

The libs and tools tier, by contrast, aren't concurrent. This is mostly 
due to historical reasons. Back in the day, the traversal order of the 
directories was hard-coded by the order directories were registered in 
(DIRS += foo bar). If you had a cross-directory dependency, you 
registered one directory before the other.

Furthermore, back when we only had export, libs, and tools tier, these 
tiers were conceptually "pre-build," "build," and "post-build" tiers. 
libs became a dumping ground for most build tasks. Fast forward a few 
years and libs is still where most of the one-off build rules live. libs 
still executes in the order directories are defined in.

Mission Misc
============

The libs tier slows down the build because it isn't executed 
concurrently. By how much? As always, "it depends." But on my machine, 
traversing the libs tier on a no-op build takes ~50% of the wall time of 
the build (~22s/45s). The inefficiency of the libs tier is even more 
pronounced during a full build (multi-core usage drops off a cliff). 
*The libs tier is clownshoes.*

Our goal is to slowly kill the libs tier by moving things elsewhere, 
where they can execute with much higher concurrency and where 
cross-directory dependencies are properly captured. That will leave us 
with a more efficient build system that executes faster.

This is where we need your help.

The tree has tons of build tasks that run during the libs tier. Many of 
them have no dependencies and can be safely moved into the misc tier.

*We need your help writing patches to move things to the misc tier.*

How You Can Help
================

1) Search for build rules in your Makefile.in that execute in the libs 
tier/target in your Makefile.in files
2) Examine the dependencies
3) If it doesn't depend on anything in the libs tier/target or doesn't 
depend on things outside the current directory, it is probably safe to 
move it to misc. (See also the warnings below.)
5) Try converting the rule to the misc tier. Add |HAS_MISC_RULE = True| 
to the moz.build in that directory.
6) Test the build locally and push it to try
7) File a bug blocking bug 1099232. Initiate a code review and flag a 
build peer (glandium, mshal, gps are preferred) for review.

For an example conversion, see 
https://hg.mozilla.org/mozilla-central/rev/b1a9e41d3f4b

For a log showing what all gets built in libs, see 
https://people.mozilla.org/~gszorc/libs.log (this is an OS X build). We 
ideally want this log to be empty.

INSTALL_TARGETS and PP_TARGETS
------------------------------

The default rule/tier for INSTALL_TARGETS and PP_TARGETS is libs. Unless 
you explicitly declare a *_TARGET variable, the rule will execute during 
libs and will be slower.

*Merely grepping for "libs" is thus not sufficient for identifying rules 
that execute during libs.*

Here is how you move an INSTALL_TARGETS rule from libs to misc.

Before:

   foo_FILES := foo.js
   foo_DEST := $(DIST)/bin
   INSTALL_TARGETS += foo

After:

   foo_FILES := foo.js
   foo_DEST := $(DIST)/bin
   foo_TARGET := misc
   INSTALL_TARGETS += foo

Things to Watch Out For
-----------------------

When evaluating whether you can convert something from libs to misc:

* Stay away from things referencing $(XPI_NAME)
* Stay away from things related to JAR manifest parsing
* Stay away from things related to packaging
* Stay away from things related to l10n

There be dragons around all of these things. It is best to let a build 
peer handle it.

If you aren't sure, don't hesitate to ask a build peer. We (the build 
peers) want help tackling low-hanging fruit: don't waste your time on 
something beyond your skill level: you likely won't derive anything but 
pain. If it doesn't work right away, seriously consider moving on and 
punting to someone with more domain knowledge.

Cross-Directory Dependencies
----------------------------

Did you find a rule that depends on another directory? For things in 
moz.build, this should "just work." But for Makefile.in rules, you need 
to explicitly declare that cross-directory dependency or the build is 
susceptible to race conditions.

To declare a cross-directory dependency for the misc tier, you'll want 
to add an entry at the bottom of config/recurse.mk.

For example, say you have a misc rule in "xpcom/system" that depends on 
a misc rule in "xpcom/base." You would add the following to recurse.mk:

   xpcom/system/misc: xpcom/base/misc

The build system integrates this into the traversal logic to ensure that 
xpcom/base is traversed before xpcom/system is evaluated.

Conclusion
----------

If you have any questions, ask in #build.

I look forward to seeing many review requests in my inbox!

Gregory
0
Gregory
11/14/2014 6:03:09 PM
mozilla.dev.builds 1760 articles. 0 followers. Post Follow

0 Replies
460 Views

Similar Articles

[PageSpeed] 41

Reply:

Similar Artilces:

superreview requested: [Bug 330053] Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. : [Attachment 218593] Make mac build complete
Robert Kaiser <kairo@kairo.at> has asked neil@parkwaycc.co.uk <neil@httl.net> for superreview: Bug 330053: Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. https://bugzilla.mozilla.org/show_bug.cgi?id=330053 Attachment 218593: Make mac build complete https://bugzilla.mozilla.org/attachment.cgi?id=218593&action=edit ------- Additional Comments from Robert Kaiser <kairo@kairo.at> This is the pathc to get suiterunner to build on Mac. Some of this is just correcting the Makefile to things we already do in xpfe/bootstrap (the Info.pli...

superreview granted: [Bug 330053] Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. : [Attachment 218593] Make mac build complete
jag (Peter Annema) <jag@tty.nl> has granted Robert Kaiser <kairo@kairo.at>'s request for superreview: Bug 330053: Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. https://bugzilla.mozilla.org/show_bug.cgi?id=330053 Attachment 218593: Make mac build complete https://bugzilla.mozilla.org/attachment.cgi?id=218593&action=edit ...

Problems building minimo, No rule to make target libpipboot.a. please help
Dear all, I'm new to porting mozilla. I've tried to cross compile minimo for an arm's platform in scratchbox. I check out the minimo project in 1.8 branch on aug 29 and build all. I've modified the mozconfig to disable crypto and patch the xpcom/reflect/xptcall/src/md/unix/xptcstubs_arm.cpp, add __attribute__((used)); When build minimo executable binary, there is an error message: c++ -o nsStaticComponents.o -c -I../../dist/include/system_wrappers -include ../../config/gcc_hidden.h -DMOZILLA_INTERNAL_API -DOSTYPE=\"Linux2.6.15-1\" -DOSARCH=\"Linux\&...

[microPATCH] make imcc build with BSD make
Index: config/gen/makefiles/imcc.in =================================================================== RCS file: /cvs/public/parrot/config/gen/makefiles/imcc.in,v retrieving revision 1.15 diff -u -r1.15 imcc.in --- config/gen/makefiles/imcc.in 22 Mar 2003 22:06:09 -0000 1.15 +++ config/gen/makefiles/imcc.in 27 Mar 2003 09:21:07 -0000 @@ -47,6 +47,8 @@ .PHONY : FORCE +FORCE : + imcparser.c imcparser.h : imcc.y FORCE $(PERL) rebuild.pl ${PQ}$(YACC) imcc.y -d -o imcparser.c${PQ} imcc.y imcparser.h imcparser.c -- You can make any kind of artificial atom - long, thin ...

Perl-5.8.6 or Perl-5.8.5 Install/Build process on AIX 5.3 Make problem make: 1254-002 Cannot find a rule to create target <command-line> from dependencies. Stop.
------=_NextPart_000_0001_01C90BEC.E9B8A770 Content-Type: multipart/related; boundary="----=_NextPart_001_0002_01C90BEC.E9B8A770" ------=_NextPart_001_0002_01C90BEC.E9B8A770 Content-Type: multipart/alternative; boundary="----=_NextPart_002_0003_01C90BEC.E9B8A770" ------=_NextPart_002_0003_01C90BEC.E9B8A770 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello all, I am trying to build perl-5.8.6.tar.gz DBI-1.50.tar.gz and DBD-Oracle-1.16.tar.gz on AIX 5.3.0.0 downloaded from cpan.org I am tryi...

Making a tryserver build as close as possible to a release build
For a user research project, I am trying to make a tryserver build which is close to a release build, at least in terms of user perception: it should have official release branding, have the crashreporter on by default. I don't think PGO is necessary, though ;-) The build should not have an update channel. What set of mozconfig/tryserver options would I use to get this? I know about http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/win32/release but I'm not exactly sure how to use that in terms of tryserver. --BDS On 01/31/13 03:38 PM...

VS Builds, and Builds, and Builds
Hello, Has anyone else had this problem?  After a while of developing, and I go to compile, VS takes forever to build and I have to end-task it.  It's definitely after I've been working for a while, then it finally has "had it" and just takes forever to build.  It's annoying as I know its not the machine (2.8 GHZ processor and 1GB RAM).  Has anyone else experienced this problem?  By the way, I do have SP1 installed. Also, after minimizing and switching around screens, when it does this, the building icon in the status bar doesn't show.  And, for more informati...

Making should be make
Name: Scott Melton Email: scott_rides_againatyahoodotcom Product: Firefox Summary: Making should be make Comments: The opening page for Bugzilla is poorly worded. http://www.mozilla.org/projects/seamonkey/start/ The second line: If you want to help making this application even better, we would encourage you to take part in the should be MAKE, not MAKING Browser Details: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070222 SeaMonkey/1.1.1 ...

firefox 6.0 build fails on executing make -f client.mk build
I am using windows 7 and VB 2010 edition. While executing command "make -f client.mk build" the error message comes as "client.mk:121: *** This source tree appears to have Windows-style line endings. To convert it to Unix-style line endings, run "python mozilla/build/ win32/mozilla-dos2unix.py". Stop." What should i do? ...

How to make web application load faster in browser after build...
Is there a way to make a web application load faster in the browser after a build?Whenever I build a web application, and want to test it in a web browser, it takes a 'long' time to initial load the newly build web application.Is there a way to make this faster?Every time I change a little bit of code, it takes more than 45 seconds to load the newly build web application.  Check debug section in web.Config - set batch="false".  There is no way to increase studio's build speed. May be add more RAM :)May the Force be with you Every time I change a little bit...

Can we make try builds default to a different profile than Nightly/Aurora/Beta/Release builds?
I sometimes ask users to test a try build to verify that the build fixes = their problem. This is an important tool, especially when the problem is = hard to reproduce or when the definition of =E2=80=9Cfix=E2=80=9D is a = bit nebulous (as it sometimes is in performance-related bugs). However, I don=E2=80=99t want to run the risk of causing users to lose = data when they test these builds! That=E2=80=99s especially true in = light of the fact that we sometimes make backwards incompatible changes = to on-disk data structures between releases (like the recent IndexedDB = changes), whic...

No rule to make target `.project' when building fennec release
I'm building sources from http://hg.mozilla.org/releases/mozilla-release/ a= nd get the following error:=20 make[5]: *** No rule to make target `.project', needed by `/Users/yu/Docume= nts/projects/mozilla/Fennec/release/obj-arm-linux-androideabi/mobile/androi= d/geckoview_library/.project'. Stop My mozconfig looks like this: ac_add_options --with-android-ndk=3D"$HOME/Development/android-ndk-r8e"=20 ac_add_options --with-android-sdk=3D"$HOME/Development/sdk/platforms/andro= id-17"=20 ac_add_options --enable-application=3Dmobile/andr...

How to "make smoke" when parrot won't make? (was Re: [perl #63922] Fresh rakudo fails to build on FreeBSD)
> I just pulled the fresh rakudo sources, Configured them with > --gen-parrot and the parrot build fails: .... > ./miniparrot config_lib.pasm > runtime/parrot/include/config.fpmc .... > Null PMC access in get_pmc_keyed_str() > gmake: *** [runtime/parrot/include/parrotlib.pbc ... It looks like I'm having the same problem on openbsd, which I posted about here a few days ago. Trying be a little more helpful, I set up a cron job to update and smoke-test parrot. The smoke test ran OK with the older revision of parrot, r37017, then I updated to the current rev, r37...

Web resources about - Help make the build faster by porting make rules to the misc tier - mozilla.dev.builds

Porting - Wikipedia, the free encyclopedia
This article is about moving software to a different system. For engine tuning, see porting (engine) . For porting of telephone numbers, see ...

Step by Step Guide to Porting Your Facebook App to Bebo
This is a guest post by Blake Commagere, creator of the “monsters” series of applications on Facebook, Bebo, and other social networks. How much ...

Porting Android on Beagle Board XM - YouTube
This video shows 3D demos on porting android OS using beagle board XM by Tenet.. BeagleBoard-xM delivers extra ARM Cortex-A8 MHz and extra memory ...

$45k stolen in phone porting scam
Scammers steal identity, redirect banking verification codes.

Learning from porting selecta
... keep, your production code or your tests?" If you have high-quality tests, you should be able to produce similar code with clean design. Porting ...

Aspyr Media talks porting games to OS X, Mac App Store & the future of Mac gaming
... to speak with Aspyr’s VP of Publishing Elizabeth Howard, with input from CEO Michael Rogers and other execs, regarding the the process of porting ...

NVIDIA Acquires Game Porting Group & Tech From Transgaming
... Windows games over to OS X. With the rise of Apple’s fortunes and the move to x86, Transgaming has been responsible for either directly porting ...

Nintendo could be porting Game Boy games to smartphones - AOL.com
A patent filed by Nintendo in June makes it look like Game Boy games might be finally coming to mobile platforms. Yup, you might finally get ...

Jetpack Joyride Developer Acquires Cross-Platform Porting Technology
... that offers cross-platform conversion options for electronic games using only their C++ code bases. With the acquisition of a dedicated porting ...

Adobe abandons development of Flash-to-iPhone porting software
A change in Apple's developer agreement has caused Adobe to halt development of technology that allows Flash applications to be ported natively ...

Resources last updated: 1/1/2016 9:44:28 PM