Help make the build faster by porting make rules to the misc tier

We added a capability to the Firefox build system that will ultimately 
lead to speeding up the build and we need your help to realize its 
potential.

Background
==========

The Firefox build executes as a pipeline of stages called tiers. The 
tiers are currently export -> compile -> misc -> libs -> tools. The 
build system iterates the tiers and builds directories in those tiers.

The export, compile, and misc tiers are mostly concurrent. If you 
execute with |make -j32| or more, they should saturate your cores. If 
you have access to a modern, multi-core machine, you realize the 
benefits of that machine.

The libs and tools tier, by contrast, aren't concurrent. This is mostly 
due to historical reasons. Back in the day, the traversal order of the 
directories was hard-coded by the order directories were registered in 
(DIRS += foo bar). If you had a cross-directory dependency, you 
registered one directory before the other.

Furthermore, back when we only had export, libs, and tools tier, these 
tiers were conceptually "pre-build," "build," and "post-build" tiers. 
libs became a dumping ground for most build tasks. Fast forward a few 
years and libs is still where most of the one-off build rules live. libs 
still executes in the order directories are defined in.

Mission Misc
============

The libs tier slows down the build because it isn't executed 
concurrently. By how much? As always, "it depends." But on my machine, 
traversing the libs tier on a no-op build takes ~50% of the wall time of 
the build (~22s/45s). The inefficiency of the libs tier is even more 
pronounced during a full build (multi-core usage drops off a cliff). 
*The libs tier is clownshoes.*

Our goal is to slowly kill the libs tier by moving things elsewhere, 
where they can execute with much higher concurrency and where 
cross-directory dependencies are properly captured. That will leave us 
with a more efficient build system that executes faster.

This is where we need your help.

The tree has tons of build tasks that run during the libs tier. Many of 
them have no dependencies and can be safely moved into the misc tier.

*We need your help writing patches to move things to the misc tier.*

How You Can Help
================

1) Search for build rules in your Makefile.in that execute in the libs 
tier/target in your Makefile.in files
2) Examine the dependencies
3) If it doesn't depend on anything in the libs tier/target or doesn't 
depend on things outside the current directory, it is probably safe to 
move it to misc. (See also the warnings below.)
5) Try converting the rule to the misc tier. Add |HAS_MISC_RULE = True| 
to the moz.build in that directory.
6) Test the build locally and push it to try
7) File a bug blocking bug 1099232. Initiate a code review and flag a 
build peer (glandium, mshal, gps are preferred) for review.

For an example conversion, see 
https://hg.mozilla.org/mozilla-central/rev/b1a9e41d3f4b

For a log showing what all gets built in libs, see 
https://people.mozilla.org/~gszorc/libs.log (this is an OS X build). We 
ideally want this log to be empty.

INSTALL_TARGETS and PP_TARGETS
------------------------------

The default rule/tier for INSTALL_TARGETS and PP_TARGETS is libs. Unless 
you explicitly declare a *_TARGET variable, the rule will execute during 
libs and will be slower.

*Merely grepping for "libs" is thus not sufficient for identifying rules 
that execute during libs.*

Here is how you move an INSTALL_TARGETS rule from libs to misc.

Before:

   foo_FILES := foo.js
   foo_DEST := $(DIST)/bin
   INSTALL_TARGETS += foo

After:

   foo_FILES := foo.js
   foo_DEST := $(DIST)/bin
   foo_TARGET := misc
   INSTALL_TARGETS += foo

Things to Watch Out For
-----------------------

When evaluating whether you can convert something from libs to misc:

* Stay away from things referencing $(XPI_NAME)
* Stay away from things related to JAR manifest parsing
* Stay away from things related to packaging
* Stay away from things related to l10n

There be dragons around all of these things. It is best to let a build 
peer handle it.

If you aren't sure, don't hesitate to ask a build peer. We (the build 
peers) want help tackling low-hanging fruit: don't waste your time on 
something beyond your skill level: you likely won't derive anything but 
pain. If it doesn't work right away, seriously consider moving on and 
punting to someone with more domain knowledge.

Cross-Directory Dependencies
----------------------------

Did you find a rule that depends on another directory? For things in 
moz.build, this should "just work." But for Makefile.in rules, you need 
to explicitly declare that cross-directory dependency or the build is 
susceptible to race conditions.

To declare a cross-directory dependency for the misc tier, you'll want 
to add an entry at the bottom of config/recurse.mk.

For example, say you have a misc rule in "xpcom/system" that depends on 
a misc rule in "xpcom/base." You would add the following to recurse.mk:

   xpcom/system/misc: xpcom/base/misc

The build system integrates this into the traversal logic to ensure that 
xpcom/base is traversed before xpcom/system is evaluated.

Conclusion
----------

If you have any questions, ask in #build.

I look forward to seeing many review requests in my inbox!

Gregory
0
Gregory
11/14/2014 6:03:09 PM
mozilla.dev.platform 6640 articles. 0 followers. Post Follow

0 Replies
434 Views

Similar Articles

[PageSpeed] 27

Reply:

Similar Artilces:

Making should be make
Name: Scott Melton Email: scott_rides_againatyahoodotcom Product: Firefox Summary: Making should be make Comments: The opening page for Bugzilla is poorly worded. http://www.mozilla.org/projects/seamonkey/start/ The second line: If you want to help making this application even better, we would encourage you to take part in the should be MAKE, not MAKING Browser Details: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070222 SeaMonkey/1.1.1 ...

Problems building minimo, No rule to make target libpipboot.a. please help
Dear all, I'm new to porting mozilla. I've tried to cross compile minimo for an arm's platform in scratchbox. I check out the minimo project in 1.8 branch on aug 29 and build all. I've modified the mozconfig to disable crypto and patch the xpcom/reflect/xptcall/src/md/unix/xptcstubs_arm.cpp, add __attribute__((used)); When build minimo executable binary, there is an error message: c++ -o nsStaticComponents.o -c -I../../dist/include/system_wrappers -include ../../config/gcc_hidden.h -DMOZILLA_INTERNAL_API -DOSTYPE=\"Linux2.6.15-1\" -DOSARCH=\"Linux\&...

Make Porting/expand-macro.pl work with a make other than `make`, also Win32 compatibility fixes
--------------040507080907060101050501 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Hello, recently it came to my attention that `Porting/expand-macro.pl` does not work if your `make` tool is named anything other than `make`. The first attached patch changes it so that it loads Config.pm and uses the configured $Config{make} from there. I presume that anybody running `Porting/expand-macro.pl` also knows about how to give `perl` the appropriate Config.pm resp. how to invoke the proper Perl. The second patch assumes that `Porti...

Perl-5.8.6 or Perl-5.8.5 Install/Build process on AIX 5.3 Make problem make: 1254-002 Cannot find a rule to create target <command-line> from dependencies. Stop.
------=_NextPart_000_0001_01C90BEC.E9B8A770 Content-Type: multipart/related; boundary="----=_NextPart_001_0002_01C90BEC.E9B8A770" ------=_NextPart_001_0002_01C90BEC.E9B8A770 Content-Type: multipart/alternative; boundary="----=_NextPart_002_0003_01C90BEC.E9B8A770" ------=_NextPart_002_0003_01C90BEC.E9B8A770 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello all, I am trying to build perl-5.8.6.tar.gz DBI-1.50.tar.gz and DBD-Oracle-1.16.tar.gz on AIX 5.3.0.0 downloaded from cpan.org I am tryi...

superreview granted: [Bug 408009] Make doGetObjectPrincipal() faster. : [ Attachment 292705] Make doGetObjectPrincipal() faster.
Boris Zbarsky (reviews very slow until spring) <bzbarsky@mit.edu> has grant= ed Johnny Stenback (:jst) <jst@mozilla.org>'s request for superreview: Bug 408009: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/show_bug.cgi?id=3D408009 Attachment 292705: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/attachment.cgi?id=3D292705&action=3Dedit ------- Additional Comments from Boris Zbarsky (reviews very slow until spr= ing) <bzbarsky@mit.edu> >+++ b/caps/src/nsScriptSecurityManager.cpp=0D >+ // nsINode. Note that th...

superreview requested: [Bug 408009] Make doGetObjectPrincipal() faster. : [ Attachment 292705] Make doGetObjectPrincipal() faster.
Johnny Stenback (:jst) <jst@mozilla.org> has asked Boris Zbarsky (reviews very slow until spring) <bzbarsky@mit.edu> for superreview: Bug 408009: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/show_bug.cgi?id=408009 Attachment 292705: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/attachment.cgi?id=292705&action=edit ------- Additional Comments from Johnny Stenback (:jst) <jst@mozilla.org> nsScriptSecurityManager::doGetObjectPrincipal() spends a good bit of time calling QI on a JSObject's private data to find out if it&#...

[microPATCH] make imcc build with BSD make
Index: config/gen/makefiles/imcc.in =================================================================== RCS file: /cvs/public/parrot/config/gen/makefiles/imcc.in,v retrieving revision 1.15 diff -u -r1.15 imcc.in --- config/gen/makefiles/imcc.in 22 Mar 2003 22:06:09 -0000 1.15 +++ config/gen/makefiles/imcc.in 27 Mar 2003 09:21:07 -0000 @@ -47,6 +47,8 @@ .PHONY : FORCE +FORCE : + imcparser.c imcparser.h : imcc.y FORCE $(PERL) rebuild.pl ${PQ}$(YACC) imcc.y -d -o imcparser.c${PQ} imcc.y imcparser.h imcparser.c -- You can make any kind of artificial atom - long, thin ...

superreview requested: [Bug 330053] Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. : [Attachment 218593] Make mac build complete
Robert Kaiser <kairo@kairo.at> has asked neil@parkwaycc.co.uk <neil@httl.net> for superreview: Bug 330053: Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. https://bugzilla.mozilla.org/show_bug.cgi?id=330053 Attachment 218593: Make mac build complete https://bugzilla.mozilla.org/attachment.cgi?id=218593&action=edit ------- Additional Comments from Robert Kaiser <kairo@kairo.at> This is the pathc to get suiterunner to build on Mac. Some of this is just correcting the Makefile to things we already do in xpfe/bootstrap (the Info.pli...

superreview granted: [Bug 330053] Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. : [Attachment 218593] Make mac build complete
jag (Peter Annema) <jag@tty.nl> has granted Robert Kaiser <kairo@kairo.at>'s request for superreview: Bug 330053: Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. https://bugzilla.mozilla.org/show_bug.cgi?id=330053 Attachment 218593: Make mac build complete https://bugzilla.mozilla.org/attachment.cgi?id=218593&action=edit ...

Help. Can I make 2 remote calls consicutivly? or must 1 return and then make the other one?
I am just starting to discover the magic of AJAX and am loving it but I am still quite green. I need to know if I can make 2 remote calls to a page method or a webservice in a row, consecutive? like that: function test(){PageMethods.Execute1RemoteCall(ObjectID, CompleteUpdate1, errorit);  PageMethods.ExecuteAnotherRemoteCall(ObjectID, CompleteUpdate2, errorit);   } function CompleteUpdate1(results, context, methodName) { dostuff();   HideWaitMessage(); } function CompleteUpdate2(results, context, methodName) { doOtherstuff();   HideWaitMessage(); ...

How to make web application load faster in browser after build...
Is there a way to make a web application load faster in the browser after a build?Whenever I build a web application, and want to test it in a web browser, it takes a 'long' time to initial load the newly build web application.Is there a way to make this faster?Every time I change a little bit of code, it takes more than 45 seconds to load the newly build web application.  Check debug section in web.Config - set batch="false".  There is no way to increase studio's build speed. May be add more RAM :)May the Force be with you Every time I change a little bit...

HELP! HELP!HELP: How to make bugzilla work for Windows7 (64 bit)
Hi There, I am in the process of installing Bugzilla at my company but not able to st= art successfully the application and like to know why: Here is the steps I took: My environment is : OS : MS Windows 7 (64 bit) VM Version : 8 1. Install Per l(5.8.1 or above) 2. Install MySQL 3. Install Apache2 4. Install Bugzilla 3.2 5. Install Perl modules 6. Bugzilla using apache 7. Perform local config file using checksetup.pl. Created bugs in mys= ql. I am not able to create the Bugzilla database by running checksetup.pl= .. Created user "bugs" as a...

make: *** No rule to make target `<command-line>', needed by `miniperlmain.o'.
Hi, I just tried to compile Perl 5.8.8 on my Linux Debian, with ./Configure -des -Dprefix=/opt/my && make And got this: First let's make sure your kit is complete. Checking... <copious amount of stuff removed> Now you must run 'make'. If you compile perl5 on a different machine or from a different object directory, copy the Policy.sh file from this object directory to the new one before you run Configure -- this will help you with most of the policy defaults. make: *** No rule to make target `<command-...

Web resources about - Help make the build faster by porting make rules to the misc tier - mozilla.dev.platform

Porting - Wikipedia, the free encyclopedia
This article is about moving software to a different system. For engine tuning, see porting (engine) . For porting of telephone numbers, see ...

Step by Step Guide to Porting Your Facebook App to Bebo
This is a guest post by Blake Commagere, creator of the “monsters” series of applications on Facebook, Bebo, and other social networks. How much ...

Porting Android on Beagle Board XM - YouTube
This video shows 3D demos on porting android OS using beagle board XM by Tenet.. BeagleBoard-xM delivers extra ARM Cortex-A8 MHz and extra memory ...

$45k stolen in phone porting scam
Scammers steal identity, redirect banking verification codes.

Learning from porting selecta
... keep, your production code or your tests?" If you have high-quality tests, you should be able to produce similar code with clean design. Porting ...

Aspyr Media talks porting games to OS X, Mac App Store & the future of Mac gaming
... to speak with Aspyr’s VP of Publishing Elizabeth Howard, with input from CEO Michael Rogers and other execs, regarding the the process of porting ...

NVIDIA Acquires Game Porting Group & Tech From Transgaming
... Windows games over to OS X. With the rise of Apple’s fortunes and the move to x86, Transgaming has been responsible for either directly porting ...

Nintendo could be porting Game Boy games to smartphones - AOL.com
A patent filed by Nintendo in June makes it look like Game Boy games might be finally coming to mobile platforms. Yup, you might finally get ...

Jetpack Joyride Developer Acquires Cross-Platform Porting Technology
... that offers cross-platform conversion options for electronic games using only their C++ code bases. With the acquisition of a dedicated porting ...

Adobe abandons development of Flash-to-iPhone porting software
A change in Apple's developer agreement has caused Adobe to halt development of technology that allows Flash applications to be ported natively ...

Resources last updated: 12/29/2015 6:45:41 AM