Help make the build faster by porting make rules to the misc tier
We added a capability to the Firefox build system that will ultimately lead to speeding up the build and we need your help to realize its potential. Background ========== The Firefox build executes as a pipeline of stages called tiers. The tiers are currently export -> compile -> misc -> libs -> tools. The build system iterates the tiers and builds directories in those tiers. The export, compile, and misc tiers are mostly concurrent. If you execute with |make -j32| or more, they should saturate your cores. If you have access to a modern, multi-core machine, you realize the benefits of that machine. The libs and tools tier, by contrast, aren't concurrent. This is mostly due to historical reasons. Back in the day, the traversal order of the directories was hard-coded by the order directories were registered in (DIRS += foo bar). If you had a cross-directory dependency, you registered one directory before the other. Furthermore, back when we only had export, libs, and tools tier, these tiers were conceptually "pre-build," "build," and "post-build" tiers. libs became a dumping ground for most build tasks. Fast forward a few years and libs is still where most of the one-off build rules live. libs still executes in the order directories are defined in. Mission Misc ============ The libs tier slows down the build because it isn't executed concurrently. By how much? As always, "it depends." But on my machine, traversing the libs tier on a no-op build takes ~50% of the wall time of the build (~22s/45s). The inefficiency of the libs tier is even more pronounced during a full build (multi-core usage drops off a cliff). *The libs tier is clownshoes.* Our goal is to slowly kill the libs tier by moving things elsewhere, where they can execute with much higher concurrency and where cross-directory dependencies are properly captured. That will leave us with a more efficient build system that executes faster. This is where we need your help. The tree has tons of build tasks that run during the libs tier. Many of them have no dependencies and can be safely moved into the misc tier. *We need your help writing patches to move things to the misc tier.* How You Can Help ================ 1) Search for build rules in your Makefile.in that execute in the libs tier/target in your Makefile.in files 2) Examine the dependencies 3) If it doesn't depend on anything in the libs tier/target or doesn't depend on things outside the current directory, it is probably safe to move it to misc. (See also the warnings below.) 5) Try converting the rule to the misc tier. Add |HAS_MISC_RULE = True| to the moz.build in that directory. 6) Test the build locally and push it to try 7) File a bug blocking bug 1099232. Initiate a code review and flag a build peer (glandium, mshal, gps are preferred) for review. For an example conversion, see https://hg.mozilla.org/mozilla-central/rev/b1a9e41d3f4b For a log showing what all gets built in libs, see https://people.mozilla.org/~gszorc/libs.log (this is an OS X build). We ideally want this log to be empty. INSTALL_TARGETS and PP_TARGETS ------------------------------ The default rule/tier for INSTALL_TARGETS and PP_TARGETS is libs. Unless you explicitly declare a *_TARGET variable, the rule will execute during libs and will be slower. *Merely grepping for "libs" is thus not sufficient for identifying rules that execute during libs.* Here is how you move an INSTALL_TARGETS rule from libs to misc. Before: foo_FILES := foo.js foo_DEST := $(DIST)/bin INSTALL_TARGETS += foo After: foo_FILES := foo.js foo_DEST := $(DIST)/bin foo_TARGET := misc INSTALL_TARGETS += foo Things to Watch Out For ----------------------- When evaluating whether you can convert something from libs to misc: * Stay away from things referencing $(XPI_NAME) * Stay away from things related to JAR manifest parsing * Stay away from things related to packaging * Stay away from things related to l10n There be dragons around all of these things. It is best to let a build peer handle it. If you aren't sure, don't hesitate to ask a build peer. We (the build peers) want help tackling low-hanging fruit: don't waste your time on something beyond your skill level: you likely won't derive anything but pain. If it doesn't work right away, seriously consider moving on and punting to someone with more domain knowledge. Cross-Directory Dependencies ---------------------------- Did you find a rule that depends on another directory? For things in moz.build, this should "just work." But for Makefile.in rules, you need to explicitly declare that cross-directory dependency or the build is susceptible to race conditions. To declare a cross-directory dependency for the misc tier, you'll want to add an entry at the bottom of config/recurse.mk. For example, say you have a misc rule in "xpcom/system" that depends on a misc rule in "xpcom/base." You would add the following to recurse.mk: xpcom/system/misc: xpcom/base/misc The build system integrates this into the traversal logic to ensure that xpcom/base is traversed before xpcom/system is evaluated. Conclusion ---------- If you have any questions, ask in #build. I look forward to seeing many review requests in my inbox! Gregory
|
0 |
|
Gregory
11/14/2014 6:03:09 PM
mozilla.dev.platform
6640 articles. 
0 followers.
0 Replies
434 Views
Similar Artilces:
Making should be make
Name: Scott Melton Email: scott_rides_againatyahoodotcom Product: Firefox Summary: Making should be make Comments: The opening page for Bugzilla is poorly worded. http://www.mozilla.org/projects/seamonkey/start/ The second line: If you want to help making this application even better, we would encourage you to take part in the should be MAKE, not MAKING Browser Details: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070222 SeaMonkey/1.1.1 ...
Problems building minimo, No rule to make target libpipboot.a. please help
Dear all, I'm new to porting mozilla. I've tried to cross compile minimo for an arm's platform in scratchbox. I check out the minimo project in 1.8 branch on aug 29 and build all. I've modified the mozconfig to disable crypto and patch the xpcom/reflect/xptcall/src/md/unix/xptcstubs_arm.cpp, add __attribute__((used)); When build minimo executable binary, there is an error message: c++ -o nsStaticComponents.o -c -I../../dist/include/system_wrappers -include ../../config/gcc_hidden.h -DMOZILLA_INTERNAL_API -DOSTYPE=\"Linux2.6.15-1\" -DOSARCH=\"Linux\&...
Make Porting/expand-macro.pl work with a make other than `make`, also Win32 compatibility fixes
--------------040507080907060101050501 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Hello, recently it came to my attention that `Porting/expand-macro.pl` does not work if your `make` tool is named anything other than `make`. The first attached patch changes it so that it loads Config.pm and uses the configured $Config{make} from there. I presume that anybody running `Porting/expand-macro.pl` also knows about how to give `perl` the appropriate Config.pm resp. how to invoke the proper Perl. The second patch assumes that `Porti...
Perl-5.8.6 or Perl-5.8.5 Install/Build process on AIX 5.3 Make problem make: 1254-002 Cannot find a rule to create target <command-line> from dependencies. Stop.
------=_NextPart_000_0001_01C90BEC.E9B8A770 Content-Type: multipart/related; boundary="----=_NextPart_001_0002_01C90BEC.E9B8A770" ------=_NextPart_001_0002_01C90BEC.E9B8A770 Content-Type: multipart/alternative; boundary="----=_NextPart_002_0003_01C90BEC.E9B8A770" ------=_NextPart_002_0003_01C90BEC.E9B8A770 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello all, I am trying to build perl-5.8.6.tar.gz DBI-1.50.tar.gz and DBD-Oracle-1.16.tar.gz on AIX 5.3.0.0 downloaded from cpan.org I am tryi...
superreview granted: [Bug 408009] Make doGetObjectPrincipal() faster. : [ Attachment 292705] Make doGetObjectPrincipal() faster.
Boris Zbarsky (reviews very slow until spring) <bzbarsky@mit.edu> has grant= ed Johnny Stenback (:jst) <jst@mozilla.org>'s request for superreview: Bug 408009: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/show_bug.cgi?id=3D408009 Attachment 292705: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/attachment.cgi?id=3D292705&action=3Dedit ------- Additional Comments from Boris Zbarsky (reviews very slow until spr= ing) <bzbarsky@mit.edu> >+++ b/caps/src/nsScriptSecurityManager.cpp=0D >+ // nsINode. Note that th...
superreview requested: [Bug 408009] Make doGetObjectPrincipal() faster. : [ Attachment 292705] Make doGetObjectPrincipal() faster.
Johnny Stenback (:jst) <jst@mozilla.org> has asked Boris Zbarsky (reviews very slow until spring) <bzbarsky@mit.edu> for superreview: Bug 408009: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/show_bug.cgi?id=408009 Attachment 292705: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/attachment.cgi?id=292705&action=edit ------- Additional Comments from Johnny Stenback (:jst) <jst@mozilla.org> nsScriptSecurityManager::doGetObjectPrincipal() spends a good bit of time calling QI on a JSObject's private data to find out if it&#...
[microPATCH] make imcc build with BSD make
Index: config/gen/makefiles/imcc.in =================================================================== RCS file: /cvs/public/parrot/config/gen/makefiles/imcc.in,v retrieving revision 1.15 diff -u -r1.15 imcc.in --- config/gen/makefiles/imcc.in 22 Mar 2003 22:06:09 -0000 1.15 +++ config/gen/makefiles/imcc.in 27 Mar 2003 09:21:07 -0000 @@ -47,6 +47,8 @@ .PHONY : FORCE +FORCE : + imcparser.c imcparser.h : imcc.y FORCE $(PERL) rebuild.pl ${PQ}$(YACC) imcc.y -d -o imcparser.c${PQ} imcc.y imcparser.h imcparser.c -- You can make any kind of artificial atom - long, thin ...
superreview requested: [Bug 330053] Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. : [Attachment 218593] Make mac build complete
Robert Kaiser <kairo@kairo.at> has asked neil@parkwaycc.co.uk <neil@httl.net> for superreview: Bug 330053: Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. https://bugzilla.mozilla.org/show_bug.cgi?id=330053 Attachment 218593: Make mac build complete https://bugzilla.mozilla.org/attachment.cgi?id=218593&action=edit ------- Additional Comments from Robert Kaiser <kairo@kairo.at> This is the pathc to get suiterunner to build on Mac. Some of this is just correcting the Makefile to things we already do in xpfe/bootstrap (the Info.pli...
superreview granted: [Bug 330053] Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. : [Attachment 218593] Make mac build complete
jag (Peter Annema) <jag@tty.nl> has granted Robert Kaiser <kairo@kairo.at>'s request for superreview: Bug 330053: Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. https://bugzilla.mozilla.org/show_bug.cgi?id=330053 Attachment 218593: Make mac build complete https://bugzilla.mozilla.org/attachment.cgi?id=218593&action=edit ...
Help. Can I make 2 remote calls consicutivly? or must 1 return and then make the other one?
I am just starting to discover the magic of AJAX and am loving it but I am still quite green. I need to know if I can make 2 remote calls to a page method or a webservice in a row, consecutive? like that: function test(){PageMethods.Execute1RemoteCall(ObjectID, CompleteUpdate1, errorit); PageMethods.ExecuteAnotherRemoteCall(ObjectID, CompleteUpdate2, errorit); } function CompleteUpdate1(results, context, methodName) { dostuff(); HideWaitMessage(); } function CompleteUpdate2(results, context, methodName) { doOtherstuff(); HideWaitMessage(); ...
How to make web application load faster in browser after build...
Is there a way to make a web application load faster in the browser after a build?Whenever I build a web application, and want to test it in a web browser, it takes a 'long' time to initial load the newly build web application.Is there a way to make this faster?Every time I change a little bit of code, it takes more than 45 seconds to load the newly build web application. Check debug section in web.Config - set batch="false". There is no way to increase studio's build speed. May be add more RAM :)May the Force be with you Every time I change a little bit...
HELP! HELP!HELP: How to make bugzilla work for Windows7 (64 bit)
Hi There, I am in the process of installing Bugzilla at my company but not able to st= art successfully the application and like to know why: Here is the steps I took: My environment is : OS : MS Windows 7 (64 bit) VM Version : 8 1. Install Per l(5.8.1 or above) 2. Install MySQL 3. Install Apache2 4. Install Bugzilla 3.2 5. Install Perl modules 6. Bugzilla using apache 7. Perform local config file using checksetup.pl. Created bugs in mys= ql. I am not able to create the Bugzilla database by running checksetup.pl= .. Created user "bugs" as a...
make: *** No rule to make target `<command-line>', needed by `miniperlmain.o'.
Hi, I just tried to compile Perl 5.8.8 on my Linux Debian, with ./Configure -des -Dprefix=/opt/my && make And got this: First let's make sure your kit is complete. Checking... <copious amount of stuff removed> Now you must run 'make'. If you compile perl5 on a different machine or from a different object directory, copy the Policy.sh file from this object directory to the new one before you run Configure -- this will help you with most of the policy defaults. make: *** No rule to make target `<command-...
Name: Scott Melton Email: scott_rides_againatyahoodotcom Product: Firefox Summary: Making should be make Comments: The opening page for Bugzilla is poorly worded. http://www.mozilla.org/projects/seamonkey/start/ The second line: If you want to help making this application even better, we would encourage you to take part in the should be MAKE, not MAKING Browser Details: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070222 SeaMonkey/1.1.1 ...
Problems building minimo, No rule to make target libpipboot.a. please help
Dear all, I'm new to porting mozilla. I've tried to cross compile minimo for an arm's platform in scratchbox. I check out the minimo project in 1.8 branch on aug 29 and build all. I've modified the mozconfig to disable crypto and patch the xpcom/reflect/xptcall/src/md/unix/xptcstubs_arm.cpp, add __attribute__((used)); When build minimo executable binary, there is an error message: c++ -o nsStaticComponents.o -c -I../../dist/include/system_wrappers -include ../../config/gcc_hidden.h -DMOZILLA_INTERNAL_API -DOSTYPE=\"Linux2.6.15-1\" -DOSARCH=\"Linux\&...
Make Porting/expand-macro.pl work with a make other than `make`, also Win32 compatibility fixes
--------------040507080907060101050501 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Hello, recently it came to my attention that `Porting/expand-macro.pl` does not work if your `make` tool is named anything other than `make`. The first attached patch changes it so that it loads Config.pm and uses the configured $Config{make} from there. I presume that anybody running `Porting/expand-macro.pl` also knows about how to give `perl` the appropriate Config.pm resp. how to invoke the proper Perl. The second patch assumes that `Porti...
Perl-5.8.6 or Perl-5.8.5 Install/Build process on AIX 5.3 Make problem make: 1254-002 Cannot find a rule to create target <command-line> from dependencies. Stop.
------=_NextPart_000_0001_01C90BEC.E9B8A770 Content-Type: multipart/related; boundary="----=_NextPart_001_0002_01C90BEC.E9B8A770" ------=_NextPart_001_0002_01C90BEC.E9B8A770 Content-Type: multipart/alternative; boundary="----=_NextPart_002_0003_01C90BEC.E9B8A770" ------=_NextPart_002_0003_01C90BEC.E9B8A770 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello all, I am trying to build perl-5.8.6.tar.gz DBI-1.50.tar.gz and DBD-Oracle-1.16.tar.gz on AIX 5.3.0.0 downloaded from cpan.org I am tryi...
superreview granted: [Bug 408009] Make doGetObjectPrincipal() faster. : [ Attachment 292705] Make doGetObjectPrincipal() faster.
Boris Zbarsky (reviews very slow until spring) <bzbarsky@mit.edu> has grant= ed Johnny Stenback (:jst) <jst@mozilla.org>'s request for superreview: Bug 408009: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/show_bug.cgi?id=3D408009 Attachment 292705: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/attachment.cgi?id=3D292705&action=3Dedit ------- Additional Comments from Boris Zbarsky (reviews very slow until spr= ing) <bzbarsky@mit.edu> >+++ b/caps/src/nsScriptSecurityManager.cpp=0D >+ // nsINode. Note that th...
superreview requested: [Bug 408009] Make doGetObjectPrincipal() faster. : [ Attachment 292705] Make doGetObjectPrincipal() faster.
Johnny Stenback (:jst) <jst@mozilla.org> has asked Boris Zbarsky (reviews very slow until spring) <bzbarsky@mit.edu> for superreview: Bug 408009: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/show_bug.cgi?id=408009 Attachment 292705: Make doGetObjectPrincipal() faster. https://bugzilla.mozilla.org/attachment.cgi?id=292705&action=edit ------- Additional Comments from Johnny Stenback (:jst) <jst@mozilla.org> nsScriptSecurityManager::doGetObjectPrincipal() spends a good bit of time calling QI on a JSObject's private data to find out if it&#...
[microPATCH] make imcc build with BSD make
Index: config/gen/makefiles/imcc.in =================================================================== RCS file: /cvs/public/parrot/config/gen/makefiles/imcc.in,v retrieving revision 1.15 diff -u -r1.15 imcc.in --- config/gen/makefiles/imcc.in 22 Mar 2003 22:06:09 -0000 1.15 +++ config/gen/makefiles/imcc.in 27 Mar 2003 09:21:07 -0000 @@ -47,6 +47,8 @@ .PHONY : FORCE +FORCE : + imcparser.c imcparser.h : imcc.y FORCE $(PERL) rebuild.pl ${PQ}$(YACC) imcc.y -d -o imcparser.c${PQ} imcc.y imcparser.h imcparser.c -- You can make any kind of artificial atom - long, thin ...
superreview requested: [Bug 330053] Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. : [Attachment 218593] Make mac build complete
Robert Kaiser <kairo@kairo.at> has asked neil@parkwaycc.co.uk <neil@httl.net> for superreview: Bug 330053: Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. https://bugzilla.mozilla.org/show_bug.cgi?id=330053 Attachment 218593: Make mac build complete https://bugzilla.mozilla.org/attachment.cgi?id=218593&action=edit ------- Additional Comments from Robert Kaiser <kairo@kairo.at> This is the pathc to get suiterunner to build on Mac. Some of this is just correcting the Makefile to things we already do in xpfe/bootstrap (the Info.pli...
superreview granted: [Bug 330053] Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. : [Attachment 218593] Make mac build complete
jag (Peter Annema) <jag@tty.nl> has granted Robert Kaiser <kairo@kairo.at>'s request for superreview: Bug 330053: Make SeaMonkey capable of generating a reasonable build with MOZ_XUL_APP set. https://bugzilla.mozilla.org/show_bug.cgi?id=330053 Attachment 218593: Make mac build complete https://bugzilla.mozilla.org/attachment.cgi?id=218593&action=edit ...
Help. Can I make 2 remote calls consicutivly? or must 1 return and then make the other one?
I am just starting to discover the magic of AJAX and am loving it but I am still quite green. I need to know if I can make 2 remote calls to a page method or a webservice in a row, consecutive? like that: function test(){PageMethods.Execute1RemoteCall(ObjectID, CompleteUpdate1, errorit); PageMethods.ExecuteAnotherRemoteCall(ObjectID, CompleteUpdate2, errorit); } function CompleteUpdate1(results, context, methodName) { dostuff(); HideWaitMessage(); } function CompleteUpdate2(results, context, methodName) { doOtherstuff(); HideWaitMessage(); ...
How to make web application load faster in browser after build...
Is there a way to make a web application load faster in the browser after a build?Whenever I build a web application, and want to test it in a web browser, it takes a 'long' time to initial load the newly build web application.Is there a way to make this faster?Every time I change a little bit of code, it takes more than 45 seconds to load the newly build web application. Check debug section in web.Config - set batch="false". There is no way to increase studio's build speed. May be add more RAM :)May the Force be with you Every time I change a little bit...
HELP! HELP!HELP: How to make bugzilla work for Windows7 (64 bit)
Hi There, I am in the process of installing Bugzilla at my company but not able to st= art successfully the application and like to know why: Here is the steps I took: My environment is : OS : MS Windows 7 (64 bit) VM Version : 8 1. Install Per l(5.8.1 or above) 2. Install MySQL 3. Install Apache2 4. Install Bugzilla 3.2 5. Install Perl modules 6. Bugzilla using apache 7. Perform local config file using checksetup.pl. Created bugs in mys= ql. I am not able to create the Bugzilla database by running checksetup.pl= .. Created user "bugs" as a...
make: *** No rule to make target `<command-line>', needed by `miniperlmain.o'.
Hi, I just tried to compile Perl 5.8.8 on my Linux Debian, with ./Configure -des -Dprefix=/opt/my && make And got this: First let's make sure your kit is complete. Checking... <copious amount of stuff removed> Now you must run 'make'. If you compile perl5 on a different machine or from a different object directory, copy the Policy.sh file from this object directory to the new one before you run Configure -- this will help you with most of the policy defaults. make: *** No rule to make target `<command-...
Web resources about - Help make the build faster by porting make rules to the misc tier - mozilla.dev.platform Porting - Wikipedia, the free encyclopedia
This article is about moving software to a different system. For engine tuning, see porting (engine) . For porting of telephone numbers, see ...
This article is about moving software to a different system. For engine tuning, see porting (engine) . For porting of telephone numbers, see ...
Step by Step Guide to Porting Your Facebook App to Bebo
This is a guest post by Blake Commagere, creator of the “monsters” series of applications on Facebook, Bebo, and other social networks. How much ...
This is a guest post by Blake Commagere, creator of the “monsters” series of applications on Facebook, Bebo, and other social networks. How much ...
Porting Android on Beagle Board XM - YouTubeThis video shows 3D demos on porting android OS using beagle board XM by Tenet.. BeagleBoard-xM delivers extra ARM Cortex-A8 MHz and extra memory ...
$45k stolen in phone porting scamScammers steal identity, redirect banking verification codes.
Learning from porting selecta... keep, your production code or your tests?" If you have high-quality tests, you should be able to produce similar code with clean design. Porting ...
Aspyr Media talks porting games to OS X, Mac App Store & the future of Mac gaming... to speak with Aspyr’s VP of Publishing Elizabeth Howard, with input from CEO Michael Rogers and other execs, regarding the the process of porting ...
NVIDIA Acquires Game Porting Group & Tech From Transgaming... Windows games over to OS X. With the rise of Apple’s fortunes and the move to x86, Transgaming has been responsible for either directly porting ...
Nintendo could be porting Game Boy games to smartphones - AOL.comA patent filed by Nintendo in June makes it look like Game Boy games might be finally coming to mobile platforms. Yup, you might finally get ...
Jetpack Joyride Developer Acquires Cross-Platform Porting Technology... that offers cross-platform conversion options for electronic games using only their C++ code bases. With the acquisition of a dedicated porting ...
Adobe abandons development of Flash-to-iPhone porting software
A change in Apple's developer agreement has caused Adobe to halt development of technology that allows Flash applications to be ported natively ...
A change in Apple's developer agreement has caused Adobe to halt development of technology that allows Flash applications to be ported natively ...
Resources last updated: 12/29/2015 6:45:41 AM