Better incremental builds - Tup backend ready for early adopters on Linux

--00000000000061c01b0572758a38
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi everyone,

Tup is a modern build system that enables fast and correct builds. I'm
pleased to announce the availability of Tup for building Firefox on Linux.
Compared to Make, building with Tup will result in faster incremental build
times and reduce the need for clobber builds. See the in-tree docs for
details [1].

We=E2=80=99re looking for some early adopters to try it out and let us know=
 if it
improves your workflow, or if you hit any blockers or barriers to adoption
that keep you on the Make backend for now.

Quick howto:

You=E2=80=99ll need to install the Tup executable, as well as the nightly
rust/cargo toolchain:

cd ~/.mozbuild && mach artifact toolchain --from-build linux64-tup
rustup install nightly
rustup default nightly

Your mozconfig needs to describe how to find the executable if it=E2=80=99s=
 not in
your PATH, and enable the Tup backend:

export TUP=3D~/.mozbuild/tup/tup
ac_add_options --enable-build-backends=3DTup

If you have any issues, feel free to file a bug blocking =E2=80=9Cbuildtup=
=E2=80=9D [2], or
contact mshal or chmanchester in #build on IRC.

[1] https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=3Dbuildtup

Thanks!

-Mike

--00000000000061c01b0572758a38
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;ma=
rgin-bottom:0pt" id=3D"gmail-docs-internal-guid-f290f52e-fb3d-5d0c-19af-c1a=
5be832920"><span style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0)=
;background-color:transparent;font-weight:400;font-style:normal;font-varian=
t:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"=
>Hi everyone,</span></p><br><p dir=3D"ltr" style=3D"line-height:1.38;margin=
-top:0pt;margin-bottom:0pt"><span style=3D"font-size:11pt;font-family:Arial=
;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:n=
ormal;font-variant:normal;text-decoration:none;vertical-align:baseline;whit=
e-space:pre-wrap">Tup is a modern build system that enables fast and correc=
t builds. I&#39;m pleased to announce the availability of Tup for building =
Firefox on Linux. Compared to Make, building with Tup will result in faster=
 incremental build times and reduce the need for clobber builds. See the in=
-tree docs for details [1].</span></p><br><p dir=3D"ltr" style=3D"line-heig=
ht:1.38;margin-top:0pt;margin-bottom:0pt"><span style=3D"font-size:11pt;fon=
t-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:40=
0;font-style:normal;font-variant:normal;text-decoration:none;vertical-align=
:baseline;white-space:pre-wrap">We=E2=80=99re looking for some early adopte=
rs to try it out and let us know if it improves your workflow, or if you hi=
t any blockers or barriers to adoption that keep you on the Make backend fo=
r now.</span></p><br><p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0p=
t;margin-bottom:0pt"><span style=3D"font-size:11pt;font-family:Arial;color:=
rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;f=
ont-variant:normal;text-decoration:none;vertical-align:baseline;white-space=
:pre-wrap">Quick howto:</span></p><br><p dir=3D"ltr" style=3D"line-height:1=
..38;margin-top:0pt;margin-bottom:0pt"><span style=3D"font-size:11pt;font-fa=
mily:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;fo=
nt-style:normal;font-variant:normal;text-decoration:none;vertical-align:bas=
eline;white-space:pre-wrap">You=E2=80=99ll need to install the Tup executab=
le, as well as the nightly rust/cargo toolchain:</span></p><br><p dir=3D"lt=
r" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style=
=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:tran=
sparent;font-weight:400;font-style:normal;font-variant:normal;text-decorati=
on:none;vertical-align:baseline;white-space:pre-wrap">cd ~/.mozbuild &amp;&=
amp; mach artifact toolchain --from-build linux64-tup</span><span style=3D"=
font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transpar=
ent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:n=
one;vertical-align:baseline;white-space:pre-wrap"><br class=3D"gmail-kix-li=
ne-break"></span><span style=3D"font-size:11pt;font-family:Arial;color:rgb(=
0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-=
variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre=
-wrap">rustup install nightly</span><span style=3D"font-size:11pt;font-fami=
ly:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font=
-style:normal;font-variant:normal;text-decoration:none;vertical-align:basel=
ine;white-space:pre-wrap"><br class=3D"gmail-kix-line-break"></span><span s=
tyle=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:=
transparent;font-weight:400;font-style:normal;font-variant:normal;text-deco=
ration:none;vertical-align:baseline;white-space:pre-wrap">rustup default ni=
ghtly</span><span style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0=
);background-color:transparent;font-weight:400;font-style:normal;font-varia=
nt:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap=
"><br class=3D"gmail-kix-line-break"><br class=3D"gmail-kix-line-break"></s=
pan></p><p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bott=
om:0pt"><span style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0);ba=
ckground-color:transparent;font-weight:400;font-style:normal;font-variant:n=
ormal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Yo=
ur mozconfig needs to describe how to find the executable if it=E2=80=99s n=
ot in your PATH, and enable the Tup backend:</span></p><br><p dir=3D"ltr" s=
tyle=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style=3D"f=
ont-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transpare=
nt;font-weight:400;font-style:normal;font-variant:normal;text-decoration:no=
ne;vertical-align:baseline;white-space:pre-wrap">export TUP=3D~/.mozbuild/t=
up/tup</span><span style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,=
0);background-color:transparent;font-weight:400;font-style:normal;font-vari=
ant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wra=
p"><br class=3D"gmail-kix-line-break"></span><span style=3D"font-size:11pt;=
font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight=
:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-al=
ign:baseline;white-space:pre-wrap">ac_add_options --enable-build-backends=
=3DTup</span></p><br><p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0p=
t;margin-bottom:0pt"><span style=3D"font-size:11pt;font-family:Arial;color:=
rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;f=
ont-variant:normal;text-decoration:none;vertical-align:baseline;white-space=
:pre-wrap">If you have any issues, feel free to file a bug blocking =E2=80=
=9Cbuildtup=E2=80=9D [2], or contact mshal or chmanchester in #build on IRC=
..</span><span style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0);ba=
ckground-color:transparent;font-weight:400;font-style:normal;font-variant:n=
ormal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><b=
r class=3D"gmail-kix-line-break"><br class=3D"gmail-kix-line-break"></span>=
</p><p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0=
pt"><span style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0);backgr=
ound-color:transparent;font-weight:400;font-style:normal;font-variant:norma=
l;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[1] <a=
 href=3D"https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html=
">https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html</a></s=
pan></p><p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bott=
om:0pt"><span style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0);ba=
ckground-color:transparent;font-weight:400;font-style:normal;font-variant:n=
ormal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[2=
] <a href=3D"https://bugzilla.mozilla.org/show_bug.cgi?id=3Dbuildtup">https=
://bugzilla.mozilla.org/show_bug.cgi?id=3Dbuildtup</a></span></p><br><p dir=
=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span =
style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color=
:transparent;font-weight:400;font-style:normal;font-variant:normal;text-dec=
oration:none;vertical-align:baseline;white-space:pre-wrap">Thanks!</span></=
p><p dir=3D"ltr" style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt=
"><span style=3D"font-size:11pt;font-family:Arial;color:rgb(0,0,0);backgrou=
nd-color:transparent;font-weight:400;font-style:normal;font-variant:normal;=
text-decoration:none;vertical-align:baseline;white-space:pre-wrap">-Mike</s=
pan></p><br></div>

--00000000000061c01b0572758a38--
0
Michael
8/2/2018 3:34:01 PM
mozilla.dev.builds 1711 articles. 0 followers. Post Follow

12 Replies
19 Views

Similar Articles

[PageSpeed] 53

This is great! A big thank you to all the folks who have worked for so long to
get us to this stage. I'm really looking forward to this work maturing and
extending to other platforms.

I've done a bit of testing on my workstation (14 core/28 thread i9, U.2
connected NVMe SSD, 64 GB DDR4 RAM) and got the following build times:

For -j28 (yes, -j28 is well into diminishing returns territory, but I wanted a
comparison of the best case scenarios for both backends):

      |      cold       |    top-level
      |  initial build  |  no-op rebuild
------------------------------------------
make  |      8m 10s     |       15s
tup   |      6m 35s     |        1.4s

And for -j8:

      |      cold       |    top-level
      |  initial build  |  no-op rebuild
------------------------------------------
make  |     11m 33s     |       15s
tup   |     10m 52s     |        1.4s

(To confirm: the rebuild numbers are the same for both -j28 and -j8.)

So a 20% reduction in build time on cold builds of a fresh tree with -j28, and
6% reduction with -j8. In addition to speeding up the build, Tup seems to
parallelize better which is great! Of course the highlight is the big reduction
in rebuild overhead which will make the edit-compile-run cycle a lot less
annoying (for me, 50s down to 37s after touching nsDocumentViewer.cpp,
rebuilding, and linking libxul, for both -j28 and -j8.)

These improvements, plus the promise of rarely having to clobber/figure out
which directories to rebuild in, put a big smile on my face. (Currently having
to turn off CARGO_INCREMENTAL, not being able to use sccache and other current
limitations, less so, but it's early days. :) )

A few notes for others trying this out (thanks cmanchester and mshal for the
help debugging some of these):

  * Make sure you use:

      --enable-build-backends=Tup

    not:

      --enable-build-backend=Tup

  * You must be using an in-tree OBJDIR.

  * MOZ_PARALLEL_BUILD isn't currently recognized, so you'll need to use
    `mach build`s -j argument to control the parallel job count for now
    ( see https://bugzilla.mozilla.org/show_bug.cgi?id=1481441 )

mshal: I think you could cross-post this to dev-platform for wider testing. I
know there are a bunch of caveats (which people should be made aware of), but
since the initial teething issues I had were figured out I've so far had no
other problems building, or rebuilding with real code changes.

Jonathan

On 02/08/2018 16:34, Michael Shal wrote:
> Hi everyone,
> 
> 
> Tup is a modern build system that enables fast and correct builds. I'm pleased
> to announce the availability of Tup for building Firefox on Linux. Compared to
> Make, building with Tup will result in faster incremental build times and reduce
> the need for clobber builds. See the in-tree docs for details [1].
> 
> 
> We’re looking for some early adopters to try it out and let us know if it
> improves your workflow, or if you hit any blockers or barriers to adoption that
> keep you on the Make backend for now.
> 
> 
> Quick howto:
> 
> 
> You’ll need to install the Tup executable, as well as the nightly rust/cargo
> toolchain:
> 
> 
> cd ~/.mozbuild && mach artifact toolchain --from-build linux64-tup
> rustup install nightly
> rustup default nightly
> 
> Your mozconfig needs to describe how to find the executable if it’s not in your
> PATH, and enable the Tup backend:
> 
> 
> export TUP=~/.mozbuild/tup/tup
> ac_add_options --enable-build-backends=Tup
> 
> 
> If you have any issues, feel free to file a bug blocking “buildtup” [2], or
> contact mshal or chmanchester in #build on IRC.
> 
> [1] https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html
> 
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=buildtup
> 
> 
> Thanks!
> 
> -Mike
> 
> 

0
Jonathan
8/7/2018 1:18:41 PM
On 8/7/18 9:18 AM, Jonathan Watt wrote:
>    * You must be using an in-tree OBJDIR.

Do you happen to know whether this is a temporary limitation or a 
permanent one?

-Boris
0
Boris
8/7/2018 2:06:49 PM
On 07/08/2018 15:06, Boris Zbarsky wrote:
> On 8/7/18 9:18 AM, Jonathan Watt wrote:
>>    * You must be using an in-tree OBJDIR.
> 
> Do you happen to know whether this is a temporary limitation or a 
> permanent one?

When I previously discussed that with mshal he said the following (edited to
replace locations specific to my setup):

"A little background about tup: by default it keeps track of everything in the
directory where 'tup init' gets run, and all subdirectories (if you look in your
top-level source directory, you should have a ".tup" directory there). So in the
case you use an out-of-tree $OBJDIR Tup doesn't see all the Tupfiles created in
that directory, since that is outside of where `tup init` gets run. I think it
might be possible to support this in the future, but that is likely a
non-trivial effort."
0
Jonathan
8/7/2018 3:27:56 PM
On 8/7/18 11:27 AM, Jonathan Watt wrote:
> When I previously discussed that with mshal he said the following (edited to
> replace locations specific to my setup):
> 
> "A little background about tup: by default it keeps track of everything in the
> directory where 'tup init' gets run, and all subdirectories

Hmm.  So if my objdirs are siblings of my srcdir, could I just run "tup 
init" in the common parent?

Being able to have objdirs outside the srcdir is pretty important in my 
case; it makes it a lot easier to search code without having the objdir 
interfere with the searches...

-Boris
0
Boris
8/7/2018 3:32:41 PM
--0000000000004d681a0572da57a2
Content-Type: text/plain; charset="UTF-8"

On Tue, Aug 7, 2018 at 9:37 AM Boris Zbarsky <bzbarsky@mit.edu> wrote:

> Being able to have objdirs outside the srcdir is pretty important in my
> case; it makes it a lot easier to search code without having the objdir
> interfere with the searches...
>

Depending on what tool you use, it should be possible to exclude objdir's
from search anyway. I know ripgrep[1] (for example) will look at
`.gitignore` in-tree and ignore any top-level directories that begin with
`obj`. I guess pretty much any tool should be able to be configured to
ignore object directories similarly.

-- Tom

[1] https://github.com/BurntSushi/ripgrep

--0000000000004d681a0572da57a2
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr">On Tue, Aug 7,=
 2018 at 9:37 AM Boris Zbarsky &lt;<a href=3D"mailto:bzbarsky@mit.edu">bzba=
rsky@mit.edu</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Being able to have objdirs outside the srcdir is pretty important in my <br=
>
case; it makes it a lot easier to search code without having the objdir <br=
>
interfere with the searches...<br></blockquote><div><br></div><div>Dependin=
g on what tool you use, it should be possible to exclude objdir&#39;s from =
search anyway. I know ripgrep[1] (for example) will look at `.gitignore` in=
-tree and ignore any top-level directories that begin with `obj`. I guess p=
retty much any tool should be able to be configured to ignore object direct=
ories similarly.</div><div><br></div><div>-- Tom<br></div><div><br></div><d=
iv>[1] <a href=3D"https://github.com/BurntSushi/ripgrep">https://github.com=
/BurntSushi/ripgrep</a><br></div></div></div>

--0000000000004d681a0572da57a2--
0
Tom
8/7/2018 3:49:25 PM
On 8/7/18 11:49 AM, Tom Prince wrote:
> Depending on what tool you use, it should be possible to exclude 
> objdir's from search anyway.

There are multiple tools, depending on what I'm doing.  "find", "grep", 
"mkid" (from idutils) are the most common ones.

"find" and "mkid" do have ways to exclude directories via command-line 
options, but not a permanent configuration that I'm aware of.

-Boris
0
Boris
8/7/2018 4:54:25 PM
--00000000000065c5da0572e71bfd
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

 >  * You must be using an in-tree OBJDIR.

Does it handle multiple in-tree objdirs? E.g. I have d64 (debug), o64
(opt), etc.

Nick


On Tue, Aug 7, 2018 at 11:18 PM, Jonathan Watt <jwatt@jwatt.org> wrote:

> This is great! A big thank you to all the folks who have worked for so
> long to
> get us to this stage. I'm really looking forward to this work maturing an=
d
> extending to other platforms.
>
> I've done a bit of testing on my workstation (14 core/28 thread i9, U.2
> connected NVMe SSD, 64 GB DDR4 RAM) and got the following build times:
>
> For -j28 (yes, -j28 is well into diminishing returns territory, but I
> wanted a
> comparison of the best case scenarios for both backends):
>
>       |      cold       |    top-level
>       |  initial build  |  no-op rebuild
> ------------------------------------------
> make  |      8m 10s     |       15s
> tup   |      6m 35s     |        1.4s
>
> And for -j8:
>
>       |      cold       |    top-level
>       |  initial build  |  no-op rebuild
> ------------------------------------------
> make  |     11m 33s     |       15s
> tup   |     10m 52s     |        1.4s
>
> (To confirm: the rebuild numbers are the same for both -j28 and -j8.)
>
> So a 20% reduction in build time on cold builds of a fresh tree with -j28=
,
> and
> 6% reduction with -j8. In addition to speeding up the build, Tup seems to
> parallelize better which is great! Of course the highlight is the big
> reduction
> in rebuild overhead which will make the edit-compile-run cycle a lot less
> annoying (for me, 50s down to 37s after touching nsDocumentViewer.cpp,
> rebuilding, and linking libxul, for both -j28 and -j8.)
>
> These improvements, plus the promise of rarely having to clobber/figure o=
ut
> which directories to rebuild in, put a big smile on my face. (Currently
> having
> to turn off CARGO_INCREMENTAL, not being able to use sccache and other
> current
> limitations, less so, but it's early days. :) )
>
> A few notes for others trying this out (thanks cmanchester and mshal for
> the
> help debugging some of these):
>
>   * Make sure you use:
>
>       --enable-build-backends=3DTup
>
>     not:
>
>       --enable-build-backend=3DTup
>
>   * You must be using an in-tree OBJDIR.
>
>   * MOZ_PARALLEL_BUILD isn't currently recognized, so you'll need to use
>     `mach build`s -j argument to control the parallel job count for now
>     ( see https://bugzilla.mozilla.org/show_bug.cgi?id=3D1481441 )
>
> mshal: I think you could cross-post this to dev-platform for wider
> testing. I
> know there are a bunch of caveats (which people should be made aware of),
> but
> since the initial teething issues I had were figured out I've so far had =
no
> other problems building, or rebuilding with real code changes.
>
> Jonathan
>
> On 02/08/2018 16:34, Michael Shal wrote:
> > Hi everyone,
> >
> >
> > Tup is a modern build system that enables fast and correct builds. I'm
> pleased
> > to announce the availability of Tup for building Firefox on Linux.
> Compared to
> > Make, building with Tup will result in faster incremental build times
> and reduce
> > the need for clobber builds. See the in-tree docs for details [1].
> >
> >
> > We=E2=80=99re looking for some early adopters to try it out and let us =
know if it
> > improves your workflow, or if you hit any blockers or barriers to
> adoption that
> > keep you on the Make backend for now.
> >
> >
> > Quick howto:
> >
> >
> > You=E2=80=99ll need to install the Tup executable, as well as the night=
ly
> rust/cargo
> > toolchain:
> >
> >
> > cd ~/.mozbuild && mach artifact toolchain --from-build linux64-tup
> > rustup install nightly
> > rustup default nightly
> >
> > Your mozconfig needs to describe how to find the executable if it=E2=80=
=99s not
> in your
> > PATH, and enable the Tup backend:
> >
> >
> > export TUP=3D~/.mozbuild/tup/tup
> > ac_add_options --enable-build-backends=3DTup
> >
> >
> > If you have any issues, feel free to file a bug blocking =E2=80=9Cbuild=
tup=E2=80=9D [2],
> or
> > contact mshal or chmanchester in #build on IRC.
> >
> > [1] https://firefox-source-docs.mozilla.org/build/buildsystem/tup.html
> >
> > [2] https://bugzilla.mozilla.org/show_bug.cgi?id=3Dbuildtup
> >
> >
> > Thanks!
> >
> > -Mike
> >
> >
>
> _______________________________________________
> dev-builds mailing list
> dev-builds@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-builds
>

--00000000000065c5da0572e71bfd
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>
&gt;=C2=A0 * You must be using an in-tree OBJDIR.</div><div><br></div><div>=
Does it handle multiple in-tree objdirs? E.g. I have d64 (debug), o64 (opt)=
, etc.<br></div><div><br></div><div>Nick<br></div><br></div><div class=3D"g=
mail_extra"><br><div class=3D"gmail_quote">On Tue, Aug 7, 2018 at 11:18 PM,=
 Jonathan Watt <span dir=3D"ltr">&lt;<a href=3D"mailto:jwatt@jwatt.org" tar=
get=3D"_blank">jwatt@jwatt.org</a>&gt;</span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">This is great! A big thank you to all the folks who have work=
ed for so long to<br>
get us to this stage. I&#39;m really looking forward to this work maturing =
and<br>
extending to other platforms.<br>
<br>
I&#39;ve done a bit of testing on my workstation (14 core/28 thread i9, U.2=
<br>
connected NVMe SSD, 64 GB DDR4 RAM) and got the following build times:<br>
<br>
For -j28 (yes, -j28 is well into diminishing returns territory, but I wante=
d a<br>
comparison of the best case scenarios for both backends):<br>
<br>
=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 cold=C2=A0 =C2=A0 =C2=A0 =C2=A0|=
=C2=A0 =C2=A0 top-level<br>
=C2=A0 =C2=A0 =C2=A0 |=C2=A0 initial build=C2=A0 |=C2=A0 no-op rebuild<br>
------------------------------<wbr>------------<br>
make=C2=A0 |=C2=A0 =C2=A0 =C2=A0 8m 10s=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =
=C2=A0 =C2=A015s<br>
tup=C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 6m 35s=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 1.4s<br>
<br>
And for -j8:<br>
<br>
=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 cold=C2=A0 =C2=A0 =C2=A0 =C2=A0|=
=C2=A0 =C2=A0 top-level<br>
=C2=A0 =C2=A0 =C2=A0 |=C2=A0 initial build=C2=A0 |=C2=A0 no-op rebuild<br>
------------------------------<wbr>------------<br>
make=C2=A0 |=C2=A0 =C2=A0 =C2=A011m 33s=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =
=C2=A0 =C2=A015s<br>
tup=C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A010m 52s=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 1.4s<br>
<br>
(To confirm: the rebuild numbers are the same for both -j28 and -j8.)<br>
<br>
So a 20% reduction in build time on cold builds of a fresh tree with -j28, =
and<br>
6% reduction with -j8. In addition to speeding up the build, Tup seems to<b=
r>
parallelize better which is great! Of course the highlight is the big reduc=
tion<br>
in rebuild overhead which will make the edit-compile-run cycle a lot less<b=
r>
annoying (for me, 50s down to 37s after touching nsDocumentViewer.cpp,<br>
rebuilding, and linking libxul, for both -j28 and -j8.)<br>
<br>
These improvements, plus the promise of rarely having to clobber/figure out=
<br>
which directories to rebuild in, put a big smile on my face. (Currently hav=
ing<br>
to turn off CARGO_INCREMENTAL, not being able to use sccache and other curr=
ent<br>
limitations, less so, but it&#39;s early days. :) )<br>
<br>
A few notes for others trying this out (thanks cmanchester and mshal for th=
e<br>
help debugging some of these):<br>
<br>
=C2=A0 * Make sure you use:<br>
<br>
=C2=A0 =C2=A0 =C2=A0 --enable-build-backends=3DTup<br>
<br>
=C2=A0 =C2=A0 not:<br>
<br>
=C2=A0 =C2=A0 =C2=A0 --enable-build-backend=3DTup<br>
<br>
=C2=A0 * You must be using an in-tree OBJDIR.<br>
<br>
=C2=A0 * MOZ_PARALLEL_BUILD isn&#39;t currently recognized, so you&#39;ll n=
eed to use<br>
=C2=A0 =C2=A0 `mach build`s -j argument to control the parallel job count f=
or now<br>
=C2=A0 =C2=A0 ( see <a href=3D"https://bugzilla.mozilla.org/show_bug.cgi?id=
=3D1481441" rel=3D"noreferrer" target=3D"_blank">https://bugzilla.mozilla.o=
rg/<wbr>show_bug.cgi?id=3D1481441</a> )<br>
<br>
mshal: I think you could cross-post this to dev-platform for wider testing.=
 I<br>
know there are a bunch of caveats (which people should be made aware of), b=
ut<br>
since the initial teething issues I had were figured out I&#39;ve so far ha=
d no<br>
other problems building, or rebuilding with real code changes.<br>
<br>
Jonathan<br>
<div><div class=3D"h5"><br>
On 02/08/2018 16:34, Michael Shal wrote:<br>
&gt; Hi everyone,<br>
&gt; <br>
&gt; <br>
&gt; Tup is a modern build system that enables fast and correct builds. I&#=
39;m pleased<br>
&gt; to announce the availability of Tup for building Firefox on Linux. Com=
pared to<br>
&gt; Make, building with Tup will result in faster incremental build times =
and reduce<br>
&gt; the need for clobber builds. See the in-tree docs for details [1].<br>
&gt; <br>
&gt; <br>
&gt; We=E2=80=99re looking for some early adopters to try it out and let us=
 know if it<br>
&gt; improves your workflow, or if you hit any blockers or barriers to adop=
tion that<br>
&gt; keep you on the Make backend for now.<br>
&gt; <br>
&gt; <br>
&gt; Quick howto:<br>
&gt; <br>
&gt; <br>
&gt; You=E2=80=99ll need to install the Tup executable, as well as the nigh=
tly rust/cargo<br>
&gt; toolchain:<br>
&gt; <br>
&gt; <br>
&gt; cd ~/.mozbuild &amp;&amp; mach artifact toolchain --from-build linux64=
-tup<br>
&gt; rustup install nightly<br>
&gt; rustup default nightly<br>
&gt; <br>
&gt; Your mozconfig needs to describe how to find the executable if it=E2=
=80=99s not in your<br>
&gt; PATH, and enable the Tup backend:<br>
&gt; <br>
&gt; <br>
&gt; export TUP=3D~/.mozbuild/tup/tup<br>
&gt; ac_add_options --enable-build-backends=3DTup<br>
&gt; <br>
&gt; <br>
&gt; If you have any issues, feel free to file a bug blocking =E2=80=9Cbuil=
dtup=E2=80=9D [2], or<br>
&gt; contact mshal or chmanchester in #build on IRC.<br>
&gt; <br>
&gt; [1] <a href=3D"https://firefox-source-docs.mozilla.org/build/buildsyst=
em/tup.html" rel=3D"noreferrer" target=3D"_blank">https://firefox-source-do=
cs.<wbr>mozilla.org/build/buildsystem/<wbr>tup.html</a><br>
&gt; <br>
&gt; [2] <a href=3D"https://bugzilla.mozilla.org/show_bug.cgi?id=3Dbuildtup=
" rel=3D"noreferrer" target=3D"_blank">https://bugzilla.mozilla.org/<wbr>sh=
ow_bug.cgi?id=3Dbuildtup</a><br>
&gt; <br>
&gt; <br>
&gt; Thanks!<br>
&gt; <br>
&gt; -Mike<br>
&gt; <br>
&gt; <br>
<br>
</div></div>______________________________<wbr>_________________<br>
dev-builds mailing list<br>
<a href=3D"mailto:dev-builds@lists.mozilla.org">dev-builds@lists.mozilla.or=
g</a><br>
<a href=3D"https://lists.mozilla.org/listinfo/dev-builds" rel=3D"noreferrer=
" target=3D"_blank">https://lists.mozilla.org/<wbr>listinfo/dev-builds</a><=
br>
</blockquote></div><br></div>

--00000000000065c5da0572e71bfd--
0
Nicholas
8/8/2018 7:03:05 AM
--0000000000008bac990572ed560e
Content-Type: text/plain; charset="UTF-8"

On Tue, Aug 7, 2018 at 11:32 AM, Boris Zbarsky <bzbarsky@mit.edu> wrote:

> On 8/7/18 11:27 AM, Jonathan Watt wrote:
>
>> When I previously discussed that with mshal he said the following (edited
>> to
>> replace locations specific to my setup):
>>
>> "A little background about tup: by default it keeps track of everything
>> in the
>> directory where 'tup init' gets run, and all subdirectories
>>
>
> Hmm.  So if my objdirs are siblings of my srcdir, could I just run "tup
> init" in the common parent?
>
> Being able to have objdirs outside the srcdir is pretty important in my
> case; it makes it a lot easier to search code without having the objdir
> interfere with the searches...


 Yes, that should work, though with a caveat - you'd probably want to make
sure that the common parent is only used for one srcdir and objdir. So this
works fine:

mydir/mozilla-central
mydir/objdir
mydir/.tup

So you would run 'tup init' manually in "mydir", and have a mozconfig like:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../objdir
ac_add_options --build-backends=Tup

I've confirmed locally that this works, though you just get an innocuous
warning when mozbuild tries to run 'tup init' inside the srcdir: "tup
warning: database already exists in directory: .../mydir". We could improve
support for this by trying to run 'tup init' automatically in the parent
dir if the objdir is a sibling of the srcdir, which shouldn't be too hard
to add.

For comparison, the "default" case of an objdir inside the srcdir looks
like this:

mydir/mozilla-central
mydir/mozilla-central/objdir
mydir/mozilla-central/.tup

Without a file monitor, tup scans everything under where ".tup" exists, so
all of "mydir" in the first example and "mydir/mozilla-central" in the
default case.

The caveat applies in what I think jwatt's case is (jwatt, correct me if
I'm wrong - I may have misunderstood your setup) where you have multiple
trees and objdirs all as siblings. For example:

mydir/mozilla-central
mydir/mozilla-inbound
mydir/mozilla-beta
mydir/objdir-central-debug
mydir/objdir-central-opt
mydir/objdir-inbound-debug
mydir/objdir-inbound-opt
....
mydir/.tup

In this case, since we don't have the file monitor integrated yet, the
initial scan would likely take way too long and you'd lose much of the
benefit of switching backends in the first place. I think we could support
this structure in the future, but it would require some changes in tup
itself, and is likely not a quick fix.

So if you already have the srcdir/objdir pairs in an isolated parent, you
should be good to go now with a manual 'tup init' in the parent, and we can
support that workflow fairly easily. If it's the latter case where many
trees are in a single common root, I don't have a good solution for that at
the moment.

-Mike

--0000000000008bac990572ed560e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On T=
ue, Aug 7, 2018 at 11:32 AM, Boris Zbarsky <span dir=3D"ltr">&lt;<a href=3D=
"mailto:bzbarsky@mit.edu" target=3D"_blank">bzbarsky@mit.edu</a>&gt;</span>=
 wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.=
8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class=3D=
"gmail-">On 8/7/18 11:27 AM, Jonathan Watt wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left:1px solid rgb(204,204,204);padding-left:1ex">
When I previously discussed that with mshal he said the following (edited t=
o<br>
replace locations specific to my setup):<br>
<br>
&quot;A little background about tup: by default it keeps track of everythin=
g in the<br>
directory where &#39;tup init&#39; gets run, and all subdirectories<br>
</blockquote>
<br></span>
Hmm.=C2=A0 So if my objdirs are siblings of my srcdir, could I just run &qu=
ot;tup init&quot; in the common parent?<br>
<br>
Being able to have objdirs outside the srcdir is pretty important in my cas=
e; it makes it a lot easier to search code without having the objdir interf=
ere with the searches...</blockquote><div><br></div><div>=C2=A0Yes, that sh=
ould work, though with a caveat - you&#39;d probably want to make sure that=
 the common parent is only used for one srcdir and objdir. So this works fi=
ne:</div><div><br></div><div>mydir/mozilla-central</div><div>mydir/objdir</=
div><div>mydir/.tup</div><div><br></div><div>So you would run &#39;tup init=
&#39; manually in &quot;mydir&quot;, and have a mozconfig like:</div><div><=
br></div><div>mk_add_options MOZ_OBJDIR=3D@TOPSRCDIR@/../objdir<br>ac_add_o=
ptions --build-backends=3DTup</div><div><br></div><div>I&#39;ve confirmed l=
ocally that this works, though you just get an innocuous warning when mozbu=
ild tries to run &#39;tup init&#39; inside the srcdir: &quot;tup warning: d=
atabase already exists in directory: .../mydir&quot;. We could improve supp=
ort for this by trying to run &#39;tup init&#39; automatically in the paren=
t dir if the objdir is a sibling of the srcdir, which shouldn&#39;t be too =
hard to add.<br></div><div><br></div><div>For comparison, the &quot;default=
&quot; case of an objdir inside the srcdir looks like this:</div><div><br><=
/div><div>mydir/mozilla-central</div><div>mydir/mozilla-central/objdir</div=
><div>mydir/mozilla-central/.tup</div><div><br></div><div>Without a file mo=
nitor, tup scans everything under where &quot;.tup&quot; exists, so all of =
&quot;mydir&quot; in the first example and &quot;mydir/mozilla-central&quot=
; in the default case.</div><div><br></div><div>The caveat applies in what =
I think jwatt&#39;s case is (jwatt, correct me if I&#39;m wrong - I may hav=
e misunderstood your setup) where you have multiple trees and objdirs all a=
s siblings. For example:</div><div><br></div><div>mydir/mozilla-central</di=
v><div>mydir/mozilla-inbound</div><div>mydir/mozilla-beta</div><div>mydir/o=
bjdir-central-debug</div><div>mydir/objdir-central-opt</div><div>mydir/objd=
ir-inbound-debug</div><div>mydir/objdir-inbound-opt</div><div>...</div><div=
>mydir/.tup<br></div><div><br></div><div>In this case, since we don&#39;t h=
ave the file monitor integrated yet, the initial scan would likely take way=
 too long and you&#39;d lose much of the benefit of switching backends in t=
he first place. I think we could support this structure in the future, but =
it would require some changes in tup itself, and is likely not a quick fix.=
</div><div><br></div><div>So if you already have the srcdir/objdir pairs in=
 an isolated parent, you should be good to go now with a manual &#39;tup in=
it&#39; in the parent, and we can support that workflow fairly easily. If i=
t&#39;s the latter case where many trees are in a single common root, I don=
&#39;t have a good solution for that at the moment.</div><div><br></div><di=
v>-Mike<br></div></div></div></div>

--0000000000008bac990572ed560e--
0
Michael
8/8/2018 2:29:27 PM
--000000000000c4cd5b0572ed7353
Content-Type: text/plain; charset="UTF-8"

On Wed, Aug 8, 2018 at 3:03 AM, Nicholas Nethercote <n.nethercote@gmail.com>
wrote:

> >  * You must be using an in-tree OBJDIR.
>
> Does it handle multiple in-tree objdirs? E.g. I have d64 (debug), o64
> (opt), etc.
>

That should work with a similar caveat that tup is going to scan everything
under where ".tup" exists, which would include both objdirs. So there will
be some additional startup overhead until we have the file monitor
integrated -- I guess at the moment it's a question of how many objdirs you
have and how much of an impact it adds to the startup overhead. I'll try it
out locally myself, but I'd be interested to hear your experiences as well!

-Mike

--000000000000c4cd5b0572ed7353
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On W=
ed, Aug 8, 2018 at 3:03 AM, Nicholas Nethercote <span dir=3D"ltr">&lt;<a hr=
ef=3D"mailto:n.nethercote@gmail.com" target=3D"_blank">n.nethercote@gmail.c=
om</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"marg=
in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"=
><span class=3D""><div>
&gt;=C2=A0 * You must be using an in-tree OBJDIR.</div><div><br></div></spa=
n><div>Does it handle multiple in-tree objdirs? E.g. I have d64 (debug), o6=
4 (opt), etc.<br></div></div></blockquote><div><br></div><div>That should w=
ork with a similar caveat that tup is going to scan everything under where =
&quot;.tup&quot; exists, which would include both objdirs. So there will be=
 some additional startup overhead until we have the file monitor integrated=
 -- I guess at the moment it&#39;s a question of how many objdirs you have =
and how much of an impact it adds to the startup overhead. I&#39;ll try it =
out locally myself, but I&#39;d be interested to hear your experiences as w=
ell!</div><div><br></div><div>-Mike<br></div></div></div></div>

--000000000000c4cd5b0572ed7353--
0
Michael
8/8/2018 2:37:37 PM
On 8/8/18 10:29 AM, Michael Shal wrote:
>   Yes, that should work, though with a caveat - you'd probably want to 
> make sure that the common parent is only used for one srcdir and objdir. 

I definitely have multiple objdirs per srcdir.  ;)  opt and debug at the 
very least; in some cases a few others.

> The caveat applies in what I think jwatt's case is (jwatt, correct me if 
> I'm wrong - I may have misunderstood your setup) where you have multiple 
> trees and objdirs all as siblings.

Right, that's not my situation.  I have basically a bunch of dirs like 
(names changed for clarity):

   dir1/srctree
   dir1/obj-debug
   dir1/obj-opt
   dir2/srctree
   dir2/obj-debug
   dir2/obj-opt
   dir2/obj-opt-asan

etc.  So all the srcdirs are siloed off from each other in my case.

Sounds like in my case tup would scan all the objdirs and the srcdir, right?

-Boris
0
Boris
8/8/2018 4:59:21 PM
--000000000000c204310572f1ca52
Content-Type: text/plain; charset="UTF-8"

On Wed, Aug 8, 2018 at 12:59 PM, Boris Zbarsky <bzbarsky@mit.edu> wrote:

> On 8/8/18 10:29 AM, Michael Shal wrote:
>
> The caveat applies in what I think jwatt's case is (jwatt, correct me if
>> I'm wrong - I may have misunderstood your setup) where you have multiple
>> trees and objdirs all as siblings.
>>
>
> Right, that's not my situation.  I have basically a bunch of dirs like
> (names changed for clarity):
>
>   dir1/srctree
>   dir1/obj-debug
>   dir1/obj-opt
>   dir2/srctree
>   dir2/obj-debug
>   dir2/obj-opt
>   dir2/obj-opt-asan
>
> etc.  So all the srcdirs are siloed off from each other in my case.
>
> Sounds like in my case tup would scan all the objdirs and the srcdir,
> right?
>

Yes, so it should work if you run 'tup init' manually in dir1 and again in
dir2. I ran some tests on my machine, and it looks like each additional
objdir only added about 100ms to the scan time. So I think with a handful
of objdirs it's not likely you'll notice a huge difference as compared to a
single objdir. I had 3 objdirs and a no-op 'mach build' was still under 2
seconds, but YMMV :)

-Mike

--000000000000c204310572f1ca52
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On W=
ed, Aug 8, 2018 at 12:59 PM, Boris Zbarsky <span dir=3D"ltr">&lt;<a href=3D=
"mailto:bzbarsky@mit.edu" target=3D"_blank">bzbarsky@mit.edu</a>&gt;</span>=
 wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bor=
der-left:1px #ccc solid;padding-left:1ex"><span class=3D"">On 8/8/18 10:29 =
AM, Michael Shal wrote:</span><span class=3D""><br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
The caveat applies in what I think jwatt&#39;s case is (jwatt, correct me i=
f I&#39;m wrong - I may have misunderstood your setup) where you have multi=
ple trees and objdirs all as siblings.<br>
</blockquote>
<br></span>
Right, that&#39;s not my situation.=C2=A0 I have basically a bunch of dirs =
like (names changed for clarity):<br>
<br>
=C2=A0 dir1/srctree<br>
=C2=A0 dir1/obj-debug<br>
=C2=A0 dir1/obj-opt<br>
=C2=A0 dir2/srctree<br>
=C2=A0 dir2/obj-debug<br>
=C2=A0 dir2/obj-opt<br>
=C2=A0 dir2/obj-opt-asan<br>
<br>
etc.=C2=A0 So all the srcdirs are siloed off from each other in my case.<br=
>
<br>
Sounds like in my case tup would scan all the objdirs and the srcdir, right=
?<br></blockquote><div><br></div><div>Yes, so it should work if you run &#3=
9;tup init&#39; manually in dir1 and again in dir2. I ran some tests on my =
machine, and it looks like each additional objdir only added about 100ms to=
 the scan time. So I think with a handful of objdirs it&#39;s not likely yo=
u&#39;ll notice a huge difference as compared to a single objdir. I had 3 o=
bjdirs and a no-op &#39;mach build&#39; was still under 2 seconds, but YMMV=
 :)</div><div><br></div><div>-Mike<br></div></div></div></div>

--000000000000c204310572f1ca52--
0
Michael
8/8/2018 7:48:17 PM
On 8/8/18 3:48 PM, Michael Shal wrote:
> Yes, so it should work if you run 'tup init' manually in dir1 and again 
> in dir2. I ran some tests on my machine, and it looks like each 
> additional objdir only added about 100ms to the scan time.

That's awesome.  ;)

-Boris
0
Boris
8/8/2018 8:41:36 PM
Reply: