=Paper=
{{Paper
|id=Vol-2510/sattose2019_paper_2
|storemode=property
|title=Lessons and Pitfalls in Building Firefox with Tup
|pdfUrl=https://ceur-ws.org/Vol-2510/sattose2019_paper_2.pdf
|volume=Vol-2510
|authors=Guillaume Maudoux,Kim Mens
|dblpUrl=https://dblp.org/rec/conf/sattose/MaudouxM19
}}
==Lessons and Pitfalls in Building Firefox with Tup==
https://ceur-ws.org/Vol-2510/sattose2019_paper_2.pdf
Lessons and Pitfalls in Building Firefox with Tup
Guillaume Maudoux and Kim Mens
UCLouvain (Belgium)
Email: {guillaume.maudoux,kim.mens}@uclouvain.be
Abstract—Build system implementations are surprisingly nu-
merous for the single common purpose of assembling software.
With this variety, picking the right one is a complex task. And
even more difficult is the migration to a new build system, with
uncertain benefits at the end. Software maintainers and release
engineers need better comparisons of build systems and precise
catagorisation on which to base an informed decision. As a first
step toward that goal, we experimented building Firefox with
Tup in replacement of Make. We report here our experience at
migrating and comparing the build systems. We also describe
interesting features of Tup and we discus Mozilla’s Firefox usage
as a benchmark for build systems.
I. INTRODUCTION
We started investigating Mozilla’s build system a year
ago, in the hope of finding a large code base for testing
different build systems in a realistic setup. We settled on
Mozilla’s Firefox because their build infrastructure is designed
in such a way that different build systems can be plugged in.
This is no coincidence, but an ongoing work at Mozilla to
update their build system. The rationale being that picking
the right replacement for Make would require testing different
alternatives, and each alternative would require a complete Fig. 1. Firefox’s build system is split in two phases and can use different
port. With a generic build definition, they can test different backends to perform builds. Depending on the intended backend, different
build definitions are generated.
build systems and let experimentations drive the selection
of the next build system. Other considerations, like gaining
control over the build definition, and using a widely known their complex codebase. Finally, we accumulated experience
language to do so were taken into account when discussing in migrating from one build system to an other, and learned
this change [1, 2]. the hard way that the process is theoretically simple but very
We started implementing the Tup backend last in August technical in practice.
2017, when Mozilla’s support for it was minimal. It later
appeared that Tup was also the next target build system for II. C ONTEXT
Mozilla itself, which led to two independent implementations For readers who may not be already familiar with these, we
(ours and theirs). Mozilla’s effort to use Tup has focused on briefly depict Mozilla’s build system, Firefox itself and the
producing reusable, clean code by fixing one issue at a time. two build systems considered this article: Make and Tup.
Our focus was on getting the build to work, regardless of a) Mozilla’s build system: Mozilla’s modular build setup
the code quality. Over the time, both implementations have works in two phases, as depicted in Figure 1. moz.build files
converged and as of August 2018, Mozilla’s implementation describe the build with a Mozilla-specific python DSL. This
should be preferred as it is now complete and is the only one format is parsed and Makefiles are generated in such a way
that will be further maintained and updated. that the remaining of the build can be handled by Make. As
This paper presents the insights we gained on three main stated before, this design accepts new emitters, or back-ends
areas. With this work, we investigated and gained an in-depth to be used in place of the Makefile generator. This is how the
knowledge of Tup’s capabilities and interals. We also learned a Tup backend was implemented. Make and Tup are examples
lot about Firefox’s build system design and their tactics to tame of backends used for compilation, but the same mechanism
— is also used to extract information from the build system to
Copyright © by the paper’s authors. Use permitted under Creative Commons be used by other tools. For example, a backend generates a
License Attribution 4.0 International (CC BY4.0) compilation database for advanced autocompletion in some
In: Anne Etien (eds.): Proceedings of the 12th Seminar on Advanced Tech-
niques Tools for Software Evolution, Bolzano, Italy, July 8-10 2019, published IDEs [3, 4]. This flexibility makes it possible to test different
at http://ceur-ws.org/ build systems and inspect Mozilla’s Firefox compilation.
b) Firefox codebase: Firefox is a libre, open-source again from scratch. A correct build system is one that always
browser with a large code base. The core application contained rebuilds what needs to be rebuilt. When an incremental build
about 4M lines of code in 2013, and was growing steadily [5]. is complete, the binaries reflect the sources exactly, and the
This does not count tests and other configuration files. At that developer need not worry about the build system’s state and
time, the full repository contained about 15M lines of code, shortcomings. She therefore can “stay focused on her project”.
and has now reached above 36M lines of code as of 2018 [6], To summarize, Tup has three claims, each of which we will
which makes it bigger than the Linux kernel and Libre Office try to verify in this paper.
according to Open Hub. It is written in a variety of languages. 1) Performance of execution,
C++ forms the core application, supplemented by JavaScript 2) Minimal incremental updates,
on top of the core engine. The repository contains also HTML, 3) Always correct build outputs.
C, python, java and has recently seen the introduction of Rust,
Mozilla’s designed systems programming language. A large These features arise from the design and careful implematation
code base, with a variety of languages forming an open-source of the algorithms detailed in [12]. They are hard to demonstrate
and well known application. All these features makes it a good by experiment as they should hold by design. We have
candidate for meaningful build systems comparison. nonetheless tested Tup’s efficiency at building Firefox and
c) Make: Speaking of build systems, Make is the refer- gathered the results presented in the next section.
ence application in that domain. It was designed at Bell Labs Minimal updates and correctness can be investigated by a
in 1976 and was derived and reimplemented many times since deep understanding of how Tup works and have real impacts
then [7, 8]. Make’s configuration language used in Makefiles on Tup usage, especially on large software systems like Fire-
is well known by a large proportion programmers. Over the fox. It must first be noted that minimal updates and correctness
years, Make has seen many disussions about its shortcomings, somehow conflict. Stricter notions of correctness will require
and even more discussions about good practices and usage. more rebuilds. While still being minimal, the build system will
Most notably, the paper “Recursive Make Considered Harm- rebuild much more than with other build systems, leading to
ful [9]” discussed the good usage of Make on large software the impression that it is uselessly recompiling more parts. This
projects, advocating for efficiency at the expense of modular- is exactly what happens with Tup.
ity. Strinking the right balance between conflicting goals is still
a challenge for current system implementations [10]. Make IV. B UILDING F IREFOX WITH T UP
has always been the build system of Firefox and is tightly Tup focuses on correctness and speed, at the expense of
integrated in its code base. expressivity or usability whenever they conflict with the former
d) Tup: From Tup’s home page, we see that “Tup is aspects. In some sense, Tup is minimalist and tries to do one
a file-based build system for Linux, OSX, and Windows. It thing well: building. The configuration language of Tup is not
inputs a list of file changes and a directed acyclic graph very expressive. It allows to specify commands, inputs and
(DAG), then processes the DAG to execute the appropriate outputs with basic support for variables and conditions. To be
commands required to update dependent files. Updates are fair, Lua integration is provided for more advanced features
performed with very little overhead since Tup implements but was not tested in this article.
powerful build algorithms to avoid doing unnecessary work. We will report here first on our experience with Tup, sepa-
This means you can stay focused on your project rather than rated in two key aspects: correctness and speed. Expressivity
on your build system” [11]. Besides optimization and careful will be discussed later on, in Section VI.
implementation, what makes Tup truly special is its ability
to trace command executions to obtain real dependencies
A. Correctness
and outputs. Validating the declared dependencies with the
observed file accesses adds a safeguard against imprecise Correctness is a key feature of build systems. It can be
declaration and potential speedups that we will discuss later. defined roughly as the property of producing valid products
despite optimizations like incremental compilation [10, 13]. In
III. R ESEARCH OBJECTIVES Tup, correctness is achieved by maintaining the state of the
From the above description of Tup, it is clear that Tup is build in a separate database and by tracing all the file accesses
focussed on performance of updates. Detailed explanations on performed by build commands.
“avoid doing unnecessary work” and “stay focussed on your Maintaining the state of the build in an external database,
project” are to be found later on Tup’s home page or in [12]. allows Tup to detect changes to the build description. When-
In a nutshell, avoiding redundant work is the aim of a minimal ever a command is added, removed or modified, Tup will
incremental build system. Minimalism ensures that up-to-date take proper actions to update the build when invoked [12]. By
outputs are maintained without running the tasks that produce design, this is difficult or impossible to achieve with stateless
them. Tup claims that it is minimal in that sense. “Stay build systems like Make. In particular, they cannot detect
focused” hides the even more complex idea of correctness. removed commands as there is no trace of them on the next
Some build system can get corrupted, or out of sync and it is invocation. This is not specific to Tup. Most (if not all) recent
not uncommon for developers to clean their build tree and start build system implementations now rely on persistent storage.
1 tup error: Unspecified output files - A command is writing to files that you didn't specify in
,→ the Tupfile. You should add them so tup knows what to expect.
2 tup error: Expected to write to file 'libxul.so' from cmd 15846 but didn't
3 tup error: Missing input dependency - a file was read from, and was not specified as an input
,→ link for the command. This is an issue because the file was created from another command,
,→ and without the input link the commands may execute out of order. You should add this file
,→ as an input, since it is possible this could randomly break in the future.
Fig. 2. Typical errors produced by Tup when inputs and outputs are not properly specified. These errors are reported respectively when a command 1) writes
to a file that was not specified as an output, 2) does not write to a file specified as an output and 3) tries to read a file that is not declared as an input.
rough super-set of these. By adding all the objects to the same
Unlike Make, Tup detects changes to the build com- Tup group linker invocations can depend solely on it, at the
mands, triggering the required command invocations. expense of introducing stages in the build, because linking
will have to wait for all the objects to be compiled before
By tracing file accesses of each and every command in the running. The real dependencies are then obtained at execution
build plan (a.k.a. build steps), Tup ensures that dependencies by tracing the command. This ensures accurate detection of
and outputs are correctly specified. In particular, Tup will updated inputs despite the over-approximation in the Tupfiles,
refuse to build a command that uses undeclared dependencies. and therefore avoids pointless rebuilds of all the libraries when
Some typical error messages can be seen on Figure 2. While a single object changes.
this allows to detect hidden dependencies and fix them, we
often stumbled upon this constraint during our migration Tup’s “groups” feature allows over-approximating the
because it interrupts the build, and the Tupfiles need to be dependencies of a build step.
generated again. Tup does not allow to bypass these issues.
Tracing is used as enforcement of the declared dependencies
and for speed optimizations as discussed later. Tup does The technical issue of specifying all the dependencies
not dynamically detect and change the build plan from the also arises in a other places such as with unified builds.
collected information. The unified builds optimization works by aggregating several
For example, we hit this constraint with fake static libraries C/C++ source file together. The compiler only needs to parse
(.a.desc files) built by Mozilla in place of the usual static the presumably large set of included headers once for the
libraries (.a files) whenever the static library will only be unified set, reducing compilation time on large rebuilds and
linked with other libraries, and not exported or otherwise on header changes [14]. This technique speeds up full rebuilds
directly used. In that case, the .a.desc file contains only a list significantly at the expense of slower incremental builds for
of the objects that should be present in the archive. That list small changes. In this case, the C/C++ compiler receives a
is then parsed and the real objects fed to the linker instead of dummy unified input file that includes the original C/C++
the .a archive itself. This avoids the creation of real .a files source files. From the point of view of Tup, the command
that can be quite time and disk-space consuming. depends on the dummy unified source file, but also on all the
This optimization conflicts with Tup’s need to know all the source files included therein.
dependencies of a command explicitly. Whenever a command This situation highlighted for us a specificity of Tup: source
links against the .a.desc, it implicitly depends on all the objects files are not required to be specified as inputs. In fact, the
listed therein and this lists needs to be explicitly expanded for set of input files is always an implicit input to all the rules.
Tup when generating Tupfiles. The rationale being that Tup is able to detect the sources
that are accessed by a particular command and source files
have no impact on command ordering. The execution order is
Tup strengthens the build description by detecting
only constrained by generated files, as consumers cannot run
undeclared and hidden dependencies. They all must
before producers. To relate this to groups, Tup behaves just as
be declared in the Tupfiles or Tup will error out.
if all the non-generated files belonged to a “sources” group,
automatically added as a dependency of all the commands.
In Mozilla’s implementation this is now avoided by skipping We experienced this style of never specyfying source files in
fake libraries. They are not generated, and linkers are fed rules, and this results in unusual, if not confusing, rules that
directly the full list of objects. This is an optimization over have no declared inputs at all.
our version. Since the full list of objects needs to be expanded
during configuration anyway, there is no reason not to feed it With Tup, there is no need to specify source files as
directly to the linker. inputs. All the commands implicitly depend on them.
In our first implementation, we avoided this issue by using
Tup groups; a feature that allows to some extent to alleviate
the need to list all the required dependencies by providing a Tup is also quite pedantic on the outputs that a command
is allowed to produce. The command must write to all its
declared outputs, and nowhere else. The issue occurred with Tup’s strong correctness guarantees come at the cost
Rust code. Due to a lack of integration at the moment, Rust of strict constraints on invoked commands and on the
libraries must be compiled in a single step, and therefore build description. There is no free lunch.
discard all outputs outside of the expected output library. This
prevents Rust from doing caching of builds, or requires to
B. Tup performance
build Rust libraries outside of the build and outside of Tup
dependency tracking. Rust also suffixes libraries with a hash Tup uses a specific data model to achieve high speeds with
of their content. To work around these issues, we had to wrap carefully crafted algorithms [12]. We first take a closer look at
Rust invocations with a cleanup script that removes extra files the three different aspects of Tup targeting performance, and
and renames the generated library. then describe our benchmark on Firefox.
First, Tup comes with a monitor that can listen to filesystem
events, collecting source changes on the fly. This optimization
Tup requires commands to produce the exact list of is nowadays used by all the build systems targetting large code
declared outputs, refusing commands that generate bases, like Bazel, Gradle or Pants. The monitor removes the
more, less or custom files on different invocations. need to walk the source tree searching for changes on each
build system invocation. This optimization is most noticeable
Even though these limitations sometimes prevent building for builds in an up-to-date workspace (a.k.a. null builds). In
with Tup, it is still possible to trick it in several ways. First, one our tests, it reduced Tup execution time from one second to
can invoke some build steps before running Tup. The produced less than a millisecond. This result probably also applies to
files are seen as plain source files by Tup. It however removes the build systems cited here-above.
Tup’s ability to detect changes and rebuild these files. In the
absence of other change detection mechanisms, these files need When everything is up-to-date and Tup’s monitor is
to be rebuilt on each invocation. The other option is to build enabled, Tup runs in less than a millisecond.
outside of the source tree, because Tup does not track files
there (unless configured to). The required outputs can then be
moved to the source tree once the command finishes. With this Also, Tup maintains the build graph in such a way that
technique, complex commands can maintain a cache across changes and rebuilds can be propagated upwards, without
builds. This is otherwise not possible with Tup constraints. The loading the whole dependency DAG [12]. Theoretically, this
potential pitfall is that build correctness relies completely in makes Tup very efficient at small rebuilds because the com-
the sub-command, as Tup has no knowledge of what happens plexity of the algorithm depends mostly on the size of the
outside of it’s monitoring. Reaching that point means that Tup update, and less on the size of the whole repository as is the
gets in the way more than it should. Commands such as Rust case with Make.
incovations require structural modifications to work properly Finally, Tup’s build graph is enriched with exact dependen-
with Tup. Mozilla’s implementation was defered until they cies detected by tracing the commands durig their execution.
updated the Rust toolchain to better support Tup [15]. This This ensures that build steps do not depend on files they do
was made possible because Rust already intended to support not need, and that there is no spurious rebuilds when such a
that usage [16], but may not be feasible with other projects or file changes. That being said, tracing has also a performance
affordable to any Tup user. penalty, and can reduce Tup’s speed on filesystem-intensive
tasks like linking Firefox’s largest binary [17].
We ran a small-scale experiment to compare build speed
Some tools are structurally conflicting with Tup. They with Make and Tup. We selected 47 consecutive pushes to
have to be modified or Tup needs to be bypassed. Firefox1 and compiled these changes incrementally in merge
order. The pushes were taken from mozilla-inbound, the
The complications induced by the strict enforcement of integration branch at Mozilla. A push is a set of commits
policies by Tup ultimately provide strong guarantees on the that are added together, and tested as one atomic change
validity of the build results. While it is possible to inadver- by Mozilla’s continuous integration system. The number of
tently omit a dependency in Make, it is nearly imposible to commits is limited to 47 because it is the largest range
do so in Tup. This makes building with Tup more robust to of consecutive commits that appeared to build correctly in
configuration errors or undetected interferences. In the long the benchmarking enviroment. Building more commits would
term, the balance leans in favor of strict build systems like have required to use different build environments, or looking
Tup because migrating to Tup requires solving tricky issues at more commit ranges. With 47 commits, we were able
only once, while an incorrect build system can produce subtle to measure the duration of 46 different incremental builds.
inconsistencies between the source code and the build products Figure 3 shows the incremental build times of Make and Tup
during any build invocation. 1 The full list of 47 pushes is accessible at https://hg.mozilla.org/integration/
mozilla-inbound/pushloghtml?startID=102811&endID=102858
Fig. 3. Duration of three incremental builds for 47 successive changesets with both Make and Tup
not visible to the mozbuild frontend.
1h We get a more detailed view of Make and Tup’s relative
performance on Figure 4. Average build times with Make
(on the left) are compared with average build times with
10min Tup (on the right) for each of the 47 commits. The graph
Build duration
is in logarithmic vertical scale, so it is difficult to get precise
values, but we see that the builds are indeed faster by a similar
1min proportion for most large builds, with Tup being slightly faster.
Looking at smaller rebuilds, we see that they get faster with
Tup, and sometimes much faster. This high speedup arises
10s when Tup detects that there is nothing to do. In that case, Tup
takes one second to search for modifications (and possibly
even faster with its monitor running, as explained above), and
1s then stops after deciding that no modifications means nothing
to compile. Make on the other hand needs to recurse into all
Make Tup the subdirectories and build the whole dependency graph to
Build backend detect the absence of anything to do. It further appears that
Make executes some build steps on each invocation, making it
Fig. 4. Relative speed of Make and Tup at building incremental changesets. impossible to measure the execution time of the Make process
The build duration of each changeset with each tool has been averaged over itself without the tasks being run. Unconditionally running
the three runs. Durations were rounded up to the next second.
tasks means that some were forced to run, most probably to
avoid issues with incomplete dependency handling in Make.
for each of these 47 commits. Three runs were made for each From that point of view, Make’s lack of correctness guarantees
build system to give an idea of the variance. The first measure translates into an execution overhead.
represents a full rebuild from clean sources, and shows how
much time is needed for a full build of Mozilla’s Firefox. With Tup is at least as good as Make on average, and much
the default Make backend, a bit more than 1h 15m are needed faster for small rebuilds.
to do so on our Intel Core i5-4310U which is a fairly recent
2 cores/4 threads CPU. All the builds were performed on the
same machine, with the same SSD drive.
On the graph, we observe a lot of short builds, thanks to V. G RASPING M OZILLA’ S BUILD DESIGN
the incremental optimizations performed by Tup and Make.
Also, the data series have the same shape, which shows that Another feature of Tup is that it allows easy extraction of
both backends compile about the same things. That being the build graph. Tup contains in its database the whole build
said, Make builds are generally slower than Tup ones on big graph that was just built. It also provides a tool to draw the
rebuilds. Thanks to the three measures, we can affirm that graph.
it is no random artifact of the build environment. The most The complete graph is not easy to understand, as it contains
probable cause is that the Make build performs more steps thousands of nodes, one for each command and for each
than the Tup build. In some places, the Tup build avoids calls file. We took some steps to simplify the graph based on
to wrappers and other fixtures used by Make. But it could also command labels and relative depth in the build tree to obtain
come from commands that are only described in Makefiles and the simplified version presented in figure 5.
python
dependentlibs.list
LINK LINK
7 commands 21 commands
7 files 21 files
LINK AR LINK
AR LINK
3 commands 2 commands 4 commands
libxpcomglue.a libnspr4.so
3 files 2 files 4 files
CXX CC AS
RUSTC
1670 commands 1144 commands 127 commands
libgkrust.a
1670 files 1144 files 127 files
python WebIDL AS
XPIDL
1 command 1 command 4 commands
xptdata.cpp
2 files 1457 files 4 files
python python XPIDL
1 command 5 commands 152 commands
2 files 11 files 2633 files
IPDL preprocess python
1 command 109 commands 70 commands
966 files 109 files 1261 files
sources
247373 files
Fig. 5. A simplified view of Firefox dependency graph. The node grouping algorithm outlines different stages of preprocessing, compiling, linking and the
final step of producing dependentlibs.list discussed in Section V. Nodes are grouped by type and by relative depth in the build system. Transitive dependencies
have been removed to avoid edges clutter, hiding in particular the fact that all the nodes depend directly on the sources.
909M gecko-dev/.tup (Tup internal state,
Tup stores the whole build plan (the build graph) in including and mostly due to Tup db)
2.2G gecko-dev (sources checkout only)
a database in an explicit form that can be queried
4.9G gecko-dev/.git (git repository)
easily. 5.8G gecko-dev/obj-tup (build outputs)
14G total
It is interesting to discover that the build can be summarized Fig. 6. Size of the main parts of our workspace after a complete build with
in a handful of nodes, and divides itself naturally in stages. Tup in a clean checkout.
There is a preprocessing and generation stage where symlinks
are created, where files are generated with python and other
custom transformations happen. Afterwards, we observe the stage, where shared objects depend on other shared objects.
classical two-stage process of compiling (AS, CC, CXX) to The graphs could be improved through better projections and
object files, and then linking into programs, static libs and simplifications. We intend to work further on these aspects to
shared libs (LINK and AR). The ultimate stage is an oddity at improve on this work and generalize our results.
Mozilla, where a dependentlibs.list is generated that lists all Extracting that graph would not have been easy with
the libraries that libxul.so depends upon. This file is used to recursive Makefiles. There have been attempts at extracting
preload all these libraries at startup, possibly for performance Makefile-based build system structure but they require to
reasons. actually run Make in verbose debug mode and parse its
This graph was the best result obtained after several at- output [18]. Tup provides a much more reliable way to access
tempts. It groups commands by the maximal depth of their that graph.
inputs. Files are merged with the command that produces Concerning that graph, it is worth noting that it requires
them. This kind of transformation does not preserve the a large amount of disk space. On our machine, after a fresh
acyclicity of the graph, as a loop appears in the linking build of Firefox, we observed the memory usage described in
EXPORTS += [ with Files('**'):
'MP3Decoder.h', BUG_COMPONENT = ('Toolkit', 'App Update')
'MP3Demuxer.h',
'MP3FrameParser.h', DIRS += ['src']
]
if CONFIG['MOZ_ENABLE_SIGNMAR']:
UNIFIED_SOURCES += [ DIRS += ['sign', 'verify']
'MP3Decoder.cpp', TEST_DIRS += ['tests']
'MP3Demuxer.cpp', elif CONFIG['MOZ_VERIFY_MAR_SIGNATURE']:
'MP3FrameParser.cpp', DIRS += ['verify']
]
# If we are building ./sign and ./verify,
FINAL_LIBRARY = 'xul' # then ./tool must come after it
DIRS += ['tool']
Fig. 7. A simple moz.build file (dom/media/mp3/moz.build) showing how
data can be encoded in this restricted python language. This format is used Fig. 8. A more advanced moz.build file, with conditionals, subdirectories and
by Mozilla to describe the build steps that need to be performed. bindings between source files and Firefox modules.
Figure 6. We see that the internal state requires an amount
of space comparable to the source checkout itself. From that the build options can depend on configuration values
quick investigations, we discovered that it is mainly due to passed in the CONFIG dictionary. These values are computed
dependencies between build steps. further investigations would at configuration time, and allow tweaking the build depending
be required to analyze how this database grows with repository on the platform and various other parameters.
sizes and programming languages, or possibly other significant
The Python format used in moz.build files was picked
factors to be discovered.
because it is easy to read and Python is well known among
The ability of Tup to inspect the build graph comes as a
developers, at Mozilla and in general [2]. This input format is
nice side-effect of using Tup as a build backend. It shows
parsed by the mozbuild toolchain that generates the right build
how using different backends can benefit Mozilla outside of
files according to the selected backend (Makefiles for make,
building per se. If Tup ends up not being used by Mozilla,
etc.) but it can also perform some complex operations on its
it remains useful for its easily inspect able build graph and
own. For example, the backend generates boilerplate unified
also for checking the dependencies declared in the moz.build
C++ files, and lists of inputs in separate files to circumvent
frontend files.
command-line length limits. The Tup backend performs more
Besides backends for building, this Mozilla-specific ar-
operations to work around Tup limitations. For example, it
chitecture opens the door to other tools instrumenting the
parses some input files to pre-compute the output file names
build system. For example, Mozilla already uses that build
that cannot be simply deduced. We call this step preprocessing,
infrastructure to extract a label for each file, in order to
and distinguish it from the parsing and generation done by
automatically relate code changes to Firefox components. It is
mozbuild. Preprocessing is mostly needed to work around
also used to generate compile-commands.json used by linters
limitations and restrictions in the build backend.
and IDE compilation databases [3]. Other tools such as linters
and software maintenance tools could be plugged in there, With its custom build description format in Python, Mozilla
making it low hanging fruit for a practical use case of new is independent from a build system in particular. Because
research tools. Python is a widely known and general purpose programming
language, it is possible to adapt it to multiple purposes and
VI. F IREFOX AS A BENCHMARK FOR BUILD SYSTEMS in particular to generate instructions for different backends as
As was already hinted before, Firefox is our candidate we did with Tup. As more backends are added, the difficulty
benchmark for comparing build systems. In this section, we to generalize the mozbuild pipeline should lower. Also, in the
explain in further details the structure of Mozilla’s build sys- process of implementing a backend for a build system, features
tem and the features that catched our attention while looking and limitations of said build system appear more clearly and
for a benchmarking project for build systems. can be collated. With several build systems executing on the
Mozilla’s build system is based on simplified python files same code base, it becomes possible to produce the meaningful
named moz.build. Examples of these files can be seen in comparisons that are not yet available. Our future work will
Figure 7 and Figure 8. They list the source files that must therefore target more build systems and will try to elicit
be compiled, the name of the libraries or programs that must distinctive features for each of them.
be produced and various options like compiler flags, headers
to be exported and such. The ad-hoc nature of this data Mozilla’s custom build framework can be easily
format allows Mozilla to encode other kind of information adapted to different build systems, and could be used
like in Figure 8 where information allows to relate files in the to compare them.
source tree with Firefox components. That example also shows
VII. R ELATED W ORK X. ACKNOWLEDGMENTS
The authors would like to thank Mike Shal for sharing
Andrey Mokhov et Al. distinguish different kinds of build knowledge about both Tup’s and Mozilla’s build systems as
systems [13]. In Mozilla at the moment, there is a recursive well as for his careful review of a previous version of this
memo/dumb configuration and preprocessing phase. This one article. Many thanks too to Marie Champenois for producing
is followed by a minimalist build system in the common and a better looking Figure 1.
strict sense. While not perfect, combining different kinds of
build systems provides the ability to work now, and to be R EFERENCES
perfectible later. This is the approach used by Mozilla in their [1] Gregory Szorc. Moving Away from Makefile’s. E-mail.
build system, and makes for a simplified migration, as it allows Aug. 22, 2012. URL: https://groups.google.com/forum/
maintaining two build systems up-to-date side-by-side. #!topic/mozilla.dev.platform/SACOnl-avMs/discussion.
Gligoric et Al. worked on migrating from one build system [2] Gregory Szorc. Moz.Build Files and the Firefox Build
to another [19]. They extracted build traces from the old one System. Feb. 2013. URL: https://gregoryszorc.com/blog/
and populated the new build system by factoring out common 2013 / 02 / 28 / moz . build - files - and - the - firefox - build -
patterns. Reusing this approach would not take advantage of system/.
the existing moz.build architecture used at Mozilla. There is [3] The Clang Team. JSON Compilation Database For-
no need to extract traces when the graph is available. mat Specification. URL: https : / / clang . llvm . org /
docs / JSONCompilationDatabase . html (visited on
09/21/2018).
VIII. R EPRODUCIBILITY
[4] Joshua Cranmer. Mach Command to Output Clang
This work was performed with the intent to make it fully Compilation Database. URL: https://bugzilla.mozilla.
reproducible. The development environment was managed org/show bug.cgi?id=904572 (visited on 09/21/2018).
with the Nix package manager in order to allow a completely [5] Ali Almossawi. How Maintainable Is the Firefox Code-
reproducible test environment [20]. All the work is available at base? May 15, 2013. URL: http : / / almossawi . com /
https://github.com/layus/gecko-tup. The results presented here firefox/prose (visited on 10/05/2018).
are all reproducible from the tools provided there. [6] The Mozilla Firefox Open Source Project on Open Hub:
Languages Page. URL: https : / / www. openhub. net / p /
This also means that our tests were performed within Nix2 ,
firefox/analyses/latest/languages summary (visited on
which imposes some strong constraints on the usual Linux
10/05/2018).
builds. Some patches were introduced to circumvent these
[7] Stuart I. Feldman. “Make — a Program for Maintaining
constraints. This does not invalidate the final results because
Computer Programs”. In: Software: Practice and Expe-
Firefox was compiled just fine. The issues were mostly with
rience 9.4 (Apr. 1, 1979), pp. 255–265. ISSN: 1097-
passing custom library paths to a build system that assumes
024X. DOI: 10.1002/spe.4380090402.
that all the system libraries comply to Filesystem Hierarchy
[8] Richard M. Stallman and Roland McGrath. GNU Make
Standard (FHS) [21], which is not the case with Nix.
- A Program for Directing Recompilation. 1991.
[9] Peter Miller. “Recursive Make Considered Harmful”.
IX. C ONCLUSION In: AUUGN. Vol. 19. AUUGN Journal of AUUG Inc 1.
AUUG, Inc., Feb. 1998, pp. 14–25.
We have conducted an experiment consisting in building [10] Guillaume Maudoux and Kim Mens. “Correct, Efficient,
Firefox with Tup. It has shown that Tup is a valid replacement and Tailored: The Future of Build Systems”. In: IEEE
for Make on this large real-life software project. While Tup Software 35.2 (2018), pp. 32–37.
does not generally bring a significant speedup at building, [11] Mike Shal. Tup Build System. URL: http://gittup.org/
its strength resides in correctness and enforcement of build tup/index.html (visited on 05/15/2018).
specifications. We argued that it may be a good investment [12] Mike Shal. Build System Rules and Algorithms. 2009.
in the long term, despite the significant refactoring that its URL : http : / / gittup . org / tup / build system rules and
introduction requires. Tup is however more efficient than Make algorithms.pdf.
at compiling small changesets. It could spare compilation time [13] Andrey Mokhov, Neil Mitchell, and Simon Peyton
if such rebuilds are frequent, as could be expected during Jones. “Build Systems à La Carte”. In: ICFP (2018).
development on a developer’s machine. Beyond its usage as a URL : https : / / www . microsoft . com / en - us / research /
build system, Tup also provides a straightforward data model uploads/prod/2018/03/build-systems.pdf.
that allows in-depth inspection of the build. In the future, we [14] Ehsan Akhgari. Unified Builds. Nov. 2013. URL: https:
intend to reuse the knowledge and tooling presented here with / / lists . mozilla . org / pipermail / dev - platform / 2013 -
Tup to compare more build systems, and look for relevant November/001998.html.
metrics to classify them. [15] Add –Build-Plan for ’cargo Build’ by Mshal · Pull
Request #5301 · Rust-Lang/Cargo. URL: https://github.
2 https://nixos.org/nix/ com/rust-lang/cargo/pull/5301 (visited on 10/18/2018).
[16] Support Producing a ”Build Plan” without Executing
Anything · Issue #3815 · Rust-Lang/Cargo. URL: https:
//github.com/rust- lang/cargo/issues/3815 (visited on
10/18/2018).
[17] Mike Shal. Linking Libxul with Tup and FUSE. Apr.
2013. URL: http://gittup.org/blog/2013/04/3- linking-
libxul-with-tup-and-fuse/.
[18] Bram Adams et al. “Makao”. In: Software Maintenance,
2007. ICSM 2007. IEEE International Conference On.
IEEE, 2007, pp. 517–518.
[19] Milos Gligoric et al. “Automated Migration of Build
Scripts Using Dynamic Analysis and Search-Based
Refactoring”. In: ACM SIGPLAN Notices 49 (Dec.
2014), pp. 599–616.
[20] Rok Garbas. Reproducible Development Environments.
Sept. 2015. URL: https://garbas.si/2015/reproducible-
development-environments.html.
[21] LSB Workgroup, The Linux Foundation. Filesystem
Hierarchy Standard, Version 3.0. Mar. 19, 2015. URL:
http://refspecs.linuxfoundation.org/FHS 3.0/fhs/index.
html.