build/documentation
Author: Boris Kolpackov
Keywords: build,system,make

# Introduction

I hear you saying: "Yeah, yeah, another marvelous make replacement with built-in support for various compilers, automatic dependency scanning, parallel and distributed builds, and XML instead of makefiles!" On the contrary, build is a software build system that is implemented on top of GNU make. It defines extensible framework for translators, delegates most of its tasks to existing tools, and uses GNU make syntax for makefiles. Build was designed with the following tasks in mind:

- configuration
- building
- testing
- installation

As distribution of software in source code is being replaced by precompiled package distributions, greater emphasis was placed on development-time conveniences. Some of the features of build include:

- position-independent makefiles
- non-recursive multi-makefile include-based structure
- leaf makefiles are full-fledged GNU makefiles, not just variable definitions
- complete dependency graph
- inter-project dependency tracking

To understand leaf makefiles in this manual you will need basic knowledge of how `make` works. However, inside, build uses some of the more advanced features of GNU make including

- order-only prerequisites
- pattern and target-specific variables
- `$(call )` and `$(eval )`
- different flavors of `include` directive

If you find usage of some of the `make` techniques obscure in this text you can always consult with the GNU make manual.

# Fundamentals

Build defines a framework in terms of which user makefiles are written. Below is the list of fundamental concepts.

Project
  is a separately distributable collection of source files. A more intuitive way of defining a project would be to say it's what you normally assign a version number to (e.g., `libhello-0.0.1`) and package as a tar ball (e.g., `libhello-0.0.1.tar.gz`). A reasonably complex project usually has a hierarchy of sub-directories under its root directory.
Component
  is part of a project for which you would normally write a makefile. For example our `libhello` could consist of two components: the library itself and a test driver.
Source root
  is a root directory of a project. For example, if we unpack our `libhello-0.0.1.tar.gz` to `/tmp` and end up with a directory `/tmp/libhello-0.0.1` then that would be this project's source root.
Source base
  is a directory of a component in a project. Continuing our example, if our `libhello` project had a subdirectory `test` then the source base of the `test` component would be `/tmp/libhello-0.0.1/test`. By definition, any source base is a sub-directory of a source root for any particular project.
Out root
  is a root directory of a project's build (or output) hierarchy. This directory signifies where generated files for the project will be placed. For example, if we decide to build `libhello` in `/tmp/libhello-i686-pc-linux-gnu` then that would be one of the possible out roots.
Out base
  is a build (or output) directory of a component in a project. If we continue our example, the out base for the `test` component would be `/tmp/libhello-i686-pc-linux-gnu/test`.

The build runtime defines four `make` variables that correspond to the last four definitions:

    src_root
    src_base
    out_root
    out_base

Below is an example of how you could use some of them.

    driver := $(out_base)/driver

    $(driver): $(out_base)/driver.o $(out_root)/libhello/libhello.so

Additionally, the build framework defines three more variables:

    bld_root
    scf_root := $(src_root)/build
    dcf_root := $(out_root)/build

`bld_root` is a root directory of the build runtime. Normally you would use it to include one of the build's files.

`scf_root` stands for static configuration root and points to a directory where project's static configuration is kept. `dcf_root` stands for dynamic configuration root and points to a directory where project's dynamic configuration is kept. We will talk more about those two variables later.

Also note that having `src_root` the same as `out_root` is perfectly valid and indicates that we are building in a source directory.

# First Example

Now we are ready to examine our first simple example. We will skip one-file C-based "hello world", however, because it is not very practical and there is not much room for improvement. Rather we will start from a `libhello` library, a test driver for it and a `hello` program that uses the library. To make our example even more realistic the `libhello` library and the `hello` program are two separate projects. And, did I mention, they are written in C++. See `examples/cxx/hello`.

## Invocation

Let's first get some user experience without looking into makefiles. Suppose we unpacked build source into `/tmp/build-0.1.12`. First let's try to build `libhello`:

    $ cd /tmp/build-0.1.12/examples/cxx/hello/libhello
    $ make

    configuring 'libhello'

    Please select the C++ compiler you would like to use:

    (1) GNU C++ (g++)
    (2) Intel C++ (icc)

    [1]: 1

    Would you like the C++ compiler to optimize generated code?

    [y]: y

    Would you like the C++ compiler to generate debug information?

    [y]: y

    configuring 'libhello'

    Please select the default library type:

    (1) archive
    (2) shared object

    [2]: 2

    configuring 'libhello'

    Please enter the g++ binary you would like to use, for example 'g++-3.4',
    '/usr/local/bin/g++' or 'distcc g++'. You can use path auto-completion.

    [g++]: g++

    Please select the optimization level you would like to use:

    (1) -O1 [Tries to reduce code size and execution time, without
             performing any optimizations that take a great deal of
             compilation time.]
    (2) -O2 [Performs nearly all supported optimizations that do not
             involve a space-speed tradeoff.]
    (3) -O3 [Optimize even more.]
    (4) -Os [Optimize for size.]

    [2]: 2
    c++ /tmp/build-0.1.12/examples/cxx/hello/libhello/libhello/hello.cxx
    ld  /tmp/build-0.1.12/examples/cxx/hello/libhello/libhello/hello.l
    c++ /tmp/build-0.1.12/examples/cxx/hello/libhello/test/driver.cxx
    ld  /tmp/build-0.1.12/examples/cxx/hello/libhello/test/driver

Here we were presented with a bunch of configuration questions. If you have `g++` in your default path you can just keep hitting enter to select defaults. If you would like to see real commands that are being executed (useful when something goes wrong) you can say `make verbose=1`.

From the messages above we can deduce that we have built the `libhello` library (`hello.l`) and the test driver for it (`test/driver`). So far so good. Now let's try to build the `hello` program:

    $ cd /tmp/build-0.1.12/examples/cxx/hello/hello
    $ make

Again you will be asked a bunch of questions you have already seen, except these two:

    Configuring external dependency on 'libhello' for 'hello driver'.

    Would you like to configure dependency on the installed
    version as opposed to the development build?

    [y]: n

    Please enter the out_root for 'libhello'.

    []: ../libhello

    Please enter the src_root for 'libhello'.

    []: ../libhello

The `hello` program depends on `libhello`. Since they are distributed as separate projects `hello` needs to know where to look for `libhello`. That's why user intervention is required. The first question is whether we would like to use an installed version of `libhello` as opposed to the not installed build. Since we haven't installed `libhello` the answer is `no`. The next two questions determine where the `out_root` of the build and the `src_root` of the source code are. Since we built `libhello` in its source directory both become `/tmp/build-0.1.12/examples/cxx/hello/libhello` or `../libhello` if we are in `/tmp/build-0.1.12/examples/cxx/hello/hello`. After answering those questions you will see something like this:

    c++ /tmp/build-0.1.12/examples/cxx/hello/hello/hello.cxx
    ld  /tmp/build-0.1.12/examples/cxx/hello/hello/hello

And that's it.

One of the nice features of build is that you can have several builds in separate directories all from the same source base. Suppose we would like to build an `-O3`-optimized statically-linked version of `hello` without debug information:

    $ mkdir /tmp/hello-i686-pc-linux-gnu
    $ cd /tmp/hello-i686-pc-linux-gnu
    $ make -f ../build-0.1.12/examples/cxx/hello/hello/makefile

    configuring 'hello driver'

    Please select the C++ compiler you would like to use:

    (1) GNU C++ (g++)
    (2) Intel C++ (icc)

    [1]: 1

    Would you like the C++ compiler to optimize generated code?

    [y]: y

    Would you like the C++ compiler to generate debug information?

    [y]: n

    Configuring external dependency on 'libhello' for 'hello driver'.

    Would you like to configure dependency on the installed
    version as opposed to the development build?

    [y]: n

    Please enter the out_root for 'libhello'.

    []: /tmp/libhello-i686-pc-linux-gnu

    Please enter the src_root for 'libhello'.

    []: ../build-0.1.12/examples/cxx/hello/libhello

    configuring 'hello driver'

    Please enter the g++ binary you would like to use, for example 'g++-3.4',
    '/usr/local/bin/g++' or 'distcc g++'. You can use path auto-completion.

    [g++]: g++

    Please select the optimization level you would like to use:

    (1) -O1 [Tries to reduce code size and execution time, without
             performing any optimizations that take a great deal of
             compilation time.]
    (2) -O2 [Performs nearly all supported optimizations that do not
             involve a space-speed tradeoff.]
    (3) -O3 [Optimize even more.]
    (4) -Os [Optimize for size.]

    [2]: 3

    configuring 'libhello'

    Please select the C++ compiler you would like to use:

    (1) GNU C++ (g++)
    (2) Intel C++ (icc)

    [1]: 1

    Would you like the C++ compiler to optimize generated code?

    [y]: y

    Would you like the C++ compiler to generate debug information?

    [y]: n

    configuring 'libhello'

    Please select the default library type:

    (1) archive
    (2) shared object

    [2]: 1

    configuring 'libhello'

    Please enter the g++ binary you would like to use, for example 'g++-3.4',
    '/usr/local/bin/g++' or 'distcc g++'. You can use path auto-completion.

    [g++]: g++

    Please select the optimization level you would like to use:

    (1) -O1 [Tries to reduce code size and execution time, without
             performing any optimizations that take a great deal of
             compilation time.]
    (2) -O2 [Performs nearly all supported optimizations that do not
             involve a space-speed tradeoff.]
    (3) -O3 [Optimize even more.]
    (4) -Os [Optimize for size.]

    [2]: 3
    c++ /tmp/build-0.1.12/examples/cxx/hello/hello/hello.cxx
    c++ /tmp/build-0.1.12/examples/cxx/hello/libhello/libhello/hello.cxx
    ar  /tmp/libhello-i686-pc-linux-gnu/libhello/hello.l
    ld  /tmp/hello-i686-pc-linux-gnu/hello

## Makefiles

Now let's take a look at the makefiles. We will start from the makefile for `libhello` (`hello/libhello/libhello/makefile`). Note, that I removed support for `install` target for now.

    include $(dir $(lastword $(MAKEFILE_LIST)))../build/bootstrap.make

    cxx_tun   := hello.cxx
    cxx_obj   := $(addprefix $(out_base)/,$(cxx_tun:.cxx=.o))
    cxx_od    := $(cxx_obj:.o=.o.d)

    hello.l             := $(out_base)/hello.l
    hello.l.cpp-options := $(out_base)/hello.l.cpp-options

    clean     := $(out_base)/.clean

    # Build.
    #
    $(hello.l): $(cxx_obj)

    $(cxx_obj): $(hello.l.cpp-options)

    $(hello.l.cpp-options): value := -I$(src_root)

    $(call -include,$(cxx_od))

    # Clean.
    #
    .PHONY: $(clean)

    $(clean): $(hello.l).clean \
              $(addsuffix .clean,$(cxx_obj)) \
              $(hello.l.cpp-options).clean

    # Aliases.
    #
    ifdef %interactive%

    .PHONY: clean

    clean: $(clean)

    endif

    # How to.
    #
    $(call include,$(bld_root)/cxx/o-l.make)
    $(call include,$(bld_root)/cxx/cxx-o.make)

Let's examine this file line-by-line:

    include $(dir $(lastword $(MAKEFILE_LIST)))../build/bootstrap.make

Every leaf makefile (i.e., one written to build a component) starts from a line that looks like this. Its sole purpose is to bootstrap the build system. In our case `$(dir $(lastword ...))` will expand to something like

    include .../hello/libhello/libhello/../build/bootstrap.make

Let's take a look at `libhello/build/bootstrap.make`:

    project_name := libhello
    include $(dir $(lastword $(MAKEFILE_LIST))).../build/bootstrap.make

The first line just initializes `project_name` with the name of the project. The second line includes the real `bootstrap.make`. See Bootstrapping section for more information on various ways of bootstrapping the build runtime.

What does `bootstrap.make` do? Besides other things (which are explained as we encounter them) it sets all those `*_root` and `*_base` variables.

Let's go back to our `libhello/makefile`. The first line should be clear by now so let's move on:

    cxx_tun   := hello.cxx
    cxx_obj   := $(addprefix $(out_base)/,$(cxx_tun:.cxx=.o))
    cxx_od    := $(cxx_obj:.o=.o.d)

This should be pretty straightforward: `cxx_tun` contains a list of translation units, `cxx_obj` contains a list of object files that will be produced from those translation units, and `cxx_od` contains a list of automatically generated dependency files for those translation units. After expansion these variables could contain the following values (note, in this example `out_root` is the same as `src_root`):

    cxx_tun   := hello.cxx
    cxx_obj   := .../hello/libhello/libhello/hello.o
    cxx_od    := .../hello/libhello/libhello/hello.o.d

Next chunk:

    hello.l             := $(out_base)/hello.l
    hello.l.cpp-options := $(out_base)/hello.l.cpp-options

    clean     := $(out_base)/.clean

`hello.l` and `clean` are two variables that store names of targets. You are probably wondering what the heck is `hello.l`? This is a library abstraction the build system provides to deal (besides other things) with archives/shared objects uniformly. A user who builds your project can specify what type of library they want without any additional effort from you. I encourage you to take a look inside `hello.l`.

`hello.l.cpp-options` is a bit trickier. `$(out_base)/hello.l.cpp-options` keeps C preprocessor options that are required to compile library files as well as any piece of code that uses it. Hopefully, it will become clear once you see the rest of the makefile.

Let's move on:

    # Build.
    #
    $(hello.l): $(cxx_obj)

    $(cxx_obj): $(hello.l.cpp-options)

    $(hello.l.cpp-options): value := -I$(src_root)

    $(call -include,$(cxx_od))

The first line tells us that `hello.l` is built from object files listed in `cxx_obj`. The second line establishes dependency of object files on C preprocessor options - in order to build an object file we will need C++ source file and C preprocessor options - quite logical. The third line sets target-specific variable `value` for `hello.l.cpp-options`: that's how `hello.l.cpp-options` gets its content. You may be wondering why do we need to add this `-I$(src_root)`. The first line of `hello.cxx` should clear things up:

    #include "libhello/hello.hxx"

And finally the fourth line includes auto-generated dependency information for each object file. You can think of `$(call -include )` as being equivalent to `-include` directive for now.

Next chunk:

    .PHONY: $(clean)

    $(clean): $(hello.l).clean \
              $(addsuffix .clean,$(cxx_obj)) \
              $(hello.l.cpp-options).clean

Let's concentrate on the last line. There we are essentially saying that cleaning `libhello` consists of cleaning the library, object files and C preprocessor options. How does this work? It is done using pattern rules. Part of the build system that defines how to build say `%.l` also defines how to clean after it: `%.l.clean`. You may want to look into `build/cxx/gnu/o-l.make` for details.

Moving on:

    # Aliases.
    #
    ifdef %interactive%

    .PHONY: clean

    clean: $(clean)

    endif

In this part we are creating a short alias (`clean`) for its long equivalent (`$(out_base)/.clean`). `%interactive%` is a system variable defined by the framework. It is initialized only if current makefile is used in interactive mode as opposed to being included.

And finally:

    # How to.
    #
    $(call include,$(bld_root)/cxx/o-l.make)
    $(call include,$(bld_root)/cxx/cxx-o.make)

In this makefile fragment we are including parts of the build system that actually know how to build. As you might have noticed, all the code that we have examined up until now was dealing with what to build. `o-l.make` defines a pattern rule to build libraries from object files. `cxx-o.make` defines a pattern rule to build object files from C++ translation units.

And this concludes our examination of the makefile for `libhello`. Next is the test driver( `hello/libhello/test/makefile`):

    include $(dir $(lastword $(MAKEFILE_LIST)))../build/bootstrap.make

    cxx_tun := driver.cxx
    cxx_obj := $(addprefix $(out_base)/,$(cxx_tun:.cxx=.o))
    cxx_od  := $(cxx_obj:.o=.o.d)

    hello.l             := $(out_root)/libhello/hello.l
    hello.l.cpp-options := $(out_root)/libhello/hello.l.cpp-options

    driver  := $(out_base)/driver.e
    clean   := $(out_base)/.clean
    test    := $(out_base)/.test

    # Build.
    #
    $(driver): $(cxx_obj) $(hello.l)

    $(cxx_obj): $(hello.l.cpp-options)

    $(call -include,$(cxx_od))

    # Test.
    #
    .PHONY: $(test)

    $(test): $(driver)
    	$<

    # Clean.
    #
    .PHONY: $(clean)

    $(clean): $(driver).clean $(addsuffix .clean,$(cxx_obj))

    # Aliases.
    #
    ifdef %interactive%

    .PHONY: clean test

    test: $(test)
    clean: $(clean)

    endif

    # How to.
    #
    $(call include,$(bld_root)/cxx/o-e.make)
    $(call include,$(bld_root)/cxx/cxx-o.make)

    # Load build information.
    #
    $(call load,$(src_root)/libhello/makefile)

Do you see anything unknown or strange in this makefile? I hope you don't. Just to re-iterate, I will remove everything we definitely saw and leave only what's new:

    hello.l             := $(out_root)/libhello/hello.l
    hello.l.cpp-options := $(out_root)/libhello/hello.l.cpp-options

    # Build.
    #
    $(driver): $(cxx_obj) $(hello.l)

    $(cxx_obj): $(hello.l.cpp-options)

    # Load build information.
    #
    $(call load,$(src_root)/libhello/makefile)

All of what's left deals in one way or the other with linking to `libhello`. The first two lines define variables that hold paths to the library and C preprocessor options (since we are in the same project we know what those paths are relative to `out_root`/`src_root`). The third line declares that driver consists of object files and should be linked with `libhello`. The next line says that we need C preprocessor options to build object files, just like we did for `libhello`. And finally the last line: what does the `load` function do? The `load` function loads rules and dependency information from the makefile specified. Note that it doesn't make variable definitions from that makefile available (or, even worse, override ones we set) - only dependencies and rules. The major benefit of doing this is in having complete dependency information along with the rules in case something gets out of date. If, for example, we change something in `libhello` and then execute `make` in `test`, `make` will be able to detect that `libhello` is out of date and will re-build it before building the test driver.

Now let's take a look at the makefile from the `hello` program (`hello/hello/makefile`). Remember that it is a separate project (I removed support for `install` target again):

    include $(dir $(lastword $(MAKEFILE_LIST)))build/bootstrap.make

    cxx_tun   := hello.cxx
    cxx_obj   := $(addprefix $(out_base)/,$(cxx_tun:.cxx=.o))
    cxx_od    := $(cxx_obj:.o=.o.d)

    hello.e   := $(out_base)/hello.e
    clean     := $(out_base)/.clean

    # Secure default target.
    #
    $(hello.e):

    # Import information about libhello.
    #
    $(call import,libhello,l: hello.l,cpp-options: hello.l.cpp-options)

    # Build.
    #
    $(hello.e): $(cxx_obj) $(hello.l)

    $(cxx_obj): $(hello.l.cpp-options)

    $(call -include,$(cxx_od))

    # Clean.
    #
    .PHONY: $(clean)

    $(clean): $(hello.e).clean $(addsuffix .clean,$(cxx_obj))

    # Aliases.
    #
    ifdef %interactive%

    .PHONY: clean

    clean: $(clean)

    endif

    # How to.
    #
    $(call include,$(bld_root)/cxx/o-e.make)
    $(call include,$(bld_root)/cxx/cxx-o.make)

Again I will remove all the parts that we are familiar with leaving only what's new:

    # Secure default target.
    #
    $(hello.e):

    # Import information about libhello.
    #
    $(call import,libhello,l: hello.l,cpp-options: hello.l.cpp-options)

Let's see what's going on here. I will start from the call to the `import` function. It is similar to the `load` function with a few exceptions. First of all, since we are importing build information from a separate project we don't know where to look for its parts; this will require dynamic configuration. Secondly, we have no way to figure out where `hello.l` and `hello.l.cpp-options` are. Thus we have ``l: hello.l'` and ``cpp-options: hello.l.cpp-options'` which essentially means "initialize variable `hello.l` with the library path and variable `hello.l.cpp-options` with the C preprocessor options file path".

One side effect of the call to `import` function is the potential loss of the default target. Thus, the first line.

The inter-project dependency importing architecture is probably the most complicated part of the build system. It will be described in a separate section.

This concludes our step-by-step examination. It was not trivial but neither is building real software.

# Inter-Project Dependencies

To be completed.

# Administrivia

## License

The build runtime (makefiles, scripts, etc.) is distributed under the terms of the GNU General Public License, version 2. Build documentation is distributed under the terms of the GNU Free Documentation License, version 1.2.

In particular, this means that any makefile, script, etc., that includes, calls, or otherwise links to the build runtime must be covered by a GPL-compatible license, should you decide to distribute them.

## Bootstrapping

There are two ways to use the build system: your project can embed all necessary files or you can require the build runtime to be installed. The first approach allows you to make your project self-content, modulo other external dependencies it might have. The disadvantage of this approach manifests itself when your project participates in a build with inter-project dependencies established: your project and the rest of the build could use different (and potentially incompatible) versions of the runtime. There is no such problem with the second approach.

My advice is to go with the installed build unless you have a compelling reason to do otherwise.

When you go with the embedded runtime you can copy all necessary files to your project's `build/` directory. When you use external build your `build/bootstrap.make` might look something like this:

    project_name := libhello
    include build-0.1/bootstrap.make

How will `make` find `build-0.1/bootstrap.make`? There are two ways this can happen: you have the runtime installed (e.g., into `/usr/local/include`) where `make` will look for it by default or you can tell `make` where to look using `-I` option in the command line or by setting `MAKEFLAGS` environment variable.

## Versioning

Build uses a three-digit versioning scheme: `X.Y.Z`. `X` is a generation number; it is incremented when a completely new design is implemented. `Y` is an interface version; two build runtimes with different `Y` have incompatible interfaces even though the ideology is the same (if ideology changes `X` will most likely be incremented). Additionally, odd `Y` indicates that it's a development release (similar to the linux kernel). Finally, `Z` signifies a release number. Releases with the same `X.Y` have the same interface.

To allow co-existence of several versions of build on the same system, the `X.Y` pair is made part of the path (e.g., `/usr/include/build-0.1`) thus when you bootstrap the runtime (in a project-specific `bootstrap.make`) you specify interface version:

    include build-0.1/bootstrap.make

Copyright © 2004-2009 Boris Kolpackov.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, version 1.2; with no Invariant Sections, no Front-Cover Texts and no Back-Cover Texts.