Chapter 8. Creating binary packages for everything in pkgsrc (bulk builds)
Prev | Part I. The pkgsrc user's guide | Next

## Chapter 8. Creating binary packages for everything in pkgsrc (bulk builds)

For a number of reasons, you may want to build binary packages for a large selected set of packages in pkgsrc, or even for all pkgsrc packages. For instance, when you have multiple machines that should run the same software, it is wasted time if they all build their packages themselves from source. Or you may want to build a list of packages you want and check them before deploying onto production systems. There is a way of getting a set of binary packages: the bulk build system, or pbulk ("p" stands for "parallel"). This chapter describes how to set it up.

## 8.1. Preparations

First of all, you have to decide whether you build all packages or a limited set of them. Full bulk builds usually consume a lot more resources, both space and time, than builds for some practical sets of packages. A number of particularly heavy packages exist that are not actually interesting to a wide audience. (The approximate resource consumption for a full bulk build is given in section Section 8.3, “Requirements of a full bulk build”.) For limited bulk builds you need to make a list of packages you want to build. Note that all their dependencies will be built, so you don't need to track them manually.

During bulk builds various packages are installed and deinstalled in `/usr/pkg` (or whatever `LOCALBASE` is), so make sure that you don't need any package during the builds. Essentially, you should provide a fresh system, either a chroot environment or something even more restrictive, depending on what the operating system provides, or dedicate the whole physical machine. As a useful side effect this makes sure that bulk builds cannot break anything in your system. There have been numerous cases where certain packages tried to install files outside the `LOCALBASE` or wanted to edit some files in `/etc`.

## 8.2. Running a bulk build

Running a bulk build works roughly as follows:

- First, build the pbulk infrastructure in a fresh pkgsrc location.

- Then, build each of the packages from a clean installation directory using the infrastructure.

### 8.2.1. Configuration

To simplify configuration, we provide the helper script `mk/pbulk/pbulk.sh`.

In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). Configure network access to fetch distribution files. Create a user with name "pbulk".

Fetch and extract pkgsrc. Use a command like one of these:

    # (cd /usr && ftp -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
    # (cd /usr && fetch -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)
    # (cd /usr && cvs -Q -z3 -d anoncvs@anoncvs.NetBSD.org:/cvsroot get -P pkgsrc)

Or any other way that fits (e.g., curl, wget).

Deploy and configure pbulk tools, e.g.:

    # sh pbulk.sh -n # use native make, no bootstrap kit needed (for use on NetBSD)
    # sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment
    # sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build

### Note

`mk.conf.frag` is a fragment of `mk.conf` that contains settings you want to apply to packages you build. For instance,

    PKG_DEVELOPER=          yes     # perform more checks
    X11_TYPE=               modular # use pkgsrc X11
    SKIP_LICENSE_CHECK=     yes     # accept all licences (useful
                                    # when building all packages)

If configured for limited list, replace the list in `/usr/pbulk/etc/pbulk.list` with your list of packages, one per line without empty lines or comments. E.g.:

    www/firefox
    mail/thunderbird
    misc/libreoffice4

At this point you can also review configuration in `/usr/pbulk/etc` and make final amendments, if wanted.

Start it:

    # /usr/pbulk/bin/bulkbuild

After it finishes, you'll have `/mnt` filled with distribution files, binary packages, and reports, plain text summary in `/mnt/bulklog/meta/report.txt`

### Note

The `pbulk.sh` script does not cover all possible use cases. While being ready to run, it serves as a good starting point to understand and build more complex setups. The script is kept small enough for better understanding.

### Note

The `pbulk.sh` script supports running unprivileged bulk build and helps configuring distributed bulk builds. Distributed bulk builds support either building in worker chroots (each node is a path to a different chroot) that replicate the target system, including the pbulk prefix, or remote machines (each node is an IP address that must be accessible over SSH without a password).

## 8.3. Requirements of a full bulk build

A complete bulk build requires lots of disk space. Some of the disk space can be read-only, some other must be writable. Some can be on remote filesystems (such as NFS) and some should be local. Some can be temporary filesystems, others must survive a sudden reboot.

- 70 GB for the distfiles (read-write, remote, temporary)

- 60 GB for the binary packages (read-write, remote, permanent)

- 1 GB for the pkgsrc tree (read-only, remote, permanent)

- 5 GB for `LOCALBASE` (read-write, local, temporary)

- 10 GB for the log files (read-write, remote, permanent)

- 5 GB for temporary files (read-write, local, temporary)

## 8.4. Bulk build variants

To ensure that pkgsrc packages work in different configurations, it makes sense to run non-default bulk builds from time to time. This section lists some ideas for bulk builds that intentionally let packages fail if they don't follow the pkgsrc style.

### 8.4.1. Detect unknown configure options

Add the following line to `mk.conf`.

    GNU_CONFIGURE_STRICT=   yes

When a package fails this additional check, the most common cause is that the configure option was valid for an older version of the package but does not apply anymore. In that case, just remove it.

### 8.4.2. Detect classes of bugs by forcing compiler warnings

The job of a compiler is not restricted to producing executable code, most compilers also detect typical programming mistakes. The pkgsrc compiler wrappers make it easy to force compiler options when the package is built. This can be used to find typical bugs across all packages that are in pkgsrc. By reporting these bugs upstream, the packages will be more reliable with the next updates.

Add some of the following lines to `mk.conf`:

    CFLAGS+=        -Werror=char-subscripts
    CFLAGS+=        -Werror=implicit-function-declaration

When a package fails to build using these stricter compiler options, document the circumstances in which the compiler produced the error message. This includes:

- The platform (`MACHINE_PLATFORM`)

- The source file

- An excerpt of the code. GCC and Clang already do this as part of the diagnostic.

- The exact error message from the compiler.

If a package produces these error messages, but the package is fine, record this in your local `mk.conf`, like this, to skip this check in the next builds:

    .if ${PKGPATH} == category/package
    # Version ${VERSION} failed on ${MACHINE_PLATFORM}:
    # error message
    # code
    # Reason why the code does not need to be fixed.
    BUILDLINK_TRANSFORM+=   rm:-Werror=char-subscripts
    .endif

If the error messages from the compiler are valid and the code needs to be fixed, prepare a local patch (see `LOCALPATCHES`) and report the bug to the upstream authors of the package, providing them with the information you collected above.

Patches that are not essential for the package to work should only be reported upstream but not committed to pkgsrc, to make future updates easier.

### 8.4.3. Force compiler options only in the build phase

When adding custom compiler flags via `CFLAGS`, these apply to all phases of the package build process. Especially in the configure phase, adding `-Werror` leads to wrong decisions. The GNU configure scripts feed many small test programs to the C compiler to see whether certain headers are available, functions are defined in a library and programs can be run. In many cases these programs would not survive a strict compiler run with `-Wall -Wextra -Werror`.

The pkgsrc infrastructure is flexible enough to support compiler options being added between the `configure` and `build` phases. It's a little more complicated than the other examples in this section but still easy enough.

The basic idea is to use the pkgsrc compiler wrapper to inject the desired compiler options. The compiler wrapper's original task is to hide unwanted directories of include files and to normalize compiler options. It does this by wrapping the compiler command and rewriting the command line. To see this in action, run bmake patch in a package directory and examine the `work/.cwrappers/config` directory. It contains individual configurations for the C compiler and the related tools. The plan is to find a hook between the configure and build phases, and to modify these configuration files at that point.

To find this hook, have a look at `mk/build/build.mk`, which contains among others the `pre-build-checks-hook`. The word `checks` doesn't quite fit, but the `pre-build-hook` sounds good enough.

The code to be included in `mk.conf` is:

    # Just a few example options.
    BUILD_ONLY_CFLAGS=      -Wall -Werror -O2 -DTEMPDIR='"/tmp"'

    .if ${BUILD_ONLY_CFLAGS:U:M*}
    pre-build-checks-hook: add-build-only-cflags

    add-build-only-cflags: .PHONY
            ${RUN} cd ${CWRAPPERS_CONFIG_DIR};      \
            ${TEST} ! -f ${.TARGET} || exit 0;      \
            for flag in ${BUILD_ONLY_CFLAGS}; do    \
                    ${ECHO} "append=$$flag" >> cc;  \
            done;                                   \
            > ${.TARGET}
    .endif

(When editing the `mk.conf`, make sure that the commands of the `add-build-only-cflags` target are indented with a tab, not with spaces.)

The condition in the `.if` statement contains the `:U` modifier to prevent parse errors if the variable should be undefined, possibly because it is only defined for a subset of the packages. The `:M*` modifier ensures that there is at least one compiler option, to prevent a syntax error in the shell parser.

The code around the `${.TARGET}` variable ensures that the additional compiler options are only appended once, even if bmake build is run multiple times. To do this, it creates a marker file.

To verify that this setup works, run bmake configure in a package directory. Up to now, everything works as usual. Examine the directory `work/.cwrappers/config` to see that the compiler options from `BUILD_ONLY_CFLAGS` are not yet added to the file `cc`. Examine the tail of the `work/.work.log` file to see that the normal compiler options are used.

Now run bmake build. This will append the options to the file `cc` and will create the marker file in the same directory. After that, the build starts as usual, but with the added compiler options. Examine the tail of the file `work/.work.log` to see that the lines starting with `[*]` don't contain the compiler options, but the corresponding lines starting with `<.>` do end with these options.

Building packages using this setup variant and fixing the resulting bugs is the same as in Section 8.4.2, “Detect classes of bugs by forcing compiler warnings”.

### 8.4.4. Use custom directories

Some directories like `PREFIX`, `VARBASE`, `PKG_SYSCONFDIR`, `PKGMANDIR`, `PKG_INFODIR` can be configured in pkgsrc. Set these to arbitrary paths during bootstrap or afterwards in `mk.conf`.

    PREFIX=         /a-random-uuid
    PKG_SYSCONFDIR= /a-random-uuid
    VARBASE=        /a-random-uuid
    PKGMANDIR=	a-random-uuid
    PKG_INFODIR=	a-random-uuid

### 8.4.5. Turn warnings into errors

When building a package, warnings are typically ignored since they just flow by and do not cause the build to fail immediately. To find these warnings, redefine them to errors in `mk.conf`.

    DELAYED_WARNING_MSG=    ${DELAYED_ERROR_MSG} "(was warning)"
    WARNING_MSG=            ${FAIL_MSG} "(was warning)"

(There are many more classes of warnings in pkgsrc, and most of them can be redefined with a simple definition like above.

If a package suggests to add `USE_TOOLS+=perl` to the package Makefile, research whether the package actually needs Perl. If it does, add `USE_TOOLS+=perl` to the package Makefile, and if it doesn't, add `TOOLS_BROKEN+=perl`.

### 8.4.6. Reject packages for which pkglint reports errors

Using pkglint as part of the regular build process is mostly a waste of time. If you want to fix some of the warnings, just run pkglint recursively on the whole pkgsrc tree. This will take a few minutes (up to 10), which is much faster than a complete bulk build.

### 8.4.7. Reject packages that contain forbidden strings

To ensure that the binary packages don't contain references to the build directory, there is already `CHECK_WRKREF`. If that variable includes the item `extra`, it is possible to define additional patterns that must not appear in any installed file. This is specified in `mk.conf`.

    CHECK_WRKREF=                   extra
    CHECK_WRKREF_EXTRA_DIRS+=       /usr/local
    CHECK_WRKREF_EXTRA_DIRS+=       /usr/pkg
    CHECK_WRKREF_EXTRA_DIRS+=	@[A-Z][A-Z]*@

The above patterns will probably generate many false positives, therefore the results need to be taken with a grain of salt.

### 8.4.8. Reject packages whose self-test fails

To run the test suites that come with each package, add this line to `mk.conf`.

    PKGSRC_RUN_TEST=        yes

Be prepared that even the most basic packages fail this test. When doing a bulk build with this, it will often abort in the early phase where the packages are scanned for their dependencies since there are cyclic dependencies. There is still a lot to do in this area.

### 8.4.9. Reject packages that use undefined shell variables

To catch typos in the shell snippets from the Makefile fragments, add the `-u` flag to most of the commands by adding this line to `mk.conf`.

    RUN=    @set -eu;

After that, ensure that none of the bulk build log files (even those for successfully built packages) contains the string `parameter not set` or whatever error message the command sh -ceu '$undefined' outputs.

See `mk/misc/common.mk` for the existing definition.

### 8.4.10. Turn off verbose logging

The build logs of a package are often quite long. This allows error messages or other interesting details to hide between the noise. To make the actual error message stand out more, add these lines to `mk.conf`.

    GNU_CONFIGURE_QUIET=    yes
    MAKE_FLAGS+=            -s

The `-s` option works for both GNU Make and BSD Make. On exotic platforms with their own make, it may be a little different.

## 8.5. Creating a multiple CD-ROM packages collection

After your pkgsrc bulk-build has completed, you may wish to create a CD-ROM set of the resulting binary packages to assist in installing packages on other machines. The `pkgtools/cdpack` package provides a simple tool for creating the ISO 9660 images. cdpack arranges the packages on the CD-ROMs in a way that keeps all the dependencies for a given package on the same CD as that package.

### 8.5.1. Example of cdpack

Complete documentation for cdpack is found in the cdpack(1) man page. The following short example assumes that the binary packages are left in `/usr/pkgsrc/packages/All` and that sufficient disk space exists in `/u2` to hold the ISO 9660 images.

    # mkdir /u2/images
    # pkg_add /usr/pkgsrc/packages/All/cdpack
    # cdpack /usr/pkgsrc/packages/All /u2/images

If you wish to include a common set of files (`COPYRIGHT`, `README`, etc.) on each CD in the collection, then you need to create a directory which contains these files, e.g.:

    # mkdir /tmp/common
    # echo "This is a README" > /tmp/common/README
    # echo "Another file" > /tmp/common/COPYING
    # mkdir /tmp/common/bin
    # echo "#!/bin/sh" > /tmp/common/bin/myscript
    # echo "echo Hello world" >> /tmp/common/bin/myscript
    # chmod 755 /tmp/common/bin/myscript

Now create the images:

    # cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images

Each image will contain `README`, `COPYING`, and `bin/myscript` in their root directories.

Prev | Up | Next
Chapter 7. Creating binary packages | Home | Chapter 9. Directory layout of the installed files