Chapter 18. Buildlink methodology
Prev | Part II. The pkgsrc developer's guide | Next

## Chapter 18. Buildlink methodology

Buildlink is a framework in pkgsrc that controls what headers and libraries are seen by a package's configure and build processes. This is implemented in a two step process:

1. Symlink headers and libraries for dependencies into `BUILDLINK_DIR`, which by default is a subdirectory of `WRKDIR`.

2. Create wrapper scripts that are used in place of the normal compiler tools that translate `-I${LOCALBASE}/include` and `-L${LOCALBASE}/lib` into references to `BUILDLINK_DIR`. The wrapper scripts also make native compiler on some operating systems look like GCC, so that packages that expect GCC won't require modifications to build with those native compilers.

This normalizes the environment in which a package is built so that the package may be built consistently despite what other software may be installed. Please note that the normal system header and library paths, e.g. `/usr/include`, `/usr/lib`, etc., are always searched -- buildlink3 is designed to insulate the package build from non-system-supplied software.

## 18.1. Converting packages to use buildlink3

The process of converting packages to use the buildlink3 framework (“bl3ifying”) is fairly straightforward. The things to keep in mind are:

1. Ensure that the build always calls the wrapper scripts instead of the actual toolchain. Some packages are tricky, and the only way to know for sure is the check `${WRKDIR}/.work.log` to see if the wrappers are being invoked.

2. Don't override `PREFIX` from within the package Makefile, e.g. Java VMs, standalone shells, etc., because the code to symlink files into `${BUILDLINK_DIR}` looks for files relative to “pkg_info -qp `pkgname`”.

3. Remember that only the `buildlink3.mk` files that you list in a package's Makefile are added as dependencies for that package.

If a dependency on a particular package is required for its libraries and headers, then we replace:

    DEPENDS+=       foo>=1.1.0:../../category/foo

with

    .include "../../category/foo/buildlink3.mk"

The buildlink3.mk files usually define the required dependencies. If you need a newer version of the dependency when using buildlink3.mk files, then you can define it in your Makefile; for example:

    BUILDLINK_API_DEPENDS.foo+=   foo>=1.1.0
    .include "../../category/foo/buildlink3.mk"

There are several `buildlink3.mk` files in `pkgsrc/mk` that handle special package issues:

- `bdb.buildlink3.mk` chooses either the native or a pkgsrc Berkeley DB implementation based on the values of `BDB_ACCEPTED` and `BDB_DEFAULT`.

- `curses.buildlink3.mk`: If the system comes with neither Curses nor NCurses, this will take care to install the `devel/ncurses` package.

- `krb5.buildlink3.mk` uses the value of `KRB5_ACCEPTED` to choose between adding a dependency on Heimdal or MIT-krb5 for packages that require a Kerberos 5 implementation.

- `motif.buildlink3.mk` checks for a system-provided Motif installation or adds a dependency on `x11/motif`. The user can set `MOTIF_TYPE` to “dt” or “motif” to choose which Motif version will be used.

- `readline.buildlink3.mk` checks for a system-provided GNU readline or editline (libedit) installation, or adds a dependency on `devel/readline`, `devel/editline`. The user can set `READLINE_DEFAULT` to choose readline implementation. If your package really needs GNU readline library, its Makefile should include `devel/readline/buildlink3.mk` instead of `readline.buildlink3.mk`.

- `oss.buildlink3.mk` defines several variables that may be used by packages that use the Open Sound System (OSS) API.

- `pgsql.buildlink3.mk` will accept any of the Postgres versions in the variable `PGSQL_VERSIONS_ACCEPTED` and default to the version `PGSQL_VERSION_DEFAULT`. See the file for more information.

- `pthread.buildlink3.mk` uses the value of `PTHREAD_OPTS` and checks for native pthreads or adds a dependency on `devel/pth` as needed.

- `xaw.buildlink3.mk` uses the value of `XAW_TYPE` to choose a particular Athena widgets library.

The comments in those `buildlink3.mk` files provide a more complete description of how to use them properly.

## 18.2. Writing `buildlink3.mk` files

A package's `buildlink3.mk` file is included by Makefiles to indicate the need to compile and link against header files and libraries provided by the package. A `buildlink3.mk` file should always provide enough information to add the correct type of dependency relationship and include any other `buildlink3.mk` files that it needs to find headers and libraries that it needs in turn.

To generate an initial `buildlink3.mk` file for further editing, Rene Hexel's `pkgtools/createbuildlink` package is highly recommended. For most packages, the following command will generate a good starting point for `buildlink3.mk` files:

    % cd pkgsrc/category/pkgdir
    % createbuildlink >buildlink3.mk

### 18.2.1. Anatomy of a buildlink3.mk file

The following real-life example `buildlink3.mk` is taken from `pkgsrc/graphics/tiff`:

    BUILDLINK_TREE+=        tiff

    .if !defined(TIFF_BUILDLINK3_MK)
    TIFF_BUILDLINK3_MK:=

    BUILDLINK_API_DEPENDS.tiff+=    tiff>=3.6.1
    BUILDLINK_ABI_DEPENDS.tiff+=    tiff>=3.7.2nb1
    BUILDLINK_PKGSRCDIR.tiff?=      ../../graphics/tiff

    .include "../../devel/zlib/buildlink3.mk"
    .include "../../graphics/jpeg/buildlink3.mk"
    .endif # TIFF_BUILDLINK3_MK

    BUILDLINK_TREE+=        -tiff

The header and footer manipulate `BUILDLINK_TREE`, which is common across all `buildlink3.mk` files and is used to track the dependency tree.

The main section is protected from multiple inclusion and controls how the dependency on `pkg` is added. Several important variables are set in the section:

- `BUILDLINK_API_DEPENDS.`pkg`` is the dependency version recorded in the installed package; this should always be set using += to ensure that we're appending to any pre-existing list of values. This variable should be set to the last version of the package that had an backwards-incompatible API change.

- `BUILDLINK_PKGSRCDIR.`pkg`` is the location of the `pkg` pkgsrc directory.

- `BUILDLINK_DEPMETHOD.`pkg`` (not shown above) controls whether we use `BUILD_DEPENDS` or `DEPENDS` to add the dependency on `pkg`. The build dependency is selected by setting `BUILDLINK_DEPMETHOD.`pkg`` to “build”. By default, the full dependency is used.

- `BUILDLINK_INCDIRS.`pkg`` and `BUILDLINK_LIBDIRS.`pkg`` (not shown above) are lists of subdirectories of `${BUILDLINK_PREFIX.`pkg`}` to add to the header and library search paths. These default to “include” and “lib” respectively.

- `BUILDLINK_CPPFLAGS.`pkg`` (not shown above) is the list of preprocessor flags to add to `CPPFLAGS`, which are passed on to the configure and build phases. The “-I” option should be avoided and instead be handled using `BUILDLINK_INCDIRS.`pkg`` as above.

The following variables are all optionally defined within this second section (protected against multiple inclusion) and control which package files are symlinked into `${BUILDLINK_DIR}` and how their names are transformed during the symlinking:

- `BUILDLINK_FILES.`pkg`` (not shown above) is a shell glob pattern relative to `${BUILDLINK_PREFIX.`pkg`}` to be symlinked into `${BUILDLINK_DIR}`, e.g. `include/*.h`.

- `BUILDLINK_FILES_CMD.`pkg`` (not shown above) is a shell pipeline that outputs to stdout a list of files relative to `${BUILDLINK_PREFIX.`pkg`}`. The resulting files are to be symlinked into `${BUILDLINK_DIR}`. By default, this takes the `+CONTENTS` of a `pkg` and filters it through `${BUILDLINK_CONTENTS_FILTER.`pkg`}`.

- `BUILDLINK_CONTENTS_FILTER.`pkg`` (not shown above) is a filter command that filters `+CONTENTS` input into a list of files relative to `${BUILDLINK_PREFIX.`pkg`}` on stdout. By default, `BUILDLINK_CONTENTS_FILTER.`pkg`` outputs the contents of the `include` and `lib` directories in the package `+CONTENTS`.

- `BUILDLINK_FNAME_TRANSFORM.`pkg`` (not shown above) is a list of sed arguments used to transform the name of the source filename into a destination filename, e.g. -e "s|/curses.h|/ncurses.h|g".

This section can additionally include any `buildlink3.mk` needed for `pkg`'s library dependencies. Including these `buildlink3.mk` files means that the headers and libraries for these dependencies are also symlinked into `${BUILDLINK_DIR}` whenever the `pkg` `buildlink3.mk` file is included. Dependencies are only added for directly include `buildlink3.mk` files.

When providing a `buildlink3.mk` and including other `buildlink3.mk` files in it, please only add necessary ones, i.e., those whose libraries or header files are automatically exposed when the package is use.

In particular, if only an executable (`bin/foo`) is linked against a library, that library does not need to be propagated in the `buildlink3.mk` file.

The following steps should help you decide if a `buildlink3.mk` file needs to be included:

- Look at the installed header files: What headers do they include? The packages providing these files must be buildlinked.

- Run `ldd` on all installed libraries and look against what other libraries they link. Some of the packages providing these probably need to be buildlinked; however, it's not automatic, since e.g. GTK on some systems pulls in the X libraries, so they will show up in the `ldd` output, while on others (like OS X) it won't. `ldd` output can thus only be used as a hint.

### 18.2.2. Updating `BUILDLINK_API_DEPENDS.`pkg`` and `BUILDLINK_ABI_DEPENDS.`pkg`` in `buildlink3.mk` files

Both variables set lower bounds for a version of this package. The two variables differ in that one describes source compatibility (API) and the other binary compatibility (ABI). The difference is that a change in the API breaks compilation of programs while changes in the ABI stop compiled programs from running.

The `BUILDLINK_API_DEPENDS.`pkg`` variable in a `buildlink3.mk` should be changed very rarely. (One possible scenario: If all packages using this package need a higher version than defined in the `buildlink3.mk`, `BUILDLINK_API_DEPENDS.`pkg`` could be updated to that higher version.)

On the other hand, changes to `BUILDLINK_ABI_DEPENDS.`pkg`` are more common. The variable will need to be updated every time the major version of one of its shared libraries is changed, or any other change where a binary built against the previous version of the package will not run against the new version any longer.

In such a case, the package's `BUILDLINK_ABI_DEPENDS.`pkg`` must be increased to require the new package version. Then the `PKGREVISION` of all packages `foo` that depend on this package need to be increased, and if they have `buildlink3.mk` files, `BUILDLINK_ABI_DEPENDS.`foo`` in their `buildlink3.mk` files must be increased to the new version as well. This is required so that a package will pull in the versions of the packages that use the new ABI and that the packages' `PKGREVISION`s uniquely identify the packages built against the new ABI. The `pkgtools/revbump` package can help with these updates.

See Section 21.1.5, “Handling dependencies” for more information about dependencies on other packages, including the `BUILDLINK_API_DEPENDS` definitions.

Please take careful consideration before adjusting `BUILDLINK_API_DEPENDS.`pkg`` or `BUILDLINK_ABI_DEPENDS.`pkg`` in a `buildlink3.mk` file as we don't want to cause unneeded package deletions and rebuilds. In many cases, new versions of packages work just fine with older dependencies.

Also, it is not needed to set `BUILDLINK_ABI_DEPENDS.`pkg`` when it is identical to `BUILDLINK_API_DEPENDS.`pkg``.

Note there is also the distinction that users are able to disable enforcement of ABI dependencies using the `USE_ABI_DEPENDS` variable, but there is no equivalent option for API dependencies.

## 18.3. Writing `builtin.mk` files

Some packages in pkgsrc install headers and libraries that coincide with headers and libraries present in the base system. Aside from a `buildlink3.mk` file, these packages should also include a `builtin.mk` file that includes the necessary checks to decide whether using the built-in software or the pkgsrc software is appropriate.

The only requirements of a builtin.mk file for `pkg` are:

1. It should set `USE_BUILTIN.`pkg`` to either “yes” or “no” after it is included.

2. It should not override any `USE_BUILTIN.`pkg`` which is already set before the `builtin.mk` file is included.

3. It should be written to allow multiple inclusion. This is very important and takes careful attention to `Makefile` coding.

### 18.3.1. Anatomy of a `builtin.mk` file

The following is the recommended template for builtin.mk files:

    .if !defined(IS_BUILTIN.foo)
    #
    # IS_BUILTIN.foo is set to "yes" or "no" depending on whether "foo"
    # genuinely exists in the system or not.
    #
    IS_BUILTIN.foo?=        no

    # BUILTIN_PKG.foo should be set here if "foo" is built-in and its package
    # version can be determined.
    #
    .  if !empty(IS_BUILTIN.foo:M[yY][eE][sS])
    BUILTIN_PKG.foo?=       foo-1.0
    .  endif
    .endif  # IS_BUILTIN.foo

    .if !defined(USE_BUILTIN.foo)
    USE_BUILTIN.foo?=       ${IS_BUILTIN.foo}
    .  if defined(BUILTIN_PKG.foo)
    .    for _depend_ in ${BUILDLINK_API_DEPENDS.foo}
    .      if !empty(USE_BUILTIN.foo:M[yY][eE][sS])
    USE_BUILTIN.foo!=                                                       \
            ${PKG_ADMIN} pmatch '${_depend_}' ${BUILTIN_PKG.foo}            \
            && ${ECHO} "yes" || ${ECHO} "no"
    .      endif
    .    endfor
    .  endif
    .endif  # USE_BUILTIN.foo

    CHECK_BUILTIN.foo?=     no
    .if !empty(CHECK_BUILTIN.foo:M[nN][oO])
    #
    # Here we place code that depends on whether USE_BUILTIN.foo is set to
    # "yes" or "no".
    #
    .endif  # CHECK_BUILTIN.foo

The first section sets `IS_BUILTIN.`pkg`` depending on if `pkg` really exists in the base system. This should not be a base system software with similar functionality to `pkg`; it should only be “yes” if the actual package is included as part of the base system. This variable is only used internally within the `builtin.mk` file.

The second section sets `BUILTIN_PKG.`pkg`` to the version of `pkg` in the base system if it exists (if `IS_BUILTIN.`pkg`` is “yes”). This variable is only used internally within the `builtin.mk` file.

The third section sets `USE_BUILTIN.`pkg`` and is required in all `builtin.mk` files. The code in this section must make the determination whether the built-in software is adequate to satisfy the dependencies listed in `BUILDLINK_API_DEPENDS.`pkg``. This is typically done by comparing `BUILTIN_PKG.`pkg`` against each of the dependencies in `BUILDLINK_API_DEPENDS.`pkg``. `USE_BUILTIN.`pkg`` must be set to the correct value by the end of the `builtin.mk` file. Note that `USE_BUILTIN.`pkg`` may be “yes” even if `IS_BUILTIN.`pkg`` is “no” because we may make the determination that the built-in version of the software is similar enough to be used as a replacement.

The last section is guarded by `CHECK_BUILTIN.`pkg``, and includes code that uses the value of `USE_BUILTIN.`pkg`` set in the previous section. This typically includes, e.g., adding additional dependency restrictions and listing additional files to symlink into `${BUILDLINK_DIR}` (via `BUILDLINK_FILES.`pkg``).

Prev | Up | Next
Chapter 17. Tools needed for building or running | Home | Chapter 19. PLIST issues