Reproducible builds

From ArchWiki

Arch Linux is currently working on making all packages reproducible. This lets users and researchers verify the distributed packages from Arch Linux. For the exact definition of reproducible builds and its benefits take a look at the project website.

Verification builds

An experimental rebuilderd instance has been setup on our own infrastructure with a status page. Rebuilderd rebuilds our repository packages and checks if they are bit for bit identical. If they are not reproducible there is either a bug in the tooling, the package is not reproducible or the package has not been built cleanly.

A list of known issues is located on /Status.

Helping out

Tooling

Help out on fixing bugs and adding features for repro.

Running a Rebuilder instance

Setting up rebuilderd to build Arch Linux packages helps to independently verify repository packages.

Verifying packages with repro and finding issues

A great way to help out is to find an unreproducible package and figuring out how it can be made reproducible.

  • Download an Arch Linux package or get one from the Arch Linux Archive
  • Run repro on the downloaded package or a package from your pacman cache. Ideally with repro -d to get diffoscope output. For example, repro -d /var/cache/pacman/pkg/curl-7.73.0-1-x86_64.pkg.tar.zst
  • Investigate if it is an issue with Arch Linux packaging or upstream, issues can be added on the status page. More information can be found on the reproducible builds website.

Work on issues in the tests.reproducible-builds.org infrastructure

Arch users can help contribute to Reproducible Build issues by looking at the continuous reproducing environment. There are various issues which can be sorted out:

  • FTBS (failed to build from source): reproduce the build failure locally and create a bug report if the package cannot be built from a clean chroot (extra-x86_64-build or multilib-build).
  • Failed to download sources, reproduce the issue (makepkg -o -d) and create a bug report on the Arch bugtracker.
  • Failed to reproduce. Locally you can reproduce packages using reprotest. Note that not all variations can be used. For simple time related testing: 
    $ reprotest --variations '+time' 'sudo extra-x86_64-build' '*.pkg.tar.zst'

There might be various reasons for a package to not be reproducible, but before digging in take a look at the upstream repository or the reproducible status in Debian

  • Failed to run tests, these failures are heavily on the testing environment. Most likely due to to LANG=C being set and Arch not supporting LANG=C.UTF-8.

If you are interested in the code which runs the continuous reproducing environment, the first build code starts here on salsa

Note: The continuous reproducing environment on tests.reproducible-builds.org does NOT reproduce official arch packages. Its purpose is only to fuzz the packages and attempt to trigger as many bugs as possible, so they can be reported upstream.

Known issues

GPG verification

There is a possible rebuild scenario where GPG keys will not verify as the packager was removed from the keyring or revoked as we use the latest keyring and a package in the archive which we need might need be signed by a revoked key we are unable to verify it and the build will fail.

Contact