Tips-and-Tricks/poudriere

Tips and tricks for poudriere on FreeBSD

Introduction

Poudriere (ports-mgmt/poudriere[-devel] in FreeBSD ports collection) is a feature-rich clean-room builder, mainly for ports.
It can have multiple builder jails (i.e.) for multiple versions of FreeBSD, multiple archs and so on.

Because of above, examples on its official site focuses on such a complexed situations.

For anyone who builds ports/pkgs locally only for the very local host itself, it is clearly overkill.

I wanted to use

* default ports tree (/usr/ports),
* default configured options (saved under /var/db/ports)
* and saved distfiles (/usr/ports/distfiles).

And more, as I'm tracking latest stable branch of base (currently stable/14), I want to use /usr/src as the source tree for jail and /usr/obj as already-built binaries to install.
But I was forced to look for infos here and there, in manpages, on the official website, googling other places.

So I record here for the procedures I've done to achieve the above configuration.
Note that the procedures below is how I did at the moment. Some options could be changed now due to the updates on poudriere[-devel].

Creating builder jail

To create buider jail called "14amd64" for stable/14 branch using /usr/src with kernel configuration file TEST15 and amd64 architecture,
poudriere jail -c -v stable/14 -a amd64 -K TEST15 -j 14amd64 -m src=/usr/src

Configuring ports tree to use

To use existing ports tree at default place (/usr/ports), null-mounted in builder jails,
poudriere ports -c -f none -M /usr/ports -m null

Configuring sccache-overlay for Rust builds is not recommended for now

Currently I don't recommend to use it. This often causes race conditions if multiple builder jails are running. sccache daemonized process fails to acknowledge in time and causes timeout on poudriere side, causing nonsense build failures.

This would NOT be a responsibility of sccache-overlay, but poudriere. It should give much more time for sccache to acknowledge. But unfortunately I couldn't have determined where in the poudriere to be tuned, including modifies to its sources. Recently, the default timeouts on sccache-overlay side was raised by ports-mgmt/sccache-overlay: Increase the timeout for server start, but still fails on certain situations.

If you want to activate sccache-overlay support regardless of above, you need to do below for preparation.
poudriere ports -c -p sccache -m null -M /usr/local/share/sccache/overlay

Configuring to share options between bare-metal builds and poudriere builder jails

To use already-saved options at default place for poudriere builder jails, too,
ln -s /var/db/ports /usr/local/etc/poudriere.d/options

Note that if you already have /usr/local/etc/poudriere.d/options directory, as you may know, delete (if empty) or rename it before doing above to create symlink.

Configuring local pkg repository built by poudriere[-devel] without installing/configuring local web server

Poudriere maintains local pkg repository by itself. But examples to use locally built repository is to configure web server to be accessed by pkg.

This is useful if there are multiple computers sharing locally built (or cached official) pkgs, but for single computer only, it's clearly overkill.

Fortunately, the protocol part of the configuration can be file:// instead of http[s]://. Moreover, pkg has functionalities to use multiple repository and prioritize them. Placing this example file in /usr/local/etc/pkg/repos/ would make pkg to use pkgs under /poudriere/data/packages/14amd64-default directory.
If required pkgs aren't exist, ones in official repository are looked for and used.
No need to disable default official repository.

Updating builder jail

To update the builder jail above using latest locally built base,
poudriere jail -u -v stable/14 -a amd64 -K TEST15 -j 14amd64 -m src=/usr/src

Updating ports

Updating ports tree

As the same ports tree is shared between bare-metal builds and poudriere builder jails, you can simply update the ports tree just as you did before you introduced poudriere.

Fetching distfiles

If I recall correctly, poudriere uses /usr/ports/distfiles/ as the distfile cache by default. (I don't remember I'd configured the setting manually.)

So you can fetch distfiles as before, or leave it for poudriere.

But my recommendation is to fetch needed distfiles as before not to be bitten by problems at fetch phase, causing builds to halt for a long time until timeouts.

Sometimes upstream repo is stalled, distinfo is missingly updated, and/or distfiles on upstream is rerolled.

Updating single or a few ports without forcible rebuilds of ports that depends on them

By default, poudriere[-devel] attempts to build ALL ports depending on ports to be rebuilt.

For example, if you have editors/libreoffice and japanese/libreoffice (as a language pack depending upon editors/libreoffice) and only editors/libreoffice is updated (for example, changed default options only), when you attempt to build editors/libreoffice, poudriere also forcibly rebuild japanese/libreoffice even though it's not at all updated.

If the ports are used by a plenty of ports, like devel/glib20, and even if the update introduced is known (to human) NOT to be affecting any ports depending upon it, poudriere forcibly and recursively rebuild every pkgs in its local repository.

Yes, it's the safest side, especially when some needed bumps are (accidentally) missed. But at the same time, in many cases, just a huge mess.

To avoid this, although it's discouraged by authors of poudriere, you can specify option "-S".

Flavors

If you want to be 100% sure that installed ports with non-default flavor, you can specify it with adding "@flavor", say, category/port@flavor form. For example, if you want to specify non-default flavor "gtk2" for textproc/fcitx5-gtk, you can specify it as textproc/fcitx5-gtk@gtk2.

Updating all but large leaf ports

Poudriere has a functionality to rebuild all updated ports installed on bare-metal environment by option "-a".

But this causes huge leaf ports (like www/chromium) to be built at the same time.

This is too much a mess when you are in hurry.

Fortunately, poudriere has an option to (re)build using list file.

I've created a small script to generate a list of all installed pkgs excluding specified ones and ones listed in MOVED without its successor. You can obtain it as poudlist-all.sh here.

Once the list is generated (by default, /poudriere/pkglist.all), you can invoke (for the examples here)

poudriere bulk -f /poudriere/pkglist.all -j 14amd64

to (re)build updated ports in the list (and any ports depending on listed ports directly and indirectly, if you don't specify "-S" option).

Note that you should run the script everytime you update the ports tree not to miss new MOVED entry. Otherwise, if any of ports you have in bare-metal environment are removed from ports tree, the list containing entries without sane origin (category/port) causes poudriere to error out and not starting actual builds.

And also, the script doesn't aware of FLAVORs. Just picks the origins of installed ports.
But don't worry.
The script automatically picks up FLAVOR used for builds. See its README.md and script itself for details. So you need to edit or drop the flavor when you want to change its flavor to be changed, including to follow changes in default flaver.

The list generated is just a text file, containing ports origins one-per-line. You can add "@flavor" at the end of the specific line.

Even more, if you want to specify multiple flavors for single origin, you can duplicate the line as needed.

Dry run (See what's done without actual builds)

You should be adviced to dry-run before actually start rebuilding to see how many and which ports are to be built. You can add option "-n" to make poudriere buil dry-run.