Experimental project containing INOFFICIAL tools to manage custom Arch Linux repositories; built on top of tools provided by the pacman and devtools packages.
Go to file
Martchus f621a87812 Update tabulate to v1.4 2021-04-18 12:00:18 +02:00
3rdparty Update tabulate to v1.4 2021-04-18 12:00:18 +02:00
cli Apply clang-format 2021-04-16 14:06:38 +02:00
libpkg Fix warnings 2021-03-22 15:08:41 +01:00
librepomgr Prevent running out of memory when signing split packages 2021-04-17 17:30:18 +02:00
pacfind Initial import 2021-02-08 23:53:56 +01:00
srv web UI: Fix links to build action details in build action table 2021-04-17 12:04:40 +02:00
.gitignore Initial import 2021-02-08 23:53:56 +01:00
.gitmodules Initial import 2021-02-08 23:53:56 +01:00
CMakeLists.txt Initial import 2021-02-08 23:53:56 +01:00
LICENSE Initial import 2021-02-08 23:53:56 +01:00
README.md Update README.md 2021-04-14 17:56:29 +02:00

README.md

Repository manager and build tool for Arch Linux

This experimental project contains inofficial tools to manage custom Arch Linux repositories. It is built on top of the official tools provided by the pacman and devtools packages.

So far it consists of:

  • libpkg: C++ library to parse Arch Linux packages and databases
    • parse pacman config
    • parse databases
    • extract and parse binary packages (.PKGINFO, dependencies on soname-level)
    • parse package source infos (.SRCINFO)
  • librepomgr: C++ library for managing custom Arch Linux repositories
  • srv: Daemon and web application for building Arch Linux packages and managing custom Arch Linux repositories
    • build packages via makechrootpkg and add them to a custom repository via repo-add
    • check AUR for package updates
    • build from locally stored PKGBUILDs and from the AUR
    • remove packages from a repository
    • move packages from one repository to another
    • check for repository issues (missing packages, package misses rebuild, …)
    • provides an HTTP API and a web UI with live streaming
  • pacfind: Tool to find the package containing a certain file
    • requires *.files-databases to be present
    • does not need the full file path and supports regex (in contrast to pacman -F)
  • cli: Command line tool to interact with srv
    • search for packages and show package details
    • show and submit build actions

Further ideas (not implemented yet):

  • distri: Tool to distribute applications from the packages in a repository
    • bundle an application with its dependencies similar to linuxdeployqt and windeployqt

Setup server

An example config files can be found in this repository, see the srv/doc directory.

Setting up a working directory

The server needs a place to temporarily store PKGBUILDs, cache files and other stuff. Just create a directory at any place with enough disk space and set the permissions so the user you start the server with can read and write there. This directory goes under working_directory in the settings file. Relative paths within the configuration file (used for other locations) will be treated relative to that directory.

Setting up a database/repository directories

The databases for the official repositories will be synced from the upstream repositories and stored in a local directory. The actual packages might be found in the pacman cache directory. Example configuration:

[database/core]
arch = x86_64
sync_from_mirror = on
dbdir = $sync_db_path/$arch
mirror = $default_mirror/$repo/os/$arch

Own repositories are not synced from a mirror and all packages are expected to be present in a dedicated local directory. It makes most sense to store packages and databases in the same directory. Example configuration:

[database/ownstuff]
path = $own_regular_db_path
files = $own_files_db_path
arch = x86_64
depends = core extra community multilib
pkgdir = $local_db_path/$repo/os/$arch
dbdir = $local_db_path/$repo/os/$arch
mirror = $local_mirror/$repo/os/$arch

The placeholder/variable values come from the previous "definitions" block, e.g.:

[definitions]
sync_db_path = sync-dbs
local_db_path = /run/media/repo/arch
own_sync_db_path = $local_db_path/$repo/os/$arch
own_regular_db_path = $local_db_path/$repo/os/$arch/$repo.db.tar.xz
own_files_db_path = $local_db_path/$repo/os/$arch/$repo.files.tar.xz
default_mirror = http://mirror.23media.de/archlinux
local_mirror = file://$local_db_path

The server obviously needs write permissions to add packages to repositories. In my setup I've just add it as group and set permissions accordingly:

sudo chown -R martchus:buildservice $local_db_path
find $local_db_path -type d -exec chmod 775 {} \+
find $local_db_path -type f -exec chmod 664 {} \+

Setting up the chroot for the build

The overall reasoning and procedure is already outlined in the Wiki.

I also like to be able to build for other architectures than x86_64 so the server expects a directory layout like this:

/the/chroot/directory
├── arch-i686
│   └── root
├── arch-x86_64
│   └── root
├── config-i686
│   ├── makepkg.conf
│   ├── pacman.conf
├── config-x86_64
│   ├── makepkg.conf
│   └── pacman.conf

So there's a "root" chroot for every architecture and config files for that architecture in dedicated directories. To achieve this one just has to call mkarchroot for each architecture. It is invoked like this:

mkarchroot -C /path/to/pacman.conf -M /path/to/makepkg.conf /directory/to/store/the/tree base-devel

Note that makechrootpkg will not use the "root" chroot directories directly. It will create a working copy using rsync -a --delete -q -W -x "$chrootdir/root/" "$copydir" (when using btrfs it creates a subvolume instead of using rsync).

When following the procedure the "root" chroot is owned by root and the working copies by the user you start the buildservice/makechrootpkg with.

Until I find a better solution the buildservice updates the config files within the "root" chroot to configure the databases used during the build. It relies on the configuration files stored within the config-* directories for that and needs permissions to update them:

sudo chown buildservice:buildservice /the/chroot/directory/arch-x86_64/root/etc/{pacman,makepkg}.conf

sudo configuration

makechrootpkg seems to use sudo internally so it is required that the user one starts the server with is allowed to use sudo. Currently it is not possible to supply the password automatically so one has to allow access without password like this:

buildservice ALL=(ALL) NOPASSWD:ALL

Sample NGINX config

To make the server publicy accessible one should use a reverse proxy. NGINX example:

proxy_http_version 1.1; # this is essential for chunked responses to work
proxy_read_timeout 1d; # prevent timeouts when serving live stream

location /buildservice/api/ {
    proxy_buffering         off; # for live streaming of logfiles
    proxy_pass              http://localhost:8090/api/;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
}
location ~ ^(?!/buildservice/api/)/buildservice/(.*)$ {
    alias /run/media/devel/projects/c++/cmake/auto-makepkg/buildservice/static/$1;
}

Sample GPG config

This minimal GPG config allows makepkg to validate signatures which might be present in some PKGBUILDs:

~/.gnupg/gpg.conf
---
utf8-strings
keyring /etc/pacman.d/gnupg/pubring.gpg
keyserver-options auto-key-retrieve
keyserver hkp://keys.gnupg.net

Notes

ccache configuration

To use ccache one can set the ccache directory in the config (ccache_dir) and install the package ccache (and mingw-w64-ccache for MinGW packages) into the chroot. Make sure the user you start the server with has permissions to read and write there. Otherwise the resulting configure errors can be confusing. Internally the server is mounting that directory like described in the wiki.

Note that ccache slows down the initial compilation. It is only useful to enable it if rebuilds with only slight changes are expected. Currently it is not possible to enable/disable ccache usage per package (e.g. via a regex).

Workflow

General

  • Use the "Reload database" build action to reload one or more databases after databases have been changes by build actions or manually. So far databases are not automatically reloaded.
  • Use the "Reload configuration" build action to apply configuration changes without restarting the service.

Building packages

  1. Ensure the databases are up to date via the "Reload databases" build action.
  2. Start the "Prepare building" build action.
    • Specify the names of packages to be built.
      • If a package already exists the pkgrel or epoch is bumped as needed so the rebuilt packages are considered as newer by pacman. A warning is logged when bumping is done.
      • Missing dependencies are pulled into the build automatically. Whether dependencies are considered "missing" or "given" depends on the specified databases (see next points).
      • Packages will be splitted into batches. The first batch contains all packages which can be built immediately. The second batch contains all packages which only depend on packages in the first batch. The third batch contains all packages which only depend on packages in the second batch and so on. That means packages do not need to be specified in the correct order. However, the order in which packages are specified is preserved within each batch so it makes still sense to specify packages in the order you would prefer to build them.
      • Cyclic dependencies can not be added to a batch. A list of cyclic "leftovers" is emitted if those exist and that is considered a failure. If this is the case you need to add a bootstrap package to break the cycle. The build system is not clever enough to pull a bootstrap package automatically into the build so it must be specified explicitely. E.g. to build mingw-w64-freetype2 and mingw-w64-harfbuzz one needs to add mingw-w64-freetype2-bootstrap explicitely to the list of packages to be built.
    • Specify exactly one destination database. The built packages are added to this database.
    • Specify source databases.
      • All packages contained by the source databases are considered as "given" and not pulled into the build as dependency.
      • If no source databases are specified the destination database and all databases the destination database is based on are used as source databases.
      • Source databases must be the destination database or a database the destination database is based on.
    • Specify a "directory" to store meta-data, logs, PKGBUILDs, sources and packages.
    • Check "Clean source directories" when previously prepared sources which are still present in the specified "directory" should be overridden.
    • When the build action has finished, have a look at the results to check whether they are as expected. If not, just restart the build action with adjusted parameters. The "directory" can generally be re-used. Use "Clean source directories" as needed. To override only a single source directory, simply delete it manually before restarting the build action.
    • You can amend PKGBUILDs created in the "directory" as needed.
  3. Start the "Conduct building" build action.
    • Specify the same "directory" as in 1.
    • Do not specify any databases. The databases as specified in 1. will be used.
    • One can optionally specify package names to build only a subset of the initially prepared build.
    • When starting the build, the following steps are performed:
      1. All sources of all packages are downloaded. The build is stopped when not all sources can be downloaded.
      2. Packages will now be built in the same order as computed when preparing. Building of the next batch is stopped when a failure happend within the previous batch.
    • When using "Build as far as possible" the build is not stopped as previously explained.
    • When using "Save chroot of failures" the chroot directory is renamed when a build failure happens and therefore not overridden by the next build. This is useful to investigate the failure later without starting the build from scratch.
    • When the build has been stopped due to failures one can simply restart the build action. Packages which have already been built will be skipped.
      • It is useful to modify build-progress.json in the "directory".
        • Set finished back to "1" and addedToRepo back to false to let the build system think a package has not been built yet. This is useful to build a package again after a non-fatal packaging mistake.
        • Set addedToRepo to true let the build system think a package has already been built. This effectively skips a package completely.
        • Set hasSources back to false to recreate the source package and download sources again as needed. This will also copy over contents of the src directory to the pkg directory again.
  4. Amend a PKGBUILD file to fix a failure within a "big rebuild".
    • Fix the package at the source, e.g. the AUR or the local package directory, then
      1. Clean the src directory within "directory" for the package(s) or use "Clean source directory" if all packages need fixing.
      2. Restart step 1. Existing sources are reused (except for the removed src directories) so it shouldn't take long. The build progress (hasSources in particular) of the affected package(s) should be resetted.
      3. Restart step 2.
    • or: Amend the PKGBUILD within the src directory.
      1. Set hasSources back to false as mentioned under 2. to retrigger building the source directory.
      2. Optionally restart step 1. to reevaluate possibly adjusted versions, split packages and dependencies. Be sure not to check "Clean source directory" as this would override your amendment.
      3. Restart step 2.

Further notes

  • One can keep using the same "directory" for different builds. Since the pkg dirs will not be cleaned up (existing files are only updated/overridden) previously downloaded source files can be reused (useful if only pkgrel changes or when building from VCS sources or when some sources just remain the same).

TODOs and further ideas for improvement

[ ] Use persistent database (e.g. lmdb or leveldb) [ ] More advanced search options [ ] Allow running tasks automatically/periodically [ ] Refresh build action details automatically while an action is pending [ ] Fix permission error when stopping a process [ ] Keep the possibility for a "soft stop" where the build action would finish the current item [ ] Show statistics like CPU and RAM usage about ongoing build processes [ ] Stop a build process which doesn't produce output after a certain time [ ] Find out why the web service sometimes gets stuck * Weirdly, restarting the client (browser) helps in these cases * Add "stress" test for live-streaming * Start process producing lots of output very fast * Let different clients connect and disconnect fast [ ] Improve test coverage [ ] Include xterm.js properly

Build instructions and dependencies

For a PKGBUILD checkout my PKGBUILDs repository.

C++ stuff

The application depends on c++utilities, reflective-rapidjson, some Boost modules and OpenSSL.

For basic instructions checkout the README file of c++utilities.

Web stuff

The only dependency is xterm.js which is bundled. However, only the JavaScript and CSS files are bundled. For development the full checkout might be useful (e.g. for TypeScript mapping). It can be retrieved using npm:

cd srv/static/node_modules
npm install xterm