diff --git a/README.md b/README.md index 287d03b..956ff14 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,17 @@ This **experimental** project contains **inofficial** tools to manage custom Arc repositories. It is built on top of the official tools provided by the `pacman` and `devtools` packages. -So far it consists of: +At this point the project is rather raw and there are many things left to implement and +to improve (checkout the TODOs section below). Currently builds are exclusively triggered +manually (although the API brings the possibility for automation), scalability is limited +through the use of an in-memory database. Builds are only conducted on one host. + +On the upside, this project can easily be used to together with other build scripts. It +doesn't care if your repository's DB file is updated by another application; just be sure +to reload the DB file (see Workflow section below). This should make an easy migration +path to use this for maintaining an existing repository. + +So far this project consists of: * libpkg: C++ library to parse Arch Linux packages and databases * parse pacman config @@ -35,7 +45,19 @@ Further ideas (not implemented yet): ## Setup server An example config files can be found in this repository, see the `srv/doc` -directory. +directory. The example config is also installed so one can easily get started. + +This is an example for using the package +[`arch-repo-manager-git`](https://github.com/Martchus/PKGBUILDs/blob/master/arch-repo-manager/git/PKGBUILD): + +``` +mkdir -p /etc/buildservice-git +cp /usr/share/buildservice-git/skel/server-config-example.conf /etc/buildservice-git/server.conf +cp /usr/share/buildservice-git/skel/presets-example.json /etc/buildservice-git/presets.json +systemctl enable --now buildservice-git.service +``` + +Note that the `-git` package is used at this point because there are no releases yet. ### Setting up a working directory The server needs a place to temporarily store PKGBUILDs, cache files and other stuff. @@ -181,7 +203,6 @@ keyserver hkp://keys.gnupg.net ``` #### Notes - * Adding the pacman keyring is actually not very useful because we need to check signatures of any upstream project and not just arch devs. * If "auto-key-retrieve" does not work, use `gpg --recv-key ` as a workaround @@ -199,10 +220,20 @@ with only slight changes are expected. Currently it is *not* possible to enable/ per package (e.g. via a regex). ## Workflow +So called "build actions" are used to conduct certain tasks, e.g. checking for updates or problems, +moving packages from one repository to another, preparing the build and conducting the build. The web +UI contains a form to create a new build action and shows the possible options. + +It would be tedious to create the same set of build actions over and over again. Hence it is also +possible to create "pre defined tasks". Such a task contains a list of build actions with pre-filled +flags and settings. By default, the build actions within a task run one after another but it is also +possible to specify that certain build actions can run in parallel. The presets are configured by +editing the presets JSON file (e.g. `/etc/buildservice-git/presets.json` in the example config). ### General -* Use the "Reload database" build action to reload one or more databases after databases have been - changes by build actions or manually. So far databases are *not* automatically reloaded. +* So far databases are *not* automatically reloaded. Use the "Reload database" build action to reload + one or more databases after databases have been changes by build actions or manually. This build + action is similar to `pacman -Sy`. * Use the "Reload configuration" build action to apply configuration changes without restarting the service. @@ -286,21 +317,23 @@ per package (e.g. via a regex). only `pkgrel` changes or when building from VCS sources or when some sources just remain the same). ## TODOs and further ideas for improvement -[ ] Use persistent database (e.g. lmdb or leveldb) -[ ] More advanced search options -[ ] Allow running tasks automatically/periodically -[ ] Refresh build action details automatically while an action is pending -[ ] Fix permission error when stopping a process -[ ] Keep the possibility for a "soft stop" where the build action would finish the current item -[ ] Show statistics like CPU and RAM usage about ongoing build processes -[ ] Stop a build process which doesn't produce output after a certain time -[ ] Find out why the web service sometimes gets stuck +* [ ] Use persistent, non-in-memory database (e.g. lmdb or leveldb) to improve scalability +* [ ] Allow triggering tasks automatically/periodically +* [ ] Allow to run `makechrootpkg` on a remote host (e.g. via SSH) to work can be spread across multiple hosts +* [ ] More advanced search options +* [ ] Refresh build action details on the web UI automatically while an action is pending +* [ ] Fix permission error when stopping a process +* [ ] Keep the possibility for a "soft stop" where the build action would finish the current item +* [ ] Show statistics like CPU and RAM usage about ongoing build processes +* [ ] Stop a build process which doesn't produce output after a certain time +* [ ] Find out why the web service sometimes gets stuck * Weirdly, restarting the client (browser) helps in these cases * Add "stress" test for live-streaming * Start process producing lots of output very fast * Let different clients connect and disconnect fast -[ ] Improve test coverage -[ ] Include `xterm.js` properly +* [ ] Improve test coverage +* [ ] Add fancy graphs for dependencies on the web UI +* [ ] Include `xterm.js` properly ## Build instructions and dependencies For a PKGBUILD checkout my [PKGBUILDs repository](https://github.com/Martchus/PKGBUILDs). diff --git a/srv/CMakeLists.txt b/srv/CMakeLists.txt index 748cbb3..f5da26b 100644 --- a/srv/CMakeLists.txt +++ b/srv/CMakeLists.txt @@ -98,7 +98,7 @@ endif () # add installs for config file template install( - FILES doc/server-config-example.conf doc/nginx-config-example.conf + FILES doc/server-config-example.conf doc/presets-example.json doc/nginx-config-example.conf DESTINATION "${CMAKE_INSTALL_PREFIX}/${META_DATA_DIR}/skel" COMPONENT config) diff --git a/srv/doc/presets-example.json b/srv/doc/presets-example.json new file mode 100644 index 0000000..dfc8fb6 --- /dev/null +++ b/srv/doc/presets-example.json @@ -0,0 +1,193 @@ +{ + "templates": { + "reload-dbs": { + "directory": "", + "packageNames": [], + "sourceDbs": [], + "destinationDbs": [ + "core", + "extra", + "community", + "multilib", + "staging", + "testing", + "community-staging", + "community-testing", + "multilib-staging", + "multilib-testing", + "kde-unstable", + "gnome-unstable", + "ownstuff", + "ownstuff-testing", + "ownstuff-staging", + "aur" + ], + "settings": {}, + "flags": 0, + "type": 4 + }, + "reload-libs": { + "directory": "", + "packageNames": [], + "sourceDbs": [], + "destinationDbs": [ + "ownstuff", + "ownstuff-testing", + "ownstuff-staging", + ], + "settings": {}, + "flags": 0, + "type": 5 + }, + "update-chroot": { + "directory": "chroot-update", + "packageNames": [], + "sourceDbs": [], + "destinationDbs": [], + "settings": { + "cmd": "…", + "shared-locks": "…" + }, + "flags": 0, + "type": 13 + }, + "update-ownstuff": { + "directory": "", + "packageNames": [], + "sourceDbs": ["aur"], + "destinationDbs": ["ownstuff"], + "settings": {}, + "flags": 0, + "type": 3 + }, + "prepare-ownstuff": { + "directory": "misc", + "packageNames": [], + "sourceDbs": ["ownstuff", "core", "extra", "community", "multilib"], + "destinationDbs": ["ownstuff"], + "settings": {}, + "flags": 2, + "type": 6 + }, + "prepare-ownstuff-testing": { + "directory": "misc", + "packageNames": [], + "sourceDbs": ["ownstuff-testing", "ownstuff", "core", "extra", "community", "multilib", "testing", "community-testing", "multilib-testing"], + "destinationDbs": ["ownstuff-testing"], + "settings": {}, + "flags": 2, + "type": 6 + }, + "prepare-ownstuff-staging": { + "directory": "misc", + "packageNames": [], + "sourceDbs": ["ownstuff-staging", "ownstuff-testing", "ownstuff", "core", "extra", "community", "multilib", "staging", "community-staging", "multilib-staging", "testing", "community-testing", "multilib-testing"], + "destinationDbs": ["ownstuff-staging"], + "settings": {}, + "flags": 2, + "type": 6 + }, + "conduct-build": { + "directory": "misc", + "packageNames": [], + "sourceDbs": [], + "destinationDbs": [], + "settings": {}, + "flags": 9, + "type": 7 + }, + "conduct-build-save-chroots-of-failures": { + "directory": "misc", + "packageNames": [], + "sourceDbs": [], + "destinationDbs": [], + "settings": {}, + "flags": 11, + "type": 7 + }, + "sync-ownstuff": { + "directory": "sync-ownstuff", + "packageNames": [], + "sourceDbs": [], + "destinationDbs": [], + "settings": { + "cmd": "rsync …" + }, + "flags": 0, + "type": 13 + } + }, + "tasks": { + "conduct-build": { + "name": "Conduct build", + "desc": "Perform build with default settings", + "category": "Building", + "actions": ["update-chroot", "conduct-build", "sync-ownstuff"] + }, + "conduct-build-save-chroots-of-failures": { + "name": "Conduct build saving chroots of failures", + "desc": "Perform build with default settings saving chroots of failures", + "category": "Building", + "actions": ["update-chroot", "conduct-build-save-chroots-of-failures", "sync-ownstuff"] + }, + "prepare-ownstuff": { + "name": "Prepare building for ownstuff", + "desc": "Prepare building packages for ownstuff (x86_64)", + "category": "Building", + "actions": ["reload-dbs", "prepare-ownstuff"] + }, + "prepare-ownstuff-testing": { + "name": "Prepare building for ownstuff-testing", + "desc": "Prepare building packages for ownstuff-testing (x86_64)", + "category": "Building", + "actions": ["reload-dbs", "prepare-ownstuff-testing"] + }, + "prepare-ownstuff-staging": { + "name": "Prepare building for ownstuff-staging", + "desc": "Prepare building packages for ownstuff-staging (x86_64)", + "category": "Building", + "actions": ["reload-dbs", "prepare-ownstuff-staging"] + }, + "build-ownstuff": { + "name": "Build for ownstuff", + "desc": "Prepare and build packages for ownstuff saving chroots of failures (x86_64)", + "category": "Building", + "actions": [ + { + "actions": ["reload-dbs", "update-chroot"], + "concurrent": true + }, + "prepare-ownstuff", + "conduct-build-save-chroots-of-failures", + "sync-ownstuff" + ] + }, + "update-ownstuff": { + "name": "Check ownstuff for updates", + "desc": "Check ownstuff for updates (x86_64)", + "category": "Repo management", + "actions": [ + "reload-dbs", + { + "actions": [ + "update-ownstuff" + ], + "concurrent": true + } + ] + }, + "sync-ownstuff": { + "name": "Sync ownstuff", + "desc": "Syncs f3l mirror of ownstuff{,-testing,-staging}", + "category": "Repo management", + "actions": ["sync-ownstuff"] + }, + "update-chroot": { + "name": "Update chroots", + "desc": "Updates the chroots (so far covers only x86_64)", + "category": "Repo management", + "actions": ["update-chroot"] + } + } +} + diff --git a/srv/doc/server-config-example.conf b/srv/doc/server-config-example.conf index a7fc0d3..f041c86 100644 --- a/srv/doc/server-config-example.conf +++ b/srv/doc/server-config-example.conf @@ -15,6 +15,7 @@ working_directory = /var/lib/buildservice/building local_pkgbuilds_dir = /run/media/devel/projects/pkgbuilds ignore_local_pkgbuilds_regex = ^(some-package-name|yet-another-package-name)$ chroot_dir = /directory/with/enough/space/for/chroot +#presets = presets.json #ccache_dir = /the/ccache/directory #package_cache_dir = /var/cache/pacman/pkg