Martchus
/
arch-repo-manager
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Improve README.md and examples

This commit is contained in:
Martchus 2021-06-03 23:01:30 +02:00
parent 3a797bc8b6
commit e1b1f99ecc
4 changed files with 244 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

65
README.md
View File

@ -4,7 +4,17 @@ This **experimental** project contains **inofficial** tools to manage custom Arc
repositories. It is built on top of the official tools provided by the `pacman` and
`devtools` packages.
So far it consists of:
At this point the project is rather raw and there are many things left to implement and
to improve (checkout the TODOs section below). Currently builds are exclusively triggered
manually (although the API brings the possibility for automation), scalability is limited
through the use of an in-memory database. Builds are only conducted on one host.
On the upside, this project can easily be used to together with other build scripts. It
doesn't care if your repository's DB file is updated by another application; just be sure
to reload the DB file (see Workflow section below). This should make an easy migration
path to use this for maintaining an existing repository.
So far this project consists of:
* libpkg: C++ library to parse Arch Linux packages and databases
* parse pacman config
@ -35,7 +45,19 @@ Further ideas (not implemented yet):
## Setup server
An example config files can be found in this repository, see the `srv/doc`
directory.
directory. The example config is also installed so one can easily get started.
This is an example for using the package
[`arch-repo-manager-git`](https://github.com/Martchus/PKGBUILDs/blob/master/arch-repo-manager/git/PKGBUILD):
```
mkdir -p /etc/buildservice-git
cp /usr/share/buildservice-git/skel/server-config-example.conf /etc/buildservice-git/server.conf
cp /usr/share/buildservice-git/skel/presets-example.json /etc/buildservice-git/presets.json
systemctl enable --now buildservice-git.service
```
Note that the `-git` package is used at this point because there are no releases yet.
### Setting up a working directory
The server needs a place to temporarily store PKGBUILDs, cache files and other stuff.
@ -181,7 +203,6 @@ keyserver hkp://keys.gnupg.net
```
#### Notes
* Adding the pacman keyring is actually not very useful because we need to check signatures
of any upstream project and not just arch devs.
* If "auto-key-retrieve" does not work, use `gpg --recv-key <KEYID>` as a workaround
@ -199,10 +220,20 @@ with only slight changes are expected. Currently it is *not* possible to enable/
per package (e.g. via a regex).
## Workflow
So called "build actions" are used to conduct certain tasks, e.g. checking for updates or problems,
moving packages from one repository to another, preparing the build and conducting the build. The web
UI contains a form to create a new build action and shows the possible options.
It would be tedious to create the same set of build actions over and over again. Hence it is also
possible to create "pre defined tasks". Such a task contains a list of build actions with pre-filled
flags and settings. By default, the build actions within a task run one after another but it is also
possible to specify that certain build actions can run in parallel. The presets are configured by
editing the presets JSON file (e.g. `/etc/buildservice-git/presets.json` in the example config).
### General
* Use the "Reload database" build action to reload one or more databases after databases have been
changes by build actions or manually. So far databases are *not* automatically reloaded.
* So far databases are *not* automatically reloaded. Use the "Reload database" build action to reload
one or more databases after databases have been changes by build actions or manually. This build
action is similar to `pacman -Sy`.
* Use the "Reload configuration" build action to apply configuration changes without restarting the
service.
@ -286,21 +317,23 @@ per package (e.g. via a regex).
only `pkgrel` changes or when building from VCS sources or when some sources just remain the same).
## TODOs and further ideas for improvement
[ ] Use persistent database (e.g. lmdb or leveldb)
[ ] More advanced search options
[ ] Allow running tasks automatically/periodically
[ ] Refresh build action details automatically while an action is pending
[ ] Fix permission error when stopping a process
[ ] Keep the possibility for a "soft stop" where the build action would finish the current item
[ ] Show statistics like CPU and RAM usage about ongoing build processes
[ ] Stop a build process which doesn't produce output after a certain time
[ ] Find out why the web service sometimes gets stuck
* [ ] Use persistent, non-in-memory database (e.g. lmdb or leveldb) to improve scalability
* [ ] Allow triggering tasks automatically/periodically
* [ ] Allow to run `makechrootpkg` on a remote host (e.g. via SSH) to work can be spread across multiple hosts
* [ ] More advanced search options
* [ ] Refresh build action details on the web UI automatically while an action is pending
* [ ] Fix permission error when stopping a process
* [ ] Keep the possibility for a "soft stop" where the build action would finish the current item
* [ ] Show statistics like CPU and RAM usage about ongoing build processes
* [ ] Stop a build process which doesn't produce output after a certain time
* [ ] Find out why the web service sometimes gets stuck
* Weirdly, restarting the client (browser) helps in these cases
* Add "stress" test for live-streaming
* Start process producing lots of output very fast
* Let different clients connect and disconnect fast
[ ] Improve test coverage
[ ] Include `xterm.js` properly
* [ ] Improve test coverage
* [ ] Add fancy graphs for dependencies on the web UI
* [ ] Include `xterm.js` properly
## Build instructions and dependencies
For a PKGBUILD checkout my [PKGBUILDs repository](https://github.com/Martchus/PKGBUILDs).

2
srv/CMakeLists.txt
View File

@ -98,7 +98,7 @@ endif ()
# add installs for config file template
install(
FILES doc/server-config-example.conf doc/nginx-config-example.conf
FILES doc/server-config-example.conf doc/presets-example.json doc/nginx-config-example.conf
DESTINATION "${CMAKE_INSTALL_PREFIX}/${META_DATA_DIR}/skel"
COMPONENT config)

193
srv/doc/presets-example.json Normal file
View File

@ -0,0 +1,193 @@
{
"templates": {
"reload-dbs": {
"directory": "",
"packageNames": [],
"sourceDbs": [],
"destinationDbs": [
"core",
"extra",
"community",
"multilib",
"staging",
"testing",
"community-staging",
"community-testing",
"multilib-staging",
"multilib-testing",
"kde-unstable",
"gnome-unstable",
"ownstuff",
"ownstuff-testing",
"ownstuff-staging",
"aur"
],
"settings": {},
"flags": 0,
"type": 4
},
"reload-libs": {
"directory": "",
"packageNames": [],
"sourceDbs": [],
"destinationDbs": [
"ownstuff",
"ownstuff-testing",
"ownstuff-staging",
],
"settings": {},
"flags": 0,
"type": 5
},
"update-chroot": {
"directory": "chroot-update",
"packageNames": [],
"sourceDbs": [],
"destinationDbs": [],
"settings": {
"cmd": "…",
"shared-locks": "…"
},
"flags": 0,
"type": 13
},
"update-ownstuff": {
"directory": "",
"packageNames": [],
"sourceDbs": ["aur"],
"destinationDbs": ["ownstuff"],
"settings": {},
"flags": 0,
"type": 3
},
"prepare-ownstuff": {
"directory": "misc",
"packageNames": [],
"sourceDbs": ["ownstuff", "core", "extra", "community", "multilib"],
"destinationDbs": ["ownstuff"],
"settings": {},
"flags": 2,
"type": 6
},
"prepare-ownstuff-testing": {
"directory": "misc",
"packageNames": [],
"sourceDbs": ["ownstuff-testing", "ownstuff", "core", "extra", "community", "multilib", "testing", "community-testing", "multilib-testing"],
"destinationDbs": ["ownstuff-testing"],
"settings": {},
"flags": 2,
"type": 6
},
"prepare-ownstuff-staging": {
"directory": "misc",
"packageNames": [],
"sourceDbs": ["ownstuff-staging", "ownstuff-testing", "ownstuff", "core", "extra", "community", "multilib", "staging", "community-staging", "multilib-staging", "testing", "community-testing", "multilib-testing"],
"destinationDbs": ["ownstuff-staging"],
"settings": {},
"flags": 2,
"type": 6
},
"conduct-build": {
"directory": "misc",
"packageNames": [],
"sourceDbs": [],
"destinationDbs": [],
"settings": {},
"flags": 9,
"type": 7
},
"conduct-build-save-chroots-of-failures": {
"directory": "misc",
"packageNames": [],
"sourceDbs": [],
"destinationDbs": [],
"settings": {},
"flags": 11,
"type": 7
},
"sync-ownstuff": {
"directory": "sync-ownstuff",
"packageNames": [],
"sourceDbs": [],
"destinationDbs": [],
"settings": {
"cmd": "rsync …"
},
"flags": 0,
"type": 13
}
},
"tasks": {
"conduct-build": {
"name": "Conduct build",
"desc": "Perform build with default settings",
"category": "Building",
"actions": ["update-chroot", "conduct-build", "sync-ownstuff"]
},
"conduct-build-save-chroots-of-failures": {
"name": "Conduct build saving chroots of failures",
"desc": "Perform build with default settings saving chroots of failures",
"category": "Building",
"actions": ["update-chroot", "conduct-build-save-chroots-of-failures", "sync-ownstuff"]
},
"prepare-ownstuff": {
"name": "Prepare building for ownstuff",
"desc": "Prepare building packages for ownstuff (x86_64)",
"category": "Building",
"actions": ["reload-dbs", "prepare-ownstuff"]
},
"prepare-ownstuff-testing": {
"name": "Prepare building for ownstuff-testing",
"desc": "Prepare building packages for ownstuff-testing (x86_64)",
"category": "Building",
"actions": ["reload-dbs", "prepare-ownstuff-testing"]
},
"prepare-ownstuff-staging": {
"name": "Prepare building for ownstuff-staging",
"desc": "Prepare building packages for ownstuff-staging (x86_64)",
"category": "Building",
"actions": ["reload-dbs", "prepare-ownstuff-staging"]
},
"build-ownstuff": {
"name": "Build for ownstuff",
"desc": "Prepare and build packages for ownstuff saving chroots of failures (x86_64)",
"category": "Building",
"actions": [
{
"actions": ["reload-dbs", "update-chroot"],
"concurrent": true
},
"prepare-ownstuff",
"conduct-build-save-chroots-of-failures",
"sync-ownstuff"
]
},
"update-ownstuff": {
"name": "Check ownstuff for updates",
"desc": "Check ownstuff for updates (x86_64)",
"category": "Repo management",
"actions": [
"reload-dbs",
{
"actions": [
"update-ownstuff"
],
"concurrent": true
}
]
},
"sync-ownstuff": {
"name": "Sync ownstuff",
"desc": "Syncs f3l mirror of ownstuff{,-testing,-staging}",
"category": "Repo management",
"actions": ["sync-ownstuff"]
},
"update-chroot": {
"name": "Update chroots",
"desc": "Updates the chroots (so far covers only x86_64)",
"category": "Repo management",
"actions": ["update-chroot"]
}
}
}

1
srv/doc/server-config-example.conf
View File

@ -15,6 +15,7 @@ working_directory = /var/lib/buildservice/building
local_pkgbuilds_dir = /run/media/devel/projects/pkgbuilds
ignore_local_pkgbuilds_regex = ^(some-package-name|yet-another-package-name)$
chroot_dir = /directory/with/enough/space/for/chroot
#presets = presets.json
#ccache_dir = /the/ccache/directory
#package_cache_dir = /var/cache/pacman/pkg