Improve README.md and examples
This commit is contained in:
parent
3a797bc8b6
commit
e1b1f99ecc
65
README.md
65
README.md
|
@ -4,7 +4,17 @@ This **experimental** project contains **inofficial** tools to manage custom Arc
|
||||||
repositories. It is built on top of the official tools provided by the `pacman` and
|
repositories. It is built on top of the official tools provided by the `pacman` and
|
||||||
`devtools` packages.
|
`devtools` packages.
|
||||||
|
|
||||||
So far it consists of:
|
At this point the project is rather raw and there are many things left to implement and
|
||||||
|
to improve (checkout the TODOs section below). Currently builds are exclusively triggered
|
||||||
|
manually (although the API brings the possibility for automation), scalability is limited
|
||||||
|
through the use of an in-memory database. Builds are only conducted on one host.
|
||||||
|
|
||||||
|
On the upside, this project can easily be used to together with other build scripts. It
|
||||||
|
doesn't care if your repository's DB file is updated by another application; just be sure
|
||||||
|
to reload the DB file (see Workflow section below). This should make an easy migration
|
||||||
|
path to use this for maintaining an existing repository.
|
||||||
|
|
||||||
|
So far this project consists of:
|
||||||
|
|
||||||
* libpkg: C++ library to parse Arch Linux packages and databases
|
* libpkg: C++ library to parse Arch Linux packages and databases
|
||||||
* parse pacman config
|
* parse pacman config
|
||||||
|
@ -35,7 +45,19 @@ Further ideas (not implemented yet):
|
||||||
|
|
||||||
## Setup server
|
## Setup server
|
||||||
An example config files can be found in this repository, see the `srv/doc`
|
An example config files can be found in this repository, see the `srv/doc`
|
||||||
directory.
|
directory. The example config is also installed so one can easily get started.
|
||||||
|
|
||||||
|
This is an example for using the package
|
||||||
|
[`arch-repo-manager-git`](https://github.com/Martchus/PKGBUILDs/blob/master/arch-repo-manager/git/PKGBUILD):
|
||||||
|
|
||||||
|
```
|
||||||
|
mkdir -p /etc/buildservice-git
|
||||||
|
cp /usr/share/buildservice-git/skel/server-config-example.conf /etc/buildservice-git/server.conf
|
||||||
|
cp /usr/share/buildservice-git/skel/presets-example.json /etc/buildservice-git/presets.json
|
||||||
|
systemctl enable --now buildservice-git.service
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that the `-git` package is used at this point because there are no releases yet.
|
||||||
|
|
||||||
### Setting up a working directory
|
### Setting up a working directory
|
||||||
The server needs a place to temporarily store PKGBUILDs, cache files and other stuff.
|
The server needs a place to temporarily store PKGBUILDs, cache files and other stuff.
|
||||||
|
@ -181,7 +203,6 @@ keyserver hkp://keys.gnupg.net
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Notes
|
#### Notes
|
||||||
|
|
||||||
* Adding the pacman keyring is actually not very useful because we need to check signatures
|
* Adding the pacman keyring is actually not very useful because we need to check signatures
|
||||||
of any upstream project and not just arch devs.
|
of any upstream project and not just arch devs.
|
||||||
* If "auto-key-retrieve" does not work, use `gpg --recv-key <KEYID>` as a workaround
|
* If "auto-key-retrieve" does not work, use `gpg --recv-key <KEYID>` as a workaround
|
||||||
|
@ -199,10 +220,20 @@ with only slight changes are expected. Currently it is *not* possible to enable/
|
||||||
per package (e.g. via a regex).
|
per package (e.g. via a regex).
|
||||||
|
|
||||||
## Workflow
|
## Workflow
|
||||||
|
So called "build actions" are used to conduct certain tasks, e.g. checking for updates or problems,
|
||||||
|
moving packages from one repository to another, preparing the build and conducting the build. The web
|
||||||
|
UI contains a form to create a new build action and shows the possible options.
|
||||||
|
|
||||||
|
It would be tedious to create the same set of build actions over and over again. Hence it is also
|
||||||
|
possible to create "pre defined tasks". Such a task contains a list of build actions with pre-filled
|
||||||
|
flags and settings. By default, the build actions within a task run one after another but it is also
|
||||||
|
possible to specify that certain build actions can run in parallel. The presets are configured by
|
||||||
|
editing the presets JSON file (e.g. `/etc/buildservice-git/presets.json` in the example config).
|
||||||
|
|
||||||
### General
|
### General
|
||||||
* Use the "Reload database" build action to reload one or more databases after databases have been
|
* So far databases are *not* automatically reloaded. Use the "Reload database" build action to reload
|
||||||
changes by build actions or manually. So far databases are *not* automatically reloaded.
|
one or more databases after databases have been changes by build actions or manually. This build
|
||||||
|
action is similar to `pacman -Sy`.
|
||||||
* Use the "Reload configuration" build action to apply configuration changes without restarting the
|
* Use the "Reload configuration" build action to apply configuration changes without restarting the
|
||||||
service.
|
service.
|
||||||
|
|
||||||
|
@ -286,21 +317,23 @@ per package (e.g. via a regex).
|
||||||
only `pkgrel` changes or when building from VCS sources or when some sources just remain the same).
|
only `pkgrel` changes or when building from VCS sources or when some sources just remain the same).
|
||||||
|
|
||||||
## TODOs and further ideas for improvement
|
## TODOs and further ideas for improvement
|
||||||
[ ] Use persistent database (e.g. lmdb or leveldb)
|
* [ ] Use persistent, non-in-memory database (e.g. lmdb or leveldb) to improve scalability
|
||||||
[ ] More advanced search options
|
* [ ] Allow triggering tasks automatically/periodically
|
||||||
[ ] Allow running tasks automatically/periodically
|
* [ ] Allow to run `makechrootpkg` on a remote host (e.g. via SSH) to work can be spread across multiple hosts
|
||||||
[ ] Refresh build action details automatically while an action is pending
|
* [ ] More advanced search options
|
||||||
[ ] Fix permission error when stopping a process
|
* [ ] Refresh build action details on the web UI automatically while an action is pending
|
||||||
[ ] Keep the possibility for a "soft stop" where the build action would finish the current item
|
* [ ] Fix permission error when stopping a process
|
||||||
[ ] Show statistics like CPU and RAM usage about ongoing build processes
|
* [ ] Keep the possibility for a "soft stop" where the build action would finish the current item
|
||||||
[ ] Stop a build process which doesn't produce output after a certain time
|
* [ ] Show statistics like CPU and RAM usage about ongoing build processes
|
||||||
[ ] Find out why the web service sometimes gets stuck
|
* [ ] Stop a build process which doesn't produce output after a certain time
|
||||||
|
* [ ] Find out why the web service sometimes gets stuck
|
||||||
* Weirdly, restarting the client (browser) helps in these cases
|
* Weirdly, restarting the client (browser) helps in these cases
|
||||||
* Add "stress" test for live-streaming
|
* Add "stress" test for live-streaming
|
||||||
* Start process producing lots of output very fast
|
* Start process producing lots of output very fast
|
||||||
* Let different clients connect and disconnect fast
|
* Let different clients connect and disconnect fast
|
||||||
[ ] Improve test coverage
|
* [ ] Improve test coverage
|
||||||
[ ] Include `xterm.js` properly
|
* [ ] Add fancy graphs for dependencies on the web UI
|
||||||
|
* [ ] Include `xterm.js` properly
|
||||||
|
|
||||||
## Build instructions and dependencies
|
## Build instructions and dependencies
|
||||||
For a PKGBUILD checkout my [PKGBUILDs repository](https://github.com/Martchus/PKGBUILDs).
|
For a PKGBUILD checkout my [PKGBUILDs repository](https://github.com/Martchus/PKGBUILDs).
|
||||||
|
|
|
@ -98,7 +98,7 @@ endif ()
|
||||||
|
|
||||||
# add installs for config file template
|
# add installs for config file template
|
||||||
install(
|
install(
|
||||||
FILES doc/server-config-example.conf doc/nginx-config-example.conf
|
FILES doc/server-config-example.conf doc/presets-example.json doc/nginx-config-example.conf
|
||||||
DESTINATION "${CMAKE_INSTALL_PREFIX}/${META_DATA_DIR}/skel"
|
DESTINATION "${CMAKE_INSTALL_PREFIX}/${META_DATA_DIR}/skel"
|
||||||
COMPONENT config)
|
COMPONENT config)
|
||||||
|
|
||||||
|
|
|
@ -0,0 +1,193 @@
|
||||||
|
{
|
||||||
|
"templates": {
|
||||||
|
"reload-dbs": {
|
||||||
|
"directory": "",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": [],
|
||||||
|
"destinationDbs": [
|
||||||
|
"core",
|
||||||
|
"extra",
|
||||||
|
"community",
|
||||||
|
"multilib",
|
||||||
|
"staging",
|
||||||
|
"testing",
|
||||||
|
"community-staging",
|
||||||
|
"community-testing",
|
||||||
|
"multilib-staging",
|
||||||
|
"multilib-testing",
|
||||||
|
"kde-unstable",
|
||||||
|
"gnome-unstable",
|
||||||
|
"ownstuff",
|
||||||
|
"ownstuff-testing",
|
||||||
|
"ownstuff-staging",
|
||||||
|
"aur"
|
||||||
|
],
|
||||||
|
"settings": {},
|
||||||
|
"flags": 0,
|
||||||
|
"type": 4
|
||||||
|
},
|
||||||
|
"reload-libs": {
|
||||||
|
"directory": "",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": [],
|
||||||
|
"destinationDbs": [
|
||||||
|
"ownstuff",
|
||||||
|
"ownstuff-testing",
|
||||||
|
"ownstuff-staging",
|
||||||
|
],
|
||||||
|
"settings": {},
|
||||||
|
"flags": 0,
|
||||||
|
"type": 5
|
||||||
|
},
|
||||||
|
"update-chroot": {
|
||||||
|
"directory": "chroot-update",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": [],
|
||||||
|
"destinationDbs": [],
|
||||||
|
"settings": {
|
||||||
|
"cmd": "…",
|
||||||
|
"shared-locks": "…"
|
||||||
|
},
|
||||||
|
"flags": 0,
|
||||||
|
"type": 13
|
||||||
|
},
|
||||||
|
"update-ownstuff": {
|
||||||
|
"directory": "",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": ["aur"],
|
||||||
|
"destinationDbs": ["ownstuff"],
|
||||||
|
"settings": {},
|
||||||
|
"flags": 0,
|
||||||
|
"type": 3
|
||||||
|
},
|
||||||
|
"prepare-ownstuff": {
|
||||||
|
"directory": "misc",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": ["ownstuff", "core", "extra", "community", "multilib"],
|
||||||
|
"destinationDbs": ["ownstuff"],
|
||||||
|
"settings": {},
|
||||||
|
"flags": 2,
|
||||||
|
"type": 6
|
||||||
|
},
|
||||||
|
"prepare-ownstuff-testing": {
|
||||||
|
"directory": "misc",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": ["ownstuff-testing", "ownstuff", "core", "extra", "community", "multilib", "testing", "community-testing", "multilib-testing"],
|
||||||
|
"destinationDbs": ["ownstuff-testing"],
|
||||||
|
"settings": {},
|
||||||
|
"flags": 2,
|
||||||
|
"type": 6
|
||||||
|
},
|
||||||
|
"prepare-ownstuff-staging": {
|
||||||
|
"directory": "misc",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": ["ownstuff-staging", "ownstuff-testing", "ownstuff", "core", "extra", "community", "multilib", "staging", "community-staging", "multilib-staging", "testing", "community-testing", "multilib-testing"],
|
||||||
|
"destinationDbs": ["ownstuff-staging"],
|
||||||
|
"settings": {},
|
||||||
|
"flags": 2,
|
||||||
|
"type": 6
|
||||||
|
},
|
||||||
|
"conduct-build": {
|
||||||
|
"directory": "misc",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": [],
|
||||||
|
"destinationDbs": [],
|
||||||
|
"settings": {},
|
||||||
|
"flags": 9,
|
||||||
|
"type": 7
|
||||||
|
},
|
||||||
|
"conduct-build-save-chroots-of-failures": {
|
||||||
|
"directory": "misc",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": [],
|
||||||
|
"destinationDbs": [],
|
||||||
|
"settings": {},
|
||||||
|
"flags": 11,
|
||||||
|
"type": 7
|
||||||
|
},
|
||||||
|
"sync-ownstuff": {
|
||||||
|
"directory": "sync-ownstuff",
|
||||||
|
"packageNames": [],
|
||||||
|
"sourceDbs": [],
|
||||||
|
"destinationDbs": [],
|
||||||
|
"settings": {
|
||||||
|
"cmd": "rsync …"
|
||||||
|
},
|
||||||
|
"flags": 0,
|
||||||
|
"type": 13
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"tasks": {
|
||||||
|
"conduct-build": {
|
||||||
|
"name": "Conduct build",
|
||||||
|
"desc": "Perform build with default settings",
|
||||||
|
"category": "Building",
|
||||||
|
"actions": ["update-chroot", "conduct-build", "sync-ownstuff"]
|
||||||
|
},
|
||||||
|
"conduct-build-save-chroots-of-failures": {
|
||||||
|
"name": "Conduct build saving chroots of failures",
|
||||||
|
"desc": "Perform build with default settings saving chroots of failures",
|
||||||
|
"category": "Building",
|
||||||
|
"actions": ["update-chroot", "conduct-build-save-chroots-of-failures", "sync-ownstuff"]
|
||||||
|
},
|
||||||
|
"prepare-ownstuff": {
|
||||||
|
"name": "Prepare building for ownstuff",
|
||||||
|
"desc": "Prepare building packages for ownstuff (x86_64)",
|
||||||
|
"category": "Building",
|
||||||
|
"actions": ["reload-dbs", "prepare-ownstuff"]
|
||||||
|
},
|
||||||
|
"prepare-ownstuff-testing": {
|
||||||
|
"name": "Prepare building for ownstuff-testing",
|
||||||
|
"desc": "Prepare building packages for ownstuff-testing (x86_64)",
|
||||||
|
"category": "Building",
|
||||||
|
"actions": ["reload-dbs", "prepare-ownstuff-testing"]
|
||||||
|
},
|
||||||
|
"prepare-ownstuff-staging": {
|
||||||
|
"name": "Prepare building for ownstuff-staging",
|
||||||
|
"desc": "Prepare building packages for ownstuff-staging (x86_64)",
|
||||||
|
"category": "Building",
|
||||||
|
"actions": ["reload-dbs", "prepare-ownstuff-staging"]
|
||||||
|
},
|
||||||
|
"build-ownstuff": {
|
||||||
|
"name": "Build for ownstuff",
|
||||||
|
"desc": "Prepare and build packages for ownstuff saving chroots of failures (x86_64)",
|
||||||
|
"category": "Building",
|
||||||
|
"actions": [
|
||||||
|
{
|
||||||
|
"actions": ["reload-dbs", "update-chroot"],
|
||||||
|
"concurrent": true
|
||||||
|
},
|
||||||
|
"prepare-ownstuff",
|
||||||
|
"conduct-build-save-chroots-of-failures",
|
||||||
|
"sync-ownstuff"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"update-ownstuff": {
|
||||||
|
"name": "Check ownstuff for updates",
|
||||||
|
"desc": "Check ownstuff for updates (x86_64)",
|
||||||
|
"category": "Repo management",
|
||||||
|
"actions": [
|
||||||
|
"reload-dbs",
|
||||||
|
{
|
||||||
|
"actions": [
|
||||||
|
"update-ownstuff"
|
||||||
|
],
|
||||||
|
"concurrent": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sync-ownstuff": {
|
||||||
|
"name": "Sync ownstuff",
|
||||||
|
"desc": "Syncs f3l mirror of ownstuff{,-testing,-staging}",
|
||||||
|
"category": "Repo management",
|
||||||
|
"actions": ["sync-ownstuff"]
|
||||||
|
},
|
||||||
|
"update-chroot": {
|
||||||
|
"name": "Update chroots",
|
||||||
|
"desc": "Updates the chroots (so far covers only x86_64)",
|
||||||
|
"category": "Repo management",
|
||||||
|
"actions": ["update-chroot"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
@ -15,6 +15,7 @@ working_directory = /var/lib/buildservice/building
|
||||||
local_pkgbuilds_dir = /run/media/devel/projects/pkgbuilds
|
local_pkgbuilds_dir = /run/media/devel/projects/pkgbuilds
|
||||||
ignore_local_pkgbuilds_regex = ^(some-package-name|yet-another-package-name)$
|
ignore_local_pkgbuilds_regex = ^(some-package-name|yet-another-package-name)$
|
||||||
chroot_dir = /directory/with/enough/space/for/chroot
|
chroot_dir = /directory/with/enough/space/for/chroot
|
||||||
|
#presets = presets.json
|
||||||
#ccache_dir = /the/ccache/directory
|
#ccache_dir = /the/ccache/directory
|
||||||
#package_cache_dir = /var/cache/pacman/pkg
|
#package_cache_dir = /var/cache/pacman/pkg
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue