Martchus
/
arch-repo-manager
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
249 Commits 5 Branches 4 Tags 27 MiB
Commit Graph

143 Commits

Author SHA1 Message Date
Martchus 218dfecf56 Avoid loading databases when modification date is <= last update 
* Do HTTP head request first when loading database from mirror to avoid
  downloading the full database all the time
* Use the last modification date of the local database file because with
  the persistent storage even local database reloads became a bit expensive
 2022-01-25 00:13:10 +01:00
Martchus 300f74fb10 lmdb: Open LMDB file after restoring cache so DBs added from cache have the storage initialized 2022-01-25 00:01:59 +01:00
Martchus 1c820fc2f6 lmdb: Fix storage initialization when config file cannot be loaded 2022-01-23 19:47:50 +01:00
Martchus f088e54c61 lmdb: Use package cache when querying packages for dependencies 
* Use package cache when querying packages for dependencies or library
  names
* Avoid manual instantiations of `std::shared_ptr<Package>`
 2022-01-21 20:35:43 +01:00
Martchus e44cbf94e5 lmdb: Enable versioning for objects we'd like to store in the DB 2022-01-18 22:48:34 +01:00
Martchus 2ffa6629c8 lmdb: Use lmdb to store packages and dependency indices 2022-01-18 21:50:53 +01:00
Martchus 0a2c41ee0c Fix shadowing warning 2021-12-07 22:48:22 +01:00
Martchus 567c3aaf56 Support "trunk" subdirectory for using official svntogit repos as source 2021-11-16 15:00:19 +01:00
Martchus 0f7718c934 Locate PKGBUILDs locally also without variant dir 
This makes it possible to checkout
https://github.com/archlinux/svntogit-packages.git and
https://github.com/archlinux/svntogit-community.git locally to build
official packages.
 2021-11-14 23:37:33 +01:00
Martchus 6f652922cd Add lookup for repo-specific config file path 
This could be useful to create another repository with packages that are
built using different compile flags, e.g. optimized for certain processors.
 2021-11-14 22:59:46 +01:00
Martchus 2adc80e5fc Override existing symlinks when adding signatures to repo 2021-11-03 16:32:38 +01:00
Martchus 893a8473ad Override existing symlinks when adding packages to repo 2021-11-03 16:06:16 +01:00
Martchus 755e1958d3 Allow overriding existing files in repository 
When `repo-add` fails but files could be copied before refusing to override
is actually very annoying because one has to delete the files before
restarting the build task.
 2021-10-07 18:12:44 +02:00
Martchus 672f3b4e97 Fix staging check for split packages 
* Consider all split packages when skipping an affected package
 2021-09-30 22:23:54 +02:00
Martchus ace478be22 Fix tests when test working directory is on another device 
* Fallback to copying if hard link creation fails due to cross device link
 2021-09-30 22:22:22 +02:00
Martchus a93924bb3a Fix cleanup of signatures 
* Record full path for the deletion of orphaned signatures
* Test for the existence of the signature symlink itself and not the target
  because the symlink target (in the any directory) might not exist anymore
  but the symlink should be removed in any case
 2021-08-29 00:32:19 +02:00
Martchus c68202bdc1 Add test for repository cleanup 2021-08-29 00:29:26 +02:00
Martchus 1cc8a3d6db Add warnings about missing DLLs to the build action's result 2021-08-28 15:06:24 +02:00
Martchus 2ad4a39aea Handle case when a split package overrides the archs from the base 2021-08-28 00:25:09 +02:00
Martchus 95ae827c74 Fix applying relative working directory multiple times 2021-07-31 14:55:49 +02:00
Martchus 1ce736d66f Fix passing the body limit to nested `cachePackages()` calls 2021-07-25 01:25:30 +02:00
Martchus 075f539480 Avoid copying strings when splitting lock names for custom command 2021-07-25 00:41:46 +02:00
Martchus f3ec908bcc Hold lock for named lock table not while acquiring named lock 
Otherwise all other attempts to acquire named locks are blocked. It should
be ok because references are not invalidated when inserting/accessing
elements of an `std::unique_map`. When cleaning locks up elements are
erased, though. Hence an additional cleanup lock is required to prevent
acquiring a named lock which has already been cleaned up.
 2021-07-25 00:40:39 +02:00
Martchus bbc7c3a533 Ignore empty packages within the cache dir 
Those are most likely leftovers from  previous failed download attempts and
can be ignored. (Existing files are overridden.)
 2021-07-17 23:58:37 +02:00
Martchus e5f4057c61 Use BufferSearch from c++utilities 2021-07-16 16:01:23 +02:00
Martchus bc8a0549b2 Log package download limit 2021-07-14 15:27:49 +02:00
Martchus 909f7bfab0 Use default package download size limit if it is set to zero 2021-07-14 15:23:18 +02:00
Martchus e0a913b16c Apply clang-format 2021-07-13 00:41:49 +02:00
Martchus c05a581322 Increase the max. download size for packages and allow configuring it 2021-07-12 15:00:43 +02:00
Martchus a2fbba9713 Allow constructing BadRequest from std::string 2021-07-07 19:46:43 +02:00
Martchus b5ca815e4c Allow excluding packages via regex when reloading library dependencies 
This recently failed because the cuda package exceeded the max. body size.
I suppose it makes generally sense to be able to exclude packages as I
don't need that package anyways.
 2021-07-07 19:14:03 +02:00
Martchus d006c9ce4f Fix error handling when instantiating `std::regex` 2021-07-07 19:09:32 +02:00
Martchus 17c5f50440 Do not exit process if config values are invalid 
This is not a good idea when re-reading the config after startup.
 2021-07-07 19:08:47 +02:00
Martchus 49aebdd8a5 Fix typos found via `codespell --skip .git -w` 2021-07-03 19:59:57 +02:00
Martchus 8b70cc5f8e Clear added/removed provides from previous builds 2021-06-22 00:02:41 +02:00
Martchus f685fddd20 Add `.pid`-equivalent for Boost.Asio's Windows-executor 2021-06-22 00:01:55 +02:00
Martchus 0a55412625 Remove unused #include 2021-06-21 23:58:34 +02:00
Martchus d9203689a0 Fix param for PrepareBuildFlags::ResetChrootSettings 2021-06-04 16:47:24 +02:00
Martchus 3a797bc8b6 Adapt test to fixed behavior of splitString*() to preserve last empty part 2021-05-16 19:43:16 +02:00
Martchus 9359004a10 Allow resetting chroot settings as well when preparing build 2021-05-16 16:49:09 +02:00
Martchus efbcef5e6e Fix error handling when makechrootpkg can not be invoked 2021-05-16 13:49:57 +02:00
Martchus b9c7f05ef1 Add missing include for condition_variable 2021-05-15 21:04:21 +02:00
Martchus 05ff6cd00f Do not set packages in conduct build actions when creating it form task 
Usually this doesn't make sense because we want to build all packages which
were determined in the previous prepare action. When specifying the
packages again in the conduct build action we'd skip packages which are
pulled into the build as dependencies which is normally not desired.
 2021-05-01 23:28:30 +02:00
Martchus 6101eaa6b6 Ensure staging flag is not unset 2021-04-28 22:19:10 +02:00
Martchus a96ba707cd Populate list of packages to be re-built also for removed libraries 2021-04-28 21:58:33 +02:00
Martchus 12050364c8 Handle signature cleanup alongside related package even if package symlink broken 2021-04-27 17:06:47 +02:00
Martchus e4755a283b Prevent running out of memory when signing split packages 
* Launch only up to 4 gpg processes at a time
* Prevent `gpg: signing failed: Nicht genügend Hauptspeicher verfügbar`
 2021-04-17 17:30:18 +02:00
Martchus 56ad086c0c Start the next job only if all start after actions were successful 2021-04-16 14:06:18 +02:00
Martchus 8e904ebfbf Fix starting first build action 
* Prevent endless recursion
* Consider starting parallel sequence also as having started an action to
  prevent starting also actions which should only start after that parallel
  sequence
 2021-04-16 14:05:25 +02:00
Martchus 80c99cb179 Support receiving a body as well when handling chunks individually 2021-04-14 17:56:29 +02:00
Martchus da9ddc2079 Apply clang-format 2021-04-14 17:56:29 +02:00
Martchus e47edcc09e Allow live-streaming via CLI 2021-04-12 14:03:47 +02:00
Martchus 338674a9e6 Fix returning output when no offset present 2021-04-05 21:11:13 +02:00
Martchus 7ab50a121f Split routes.cpp 2021-04-05 20:02:00 +02:00
Martchus b73bee87b5 Add test for posting build actions from pre-defined build task 2021-04-05 19:49:04 +02:00
Martchus ff1e955bde Consider concurrent flag when starting build actions from task 
This allows concurrent build actions within the same task.
 2021-04-05 12:10:42 +02:00
Martchus 7f0ec081c6 Allow specifying build tasks with more complicated build actions 
The new data structures (e.g. concurrent flag) aren't actually used yet.
 2021-04-02 00:14:00 +02:00
Martchus 6f2c6cabc2 cli: Add more build action operations 2021-03-27 21:37:28 +01:00
Martchus bfbde3f957 Refactor client code 2021-03-27 19:55:24 +01:00
Martchus fa969270c1 Fix warnings 2021-03-22 15:08:41 +01:00
Martchus ac7f38e8fa Do not access moved-from object in CleanRepository::run() 2021-03-16 15:36:57 +01:00
Martchus 9581716d8b Delete signature in the src repo when moving package 2021-03-16 15:16:03 +01:00
Martchus 307322ab11 Handle signatures when cleaning up repository 2021-03-15 19:22:28 +01:00
Martchus 1c75e8f957 Allow signing packages manually after the build 
Simply adding `--sign` to the `makepkg` flags doesn't work because it would
require setting up GPG within the chroot environment (of `makechrootpkg`).

When debugging it is anyways annoying that `makepkg` sends the `gpg` output
to `/dev/null`. This way the logs are preserved.
 2021-03-14 21:53:51 +01:00
Martchus 45bf4fa234 Handle signature file when adding/removing/moving/checking/cleaning packages 2021-03-14 01:53:40 +01:00
Martchus 94b39f55dc Lock working copy directory as well (as root directory) 2021-03-06 17:08:04 +01:00
Martchus 29c0741984 Allow registration of multiple data handlers 2021-03-06 16:44:42 +01:00
Martchus cbfa8d8298 Use locks when checking/cleaning repository 2021-03-05 23:11:48 +01:00
Martchus 556a749362 Improve lock-related code in conductbuild.cpp 2021-03-02 18:30:40 +01:00
Martchus 4416922397 Fix clearing locks 2021-03-02 18:30:15 +01:00
Martchus f147dc3179 Fix locks for global resources by implementing a mutex not tied to a thread 
* Can not use a normal mutex because we don't want to tie the resources to
  a specific thread (and instead e.g. to a build action which might not be
  executed by a single thread)
* A semaphore would do that but libstdc++ only supports it as of GCC 11 and
  besides it wouldn't distinguish between shared and exclusive locking
 2021-02-26 01:39:21 +01:00
Martchus 930d449333 Fix handling locks in ConductBuild 2021-02-25 23:20:07 +01:00
Martchus 0760860c6d Log lock acquisitions/releases 2021-02-25 22:12:56 +01:00
Martchus aae5bf81f3 Allow specifying locks when executing custom command 2021-02-23 20:57:07 +01:00
Martchus b7f27cb0a0 Use locks to prevent multiple actions accessing the same db files and chroot dirs 2021-02-22 23:44:06 +01:00
Martchus bc993f1d78 Add locking system so build actions can acquire named locks 2021-02-20 23:25:31 +01:00
Martchus 120fbf993b Fix param for ignoring dependencies 2021-02-20 12:25:08 +01:00
Martchus 046f963f4b Improve error handling when parsing config, fix specifying IP address 2021-02-18 15:21:05 +01:00
Martchus f040418bdd Avoid checking for failed dependencies when processing the last package of batch 2021-02-13 12:54:01 +01:00
Martchus 97001740e2 Do enable staging in the current batch when processing the very first package 
Likely not really relevant because the function should only be called when
processing the 2nd or further packages. However, this way it seems a little
bit less error prone.
 2021-02-13 12:52:54 +01:00
Martchus b556c39cd7 Skip the current package when checking for failed dependencies 
We usually wouldn't expect a package to depend on itself but the different
packages build as part of one split-package might do.
 2021-02-13 12:30:13 +01:00
Martchus 3fe15fb081 Add missing newline when logging failed makechrootpkg invocation 2021-02-13 12:28:42 +01:00
Martchus ab6b940d8e Acknowledge restoring config from cache file correctly 2021-02-09 14:59:40 +01:00
Martchus 5127ca351f Allow ignoring certain dependencies/libraries when checking for problems 2021-02-08 23:54:03 +01:00
Martchus b06a1dfcd2 Allow considering the regular package when checking for updates 2021-02-08 23:54:03 +01:00
Martchus 3894d911d7 Fix posting build action to start it later; add basic test for the route 2021-02-08 23:54:03 +01:00
Martchus 0c8f0b04c2 Experiment with Boost.Beast to use version of async_accept accepting IO ctx 
This would make more sense instead of using the (copy of the) same socket
object all the time.
 2021-02-08 23:54:03 +01:00
Martchus 6b274ccc88 Improve log messages 2021-02-08 23:54:03 +01:00
Martchus 17ab860c8b Don't log all build-action specific output to stderr as well 2021-02-08 23:54:03 +01:00
Martchus cfdf980511 Avoid constructing extra string in ConductBuild::dumpBuildProgress() 2021-02-08 23:54:03 +01:00
Martchus aa356944da Log process IDs of started sub processes 2021-02-08 23:54:02 +01:00
Martchus b647fc79ab Fix parameter of "Keep pkgrel-epoch" option 2021-02-08 23:54:02 +01:00
Martchus c1554f4c87 Initial import 2021-02-08 23:53:56 +01:00