Martchus
/
arch-repo-manager
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
286 Commits 5 Branches 4 Tags 27 MiB
Commit Graph

106 Commits

Author SHA1 Message Date
Martchus 55c7c62c7c Speed up returning build actions table by avoiding deserializing whole obj 2022-05-31 20:54:02 +02:00
Martchus 645bb0fe54 Use arch-specific sub-directory within cache dir when conducting build 2022-05-31 20:27:42 +02:00
Martchus f2902f1d08 Apply clang-format 2022-05-31 20:13:21 +02:00
Martchus ecebf9fbde Fix showing arch of packages after optimization 
* Move the binary package arch truly into the base package
 2022-05-31 20:13:15 +02:00
Martchus 68d67f543f Refactor to optimize deserialization of base-data 
* Break backwards compatibility
* Allow to deserialize only base-data of packages and build
  actions to potentially speed up showing tables
* Speed up package search in many cases by only deserializing
  base-data (unless details are actually wanted)
 2022-05-31 19:04:57 +02:00
Martchus 40cfe9883c Ensure there are no more leftovers in "clean-repository-" DBs 2022-05-15 00:07:39 +02:00
Martchus 5a6d68b318 Fix using files DB in CleanRepository 2022-05-07 01:59:14 +02:00
Martchus c2d5e2f749 Use DB path when checking timestamps in repo cleanup 
Otherwise it can break if files DBs are used.
 2022-05-02 22:23:35 +02:00
Martchus 26caa78956 Reduce memory usage when loading packages 
* Avoid creation of map with all archive contents; instead parse packages
  while walking though the archive
* Avoid instantiation of std::string in come cases (using std::string_view)
* Reuse libarchive's archive entry when walking though archive
* Use visitor-based database parser in all places to avoid intermediate big
  array with all package objects
 2022-04-20 22:47:10 +02:00
Martchus 1f3f0b0df1 Avoid high memory usage when reloading database 
Especially when enabling files DBs it is quite problematic to store all
package objects of a database within one big array. This change avoids the
array and instead adds the packages directly. The disadvantage is that
clearing the database isn't as simple anymore.
 2022-04-19 00:23:46 +02:00
Martchus 08421fa719 Don't rely on local pkg dir when reloading library deps 
Since 11f1849fde the cache dir is also set as local pkg dir
so we cannot rely on packages really existing there.
 2022-04-07 21:35:05 +02:00
Martchus d5e2b5489d Provide filename for artefact downloads 2022-03-15 23:22:34 +01:00
Martchus 69a81f7583 Remove unused function declaration 2022-03-15 22:27:50 +01:00
Martchus 83c7426fce Allow specifying output path for license info 2022-03-15 22:23:34 +01:00
Martchus 13cccd3bc9 Improve creating license info 2022-03-15 01:10:15 +01:00
Martchus f87829ba85 Save state after reloading configuration 2022-03-15 01:07:51 +01:00
Martchus 59cd593a3b Display template name 2022-03-12 22:46:42 +01:00
Martchus 70a98ffedc Make name of build action to execute custom command shorter 2022-03-12 21:37:59 +01:00
Martchus 55f081153e Add command to delete invalid build actions from DB 2022-03-07 23:36:48 +01:00
Martchus 7ade757c8d Avoid locking whole config when updating DBs 
* Only lock the config for writing the reloading the config file
* Make sure all write operations to the database acquire an "update mutex"
  to ensure only one write operation happens at a time
* Do *not* acquire any additional locks when reading from a database as it
  should be safe to do so even when a write operation happens because
    * LMDB read and write transactions can happen at the same time
    * The package cache has its own mutex anyways
    * Write ops to the package cache try to lock the "update mutex" to
      prevent writing "old" data to the cache during updates
* Make "lastUpdate" atomic to avoid locking the config when accessing it
 2022-03-05 18:55:52 +01:00
Martchus 9b6be51dd3 Allow reloading library dependencies of specific packages 2022-03-04 21:51:25 +01:00
Martchus 4293eb92bf Ignore libs for cross-compilation when checking any packages for binaries 
Not really generic yet (e.g. doesn't cover `aarch64-linux-gnu-glibc`) but
good enough for now.
 2022-03-04 21:50:56 +01:00
Martchus c633eea607 Add any-packages containing binaries as problem when checking for errors 2022-03-03 23:08:14 +01:00
Martchus c2e7f40767 Fix typos 2022-03-01 00:43:06 +01:00
Martchus 2071327a71 Fix building packages using signed git repositories as source 2022-02-25 23:09:16 +01:00
Martchus ff7f039519 Ensure problems of repos with different archs but the same name aren't mixed up 2022-02-22 22:50:51 +01:00
Martchus 382169ab6e Fix varoius build action related problems 
* Fix crashes in some situations
* Fix aborting reloading library dependencies
 2022-02-22 22:49:40 +01:00
Martchus 5c62014b3b Fix variable name in fallback code when BOOST_ASIO_HAS_FILE 2022-02-22 20:22:00 +01:00
Martchus 839a4b0831 Fix wrong variable name in `buildactionlivestreaming.cpp` 2022-02-22 00:05:35 +01:00
Martchus 8ea2f5fd62 Reintroduce cache file as it is still required for "lastUpdate" for DBs 
Maybe it makes more sense to use LMDB for this as well. On the other hand
it is an easy way of keeping the remaining state persistent.
 2022-02-20 20:17:42 +01:00
Martchus 9c4f0277bd Unlock output lock early 
The following things are read-only or guarded by other mutex.
 2022-02-20 19:51:46 +01:00
Martchus bdbfd42e5e Avoid package object being reused after 3d7310af3 2022-02-20 19:50:44 +01:00
Martchus 334bb1faa9 Use io_uring for async file I/O 2022-02-20 18:44:07 +01:00
Martchus 8115721913 Fix handling error computing db order when reloading dependencies 2022-02-20 00:18:49 +01:00
Martchus 93afb3883d Write build action "output" log to a logfile like for sub-processes 2022-02-20 00:11:23 +01:00
Martchus b3b396df5f Allow ignoring copying errors due to existing files when moving packages 2022-02-16 19:47:25 +01:00
Martchus 557fd1a738 lmdb: Use lmdb to store build actions 2022-02-13 21:04:54 +01:00
Martchus e582d5f406 Remove useless blank line 2022-02-03 22:07:22 +01:00
Martchus 66f59fecb9 Make storing lib deps more efficient and allow for intermediate reads 2022-01-31 21:32:07 +01:00
Martchus 218dfecf56 Avoid loading databases when modification date is <= last update 
* Do HTTP head request first when loading database from mirror to avoid
  downloading the full database all the time
* Use the last modification date of the local database file because with
  the persistent storage even local database reloads became a bit expensive
 2022-01-25 00:13:10 +01:00
Martchus f088e54c61 lmdb: Use package cache when querying packages for dependencies 
* Use package cache when querying packages for dependencies or library
  names
* Avoid manual instantiations of `std::shared_ptr<Package>`
 2022-01-21 20:35:43 +01:00
Martchus e44cbf94e5 lmdb: Enable versioning for objects we'd like to store in the DB 2022-01-18 22:48:34 +01:00
Martchus 2ffa6629c8 lmdb: Use lmdb to store packages and dependency indices 2022-01-18 21:50:53 +01:00
Martchus 0a2c41ee0c Fix shadowing warning 2021-12-07 22:48:22 +01:00
Martchus 567c3aaf56 Support "trunk" subdirectory for using official svntogit repos as source 2021-11-16 15:00:19 +01:00
Martchus 0f7718c934 Locate PKGBUILDs locally also without variant dir 
This makes it possible to checkout
https://github.com/archlinux/svntogit-packages.git and
https://github.com/archlinux/svntogit-community.git locally to build
official packages.
 2021-11-14 23:37:33 +01:00
Martchus 6f652922cd Add lookup for repo-specific config file path 
This could be useful to create another repository with packages that are
built using different compile flags, e.g. optimized for certain processors.
 2021-11-14 22:59:46 +01:00
Martchus 2adc80e5fc Override existing symlinks when adding signatures to repo 2021-11-03 16:32:38 +01:00
Martchus 893a8473ad Override existing symlinks when adding packages to repo 2021-11-03 16:06:16 +01:00
Martchus 755e1958d3 Allow overriding existing files in repository 
When `repo-add` fails but files could be copied before refusing to override
is actually very annoying because one has to delete the files before
restarting the build task.
 2021-10-07 18:12:44 +02:00