Martchus
/
arch-repo-manager
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
455 Commits 5 Branches 4 Tags 27 MiB
Commit Graph

128 Commits

Author SHA1 Message Date
Martchus 6965d9b40b Pass package pointer by reference when parsing database file 2022-04-20 23:14:49 +02:00
Martchus 26caa78956 Reduce memory usage when loading packages 
* Avoid creation of map with all archive contents; instead parse packages
  while walking though the archive
* Avoid instantiation of std::string in come cases (using std::string_view)
* Reuse libarchive's archive entry when walking though archive
* Use visitor-based database parser in all places to avoid intermediate big
  array with all package objects
 2022-04-20 22:47:10 +02:00
Martchus 1f3f0b0df1 Avoid high memory usage when reloading database 
Especially when enabling files DBs it is quite problematic to store all
package objects of a database within one big array. This change avoids the
array and instead adds the packages directly. The disadvantage is that
clearing the database isn't as simple anymore.
 2022-04-19 00:23:46 +02:00
Martchus f07ef9f147 Remove disabled code 2022-04-18 19:23:18 +02:00
Martchus edb014011c Make package search more efficient 
* Avoid keeping list of all packages in memory by pushing each package
  directly to the JSON array
* Avoid check for database before pushing each package; do it only before
  reaching the next database
* Remove no longer required functions
 2022-04-17 23:39:25 +02:00
Martchus 39c1261efb Apply clang-format 2022-04-07 23:19:24 +02:00
Martchus 4fc3bf0928 Fix version comparison when epoch is only present in one version 2022-03-29 22:08:20 +02:00
Martchus f5394db6b1 Avoid crash when searching for empty dependency/library 2022-03-20 17:21:41 +01:00
Martchus 13cccd3bc9 Improve creating license info 2022-03-15 01:10:15 +01:00
Martchus af66549d87 Avoid possibly accessing packageCacheDirs when it is empty 2022-03-15 01:08:19 +01:00
Martchus 11f1849fde Keep local pkg paths deduced from pacman config 
It makes the setup easier and shouldn't hurt otherwise.
 2022-03-15 01:07:36 +01:00
Martchus 0e04235d0e Generate meaningful error message if path for "licenses" package not configured 2022-03-14 01:07:26 +01:00
Martchus 30e3d0343e Fix condition in `Package::isPkgInfoFileOrBinary()` 2022-03-10 23:04:18 +01:00
Martchus 6699bfa49e lmdb: Avoid storing empty keys possibly leading to `MDB_BAD_VALSIZE` 
This might prevent errors like:

```
Unable to parse retrieved database file for "community-staging@x86_64": Getting data: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size
```
 2022-03-10 22:45:11 +01:00
Martchus 7814b8a336 Avoid installing static library of lmdb-safe 2022-03-08 00:47:38 +01:00
Martchus 7ade757c8d Avoid locking whole config when updating DBs 
* Only lock the config for writing the reloading the config file
* Make sure all write operations to the database acquire an "update mutex"
  to ensure only one write operation happens at a time
* Do *not* acquire any additional locks when reading from a database as it
  should be safe to do so even when a write operation happens because
    * LMDB read and write transactions can happen at the same time
    * The package cache has its own mutex anyways
    * Write ops to the package cache try to lock the "update mutex" to
      prevent writing "old" data to the cache during updates
* Make "lastUpdate" atomic to avoid locking the config when accessing it
 2022-03-05 18:55:52 +01:00
Martchus 1f3abd5596 Distinguish Android libs from GNU/Linux libs 2022-03-04 21:48:28 +01:00
Martchus 3d3ba1457a Adapt tests 2022-03-04 09:30:19 +01:00
Martchus 8e18b4747f Assume the filename as library name for ELF libraries without soname field 2022-03-03 23:00:16 +01:00
Martchus 6456273951 Fix early return for special handling of `mingw-w64-crt` 2022-03-03 22:43:31 +01:00
Martchus cbfdf2236e Skip DLLs provided by Windows when checking for unresolvable DLLs 2022-03-03 22:40:48 +01:00
Martchus c2e7f40767 Fix typos 2022-03-01 00:43:06 +01:00
Martchus 22cfb4bb29 Use std::string_view when parsing pkg info 2022-02-25 23:08:12 +01:00
Martchus fd35aa0da6 Parse sources and size from pkg info 2022-02-25 23:07:55 +01:00
Martchus 90ff9678fe Limit results returned by the API for better scalability 2022-02-25 00:29:43 +01:00
Martchus 9e15129d9d Ignore null-dereference warning 
It is about
```
return result_type{ storageEntry.id, storageEntry.ref.relatedStorage };
```
but it isn't clear why the compiler thinks there's a problem.
 2022-02-22 22:46:53 +01:00
Martchus bdbfd42e5e Avoid package object being reused after 3d7310af3 2022-02-20 19:50:44 +01:00
Martchus d11ba43bf0 lmdb: Avoid getting packages with empty name, can lead to exception 2022-02-20 19:28:44 +01:00
Martchus a7de520549 Use lazy-deserialization and package cache when searching in package name 2022-02-19 00:11:59 +01:00
Martchus 3d7310af30 lmdb: Get `std::shared_ptr` directly from iterator 2022-02-18 19:23:00 +01:00
Martchus 68c11be6bb Call `addDepsAndProvidesFromOtherPackage` *before* clearing existing packages 2022-02-16 19:54:21 +01:00
Martchus 557fd1a738 lmdb: Use lmdb to store build actions 2022-02-13 21:04:54 +01:00
Martchus 3c4f81bd55 lmdb: Declare use of lmdb on level of lmdb-safe 2022-02-05 21:52:20 +01:00
Martchus ff8b644221 Avoid redundant code in PackageUpdaterPrivate::submit() applying previous commit in all versions 2022-02-04 17:57:11 +01:00
Martchus e5a3b5f311 Fix accidentally removing dependency from index when already provided by existing package 2022-02-03 22:10:22 +01:00
Martchus 90379678ad Improve error handling when opening archive 
* Treat an empty archive buffer/file as error as this should never happen
  and it is better to fail early than replacing existing data with nothing
* Query error message also when opening archive from file
 2022-02-02 21:16:56 +01:00
Martchus d1d65dac80 Use `CacheRef(storage, entry)` c'tor as it is generally preferred 
Passing any string here is more likely a mistake, see previous commit
 2022-02-01 22:59:09 +01:00
Martchus 2a256e6ac1 Fix using dangling ref of package name as hash key 2022-02-01 22:57:13 +01:00
Martchus 3f9f5eae3f Avoid exceeding txn limit when pulling deps by avoiding recursion 2022-02-01 21:51:19 +01:00
Martchus 82eaae5d36 Remove unused object 2022-01-31 21:39:57 +01:00
Martchus 66f59fecb9 Make storing lib deps more efficient and allow for intermediate reads 2022-01-31 21:32:07 +01:00
Martchus 162f56c5e3 Fix passing around different versions of PackageSpec 
The way this is passed has gotten a bit inconsistent; let's just make it
compatible both ways.
 2022-01-31 21:32:07 +01:00
Martchus 92f83fadb6 Fix/simplify move c'tor/assignment of Database 2022-01-31 20:48:02 +01:00
Martchus eafa797262 lmdb: Avoid installing lmdb-safe when building it as part of this project 
* Build lmdb-safe always as static library
* Disable install targets of lmdb-safe
 2022-01-30 21:11:39 +01:00
Martchus ecf25ca260 lmdb: Build lmdb-safe as own CMake project/library 2022-01-30 01:28:03 +01:00
Martchus cd98c39c54 lmdb: Test clearing all packages 2022-01-26 00:47:12 +01:00
Martchus 0e61a726e1 Set `lastUpdate` to zero when clearing packages 
Since reloading packages is now avoided clearing packages should not itself
count as update as it would prevent loading packages again.
 2022-01-26 00:46:36 +01:00
Martchus 66d3e7b301 lmdb: Remove unused function 2022-01-26 00:44:07 +01:00
Martchus 45922b47ec lmdb: Allow configuring cache limit 2022-01-26 00:41:53 +01:00
Martchus 218dfecf56 Avoid loading databases when modification date is <= last update 
* Do HTTP head request first when loading database from mirror to avoid
  downloading the full database all the time
* Use the last modification date of the local database file because with
  the persistent storage even local database reloads became a bit expensive
 2022-01-25 00:13:10 +01:00
Martchus fe09463b0a lmdb: Fix replacing existing packages when updating package via cache 2022-01-25 00:04:25 +01:00
Martchus 26f787fb42 lmdb: Do not make lmdb a public dependency as it is only used within libpkg 2022-01-23 19:47:28 +01:00
Martchus f088e54c61 lmdb: Use package cache when querying packages for dependencies 
* Use package cache when querying packages for dependencies or library
  names
* Avoid manual instantiations of `std::shared_ptr<Package>`
 2022-01-21 20:35:43 +01:00
Martchus fcdd4fa7b3 lmdb: Fix warning about type conversion 2022-01-21 20:29:52 +01:00
Martchus 9af3e4783f lmdb: Remove obsolete code 2022-01-21 20:29:25 +01:00
Martchus f73753792c lmdb: Use caching when finding package via ID 2022-01-20 23:33:02 +01:00
Martchus 231571f2d1 lmdb: Generalize caching to be able to use it for other types than packages 2022-01-19 23:33:07 +01:00
Martchus a8afccf081 lmdb: Set limit for at least 6 DBs (requiring each 10 LMDB databases) 
This makes setting up tests easier.
 2022-01-18 22:54:49 +01:00
Martchus e44cbf94e5 lmdb: Enable versioning for objects we'd like to store in the DB 2022-01-18 22:48:34 +01:00
Martchus fc32f790fb lmdb: Use namespace and have reflective-rapidjson code in its own header 2022-01-18 22:25:28 +01:00
Martchus f4774cac56 lmdb: Remove obsolete code 2022-01-18 22:11:54 +01:00
Martchus 2ffa6629c8 lmdb: Use lmdb to store packages and dependency indices 2022-01-18 21:50:53 +01:00
Martchus 5ad89d964f lmdb: Exclude lmdb-safe from tidying 2021-12-07 22:48:42 +01:00
Martchus c4a32feb48 lmdb: Configure project to use lmdb and lmdb-safe 2021-12-07 22:48:42 +01:00
Martchus 1cc8a3d6db Add warnings about missing DLLs to the build action's result 2021-08-28 15:06:24 +02:00
Martchus 2ad4a39aea Handle case when a split package overrides the archs from the base 2021-08-28 00:25:09 +02:00
Martchus e0a913b16c Apply clang-format 2021-07-13 00:41:49 +02:00
Martchus c10df26871 Adapt to versioning changes in reflective rapidjson 2021-07-13 00:41:25 +02:00
Martchus 5ec527bebb Fix typo in test suite consistently 2021-07-07 20:13:22 +02:00
Martchus 49aebdd8a5 Fix typos found via `codespell --skip .git -w` 2021-07-03 19:59:57 +02:00
Martchus fa969270c1 Fix warnings 2021-03-22 15:08:41 +01:00
Martchus bc993f1d78 Add locking system so build actions can acquire named locks 2021-02-20 23:25:31 +01:00
Martchus 4ac9e729d0 Exclude current db from order computation when detecting unresolved packages 2021-02-09 11:59:23 +01:00
Martchus 7a8601010e Fix dependency lookup in Database::replacePackages() 2021-02-08 23:54:03 +01:00
Martchus 5127ca351f Allow ignoring certain dependencies/libraries when checking for problems 2021-02-08 23:54:03 +01:00
Martchus 6a9589b4a1 Fix dependency matching if pkgrel contains a dot 2021-02-08 23:54:03 +01:00
Martchus b06a1dfcd2 Allow considering the regular package when checking for updates 2021-02-08 23:54:03 +01:00
Martchus c1554f4c87 Initial import 2021-02-08 23:53:56 +01:00