Commit Graph

9 Commits

Author SHA1 Message Date
Martchus 26caa78956 Reduce memory usage when loading packages
* Avoid creation of map with all archive contents; instead parse packages
  while walking though the archive
* Avoid instantiation of std::string in come cases (using std::string_view)
* Reuse libarchive's archive entry when walking though archive
* Use visitor-based database parser in all places to avoid intermediate big
  array with all package objects
2022-04-20 22:47:10 +02:00
Martchus 1f3f0b0df1 Avoid high memory usage when reloading database
Especially when enabling files DBs it is quite problematic to store all
package objects of a database within one big array. This change avoids the
array and instead adds the packages directly. The disadvantage is that
clearing the database isn't as simple anymore.
2022-04-19 00:23:46 +02:00
Martchus 7ade757c8d Avoid locking whole config when updating DBs
* Only lock the config for writing the reloading the config file
* Make sure all write operations to the database acquire an "update mutex"
  to ensure only one write operation happens at a time
* Do *not* acquire any additional locks when reading from a database as it
  should be safe to do so even when a write operation happens because
    * LMDB read and write transactions can happen at the same time
    * The package cache has its own mutex anyways
    * Write ops to the package cache try to lock the "update mutex" to
      prevent writing "old" data to the cache during updates
* Make "lastUpdate" atomic to avoid locking the config when accessing it
2022-03-05 18:55:52 +01:00
Martchus 8ea2f5fd62 Reintroduce cache file as it is still required for "lastUpdate" for DBs
Maybe it makes more sense to use LMDB for this as well. On the other hand
it is an easy way of keeping the remaining state persistent.
2022-02-20 20:17:42 +01:00
Martchus 218dfecf56 Avoid loading databases when modification date is <= last update
* Do HTTP head request first when loading database from mirror to avoid
  downloading the full database all the time
* Use the last modification date of the local database file because with
  the persistent storage even local database reloads became a bit expensive
2022-01-25 00:13:10 +01:00
Martchus fa969270c1 Fix warnings 2021-03-22 15:08:41 +01:00
Martchus 0760860c6d Log lock acquisitions/releases 2021-02-25 22:12:56 +01:00
Martchus b7f27cb0a0 Use locks to prevent multiple actions accessing the same db files and chroot dirs 2021-02-22 23:44:06 +01:00
Martchus c1554f4c87 Initial import 2021-02-08 23:53:56 +01:00