lmdb-safe/README.md

# lmdb-safe
A safe modern & performant C++ wrapper of LMDB.  
Requires C++17, or C++11 + Boost.

[LMDB](http://www.lmdb.tech/doc/index.html) is an outrageously fast
key/value store with semantics that make it highly interesting for many
applications.  Of specific note, besides speed, is the full support for
transactions and good read/write concurrency.  LMDB is also famed for its
robustness..  **when used correctly**.

The design of LMDB is elegant and simple, which aids both the performance
and stability. The downside of this elegant design is a [nontrivial set of
rules](http://www.lmdb.tech/doc/starting.html)
that [need to be followed](http://www.lmdb.tech/doc/group__mdb.html) to not break things. In other words, LMDB delivers
great things but only if you use it exactly right. This is by [conscious
design](https://twitter.com/hyc_symas/status/1056168832606392320). 

Among the things to keep in mind when using LMDB natively:

 * Never open a database file more than once anywhere in your process
 * Never open more than one transaction within a thread
   * .. unless they are all read-only and have MDB_NOTLS set
 * When opening a named database, no other threads may do that at the same time
 * Cursors within RO transactions need freeing, but cursors within RW
 transactions must not be freed. 
 * A new transaction may indicate the database has grown, and you need to
   restart the transaction then.

Breaking these rules may cause no immediate errors, but can lead to silent
data corruption, missing updates, or random crashes. Again, this is not an
actual bug in LMDB, it means that LMDB expects you to use it according to
its exact rules. And who are we to disagree?

The `lmdb-safe` library aims to deliver the full LMDB performance while
programmatically making sure the LMDB semantics are adhered to, with very
limited overhead.

Most common LMDB functionality is wrapped within this library but the native
MDB handles are all available should you want to use functionality we did
not (yet) cater for.

In addition, on top of `lmdb-safe`, a type-safe ["Object Relational
Mapping"](https://en.wikipedia.org/wiki/Object-relational_mapping) interface
is also available. This auto-generates indexes and allows for the insertion,
deletion and iteration of objects. 

# Status
Fresh. If using this tiny library, be aware things might change
rapidly. To use, add `lmdb-safe.cc` and `lmdb-safe.hh` to your project. In
addition, add `lmdb-typed.hh` to use the ORM.

# Philosophy
This library tries to not restrict your use of LMDB, nor make it slower,
except on operations that should be rare. The native LMDB handles
(Environment, DBI, Transactions & Cursors) are all available for your direct
use if need be.

When using `lmdb-safe`, errors "that should never happen" are turned into
exceptions. An error that merely indicates that a key can not be found is
passed on as a regular LMDB error code.

# Example
The following example has no overhead compared to native LMDB, but already
exhibits several ways in which lmdb-safe automates LMDB constraints:
```
  auto env = getMDBEnv("./database", 0, 0600);
  auto dbi = env->openDB("example", MDB_CREATE);
  auto txn = env->getRWTransaction();
```

The first line requests an LMDB environment for a database hosted in
`./database`.  **Within LMDB, it is not allowed to open a database file more
than once**, not even from other threads, not even when using a different
LMDB handle.  `getMDBEnv` keeps a registry of LMDB environments, keyed to
the exact inode & flags.  If another part of your process requests access to
the same inode, it will get the same environment. `MDBEnv` is threadsafe.

On the second line, a database is opened within our environment. The
semantics of opening or creating a database within LMDB are tricky. With
some loss of generality, `MDBEnv::openDB` will create a transaction for you
to open the database, and close it too. Most of the time this is what you
want. It is also possible to open a database within a transaction manually.

The third line opens a read/write transaction using the Resource Acquisition
Is Initialization (RAII) technique. If `txn` goes out of scope, the
transaction is aborted automatically. To commit or abort, use `commit()` or
`abort()`, after which going out of scope has no further effect.

```
  txn->put(dbi, "lmdb", "great");

  string_view data;
  if(!txn->get(dbi, "lmdb", data)) {
    cout<< "Within RW transaction, found that lmdb = " << data <<endl;
  }
  else
    cout<<"Found nothing" << endl;

  txn->commit();
```

LMDB is so fast because it does not copy data unless it really needs to.
Memory bandwidth is a huge determinant of performance on modern CPUs. This
wrapper agrees, and using modern C++ makes it possible to seamlessly use
'views' on data without copying them. Using these techniques, the call to
`txn.put()` sets the "lmdb" string to "great", without making additional
copies. 

We employ the same technique to request the value of "lmdb", which is made
available to us as a read-only view, straight onto the memory mapped data on
disk. 

In the final line, we commit the transaction, after which it also becomes
available for other threads and processes. 

A slightly expanded version of this code can be found in
[basic-example.cc](basic-example.cc).

# Input and output of values
The basic data unit of LMDB is `MDB_val` which describes a slab of memory.
Within LMDB, `MDB_val` is used for both input and output. For safety
purposes, in this library we split this up into `MDBInValue` and
`MDBOutValue`. Once split, we can add some very convenient semantics to these
classes.

For example, to store `double` values for 64 bit IDs:

```
  auto txn = env->getRWTransaction();
  uint64_t id=12345678901;
  double score=3.14159;
  txn->put(dbi, id, score);
  txn->commit();
```

Behind the scenes, the `id` and `score` values are wrapped by `MDBInVal`
which converts these values into byte strings. To retrieve these values
works similarly:

```
  auto txn = env->getRWTransaction();
  uint64_t id=12345678901;
  MDBOutValue val;

  txn->get(dbi, id, val);

  cout << "Score: " << val.get<double>() << "\n";
```

Note that on retrieval, we have to specify the type of the value stored.
This allows the conversion back from a byte string into the native type.
`MDBOutValue` also tests if the length of the data matches the type.

## Details
The automatic conversion to and from the `MDBVal`s is implemented strictly
for:

 * Integer and floating point types
 * std::string
 * std::string_view

However, if you explicitly ask for it, it is also possible to serialize
`struct`s:

```
struct Coordinate
{
	double x,y;
};

C c{12.0, 13.0};

txn->put(dbi, MDBInVal::fromStruct(c), 12.0);

MDBOutVal res;
txn->get(dbi, MDBInVal::fromStruct(c), res);

auto c1 = res.get_struct<Coordinate>();
```

# Cursors, transactions
This example shows how to use cursors and how to mix `lmdb-safe` with direct
calls to mdb.

```
  auto env = getMDBEnv("./database", 0, 0600);
  auto dbi = env->openDB("huge", MDB_CREATE);
  auto txn = env->getRWTransaction();

  unsigned int limit=20000000;
```

This is the usual opening sequence.

```
  auto cursor=txn->getCursor(dbi);
  MDBOutVal key, data;
  int count=0;
  cout<<"Counting records.. "; cout.flush();
  while(!cursor.get(key, data, count ? MDB_NEXT : MDB_FIRST)) {
    count++;
  }
  cout<<"Have "<<count<<"!"<<endl;
```

This describes how we generate a cursor for the `huge` database and iterate
over it to count the number of keys in there. We pass two LMDB native
`MDB_val` structs to the cursor `get` function. These do not get copies of
all the millions of potential keys in the `huge` database - they only
contain pointers to that data. Because of this, we can count 20 million
records in under a second (!).
  
```
  cout<<"Clearing records.. "; cout.flush();
  mdb_drop(*txn, dbi, 0); // clear records
  cout<<"Done!"<<endl;
```

Here we drop al keys from the database, which too happens nearly
instantaneously. Note that we pass our `txn` (which is a class) to the
native `mdb_drop` function which we did not wrap. This is possible because
`txn` converts to an `MDB_env*` if needed.

```
  cout << "Adding "<<limit<<" values  .. "; cout.flush();
  for(unsigned int n = 0 ; n < limit; ++n) {
    txn->put(dbi, n, n);
  }
  cout <<"Done!"<<endl;
  cout <<"Calling commit.. "; cout.flush();
  txn->commit();
  cout<<"Done!"<<endl;
```

Here we add 20 million value & then commit the `mdb_drop` and the 20 million
puts.  All this happened in less than 20 seconds.

Had we created our database with the `MDB_INTEGERKEY` option and added the
`MDB_APPEND` flag to `txn.put`, the whole process would have taken around 5
seconds.

# lmdb-typed
The `lmdb-safe` interface may be safe in one sense, but it is still a
key-value store, allowing the user to store any key and any value.
Frequently we have specific needs: to store objects and find them using
different keys. Doing so manually is cumbersome and error-prone, as all
indexes (for rapid retrieval) need to be carefully maintained by hand.

Inspired by Boost MultiIndex, `lmdb-typed` builds on `lmdb-safe` to create,
populate and use indexes for rapidly retrieving objects. As an example,
let's say we want to store the following struct:

```
struct DNSResourceRecord
{
  string qname;          // index
  uint16_t qtype{0};
  uint32_t domain_id{0}; // index
  string content;
  uint32_t ttl{0};
  string ordername;      // index
  bool auth{true};
};
```

And we want to do so based on the `qname`, `domain_id` or `ordername`
fields. First, we have to make sure DNSResourceRecord can be (de)serialized
to/from a string. For that you need to implement `serToString(const T& t)` and
`serFromString(const string_view& str, T& ret)` like it is shown in
`lmdb-reflective.hh` which uses
[Reflective RapidJSON](https://github.com/Martchus/reflective-rapidjson)'s
binary (de)serializer to ease that task. You can also use Boost.Serialization.
It might also be helpful to utilize Boost.Hana, e.g. in combination with the
previously mentioned libraries.

Next up, we need to define our "Object Relational Mapper":

```
  TypedDBI<DNSResourceRecord, 
           index_on<DNSResourceRecord, string,   &DNSResourceRecord::qname>,
           index_on<DNSResourceRecord, uint32_t, &DNSResourceRecord::domain_id>,
           index_on<DNSResourceRecord, string,   &DNSResourceRecord::ordername>
           > tdbi(getMDBEnv("./typed.lmdb", MDB_NOSUBDIR, 0600), "records");

```
This defines that we create a database called `records` in the file
`./typed.lmdb`. We also state that this database stores `DNSResourceRecord`
objects, and that we want three indexes. Note that this syntax is reasonable
similar to that used by Boost::MultiIndex.

Next up, we can insert some objects:

```
auto txn = tdbi.getRWTransaction();
DNSResourceRecord rr{"www.powerdns.com", 1, domain_id, "1.2.3.4", 0, "www"};
// populate rr
auto id = txn.insert(rr);
txn.commit();
```

Internally, the opening of `tdbi` above created four databases: `records`,
`records_0`, `records_1` and `records_2`. On insert, a serialized form of
`rr` was stored in the `records` table, with the key containing the
(assigned) id value.

In addition, in `records_1`, the qname was added as key, with the `id` field
as value. And similarly for `domain_id` and `ordername`. So the indexes all
point to the id field, which we can find in the `records` database.

To retrieve, we can use any of the indexes:

```
auto txn = tdbi.getROTransaction(); 
DNSResourceRecord rr;
txn.get(id, rr);
txn.get<0>("www.powerdns.com", rr);
txn.get<1>(domain_id, rr);
txn.get<2>("www", rr);
```

As long as we inserted only the one `DNSResourceRecord` from above, all four
`get` calls find the same `rr`.

In the more interesting case where we inserted more DNS records, we could
iterate over all items with `domain_id = 4` as follows:

```
for(auto [iter, end] = txn.equal_range<1>(4): iter != end; ++iter) {
	cout << iter->qname << "\n";
}
```

To delete an item, use `txn.del(12)`, which will remove the record with id
12 from the main database and also from all the indexes.