v0.15.0

Genji v0.15.0 is out and comes a few big architectural changes 🚀

Goodbye Bolt and Badger, hello Pebble

Historically, Genji started as a layer on top of the amazing Bolt and Badger.
As features were being added, Genji is pretending to become more and more like a full featured SQL database, and thus needed more flexibility than what Bolt and Badger could offer.

Things like custom key comparators (to control how keys are organized in the tree), snapshots, concurrent read/writes (in Bolt only), were missing greatly. Both came with their own transaction mechanism which were a bit limiting for Genji (limited number of writes per transaction, no support for transaction promotion, etc.).
Also, having to maintain both data layers came with lots of challenges that we believe wasn't worth it.

That's why we decided to use a single backend: Pebble.

Pebble comes with lots of nice features, here are the ones we consider particularly interesting for Genji:

• No transactions (this is a good thing, we have our own now!)
• Batches
• Snapshots
• Can also be used in-memory only
• Full control on how the tree is organized

New transaction mechanism

Pebble doesn't come with its own transactions mechanism, so we had to implement our own, which comes will lots of advantages:

• Reads don't block writes, writes don't block reads: Read transactions always operate on the latest snapshot and don't require any locking. Write transactions use small in-memory batches which are atomically persisted alongside a rollback segment. The rollback segment contains undo changes to apply if the transaction is rolled back or if the database crashes.
• Fully serializable transactions: Genji supports multiple readers at a time and a single writer. This doesn't change from the previous releases, however we believe having a single writer should be more than enough in a embedded setting.
• Arbitrarily large transactions: Write transactions can contain a virtually unlimited number of changes while guaranteeing they won't be taking more than 10MB of ram (hardcoded value for this release). This allows to run Genji on small devices without worrying about the ram usage.
• Super fast writes: This release doesn't come with any official benchmark but our own biased ones showed that when inserting large documents, writes are twice as fast as the amazing SQLite (with WAL mode). Again, this is on a very specific benchmark, on a very specific device. But it is promising.

New CREATE TABLE API

The CREATE TABLE statement introduces new rules for table creation that gives much more control.

By default, a table now only inserts fields that have been declared. Any other field will be ignored and not inserted.

CREATE TABLE foo (a INTEGER, b TEXT);
INSERT INTO foo (a, b, c) VALUES (1, 'hello', 'not inserted');
INSERT INTO foo VALUES {a: 2, b: 'bye', c: 'not inserted'};
SELECT * FROM foo;

{
  "a": 1,
  "b": "hello"
}
{
  "a": 2,
  "b": "bye"
}

If you want to insert any field, and only care about declaring a few ones, you can use the ... notation.

CREATE TABLE foo (a INTEGER, b TEXT, ...);
INSERT INTO foo (a, b, c) VALUES (1, 'hello', 'inserted');
SELECT * FROM foo;

{
  "a": 1,
  "b": "hello"
  "c": "inserted"
}

Nested fields are now declared like this:

CREATE TABLE user (
    name TEXT,
    address (
        number INTEGER,
        street TEXT,
        zipcode TEXT NOT NULL
    )
);
INSERT INTO user (name, address) VALUES (
    "Ilias",
    {
        number: 10,
        street: "rue de la paix",
        zipcode: "12345"
    }
);

Drop support for WASM and TinyGo

Historically, Genji has invested a lot of efforts in supporting TinyGo in the hopes of being compiled in WASM and being used in the browser.
However, because of the lack of support for certain packages of the standard library (like reflect, fmt, encoding/jsom, etc.), it was very difficult for us to keep maintaining compatibility across different releases.
Even when we managed to compile, TinyGo would produce very large files (> 3MB), which defeats the purpose of using it.
We may reconsider this in the future as the support for Wasm in TinyGo becomes more mature.

Breaking changes and data format stability

⚠️ This release introduces a big refactoring and thus is not compatible with previous versions.
⚠️ It also only support Go 1.18+ as it relies on the new generics feature introduced in Go 1.18.

However, the database format is almost stable and future releases should not introduce any breaking changes.
If that happens, we will over communicate and provide ways to make it transparent as much as possible.

Our goal is to reach v1 very soon 🚀

Other notable changes

• index: treat all NULL values differently in UNIQUE indexes by @asdine in 2d4df65
• index: use index with BETWEEN operator by @asdine in a304e80
• index: prevent index creation on undeclared paths by @asdine in 3f32bcb
• cli: fix genji restore command with certains whitespace characters by @tzzed in #439
• cli: add .timer shell command by @tzzed in #445
• cli: ensure the inserts are run in the same transaction in genji insert by @tzzed in b4606f7
• sql: fix UPDATE behavior with primary keys by @asdine in f81f3f1
• sql: support params in LIMIT and OFFSET by @asdine in 6178d1a

Full Changelog: v0.14.0...v0.15.0