steps required to support additional databases

[nkosi23]

Hello!

Since there are talks about supporting SQL Server, I am wondering if as of now Marten has a pluggable system enabling users to add and register support for additional databases?

My idea is to add a provider for the CouchDb database. I feel that since it is a "native" json-based document db, adding support for this shouldn't be too complicated if a pluggable system is already in place.

Now in terms of why: CouchDB is great for document storage (absolutely no migrations scripts are ever needed, documents are stored in json, a real-time continuous change feed available over http for every database, great clustering and multi-master scaleout story etc....) But has zero SQL support so is very limited in terms of aggregation queries. It also lacks the event sourcing tooling available in Marten.

Marten on the other hand works natively with postgres, which enables extremely flexible ways to query data. In a nutshell, my thinking is to use Marten as a sort of Mediator to facilitate a CQRS implementation, so that most writes (namely events) would be routed to CouchDb, while some, most or all model snapshots would be stored in postgresql to ease querying. The goal would basically be to use couchdb for the write side and some or most of the read side, while using postgresql for some or most of the read side.

By default I would configure most snapshots to be stored in CouchDb, while some snapshots would be stored in postgresql where advanced aggregation queries are needed whenever the need arises (which event sourcing enables to do quite easily). This would be a best of both worlds kind of thing.

Another benefit of this approach is that it could potentially also enable MartenDb users to do major version postgresql upgrades more easily and safely at scale as they would only amount to rebuilding model snapshots on a new postgresql instance running the new version. A zero downtime switchover could even be achievable relatively easily thanks to the snapshot daemon that would continually keep the new instance in sync with the latest events ingested by couchdb.

Having worked a lot with NoSQL databases and CouchDb in particular, while the development-time story of marten is great, migrations are still needed when deploying to production and it makes putting together a continuous deployment pipeline more challenging (this is the friendly feedback of a newcomer).

I feel that that Marten may be lacking something equivalent to EntityFramework's modelbuilder class whose key benefit is that it allows developers to have a representation of the state of the schema of the production database without needing to have rights to connect directly to the production database to make diffs against it. This makes letting the application automatically apply migrations on startup way less risky and way more acceptable, while keeping the deployment pipeline simple. I've been scratching my head a lot on Marten's migrations management workflow for the past few weeks but haven't been able to figure out to have something similar to what EF's modelbuilder provides (an other benefit being that it is easy to store and diff in source control). So deploying my application is quite a pain.

Right now, since I was big on using EF, then big on using CouchDb, since using marten this is the first time I need to figure out a deployment process to manually check and apply sql migrations, and I feel this is a huge step backward in my productivity. So in addition to performance and scalability, an additional goal pursued by the couchdb integration would be to keep the number of schema changes on the postgresql side very small.

[jeremydmiller]

Old but still relevant: https://jeremydmiller.com/2018/03/07/marten-2-6-2-7-and-why-we-dont-support-sql-server-yet

No way for you to know this as a newcomer, but we've discussed this ad nauseam since Marten started in 2015, and I still don't believe there's any remotely simple way forward to making Marten pluggable for different database engines as it's using PostgreSQL specific functionality in a helluva lot of places that doesn't necessarily even have equivalents in Sql Server or other engines.

I think there's some real possibility of just the event sourcing support being ported to Sql Server with maybe some simplistic saving & loading documents by their primary key, but that's dependent on a commercial sponsor.

If you really just want Marten to act like EF Core because that's what you're used to, someone has gone pretty far in making "dotnet ef" usable with Marten as is. I don't remember the link for that offhand though.

You can use Marten's schema management to drop out migration deltas. So spin up a temp database container, run your migration files, then have Marten drop out a delta migration file for the difference between the real database and what's configured.

If you want the ModelBuilder approach, there is already a strong typed model as part of the Weasel library if you feel like taking on a code centric approach.

[nkosi23]

Thanks a lot for taking the time to share these insights with me Jeremy!

[nkosi23]

@jeremydmiller sorry, could you please give me pointers regarding how to use the strong typed model provided by Weasel? I went through the docs and even Weasel's source code but couldn't find my way on how to pull together a modelbuilder-like workflow

[jeremydmiller]

Using the Table model is all I meant