Last Push to V4 Release

[jeremydmiller]

Just to have an open forum to talk through what's left to do for V4, and what we might cut from scope to V4.1 so we can get this thing out the door. I (Jeremy) am writing the first shot at this just to facilitate conversation. With as long as V4 development has gone on, how long it was delayed starting in the first place, and me running out of steam a little bit, I think we're approaching the time to triage remaining scope and only play things that are either breaking changes or really important like the new website. I also think we (me) need to leave some energy to be writing blog posts and sample applications at the end of this to promote V4 to make all the work worth it.

The open issues left for V4 are here.

After going through the issues, here's how I think I'd break things down:

Documentation

I think this is the biggest area of remaining effort, and probably what's most important

• Definitely in -- the move to using VitePress for the documentation and getting a new, spiffier layout and styling. I'd say this is Rebrand look & feel of the Marten documentation #1401
• In: Create a page with links to the articles about Marten #1395, Create a page with links to other projects using Marten #1396, Create a page with recommendations from our users #1398
• Add a sample of an aggregated projection across streams to the docs #521 will just be the new documentation on the V4 ViewProjection
• V4 Documentation Updates #1555 is just a reminder of things that need to be doc'd

Event Store

Discuss

• Mechanism to Import Events #1435 -- what does this really mean? could fall to >4.0
• #1611 -- seems important, folks ask for it, might require breaking changes
• Event schema versioning and upcasting #1613 -- need more investigation. Jeremy thinks this could be added later without breaking changes
• Recipe for easy testing of event projections #1615 -- I think this is more about documentation than anything new, but we need to explore it anyway
• Improved transaction and concurrency prevention for appending events #1665 -- Not huge, kinda need to do for backward compatibility
• Re-evaluate the AggregateType on Stream #1671 -- eliminate the aggregate type definition in StartStream<T>()? Doesn't do anything today. Just a marker that isn't paid attention to otherwise. Breaking change
• Re-evaluate the Marten CLI "projection" command #1677 -- I want this personally, but I wanna make some Oakton changes first
• Extra validation and edge case combinations in the Event Store + Aggregated Projections #1689 -- I'm not 100% sure what there is to do, or if we should. Smells like edge cases users could do w/ custom projections
• Projection Shard Work Distribution in the Async Daemon #1717 -- sharding the projection async work. I think this would be hugely valuable, but a bridge too far

Do for V4

• Event Metadata should be in. Event Metadata #780. The only thing I'd add here is to say I'd still like to a do a sample app before v4 off to the side with Jasper + Marten that supports the open telemetry spec from HTTP endpoints & messaging endpoints with Marten activity being logged as part of that. I think that would help prove out the logging and metadata for Marten v4
• Partitioning support for Event Store #770, partitioning support for the event & stream tables
• Archiving Event Streams #1612 -- Archiving event streams. Seems relatively small the way Oskar, Babu, and Jeremy discussed doing it. Big advantage for scalability
• It's not possible to use Task in the class name in v4 #1647 -- Lamar bug
• Self-Aggregation does not work if the aggregate is an inner class #1679 -- Lamar bug
• Async Daemon Leader Election #1718 -- at least for the hot/cold failover mode. It's doable w/ advisory locks. Gets us to where folks can incorporate the async daemon better than they do today in app clusters
• Inline projection work is getting cut off in V4 with a large number of events #1723 -- might be a bug, might not be an issue. Just needs investigation
• Poison Message detection for the async projections #1731 -- we don't have this in V3, but it seems important

Push off to later

• Add Options property to IDocumentStore interface #1230. If this is just a copy now of EventStoreDb's RSS subscription, I'm not enthusiastic about taking this on in Marten core
• V4 projections and DI containers #1670 -- I vote later
• New Linq selector to apply a live aggregation against any IQueryable<IEvent> #1680 & Allow self-aggregate of arbitrary events to an aggregate type with no identity #1681 -- nice to do
• Support the usage of private Apply() methods in self-aggregators #1682 -- I think this is more work than it's worth
• Allow dependencies between projections #1714 -- my vote is to push this off. Really to be able to use build products in one projection in another projection
• Zero Downtime Projection Rebuilds #1720 & Projection Shard Work Distribution in the Async Daemon #1717 -- Definitely going to come later. I think I'm back to wanting to do this inside of Jasper rather than just Marten. And it's big

Extracting the Database Helpers to its Own Library

This is #1467. To be honest, I want to do this because I want to reuse a lot of the code here in Jasper, and it is generic enough to move out of Marten itself. I'm thinking of pulling out CommandBuilder, the IFeatureSchema types, and utility extension methods on Npgsql.

Logging

• Let's do Use the standard .Net ILogger in the default Marten logging? #1686 as a quick win

Multi-Tenancy

#1451 & #1460 are both relatively easy. I've just been putting them off thinking we could get the folks who reported them to help

Miscellaneous

To discuss

• Proposal for v4: Postgres RLS feature via Marten #1539
• Add Options property to IDocumentStore interface #1230 -- if we do this, we need to put a read only interface over the top of StoreOptions
• Ability to stream json data directly to another stream #585 -- I'd really like to see this in V4, but it's an additive change
• Making Marten Open Source model sustainable #1391 -- is there any way to "close" this, or do we convert that to an ongoing discussion?
• Fixes #1386 - not being able to serialize IReadonlyCollection<int> while using CollectionStorage.AsArray #1390 -- same as the above issue
• Create Marten Newsletter #1397 -- seems doable
• F#: Discriminated Unions are not supported for aggregates #1283 -- there's demand from our F# users, but it's an additive change that could come later
• Re-evaluate the StoreOptions.Schema.For<T>() fluent interface #1564 -- would be breaking changes

In for V4

• Internal/public accessor hunt #1470, the big sweep to move any possible type into internal we can to reduce the public API space
• Document updates need to apply to the Patch API operations #1570 -- improvement to patching

Later

• Use unique index inference for upsert constraint #1262. I don't honestly understand this one yet
• Patch Method truncates long Id #1315 -- it's a limitation of the serialization, not Marten per se. I'm not sure we can fix this
• Linq helper to query events by metadata header values #1618 & It's not possible to use Task in the class name in v4 #1647 -- Linq helpers for raw event selectors. I vote this goes later
• Support "ToAsyncEnumerable()"  #1660 -- I think it would be cool, but it's not a breaking change, so later?
• IDocumentSession.DeleteMany(id list) operation? #1661 -- I vote later. Could be a way to optimize the async projections too

[oskardudycz]

@jeremydmiller -
Documentation

For me, the most important is to deliver new docs together with the version switcher. If it contains the features that we had, then from my perspective we can polish it later. Based on what @mysticmind wash showing, it'll be still a vast improvement and "wow effect" for marketing. Pages with links to blog posts are nice to have for me. We should have it asap, but that's not a blocker for me.

I'd be happy to work on the Event Sourcing docs. We should go with our user a journey from the basic concepts like events, streams, and appending and aggregation. Then explain snapshots (inline projections) and projections. I planned to do it earlier, but work on v4 stopped me as it could be made obsolete. Of course, @jeremydmiller, if you prefer to do it on your own, I won't interfere with that.

Event Store
Discuss

• Mechanism to Import Events #1435 -- I think we need to cover two main scenarios - migration of the data from other event store or relational/document data and integration with queuing systems. Probably we should have a middleware that would allow the user to do some data transformations. For the event store, we should provide either the custom version or aggregate by different criteria. This is especially important for queues integration, as they may not maintain order. So allow the user to insert raw event data without going through all checks and logic.
• #1611 - could you elaborate what's projection snapshot? I don't know this request and this concept
• Event schema versioning and upcasting #1613 - yes, this can be added in v4.1 this shouldn't be a breaking change, as it's adding a middleware/pipeline to the deserialisation part. You can check my recent discussion on this topic https://discuss.eventstore.com/t/deserialisation-in-c/3091
• Recipe for easy testing of event projections #1615 - Agree, we should provide documentation and samples, and it should be enough
• Improved transaction and concurrency prevention for appending events #1665 - I think that this should be a part of V4 - @ericgreenmix found that for the high load without that we may have concurrency issues
• Re-evaluate the AggregateType on Stream #1671 - I don't think that we should remove it. We could make it opt-in, but I think we should keep it. What's more, if we allow having AggregateType on the events table, together with partitioning (Partitioning support for Event Store #770), it'll add the possibility to have events partitioned by aggregate type. The queue integration/outbox pattern will open case, e.g. to publish aggregate events to the same Kafka topic.
• Re-evaluate the Marten CLI "projection" command #1677 - Btw. do we know how many people are using Marten CLI? TBT I've never used it.
• Extra validation and edge case combinations in the Event Store + Aggregated Projections #1689 - I was using that in my project to make sure that I won't be (re)creating projection on some race conditions or connection with deleting events. Especially with integration with events from other modules by message bus, it's useful. I agree that's related to edge cases, but it can save you some time and data integrity
• Projection Shard Work Distribution in the Async Daemon #1717 - I think that's a nice to have for v4

Do for V4

• Event Metadata #780 - Yes, that's a must-have. This is the most significant from the features left in ToDo. @mysticmind do you still plan to work on that?
• Partitioning support for Event Store #770 - I think we can deliver that in a non-breaking manner as opt-in potentially in v4.1. I'd be happy to work on that.
• Archiving Event Streams #1612 - Yes, together with Partitioning support for Event Store #770, this would be nice to have. I think that this can also be part of v4.1
• It's not possible to use Task in the class name in v4 #1647, Self-Aggregation does not work if the aggregate is an inner class #1679 -- Looks like both are fixed - nice @jeremydmiller
• Async Daemon Leader Election #1718 - That looks quite important, however doing proper tests may take a lot of time, I'm okay with moving it to v4.1
• Inline projection work is getting cut off in V4 with a large number of events #1723 - yup, we need to understand that at least
• Poison Message detection for the async projections #1731 - It's essential, but I think that we could deliver that later in v4.1 in a non-breaking manner

Push off to later

• Add Options property to IDocumentStore interface #1230 - @jeremydmiller are you sure that's the right issue number?
• V4 projections and DI containers #1670 - yes, we can add it later
• New Linq selector to apply a live aggregation against any IQueryable<IEvent> #1680 & Allow self-aggregate of arbitrary events to an aggregate type with no identity #1681 - nice to do - I agree
• Support the usage of private Apply() methods in self-aggregators #1682 - What is "self-aggregators"?
• Allow dependencies between projections #1714 - yes, this is an edge case that probably should be solved on the application layer (usually).
• Zero Downtime Projection Rebuilds #1720 & Projection Shard Work Distribution in the Async Daemon #1717 - For me, sharding is not so necessary. Zero downtime - if I understand that correctly is a must-have eventually. However, we may push it for the next release

[oskardudycz]

Extracting the Database Helpers to its Own Library

• Extract Generic Database Helpers to its own Library #1467 - I'm okay with that, but this should probably go to v4 as it'll be a breaking change

Logging

• Use the standard .Net ILogger in the default Marten logging? #1686 - I see this is done - sweet!

Multi-Tenancy

• Allow update/delete of tenanted document from session with no TenantId #1451 & Tenanted documents storing in non-tenant session #1460 - I think that we should discuss the elegant approach to the API before doing the work

Miscellaneous

To discuss

• Proposal for v4: Postgres RLS feature via Marten #1539 - this would be cool to have - @mysticmind do you think this could be included later as non-breaking opt-in?
• Add Options property to IDocumentStore interface #1230 - I'm okay with putting the read-only interface over the top of StoreOptions
• Ability to stream json data directly to another stream #585 - This would be cool from the marketing perspective. I could try to tackle that after my work on System.Text.Json. I agree that this should not block the v4 release if it's not ready
• Making Marten Open Source model sustainable #1391 - I think that as we're not planning to do any action on this topic, then we can close it...
• Fixes #1386 - not being able to serialize IReadonlyCollection<int> while using CollectionStorage.AsArray #1390 - Are you sure that you linked the right issue?
• Create Marten Newsletter #1397 - yes, it's doable, the question is if we want to do it, and what would be our goal for that (as we plan to close Making Marten Open Source model sustainable #1391)
• F#: Discriminated Unions are not supported for aggregates #1283 - I think that with new projections etc. Marten has good potential in the functional community. It's important but can go in v4.1
• Re-evaluate the StoreOptions.Schema.For<T>() fluent interface #1564 - I agree, it would be good to clean up that in v4

In for V4

• Internal/public accessor hunt #1470 - yes, we need to make sure that we cleared the API and only stuff that should be public is public, so we could provide more changes in a non-breaking way not being blocked by "maybe someone is using it."
• Document updates need to apply to the Patch API operations #1570 - could you elaborate? I'm not following what's the case here

Later

• Use unique index inference for upsert constraint #1262 - it might be just a simple syntax change, as it looks breaking we may consider that for v4
• Patch Method truncates long Id #1315 - We could store long as string
• Linq helper to query events by metadata header values #1618 & It's not possible to use Task in the class name in v4 #1647 - Agree, nice to have
• Support "ToAsyncEnumerable()"  #1660 - can go later - agreed.
• IDocumentSession.DeleteMany(id list) operation? #1661 - yes, can go later

[oskardudycz]

Ah, I think that my PR with System.Text.Json should go in - it's breaking #1685

[Hawxy]

Excited for the new docs :). Would it be worth having a section dedicated to "marten in production" scenarios? ie an example of how a CI/CD pipeline should be setup to deal with marten's schema changes over time.

#1539 - Would be great for those of us with clients that have security-mature policies and are uncertain about using a multi-tenanted single-database solution.

#770 + #1612 - Highly valuable for long-term scalability, yes please!

[jeremydmiller]

Hey all, it's time to once again play "what's it going to take to finish V4?"

I've got a relatively free day tomorrow at work, and I can throw maybe 3-4 hours of uninterrupted time at Marten. Here's what I see is left besides the new documentation for development:

• Event Metadata (Event Metadata #780). @mysticmind Do you still want to take this? I could divert and pound this out if you want.
• Add Options property to IDocumentStore interface #1230 - I can take this. Shouldn't be any big deal
• @oskardudycz is already in flight with Partitioning support for Event Store #770
• I'm still interested in doing Extract Generic Database Helpers to its own Library #1467, but we shouldn't tough that until Partitioning support for Event Store #770 is done
• Internal/public accessor hunt #1470 is a last minute sweep of the codebase, and I don't anticipate that being a big deal
• F#: Discriminated Unions are not supported for aggregates #1283 - the F# discriminated union querying. I think this could actually be done now w/ a Marten add on, but it'd still be good to prove that out. I kinda want some F# dev help here
• Use unique index inference for upsert constraint #1262 -- the title might be misleading. It's really "Make Marten work with TimescaleDB". I wrote that I thought it was a breaking change because it would change the upsert functions. I want to vote to move this to 4.1
• Proposal for v4: Postgres RLS feature via Marten #1539 -- something similar came up recently where it would be advantageous to set up SQL operations to take place upon a session being initialized
• Patch Method truncates long Id #1315 -- @oskardudycz suggested storing longs as strings. We didn't attempt to tackle any kind of optimized patching story in V4. I'd like to make anything around patching be a V4.1 thing and put this off
• Mechanism to Import Events #1435 - I'm thinking we do something w/ binary copy to make it fast. I'm voting to push this to post V4
• Archiving Event Streams #1612 - Archiving event streams. As we discussed on a call, I think this in combination w/ Partitioning support for Event Store #770 is a big win for scalability. I'd be happy to take that on whenever, but I don't know if it would be better to let Partitioning support for Event Store #770 go first
• #1611 - We haven't talked about this at all, but I think we should on our next Marten team call. I don't have any thoughts on how to do this or requirements offhand.
• Recipe for easy testing of event projections #1615 - Recipe for testing event projections. I'm going to take this tomorrow, and do something similar to Alba for projections
• Poison Message detection for the async projections #1731 - I've got some ideas around this one, and I don't think it's big. This will be in my next wave of work
• Async Daemon Leader Election #1718 - the Async Daemon leader election using advisory locks. I don't think this is very much code at all, but it's gonna be a pain to test. I'm happy to work on this because it'll be similar to work I've done in Jasper
• Marten v4 memory consumption problem #1735 - the memory usage w/ Roslyn. At the very worst, I have a workaround. Really just need to run the tests with memory usage tracing and see what's hanging on to the memory

I'm sure we'll come up with other stuff, plus the documentation & new website.

Thoughts???

[jeremydmiller]

And I completely missed #1613. I think I'd ask @oskardudycz to speak to that next time we all jump on a call

[mysticmind]

@jeremydmiller with regards to #780, I was planning to take a crack at it by early next week, besides the plan was to understand the meta data portion so that I can also add the documentation for it. If you think, you can take it up, please do, don't want to keep it as a blocker for v4.

[jeremydmiller]

@mysticmind If you could take #780 and the docs on metadata, that would be a huge contribution. There's plenty of other things for me to be doing in the meantime. Yell if you have any questions or suggestions about what's already been done on the metadata.

I don't think this would help timewise, but I'd love to switch back to Jasper soon, push through an OpenTelemetry implementation there, then drop Marten into that and prove that the new metadata support in Marten works well with OT tracing.

[mysticmind]

@jeremydmiller sure, will get it done 👍

[jeremydmiller]

Just time for an update on V4, because it's taken much, much longer than I think anyone would have hoped for. The milestone is getting smaller at least.

Me, personally, I'd like to declare success on my development work by mid-April and focus on documentation updates. In a perfect world, I'd love for us to shoot for v4 escaping into the wild by the end of April. Commence laughing at me.

Here's how I see the remaining development work:

• @mysticmind is working on the Event Metadata story (Event Metadata #780) and I think that's close
• @Hawxy is adding some nullability checks throughout the codebase, and we'll need to do that before V4 since it changes the public API
• @JohnCampionJr is working on the F# discriminated union request F#: Discriminated Unions are not supported for aggregates #1283
• @oskardudycz is working on Allow users to supply custom IEventSlicer for custom aggregation strategies #1784 that's necessary for us to support event projections where one event could apply to multiple projected views
• We'll need to tackle the usage of Postgresql table partitioning in our schema management. Once that works, that opens up event store partitioning (Partitioning support for Event Store #770) and the event store stream archiving (Archiving Event Streams #1612). Somewhat related to that is moving the generic ADO.Net helper and Postgresql schema management functionality to a new library (Extract Generic Database Helpers to its own Library #1467). I'm still interested in making this happen, but I think this happens right after we make our next alpha release. I may start that work in a branch soon.
• The mechanism to import events is probably dependent upon Event Metadata #780 and also Archiving Event Streams #1612 and Partitioning support for Event Store #770. There is user demand for this, but it could float to 4.1 in my opinion because it's not a breaking change.
• We'll do the internal accessor hunt (Internal/public accessor hunt #1470) and fill in Xml API comments at the last minute. I'm not too worried about that.
• It's not on the milestone yet, but near the end let's do a find and replace on .ConfigureAwait(false) and wipe that out of the codebase
• I (@jeremydmiller) will take a look at the memory usage issue (Marten v4 memory consumption problem #1735) with dotProfile before V4. I'm not too worried about this, and if we can't fix it, I've got a workaround in mind from Jasper to pre-build the generated source code ahead of time.
• Proposal for v4: Postgres RLS feature via Marten #1539 would be a cool feature, and might not be very difficult.
• I think we need to have a conversation about some of the remaining event store work. We have no analysis done yet on Projection Snapshots #1611 (basically sampling projected views by persisting it every so many versions).
• We also need to talk a little more about event versioning and upcasting (Event schema versioning and upcasting #1613). I think this could be done w/o any breaking changes and might be a candidate for 4.1 if we can't squeeze it in for 4.0

[johncj-improving]

Is that a typo wrt #1283 or is someone else also working on it?

[jeremydmiller]

Typo on my part, sorry about that:)

[johncj-improving]

My fix for #1283 is completely external to the Marten codebase. There's no need to hold up the release of 4.0 for it. In fact, it is somewhat dependent on the completion of 4.0. In particular, the implementation of #780 and #1784 will have an impact on my solution.

[jeremydmiller]

The Very, Very Last Push

At this point I want to cut bait, push every non-breaking change to 4.0, and start writing documentation updates.

I did some more research on using table partitioning against the event table, and I think it's a viable strategy. I also think it would tie in very well with some of the "async daemon sharding" stories that have already gotten pushed to the 4.1 milestone. What I propose at this point is to:

• Let @oskardudycz finish Allow users to supply custom IEventSlicer for custom aggregation strategies #1784 in a simple way for now
• Knock out all the documentation related issues in the 4.0 milestone next week (https://github.com/JasperFx/marten/milestone/28)
• Do the "Archiving Event Streams" story (Archiving Event Streams #1612) in a simplistic way just to get the new is_archived column and related changes in place, but punt on table partitioning until 4.1. I've gone back and forth, but I vote to do the event store archiving with the assumption that it works with partitioning. We should discuss this decision to punt the partitioning
• I'll deal with the memory issue this or next week (Marten v4 memory consumption problem #1735). I have a workaround strategy to pre-generate the source code ahead of time as a last ditch effort to fix this.
• Push Partitioning support for Event Store #770, Proposal for v4: Postgres RLS feature via Marten #1539 (I think this could be mostly done w/ the ISessionFactory thing, plus some extensions to Weasel for RLS), Mechanism to Import Events #1435, #1611 to 4.1.
• Move PLV8 related stuff to an add on library Extract all features that depend on PLV8 to an addon library #1812. This would impact the Patch() functionality!
• New little story on projection configuration usability: Improved Projection Registration #1813
• Cut a Marten 4.0.0-rc-1 at the end of this week
• Shoot for the new website to be ready-ish by the end of next week

The 4.1 release would end up being a lot of optimization of the event store functionality that's getting missed by 4.0. I think that 4.2 could be doing an optimized, non-plv8 using version of the Patch() API.

4.0 has been in flight for over a year of coding and 2 years of planning, and it's starting to feel like Duke Nukem Forever. I think I want to move to documentation and the inevitable last minute API changes that should result from doing the documentation and sample apps.

There's an artificial time limit here. I'm starting a new job the last week of May, and I'd dearly love V4 to basically be done before I start the new job. I think I'll have more energy for OSS in my new role, but who knows?

[oskardudycz]

The plan looks reasonable 👍 I don't have points where I'd disagree 😉