Move towards using MultiWriterIdGenerator everywhere
#17226
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erikjohnston
force-pushed
the
erikj/use_multi_writer_id_generator
branch
from
8dfeccf
to
9ccffef
Compare
erikjohnston
force-pushed
the
erikj/use_multi_writer_id_generator
branch
from
9ccffef
to
c2ba6d5
Compare
clokep
reviewed
May 21, 2024
Comment on lines
194
to
195
| self._stream_id_gen: AbstractStreamIdGenerator | ||
| self._backfill_id_gen: AbstractStreamIdGenerator |
There was a problem hiding this comment.
You can probably let these just be MultiWriterIdGenerator now instead of defining them as AbstractStreamIdGenerator? (Applies throughout the diff.)
(Overall love the simplification! 🎉 )
There was a problem hiding this comment.
Yeah I tried to remove these and mypy went 💥 I'm sure we can get rid of them, but I didn't want to go down that rabbithole right now
This was referenced May 22, 2024
reivilibre
approved these changes
May 29, 2024
| FROM %(table)s | ||
| """ % { | ||
| "greatest_func": greatest_func, | ||
| "id": id_column, | ||
| "table": table, | ||
| "agg": "MAX" if self._positive else "-MIN", |
There was a problem hiding this comment.
the inconsistency here is killing me, but apparently this is indeed how Postgres works
There was a problem hiding this comment.
Between the aggregation functionMAX and the standard function GREATEST?
Co-authored-by: reivilibre <oliverw@matrix.org>
H-Shay
pushed a commit
to H-Shay/hq_synapse
that referenced
this pull request
May 31, 2024
) There is a problem with `StreamIdGenerator` where it can go backwards over restarts when a stream ID is requested but then not inserted into the DB. This is problematic if we want to land element-hq#17215, and is generally a potential cause for all sorts of nastiness. Instead of trying to fix `StreamIdGenerator`, we may as well move to `MultiWriterIdGenerator` that does not suffer from this problem (the latest positions are stored in `stream_positions` table). This involves adding SQLite support to the class. This only changes id generators that were already using `MultiWriterIdGenerator` under postgres, a separate PR will move the rest of the uses of `StreamIdGenerator` over.
Mic92
pushed a commit
to Mic92/synapse
that referenced
this pull request
Jun 14, 2024
) There is a problem with `StreamIdGenerator` where it can go backwards over restarts when a stream ID is requested but then not inserted into the DB. This is problematic if we want to land element-hq#17215, and is generally a potential cause for all sorts of nastiness. Instead of trying to fix `StreamIdGenerator`, we may as well move to `MultiWriterIdGenerator` that does not suffer from this problem (the latest positions are stored in `stream_positions` table). This involves adding SQLite support to the class. This only changes id generators that were already using `MultiWriterIdGenerator` under postgres, a separate PR will move the rest of the uses of `StreamIdGenerator` over.
3 tasks
yingziwu
added a commit
to yingziwu/synapse
that referenced
this pull request
Jun 26, 2024
- Fix the building of binary wheels for macOS by switching to macOS 12 CI runners. ([\#17319](element-hq/synapse#17319)) - When rolling back to a previous Synapse version and then forwards again to this release, don't require server operators to manually run SQL. ([\#17305](element-hq/synapse#17305), [\#17309](element-hq/synapse#17309)) - Use the release branch for sytest in release-branch PRs. ([\#17306](element-hq/synapse#17306)) - Fix bug where one-time-keys were not always included in `/sync` response when using workers. Introduced in v1.109.0rc1. ([\#17275](element-hq/synapse#17275)) - Fix bug where `/sync` could get stuck due to edge case in device lists handling. Introduced in v1.109.0rc1. ([\#17292](element-hq/synapse#17292)) - Add the ability to auto-accept invites on the behalf of users. See the [`auto_accept_invites`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#auto-accept-invites) config option for details. ([\#17147](element-hq/synapse#17147)) - Add experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync/e2ee` endpoint for to-device messages and device encryption info. ([\#17167](element-hq/synapse#17167)) - Support [MSC3916](matrix-org/matrix-spec-proposals#3916) by adding unstable media endpoints to `/_matrix/client`. ([\#17213](element-hq/synapse#17213)) - Add logging to tasks managed by the task scheduler, showing CPU and database usage. ([\#17219](element-hq/synapse#17219)) - Fix deduplicating of membership events to not create unused state groups. ([\#17164](element-hq/synapse#17164)) - Fix bug where duplicate events could be sent down sync when using workers that are overloaded. ([\#17215](element-hq/synapse#17215)) - Ignore attempts to send to-device messages to bad users, to avoid log spam when we try to connect to the bad server. ([\#17240](element-hq/synapse#17240)) - Fix handling of duplicate concurrent uploading of device one-time-keys. ([\#17241](element-hq/synapse#17241)) - Fix reporting of default tags to Sentry, such as worker name. Broke in v1.108.0. ([\#17251](element-hq/synapse#17251)) - Fix bug where typing updates would not be sent when using workers after a restart. ([\#17252](element-hq/synapse#17252)) - Update the LemonLDAP documentation to say that claims should be explicitly included in the returned `id_token`, as Synapse won't request them. ([\#17204](element-hq/synapse#17204)) - Improve DB usage when fetching related events. ([\#17083](element-hq/synapse#17083)) - Log exceptions when failing to auto-join new user according to the `auto_join_rooms` option. ([\#17176](element-hq/synapse#17176)) - Reduce work of calculating outbound device lists updates. ([\#17211](element-hq/synapse#17211)) - Improve performance of calculating device lists changes in `/sync`. ([\#17216](element-hq/synapse#17216)) - Move towards using `MultiWriterIdGenerator` everywhere. ([\#17226](element-hq/synapse#17226)) - Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`. ([\#17229](element-hq/synapse#17229)) - Change the `allow_unsafe_locale` config option to also apply when setting up new databases. ([\#17238](element-hq/synapse#17238)) - Fix errors in logs about closing incorrect logging contexts when media gets rejected by a module. ([\#17239](element-hq/synapse#17239), [\#17246](element-hq/synapse#17246)) - Clean out invalid destinations from `device_federation_outbox` table. ([\#17242](element-hq/synapse#17242)) - Stop logging errors when receiving invalid User IDs in key querys requests. ([\#17250](element-hq/synapse#17250)) * Bump anyhow from 1.0.83 to 1.0.86. ([\#17220](element-hq/synapse#17220)) * Bump bcrypt from 4.1.2 to 4.1.3. ([\#17224](element-hq/synapse#17224)) * Bump lxml from 5.2.1 to 5.2.2. ([\#17261](element-hq/synapse#17261)) * Bump mypy-zope from 1.0.3 to 1.0.4. ([\#17262](element-hq/synapse#17262)) * Bump phonenumbers from 8.13.35 to 8.13.37. ([\#17235](element-hq/synapse#17235)) * Bump prometheus-client from 0.19.0 to 0.20.0. ([\#17233](element-hq/synapse#17233)) * Bump pyasn1 from 0.5.1 to 0.6.0. ([\#17223](element-hq/synapse#17223)) * Bump pyicu from 2.13 to 2.13.1. ([\#17236](element-hq/synapse#17236)) * Bump pyopenssl from 24.0.0 to 24.1.0. ([\#17234](element-hq/synapse#17234)) * Bump serde from 1.0.201 to 1.0.202. ([\#17221](element-hq/synapse#17221)) * Bump serde from 1.0.202 to 1.0.203. ([\#17232](element-hq/synapse#17232)) * Bump twine from 5.0.0 to 5.1.0. ([\#17225](element-hq/synapse#17225)) * Bump types-psycopg2 from 2.9.21.20240311 to 2.9.21.20240417. ([\#17222](element-hq/synapse#17222)) * Bump types-pyopenssl from 24.0.0.20240311 to 24.1.0.20240425. ([\#17260](element-hq/synapse#17260))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a problem with
StreamIdGeneratorwhere it can go backwards over restarts when a stream ID is requested but then not inserted into the DB. This is problematic if we want to land #17215, and is generally a potential cause for all sorts of nastiness.Instead of trying to fix
StreamIdGenerator, we may as well move toMultiWriterIdGeneratorthat does not suffer from this problem (the latest positions are stored instream_positionstable). This involves adding SQLite support to the class.This only changes id generators that were already using
MultiWriterIdGeneratorunder postgres, a separate PR will move the rest of the uses ofStreamIdGeneratorover.Reviewable commit-by-commit.