Prepare for authenticated media freeze #17433

H-Shay · 2024-07-11T04:23:21Z

As part of the rollout of MSC3916 this PR adds support for designating authenticated media and ensuring that authenticated media is not served over unauthenticated endpoints.

Should be reviewable commit-by-commit.

…mote_media_cache`

…edia

… demands it

H-Shay · 2024-07-11T04:32:01Z

synapse/config/repository.py

@@ -272,6 +272,11 @@ def read_config(self, config: JsonDict, **kwargs: Any) -> None:
                remote_media_lifetime
            )

+        self.authenticate_new_media = config.get("authenticate_new_media", False)
+        self.enforce_authenticated_media = config.get(
+            "enforce_authenticated_media", False


Two questions:

Should this be one config option? Something like enable_authenticated_media?

Should this option be documented in the config manual, since we are going to switch it on default and strongly encourage it's use - perhaps documenting it will make it seem more optional than we'd like?

I think documenting any config option (except experimental ones) is worth doing, even if you discourage changing it and outline the plans for the default value of the option. May also be worth saying when the option is expected to be removed (which I imagine we will do?)

Yup, agreed. I'm sure there will be some people that will need to fiddle with the config options (e.g. those using custom clients), so having them documented would be good.

Should this be one config option? Something like enable_authenticated_media?

I'm in two minds a bit here. Having two gives us more flexibility, but not actually sure of the use case. The only thing that comes to mind is if we change the defaults at different times, but then I think we wouldn't want to do that at all.

I also went back and forth about making it one config option, and I think in the end it makes slightly more sense to make it one, essentially for the reasons you listed - I couldn't really think of a case where the two would be set independently of each other. I am going to switch it to one - this will also make documentation clearer. If anyone comes up with a reason to switch it back let me know...

H-Shay · 2024-07-11T04:33:30Z

@erikjohnston - hoping you'll be able to take a look and let me know if this is in alignment with what we discussed

erikjohnston · 2024-07-16T10:19:00Z

@H-Shay This looks good to me, but would like someone else on the team to review the approach/code just in case.

reivilibre

nothing major, just tidbits

reivilibre · 2024-07-16T11:13:03Z

synapse/storage/schema/main/delta/86/01_authenticate_media.sql

+ALTER TABLE remote_media_cache ADD COLUMN authenticated BOOLEAN;
+ALTER TABLE local_media_repository ADD COLUMN authenticated BOOLEAN;


should we have a DEFAULT value here, and make it NOT NULL?

It would be nice to add a comment describing the semantics of the value here, even briefly. If it's a nullable column, that should be described too, which is why I'm tempted to suggest making this NOT NULL.

I guess the default could be False, although the code essentially assumes any media item for which this column isn't True is unauthenticated so I think the null works (but is probably unclear, as you are pointing out)- setting the default might make that clearer. Essentially all existing media in the table should be either null or False once the column is added, but I can make that clearer.

Note that setting a DEFAULT means that the table needs to get rewritten to insert the new values, which takes a while. Leaving it as NULLABLE is somewhat annoying but by far the easiest.

This isn't the case anymore since Postgres 11: https://www.dbi-services.com/blog/postgresql-11-instant-add-column-with-a-non-null-default-value/

synapse/media/media_repository.py

synapse/media/thumbnailer.py

synapse/storage/schema/main/delta/86/01_authenticate_media.sql

reivilibre

I'm happy with this, just some fiddling with the documentation

reivilibre · 2024-07-17T10:39:10Z

docs/usage/configuration/config_documentation.md

@@ -1863,6 +1863,16 @@ federation_rr_transactions_per_room_per_second: 40
 ## Media Store
 Config options related to Synapse's media store.

+---
+### `enable_authenticated_media`


maybe this would be clearer as new_media_requires_authentication or something like that? It seems like once the media gets the flag, it stays in that state forever, so the current name doesn't give quite the right impression.

Thanks for writing this up though, this is way clearer about what we're hoping this feature will do, it's worth it just for that alone.

mrh, actually, it looks like the two options were merged and this option is named properly, but it'd probably be worth pointing out that disabling this option means all media, including those flagged as 'authenticated media', will be available over the unauthenticated endpoint? Since we disable the auth checks when this is turned off.

reivilibre · 2024-07-17T10:39:28Z

docs/usage/configuration/config_documentation.md

+
+Example configuration:
+```yaml
+enable_authenticated_media


Suggested change

enable_authenticated_media

enable_authenticated_media: true

reivilibre · 2024-07-17T10:40:09Z

docs/usage/configuration/config_documentation.md

+
+When set to true, all subsequent media uploads will be marked as authenticated, and will not be available over legacy
+unauthenticated media endpoints (`/_matrix/media/(r0|v3|v1)/download` and `/_matrix/media/(r0|v3|v1)/thumbnail`) - requests for authenticated media over these endpoints will result in a 404. All media, including authenticated media, will be available over the authenticated media endpoints `_matrix/client/v1/media/download` and `_matrix/client/v1/media/thumbnail`. Media uploaded prior to setting this option to true will still be available over the legacy endpoints.
+


Suggested change

Defaults to false, but this will change to true in a future Synapse release.

reivilibre · 2024-07-17T10:41:49Z

docs/usage/configuration/config_documentation.md

@@ -1863,6 +1863,16 @@ federation_rr_transactions_per_room_per_second: 40
 ## Media Store
 Config options related to Synapse's media store.

+---


(on the fence; do you think we ought to add an upgrade notes entry about this change, so people can get ahead of the curve and set the option to false explicitly if they want to make sure their special client won't be bitten by the upcoming freeze? It'd also be a good place to just lay out the plan a bit, since I think for most who don't read TWIM etc, the upcoming changes may not be obvious?)

I'd like to get @turt2live's opinion on this - perhaps it would make sense to link to the blog post announcing the freeze? I think for now though we can merge this (I'd like to get it into the next rc so people can test against it on beta.matrix.org) and figure out how to handle the upgrade notes, etc soon.

Server operators are going to be increasingly incentivized to ensure this option is actually set to true rather than false, regardless of client breakage. When we change the default to true we should definitely call out matrix.org's plan as reference, but I wouldn't include language about setting it to false.

From brief discussion in backend internal: we should provide users with instructions on how to set it to false, but we can have the changelog read as eventually removing the option (making the feature inevitable)

synapse/storage/schema/main/delta/86/01_authenticate_media.sql

H-Shay · 2024-07-17T18:03:14Z

Doc changes are done, @reivilibre can you hit the merge button if you are satisfied with the result?

@Michael-Hollister

This security release is to update our locked dependency on Twisted to 24.7.0rc1, which includes a security fix for [CVE-2024-41671 / GHSA-c8m8-j448-xjx7: Disordered HTTP pipeline response in twisted.web, again](GHSA-c8m8-j448-xjx7). Note that this security fix is also available as **Synapse 1.111.1**, which does not include the rest of the changes in Synapse 1.112.0. This issue means that, if multiple HTTP requests are pipelined in the same TCP connection, Synapse can send responses to the wrong HTTP request. If a reverse proxy was configured to use HTTP pipelining, this could result in responses being sent to the wrong user, severely harming confidentiality. With that said, despite being a high severity issue, **we consider it unlikely that Synapse installations will be affected**. The use of HTTP pipelining in this fashion would cause worse performance for clients (request-response latencies would be increased as users' responses would be artificially blocked behind other users' slow requests). Further, Nginx and Haproxy, two common reverse proxies, do not appear to support configuring their upstreams to use HTTP pipelining and thus would not be affected. For both of these reasons, we consider it unlikely that a Synapse deployment would be set up in such a configuration. Despite that, we cannot rule out that some installations may exist with this unusual setup and so we are releasing this security update today. **pip users:** Note that by default, upgrading Synapse using pip will not automatically upgrade Twisted. **Please manually install the new version of Twisted** using `pip install Twisted==24.7.0rc1`. Note also that even the `--upgrade-strategy=eager` flag to `pip install -U matrix-synapse` will not upgrade Twisted to a patched version because it is only a release candidate at this time. - Upgrade locked dependency on Twisted to 24.7.0rc1. ([\#17502](element-hq/synapse#17502)) Please note that this release candidate does not include the security dependency update included in version 1.111.1 as this version was released before 1.111.1. The same security fix can be found in the full release of 1.112.0. - Add to-device extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17416](element-hq/synapse#17416)) - Populate `name`/`avatar` fields in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17418](element-hq/synapse#17418)) - Populate `heroes` and room summary fields (`joined_count`, `invited_count`) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17419](element-hq/synapse#17419)) - Populate `is_dm` room field in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17429](element-hq/synapse#17429)) - Add room subscriptions to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17432](element-hq/synapse#17432)) - Prepare for authenticated media freeze. ([\#17433](element-hq/synapse#17433)) - Add E2EE extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17454](element-hq/synapse#17454)) - Add configurable option to always include offline users in presence sync results. Contributed by @Michael-Hollister. ([\#17231](element-hq/synapse#17231)) - Fix bug in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint when using room type filters and the user has one or more remote invites. ([\#17434](element-hq/synapse#17434)) - Order `heroes` by `stream_ordering` as the Matrix specification states (applies to `/sync`). ([\#17435](element-hq/synapse#17435)) - Fix rare bug where `/sync` would break for a user when using workers with multiple stream writers. ([\#17438](element-hq/synapse#17438)) - Update the readme image to have a white background, so that it is readable in dark mode. ([\#17387](element-hq/synapse#17387)) - Add Red Hat Enterprise Linux and Rocky Linux 8 and 9 installation instructions. ([\#17423](element-hq/synapse#17423)) - Improve documentation for the [`default_power_level_content_override`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#default_power_level_content_override) config option. ([\#17451](element-hq/synapse#17451)) - Make sure we always use the right logic for enabling the media repo. ([\#17424](element-hq/synapse#17424)) - Fix argument documentation for method `RateLimiter.record_action`. ([\#17426](element-hq/synapse#17426)) - Reduce volume of 'Waiting for current token' logs, which were introduced in v1.109.0. ([\#17428](element-hq/synapse#17428)) - Limit concurrent remote downloads to 6 per IP address, and decrement remote downloads without a content-length from the ratelimiter after the download is complete. ([\#17439](element-hq/synapse#17439)) - Remove unnecessary call to resume producing in fake channel. ([\#17449](element-hq/synapse#17449)) - Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to bump room when it is created. ([\#17453](element-hq/synapse#17453)) - Speed up generating sliding sync responses. ([\#17458](element-hq/synapse#17458)) - Add cache to `get_rooms_for_local_user_where_membership_is` to speed up sliding sync. ([\#17460](element-hq/synapse#17460)) - Speed up fetching room keys from backup. ([\#17461](element-hq/synapse#17461)) - Speed up sorting of the room list in sliding sync. ([\#17468](element-hq/synapse#17468)) - Implement handling of `$ME` as a state key in sliding sync. ([\#17469](element-hq/synapse#17469)) * Bump bytes from 1.6.0 to 1.6.1. ([\#17441](element-hq/synapse#17441)) * Bump hiredis from 2.3.2 to 3.0.0. ([\#17464](element-hq/synapse#17464)) * Bump jsonschema from 4.22.0 to 4.23.0. ([\#17444](element-hq/synapse#17444)) * Bump matrix-org/done-action from 2 to 3. ([\#17440](element-hq/synapse#17440)) * Bump mypy from 1.9.0 to 1.10.1. ([\#17445](element-hq/synapse#17445)) * Bump pyopenssl from 24.1.0 to 24.2.1. ([\#17465](element-hq/synapse#17465)) * Bump ruff from 0.5.0 to 0.5.4. ([\#17466](element-hq/synapse#17466)) * Bump sentry-sdk from 2.6.0 to 2.8.0. ([\#17456](element-hq/synapse#17456)) * Bump sentry-sdk from 2.8.0 to 2.10.0. ([\#17467](element-hq/synapse#17467)) * Bump setuptools from 67.6.0 to 70.0.0. ([\#17448](element-hq/synapse#17448)) * Bump twine from 5.1.0 to 5.1.1. ([\#17443](element-hq/synapse#17443)) * Bump types-jsonschema from 4.22.0.20240610 to 4.23.0.20240712. ([\#17446](element-hq/synapse#17446)) * Bump ulid from 1.1.2 to 1.1.3. ([\#17442](element-hq/synapse#17442)) * Bump zipp from 3.15.0 to 3.19.1. ([\#17427](element-hq/synapse#17427))

@Michael-Hollister

# Synapse 1.112.0 (2024-07-30) This security release is to update our locked dependency on Twisted to 24.7.0rc1, which includes a security fix for [CVE-2024-41671 / GHSA-c8m8-j448-xjx7: Disordered HTTP pipeline response in twisted.web, again](GHSA-c8m8-j448-xjx7). Note that this security fix is also available as **Synapse 1.111.1**, which does not include the rest of the changes in Synapse 1.112.0. This issue means that, if multiple HTTP requests are pipelined in the same TCP connection, Synapse can send responses to the wrong HTTP request. If a reverse proxy was configured to use HTTP pipelining, this could result in responses being sent to the wrong user, severely harming confidentiality. With that said, despite being a high severity issue, **we consider it unlikely that Synapse installations will be affected**. The use of HTTP pipelining in this fashion would cause worse performance for clients (request-response latencies would be increased as users' responses would be artificially blocked behind other users' slow requests). Further, Nginx and Haproxy, two common reverse proxies, do not appear to support configuring their upstreams to use HTTP pipelining and thus would not be affected. For both of these reasons, we consider it unlikely that a Synapse deployment would be set up in such a configuration. Despite that, we cannot rule out that some installations may exist with this unusual setup and so we are releasing this security update today. **pip users:** Note that by default, upgrading Synapse using pip will not automatically upgrade Twisted. **Please manually install the new version of Twisted** using `pip install Twisted==24.7.0rc1`. Note also that even the `--upgrade-strategy=eager` flag to `pip install -U matrix-synapse` will not upgrade Twisted to a patched version because it is only a release candidate at this time. ### Internal Changes - Upgrade locked dependency on Twisted to 24.7.0rc1. ([\#17502](element-hq/synapse#17502)) # Synapse 1.112.0rc1 (2024-07-23) Please note that this release candidate does not include the security dependency update included in version 1.111.1 as this version was released before 1.111.1. The same security fix can be found in the full release of 1.112.0. ### Features - Add to-device extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17416](element-hq/synapse#17416)) - Populate `name`/`avatar` fields in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17418](element-hq/synapse#17418)) - Populate `heroes` and room summary fields (`joined_count`, `invited_count`) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17419](element-hq/synapse#17419)) - Populate `is_dm` room field in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17429](element-hq/synapse#17429)) - Add room subscriptions to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17432](element-hq/synapse#17432)) - Prepare for authenticated media freeze. ([\#17433](element-hq/synapse#17433)) - Add E2EE extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17454](element-hq/synapse#17454)) ### Bugfixes - Add configurable option to always include offline users in presence sync results. Contributed by @Michael-Hollister. ([\#17231](element-hq/synapse#17231)) - Fix bug in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint when using room type filters and the user has one or more remote invites. ([\#17434](element-hq/synapse#17434)) - Order `heroes` by `stream_ordering` as the Matrix specification states (applies to `/sync`). ([\#17435](element-hq/synapse#17435)) - Fix rare bug where `/sync` would break for a user when using workers with multiple stream writers. ([\#17438](element-hq/synapse#17438)) ### Improved Documentation - Update the readme image to have a white background, so that it is readable in dark mode. ([\#17387](element-hq/synapse#17387)) - Add Red Hat Enterprise Linux and Rocky Linux 8 and 9 installation instructions. ([\#17423](element-hq/synapse#17423)) - Improve documentation for the [`default_power_level_content_override`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#default_power_level_content_override) config option. ([\#17451](element-hq/synapse#17451)) ### Internal Changes - Make sure we always use the right logic for enabling the media repo. ([\#17424](element-hq/synapse#17424)) - Fix argument documentation for method `RateLimiter.record_action`. ([\#17426](element-hq/synapse#17426)) - Reduce volume of 'Waiting for current token' logs, which were introduced in v1.109.0. ([\#17428](element-hq/synapse#17428)) - Limit concurrent remote downloads to 6 per IP address, and decrement remote downloads without a content-length from the ratelimiter after the download is complete. ([\#17439](element-hq/synapse#17439)) - Remove unnecessary call to resume producing in fake channel. ([\#17449](element-hq/synapse#17449)) - Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to bump room when it is created. ([\#17453](element-hq/synapse#17453)) - Speed up generating sliding sync responses. ([\#17458](element-hq/synapse#17458)) - Add cache to `get_rooms_for_local_user_where_membership_is` to speed up sliding sync. ([\#17460](element-hq/synapse#17460)) - Speed up fetching room keys from backup. ([\#17461](element-hq/synapse#17461)) - Speed up sorting of the room list in sliding sync. ([\#17468](element-hq/synapse#17468)) - Implement handling of `$ME` as a state key in sliding sync. ([\#17469](element-hq/synapse#17469)) ### Updates to locked dependencies * Bump bytes from 1.6.0 to 1.6.1. ([\#17441](element-hq/synapse#17441)) * Bump hiredis from 2.3.2 to 3.0.0. ([\#17464](element-hq/synapse#17464)) * Bump jsonschema from 4.22.0 to 4.23.0. ([\#17444](element-hq/synapse#17444)) * Bump matrix-org/done-action from 2 to 3. ([\#17440](element-hq/synapse#17440)) * Bump mypy from 1.9.0 to 1.10.1. ([\#17445](element-hq/synapse#17445)) * Bump pyopenssl from 24.1.0 to 24.2.1. ([\#17465](element-hq/synapse#17465)) * Bump ruff from 0.5.0 to 0.5.4. ([\#17466](element-hq/synapse#17466)) * Bump sentry-sdk from 2.6.0 to 2.8.0. ([\#17456](element-hq/synapse#17456)) * Bump sentry-sdk from 2.8.0 to 2.10.0. ([\#17467](element-hq/synapse#17467)) * Bump setuptools from 67.6.0 to 70.0.0. ([\#17448](element-hq/synapse#17448)) * Bump twine from 5.1.0 to 5.1.1. ([\#17443](element-hq/synapse#17443)) * Bump types-jsonschema from 4.22.0.20240610 to 4.23.0.20240712. ([\#17446](element-hq/synapse#17446)) * Bump ulid from 1.1.2 to 1.1.3. ([\#17442](element-hq/synapse#17442)) * Bump zipp from 3.15.0 to 3.19.1. ([\#17427](element-hq/synapse#17427)) # Synapse 1.111.0 (2024-07-16) No significant changes since 1.111.0rc2. # Synapse 1.111.0rc2 (2024-07-10) ### Bugfixes - Fix bug where using `synapse.app.media_repository` worker configuration would break the new media endpoints. ([\#17420](element-hq/synapse#17420)) ### Improved Documentation - Document the new federation media worker endpoints in the [upgrade notes](https://element-hq.github.io/synapse/v1.111/upgrade.html) and [worker docs](https://element-hq.github.io/synapse/v1.111/workers.html). ([\#17421](element-hq/synapse#17421)) ### Internal Changes - Route authenticated federation media requests to media repository workers in Complement tests. ([\#17422](element-hq/synapse#17422)) # Synapse 1.111.0rc1 (2024-07-09) ### Features - Add `rooms` data to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17320](element-hq/synapse#17320)) - Add `room_types`/`not_room_types` filtering to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17337](element-hq/synapse#17337)) - Return "required state" in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17342](element-hq/synapse#17342)) - Support [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md) by adding [`_matrix/client/v1/media/download`](https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediadownloadservernamemediaid) endpoint. ([\#17365](element-hq/synapse#17365)) - Support [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/rav/authentication-for-media/proposals/3916-authentication-for-media.md) by adding [`_matrix/client/v1/media/thumbnail`](https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediathumbnailservernamemediaid), [`_matrix/federation/v1/media/thumbnail`](https://spec.matrix.org/v1.11/server-server-api/#get_matrixfederationv1mediathumbnailmediaid) endpoints and stabilizing the remaining [`_matrix/client/v1/media`](https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediaconfig) endpoints. ([\#17388](element-hq/synapse#17388)) - Add `rooms.bump_stamp` for easier client-side sorting in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17395](element-hq/synapse#17395)) - Forget all of a user's rooms upon deactivation, preventing local room purges from being blocked on deactivated users. ([\#17400](element-hq/synapse#17400)) - Declare support for [Matrix 1.11](https://matrix.org/blog/2024/06/20/matrix-v1.11-release/). ([\#17403](element-hq/synapse#17403)) - [MSC3861](matrix-org/matrix-spec-proposals#3861): allow overriding the introspection endpoint. ([\#17406](element-hq/synapse#17406)) ### Bugfixes - Fix rare race which caused no new to-device messages to be received from remote server. ([\#17362](element-hq/synapse#17362)) - Fix bug in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint when using an old database. ([\#17398](element-hq/synapse#17398)) ### Improved Documentation - Clarify that `url_preview_url_blacklist` is a usability feature. ([\#17356](element-hq/synapse#17356)) - Fix broken links in README. ([\#17379](element-hq/synapse#17379)) - Clarify that changelog content *and file extension* need to match in order for entries to merge. ([\#17399](element-hq/synapse#17399)) ### Internal Changes - Make the release script create a release branch for Complement as well. ([\#17318](element-hq/synapse#17318)) - Fix uploading packages to PyPi. ([\#17363](element-hq/synapse#17363)) - Add CI check for the README. ([\#17367](element-hq/synapse#17367)) - Fix linting errors from new `ruff` version. ([\#17381](element-hq/synapse#17381), [\#17411](element-hq/synapse#17411)) - Fix building debian packages on non-clean checkouts. ([\#17390](element-hq/synapse#17390)) - Finish up work to allow per-user feature flags. ([\#17392](element-hq/synapse#17392), [\#17410](element-hq/synapse#17410)) - Allow enabling sliding sync per-user. ([\#17393](element-hq/synapse#17393)) ### Updates to locked dependencies * Bump certifi from 2023.7.22 to 2024.7.4. ([\#17404](element-hq/synapse#17404)) * Bump cryptography from 42.0.7 to 42.0.8. ([\#17382](element-hq/synapse#17382)) * Bump ijson from 3.2.3 to 3.3.0. ([\#17413](element-hq/synapse#17413)) * Bump log from 0.4.21 to 0.4.22. ([\#17384](element-hq/synapse#17384)) * Bump mypy-zope from 1.0.4 to 1.0.5. ([\#17414](element-hq/synapse#17414)) * Bump pillow from 10.3.0 to 10.4.0. ([\#17412](element-hq/synapse#17412)) * Bump pydantic from 2.7.1 to 2.8.2. ([\#17415](element-hq/synapse#17415)) * Bump ruff from 0.3.7 to 0.5.0. ([\#17381](element-hq/synapse#17381)) * Bump serde from 1.0.203 to 1.0.204. ([\#17409](element-hq/synapse#17409)) * Bump serde_json from 1.0.117 to 1.0.120. ([\#17385](element-hq/synapse#17385), [\#17408](element-hq/synapse#17408)) * Bump types-setuptools from 69.5.0.20240423 to 70.1.0.20240627. ([\#17380](element-hq/synapse#17380)) # Synapse 1.110.0 (2024-07-03) No significant changes since 1.110.0rc3. # Synapse 1.110.0rc3 (2024-07-02) ### Bugfixes - Fix bug where `/sync` requests could get blocked indefinitely after an upgrade from Synapse versions before v1.109.0. ([\#17386](element-hq/synapse#17386), [\#17391](element-hq/synapse#17391)) ### Internal Changes - Limit size of presence EDUs to 50 entries. ([\#17371](element-hq/synapse#17371)) - Fix building debian package for debian sid. ([\#17389](element-hq/synapse#17389)) # Synapse 1.110.0rc2 (2024-06-26) ### Internal Changes - Fix uploading packages to PyPi. ([\#17363](element-hq/synapse#17363)) # Synapse 1.110.0rc1 (2024-06-26) ### Features - Add initial implementation of an experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17187](element-hq/synapse#17187)) - Add experimental support for [MSC3823](matrix-org/matrix-spec-proposals#3823) - Account suspension. ([\#17255](element-hq/synapse#17255)) - Improve ratelimiting in Synapse. ([\#17256](element-hq/synapse#17256)) - Add support for the unstable [MSC4151](matrix-org/matrix-spec-proposals#4151) report room API. ([\#17270](element-hq/synapse#17270), [\#17296](element-hq/synapse#17296)) - Filter for public and empty rooms added to Admin-API [List Room API](https://element-hq.github.io/synapse/latest/admin_api/rooms.html#list-room-api). ([\#17276](element-hq/synapse#17276)) - Add `is_dm` filtering to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17277](element-hq/synapse#17277)) - Add `is_encrypted` filtering to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17281](element-hq/synapse#17281)) - Include user membership in events served to clients, per [MSC4115](matrix-org/matrix-spec-proposals#4115). ([\#17282](element-hq/synapse#17282)) - Do not require user-interactive authentication for uploading cross-signing keys for the first time, per [MSC3967](matrix-org/matrix-spec-proposals#3967). ([\#17284](element-hq/synapse#17284)) - Add `stream_ordering` sort to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17293](element-hq/synapse#17293)) - `register_new_matrix_user` now supports a --password-file flag, which is useful for scripting. ([\#17294](element-hq/synapse#17294)) - `register_new_matrix_user` now supports a --exists-ok flag to allow registration of users that already exist in the database. This is useful for scripts that bootstrap user accounts with initial passwords. ([\#17304](element-hq/synapse#17304)) - Add support for via query parameter from [MSC4156](matrix-org/matrix-spec-proposals#4156). ([\#17322](element-hq/synapse#17322)) - Add `is_invite` filtering to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17335](element-hq/synapse#17335)) - Support [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md) by adding a federation /download endpoint. ([\#17350](element-hq/synapse#17350)) ### Bugfixes - Fix searching for users with their exact localpart whose ID includes a hyphen. ([\#17254](element-hq/synapse#17254)) - Fix wrong retention policy being used when filtering events. ([\#17272](element-hq/synapse#17272)) - Fix bug where OTKs were not always included in `/sync` response when using workers. ([\#17275](element-hq/synapse#17275)) - Fix a long-standing bug where an invalid 'from' parameter to [`/notifications`](https://spec.matrix.org/v1.10/client-server-api/#get_matrixclientv3notifications) would result in an Internal Server Error. ([\#17283](element-hq/synapse#17283)) - Fix edge case in `/sync` returning the wrong the state when using sharded event persisters. ([\#17295](element-hq/synapse#17295)) - Add initial implementation of an experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17301](element-hq/synapse#17301)) - Fix email notification subject when invited to a space. ([\#17336](element-hq/synapse#17336)) ### Improved Documentation - Add missing quotes for example for `exclude_rooms_from_sync`. ([\#17308](element-hq/synapse#17308)) - Update header in the README to visually fix the the auto-generated table of contents. ([\#17329](element-hq/synapse#17329)) - Fix stale references to the Foundation's Security Disclosure Policy. ([\#17341](element-hq/synapse#17341)) - Add default values for `rc_invites.per_issuer` to docs. ([\#17347](element-hq/synapse#17347)) - Fix an error in the docs for `search_all_users` parameter under `user_directory`. ([\#17348](element-hq/synapse#17348)) ### Internal Changes - Remove unused `expire_access_token` option in the Synapse Docker config file. Contributed by @AaronDewes. ([\#17198](element-hq/synapse#17198)) - Use fully-qualified `PersistedEventPosition` when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation. ([\#17265](element-hq/synapse#17265)) - Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. ([\#17266](element-hq/synapse#17266)) - Handle OTK uploads off master. ([\#17271](element-hq/synapse#17271)) - Don't try and resync devices for remote users whose servers are marked as down. ([\#17273](element-hq/synapse#17273)) - Re-organize Pydantic models and types used in handlers. ([\#17279](element-hq/synapse#17279)) - Expose the worker instance that persisted the event on `event.internal_metadata.instance_name`. ([\#17300](element-hq/synapse#17300)) - Update the README with Element branding, improve headers and fix the #synapse:matrix.org support room link rendering. ([\#17324](element-hq/synapse#17324)) - Change path of the experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync implementation to `/org.matrix.simplified_msc3575/sync` since our simplified API is slightly incompatible with what's in the current MSC. ([\#17331](element-hq/synapse#17331)) - Handle device lists notifications for large accounts more efficiently in worker mode. ([\#17333](element-hq/synapse#17333), [\#17358](element-hq/synapse#17358)) - Do not block event sending/receiving while calculating large event auth chains. ([\#17338](element-hq/synapse#17338)) - Tidy up `parse_integer` docs and call sites to reflect the fact that they require non-negative integers by default, and bring `parse_integer_from_args` default in alignment. Contributed by Denis Kasak (@dkasak). ([\#17339](element-hq/synapse#17339)) ### Updates to locked dependencies * Bump authlib from 1.3.0 to 1.3.1. ([\#17343](element-hq/synapse#17343)) * Bump dawidd6/action-download-artifact from 3.1.4 to 5. ([\#17289](element-hq/synapse#17289)) * Bump dawidd6/action-download-artifact from 5 to 6. ([\#17313](element-hq/synapse#17313)) * Bump docker/build-push-action from 5 to 6. ([\#17312](element-hq/synapse#17312)) * Bump jinja2 from 3.1.3 to 3.1.4. ([\#17287](element-hq/synapse#17287)) * Bump lazy_static from 1.4.0 to 1.5.0. ([\#17355](element-hq/synapse#17355)) * Bump msgpack from 1.0.7 to 1.0.8. ([\#17317](element-hq/synapse#17317)) * Bump netaddr from 1.2.1 to 1.3.0. ([\#17353](element-hq/synapse#17353)) * Bump packaging from 24.0 to 24.1. ([\#17352](element-hq/synapse#17352)) * Bump phonenumbers from 8.13.37 to 8.13.39. ([\#17315](element-hq/synapse#17315)) * Bump regex from 1.10.4 to 1.10.5. ([\#17290](element-hq/synapse#17290)) * Bump requests from 2.31.0 to 2.32.2. ([\#17345](element-hq/synapse#17345)) * Bump sentry-sdk from 2.1.1 to 2.3.1. ([\#17263](element-hq/synapse#17263)) * Bump sentry-sdk from 2.3.1 to 2.6.0. ([\#17351](element-hq/synapse#17351)) * Bump tornado from 6.4 to 6.4.1. ([\#17344](element-hq/synapse#17344)) * Bump mypy from 1.8.0 to 1.9.0. ([\#17297](element-hq/synapse#17297)) * Bump types-jsonschema from 4.21.0.20240311 to 4.22.0.20240610. ([\#17288](element-hq/synapse#17288)) * Bump types-netaddr from 1.2.0.20240219 to 1.3.0.20240530. ([\#17314](element-hq/synapse#17314)) * Bump types-pillow from 10.2.0.20240423 to 10.2.0.20240520. ([\#17285](element-hq/synapse#17285)) * Bump types-pyyaml from 6.0.12.12 to 6.0.12.20240311. ([\#17316](element-hq/synapse#17316)) * Bump typing-extensions from 4.11.0 to 4.12.2. ([\#17354](element-hq/synapse#17354)) * Bump urllib3 from 2.0.7 to 2.2.2. ([\#17346](element-hq/synapse#17346))

H-Shay added 5 commits July 10, 2024 21:18

add column authenticated to tables local_media_repository and `re…

99840b3

…mote_media_cache`

add config options to set and enforce authenticated media

7569c77

get and set authenticated status when storing/fetching local/remote m…

f5ec081

…edia

disallow authenticated media over unauthenticated endpoints if config…

87ae2c2

… demands it

tests

0c787b4

H-Shay requested a review from a team as a code owner July 11, 2024 04:23

newsfragment

11cfc0a

H-Shay commented Jul 11, 2024

View reviewed changes

H-Shay changed the title ~~Prepare for authenticated media~~ Prepare for authenticated media freeze Jul 11, 2024

reivilibre requested changes Jul 16, 2024

View reviewed changes

reivilibre reviewed Jul 16, 2024

View reviewed changes

synapse/storage/schema/main/delta/86/01_authenticate_media.sql Show resolved Hide resolved

requested changes

bfee314

github-actions bot deployed to PR Documentation Preview July 16, 2024 23:46 Active

H-Shay requested a review from reivilibre July 17, 2024 00:48

reivilibre approved these changes Jul 17, 2024

View reviewed changes

request doc changes

f532149

github-actions bot deployed to PR Documentation Preview July 17, 2024 18:03 Active

reivilibre merged commit dc8ddc6 into element-hq:develop Jul 22, 2024
41 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prepare for authenticated media freeze #17433

Prepare for authenticated media freeze #17433

H-Shay commented Jul 11, 2024

H-Shay Jul 11, 2024 •

edited

Loading

reivilibre Jul 11, 2024

erikjohnston Jul 16, 2024

erikjohnston Jul 16, 2024

H-Shay Jul 16, 2024

H-Shay commented Jul 11, 2024

erikjohnston commented Jul 16, 2024

reivilibre left a comment

reivilibre Jul 16, 2024

H-Shay Jul 16, 2024

erikjohnston Jul 17, 2024

sandhose Jul 17, 2024

erikjohnston Jul 17, 2024

reivilibre left a comment

reivilibre Jul 17, 2024

reivilibre Jul 17, 2024

reivilibre Jul 17, 2024

reivilibre Jul 17, 2024

reivilibre Jul 17, 2024

H-Shay Jul 17, 2024

turt2live Jul 23, 2024 •

edited

Loading

turt2live Jul 24, 2024

H-Shay commented Jul 17, 2024

		ALTER TABLE remote_media_cache ADD COLUMN authenticated BOOLEAN;
		ALTER TABLE local_media_repository ADD COLUMN authenticated BOOLEAN;


		When set to true, all subsequent media uploads will be marked as authenticated, and will not be available over legacy
		unauthenticated media endpoints (`/_matrix/media/(r0\|v3\|v1)/download` and `/_matrix/media/(r0\|v3\|v1)/thumbnail`) - requests for authenticated media over these endpoints will result in a 404. All media, including authenticated media, will be available over the authenticated media endpoints `_matrix/client/v1/media/download` and `_matrix/client/v1/media/thumbnail`. Media uploaded prior to setting this option to true will still be available over the legacy endpoints.


	Defaults to false, but this will change to true in a future Synapse release.

Prepare for authenticated media freeze #17433

Prepare for authenticated media freeze #17433

Conversation

H-Shay commented Jul 11, 2024

H-Shay Jul 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

H-Shay commented Jul 11, 2024

erikjohnston commented Jul 16, 2024

reivilibre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reivilibre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turt2live Jul 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

H-Shay commented Jul 17, 2024

H-Shay Jul 11, 2024 •

edited

Loading

turt2live Jul 23, 2024 •

edited

Loading