Support rendering previews with data: URLs in them #11767

clokep · 2022-01-18T20:40:50Z

I noticed while reviewing/testing #11669 that if a webpage has a data: on it (which we choose for the og:image response) then the URL preview breaks. An example page for this is https://www.reddit.com/r/matrixdotorg/comments/s5m47u/cant_connect_to_my_home_server_in_element/

The image which ends up being chosen is the reddit logo, which they embed as a data: URL. After this PR we get a working preview:

The only thing that is a bit weird about the current implementation is that it will allow Synapse to directly render data: URLs if requests, e.g. using the "HTML document" example from MDN and URL encoding it: This is no longer true and only image or oEmbed URLs found in HTML would be parsed if they're data: URLs.

$ curl --header "Authorization: Bearer $TOK" http://localhost:8080/_matrix/media/r0/preview_url\?url\=data%3Atext%2Fhtml%2C%253Ch1%253EHello%252C%2520World%2521%253C%252Fh1%253E
{"og:title":"Hello, World!","og:description":"Hello, World!"}

~~I don't think there's any risk to doing this since we're not rendering JavaScript, but it feels a bit icky.~~

The code should probably handle if you embed something weird (e.g. HTML) as the src of the image, however. (Note that "other" schemes usually get rejected via deep in Twisted internals for Agent). Without this change the error is obscure since the rebase_url method is broken and you end up trying to preview data:www.reddit.com/r/matrixdotorg/comments/s5m47u/cant_connect_to_my_home_server_in_element/image/png;base64,<the base 64 data>.

Should be reviewable commit-by-commit.

synapse/rest/media/v1/preview_url_resource.py

clokep · 2022-01-18T20:48:34Z

synapse/rest/media/v1/preview_url_resource.py

+                    download_name,
+                    expires,
+                    etag,
+                ) = await self._parse_data_url(url, f)
            else:


We might want to assert HTTP(S) schemes here as additional error checking.

Yeah I feel like I'd be happier having a few select allowed protocols rather than singling out data:; not sure it really makes sense to ask us to preview an ftp: URI either for example.

Yep, note that those will fail right now (we won't try them), but it would be better to be explicit, I think!

I think this might actually be tougher to get correct than I initially though since we seem to support previewing scheme-less URLs. I'm not sure it makes sense to refactor that as part of this.

Yeah, fair enough.

clokep · 2022-01-18T21:52:41Z

The only thing that is a bit weird about the current implementation is that it will allow Synapse to directly render data: URLs if requests

We could probably avoid this with a flag, now that I think a bit more about it. So directly given URLs couldn't be data:. I'm not sure if this offers any real protection though.

DMRobertson

A few drive-by comments.

Not sure of the security ramifications here; I'll stick into the queue for a second opinion. But FWIW: instead of downloading an arbitrary data blob from someone else, we have an arbitrary binary blob provided to us via a URL---is there any difference? If Synapse itself was processing the data blob generically, then I could forsee problems like the billion laughs attack or some invalid image data designed to trip up an image renderer.

I agree with you on the ickiness. It does seem... odd and confused to allow someone to say "please preview this data that I already have".

synapse/rest/media/v1/preview_html.py

DMRobertson · 2022-01-19T18:36:57Z

synapse/rest/media/v1/preview_url_resource.py

+
+        length = output_stream.seek(1)
+
+        media_type = url_info.headers.get_content_type()


Should we cross reference this with the mime-type in the data url?

(Do we want to have some kind of whitelist of trusted mime types which are okay to preview?)

This is the mime-type from the data URL, it gets parsed out for us and shoved into a headers property for some reason.

The docs for urlopen have some info on how data URLs are handled:

For FTP, file, and data URLs [...], this function returns a urllib.response.addinfourl object.

The addinfourl object has a headers property:

Returns the headers of the response in the form of an EmailMessage instance.

The EmailMessage object has a get_content_type() property:

Return the message’s content type, coerced to lower case of the form maintype/subtype. If there is no Content-Type header in the message return the value returned by get_default_type(). If the Content-Type header is invalid, return text/plain.

This all doesn't really make sense for data URLs, but from trial and error it seems to do the correct thing.

I added a clarifying comment in 116015f -- hopefully that helps?

synapse/rest/media/v1/preview_url_resource.py

clokep · 2022-01-19T20:10:18Z

Not sure of the security ramifications here; I'll stick into the queue for a second opinion. But FWIW: instead of downloading an arbitrary data blob from someone else, we have an arbitrary binary blob provided to us via a URL---is there any difference? If Synapse itself was processing the data blob generically, then I could forsee problems like the billion laughs attack or some invalid image data designed to trip up an image renderer.

I agree with you on the ickiness. It does seem... odd and confused to allow someone to say "please preview this data that I already have".

Thanks for the review! I agree that it isn't too different from what we're doing. It is worth investigating how urlopen would work in some situations though (I'll take a look at that!) In particular this might not handle our max download size properly...but if it is embedded in something we already downloaded then by definition it must be smaller than the max download size. I think.

I've updated the PR to not allow directly previewing data: URLs because I don't think there's a valid use-case there.

reivilibre

Seems sensible enough — just a few points

reivilibre · 2022-01-21T09:47:34Z

synapse/rest/media/v1/preview_html.py

@@ -321,14 +321,33 @@ def _iterate_over_text(


 def rebase_url(url: str, base: str) -> str:
-    base_parts = list(urlparse.urlparse(base))
+    """
+    Resolves a potentially relative `url` against an absolute `base` URL.


Do you know how this differs from urljoin in the standard lib?
Not necessarily opposed to rolling our own, but it'd be nice to know if we had specific motivation for doing so.

I didn't know of it, but I'll look into it!

I think this will work OK, but I'd rather punt this to a separate PR since it is pretty unrelated to this work. Would that be OK?

synapse/rest/media/v1/preview_url_resource.py

reivilibre · 2022-01-21T09:49:43Z

synapse/rest/media/v1/preview_url_resource.py

+                    download_name,
+                    expires,
+                    etag,
+                ) = await self._parse_data_url(url, f)
            else:


Yeah I feel like I'd be happier having a few select allowed protocols rather than singling out data:; not sure it really makes sense to ask us to preview an ftp: URI either for example.

reivilibre

Thanks!

@totallynotvaishnav

Synapse 1.52.0rc1 (2022-02-01) ============================== Features -------- - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11621](matrix-org/synapse#11621), [\#11788](matrix-org/synapse#11788), [\#11789](matrix-org/synapse#11789)) - Add an admin API to reset connection timeouts for remote server. ([\#11639](matrix-org/synapse#11639)) - Add an admin API to get a list of rooms that federate with a given remote homeserver. ([\#11658](matrix-org/synapse#11658)) - Add a config flag to inhibit M_USER_IN_USE during registration. ([\#11743](matrix-org/synapse#11743)) - Add a module callback to set username at registration. ([\#11790](matrix-org/synapse#11790)) - Allow configuring a maximum file size as well as a list of allowed content types for avatars. ([\#11846](matrix-org/synapse#11846)) Bugfixes -------- - Include the bundled aggregations in the `/sync` response, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\#11612](matrix-org/synapse#11612)) - Fix a long-standing bug when previewing Reddit URLs which do not contain an image. ([\#11767](matrix-org/synapse#11767)) - Fix a long-standing bug that media streams could cause long-lived connections when generating URL previews. ([\#11784](matrix-org/synapse#11784)) - Include a `prev_content` field in state events sent to Application Services. Contributed by @totallynotvaishnav. ([\#11798](matrix-org/synapse#11798)) - Fix a bug introduced in Synapse 0.33.3 causing requests to sometimes log strings such as `HTTPStatus.OK` instead of integer status codes. ([\#11827](matrix-org/synapse#11827)) Improved Documentation ---------------------- - Update pypi installation docs to indicate that we now support Python 3.10. ([\#11820](matrix-org/synapse#11820)) - Add missing steps to the contribution submission process in the documentation. Contributed by @sequentialread. ([\#11821](matrix-org/synapse#11821)) - Remove not needed old table of contents in documentation. ([\#11860](matrix-org/synapse#11860)) - Consolidate the `access_token` information at the top of each relevant page in the Admin API documentation. ([\#11861](matrix-org/synapse#11861)) Deprecations and Removals ------------------------- - Drop support for Python 3.6, which is EOL. ([\#11683](matrix-org/synapse#11683)) - Remove the `experimental_msc1849_support_enabled` flag as the features are now stable. ([\#11843](matrix-org/synapse#11843)) Internal Changes ---------------- - Preparation for database schema simplifications: add `state_key` and `rejection_reason` columns to `events` table. ([\#11792](matrix-org/synapse#11792)) - Add `FrozenEvent.get_state_key` and use it in a couple of places. ([\#11793](matrix-org/synapse#11793)) - Preparation for database schema simplifications: stop reading from `event_reference_hashes`. ([\#11794](matrix-org/synapse#11794)) - Drop unused table `public_room_list_stream`. ([\#11795](matrix-org/synapse#11795)) - Preparation for reducing Postgres serialization errors: allow setting transaction isolation level. Contributed by Nick @ Beeper. ([\#11799](matrix-org/synapse#11799), [\#11847](matrix-org/synapse#11847)) - Docker: skip the initial amd64-only build and go straight to multiarch. ([\#11810](matrix-org/synapse#11810)) - Run Complement on the Github Actions VM and not inside a Docker container. ([\#11811](matrix-org/synapse#11811)) - Log module names at startup. ([\#11813](matrix-org/synapse#11813)) - Improve type safety of bundled aggregations code. ([\#11815](matrix-org/synapse#11815)) - Correct a type annotation in the event validation logic. ([\#11817](matrix-org/synapse#11817), [\#11830](matrix-org/synapse#11830)) - Minor updates and documentation for database schema delta files. ([\#11823](matrix-org/synapse#11823)) - Workaround a type annotation problem in `prometheus_client` 0.13.0. ([\#11834](matrix-org/synapse#11834)) - Minor performance improvement in room state lookup. ([\#11836](matrix-org/synapse#11836)) - Fix some indentation inconsistencies in the sample config. ([\#11838](matrix-org/synapse#11838)) - Add type hints to `tests/rest/admin`. ([\#11851](matrix-org/synapse#11851))

@totallynotvaishnav

Synapse 1.52.0 (2022-02-08) =========================== No significant changes since 1.52.0rc1. During the making of this release, the developers of Twisted have released [Twisted 22.1.0](https://github.com/twisted/twisted/releases/tag/twisted-22.1.0), which fixes [a security issue](GHSA-92x2-jw7w-xvvx) within Twisted. We do not believe Synapse to be vulnerable to any security problem caused by this issue, though we advise server administrators to update their local version of Twisted if they can. Synapse 1.52.0rc1 (2022-02-01) ============================== Features -------- - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11621](matrix-org/synapse#11621), [\#11788](matrix-org/synapse#11788), [\#11789](matrix-org/synapse#11789)) - Add an admin API to reset connection timeouts for remote server. ([\#11639](matrix-org/synapse#11639)) - Add an admin API to get a list of rooms that federate with a given remote homeserver. ([\#11658](matrix-org/synapse#11658)) - Add a config flag to inhibit `M_USER_IN_USE` during registration. ([\#11743](matrix-org/synapse#11743)) - Add a module callback to set username at registration. ([\#11790](matrix-org/synapse#11790)) - Allow configuring a maximum file size as well as a list of allowed content types for avatars. ([\#11846](matrix-org/synapse#11846)) Bugfixes -------- - Include the bundled aggregations in the `/sync` response, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\#11612](matrix-org/synapse#11612)) - Fix a long-standing bug when previewing Reddit URLs which do not contain an image. ([\#11767](matrix-org/synapse#11767)) - Fix a long-standing bug that media streams could cause long-lived connections when generating URL previews. ([\#11784](matrix-org/synapse#11784)) - Include a `prev_content` field in state events sent to Application Services. Contributed by @totallynotvaishnav. ([\#11798](matrix-org/synapse#11798)) - Fix a bug introduced in Synapse 0.33.3 causing requests to sometimes log strings such as `HTTPStatus.OK` instead of integer status codes. ([\#11827](matrix-org/synapse#11827)) Improved Documentation ---------------------- - Update pypi installation docs to indicate that we now support Python 3.10. ([\#11820](matrix-org/synapse#11820)) - Add missing steps to the contribution submission process in the documentation. Contributed by @sequentialread. ([\#11821](matrix-org/synapse#11821)) - Remove not needed old table of contents in documentation. ([\#11860](matrix-org/synapse#11860)) - Consolidate the `access_token` information at the top of each relevant page in the Admin API documentation. ([\#11861](matrix-org/synapse#11861)) Deprecations and Removals ------------------------- - Drop support for Python 3.6, which is EOL. ([\#11683](matrix-org/synapse#11683)) - Remove the `experimental_msc1849_support_enabled` flag as the features are now stable. ([\#11843](matrix-org/synapse#11843)) Internal Changes ---------------- - Preparation for database schema simplifications: add `state_key` and `rejection_reason` columns to `events` table. ([\#11792](matrix-org/synapse#11792)) - Add `FrozenEvent.get_state_key` and use it in a couple of places. ([\#11793](matrix-org/synapse#11793)) - Preparation for database schema simplifications: stop reading from `event_reference_hashes`. ([\#11794](matrix-org/synapse#11794)) - Drop unused table `public_room_list_stream`. ([\#11795](matrix-org/synapse#11795)) - Preparation for reducing Postgres serialization errors: allow setting transaction isolation level. Contributed by Nick @ Beeper. ([\#11799](matrix-org/synapse#11799), [\#11847](matrix-org/synapse#11847)) - Docker: skip the initial amd64-only build and go straight to multiarch. ([\#11810](matrix-org/synapse#11810)) - Run Complement on the Github Actions VM and not inside a Docker container. ([\#11811](matrix-org/synapse#11811)) - Log module names at startup. ([\#11813](matrix-org/synapse#11813)) - Improve type safety of bundled aggregations code. ([\#11815](matrix-org/synapse#11815)) - Correct a type annotation in the event validation logic. ([\#11817](matrix-org/synapse#11817), [\#11830](matrix-org/synapse#11830)) - Minor updates and documentation for database schema delta files. ([\#11823](matrix-org/synapse#11823)) - Workaround a type annotation problem in `prometheus_client` 0.13.0. ([\#11834](matrix-org/synapse#11834)) - Minor performance improvement in room state lookup. ([\#11836](matrix-org/synapse#11836)) - Fix some indentation inconsistencies in the sample config. ([\#11838](matrix-org/synapse#11838)) - Add type hints to `tests/rest/admin`. ([\#11851](matrix-org/synapse#11851))

@totallynotvaishnav

Synapse 1.53.0 (2022-02-22) =========================== No significant changes. Synapse 1.53.0rc1 (2022-02-15) ============================== Features -------- - Add experimental support for sending to-device messages to application services, as specified by [MSC2409](matrix-org/matrix-spec-proposals#2409). ([\#11215](matrix-org/synapse#11215), [\#11966](matrix-org/synapse#11966)) - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11655](matrix-org/synapse#11655)) - Experimental support for [MSC3666](matrix-org/matrix-spec-proposals#3666): including bundled aggregations in server side search results. ([\#11837](matrix-org/synapse#11837)) - Enable cache time-based expiry by default. The `expiry_time` config flag has been superseded by `expire_caches` and `cache_entry_ttl`. ([\#11849](matrix-org/synapse#11849)) - Add a callback to allow modules to allow or forbid a 3PID (email address, phone number) from being associated to a local account. ([\#11854](matrix-org/synapse#11854)) - Stabilize support and remove unstable endpoints for [MSC3231](matrix-org/matrix-spec-proposals#3231). Clients must switch to the stable identifier and endpoint. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#stablisation-of-msc3231) for more information. ([\#11867](matrix-org/synapse#11867)) - Allow modules to retrieve the current instance's server name and worker name. ([\#11868](matrix-org/synapse#11868)) - Use a dedicated configurable rate limiter for 3PID invites. ([\#11892](matrix-org/synapse#11892)) - Support the stable API endpoint for [MSC3283](matrix-org/matrix-spec-proposals#3283): new settings in `/capabilities` endpoint. ([\#11933](matrix-org/synapse#11933), [\#11989](matrix-org/synapse#11989)) - Support the `dir` parameter on the `/relations` endpoint, per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#11941](matrix-org/synapse#11941)) - Experimental implementation of [MSC3706](matrix-org/matrix-spec-proposals#3706): extensions to `/send_join` to support reduced response size. ([\#11967](matrix-org/synapse#11967)) Bugfixes -------- - Fix [MSC2716](matrix-org/matrix-spec-proposals#2716) historical messages backfilling in random order on remote homeservers. ([\#11114](matrix-org/synapse#11114)) - Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#11890](matrix-org/synapse#11890)) - Fix a long-standing bug where some unknown endpoints would return HTML error pages instead of JSON `M_UNRECOGNIZED` errors. ([\#11930](matrix-org/synapse#11930)) - Implement an allow list of content types for which we will attempt to preview a URL. This prevents Synapse from making useless longer-lived connections to streaming media servers. ([\#11936](matrix-org/synapse#11936)) - Fix a long-standing bug where pagination tokens from `/sync` and `/messages` could not be provided to the `/relations` API. ([\#11952](matrix-org/synapse#11952)) - Require that modules register their callbacks using keyword arguments. ([\#11975](matrix-org/synapse#11975)) - Fix a long-standing bug where `M_WRONG_ROOM_KEYS_VERSION` errors would not include the specced `current_version` field. ([\#11988](matrix-org/synapse#11988)) Improved Documentation ---------------------- - Fix typo in User Admin API: unpind -> unbind. ([\#11859](matrix-org/synapse#11859)) - Document images returned by the User List Media Admin API can include those generated by URL previews. ([\#11862](matrix-org/synapse#11862)) - Remove outdated MSC1711 FAQ document. ([\#11907](matrix-org/synapse#11907)) - Correct the structured logging configuration example. Contributed by Brad Jones. ([\#11946](matrix-org/synapse#11946)) - Add information on the Synapse release cycle. ([\#11954](matrix-org/synapse#11954)) - Fix broken link in the README to the admin API for password reset. ([\#11955](matrix-org/synapse#11955)) Deprecations and Removals ------------------------- - Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error. ([\#11895](matrix-org/synapse#11895)) - Remove deprecated `user_may_create_room_with_invites` spam checker callback. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#removal-of-user_may_create_room_with_invites) for more information. ([\#11950](matrix-org/synapse#11950)) - No longer build `.deb` packages for Ubuntu 21.04 Hirsute Hippo, which has now EOLed. ([\#11961](matrix-org/synapse#11961)) Internal Changes ---------------- - Enhance user registration test helpers to make them more useful for tests involving application services and devices. ([\#11615](matrix-org/synapse#11615), [\#11616](matrix-org/synapse#11616)) - Improve performance when fetching bundled aggregations for multiple events. ([\#11660](matrix-org/synapse#11660), [\#11752](matrix-org/synapse#11752)) - Fix type errors introduced by new annotations in the Prometheus Client library. ([\#11832](matrix-org/synapse#11832)) - Add missing type hints to replication code. ([\#11856](matrix-org/synapse#11856), [\#11938](matrix-org/synapse#11938)) - Ensure that `opentracing` scopes are activated and closed at the right time. ([\#11869](matrix-org/synapse#11869)) - Improve opentracing for incoming federation requests. ([\#11870](matrix-org/synapse#11870)) - Improve internal docstrings in `synapse.util.caches`. ([\#11876](matrix-org/synapse#11876)) - Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes. ([\#11878](matrix-org/synapse#11878)) - Convert `ApplicationServiceTestCase` to use `simple_async_mock`. ([\#11880](matrix-org/synapse#11880)) - Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled. ([\#11884](matrix-org/synapse#11884)) - Disable coverage calculation for olddeps build. ([\#11888](matrix-org/synapse#11888)) - Preparation to support sending device list updates to application services. ([\#11905](matrix-org/synapse#11905)) - Add a test that checks users receive their own device list updates down `/sync`. ([\#11909](matrix-org/synapse#11909)) - Run Complement tests sequentially. ([\#11910](matrix-org/synapse#11910)) - Various refactors to the application service notifier code. ([\#11911](matrix-org/synapse#11911), [\#11912](matrix-org/synapse#11912)) - Tests: replace mocked `Authenticator` with the real thing. ([\#11913](matrix-org/synapse#11913)) - Various refactors to the typing notifications code. ([\#11914](matrix-org/synapse#11914)) - Use the proper type for the `Content-Length` header in the `UploadResource`. ([\#11927](matrix-org/synapse#11927)) - Remove an unnecessary ignoring of type hints due to fixes in upstream packages. ([\#11939](matrix-org/synapse#11939)) - Add missing type hints. ([\#11953](matrix-org/synapse#11953)) - Fix an import cycle in `synapse.event_auth`. ([\#11965](matrix-org/synapse#11965)) - Unpin `frozendict` but exclude the known bad version 2.1.2. ([\#11969](matrix-org/synapse#11969)) - Prepare for rename of default Complement branch. ([\#11971](matrix-org/synapse#11971)) - Fetch Synapse's version using a helper from `matrix-common`. ([\#11979](matrix-org/synapse#11979)) Synapse 1.52.0 (2022-02-08) =========================== No significant changes since 1.52.0rc1. Note that [Twisted 22.1.0](https://github.com/twisted/twisted/releases/tag/twisted-22.1.0) has recently been released, which fixes a [security issue](GHSA-92x2-jw7w-xvvx) within the Twisted library. We do not believe Synapse is affected by this vulnerability, though we advise server administrators who installed Synapse via pip to upgrade Twisted with `pip install --upgrade Twisted` as a matter of good practice. The Docker image `matrixdotorg/synapse` and the Debian packages from `packages.matrix.org` are using the updated library. Synapse 1.52.0rc1 (2022-02-01) ============================== Features -------- - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11621](matrix-org/synapse#11621), [\#11788](matrix-org/synapse#11788), [\#11789](matrix-org/synapse#11789)) - Add an admin API to reset connection timeouts for remote server. ([\#11639](matrix-org/synapse#11639)) - Add an admin API to get a list of rooms that federate with a given remote homeserver. ([\#11658](matrix-org/synapse#11658)) - Add a config flag to inhibit `M_USER_IN_USE` during registration. ([\#11743](matrix-org/synapse#11743)) - Add a module callback to set username at registration. ([\#11790](matrix-org/synapse#11790)) - Allow configuring a maximum file size as well as a list of allowed content types for avatars. ([\#11846](matrix-org/synapse#11846)) Bugfixes -------- - Include the bundled aggregations in the `/sync` response, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\#11612](matrix-org/synapse#11612)) - Fix a long-standing bug when previewing Reddit URLs which do not contain an image. ([\#11767](matrix-org/synapse#11767)) - Fix a long-standing bug that media streams could cause long-lived connections when generating URL previews. ([\#11784](matrix-org/synapse#11784)) - Include a `prev_content` field in state events sent to Application Services. Contributed by @totallynotvaishnav. ([\#11798](matrix-org/synapse#11798)) - Fix a bug introduced in Synapse 0.33.3 causing requests to sometimes log strings such as `HTTPStatus.OK` instead of integer status codes. ([\#11827](matrix-org/synapse#11827)) Improved Documentation ---------------------- - Update pypi installation docs to indicate that we now support Python 3.10. ([\#11820](matrix-org/synapse#11820)) - Add missing steps to the contribution submission process in the documentation. Contributed by @sequentialread. ([\#11821](matrix-org/synapse#11821)) - Remove not needed old table of contents in documentation. ([\#11860](matrix-org/synapse#11860)) - Consolidate the `access_token` information at the top of each relevant page in the Admin API documentation. ([\#11861](matrix-org/synapse#11861)) Deprecations and Removals ------------------------- - Drop support for Python 3.6, which is EOL. ([\#11683](matrix-org/synapse#11683)) - Remove the `experimental_msc1849_support_enabled` flag as the features are now stable. ([\#11843](matrix-org/synapse#11843)) Internal Changes ---------------- - Preparation for database schema simplifications: add `state_key` and `rejection_reason` columns to `events` table. ([\#11792](matrix-org/synapse#11792)) - Add `FrozenEvent.get_state_key` and use it in a couple of places. ([\#11793](matrix-org/synapse#11793)) - Preparation for database schema simplifications: stop reading from `event_reference_hashes`. ([\#11794](matrix-org/synapse#11794)) - Drop unused table `public_room_list_stream`. ([\#11795](matrix-org/synapse#11795)) - Preparation for reducing Postgres serialization errors: allow setting transaction isolation level. Contributed by Nick @ Beeper. ([\#11799](matrix-org/synapse#11799), [\#11847](matrix-org/synapse#11847)) - Docker: skip the initial amd64-only build and go straight to multiarch. ([\#11810](matrix-org/synapse#11810)) - Run Complement on the Github Actions VM and not inside a Docker container. ([\#11811](matrix-org/synapse#11811)) - Log module names at startup. ([\#11813](matrix-org/synapse#11813)) - Improve type safety of bundled aggregations code. ([\#11815](matrix-org/synapse#11815)) - Correct a type annotation in the event validation logic. ([\#11817](matrix-org/synapse#11817), [\#11830](matrix-org/synapse#11830)) - Minor updates and documentation for database schema delta files. ([\#11823](matrix-org/synapse#11823)) - Workaround a type annotation problem in `prometheus_client` 0.13.0. ([\#11834](matrix-org/synapse#11834)) - Minor performance improvement in room state lookup. ([\#11836](matrix-org/synapse#11836)) - Fix some indentation inconsistencies in the sample config. ([\#11838](matrix-org/synapse#11838)) - Add type hints to `tests/rest/admin`. ([\#11851](matrix-org/synapse#11851))

@totallynotvaishnav

Synapse 1.52.0 (2022-02-08) =========================== No significant changes since 1.52.0rc1. During the making of this release, the developers of Twisted have released [Twisted 22.1.0](https://github.com/twisted/twisted/releases/tag/twisted-22.1.0), which fixes [a security issue](GHSA-92x2-jw7w-xvvx) within Twisted. We do not believe Synapse to be vulnerable to any security problem caused by this issue, though we advise server administrators to update their local version of Twisted if they can. Synapse 1.52.0rc1 (2022-02-01) ============================== Features -------- - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\matrix-org#11621](matrix-org#11621), [\matrix-org#11788](matrix-org#11788), [\matrix-org#11789](matrix-org#11789)) - Add an admin API to reset connection timeouts for remote server. ([\matrix-org#11639](matrix-org#11639)) - Add an admin API to get a list of rooms that federate with a given remote homeserver. ([\matrix-org#11658](matrix-org#11658)) - Add a config flag to inhibit `M_USER_IN_USE` during registration. ([\matrix-org#11743](matrix-org#11743)) - Add a module callback to set username at registration. ([\matrix-org#11790](matrix-org#11790)) - Allow configuring a maximum file size as well as a list of allowed content types for avatars. ([\matrix-org#11846](matrix-org#11846)) Bugfixes -------- - Include the bundled aggregations in the `/sync` response, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\matrix-org#11612](matrix-org#11612)) - Fix a long-standing bug when previewing Reddit URLs which do not contain an image. ([\matrix-org#11767](matrix-org#11767)) - Fix a long-standing bug that media streams could cause long-lived connections when generating URL previews. ([\matrix-org#11784](matrix-org#11784)) - Include a `prev_content` field in state events sent to Application Services. Contributed by @totallynotvaishnav. ([\matrix-org#11798](matrix-org#11798)) - Fix a bug introduced in Synapse 0.33.3 causing requests to sometimes log strings such as `HTTPStatus.OK` instead of integer status codes. ([\matrix-org#11827](matrix-org#11827)) Improved Documentation ---------------------- - Update pypi installation docs to indicate that we now support Python 3.10. ([\matrix-org#11820](matrix-org#11820)) - Add missing steps to the contribution submission process in the documentation. Contributed by @sequentialread. ([\matrix-org#11821](matrix-org#11821)) - Remove not needed old table of contents in documentation. ([\matrix-org#11860](matrix-org#11860)) - Consolidate the `access_token` information at the top of each relevant page in the Admin API documentation. ([\matrix-org#11861](matrix-org#11861)) Deprecations and Removals ------------------------- - Drop support for Python 3.6, which is EOL. ([\matrix-org#11683](matrix-org#11683)) - Remove the `experimental_msc1849_support_enabled` flag as the features are now stable. ([\matrix-org#11843](matrix-org#11843)) Internal Changes ---------------- - Preparation for database schema simplifications: add `state_key` and `rejection_reason` columns to `events` table. ([\matrix-org#11792](matrix-org#11792)) - Add `FrozenEvent.get_state_key` and use it in a couple of places. ([\matrix-org#11793](matrix-org#11793)) - Preparation for database schema simplifications: stop reading from `event_reference_hashes`. ([\matrix-org#11794](matrix-org#11794)) - Drop unused table `public_room_list_stream`. ([\matrix-org#11795](matrix-org#11795)) - Preparation for reducing Postgres serialization errors: allow setting transaction isolation level. Contributed by Nick @ Beeper. ([\matrix-org#11799](matrix-org#11799), [\matrix-org#11847](matrix-org#11847)) - Docker: skip the initial amd64-only build and go straight to multiarch. ([\matrix-org#11810](matrix-org#11810)) - Run Complement on the Github Actions VM and not inside a Docker container. ([\matrix-org#11811](matrix-org#11811)) - Log module names at startup. ([\matrix-org#11813](matrix-org#11813)) - Improve type safety of bundled aggregations code. ([\matrix-org#11815](matrix-org#11815)) - Correct a type annotation in the event validation logic. ([\matrix-org#11817](matrix-org#11817), [\matrix-org#11830](matrix-org#11830)) - Minor updates and documentation for database schema delta files. ([\matrix-org#11823](matrix-org#11823)) - Workaround a type annotation problem in `prometheus_client` 0.13.0. ([\matrix-org#11834](matrix-org#11834)) - Minor performance improvement in room state lookup. ([\matrix-org#11836](matrix-org#11836)) - Fix some indentation inconsistencies in the sample config. ([\matrix-org#11838](matrix-org#11838)) - Add type hints to `tests/rest/admin`. ([\matrix-org#11851](matrix-org#11851))

clokep added 5 commits January 18, 2022 14:31

Add a comment.

ff7c377

Rename _download_url to _handle_url.

5b09001

Split downloading a URL from storing it.

fb448dd

Handle data URLs.

9b98fea

Rename test file.

f07df94

clokep requested a review from a team as a code owner January 18, 2022 20:40

clokep added 2 commits January 18, 2022 15:42

Newsfragment

6cd9b5b

Fix-up comments.

88917c0

clokep commented Jan 18, 2022

View reviewed changes

synapse/rest/media/v1/preview_url_resource.py Outdated Show resolved Hide resolved

clokep commented Jan 18, 2022

View reviewed changes

DMRobertson reviewed Jan 19, 2022

View reviewed changes

DMRobertson requested a review from a team January 19, 2022 18:38

clokep added 6 commits January 19, 2022 14:31

Properly use seek() APIs.

b9a1c66

Clarify where media_type comes from.

116015f

Add docstring and tests for rebase_url.

3b38444

Do not convert base_parts to a list.

e042244

Merge remote-tracking branch 'origin/develop' into clokep/data-urls

deeec27

Do not support directly previewing data URLs.

5124a94

Assert the image size.

d38a07c

reivilibre suggested changes Jan 21, 2022

View reviewed changes

Use a class for return values.

043b4c1

clokep requested a review from reivilibre January 21, 2022 18:22

Merge remote-tracking branch 'origin/develop' into clokep/data-urls

2042da1

reivilibre approved these changes Jan 24, 2022

View reviewed changes

clokep merged commit 807efd2 into develop Jan 24, 2022

clokep deleted the clokep/data-urls branch January 24, 2022 13:58

clokep mentioned this pull request Mar 14, 2022

Clean-up logic for rebasing URLs #12219

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support rendering previews with data: URLs in them #11767

Support rendering previews with data: URLs in them #11767

clokep commented Jan 18, 2022 •

edited

Loading

clokep Jan 18, 2022

reivilibre Jan 21, 2022

clokep Jan 21, 2022

clokep Jan 21, 2022

reivilibre Jan 24, 2022

clokep commented Jan 18, 2022

DMRobertson left a comment

DMRobertson Jan 19, 2022

clokep Jan 19, 2022

clokep Jan 19, 2022

clokep commented Jan 19, 2022

reivilibre left a comment

reivilibre Jan 21, 2022

clokep Jan 21, 2022

clokep Jan 21, 2022

reivilibre Jan 24, 2022

reivilibre Jan 21, 2022

reivilibre left a comment


		length = output_stream.seek(1)

		media_type = url_info.headers.get_content_type()

Support rendering previews with data: URLs in them #11767

Support rendering previews with data: URLs in them #11767

Conversation

clokep commented Jan 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep commented Jan 18, 2022

DMRobertson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep commented Jan 19, 2022

reivilibre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reivilibre left a comment

Choose a reason for hiding this comment

clokep commented Jan 18, 2022 •

edited

Loading