Add config option to prevent media downloads from listed domains. #15197

turt2live · 2023-03-02T22:27:02Z

See config documentation for more details on the feature description.

This could probably be implemented as an antispam or similar module, however this is also tooling that we'd like available to everyone. Instead of trying to bundle a module into every deployment, let's just put it in the core project :)

This is Trust & Safety (T&S) tooling to combat a specific type of abuse. There are no active incidents related to this, but is something we'd like available to us just in case.

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Pull request includes a sign off
Code style is correct
(run the linters)

tests/rest/media/test_domain_blocking.py

synapse/media/media_repository.py

tests/rest/media/test_domain_blocking.py

Twi1ightSparkle · 2023-03-06T15:07:07Z

Will this also support wildcards? I.e.,

prevent_downloads_from:
  - '*.example.org'

Or, would

prevent_downloads_from:
  - example.org

automatically deny example.org and all possible subdomains?

davidegirardi · 2023-03-06T15:12:46Z

Wildcard are really important because server1.example.tld and server2.example.tld could be matrix servers for 2 different domains. Moreover, something.domain.tld and somethingelse.something.domain.tld could be separated zones managed by different organisations.

turt2live · 2023-03-06T15:15:57Z

Wildcards would be a different PR.

davidegirardi · 2023-03-06T15:17:46Z

Since you block servers and not domains (D'OH!) it makes perfect sense.

turt2live · 2023-03-06T16:46:46Z

it's more the cost of implementation versus the benefit: we don't see a particular profile of abuse that needs wildcard bans here, making the implementation+maintenance effort (and associated performance loss) not worth it. If needed, we can always add it in a pinch.

davidegirardi · 2023-03-07T07:07:48Z

I think a list without wildcards makes perfect sense since we target servers, not domains. There's no ambiguity this way.

tests/rest/media/test_domain_blocking.py

reivilibre · 2023-03-29T08:01:46Z

I think a list without wildcards makes perfect sense since we target servers, not domains. There's no ambiguity this way.

though it'd possibly be quite easy to serve up media on wildcard domains. I guess this is just a theoretical risk until someone does, though.

changelog.d/15197.feature

docs/usage/configuration/config_documentation.md

synapse/media/media_repository.py

synapse/rest/media/thumbnail_resource.py

tests/rest/media/test_domain_blocking.py

synapse/config/repository.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

dklimpel · 2023-05-08T18:05:29Z

docs/usage/configuration/config_documentation.md

@@ -1768,6 +1768,30 @@ Example configuration:
 max_image_pixels: 35M
 ```
 ---
+### `prevent_media_downloads_from`


I was just thinking about the name. This is a block list, if you later want to use the concept with allow lists, you have to rename the parameter.

Wouldn't it be better to adapt the name to the current parameters for allowing and disallowing federation or url previews?

Allow lists aren't currently planned for this feature - I'd favour a config migration (deprecation) later on if we decide to go that route.

clokep

LGTM! 🎉

catfromplan9 · 2023-05-12T08:09:59Z

Can clients still pull the media from the host server? I have people using matrix.org for abuse and I do not want to prevent matrix.org downloads nor do I want to email matrix.org every time theres illegal media

catfromplan9 · 2023-05-12T08:27:17Z

The type of abuse described can happen, but it will likely never be media coming from an abusive server. To really prevent this attack, you would need to index every single public matrix server and block media federation. This PR is a great step towards a solution but it needs more fine tuned control such as exists on many ActivityPub platforms

catfromplan9 · 2023-05-12T08:31:03Z

I think adding a boolean federate media flag in addition to this would be beneficial in cases where attackers are determined. Then implement a client option as I describe in my Element feature suggestion to disable fetching media federated or a server request to client to fetch media using the server the media is located on

…5197) This stops media (and thumbnails) from being accessed from the listed domains. It does not delete any already locally cached media, but will prevent accessing it. Note that admin APIs are unaffected by this change.

turt2live · 2023-05-12T14:07:57Z

Clients should not be reaching out to the original server the media was hosted on. That's abusive in its own right.

This PR is to target a different abusive case than the one you're describing. I can't go into details here though, sorry.

Please report any illegal material found originating from matrix.org to abuse@matrix.org via email. It's not the most efficient system at the moment, but it is the most reliable.

catfromplan9 · 2023-05-12T14:56:52Z

Clients should not be reaching out to the original server the media was hosted on. That's abusive in its own right.

This PR is to target a different abusive case than the one you're describing. I can't go into details here though, sorry.

Please report any illegal material found originating from matrix.org to abuse@matrix.org via email. It's not the most efficient system at the moment, but it is the most reliable.

There is honestly not much point to even reporting it. I will do it, sure, but these are throwaway accounts doing this. In the time it takes for a report to process, the burner accounts already did their jobs

turt2live · 2023-05-12T16:26:07Z

The team does pattern analysis in these cases, but this is highly offtopic for this PR. Not reporting burner accounts just means they're able to get away with it.

catfromplan9 · 2023-05-12T16:30:37Z

Yeah, sorry for flooding this PR with off topic. I just wish to see an option to disable media downloading from my server. A way to host my matrix without doubling as a file upload site and having to deal with the burdens associated with that. Maybe eventually, I can hope a PR that does what im looking for is merged

JokerGermany · 2023-05-23T11:45:09Z

Can I use this if i only want to upload media from my server, but don't want to download media from any other server? (The Server is only used for bridges/bots)

turt2live · 2023-05-23T14:58:55Z

Folks, I'm locking this. Please use the existing support rooms for questions/future extensions.

turt2live added 2 commits March 2, 2023 15:24

Prevent downloads from configured domains

aa7c7af

Changelog

5215707

github-actions bot deployed to PR Documentation Preview March 2, 2023 22:27 View deployment

github-actions bot deployed to PR Documentation Preview March 2, 2023 22:28 View deployment

Define a variable that we use

960650b

github-actions bot deployed to PR Documentation Preview March 2, 2023 22:36 View deployment

dklimpel reviewed Mar 3, 2023

View reviewed changes

tests/rest/media/test_domain_blocking.py Show resolved Hide resolved

synapse/media/media_repository.py Show resolved Hide resolved

tests/rest/media/test_domain_blocking.py Outdated Show resolved Hide resolved

Remove dollar signs

f7fa9f7

github-actions bot deployed to PR Documentation Preview March 3, 2023 16:10 View deployment

turt2live added 2 commits March 9, 2023 10:58

Merge branch 'develop' into travis/block-media-by-domain

5c6e2ec

Try writing the file too?

0d2ea49

github-actions bot deployed to PR Documentation Preview March 9, 2023 18:04 View deployment

I guess await doesn't work

c4855c2

github-actions bot deployed to PR Documentation Preview March 9, 2023 18:08 View deployment

Access properties that might work

d9daa63

github-actions bot deployed to PR Documentation Preview March 9, 2023 18:13 View deployment

Appease CI

33f3107

github-actions bot deployed to PR Documentation Preview March 9, 2023 18:19 View deployment

turt2live commented Mar 9, 2023

View reviewed changes

tests/rest/media/test_domain_blocking.py Show resolved Hide resolved

turt2live marked this pull request as ready for review March 9, 2023 18:35

turt2live requested a review from a team as a code owner March 9, 2023 18:35

Merge branch 'develop' into travis/block-media-by-domain

b292cd1

github-actions bot deployed to PR Documentation Preview March 28, 2023 08:11 View deployment

Fix up tests

298c96c

clokep self-assigned this Mar 29, 2023

clokep reviewed Mar 29, 2023

View reviewed changes

turt2live and others added 3 commits May 8, 2023 10:59

Apply suggestions from code review

bd13239

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Finish field rename

2dd8d5d

Merge branch 'develop' into travis/block-media-by-domain

e339fa9

github-actions bot deployed to PR Documentation Preview May 8, 2023 17:03 View deployment

turt2live added 2 commits May 8, 2023 11:10

Fix imports

673e4fc

Fix code style

9c6e0b5

github-actions bot deployed to PR Documentation Preview May 8, 2023 17:17 View deployment

Clean up imports

2987d9a

github-actions bot deployed to PR Documentation Preview May 8, 2023 17:22 View deployment

turt2live requested a review from clokep May 8, 2023 17:42

dklimpel reviewed May 8, 2023

View reviewed changes

clokep approved these changes May 9, 2023

View reviewed changes

clokep merged commit ab4535b into develop May 9, 2023

clokep deleted the travis/block-media-by-domain branch May 9, 2023 18:08

dklimpel mentioned this pull request May 12, 2023

Disable media federation #15579

Open

catfromplan9 mentioned this pull request May 12, 2023

Disable media federation element-hq/element-meta#1591

Open

matrix-org locked as resolved and limited conversation to collaborators May 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add config option to prevent media downloads from listed domains. #15197

Add config option to prevent media downloads from listed domains. #15197

turt2live commented Mar 2, 2023 •

edited

Loading

Twi1ightSparkle commented Mar 6, 2023

davidegirardi commented Mar 6, 2023

turt2live commented Mar 6, 2023

davidegirardi commented Mar 6, 2023

turt2live commented Mar 6, 2023

davidegirardi commented Mar 7, 2023

reivilibre commented Mar 29, 2023

dklimpel May 8, 2023

turt2live May 8, 2023

clokep left a comment

catfromplan9 commented May 12, 2023

catfromplan9 commented May 12, 2023

catfromplan9 commented May 12, 2023 •

edited

Loading

turt2live commented May 12, 2023

catfromplan9 commented May 12, 2023

turt2live commented May 12, 2023

catfromplan9 commented May 12, 2023

JokerGermany commented May 23, 2023

turt2live commented May 23, 2023

Add config option to prevent media downloads from listed domains. #15197

Add config option to prevent media downloads from listed domains. #15197

Conversation

turt2live commented Mar 2, 2023 • edited Loading

Pull Request Checklist

Twi1ightSparkle commented Mar 6, 2023

davidegirardi commented Mar 6, 2023

turt2live commented Mar 6, 2023

davidegirardi commented Mar 6, 2023

turt2live commented Mar 6, 2023

davidegirardi commented Mar 7, 2023

reivilibre commented Mar 29, 2023

dklimpel May 8, 2023

Choose a reason for hiding this comment

turt2live May 8, 2023

Choose a reason for hiding this comment

clokep left a comment

Choose a reason for hiding this comment

catfromplan9 commented May 12, 2023

catfromplan9 commented May 12, 2023

catfromplan9 commented May 12, 2023 • edited Loading

turt2live commented May 12, 2023

catfromplan9 commented May 12, 2023

turt2live commented May 12, 2023

catfromplan9 commented May 12, 2023

JokerGermany commented May 23, 2023

turt2live commented May 23, 2023

turt2live commented Mar 2, 2023 •

edited

Loading

catfromplan9 commented May 12, 2023 •

edited

Loading