From 5afe6c31aae72c9df88b15c1f116f6a0cf7435b9 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Tue, 23 Aug 2022 13:09:43 +0200 Subject: [PATCH 01/14] MSC3874: Filtering threads from the /messages endpoint Signed-off-by: Janne Mareike Koschinski --- .../3874-messages-endpoint-threads-filter.md | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 proposals/3874-messages-endpoint-threads-filter.md diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md new file mode 100644 index 0000000000..43512f740e --- /dev/null +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -0,0 +1,75 @@ +# MSC3874 Loading Messages excluding Threads + +## Motivation + +In our beta deployment of threads, it's become a noticeable issue where a room may have activity in a long-running +thread while the main timeline is inactive. If a user starts their client and tries to paginate through the main +timeline, loading will feel sluggish, as lots of events have to be loaded that won’t be displayed. + +## Proposal + +### Allow filtering the /messages API to not include threaded messages + +This proposal recommends extending the existing [Event filters] are extended with a new filter, named +`not_related_by_rel_types`, which acts exactly like the opposite of the existing `related_by_rel_types` filter. + +This means, if this filter is specified, only message which match none of the given relation types will be returned. + +``` +GET /_matrix/client/v3/rooms/!room_id:domain/messages?filter=... +``` + +The filter string includes the new fields, above. In this example, the URL encoded JSON is presented unencoded and +formatted for legibility: + +```jsonc +{ + "types": ["m.room.message"], + "not_related_by_rel_types": ["m.thread"] +} +``` + +Note that the newly added filtering parameters return events based on information in related events. Consider the +following events in a room: + +* `A`: a `m.room.message` event sent by `alice` +* `B`: a `m.room.message` event sent by `bob` +* `C`: a `m.room.message` event sent by `charlie` which relates to `A` with type `m.thread` + +Using a filter of `"not_related_by_rel_types": ["m.thread"]` would return only event `B` as it has no event which +relates to it via `m.thread`. + +### Server capabilities + +Threads might have sporadic support across servers, to simplify feature detections for clients, a homeserver must +advertise unstable support for threads as part of the `/versions` API: + +```jsonc +{ + "unstable_features": { + "org.matrix.msc3874": true, + // ... + } +} +``` + +## Potential issues + +This proposal moves the loading and processing of these hidden events onto the server. Depending on the server’s +architecture, this may have a non-negligible performance impact. + +## Alternatives + +- A suitable workaround, depending on the ratio of thread-messages compared to main timeline messages in a room, may be + an increase of the page size + +## Dependencies + +- [MSC2674](https://github.com/matrix-org/matrix-doc/pull/2674) +- [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675) +- [MSC3567](https://github.com/matrix-org/matrix-doc/pull/3567) +- [MSC3676](https://github.com/matrix-org/matrix-doc/pull/3676) +- [MSC3440](https://github.com/matrix-org/matrix-doc/pull/344ß) + + +[Event filters]: https://spec.matrix.org/v1.2/client-server-api/#filtering From 04ece963926060023363ae9eeead33786ce4c20c Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Wed, 24 Aug 2022 13:57:10 +0200 Subject: [PATCH 02/14] Avoid accidental double spaces Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 43512f740e..130999cd7b 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -19,7 +19,7 @@ This means, if this filter is specified, only message which match none of the gi GET /_matrix/client/v3/rooms/!room_id:domain/messages?filter=... ``` -The filter string includes the new fields, above. In this example, the URL encoded JSON is presented unencoded and +The filter string includes the new fields, above. In this example, the URL encoded JSON is presented unencoded and formatted for legibility: ```jsonc @@ -29,20 +29,20 @@ formatted for legibility: } ``` -Note that the newly added filtering parameters return events based on information in related events. Consider the +Note that the newly added filtering parameters return events based on information in related events. Consider the following events in a room: * `A`: a `m.room.message` event sent by `alice` * `B`: a `m.room.message` event sent by `bob` * `C`: a `m.room.message` event sent by `charlie` which relates to `A` with type `m.thread` -Using a filter of `"not_related_by_rel_types": ["m.thread"]` would return only event `B` as it has no event which +Using a filter of `"not_related_by_rel_types": ["m.thread"]` would return only event `B` as it has no event which relates to it via `m.thread`. ### Server capabilities Threads might have sporadic support across servers, to simplify feature detections for clients, a homeserver must -advertise unstable support for threads as part of the `/versions` API: +advertise unstable support for threads as part of the `/versions` API: ```jsonc { From 0061a089d6200ec2417379ab0a158e87f5b0fe1f Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Wed, 24 Aug 2022 13:57:21 +0200 Subject: [PATCH 03/14] Rephrase and clarify motivation Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 130999cd7b..7dd7a69d18 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -2,9 +2,11 @@ ## Motivation -In our beta deployment of threads, it's become a noticeable issue where a room may have activity in a long-running -thread while the main timeline is inactive. If a user starts their client and tries to paginate through the main -timeline, loading will feel sluggish, as lots of events have to be loaded that won’t be displayed. +In Element's beta implementation of threads, it's become a noticeable issue where a room may have activity in a +long-running thread while the main timeline is inactive. If a user starts their client and tries to paginate through the +main timeline, loading will feel sluggish, as lots of events have to be loaded that won’t be displayed. + +This proposal would allow reducing the number of requests and amount of data transmitted. ## Proposal From 5ef2e9eefbac90c76a4edb762cb9c62b8edeb275 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Wed, 24 Aug 2022 13:57:31 +0200 Subject: [PATCH 04/14] Correct specified endpoint Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 7dd7a69d18..37f073e635 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -18,7 +18,7 @@ This proposal recommends extending the existing [Event filters] are extended wit This means, if this filter is specified, only message which match none of the given relation types will be returned. ``` -GET /_matrix/client/v3/rooms/!room_id:domain/messages?filter=... +GET /_matrix/client/unstable/org.matrix.msc3874/rooms/!room_id:domain/messages?filter=... ``` The filter string includes the new fields, above. In this example, the URL encoded JSON is presented unencoded and From 6fa06d841a0be95ad2e16875c15f07b1141789dd Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Wed, 24 Aug 2022 13:57:40 +0200 Subject: [PATCH 05/14] Clarify and describe limitations Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 37f073e635..4290aaa69e 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -60,6 +60,11 @@ advertise unstable support for threads as part of the `/versions` API: This proposal moves the loading and processing of these hidden events onto the server. Depending on the server’s architecture, this may have a non-negligible performance impact. +## Limitations + +This proposal only considers events which have a direct relationship with the thread itself. Events such as reactions +don’t, so they won’t be able to be filtered by this proposal. + ## Alternatives - A suitable workaround, depending on the ratio of thread-messages compared to main timeline messages in a room, may be From 86c88b447c36ff343833bdefbfadb81644571891 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Thu, 25 Aug 2022 14:42:40 +0200 Subject: [PATCH 06/14] Improve formatting Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 4290aaa69e..cd3c7c83ae 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -10,7 +10,7 @@ This proposal would allow reducing the number of requests and amount of data tra ## Proposal -### Allow filtering the /messages API to not include threaded messages +### Allow filtering the `/messages` API to not include threaded messages This proposal recommends extending the existing [Event filters] are extended with a new filter, named `not_related_by_rel_types`, which acts exactly like the opposite of the existing `related_by_rel_types` filter. From ceff9c9031d306beeeeb67248eeedb3d87e44640 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Thu, 25 Aug 2022 14:42:50 +0200 Subject: [PATCH 07/14] Mark already merged PRs Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index cd3c7c83ae..85c168d05c 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -72,11 +72,11 @@ don’t, so they won’t be able to be filtered by this proposal. ## Dependencies -- [MSC2674](https://github.com/matrix-org/matrix-doc/pull/2674) -- [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675) -- [MSC3567](https://github.com/matrix-org/matrix-doc/pull/3567) -- [MSC3676](https://github.com/matrix-org/matrix-doc/pull/3676) -- [MSC3440](https://github.com/matrix-org/matrix-doc/pull/344ß) +- [MSC2674](https://github.com/matrix-org/matrix-doc/pull/2674) ✓ +- [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675) ✓ +- [MSC3567](https://github.com/matrix-org/matrix-doc/pull/3567) ✓ +- [MSC3676](https://github.com/matrix-org/matrix-doc/pull/3676) ✓ +- [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440) ✓ [Event filters]: https://spec.matrix.org/v1.2/client-server-api/#filtering From df5417658178c1307c257e0c4a8c222c5a2e4f71 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Thu, 25 Aug 2022 14:42:59 +0200 Subject: [PATCH 08/14] Use production URL again Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 85c168d05c..8071d0cde8 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -18,7 +18,7 @@ This proposal recommends extending the existing [Event filters] are extended wit This means, if this filter is specified, only message which match none of the given relation types will be returned. ``` -GET /_matrix/client/unstable/org.matrix.msc3874/rooms/!room_id:domain/messages?filter=... +GET /_matrix/client/v3/rooms/!room_id:domain/messages?filter=... ``` The filter string includes the new fields, above. In this example, the URL encoded JSON is presented unencoded and From 6fff09540ac8c47d8e3229b30a53905a333081bf Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Thu, 25 Aug 2022 14:43:16 +0200 Subject: [PATCH 09/14] Correct issue where rel_types and related_by_rel_types got mixed up Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 8071d0cde8..dc8b6909ee 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -13,7 +13,7 @@ This proposal would allow reducing the number of requests and amount of data tra ### Allow filtering the `/messages` API to not include threaded messages This proposal recommends extending the existing [Event filters] are extended with a new filter, named -`not_related_by_rel_types`, which acts exactly like the opposite of the existing `related_by_rel_types` filter. +`not_rel_types`, which acts exactly like the opposite of the existing `rel_types` filter. This means, if this filter is specified, only message which match none of the given relation types will be returned. @@ -27,7 +27,7 @@ formatted for legibility: ```jsonc { "types": ["m.room.message"], - "not_related_by_rel_types": ["m.thread"] + "not_rel_types": ["m.thread"] } ``` @@ -38,7 +38,7 @@ following events in a room: * `B`: a `m.room.message` event sent by `bob` * `C`: a `m.room.message` event sent by `charlie` which relates to `A` with type `m.thread` -Using a filter of `"not_related_by_rel_types": ["m.thread"]` would return only event `B` as it has no event which +Using a filter of `"not_rel_types": ["m.thread"]` would return only event `B` as it has no event which relates to it via `m.thread`. ### Server capabilities From 9476b7f84faee94bd301c9735f8ccf88580aa0b4 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Thu, 25 Aug 2022 14:44:15 +0200 Subject: [PATCH 10/14] Add mobile reasoning to motivation Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index dc8b6909ee..13d43e88e6 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -6,7 +6,8 @@ In Element's beta implementation of threads, it's become a noticeable issue wher long-running thread while the main timeline is inactive. If a user starts their client and tries to paginate through the main timeline, loading will feel sluggish, as lots of events have to be loaded that won’t be displayed. -This proposal would allow reducing the number of requests and amount of data transmitted. +This proposal would allow reducing the number of requests and amount of data transmitted. Especially for mobile usage +this would allow clients significant usability improvements with threads. ## Proposal From e28ba77a8d7d591729dfb72204e3e2d4c48e80a8 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Fri, 26 Aug 2022 11:33:22 +0200 Subject: [PATCH 11/14] Clarify example Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 13d43e88e6..57ee70ed45 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -39,8 +39,8 @@ following events in a room: * `B`: a `m.room.message` event sent by `bob` * `C`: a `m.room.message` event sent by `charlie` which relates to `A` with type `m.thread` -Using a filter of `"not_rel_types": ["m.thread"]` would return only event `B` as it has no event which -relates to it via `m.thread`. +Using a filter of `"not_rel_types": ["m.thread"]` would return only events `A` and `B` as they do not have a relation of +`m.thread` in them. Thread roots are returned in the same way as messages which are not part of threads at all. ### Server capabilities From b14510fc27b66367546ff7e4b94d61af77a97e45 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Fri, 26 Aug 2022 11:33:31 +0200 Subject: [PATCH 12/14] Correct proposal wording Signed-off-by: Janne Mareike Koschinski --- proposals/3874-messages-endpoint-threads-filter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 57ee70ed45..54d4931927 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -13,8 +13,8 @@ this would allow clients significant usability improvements with threads. ### Allow filtering the `/messages` API to not include threaded messages -This proposal recommends extending the existing [Event filters] are extended with a new filter, named -`not_rel_types`, which acts exactly like the opposite of the existing `rel_types` filter. +This proposal recommends extending the existing [Event filters] are extended with a new filter, named `not_rel_types`. +If this filter is specified, only messages which match none of the given relation types will be returned. This means, if this filter is specified, only message which match none of the given relation types will be returned. From 747daf153952968c31ccf987915e539f289e9aea Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Tue, 18 Oct 2022 14:34:47 +0200 Subject: [PATCH 13/14] Address comments on proposal Signed-off-by: Janne Mareike Koschinski --- .../3874-messages-endpoint-threads-filter.md | 54 +++++++++++-------- 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 54d4931927..2f88e48d26 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -11,12 +11,15 @@ this would allow clients significant usability improvements with threads. ## Proposal -### Allow filtering the `/messages` API to not include threaded messages +### Allow filtering the `/messages` API by event relation type -This proposal recommends extending the existing [Event filters] are extended with a new filter, named `not_rel_types`. -If this filter is specified, only messages which match none of the given relation types will be returned. +This proposal recommends extending the existing [Event filters] are extended with new filters, named `not_rel_types` and +`rel_types`. If the `rel_types` filter is specified, messages which match any of the given relation types will be +returned. If the `not_rel_types` filter is specified, only messages which match none of the given relation types will be +returned. -This means, if this filter is specified, only message which match none of the given relation types will be returned. +If a relation type is present in both of these fields, `not_rel_types` takes precedence and messages with this type will +not be returned. ``` GET /_matrix/client/v3/rooms/!room_id:domain/messages?filter=... @@ -28,7 +31,8 @@ formatted for legibility: ```jsonc { "types": ["m.room.message"], - "not_rel_types": ["m.thread"] + "not_rel_types": ["m.thread"], + "rel_types": ["m.edit"] } ``` @@ -42,10 +46,20 @@ following events in a room: Using a filter of `"not_rel_types": ["m.thread"]` would return only events `A` and `B` as they do not have a relation of `m.thread` in them. Thread roots are returned in the same way as messages which are not part of threads at all. -### Server capabilities +## Potential issues + +This proposal moves the loading and processing of these hidden events onto the server. Depending on the server’s +architecture, this may have a non-negligible performance impact. + +## Limitations -Threads might have sporadic support across servers, to simplify feature detections for clients, a homeserver must -advertise unstable support for threads as part of the `/versions` API: +While the client can effectively filter out noisy threads, it's not as easy to filter out events adjacent to threads +such as reactions. A more performant implementation is best left for a future MSC. + +## Unstable prefix and versioning + +Relation filters might have sporadic support across servers, to simplify feature detections for clients, a homeserver +must advertise unstable support for these filters as part of the `/versions` API: ```jsonc { @@ -56,20 +70,18 @@ advertise unstable support for threads as part of the `/versions` API: } ``` -## Potential issues - -This proposal moves the loading and processing of these hidden events onto the server. Depending on the server’s -architecture, this may have a non-negligible performance impact. +Unstable implementations should prefix the filter attributes with `org.matrix.msc3874`, e.g., -## Limitations - -This proposal only considers events which have a direct relationship with the thread itself. Events such as reactions -don’t, so they won’t be able to be filtered by this proposal. - -## Alternatives - -- A suitable workaround, depending on the ratio of thread-messages compared to main timeline messages in a room, may be - an increase of the page size +```http request +GET /_matrix/client/v3/rooms/!room_id:domain/messages?filter=... +``` +In this example, the URL encoded JSON is presented unencoded and formatted for legibility: +```jsonc +{ + "types": ["m.room.message"], + "org.matrix.msc3874.not_rel_types": ["m.thread"] +} +``` ## Dependencies From 8d607ca5b1d74255ce98d1e7bbabf45c31a3b185 Mon Sep 17 00:00:00 2001 From: Janne Mareike Koschinski Date: Wed, 19 Oct 2022 15:11:00 +0200 Subject: [PATCH 14/14] Apply suggestions from code review Co-authored-by: Patrick Cloke --- proposals/3874-messages-endpoint-threads-filter.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/3874-messages-endpoint-threads-filter.md b/proposals/3874-messages-endpoint-threads-filter.md index 2f88e48d26..cf80c90ef5 100644 --- a/proposals/3874-messages-endpoint-threads-filter.md +++ b/proposals/3874-messages-endpoint-threads-filter.md @@ -16,7 +16,7 @@ this would allow clients significant usability improvements with threads. This proposal recommends extending the existing [Event filters] are extended with new filters, named `not_rel_types` and `rel_types`. If the `rel_types` filter is specified, messages which match any of the given relation types will be returned. If the `not_rel_types` filter is specified, only messages which match none of the given relation types will be -returned. +returned, this includes events without a relation. If a relation type is present in both of these fields, `not_rel_types` takes precedence and messages with this type will not be returned. @@ -36,15 +36,14 @@ formatted for legibility: } ``` -Note that the newly added filtering parameters return events based on information in related events. Consider the -following events in a room: +The newly added filtering parameters return events based on information in the `m.relates_to` field. Consider the following events in a room: * `A`: a `m.room.message` event sent by `alice` * `B`: a `m.room.message` event sent by `bob` * `C`: a `m.room.message` event sent by `charlie` which relates to `A` with type `m.thread` Using a filter of `"not_rel_types": ["m.thread"]` would return only events `A` and `B` as they do not have a relation of -`m.thread` in them. Thread roots are returned in the same way as messages which are not part of threads at all. +`m.thread` in them. ## Potential issues