From a72bf5e4d4f06ccca3299b5e2f5e6c9fa977f06d Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Thu, 7 May 2020 19:34:24 +0300 Subject: [PATCH 01/11] Proposal to use body field as media caption --- proposals/2530-body-as-caption.md | 36 +++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 proposals/2530-body-as-caption.md diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md new file mode 100644 index 0000000000..86078628c9 --- /dev/null +++ b/proposals/2530-body-as-caption.md @@ -0,0 +1,36 @@ +# Body field as media caption + +This is an alternative to [MSC2529](https://github.com/matrix-org/matrix-doc/pull/2529). + +## Proposal + +This proposal defines a new field `filename` and allows the `format` and +`formatted_body` fields (from `m.text`) for all media msgtypes (`m.image`, +`m.audio`, `m.video`, `m.file`). + +If the `filename` field is present in a media message, clients should treat +`body` as a caption instead of a file name. The `formatted_body` field should +also be supported and work the same way as they do in `m.text` messages. + +## Tradeoffs + +While this approach isn't backwards-compatible like MSC2529, it's simpler to +implement as it doesn't require handling relations. Additionally, some clients +already render the body as a file name, so the caption would already be visible +in those cases. + +The format proposed by MSC2529 would also make it technically possible to use +other message types without changing the format of the events. + +## Potential issues + +The spec for `m.file` already defines a `filename` field, but the spec also +says `body` should contain the file name. This means old `m.file` messages +could be misinterpreted to contain a caption, even though it's just the file +name. To avoid this problem, we could require that `body` and `filename` must +be different for `body` to be interpreted as a caption. + +Like MSC2529, this would be obsoleted by [extensible events](https://github.com/matrix-org/matrix-doc/pull/1767). +In my opinion, unless extensible events are planned to be pushed through in the +near future, we shouldn't block basic features like captions on extensible +events hopefully eventually happening. From 6691780b2bcc522050fab5a214fa0a9877e85b05 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Thu, 7 May 2020 19:44:33 +0300 Subject: [PATCH 02/11] Add paragraph about relation-based captions being difficult for bridges --- proposals/2530-body-as-caption.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index 86078628c9..9162ac62e4 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -12,15 +12,22 @@ If the `filename` field is present in a media message, clients should treat `body` as a caption instead of a file name. The `formatted_body` field should also be supported and work the same way as they do in `m.text` messages. -## Tradeoffs +## Comparison to MSC2529 While this approach isn't backwards-compatible like MSC2529, it's simpler to -implement as it doesn't require handling relations. Additionally, some clients -already render the body as a file name, so the caption would already be visible -in those cases. +implement as it doesn't require handling relations. + +Implementing relation-based captions would be especially difficult for bridges. +It would require either waiting a predefined amount of time for the caption to +come through, or editing the message on the target platform (if edits are +supported). + +Additionally, some clients already render the body (as a file name), so the +new-style caption would already be visible in those cases. The format proposed by MSC2529 would also make it technically possible to use -other message types without changing the format of the events. +other message types without changing the format of the events, which is not +possible with this proposal. ## Potential issues From 2cf7abb5450c3ff2f08fe46534c08727cabbaa20 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Fri, 8 May 2020 18:30:37 +0300 Subject: [PATCH 03/11] Clarify how to treat body when filename is not present --- proposals/2530-body-as-caption.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index 9162ac62e4..063e52cee6 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -12,6 +12,11 @@ If the `filename` field is present in a media message, clients should treat `body` as a caption instead of a file name. The `formatted_body` field should also be supported and work the same way as they do in `m.text` messages. +The current spec is inconsistent with how `body` should be handled, but in +practice, clients (or at least Riot) use it as the file name. Therefore, this +proposal suggests that clients should treat `body` as the file name if the +`filename` field is not present. + ## Comparison to MSC2529 While this approach isn't backwards-compatible like MSC2529, it's simpler to From a7047f8222507b5ad206b1414d514d39cf73cad1 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Wed, 19 Oct 2022 20:38:53 +0300 Subject: [PATCH 04/11] Refactor proposal text --- proposals/2530-body-as-caption.md | 76 ++++++++++++++++++------------- 1 file changed, 45 insertions(+), 31 deletions(-) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index 063e52cee6..fabbdfd1bd 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -1,48 +1,62 @@ # Body field as media caption -This is an alternative to [MSC2529](https://github.com/matrix-org/matrix-doc/pull/2529). - ## Proposal -This proposal defines a new field `filename` and allows the `format` and -`formatted_body` fields (from `m.text`) for all media msgtypes (`m.image`, +This proposal allows the `filename` field from `m.file`, and the `format` and +`formatted_body` fields from `m.text` for all media msgtypes (`m.image`, `m.audio`, `m.video`, `m.file`). If the `filename` field is present in a media message, clients should treat -`body` as a caption instead of a file name. The `formatted_body` field should -also be supported and work the same way as they do in `m.text` messages. +`body` as a caption instead of a file name. The `format`/`formatted_body` +fields should also be supported and work the same way as they do in `m.text` +messages. + +The current spec is somewhat ambiguous as to how `body` should be handled and +the definition varies across different message types. In practice, clients +(or at least Element) use it as the file name. As a part of adding captions, +the `body` field for all message types is explicitly defined to be used as the +file name when the `filename` field is not present. -The current spec is inconsistent with how `body` should be handled, but in -practice, clients (or at least Riot) use it as the file name. Therefore, this -proposal suggests that clients should treat `body` as the file name if the -`filename` field is not present. +For `m.file` messages, the current spec confusingly defines both `filename` and +`body` as the file name. In order to avoid (old) messages with both fields from +being misinterpreted as having captions, the `body` field should not be used as +a caption when it's equal to `filename`. + +## Potential issues -## Comparison to MSC2529 +In clients that don't show the file name anywhere, the caption would not be +visible at all. However, extensible events would run into the same issue. +Clients having captions implemented beforehand may even help eventually +implementing extensible events. -While this approach isn't backwards-compatible like MSC2529, it's simpler to -implement as it doesn't require handling relations. +## Alternatives -Implementing relation-based captions would be especially difficult for bridges. -It would require either waiting a predefined amount of time for the caption to -come through, or editing the message on the target platform (if edits are -supported). +### [MSC2529](https://github.com/matrix-org/matrix-spec-proposals/pull/2529) -Additionally, some clients already render the body (as a file name), so the -new-style caption would already be visible in those cases. +MSC2529 would allow existing clients to render captions without any changes, +but the use of relations makes implementation more difficult, especially for +bridges. It would require either waiting a predefined amount of time for the +caption to come through, or editing the message on the target platform (if +edits are supported). The format proposed by MSC2529 would also make it technically possible to use -other message types without changing the format of the events, which is not -possible with this proposal. +other message types as captions without changing the format of the events, +which is not possible with this proposal. -## Potential issues +### Extensible events + +Like MSC2529, this would be obsoleted by [extensible events](https://github.com/matrix-org/matrix-spec-proposals/pull/3552). +However, fully switching to extensible events requires significantly more +implementation work, and it may take years for the necessary time to be +allocated for that. + +### Security considerations + +This proposal doesn't involve any security-sensitive components. -The spec for `m.file` already defines a `filename` field, but the spec also -says `body` should contain the file name. This means old `m.file` messages -could be misinterpreted to contain a caption, even though it's just the file -name. To avoid this problem, we could require that `body` and `filename` must -be different for `body` to be interpreted as a caption. +### Unstable prefix -Like MSC2529, this would be obsoleted by [extensible events](https://github.com/matrix-org/matrix-doc/pull/1767). -In my opinion, unless extensible events are planned to be pushed through in the -near future, we shouldn't block basic features like captions on extensible -events hopefully eventually happening. +The fields being added already exist in other msgtypes, so unstable prefixes +don't seem necessary. Additionally, using `body` as a caption could already be +considered spec-compliant due to the ambiguous definition of the field, and +only adding unstable prefixes for the other fields would be silly. From aebe61af323c106fe88c5c58efe304741a020071 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Wed, 19 Oct 2022 20:39:42 +0300 Subject: [PATCH 05/11] Fix heading size --- proposals/2530-body-as-caption.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index fabbdfd1bd..c81e60bf1f 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -50,11 +50,11 @@ However, fully switching to extensible events requires significantly more implementation work, and it may take years for the necessary time to be allocated for that. -### Security considerations +## Security considerations This proposal doesn't involve any security-sensitive components. -### Unstable prefix +## Unstable prefix The fields being added already exist in other msgtypes, so unstable prefixes don't seem necessary. Additionally, using `body` as a caption could already be From 76d5b54d3da176e49819c72c77d7211be4643fee Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 Jan 2024 22:43:18 +0200 Subject: [PATCH 06/11] Add problem statement --- proposals/2530-body-as-caption.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index c81e60bf1f..a431b65d2f 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -1,5 +1,15 @@ # Body field as media caption +When sending images or other attachments, users often want to include text to +convey additional information. Most chat platforms offer media captions as a +first-class feature, allowing users to choose the attachment and write text, +then send both together in one message. + +Matrix currently does not enable this on the protocol level: at best, clients +can emulate the behavior by sending two messages quickly; at worst, the user +has to do that manually. Sending separate messages means it's possible for +the second message to be delayed or lost if something goes wrong. + ## Proposal This proposal allows the `filename` field from `m.file`, and the `format` and From 675198da80bf1d378373607b514775187a8acaab Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 Jan 2024 22:43:52 +0200 Subject: [PATCH 07/11] Add links to and quotes from current spec --- proposals/2530-body-as-caption.md | 34 +++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index a431b65d2f..a4aec308b8 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -12,8 +12,8 @@ the second message to be delayed or lost if something goes wrong. ## Proposal -This proposal allows the `filename` field from `m.file`, and the `format` and -`formatted_body` fields from `m.text` for all media msgtypes (`m.image`, +This proposal allows the `filename` field from [`m.file`], and the `format` and +`formatted_body` fields from [`m.text`] for all media msgtypes (`m.image`, `m.audio`, `m.video`, `m.file`). If the `filename` field is present in a media message, clients should treat @@ -22,15 +22,27 @@ fields should also be supported and work the same way as they do in `m.text` messages. The current spec is somewhat ambiguous as to how `body` should be handled and -the definition varies across different message types. In practice, clients -(or at least Element) use it as the file name. As a part of adding captions, -the `body` field for all message types is explicitly defined to be used as the -file name when the `filename` field is not present. - -For `m.file` messages, the current spec confusingly defines both `filename` and -`body` as the file name. In order to avoid (old) messages with both fields from -being misinterpreted as having captions, the `body` field should not be used as -a caption when it's equal to `filename`. +the definition varies across different message types. The current spec for +[`m.image`] describes `body` as + +> A textual representation of the image. This could be the alt text of the +> image, the filename of the image, or some kind of content description for +> accessibility e.g. ‘image attachment’. + +In practice, clients (or at least Element) use it as the file name. As a part +of adding captions, the `body` field for all message types is explicitly +defined to be used as the file name when the `filename` field is not present. + +For `m.file` messages, the [current (v1.9) spec][`m.file`] confusingly defines +`filename` as "The original filename of the uploaded file" and simultaneously +recommends that `body` is "the filename of the original upload", effectively +saying both fields should have the file name. In order to avoid (old) messages +with both fields from being misinterpreted as having captions, the `body` field +should not be used as a caption when it's equal to `filename`. + +[`m.file`]: https://spec.matrix.org/v1.9/client-server-api/#mfile +[`m.text`]: https://spec.matrix.org/v1.9/client-server-api/#mtext +[`m.image`]: https://spec.matrix.org/v1.9/client-server-api/#mimage ## Potential issues From 5b3d74e579c24947ca8e89a48555759b9f32e057 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 Jan 2024 23:02:30 +0200 Subject: [PATCH 08/11] Adjust wording and quote m.audio body spec --- proposals/2530-body-as-caption.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index a4aec308b8..cf07dbb3eb 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -17,9 +17,10 @@ This proposal allows the `filename` field from [`m.file`], and the `format` and `m.audio`, `m.video`, `m.file`). If the `filename` field is present in a media message, clients should treat -`body` as a caption instead of a file name. The `format`/`formatted_body` -fields should also be supported and work the same way as they do in `m.text` -messages. +`body` as a caption instead of a file name. If the `format`/`formatted_body` +fields are present in addition to `filename` and `body`, then they should take +priority as the caption text. Formatted text in media captions is rendered the +same way as formatted text in `m.text` messages. The current spec is somewhat ambiguous as to how `body` should be handled and the definition varies across different message types. The current spec for @@ -29,6 +30,11 @@ the definition varies across different message types. The current spec for > image, the filename of the image, or some kind of content description for > accessibility e.g. ‘image attachment’. +while [`m.audio`] describes it as + +> A description of the audio e.g. ‘Bee Gees - Stayin’ Alive’, or some kind of +> content description for accessibility e.g. ‘audio attachment’. + In practice, clients (or at least Element) use it as the file name. As a part of adding captions, the `body` field for all message types is explicitly defined to be used as the file name when the `filename` field is not present. @@ -37,12 +43,13 @@ For `m.file` messages, the [current (v1.9) spec][`m.file`] confusingly defines `filename` as "The original filename of the uploaded file" and simultaneously recommends that `body` is "the filename of the original upload", effectively saying both fields should have the file name. In order to avoid (old) messages -with both fields from being misinterpreted as having captions, the `body` field +with both fields being misinterpreted as having captions, the `body` field should not be used as a caption when it's equal to `filename`. [`m.file`]: https://spec.matrix.org/v1.9/client-server-api/#mfile [`m.text`]: https://spec.matrix.org/v1.9/client-server-api/#mtext [`m.image`]: https://spec.matrix.org/v1.9/client-server-api/#mimage +[`m.audio`]: https://spec.matrix.org/v1.9/client-server-api/#maudio ## Potential issues From cca27fddd0f48cea25aaf1f9716b9faf0e48be98 Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 23 Jan 2024 23:04:08 +0200 Subject: [PATCH 09/11] Clarify that m.location and m.sticker are out of scope for this proposal --- proposals/2530-body-as-caption.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index cf07dbb3eb..f61d168f9d 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -14,7 +14,9 @@ the second message to be delayed or lost if something goes wrong. This proposal allows the `filename` field from [`m.file`], and the `format` and `formatted_body` fields from [`m.text`] for all media msgtypes (`m.image`, -`m.audio`, `m.video`, `m.file`). +`m.audio`, `m.video`, `m.file`). This proposal does not affect the `m.location` +msgtype, nor the separate `m.sticker` event type: stickers already use `body` +as a description, and locations don't have file names. If the `filename` field is present in a media message, clients should treat `body` as a caption instead of a file name. If the `format`/`formatted_body` @@ -36,7 +38,7 @@ while [`m.audio`] describes it as > content description for accessibility e.g. ‘audio attachment’. In practice, clients (or at least Element) use it as the file name. As a part -of adding captions, the `body` field for all message types is explicitly +of adding captions, the `body` field for all media message types is explicitly defined to be used as the file name when the `filename` field is not present. For `m.file` messages, the [current (v1.9) spec][`m.file`] confusingly defines From ec2305080b962d720ed8ee570e692cc851fa79bc Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 6 Feb 2024 21:34:57 +0200 Subject: [PATCH 10/11] Add examples and summary of changes --- proposals/2530-body-as-caption.md | 51 +++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index f61d168f9d..7d7758f393 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -53,6 +53,54 @@ should not be used as a caption when it's equal to `filename`. [`m.image`]: https://spec.matrix.org/v1.9/client-server-api/#mimage [`m.audio`]: https://spec.matrix.org/v1.9/client-server-api/#maudio +### Examples +
+Image with caption + +```json +{ + "msgtype": "m.image", + "url": "mxc://maunium.net/HaIrXlnKfEEHvMNKzuExiYlv" + "filename": "cat.jpeg", + "body": "this is a cat picture :3", + "info": { + "w": 479, + "h": 640, + "mimetype": "image/jpeg", + "size": 27253 + }, + "m.mentions": {} +} +``` + +
+
+File with formatted caption + +```json +{ + "msgtype": "m.file", + "url": "mxc://maunium.net/TizWsLhHfDCETKRXdDwHoAGn", + "filename": "hello.txt", + "body": "this caption is longer than the file itself 🤔", + "format": "org.matrix.custom.html", + "formatted_body": "this caption is longer than the file itself 🤔", + "info": { + "mimetype": "text/plain", + "size": 14 + }, + "m.mentions": {} +} +``` + +
+ +### Summary +* `filename` is defined for all media msgtypes. +* `body` is defined to be a caption when `filename` is present and not equal to `body`. + * `format` and `formatted_body` are allowed as well for formatted captions. +* `body` is defined to be the file name when `filename` is not present. + ## Potential issues In clients that don't show the file name anywhere, the caption would not be @@ -60,6 +108,9 @@ visible at all. However, extensible events would run into the same issue. Clients having captions implemented beforehand may even help eventually implementing extensible events. +Old clients may default to using the caption as the file name when the user +wants to download a file, which will be somewhat weird UX. + ## Alternatives ### [MSC2529](https://github.com/matrix-org/matrix-spec-proposals/pull/2529) From cbcd0dc6cece8141736b98f477d6c26a487a996d Mon Sep 17 00:00:00 2001 From: Tulir Asokan Date: Tue, 6 Feb 2024 21:37:25 +0200 Subject: [PATCH 11/11] Fix JSON syntax in example --- proposals/2530-body-as-caption.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/2530-body-as-caption.md b/proposals/2530-body-as-caption.md index 7d7758f393..246b3f97df 100644 --- a/proposals/2530-body-as-caption.md +++ b/proposals/2530-body-as-caption.md @@ -60,7 +60,7 @@ should not be used as a caption when it's equal to `filename`. ```json { "msgtype": "m.image", - "url": "mxc://maunium.net/HaIrXlnKfEEHvMNKzuExiYlv" + "url": "mxc://maunium.net/HaIrXlnKfEEHvMNKzuExiYlv", "filename": "cat.jpeg", "body": "this is a cat picture :3", "info": {