Issue with adding attachments with API

[mshedsilegx]

1- OTS server version 1.10.0
acceptedFileTypes: 'image/*,text/plain,text/csv,application/pdf,application/xml,application/json,application/zip,application/x-pkcs12,application/x-pem-file'

2- Works fine with the Web UI

3- Does not work for me with API
ots-cli create -u USER:PASSWORD --secret-from pem/file.pem --instance https://host.domain.com/ots -f test.txt

INFO[0000] reading secret content...
INFO[0000] attaching file... file=test.txt
INFO[0000] creating the secret...
Error: sanity checking secret: attachment type is not allowed

[Luzifer]

# ots-cli create -f cert.pem --no-text --log-level=debug
INFO[0000] reading secret content...
INFO[0000] attaching file...                             file=cert.pem
INFO[0000] creating the secret...
DEBU[0000] attachment type not allowed                   content-type=application/x-x509-ca-cert
Error: sanity checking secret: attachment type is not allowed

That's the issue with mime-type guessing: You're (probably) missing application/x-x509-ca-cert.

Also that result might vary depending on the system the tool is executed on: https://pkg.go.dev/mime#TypeByExtension

(Please note: The debug-logging is not yet available in any released version, I just applied some gaffa-tape and a proper version will be available in v1.11.x)

[mshedsilegx]

Recap:
1- Web UI is working just fine as is
2- API always returns attachment type is not allowed, no matter what type file is being attached, even plain text
Examples:

ots-cli create --secret-from test.txt --instance https://host.domain.com/ots -f test.txt

with test.txt being a plain text file containing:
THIS IS A TEST

INFO[0000] reading secret content...
INFO[0000] attaching file... file=test.txt
INFO[0000] creating the secret...
Error: sanity checking secret: attachment type is not allowed

ots-cli create ---no-text --instance https://host.domain.com/ots -f test.txt

INFO[0000] reading secret content...
INFO[0000] attaching file... file=test.txt
INFO[0000] creating the secret...
Error: sanity checking secret: attachment type is not allowed

Am I missing something ? I cannot make it work.

[Luzifer]

API always returns attachment type is not allowed

API does not return anything: There is no and cannot be any server-side checking as nothing is sent to the server in plain! - The check is only done in ots-cli locally!

Am I missing something ? I cannot make it work.

You could try building the ots-cli from this PR: #155 (branch 154-add-debug-logging-for-rejected-attachment-types) - That one has the debug logging I used above.

Afterwards it should tell you the detected mime type when rejecting an attachment using ots-cli create -f test.txt --no-text --log-level=debug with text/* allowed (ots.fyi) I've no issues attaching a .txt file so I assume the mime-type detection does something stupid and does not detect text/plain but something else.

[Luzifer]

Was able to reproduce:

# ots-cli create --instance=http://localhost:4000 --no-text -f test.txt --log-level=debug
INFO[0000] reading secret content...
INFO[0000] attaching file...                             file=test.txt
INFO[0000] creating the secret...
DEBU[0000] attachment type not allowed                   content-type="text/plain; charset=utf-8"
Error: sanity checking secret: attachment type is not allowed

Added a fix (stripping off everything after a ;) to #155.

[mshedsilegx]

Things are still not working for me.

1- OTS server version 1.10.0-dc47bf0

acceptedFileTypes: 'image/*,text/plain,text/csv,application/pdf,application/xml,application/json,application/zip,application/x-pkcs12,application/x-pem-file'

2- ots-cli create --instance https://host.domain.com/ots --no-text -f file.p12

INFO[0000] reading secret content...
INFO[0000] attaching file...                             file=file.p12
INFO[0000] creating the secret...
Error: sanity checking secret: attachment type is not allowed

As you can see, MIME type application/x-pkcs12 is allowed.

3- With the exact same settings, the WEB UI works fine. So the issue is with the OTS cli.

[Luzifer]

I can't guess what the detected mime-type on your system is. Until the next release you can build ots-cli from the current master and use --log-level=debug to have it print the detected mime-type.

Like stated above: The mime-type detection entirely depends on your local system and will differ at least between major operating systems but also might differ between two recent Linux systems.

[mshedsilegx]

Fair point, see below

INFO[0000] reading secret content...
INFO[0000] attaching file...                             file=file.p12
INFO[0000] creating the secret...
DEBU[0000] attachment type not allowed                   content-type=application/pkcs12
Error: sanity checking secret: attachment type is not allowed

I thought the standard was: application/x-pkcs12
Strange that it is working fine from the Web UI and not ots-cli CMD

[Luzifer]

Yeah the browsers are doing their own magic… 🤷🏻‍♀️

A .pfx I do have lying around is detected as application/x-pkcs12 so yeah, my system agrees on that… Seems the mime types "database" on your system for some reason has it added as application/pkcs12…

[mshedsilegx]

Interesting, you're right. I'll look this up in the future to make sure it matches.

grep "pkcs12" /etc/mime.types
-----
application/pkcs12                              p12 pfx

[Luzifer]

# rg '(p12|pfx)' /usr/local/share/mime/globs2 /usr/share/mime/globs2 /etc/mime.types /etc/apache2/mime.types /etc/apache/mime.types
/usr/local/share/mime/globs2: No such file or directory (os error 2)
/etc/apache2/mime.types: No such file or directory (os error 2)
/etc/apache/mime.types: No such file or directory (os error 2)
/etc/mime.types
324:application/pkcs12				p12 pfx

/usr/share/mime/globs2
10:75:application/x-pkcs12:*.p12
24:55:application/x-pkcs12:*.pfx
719:50:application/pkcs12:*.pfx
963:50:application/pkcs12:*.p12

Well.

[mshedsilegx]

Agree, I included both:

 application/x-pkcs12
 application/pkcs12

[Luzifer]

An then it's probably what is encountered first in https://pkg.go.dev/mime#TypeByExtension - first one wins and that might be either of them depending on the order of the file and the system and where the mars stands in relation to jupiter…

[mshedsilegx]

1- Can we display
[SUCCESS]]

DEBU[0000] attachment type detected                   content-type=__MIME_TYPE__

[FAILURE]

DEBU[0000] attachment type not allowed                   content-type=__MIME_TYPE__

with log-level=debug
I think it is valuable information in both cases

2- If attachment type is unauthorized, can we please not diplay the help reminder. I think the error message is sufficient IMO.

Error: sanity checking secret: attachment type is not allowed
Usage:
   create [-f file]... [--instance url] [--secret-from file] [flags]

Examples:
echo "I'm a very secret secret" | ots-cli create

Flags:
      --expire duration      When to expire the secret (0 to use server-default)
  -f, --file strings         File(s) to attach to the secret
  -H, --header strings       Headers to include in the request (i.e. 'Authorization: Token ...')
  -h, --help                 help for create
      --instance string      Instance to create the secret with (default "https://ots.fyi/")
      --no-text              Disable secret read (create a secret with only files)
      --secret-from string   File to read the secret content from ("-" for STDIN) (default "-")
  -u, --user string          Username / Password for basic auth, specified as 'user:pass'

Global Flags:
      --log-level string   Level to use for logging (trace, debug, info, warn, error, fatal) (default "info")

[mshedsilegx]

Another idea would be to have a flag in ots to display all translated MIME types resolved from acceptedFileTypes. I think it would really help if possible.

[Luzifer]

1- Can we display

Sure. Adding a debug message about what gets attached is no issue.

2- If attachment type is unauthorized, can we please not diplay the help reminder. I think the error message is sufficient IMO.

Just had a look, is possible. Keeping this as info for me:
https://github.com/spf13/cobra/issues/340#issuecomment-374617413

Another idea would be to have a flag in ots to display all translated MIME types resolved from acceptedFileTypes.

I don't think that's easily possible as the resolve is either by file extension or by mime type directly but as we support wildcards and there is no list I can get from the mime stdlib I currently don't see how that would work out.

[Luzifer]

# ots-cli create -f ~/Downloads/test.pdf --no-text --log-level=debug 
INFO[0000] reading secret content...                    
INFO[0000] attaching file...                             file=/home/luzifer/Downloads/test.pdf
INFO[0000] creating the secret...                       
DEBU[0000] attachment allowed                            allowed_by=application/pdf content-type=application/pdf
INFO[0000] secret created, see URL below                 expires-at="2023-12-08 17:52:39.550321453 +0000 UTC"
https://ots.fyi/#...

# ots-cli create -f ~/Downloads/test.txt --no-text --log-level=debug
INFO[0000] reading secret content...                    
INFO[0000] attaching file...                             file=/home/luzifer/Downloads/test.txt
INFO[0000] creating the secret...                       
DEBU[0000] attachment allowed                            allowed_by="text/*" content-type=text/plain
INFO[0000] secret created, see URL below                 expires-at="2023-12-08 17:52:49.31302586 +0000 UTC"
https://ots.fyi/#...

# ots-cli create -f ~/Downloads/test.mp4 --no-text --log-level=debug 
INFO[0000] reading secret content...                    
INFO[0000] attaching file...                             file=/home/luzifer/Downloads/test.mp4
INFO[0000] creating the secret...                       
DEBU[0000] attachment type not allowed                   content-type=video/mp4
Error: sanity checking secret: attachment type is not allowed

[mshedsilegx]

Thank you for all your support. This solution is quickly becoming the best available product for OTS.