EPIC: Universal share prefix

[evan-forbes]

We want to switch to more universal share encoding format, similar to what is discussed in celestiaorg/celestia-core#839. The main benefit of this approach being that parsing shares should be more similar in that we add a varint length delimiter to the total length of contiguous shares.

nid (8 bytes) | info byte | message length (varint) | share data for shares at the start of a message;
nid (8 bytes) | info byte | share data for other shares.

This means that the current format for contiguous shares (reserved namespace shares) will look like

starting shares (last info bit == 1)
nid (8 bytes) | info byte | message length (varint) | reserved byte | share data
non starting shares (last info bit == 0)
nid (8 bytes) | info byte | reserved byte | share data

note that depending on what we decide for celestiaorg/celestia-core#842, this issue might be moved to core.

[rootulp]

Implementation

• Implement InfoReservedByte #671
• Implement ParseInfoReservedByte #689
• Add InfoReservedByte to compact and sparse shares #691
• Rename InfoReservedByte to InfoByte #787
• Encode data length varint into first compact share #725
• Define a ParseShares API #723
• ParseShare should return an error if total sequence length varint doesn't match actual #839
• Specify share version in MsgWirePayForBlob #936
• All shares must contain a share version that belongs to a list of supported versions #830
• Test: verify that modifications to share sequence indicator in reserved namespace result in different DataHash #831

Docs

• README for pkg/shares #718
• ADR-007: Universal Share Prefix #660
• specs: universal share prefix #729

[rootulp]

Proposal to rename | message length (varint) | to | data length (varint) | because in universal share encoding, it applies to message shares and non-message shares.

[evan-forbes]

I like that idea. our whole naming scheme has kinda been thrown off since we changed PayForMessage to PayForData (which was a really good change imo). Might just want to refactor the Message struct to ShareData or something and stick everything in that.

[rootulp]

I've been confused on this because I've been using Message for message shares and did not think that term applied to shares in the reserved namespace (i.e. transactions, ISRs, evidence). See #660 (comment)

If we've been using Message to also describe data in reserved namespaces, I totally agree with your suggestion and will create an issue for it.

Update: created celestiaorg/celestia-core#848