feat: add ledger sync protocol

[bhgomes]

Description

This PR adds the pull method for use by wallets to request the diff from the latest state.

Unit Test Spec

Use sudo to set the storage to random UTXO and void numbers: for each round add random number (1 to 2) of random (UTXO, EncryptedNote) pairs and a random number (0 to 2) of random VoidNumbers:

• Shards: Add random (UTXO, EncryptedNote) pairs with a random first key and increasing second key
• InsertionOrderedVoidNumbers: Add random VoidNumbers with increasing key

Checkpoint Description:

• receiver_index: [usize; 256]: Lengths of each first key in Shards
• sender_index: usize: Length of InsertionOrderedVoidNumber

Testing correctness of Normal Usage:

1. Add random elements to ledger
2. Select a random checkpoint that's below the current one
3. Pull the latest state and check that it returns the data items (in order!) up to the checkpoint returned in the diff
4. If should_continue == true then pull should be called again with the returned checkpoint (GOTO step 3)
5. If it passes GOTO step 1

Testing correctness of Abnormal Usage: Same as the previous test except try to get from a checkpoint later than the current one. The pull should return nothing.

TODO

• Implement diffing algorithm
• Add RPC interface
• Add Unit tests / Simulation testing

---

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

• Linked to Github issue with discussion and accepted design OR have an explanation in the PR that describes this work.
• Wrote unit tests.
• Updated relevant documentation in the code.
• Added one line describing your change in <branch>/CHANGELOG.md
• Re-reviewed Files changed in the Github PR explorer.
• If runtime changes, need to update the version numbers properly:
  • authoring_version: The version of the authorship interface. An authoring node will not attempt to author blocks unless this is equal to its native runtime.
  • spec_version: The version of the runtime specification. A full node will not attempt to use its native runtime in substitute for the on-chain Wasm runtime unless all of spec_name, spec_version, and authoring_version are the same between Wasm and native.
  • impl_version: The version of the implementation of the specification. Nodes are free to ignore this; it serves only as an indication that the code is different; as long as the other two versions are the same then while the actual code may be different, it is nonetheless required to do the same thing. Non-consensus-breaking optimizations are about the only changes that could be made which would result in only the impl_version changing.
  • transaction_version: The version of the extrinsics interface. This number must be updated in the following circumstances: extrinsic parameters (number, order, or types) have been changed; extrinsics or pallets have been removed; or the pallet order in the construct_runtime! macro or extrinsic order in a pallet has been changed. You can run the metadata_diff.yml workflow for help. If this number is updated, then the spec_version must also be updated
• Verify benchmarks & weights have been updated for any modified runtime logics
• If importing a new pallet, choose a proper module index for it, and allow it in BaseFilter. Ensure every extrinsic works from front-end. If there's corresponding tool, ensure both work for each other.
• If needed, update our Javascript/Typescript APIs. These APIs are officially used by exchanges or community developers.
• If modifying existing runtime storage items, make sure to implement storage migrations for the runtime and test them with try-runtime. This includes migrations inherited from upstream changes, and you can search the diffs for modifications of #[pallet::storage] items to check for any.

[github-actions]

Please signoff on all commits with your name, email and gpg key for the Developer's Certificate of Origin.
git config --global user.name your_name
git config --global user.email you_email
COMMAND : git commit -s -S -m your_commit_message
-s = Signed-off-by
-S = Verify commit using gpg key
If there is more than one commit in your pull request and your git client is modern enough (2.13+), rebase the required number of commits with --signoff:
git rebase --signoff HEAD~<number_of_commits>
Then, force push:
git push -f origin <your_branch>
For instructions on managing gpg signature verification please visit: https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

[Dengjianping]

When runtime team integrate v0.9.22(jsonrpsee), we better mark storage method as blocking.
But not there's any rpc method like this in manta-pay.
paritytech/substrate#11459

[stechu]

When runtime team integrate v0.9.22(jsonrpsee), we better mark storage method as blocking. But not there's any rpc method like this in manta-pay. paritytech/substrate#11459

Indeed. Some context here: paritytech/substrate#11401

[Dengjianping]

LGTM!

[Garandor]

Partial review, I did not look at node and tests changes yet.

As a note for the future, there are a lot of unrelated clippy/style improvements (e.g. in collator-selection) interspersed with the feature changes

In a big PR like this, I'd recommend to separate these out into a separate "cleanup" PR to contain the scope of changes and in this case, make the actual rpc change easier to locate and reason about.

@Dengjianping : manta-pay looks like it's gonna need a rewrite for the 0. 9.22 jsonrpsee migration.

[Dengjianping]

Correct, I just added a comment to #571.

[Dengjianping]

@Garandor You just reminded me that we might do storage migration for manta-pay, but I didn't find the feature try-runtime in manta-pay.

Can we add this feature to manta-pay?

[Garandor]

Finished looking through the rest of the code.

The node looks great, thanks for the extensible design and all your refactoring of the messy code in service @bhgomes and/or @stechu
We'll soon get rid of all the aura-cumulus code in there as well and then this file will have some semblance of readability.

[Garandor]

Can we add this feature to manta-pay?

Agree, it should be added along with the migration.
And migrating is mandatory, otherwise the new void numbers storage will be empty and ... all spent UTXOs become spendable again?

[Garandor]

one more question, then LGTM

I logged a followup to re-align the chainspec files across all runtimes
#581

[Garandor]

One more question: The unit tests @bhgomes describes are not really implemented here. Is there a tracking issue to follow up?

[stechu]

One more question: The unit tests @bhgomes describes are not really implemented here. Is there a tracking issue to follow up?

#579

[Garandor]

thanks for the new naming, it all LGTM except for the migration issue.

@Dengjianping actually alluded to that before that we should add the try_runtime feature to manta-pay so we can verify the runtime upgrade will succeed before actually issuing it to the chain.

I'll give it a shot to implement it so we can merge this PR today

[Garandor]

After discussion, migration of asset_manager will be discussed in #572, so this is good to go