Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Deep tree structure with Updates #13

Merged
merged 29 commits into from
Nov 3, 2022

Conversation

Manav-Aggarwal
Copy link
Member

@Manav-Aggarwal Manav-Aggarwal commented Nov 1, 2022

This PR is just a reference to #5 which adds a deep tree structure with updates, however, it was forked off of master which uses cosmos-db as the underlying DB, however, the default branch in cosmos-sdk-rollmint: release/v0.46.x-rollmint uses iavl 0.19.4 which corresponds to release/v0.19.x in this repo and uses tm-db. Also, it was opened by @tzdybal so he could not approve it. We'll use this PR as a proxy for that PR in order for approval.

Please look at #5 for relevant comments.

@Manav-Aggarwal Manav-Aggarwal changed the title Manav/dst with updates feat: Create deep tree structure using an ICS23-style approach: Updates Nov 1, 2022
@Manav-Aggarwal Manav-Aggarwal changed the title feat: Create deep tree structure using an ICS23-style approach: Updates feat: Deep tree structure with Updates Nov 1, 2022
@Manav-Aggarwal Manav-Aggarwal linked an issue Nov 1, 2022 that may be closed by this pull request
6 tasks
@Manav-Aggarwal Manav-Aggarwal self-assigned this Nov 1, 2022
@Manav-Aggarwal Manav-Aggarwal force-pushed the manav/dst_with_updates branch 2 times, most recently from a04e42b to 397bd4d Compare November 1, 2022 16:23
@Manav-Aggarwal Manav-Aggarwal marked this pull request as ready for review November 1, 2022 16:26
deepsubtree.go Outdated Show resolved Hide resolved
deepsubtree.go Outdated Show resolved Hide resolved
deepsubtree.go Outdated Show resolved Hide resolved
deepsubtree.go Outdated Show resolved Hide resolved
@Manav-Aggarwal Manav-Aggarwal changed the base branch from release/v0.19.x to deepsubtrees November 3, 2022 10:28
@codecov-commenter
Copy link

codecov-commenter commented Nov 3, 2022

Codecov Report

❗ No coverage uploaded for pull request base (deepsubtrees@f4499a9). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             deepsubtrees      #13   +/-   ##
===============================================
  Coverage                ?   64.24%           
===============================================
  Files                   ?       28           
  Lines                   ?     4645           
  Branches                ?        0           
===============================================
  Hits                    ?     2984           
  Misses                  ?     1278           
  Partials                ?      383           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

MSevey
MSevey previously requested changes Nov 3, 2022
deepsubtree.go Outdated

// Link the given node if it is not linked yet
// If already linked, return an error in case connection was made incorrectly
// Note: GetNode returns nil if the hash passed into it is empty which is expected
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a valid reason to ignore the error because this doesn't handles the case of a corrupt hash and nil node, only the case when node is nil and hash is nil, so GetNode is returning nil, ErrNodeMissingHash. But we would never hit that case because we are checking for > 0 at the start. So we know that if we are calling GetNode we are calling it with a non empty hash.

Therefore the errors we should be getting from GetNode should be an error getting the node or making the node, both of which seem like errors we should care about.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the comment. The error starting with Value missing for hash is expected with a deep subtree since it does not contain all nodes but can contain left hashes. ErrMissingHash will never be true since we are checking > 0 like you said.

if len(node.leftHash) > 0 {
if node.leftNode == nil {
node.leftNode, _ = dst.ndb.GetNode(node.leftHash)
} else if !bytes.Equal(node.leftNode.hash, node.leftHash) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building off of my other comment, we should either explain why it is ok to return without an error and still a nil node.leftNode in this case, or we should update this to the following as an additional protection against mismatches.

Suggested change
} else if !bytes.Equal(node.leftNode.hash, node.leftHash) {
}
if !bytes.Equal(node.leftNode.hash, node.leftHash) {

Copy link
Member Author

@Manav-Aggarwal Manav-Aggarwal Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an invalid code since if node.leftNode == nil is true, then node.leftNode.hash will crash. GetNode can return a nil value for node.leftNode and that's completely fine.

@Manav-Aggarwal Manav-Aggarwal dismissed MSevey’s stale review November 3, 2022 21:43

Code suggested would cause crashes.

@Manav-Aggarwal Manav-Aggarwal merged commit 690c59a into deepsubtrees Nov 3, 2022
Manav-Aggarwal added a commit that referenced this pull request Dec 20, 2022
* ICS23-style approach for DeepSubTree creation

* Fix deepsubtree, all checks pass

* Update documentation for deep subtrees

* Fix deepsubtree to accomodate left/right paths both

* Refactor test code

* Refactor TestDeepSubtreeStepByStep

* Address comments

* Refactor code

* Disable gocritic and unused

* Add description

* Refactor code to check err in Set

* Modify traversal conditions to be clearer

* feat: build deep subtree from ICS23 inclusion proofs (#9)

* feat: build deep subtree from ICS23 inclusion proofs

* feat: handle non-existence proofs

* linter: goimports deepsubtree.go

* refactor: addExistenceProofProof -> addExistenceProof

* fix: un-hardcode size of byte array

* Add more comments

* Refactor ndb.Commit call outside for loop

* verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first

* Add strings and use strings.Repeat

* Delete addPath and AddPath

* Remove print statements

* Refactor recomputeHash usage

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* return err directly

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* Address linting checks

* Use tm-db instead of cosmos-db

* Fix linter

* Update comment

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* Add error checking for node hash

* turn lengthByte into a const

* Refactor linkNode

* Update comment

Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>
Co-authored-by: Matthew Sevey <mjsevey@gmail.com>
Manav-Aggarwal added a commit that referenced this pull request Dec 20, 2022
* ICS23-style approach for DeepSubTree creation

* Fix deepsubtree, all checks pass

* Update documentation for deep subtrees

* Fix deepsubtree to accomodate left/right paths both

* Refactor test code

* Refactor TestDeepSubtreeStepByStep

* Address comments

* Refactor code

* Disable gocritic and unused

* Add description

* Refactor code to check err in Set

* Modify traversal conditions to be clearer

* feat: build deep subtree from ICS23 inclusion proofs (#9)

* feat: build deep subtree from ICS23 inclusion proofs

* feat: handle non-existence proofs

* linter: goimports deepsubtree.go

* refactor: addExistenceProofProof -> addExistenceProof

* fix: un-hardcode size of byte array

* Add more comments

* Refactor ndb.Commit call outside for loop

* verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first

* Add strings and use strings.Repeat

* Delete addPath and AddPath

* Remove print statements

* Refactor recomputeHash usage

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* return err directly

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* Address linting checks

* Use tm-db instead of cosmos-db

* Fix linter

* Update comment

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* Add error checking for node hash

* turn lengthByte into a const

* Refactor linkNode

* Update comment

Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>
Co-authored-by: Matthew Sevey <mjsevey@gmail.com>
Manav-Aggarwal added a commit that referenced this pull request Jan 18, 2023
* feat: Deep tree structure with Updates (#13)

* ICS23-style approach for DeepSubTree creation

* Fix deepsubtree, all checks pass

* Update documentation for deep subtrees

* Fix deepsubtree to accomodate left/right paths both

* Refactor test code

* Refactor TestDeepSubtreeStepByStep

* Address comments

* Refactor code

* Disable gocritic and unused

* Add description

* Refactor code to check err in Set

* Modify traversal conditions to be clearer

* feat: build deep subtree from ICS23 inclusion proofs (#9)

* feat: build deep subtree from ICS23 inclusion proofs

* feat: handle non-existence proofs

* linter: goimports deepsubtree.go

* refactor: addExistenceProofProof -> addExistenceProof

* fix: un-hardcode size of byte array

* Add more comments

* Refactor ndb.Commit call outside for loop

* verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first

* Add strings and use strings.Repeat

* Delete addPath and AddPath

* Remove print statements

* Refactor recomputeHash usage

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* return err directly

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* Address linting checks

* Use tm-db instead of cosmos-db

* Fix linter

* Update comment

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* Add error checking for node hash

* turn lengthByte into a const

* Refactor linkNode

* Update comment

Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>
Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* feat: Support Adds in a Deep Subtree (#8)

* ICS23-style approach for DeepSubTree creation

* Fix deepsubtree, all checks pass

* Update documentation for deep subtrees

* Fix deepsubtree to accomodate left/right paths both

* Refactor test code

* Refactor TestDeepSubtreeStepByStep

* Address comments

* Refactor code

* Disable gocritic and unused

* Add description

* Refactor code to check err in Set

* Modify traversal conditions to be clearer

* feat: build deep subtree from ICS23 inclusion proofs (#9)

* feat: build deep subtree from ICS23 inclusion proofs

* feat: handle non-existence proofs

* linter: goimports deepsubtree.go

* refactor: addExistenceProofProof -> addExistenceProof

* fix: un-hardcode size of byte array

* Add more comments

* Refactor ndb.Commit call outside for loop

* verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first

* Add strings and use strings.Repeat

* Delete addPath and AddPath

* Remove print statements

* Refactor recomputeHash usage

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* return err directly

Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* Address linting checks

* Use tm-db instead of cosmos-db

* Fix linter

* Add test to check add operation in deepsubtree: WIP

* Add insert key functionality WIP

* Try replicating non-inclusion proofs

* Add sibling nodes to nonExistenceProofs

* Refactor test code

* Finish adding add functionality to dst

* Add Remove functionality to dst

* Fix linting errors

* Fix GetSiblingNode

* Remove more print statements

* Add comment for each case in recursive set

* Change which methods are exported and document exported methods

* feat: Support Empty Hashes and Add constructor (#11)

* Export siblings

* Add deepsubtree constructor

* Support empty root hashes

* Use working hash instead of root.hash

* Use tm-db instead of cosmos-db

* Return nil in BuildTree

* Address comments

* Address more comments

Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>
Co-authored-by: Matthew Sevey <mjsevey@gmail.com>

* Add randomized tests for adding/removing keys (#16)

* Add go fuzz tests

* Add membership proof for existing keys

* Build tree after adding membership proof

* Make batch add fuzz tests work

* Do not commit version twice for dst

* Save version out of dst.Set condition

* Set rootHash to workingHash

* Fix edge cases

* Refactor DST Non-Existence Proof

* Change cacheSize

* Add test data and sibling nodes for each node in path

* Fix GetSiblingNodes

* Add more test data

* testing: fuzz: failing test case

* Use set for keys

* Add more test data

* Refactor existence proofs code

* Add required info for remove operation

* Add children of siblings as well

* Remove debug code

* Add testdata that breaks current code

* Fix bug

* Add failing testcase

* Add breaking testcase

* IAVL with tracing

* Fuzz tests pass

* Refactor tracing code

* Remove redundant code

* Remove working hash in calls to AddExistenceProof

* Clean up flag

* Make build tree a private method

* Add back whitespace in node.go

* Add new ci for fuzz

* Refactor more

* Refactor out getKey method

* Change name to reapInclusionProofs

* Refactor set and remove in DST

* Factor out add existence proofs from Remove DST for consistency with Set

* Refactor into testContext

* Clean up setInDST

Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>

* Add tracing to Trees and Deep Subtrees (#18)

* Add go fuzz tests

* Add membership proof for existing keys

* Build tree after adding membership proof

* Make batch add fuzz tests work

* Do not commit version twice for dst

* Save version out of dst.Set condition

* Set rootHash to workingHash

* Fix edge cases

* Refactor DST Non-Existence Proof

* Change cacheSize

* Add test data and sibling nodes for each node in path

* Fix GetSiblingNodes

* Add more test data

* testing: fuzz: failing test case

* Use set for keys

* Add more test data

* Refactor existence proofs code

* Add required info for remove operation

* Add children of siblings as well

* Remove debug code

* Add testdata that breaks current code

* Fix bug

* Add failing testcase

* Add breaking testcase

* IAVL with tracing

* Fuzz tests pass

* Refactor tracing code

* Remove redundant code

* Remove working hash in calls to AddExistenceProof

* Clean up flag

* Make build tree a private method

* Add back whitespace in node.go

* Add new ci for fuzz

* Refactor more

* Refactor out getKey method

* Change name to reapInclusionProofs

* Refactor set and remove in DST

* Factor out add existence proofs from Remove DST for consistency with Set

* Refactor into testContext

* Clean up setInDST

* Add method for get

* Export methods

* Add witness data to deep subtree

* Verify operation in witness data

* Refactor and verify operation for get and remove

* Add set witness data

* Add tracing to tree

* add getter for witness data

* Verify existence proofs in dst

* Cleanup

* Reset witness data on tracing enabled

* Add node to keysAccessed even not in cache

* Add initial root hash

* Refactor GetInitialRootHash

* Modify GetInitialRootHash

* Add test data

* Add get to dst tests: fails right now

* Refactor and add tracing for root key as well

* Add docs

* Add more docs

* Update comments

* Update log message

Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>

* allocate length

Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>

* fix: remove `RangeProofs` (cosmos#586)

Co-authored-by: colin axnér <25233464+colin-axner@users.noreply.github.com>
Co-authored-by: Marko <marbar3778@yahoo.com>

* Update go.mod file

* Add new line at end of .golangci.yml

Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl>
Co-authored-by: Matthew Sevey <mjsevey@gmail.com>
Co-authored-by: cool-developer <51834436+cool-develope@users.noreply.github.com>
Co-authored-by: colin axnér <25233464+colin-axner@users.noreply.github.com>
Co-authored-by: Marko <marbar3778@yahoo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create deep tree structure using an ICS23-style approach: Updates
4 participants