Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up shrinking in some cases #1192

Merged
merged 1 commit into from
Feb 29, 2024
Merged

Conversation

samalws-tob
Copy link
Collaborator

@samalws-tob samalws-tob commented Feb 28, 2024

This PR changes the process used to shrink a contract call. For a given contract call, instead of trying to shrink all the parameters at once, it picks a subset of the parameters and shrinks those. Subset size is chosen out of [1,2,rng(0,n),n]; this means that sometimes we will try to shrink just one parameter, sometimes we will try to shrink all parameters, etc.
I'm fairly sure this fixes #1028 , and it also definitely fixes the following proof-of-concept bug, which the current echidna code on the master branch is unable to shrink:

contract Test {
  function foo(bool a, uint256 b) public {
    assert(!a); // `b` won't shrink to 0 because echidna always tries to shrink `a` to False at the same time
  }
}

Summary by CodeRabbit

  • Refactor
    • Improved the logic for shrinking values in ABI calls, ensuring more efficient and uniform shrinking probabilities.

Copy link

coderabbitai bot commented Feb 28, 2024

Walkthrough

The update to the Echidna.ABI module, particularly the shrinkAbiCall function, introduces a refined logic for value shrinking, which now includes a method of random selection and ensures uniform probabilities for shrinking. This change aims to enhance the efficiency and speed of the shrinking process, addressing issues of slow performance in certain scenarios.

Changes

File(s) Change Summary
lib/Echidna/ABI.hs Refactored shrinkAbiCall to include sophisticated logic for value shrinking, with random selection and uniform shrinking probabilities.

Assessment against linked issues

Objective Addressed Explanation
Address slow shrinking in rare situations (#1028)
Investigate slow shrinking with increased worker count (#1105) The change does not directly address the issue of worker count affecting shrinking speed.

Related issues

Poem

In the code where the bits gently hop,
A rabbit worked without a stop.
🐇 With a tweak and a twist, it refined,
Shrinking logic, redefined.
🌟 No more waiting, no more stall,
Echidna's calls, now swift for all.
In the digital fields, it leaps, a sight so grand,
A rabbit's touch, making code so efficient and grand.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit-tests for this file.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit tests for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository from git and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit tests.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 1

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between e72975c and f39dc29.
Files selected for processing (1)
  • lib/Echidna/ABI.hs (2 hunks)

Comment on lines 277 to 301
shrinkAbiCall (name, vals) = do
let numShrinkable = length $ filter canShrinkAbiValue vals

halfwayVal <- getRandomR (0, numShrinkable)
-- This list was made arbitrarily. Feel free to change
let numToShrinkOptions = [1, 2, halfwayVal, numShrinkable]

numToShrink <- min numShrinkable <$> uniform numToShrinkOptions
shrunkVals <- shrinkVals (fromIntegral numShrinkable) (fromIntegral numToShrink) vals
pure (name, shrunkVals)
where
shrinkVals _ 0 l = pure l
shrinkVals _ _ [] = pure []
shrinkVals numShrinkable numToShrink (h:t)
| not (canShrinkAbiValue h) = (h:) <$> shrinkVals numShrinkable numToShrink t
| otherwise = do
-- We want to pick which ones to shrink uniformly from the vals list.
-- Odds of shrinking one element is numToShrink/numShrinkable.
shouldShrink <- fromList [(True, numToShrink), (False, numShrinkable-numToShrink)]
h' <- if shouldShrink then shrinkAbiValue h else pure h
let
numShrinkable' = numShrinkable-1
numToShrink' = if shouldShrink then numToShrink-1 else numToShrink
(h':) <$> shrinkVals numShrinkable' numToShrink' t
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The refactoring of the shrinkAbiCall function introduces a more nuanced approach to shrinking contract call parameters by selecting a subset of parameters to shrink. This approach is expected to enhance the efficiency of the shrinking process, especially in scenarios where shrinking all parameters simultaneously leads to suboptimal outcomes. The implementation uses randomness and uniform probabilities in selecting the subset size and the parameters to shrink, which aligns with the PR objectives. However, there are a few points to consider:

  1. Randomness and Reproducibility: The use of randomness in the shrinking process can introduce variability in the outcomes. It's important to ensure that this variability does not compromise the reproducibility of test results. Consider documenting the seed used for random number generation or providing a mechanism to set it, ensuring consistent results across runs when needed.

  2. Efficiency and Performance: While the new approach aims to improve efficiency, it's crucial to assess its performance impact comprehensively. The introduction of randomness and the selection process for the subset of parameters to shrink could potentially introduce overhead. Conducting benchmark tests comparing the new and old strategies in various scenarios would be beneficial to quantify the improvements and identify any cases where the new approach might not perform as expected.

  3. Edge Cases and Robustness: The logic for determining the subset of parameters to shrink (lines 280-284) and the subsequent shrinking process (lines 290-300) should be thoroughly tested to ensure robustness. Consider edge cases such as functions with a large number of parameters, parameters that cannot be further shrunk, and scenarios where the chosen subset size is larger than the number of shrinkable parameters.

  4. Documentation and Comments: The comments in the code provide some insight into the logic and intentions behind the changes. However, expanding the documentation to include more details about the rationale, expected outcomes, and potential limitations of the new shrinking strategy would be helpful for future maintainers and users of the Echidna framework.

Overall, the changes to the shrinkAbiCall function represent a promising improvement to the Echidna testing framework's shrinking process. Addressing the points mentioned above can further enhance its effectiveness, reliability, and usability.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between e72975c and f4cb053.
Files selected for processing (1)
  • lib/Echidna/ABI.hs (2 hunks)
Files skipped from review as they are similar to previous changes (1)
  • lib/Echidna/ABI.hs

@arcz
Copy link
Member

arcz commented Feb 28, 2024

This is quite a fancy solution however I feel it's a bit too complex. Wouldn't something like this work too:

shrinkAbiCall (func, vals) = do
  vals' <- forM vals $ \val -> do
    chance :: Float <- getRandomR (0,1)
    -- 50% chance of shrinking a value
    if chance < 0.5 then shrinkAbiValue val
                    else pure val
  pure (func, vals')

@ggrieco-tob
Copy link
Member

This is quite a fancy solution however I feel it's a bit too complex. Wouldn't something like this work too:

shrinkAbiCall (func, vals) = do
  vals' <- forM vals $ \val -> do
    chance :: Float <- getRandomR (0,1)
    -- 50% chance of shrinking a value
    if chance < 0.5 then shrinkAbiValue val
                    else pure val
  pure (func, vals')

Actually, we don't need floats here, let's keep everything in integers if possible.

@samalws-tob
Copy link
Collaborator Author

The reason I wanted to avoid doing that is if we have a function with a lot of arguments and we can shrink only one of them, the odds of that one (and no others) getting picked by the simplified algo is 1/2^n

@arcz
Copy link
Member

arcz commented Feb 29, 2024

Ok, let's see how it performs and adjust if we feel the need

@arcz arcz merged commit d9f5016 into crytic:master Feb 29, 2024
17 of 18 checks passed
@arcz arcz mentioned this pull request Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug-Candidate]: Shrinking extremely slow in some rare situations
3 participants