Add benchmark report #463
Merged
Add benchmark report #463
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AntonOsika
reviewed
Jul 2, 2023
|
Thanks for the PR! Suggested a small clarification to the print statement. I agree with Was unsure if I wanted to populate it if the user didn't explicitly set it, but now I think it should. |
70ziko
pushed a commit
to 70ziko/gpt-engineer
that referenced
this pull request
Oct 25, 2023
* Add benchmark report * Update scripts/benchmark.py --------- Co-authored-by: Artem Moskvin <artemm@spotify.com> Co-authored-by: Anton Osika <anton.osika@gmail.com>
|
This series of comments smells spammy. @MrDaVinC please respond to this comment to substantiate your comments. |
User blocked. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses #240 by adding the benchmark report after running the evaluation step. Additionally, this report can be inserted in the
benchmark/RESULTS.mdfile if needed. The insertion is a bit hacky now because it is based on certain assumptions around the results file structure. I believe there's a better way of doing it so any feedback is appreciated.On a more general note, I was a bit confused by the difference between the
WorksandPerfectcriteria. The current evaluation logic leavesWorksempty if a user replies 'yes' toPerfectwhich is a bit misleading. I think if the code works it works and ideally we would measure that using tests which is probably part of the roadmap. Anyway, please take a look and let me know what y'all think 🙏