-
Notifications
You must be signed in to change notification settings - Fork 313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cache source artifacts #992
Cache source artifacts #992
Conversation
With this commit we add a cache for source artifacts. It can be enabled by setting `cache = true` in the `source` section of `rally.ini`. Caching works based on the specified revision. We don't implement an eviction policy for now but assume that an external process cleans up cached artifacts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks and works great!
I left a question.
esrally/mechanic/supplier.py
Outdated
# this can be None if the Elasticsearch does not reside in a git repo and the user has only | ||
# copied all source files. In that case, we cannot resolve a revision hash and thus we cannot cache. | ||
if self.cached_path: | ||
shutil.copy(original_path, self.cached_path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if there is some issue, out of disk space, file in use etc? Should we catch IOError
, log it, and fallback to no caching?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I've pushed 937595e.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
With this commit we create the artifact cache directory if it does not exist yet. Otherwise, the artifact cannot be cached making the cache ineffective. Relates elastic#992
With this commit we create the artifact cache directory if it does not exist yet. Otherwise, the artifact cannot be cached making the cache ineffective. Relates #992
With this commit we add the artifact cache introduced by elastic#992 also for plugins and implement a cache cleanup mechanism that cleans up artifacts after seven days by default (can be configured with `cache.days`). As artifact caching can now replace the pipeline `from-sources-skip-build` we deprecate it along with `from-sources-complete` and introduce a new pipeline `from-sources` instead.
With this commit we add the artifact cache introduced by #992 also for plugins and implement a cache cleanup mechanism that cleans up artifacts after seven days by default (can be configured with `cache.days`). As artifact caching can now replace the pipeline `from-sources-skip-build` we deprecate it along with `from-sources-complete` and introduce a new pipeline `from-sources` instead.
With this commit we add a cache for source artifacts. It can be enabled
by setting
cache = true
in thesource
section ofrally.ini
.Caching works based on the specified revision. We don't implement an
eviction policy for now but assume that an external process cleans up
cached artifacts.