Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --db-index, --db-file, --taxonomy, --db-pipe for piped db files #89

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

yesimon
Copy link

@yesimon yesimon commented Oct 24, 2017

This allows loading the database from a pipe instead of a file. This is useful in cloud platforms like AWS/GCP to load the database from an object store download and decompress on the fly, which is both faster and cheaper than storing on networked disk.

Also allows processing multiple inputs in one run. This is more important for the loading from pipe scenario because the database files are not implicitly cached in the page cache between runs.

Allow kraken to operate on multiple inputs -> outputs
@tomkinsc
Copy link

👍 It would be great to see this merged in.

@jenniferlu717
Copy link
Collaborator

I'll take a look at this ASAP (hopefully today, if not by the end of this week)

@clintval
Copy link

clintval commented Jan 5, 2018

+1 Interested in this feature. Thanks!

@maxmaliska
Copy link

+1 also interested in this feature!

@nh13
Copy link

nh13 commented Feb 7, 2018

@DerrickWood also interest on this end.

@clintval
Copy link

@jenniferlu717 Any chance of a review or merge?

@jenniferlu717
Copy link
Collaborator

Apologies....we've been working on releasing Kraken 2.0 ASAP and haven't had a chance to review this. It has a number of conflicts with the current code so merging is a bit more difficult but I will try to look at it soon.....

@yesimon
Copy link
Author

yesimon commented May 11, 2018

In that case perhaps it's better for me to wait until 2.0 before resolving conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants