Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relative tmp address #19

Open
fataltes opened this issue Apr 10, 2020 · 3 comments
Open

Relative tmp address #19

fataltes opened this issue Apr 10, 2020 · 3 comments

Comments

@fataltes
Copy link

Hi @rizkg and others,

First of all, thank you for your awesome MPHF implementation. It's fast, and easy to use.

There is a need for generating some temporary files on disk during the construction. But the address used for temp files is relative to where one is running the code from. This happened to make our run fail a couple of times, while we were running the code from a disk that was full but giving the output address on another disk that had enough space (also, the binary was not in the same disk as the one we were running our program from). That took us a while to wonder what is causing this failure.

If not wrong, I think the part that one can change root address for temp files to where the final output MPHF is stored should be here: https://github.com/rizkg/BBHash/blob/master/BooPHF.h#L1377
We fixed this by allowing a prefix for the temporary directory to be passed to the constructor. But you as authors might have a better way to do it.

Either way, it seems it would be useful to allow, at least optionally, to override the current behavior.

Thanks,
Fatemeh

@rchikhi
Copy link
Collaborator

rchikhi commented Apr 11, 2020

Hi Fatemeh, giving a prefix to the constructor sounds to me like a great way to fix this.

@rchikhi
Copy link
Collaborator

rchikhi commented Apr 11, 2020

I'll note also that BBHash has a mode where it doesn't use the disk. I believe it's writeEach=false in the constructor

@fataltes
Copy link
Author

Thanks @rchikhi ,
you're right. It has the mode. But I thought this makes it use maximum memory and we would like to limit memory usage by allowing intermediate disk usage.

@fataltes fataltes reopened this Apr 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants