-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Single node pipeline processing #58
Comments
Hi @manuparra, it's generally on the road map to support other non-SLURM platforms, although I must admit that single node/VM support isn't a strong part of that, as multi-node processing is a crucial design of the pipeline. However, I believe one can use MPI on a single node/VM anyway, assuming there are enough cores to make it worthwhile. And a few tasks like tclean can also make use of multiple cores through OpenMP. So, with a few tweaks, I have used the pipeline successfully on a single VM. However, that was before version 1.1, in which we introduced SPW splitting. To do this, one can take the sbatch scripts written and simply run them as bash scripts, since the
And you might also want to look at redirecting the output to a log within your sbatch scripts, with something like this:
However, doing this with SPW splitting (where Another trick that might be useful is the Are you thinking of doing some of this development yourself? What's the platform and software you're using? I'd be happy to walk you through doing some of these things if it's useful. |
Hi @manuparra, I suppose you could consider a high level script that runs the custom Of course, beyond that, you could abandon the bash approach altogether and use Python or something else, but that may require quite a bit of development. |
Hi all, I've been reviewing the code and I see that the pipeline scripts are tightly coupled to their execution in
slurm
(of course this is a pipeline for this kind of clusters). We are doing some tests to run the pipeline withoutslurm
, directly using singularity, but we see that there are references to variables and of course to procedures of theslurm
structure, which makes this complicated.Do you think it could be ported to a model without
slurm
?The text was updated successfully, but these errors were encountered: