Skip to content

Commit

Permalink
update euler guide (#794)
Browse files Browse the repository at this point in the history 
* jupyterhub is a fully integrated service on euler now

* typo
  • Loading branch information
@emanuel-schmid
emanuel-schmid authored Dec 6, 2023
1 parent 77ac8fb commit 3851c48
Showing 1 changed file with 27 additions and 129 deletions.
156 changes: 27 additions & 129 deletions doc/guide/Guide_Euler.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,14 @@
"\n",
"### 1. Load dependencies\n",
"\n",
"Use the new software stack. Unless you have already done so, run `set_software_stack.sh new`.\n",
"\n",
"```bash\n",
"env2lmod\n",
"module load gcc/8.2.0 python/3.9.9 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.22.1 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5\n",
"module load gcc/8.2.0 python/3.10.4 hdf5/1.10.1 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.31.0 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5\n",
"```\n",
"\n",
"You need to execute these two lines every time you login to Euler before Climada can be used. To safe yourself from doing it manually, one can append these lines to the ~/.bashrc script, which is automatically executed upon logging in to Euler."
"You need to execute this every time you login to Euler before Climada can be used. \n",
"To safe yourself from doing it manually, append these lines to the ~/.bashrc script, which is automatically executed upon logging in to Euler."
]
},
{
Expand All @@ -91,9 +93,11 @@
"```bash\n",
"python -c 'import climada; print(climada.__file__)'\n",
"```\n",
" should output something like this:\n",
"\n",
"should output something like this:\n",
"\n",
"```bash\n",
"/cluster/apps/nss/gcc-8.2.0/python/3.9.9/x86_64/lib64/python3.9/site-packages/climada/__init__.py\n",
"/cluster/apps/nss/gcc-8.2.0/python/3.10.4/x86_64/lib64/python3.10/site-packages/climada/__init__.py\n",
"```"
]
},
Expand Down Expand Up @@ -419,162 +423,56 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"It is possible to run a Jupyter Notebook on Euler and access it through a browser from your local machine.\n",
"To make this possible you have to do some preparation steps on both, your local machine and on Euler.\n",
"\n",
"**Windows Users:** to run the commands given below on your local computer you have to use `git-bash` (or WSL `bash` if Ubuntu on Windows is installed)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 1. Get the _Jupyter on Euler_ scripts and documentation\n",
"on your local computer, run these commands:\n",
"\n",
"```bash\n",
"git clone git@gitlab.ethz.ch:sfux/Jupyter-on-Euler-or-Leonhard-Open.git\n",
"cd Jupyter-on-Euler-or-Leonhard-Open\n",
"git checkout add_slurm\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 2. Create an SSH key pair\n",
"It is possible to run a Jupyter Notebook on Euler within a JupytherHub instance running as an interactive slurm job.\n",
"See the documentation https://scicomp.ethz.ch/wiki/JupyterHub.\n",
"\n",
"on your local computer, run this command:\n",
"For using climada inside the jupyter notebook, create a `.jupyterlabrc` file in your Euler home directory with the following content:\n",
"\n",
"```bash\n",
"ssh-keygen ssh-keygen -t ed25519 -f $HOME/.ssh/id_ed25519_euler\n",
"```\n",
"\n",
"When prompted, enter a _strong_ passphrase.\n",
"\n",
"Then, run\n",
"\n",
"```bash\n",
"ssh-add $HOME/.ssh/id_ed25519_euler\n",
"module purge\n",
"module load StdEnv gcc/8.2.0 python_gpu/3.10.4 eth_proxy r/4.2.2 julia/1.8.5 hdf5/1.10.1 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.31.0 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5\n",
"```\n",
"\n",
"and provide the passphrase from above.\n",
"\n",
"**Windows Users:** `ssh-add` is working together with an SSH-Agent, which is not running out of the box in `git-bash`. Follow the instructions [here](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/working-with-ssh-key-passphrases) to make it available."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 3. Copy the public key into the authorized_keys file on Euler\n",
"\n",
"```bash\n",
"ssh-copy-id -i $HOME/.ssh/id_ed25519_euler.pub USERNAME@euler.ethz.ch\n",
"```\n",
"\n",
"replace `USERNAME` with your ETH user name. When prompted to enter the passwort provide your LADP password (i.g., it is the same as your E-mail password).\n",
"\n",
"At this point you should be able to access Euler through `ssh` without password:\n",
"\n",
"```bash\n",
"ssh USERNAME@euler.ethz.ch\n",
"```\n",
"should bring you directly to the Euler cluster."
"Then login to https://jupyter.euler.hpc.ethz.ch/ and start a JupyterLab server."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 4. Prepare Euler environment\n",
"### Using a virtual environment in a Jupyter notebook\n",
"\n",
"It is highly recommanded to use a python environment and a github clone as described above under **Working with Git Branches**.\n",
"By default the pre-installed climada version is running in your notebooks. If you want to use climada from source you can simply install a python kernel from the `climada_venv` environment, see [Working with Git branches](#Working-with-Git-branches)\n",
"\n",
"Login to Euler.\n",
"\n",
"Unless you have done so already you need to permanently switch to the new software stack:\n",
"```bash\n",
"set_software_stack.sh new\n",
"```\n",
"\n",
"Unless you have done so already you need to append the default module selection in your `~/.bashrc` file:\n",
"```bash\n",
"echo >> ~/.bashrc\n",
"echo module load gcc/8.2.0 python/3.9.9 gdal/3.4.3 geos/3.9.1 proj/8.2.1 libspatialindex/1.9.3 netcdf/4.6.3 eccodes/2.22.1 zlib/1.2.9 libtiff/4.2.0 sqlite/3.35.5 >> ~/.bashrc\n",
"```\n",
"Install an IPyhton-kernel:\n",
"\n",
"Activate your python environment:\n",
"```bash\n",
"source /cluster/project/climate/$USER/climada_venv/bin/activate\n",
"```\n",
"\n",
"Install an IPyhton-kernel:\n",
"```\n",
"python -m ipykernel install --user --name climada_venv\n",
"``` \n",
"\n",
"Logout."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 5. Adjust the script\n",
"\n",
"Open the script `start_jupyter_nb.sh` in an editor and change the gcc and python version numbers in lines 421/422 so they look like this:\n",
"```\n",
"[421] JNB_MODULE_COMMAND=\"gcc/8.2.0 r/4.0.2 python/3.9.9 eth_proxy $JNB_JULIA\"\n",
"[422] echo -e \"Using new software stack (gcc/8.2.0 python/3.9.9 eth_proxy $JNB_JULIA)\"\n",
"\n",
"```\n",
"\n",
"make the script executable:\n",
"```bash\n",
"chmod +x start_jupyter_nb.sh\n",
"```"
"Start a new JupyterLab server, the `climada_venv` kernel should appear in the list of available kernels in JupyterHub."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### 6. Start a Jupyter Notebook on Euler\n",
"\n",
"At this point everything is ready. You should be able to start a Jupter Notebook remotely running on Euler.\n",
"## Trouble shooting\n",
"\n",
"Check out the script:\n",
"```bash\n",
"./start_jupyter_nb.sh\n",
"```\n",
"### 1. Python Module not found or available\n",
"\n",
"Run Jupyter on Euler on a single cpu (default) with 1G of memory (default) in your projects directory with the slurm batch system:\n",
"```bash\n",
"./start_jupyter_nb.sh -u NETHZID -b SLURM -w /cluster/project/climate/USERNAME\n",
"```\n",
"- Make sure your python environment is activated.\n",
"- Run `pip install --upgrade MISSING_MODULE`.\n",
"\n",
"If all went fine your default browser will be opened after some time with Jupyter running in your project directory.\\\n",
"When opening or creating a notebook make sure you select the `climada_venv` kernel."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. Upgrading from Python 3.9 to 3.10\n",
"\n",
"## Trouble shooting\n",
"Virtual environments created with Python 3.9 are i.g. not working for Python 3.10.\n",
"In particular Python kernels from 3.9 environments will fail to connect in a Jupyter notebook on https://jupyter.euler.hpc.ethz.ch/.\n",
"\n",
"### 1. Python Module not found or available\n",
"- Make sure your python environment is activated.\n",
"- Run `pip install --upgrade MISSING_MODULE`."
"- It's suggested to create new environments and remove the old kernels from `~/.local/share/jupyter/kernels/`."
]
}
],
Expand Down

0 comments on commit 3851c48

Please sign in to comment.