Argos crop

NB: *** If you ran the cropping tool before and already have your data stored in your vantage mount you could skip step 3 and go straight to the 'Epoch training and Prediction' section below. This involves copying the data from your vantage6 mount back to a local folder. If you are not comfortable or can't find your vantage mount, please run the cropping tool again as instructed in step 3 ***

This code requires that your data is already pulled from XNAT and stored in a folder on your host machine. If you have not done so yet, please run the XNAT extraction code first or contact Leonard Wee for instructions. This folder should contain 2 subfolders; pre-process-TRAIN, and pre-process-VALIDATE.

Step 0: *** If you previously done our ARGOS GPU standalone test you can skip steps 1 and 2, because it was done during that test. ***

Step 1: If you have not already done so, install an nvidia driver:
1: Identify the type of GPU in your system by using the console and typing: ubuntu-drivers devices
2: Your GPU and a list of drivers should appear. Please make a printscreen of this output and send it to us. We recommend installing the latest recommended version or a version >= 418.81.07. For example on a Tesla M60 we would type: sudo apt install nvidia-driver-490
3: Reboot your Ubuntu machine.

Step 2: The next step is to install Docker-CE by typing the commands below or follow the official instructions here (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker): curl https://get.docker.com | sh && sudo systemctl --now enable docker

Next we install the NVIDIA Container toolkit:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list |sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update: sudo apt-get update
Install the nvidia docker container: sudo apt-get install -y nvidia-docker2
Restart the docker daemon: sudo systemctl restart docker
We can test if the installation was succesful by typing: sudo docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

Step 3: Download and unzip the argos-crop-main.zip
Change your working directory to the argos-crop-main folder, for example, on my machine it is /home/ubuntu/argos-crop-main :\

cd {your_path_here}/argos-crop-main

Next we build the container:\

sudo docker build -f Dockerfile -t crops .

Please read the following instructions fully before executing.
To run the preprocessing you need to adjust the path in this command to the directory where the folders called "pre-process-TRAIN" and "pre-process-VALIDATE" are found. You would have done these by running Leonard's batch conversion to NRRD script and then extracted them from XNAT (see Leonard's guidance videos):\

sudo docker run --gpus all -v {/path/to/your/data}:/home/leroy/app/data -ti crops python3 -u crop_lung_volume.py

NB : Change the above "{/path/to/your/data}" to your own actual full path. For example, on my machine it is /home/ubuntu/xnat-docker-compose-master/pyradiomics-master/o-raw\

This script will search for lung slices and only select those for deep learning. It will create them in a new folder called "Train" and "Validate". It will put these at the same location where "pre-process-TRAIN" and "pre-process-VALIDATE" were located. Additionally, it will write 2 csv files (train_list.csv and validation_list.csv) with patient folder names and image shapes (e.g. patient-001, (512, 512, 89)). Please check if these csv files are okay to send, and send them to Leonard Wee.

---

Epoch training and Prediction

To run our run_online_epoch.py and predict_full.py scripts, please first COPY your Train and Validation folders from the Vantage6 mount to a directory where you have permissions (e.g. /home/...) or run the cropping tool from step 3 above again. To gain access to the vantage6 mount you need 'sudo su' privileges. These 3 scripts will save 6 .csv files. Please note that 2 of these scripts can have a runtime of several hours.

Change directory to the folder containing this code and then build the container:

sudo docker build -f Dockerfile -t crops .

Please run this check first (~5 minutes). This loads all the images and saves 2 csv files with image shapes. Any strange results in this list might need to be removed first. If you get a 'Permission Denied' error where a .csv file can't be saved, we suggest changing the permission configuration of your data folder '{/home/ubuntu/ARGOS_Data}' e.g. chmod -R 777 ./

sudo docker run -v {/home/ubuntu/ARGOS_Data}:/home/leroy/app/data -ti crops python3 -u test_loading.py

sudo docker run --gpus all -v {/home/ubuntu/ARGOS_Data}:/home/leroy/app/data -ti crops python3 -u run_online_epoch.py

NB: Please change "{/home/ubuntu/ARGOS_Data}" to your new path.

sudo docker run --gpus all -v /home/ubuntu/ARGOS_Data:/home/leroy/app/data -ti crops python3 -u predict_full.py

Once you are finished running everything you can change the permissions back with chmod -R 755 ./