This document extends the basic Getting Started with NCI guide by focusing on practical workflows for managing multiple project-specific environments on NCI’s Gadi supercomputer. Python virtual environments are used throughout as a representative example.

It is intended for research scenarios where different projects require incompatible Python dependencies, and where jobs are submitted through PBS (both batch and interactive), including GPU-enabled workloads.

1. Managing Python Environments on NCI

On NCI systems, the home directory has a strict quota limit (typically 10 GB), which makes it unsuitable for storing large Python environments with many dependencies.

During active development, users often create Python virtual environments under /scratch/$USER, which provides larger temporary storage and higher I/O performance.

Note:
/scratch is a temporary filesystem subject to automatic cleanup.

Please note that auto-purge policy is in place for /scratch. More information is available at:

Gadi /scratch File Management - NCI Help - Opus - NCI Confluence

2. Creating Project-Specific Virtual Environments

Each project should use an isolated virtual environment to avoid dependency conflicts.

Project A Environment

module load python/3.9 python -m venv /scratch/$USER/envs/projA-py39 source /scratch/$USER/envs/projA-py39/bin/activate cd ~/projA pip install -r requirements.txt deactivate

Project B Environment (Different Dependencies)

module load python/3.9 # or another required version python -m venv /scratch/$USER/envs/projB-py39 source /scratch/$USER/envs/projB-py39/bin/activate cd ~/projB pip install -r requirements.txt deactivate

3. Submitting Batch Jobs with the Correct Environment

When submitting jobs via PBS, the required Python environment must be explicitly activated within the job script.

Project A Batch Job

#!/bin/bash #PBS -N projA_job #PBS -q gpuvolta #PBS -l ncpus=4 #PBS -l ngpus=1 #PBS -l mem=32GB #PBS -l walltime=02:00:00 #PBS -j oe cd ~/projA module load python/3.9 source /scratch/$USER/envs/projA-py39/bin/activate python train_A.py

Project B Batch Job

#!/bin/bash #PBS -N projB_job #PBS -q gpuvolta #PBS -l ncpus=4 #PBS -l ngpus=1 #PBS -l mem=32GB #PBS -l walltime=02:00:00 #PBS -j oe cd ~/projB module load python/3.9 source /scratch/$USER/envs/projB-py39/bin/activate python train_B.py

By explicitly activating the appropriate environment in each job script, users
can safely run multiple projects with different dependencies on the same system.

4. Interactive Jobs for Development and Debugging

During development, debugging, or environment testing, it is often more convenient to request an interactive job.

run_interactive.pbs:

bash #!/bin/bash #PBS -N projB_job #PBS -q gpuvolta #PBS -l ncpus=4 #PBS -l ngpus=1 #PBS -l mem=32GB #PBS -l walltime=02:00:00 #PBS -j oe qsub -I run_interactive.pbs

After entering interactive mode, you need to import the environment first and then set up the virtual environment:

module load python/3.9 source /scratch/$USER/envs/projA-py39/bin/activate