I’m about a year and a half into an HPC role that has me administering a 35 node Ubuntu cluster. 1 head node, 1 viz node, 1 admin node, 32 compute nodes. My background previously was developing machine & deep learning models and general data science. Currently, it has a proprietary scheduler and extensive scripting written to create chroot jails and an overlayfs populated with the contents of squashfs containers on the compute nodes.
I have a complete new set of hardware soon to be setup (yay!) which we will be using a PBS & Singularity workflow. After upgrading to a recent version of Singularity I have been able to successfully run the squashfs containers in Singularity. Now I’m trying to understand how to run PBS commands in the containers.
Do I need to use hooks in PBS to the Singularity containers? Create a bootstrapped script? Any advice or conversation is welcomed.
Bit more detail
I’m needing to use some additional features. Specifically:
Binded Directories/Mounts (Singularity feature)
Persistent Overlay (Singularity feature)