So okay, I know there is documentation with two documented approaches:
Basic Scheduling: you declare a resource, but it is mainly for fun, as each job will exclusivly use a complete node no matter how many gpus are requested. So a dedicated queue would work almost the same.
Advanced Scheduling: You declare one vnodes per GPU, but in this case, is it possible to qsub –lselect=1:ncpus=1:ngpus=2 ?
In my cluster I have nodes with 4GPUS, I want to be able to have several individual jobs booking one GPU on one given node, but also to have jobs booking the entire node easily. Is this possible?
*** Edit ***
What I want is to be able to do qsub –lselect=1:ncpus=1:ngpus=2 or two times qsub –lselect=1:ncpus=1:ngpus=1 on the same node.
I would recommend that you look into the cgroup hook. If I recall correctly it identifies nodes with GPUs and can set the resources on the server. It also makes sure that jobs that requests GPUs have access to them and jobs that just request the cores cannot access the GPUs.
We’ve got problem in the past in trying using cgroup with PBS. We’re still on PBS 13 version, and with the hook we had and CentOs7, things were not working properly.
Isn’t there some hook to set CUDA_VISIBLE_DEVICE or something like that if we don’t want to use cgroup?
If you have 2 GPU cards on the system, create configv2 file, which creates 2 vnodes , assign CUDA_VISIBLE_DEVICE=0 to vnode[0] and CUDA_VISIBLE_DEVICE=1 to vnode[1] using a runjob hook
You need to write a execjob_launch (to set the correct CUDA_VISIBLE_DEVICE environment variable by querying underlying gpu cards) and execjob_end (to remove the correct variable assigned to CUDA_VISIBLE_DEVICE)
It may or may not work, depending on which version of the cgroups hook is being used with version 13. The latest 19.1 cgroups hook, for example, adds event handling for the new execjob_resize event, which will cause errors when used with version 18 or earlier where that event does not exist in the server etc. Further, PBS Pro 13.x used Python 2.5, later releases 2.7, and there are some Python 2.7-isms in more modern revisions of the cgroups hook.
While these are not insurmountable problems, there may be other unknown problems, this is not anything that is tested. Best bet is to upgrade to 18/19 to get a tested integration.
I’m also trying to configure a single node GPU cluster with 16 GPUs for advanced GPU scheduling, at the moment I’m reading through the Big Book.
I’ve summarized my understanding, before I start the configuration I would like to verify that I’m doing the right thing. Please correct me if I’m wrong:
4 Run jobs
CUDA_VISIBLE_DEVICES will be automatically set by the PBS server, multiple users should be able to run jobs if the GPU request can be satisfied.
In section 3.2 is it correct that the GPUs must be numbered from 0 to 15?