I am fairly new to PBS and I have a series of questions about how I can control process distribution in PBS. For all of these questions, lets assume I have a 16 node cluster, each node has 32 cores.
For the first set of questions lets assume I want to do qsub -l select-17:ncpus=4 -l place=<WHAT GOES HERE> or if the answer isn’t place= how do I get what I want.
Scenario 1: I want the minimum possible set of nodes, which in this case would end up being 2 complete nodes and a 3rd node with one process using 4 cores. I tried place=pack, but AFAICT that only works for a single node, so I would call this packN or min_nodes.
Feb 19: Updated Scenario 2 because I realized my description ended up with 33 processes, not 17 as it should have.
Scenario2: I want the maximum possible set of nodes, which would be 1 processes of 4 cores on each of the 16 nodes with one of the 16 having a 2nd 4 core process. I tried place=scatter, but the job never runs because it says there are not enough nodes. I would call this scatterN or maxnodes.
Once I have the above, I want to start adding in mpiprocs and I want to control the sequence in the pbsnodes file.
AFAICT it will always pack them, so in Scenario 1 above, assuming I got nodes 1-3 I would get n1 on the first 8 lines, n2 on the next 8, and then one line with n3. That is often what we want, but I would also like to be able to get round robin: n1,n2,n3,n1,n2,n1…n2.
In Scenario 2: I would get n1,n1,n1,n2,n2,n3.n3…n16,n16. Again, often that is fine, but I would like to also be able to get n1,n2…n16,n1,n2…n16,n1
Maybe I am missing something obvious or maybe this just can’t be done, but I would appreciate any ideas or input you all may have.