Hi,
Can 2 mpi programs use the same processor of same node simultaneously? I have been trying to make the best use of cpus, so trying to use over-subscription of cpus, to achieve the desired goal. But could not find any configuration in openpbs for the same.
Also, is there any way to increase the cpu allocated to a job in middle way, suppose previous job stops running, so its nodes became available, then can those be allocated to this other job that is running?
AFAIK, PBS does not allow for oversubscription of cpus, you could qrun both jobs on the same node manually, bypassing the PBS scheduler altogether. Another way might be to qsub with ncpus=1, even though your MPI job will use more (won’t work if you are using cgroups).
As Adarsh mentioned, you can’t provide a job more resources once it’s already running. Can you mention why you desire this? Maybe your use case can be fulfilled via another route.
I’ll try using qrun for oversubscription.
I need this as basic setup for my research work. I need to extend openpbs [as torque has been extended : paper.pdf (illinois.edu)] to support malleable jobs, which can adapt to the resources available at runtime. Is there any developers document for openpbs or a documentation for the openpbs code?
Please check this:
The pcpus resource represents the physical CPUs pbs_mom has detected on the system. The resources_available.ncpus resource is what the scheduler uses when scheduling jobs. If you set resources_available.ncpus to a higher value you can oversubscribe/undersubscribe the system.
Thankyou so much for suggesting this.
Also I was thinking of using hooks to increase cpus allocated to the job at runtime. As we can execute the hooks periodically, we can check if any resource became available. If the resources are available, we can use pbs.select.increment_chunks() to increase the cpus allocated to the job that is already running. I am new to pbs and hooks. Can you please suggest if this is the correct approach
I might be wrong, but our documentation says that increment_chunks() can only work when the job has not run yet, so this might not work.
PBS has a feature which does the opposite: you can release nodes from a running job … but adding more to a running job is really something that should go through the scheduler otherwise you’ll have 2 brains trying to allocate nodes to jobs, and they will clash. So, maybe we could think of adding a feature to PBS which allows a running job to be altered, the scheduler could look at the alter request, try to find the additional resources for it and have the server just instruct the execution daemon to add more resources to an additional job … we can already do this for reservations, so it might not be a big change, but I might be overlooking some complication. What do you think @bhroam ?
Certain function calls are only available in certain hooks. There is a table in the hooks guide which tells you what functions can be used where. I highly suspect that incrament_hooks() is meant to be used in a queue job hook only. It’s meant to be used with the feature that @agrawalravi90 talked about where we reduce the number of nodes at runtime. This is meant so we over allocate nodes, and then perform a health check on them. We’ll find the number of nodes we really need, and release the rest.
As for oversubscription, PBS doesn’t support anything past what @mkaro suggested. You can raise and lower the number of resources_available.ncpus on the nodes. This will tell the scheduler how many resources to use.
A job can’t be modified after it started running. The best you can do is preempt one to run another, or raise/lower the number of resources_available.ncpus in a hook.
Bhroam
Really appreciate your help regarding this feature. Yes, the new allocation of cpus should be via scheduler only. If that feature could be added, it will be really helpful for me. We can have an initial option as well describing the maximum cpus that the job can use efficiently. I am not aware about the reservation, will look into it.