Failed to assign resources when hypertheading is enabled

Users/Site Administrators
1

Hi, I need help about how to execute jobs in an hyperthreading node.
My configuration is: a single node and hyperthreading active which provides 36 cores / 72 threads.

By default pbsnodes shows that the node has 36 available cores but I want to submit the maximum of sequential jobs using hyperthreading: 72

In order to allow execute in all of the threads I configure the node with:
(base) [root@node01 ~]# pbsnodes -a
node01
Mom = node01
Port = 15002
pbs_version = 20.0.0
ntype = PBS
state = free
pcpus = 72
resources_available.arch = linux
resources_available.host = node01
resources_available.hpmem = 0b
resources_available.mem = 385555mb
resources_available.ncpus = 72
resources_available.ngpus = 2
resources_available.vmem = 389587mb
resources_available.vnode = node01
resources_assigned.accelerator_memory = 0kb
resources_assigned.hbmem = 0kb
resources_assigned.mem = 0kb
resources_assigned.naccelerators = 0
resources_assigned.ncpus = 0
resources_assigned.vmem = 0kb
resv_enable = True
sharing = default_shared
last_state_change_time = Tue Apr 20 15:21:50 2021
last_used_time = Wed Apr 21 10:05:23 2021

However when I submit a number of single jobs (echo “sleep 60” | qsub -l select=1:ncpus=1) greater than 36 only the first 36 jobs are in running status, the rest of jobs are changed to hold status.
Checking the mom logs I can read something like:

pbs_python;Hook;pbs_python;Processing error in pbs_cgroups handling execjob_begin event for job 947.node01: CgroupProcessingError (‘Failed to assign resources’,)
pbs_version = 20.0.0
I don’t know where is the problem.
Thanks in advance.

2

Could you please disable the cgroups hook and try.

qmgr -c “set hook pbs_cgroups enabled=false”

Note: i am not sure whether you are using cgroups for requirements.
Please type the above line, sometimes copy paste has some special characters.

3

Yes that worked.
It means that the cgroup constraints are disabled for all PBS jobs submitted to the system? cpu, mem, etc ?

I tried enabling pbs_cgroups and use_hyperthreads = true but not working.

Below my pbs_cgroups.json configuration:
{
“cgroup_prefix” : “pbs_jobs”,
“exclude_hosts” : [],
“exclude_vntypes” : [“no_cgroups”],
“run_only_on_hosts” : [],
“periodic_resc_update” : true,
“vnode_per_numa_node” : false,
“online_offlined_nodes” : true,
“use_hyperthreads” : false,
“ncpus_are_cores” : false,
“cgroup” : {
“cpuacct” : {
“enabled” : true,
“exclude_hosts” : [],
“exclude_vntypes” : []
},
“cpuset” : {
“enabled” : true,
“exclude_cpus” : [],
“exclude_hosts” : [],
“exclude_vntypes” : [],
“mem_fences” : true,
“mem_hardwall” : false,
“memory_spread_page” : false
},
“devices” : {
“enabled” : true,
“exclude_hosts” : [],
“exclude_vntypes” : [],
“allow” : [
“c 195:* m”,
“c 136:* rwm”,
[“infiniband/rdma_cm”,“rwm”],
[“fuse”,“rwm”],
[“net/tun”,“rwm”],
[“tty”,“rwm”],
[“ptmx”,“rwm”],
[“console”,“rwm”],
[“null”,“rwm”],
[“zero”,“rwm”],
[“full”,“rwm”],
[“random”,“rwm”],
[“urandom”,“rwm”],
[“cpu/0/cpuid”,“rwm”,"*"],
[“nvidia-modeset”, “rwm”],
[“nvidia-uvm”, “rwm”],
[“nvidia-uvm-tools”, “rwm”],
[“nvidiactl”, “rwm”],
“b : rwm”,
“c : rwm”
]
},
“hugetlb” : {
“enabled” : false,
“exclude_hosts” : [],
“exclude_vntypes” : [],
“default” : “0MB”,
“reserve_percent” : 0,
“reserve_amount” : “0MB”
},
“memory” : {
“enabled” : true,
“exclude_hosts” : [],
“exclude_vntypes” : [],
“soft_limit” : false,
“default” : “256MB”,
“reserve_percent” : 0,
“reserve_amount” : “64MB”
},
“memsw” : {
“enabled” : false,
“exclude_hosts” : [],
“exclude_vntypes” : [],
“default” : “256MB”,
“reserve_percent” : 0,
“reserve_amount” : “64MB”
}
}
}

4

Please check whether you are using the latest cgroups hook/configuration file:

Please check the cgroup documentation at : https://www.altair.com/pdfs/pbsworks/PBSAdminGuide2021.1.pdf