Hi everyone,
I am trying to configure the pbs_cgroups hook for resource isolation on a compute node, but I am hitting a roadblock with the cpuset subsystem on Cgroups v2.
Environment:
-
OS: Rocky Linux 9.6 (Linux version 5.14.0-570.21.1.el9_6.x86_64)
-
OpenPBS Version: pbs_version = 22.05.11
-
Cgroups: v2 (Unified Hierarchy)
The Problem: When a job is submitted to the node, the MoM rejects it, and I get the following error in the mom_logs:
Unexpected error in pbs_cgroups handling execjob_begin event for job 176709 (system hold set): CgroupConfigError (‘enabled: cgroups enabled but not mounted for subsystem cpuset’,)
Context & Configurations: I verified that the host is indeed running Cgroups v2 and that the cpuset controller is available at the root level:
$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc
Since it’s v2, I already removed the legacy cpuacct from my configuration. Here is one part of the pbs_cgroups.json I imported into the master:
{
“cgroup_prefix”: “pbs_jobs”,
“enabled”: true,
“run_only_on_hosts”: [“compute-0-X”],
“exclude_vntypes”: [“no_cgroup”],
“periodic_resc_update”: true,
“vnode_per_numa_node”: true,
“use_hyperthreads”: false,
“ncpus_are_cores”: true,
“discover_gpus”: true,
“nvidia-smi”: “/usr/bin/nvidia-smi”,
“manage_rlimit_as”: true,
“cgroup”: {
“cpuacct”: {
“enabled”: false,
“exclude_vntypes”: [“no_cgroup”]
},
“cpuset”: {
“enabled”: true,
“exclude_cpus”: ,
“exclude_vntypes”: [“no_cgroup”],
“allow_zero_cpus”: false,
“mem_fences”: false,
“mem_hardwall”: true,
“memory_spread_page”: true
},
…
}
My Question: Even though cpuset is listed in cgroup.controllers, the hook says it’s not mounted. Does the PBS daemon require a manual systemd override to allow the delegation of the cpuset controller to the MoM on Cgroups v2? Or is there something wrong with my JSON configuration?
Any guidance would be greatly appreciated! Thanks.