# Cgroups not cleaning up after job kill, further jobs unable to assign resources **URL:** \ **Category:** Users/Site Administrators\ **Created:** [June 29, 2023, 10:57am UTC](https://community.openpbs.org/t/cgroups-not-cleaning-up-after-job-kill-further-jobs-unable-to-assign-resources/3599 "2023-06-29T10:57:44Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\ **Post date:** [June 29, 2023, 10:57am UTC](https://community.openpbs.org/t/cgroups-not-cleaning-up-after-job-kill-further-jobs-unable-to-assign-resources/3599/1 "2023-06-29T10:57:44Z")
Hi, We have an issue when a job was killed. It took more than the kill\_delay of 180s and a force exit was initiated. 06/26/2023 18:57:25;0008;pbs\_mom;Job;16033.hpc-p-fea-pbs1;kill\_job 06/26/2023 18:57:25;0080;pbs\_mom;Job;16033.hpc-p-fea-pbs1;task 00000001 terminated * * * 06/26/2023 19:00:27;0080;pbs\_mom;Job;16033.hpc-p-fea-pbs1;task 00000001 force exited 06/26/2023 19:00:27;0008;pbs\_mom;Job;16033.hpc-p-fea-pbs1;Terminated 06/26/2023 19:00:28;0100;pbs\_mom;Job;16033.hpc-p-fea-pbs1;task 00000001 cput=23:52:59 06/26/2023 19:00:28;0008;pbs\_mom;Job;16033.hpc-p-fea-pbs1;kill\_job For some unknown reason the processes survived the force exit. 06/26/2023 19:00:50;0100;pbs\_python;Hook;pbs\_python;\_kill\_tasks: PID 37446 survived: [‘Name:\tstandard’, ‘State:\tD (disk sleep)’, ‘Uid:\t1570\t1570\t1570\t1570’] 06/26/2023 19:00:50;0100;pbs\_python;Hook;pbs\_python;\_kill\_tasks: PID 37447 survived: [‘Name:\tstandard’, ‘State:\tD (disk sleep)’, ‘Uid:\t1570\t1570\t1570\t1570’] 06/26/2023 19:00:50;0002;pbs\_python;Hook;pbs\_python;cgroup still has 15 tasks: /sys/fs/cgroup/cpuset/pbs\_jobs.service/jobid/16033.hpc-p-fea-pbs1 06/26/2023 19:00:50;0100;pbs\_python;Hook;pbs\_python;delete: Unable to delete cgroup for job 16033.hpc-p-fea-pbs1 PBS attempted to take the node offline: 06/26/2023 19:00:50;0100;pbs\_python;Hook;pbs\_python;take\_node\_offline: Taking vnode(s) offline 06/26/2023 19:00:50;0100;pbs\_python;Hook;pbs\_python;take\_node\_offline: Hook pbs\_cgroups: Unable to clean up one or more cgroups; offlining hpc-p-fea-n127 06/26/2023 19:00:50;0100;pbs\_python;Hook;pbs\_python;take\_node\_offline: Offline file already exists, not overwriting Appears that process then must have cleared: 06/26/2023 19:00:50;0100;pbs\_python;Hook;pbs\_python;Hook ended: pbs\_cgroups, job ID 16033.hpc-p-fea-pbs1, event\_type 512 (elapsed time: 10.7606) 06/26/2023 19:00:50;0008;pbs\_mom;Job;16033.hpc-p-fea-pbs1;no active tasks The node is then brough back online: 06/26/2023 19:02:25;0080;pbs\_python;Hook;pbs\_python;bring\_node\_online: Vnode hpc-p-fea-n127 will be brought back online Jobs are then attempted to launch on the node: 06/26/2023 19:06:28;0008;pbs\_mom;Job;16033.hpc-p-fea-pbs1;no active tasks 06/26/2023 19:07:48;0100;pbs\_mom;Req;;Type 1 request received from root@172.26.92.5:15001, sock=185 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;main: Event type is execjob\_begin, job ID is 16041.hpc-p-fea-pbs1 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;create\_job: Creating directory /sys/fs/cgroup/systemd/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1/ 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;create\_job: Creating directory /sys/fs/cgroup/cpu,cpuacct/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1/ 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;create\_job: Creating directory /sys/fs/cgroup/cpuset/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1/ 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;create\_job: Creating directory /sys/fs/cgroup/devices/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1/ 06/26/2023 19:07:49;0080;pbs\_python;Hook;pbs\_python;\_setup\_subsys\_devices: Entry not added to devices.allow: [‘nvidiactl’, ‘rwm’, ‘\*’] 06/26/2023 19:07:49;0080;pbs\_python;Hook;pbs\_python;\_setup\_subsys\_devices: Entry not added to devices.allow: [‘nvidia-uvm’, ‘rwm’] 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;create\_job: Creating directory /sys/fs/cgroup/memory/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1/ 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;configure\_job: mem not requested, assigning 134321225728 to cgroup 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;configure\_job: vmem not requested, assigning 138756698112 to cgroup 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;configure\_job: Failed to assign job resources 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;configure\_job: Resyncing local job data 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;\: Assignment of resources failed for 16041.hpc-p-fea-pbs1, attempting cleanup 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;cleanup\_orphans: Removing orphaned cgroup: /sys/fs/cgroup/systemd/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;\_remove\_cgroup: Removing directory /sys/fs/cgroup/systemd/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;cleanup\_orphans: Removing orphaned cgroup: /sys/fs/cgroup/cpu,cpuacct/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;\_remove\_cgroup: Removing directory /sys/fs/cgroup/cpu,cpuacct/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;cleanup\_orphans: Removing orphaned cgroup: /sys/fs/cgroup/devices/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;\_remove\_cgroup: Removing directory /sys/fs/cgroup/devices/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;cleanup\_orphans: Removing orphaned cgroup: /sys/fs/cgroup/memory/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;\_remove\_cgroup: Removing directory /sys/fs/cgroup/memory/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;cleanup\_orphans: Removing orphaned cgroup: /sys/fs/cgroup/cpuset/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;\_remove\_cgroup: Removing directory /sys/fs/cgroup/cpuset/pbs\_jobs.service/jobid/16041.hpc-p-fea-pbs1.orphan 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;configure\_job: Requeuing job 16041.hpc-p-fea-pbs1 06/26/2023 19:07:49;0080;pbs\_python;Hook;pbs\_python;[‘Traceback (most recent call last):’, ’ File “”, line 6165, in main’, ’ File “”, line 1016, in invoke\_handler’, ’ File “”, line 1065, in \_execjob\_begin\_handler’, ’ File “”, line 5059, in configure\_job’, ‘CgroupProcessingError: Failed to assign resources’] 06/26/2023 19:07:49;0001;pbs\_python;Hook;pbs\_python;Processing error in pbs\_cgroups handling execjob\_begin event for job 16041.hpc-p-fea-pbs1: CgroupProcessingError (‘Failed to assign resources’,) 06/26/2023 19:07:49;0100;pbs\_python;Hook;pbs\_python;Hook ended: pbs\_cgroups, job ID 16041.hpc-p-fea-pbs1, event\_type 64 (elapsed time: 0.5809) 06/26/2023 19:07:49;0100;pbs\_mom;Hook;pbs\_cgroups;execjob\_begin request rejected by ‘pbs\_cgroups’ 06/26/2023 19:07:49;0008;pbs\_mom;Job;16041.hpc-p-fea-pbs1;Processing error in pbs\_cgroups handling execjob\_begin event for job 16041.hpc-p-fea-pbs1: CgroupProcessingError (‘Failed to assign resources’,) 06/26/2023 19:07:49;0008;pbs\_mom;Job;16033.hpc-p-fea-pbs1;no active tasks Failed to assign resource as Job 16033 still has files on the node i.e: /sys/fs/cgroup/systemd/pbs\_jobs.service/jobid/16033.hpc-p-fea-pbs1/ Anyone know the best method to clean this up, guessing to restart the pbs service on the node? More importantly how to stop it happening again in the future? Some method to keep the node offline if the job files have not be cleaned? Many thanks. ---
**Author:** ![alexis.cousein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/alexis.cousein/32/265_2.png) [@alexis.cousein](https://community.openpbs.org/u/alexis.cousein)\ **Post date:** [July 3, 2023, 11:08pm UTC](https://community.openpbs.org/t/cgroups-not-cleaning-up-after-job-kill-further-jobs-unable-to-assign-resources/3599/2 "2023-07-03T23:08:20Z")
> [@CGully](#): > > More importantly how to stop it happening again in the future? Modify the cgroup hook to move the cgroups to .orphan (Cfr. cleanup\_orphans) in execjob\_end on failure. It will keep the node offline until the orphan cgroup can be cleaned. But the root cause is a process hung in the kernel doing I/O (the unkillable process is in state D). Filesystem client kernel modules are supposed to leave tasks in interruptible state after some time, this one apparently does not. Not much PBS can do. There is a flag in the cgroup config file that controls whether nodes should be onlined again by the hook if they were offlined by it. Set that to false and you’ll be able to sanity check a node before putting it online again (if you do best also remove the file the hook sets to mark it offlined the node; not hard to find in the mom\_priv tree).