# Nodes resources not getting released to the resource pool after the job is run and exited

**URL:** <https://community.openpbs.org/t/nodes-resources-not-getting-released-to-the-resource-pool-after-the-job-is-run-and-exited/3280>\
**Category:** Users/Site Administrators\
**Created:** [September 15, 2022, 8:20am UTC](https://community.openpbs.org/t/nodes-resources-not-getting-released-to-the-resource-pool-after-the-job-is-run-and-exited/3280 "2022-09-15T08:20:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![anieshmathew](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@anieshmathew](https://community.openpbs.org/u/anieshmathew)\
**Post date:** [September 15, 2022, 8:20am UTC](https://community.openpbs.org/t/nodes-resources-not-getting-released-to-the-resource-pool-after-the-job-is-run-and-exited/3280/1 "2022-09-15T08:20:39Z")

</div>

Hi,

We have 4 worker nodes of which 2 are down. We have run the multiple cases on 2 available nodes. However, all of a sudden the same job which ran earlier goes into a que and mentions " comment = Not Running: Not enough free nodes available"

I can see that the pbsnodes -a still shows old job ids and is currently in “state = job-busy”. I am unsure why the old jobs ids are still being shown when the jobs have run and exited.

The nodes have 8 cpus and the old jobs have taken 0 to 7 from each node. There is something that is not releasing the resources of old jobs back to the pool. How can we fix this issue?

```auto
`[root@mgt1 pbs]# qstat -answ1

mgt1:
                                                                                                   Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
------------------------------ --------------- --------------- --------------- -------- ---- ----- ------ ----- - -----
288.mgt1 hpcuser workq perl_job 14898 2 12 -- 00:30 E 00:00:01 node01/0*6+node03/0*6
   Job run at Wed Sep 14 at 19:10 on (node01:ncpus=6)+(node03:ncpus=6)
289.mgt1 hpcuser workq perl_job 33051 2 12 -- 00:30 E 00:00:01 node01/1*6+node03/1*6
   Job run at Thu Sep 15 at 10:16 on (node01:ncpus=6)+(node03:ncpus=6)
290.mgt1 hpcuser workq perl_job 33120 2 12 -- 00:30 E 00:00:01 node01/2*6+node03/2*6
   Job run at Thu Sep 15 at 10:16 on (node01:ncpus=6)+(node03:ncpus=6)
291.mgt1 hpcuser workq perl_job 33206 2 12 -- 00:30 E 00:00:01 node01/3*6+node03/3*6
   Job run at Thu Sep 15 at 10:18 on (node01:ncpus=6)+(node03:ncpus=6)
292.mgt1 hpcuser workq perl_job 33390 2 12 -- 00:30 E 00:00:01 node01/4*6+node03/4*6
   Job run at Thu Sep 15 at 10:25 on (node01:ncpus=6)+(node03:ncpus=6)
293.mgt1 hpcuser workq perl_job 33503 2 12 -- 00:30 E 00:00:00 node01/5*6+node03/5*6
   Job run at Thu Sep 15 at 10:28 on (node01:ncpus=6)+(node03:ncpus=6)
294.mgt1 hpcuser workq perl_job 33572 2 12 -- 00:30 E 00:00:01 node01/6*6+node03/6*6
   Job run at Thu Sep 15 at 10:29 on (node01:ncpus=6)+(node03:ncpus=6)
295.mgt1 hpcuser workq perl_job 33641 2 12 -- 00:30 E 00:00:03 node01/7*6+node03/7*6
   Job run at Thu Sep 15 at 10:30 on (node01:ncpus=6)+(node03:ncpus=6)
310.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 Q -- --
   Not Running: Not enough free nodes available
(base) [root@mgt1 pbs]#
`

```

```auto

> [root@mgt1 ~]# pbsnodes -a
node01
     Mom = node01
     Port = 15002
     pbs_version = 19.1.1
     ntype = PBS
     state = job-busy
     pcpus = 48
     jobs = 288.mgt1/0, 288.mgt1/1, 288.mgt1/2, 288.mgt1/3, 288.mgt1/4, 288.mgt1/5, 289.mgt1/6, 289.mgt1/7, 289.mgt1/8, 289.mgt1/9, 289.mgt1/10, 289.mgt1/11, 290.mgt1/12, 290.mgt1/13, 290.mgt1/14, 290.mgt1/15, 290.mgt1/16, 290.mgt1/17, 291.mgt1/18, 291.mgt1/19, 291.mgt1/20, 291.mgt1/21, 291.mgt1/22, 291.mgt1/23, 292.mgt1/24, 292.mgt1/25, 292.mgt1/26, 292.mgt1/27, 292.mgt1/28, 292.mgt1/29, 293.mgt1/30, 293.mgt1/31, 293.mgt1/32, 293.mgt1/33, 293.mgt1/34, 293.mgt1/35, 294.mgt1/36, 294.mgt1/37, 294.mgt1/38, 294.mgt1/39, 294.mgt1/40, 294.mgt1/41, 295.mgt1/42, 295.mgt1/43, 295.mgt1/44, 295.mgt1/45, 295.mgt1/46, 295.mgt1/47
     resources_available.arch = linux
     resources_available.host = node01
     resources_available.mem = 65184884kb
     resources_available.ncpus = 48
     resources_available.vnode = node01
     resources_assigned.accelerator_memory = 0kb
     resources_assigned.hbmem = 0kb
     resources_assigned.mem = 0kb
     resources_assigned.naccelerators = 0
     resources_assigned.ncpus = 48
     resources_assigned.vmem = 0kb
     resv_enable = True
     sharing = default_shared
     last_state_change_time = Thu Sep 15 12:57:01 2022
     last_used_time = Mon Sep 5 17:43:15 2022

node02
     Mom = node02
     Port = 15002
     pbs_version = 19.1.1
     ntype = PBS
     state = state-unknown,down
     pcpus = 48
     resources_available.host = node02
     resources_available.ncpus = 48
     resources_available.vnode = node02
     resources_assigned.accelerator_memory = 0kb
     resources_assigned.hbmem = 0kb
     resources_assigned.mem = 0kb
     resources_assigned.naccelerators = 0
     resources_assigned.ncpus = 0
     resources_assigned.vmem = 0kb
     comment = node down: communication closed
     resv_enable = True
     sharing = default_shared
     last_state_change_time = Thu Sep 15 12:57:01 2022
     last_used_time = Fri Nov 20 12:59:37 2020

node03
     Mom = node03
     Port = 15002
     pbs_version = 19.1.1
     ntype = PBS
     state = job-busy
     pcpus = 48
     jobs = 288.mgt1/0, 288.mgt1/1, 288.mgt1/2, 288.mgt1/3, 288.mgt1/4, 288.mgt1/5, 289.mgt1/6, 289.mgt1/7, 289.mgt1/8, 289.mgt1/9, 289.mgt1/10, 289.mgt1/11, 290.mgt1/12, 290.mgt1/13, 290.mgt1/14, 290.mgt1/15, 290.mgt1/16, 290.mgt1/17, 291.mgt1/18, 291.mgt1/19, 291.mgt1/20, 291.mgt1/21, 291.mgt1/22, 291.mgt1/23, 292.mgt1/24, 292.mgt1/25, 292.mgt1/26, 292.mgt1/27, 292.mgt1/28, 292.mgt1/29, 293.mgt1/30, 293.mgt1/31, 293.mgt1/32, 293.mgt1/33, 293.mgt1/34, 293.mgt1/35, 294.mgt1/36, 294.mgt1/37, 294.mgt1/38, 294.mgt1/39, 294.mgt1/40, 294.mgt1/41, 295.mgt1/42, 295.mgt1/43, 295.mgt1/44, 295.mgt1/45, 295.mgt1/46, 295.mgt1/47
     resources_available.arch = linux
     resources_available.host = node03
     resources_available.mem = 65184884kb
     resources_available.ncpus = 48
     resources_available.vnode = node03
     resources_assigned.accelerator_memory = 0kb
     resources_assigned.hbmem = 0kb
     resources_assigned.mem = 0kb
     resources_assigned.naccelerators = 0
     resources_assigned.ncpus = 48
     resources_assigned.vmem = 0kb
     resv_enable = True
     sharing = default_shared
     last_state_change_time = Thu Sep 15 12:57:01 2022
     last_used_time = Mon Sep 5 17:31:19 2022

node04
     Mom = node04
     Port = 15002
     pbs_version = 19.1.1
     ntype = PBS
     state = state-unknown,down
     pcpus = 24
     resources_available.host = node04
     resources_available.ncpus = 24
     resources_available.vnode = node04
     resources_assigned.accelerator_memory = 0kb
     resources_assigned.hbmem = 0kb
     resources_assigned.mem = 0kb
     resources_assigned.naccelerators = 0
     resources_assigned.ncpus = 0
     resources_assigned.vmem = 0kb
     comment = node down: communication closed
     resv_enable = True
     sharing = default_shared
     last_state_change_time = Thu Sep 15 12:57:01 2022
     last_used_time = Sat Feb 15 10:25:30 2020

```

The job runs when I manually delete an old job. Hence please let us know how the resource can be allocated back to the pool when the job finishes so we do not have to manually delete an old job. The status currently is E for jobs that have run and exited. It seems if the status changes to F the resources will be available to the resource pool without any manual intervention.

The pbs subbmission script is as below

```auto

> [hpcuser@mgt1 TEST_PBS]$ cat test.pl
foreach my $n(1..580000)
{
        print "$n\n";
}
(base) [hpcuser@mgt1 TEST_PBS]$ cat pbs_script.pbs
#!/bin/bash
#PBS -l nodes=2:ncpus=6
#PBS -l walltime=00:30:00
#PBS -N perl_job
cd /nfsshare/home/hpcuser/MBTD_Genetics/wilson/TEST_PBS

perl /nfsshare/home/hpcuser/MBTD_Genetics/wilson/TEST_PBS/test.pl > ./Result

(base) [hpcuser@mgt1 TEST_PBS]$ qsub pbs_script.pbs

```

Regards,  
Aniesh Mathew

---

<div class="post-metadata">

**Author:** ![anieshmathew](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@anieshmathew](https://community.openpbs.org/u/anieshmathew)\
**Post date:** [September 15, 2022, 9:09am UTC](https://community.openpbs.org/t/nodes-resources-not-getting-released-to-the-resource-pool-after-the-job-is-run-and-exited/3280/2 "2022-09-15T09:09:59Z")

</div>

Hi,

As I mentioned earlier the status of the job is E. When I delete a job, it changes to state F and the resources become available for job submission.

Is there anything that needs to be added in my batch script?

```auto

> [root@mgt1 pbs]# qstat -H

mgt1:
                                                            Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
250.mgt1 hpcuser workq run.sh 66370 1 1 -- -- F 00:00
251.mgt1 hpcuser workq HelloJob 66587 1 3 1gb -- F 00:00
252.mgt1 hpcuser workq HelloJob 66665 1 3 1gb -- F 00:00
253.mgt1 hpcuser workq HelloJob 66735 1 3 1gb -- F 00:00
254.mgt1 hpcuser workq HelloJob 66805 1 3 1gb -- F 00:00
255.mgt1 hpcuser workq run_trimmo 73846 1 1 -- -- F 00:00
257.mgt1 hpcuser workq STDIN -- 4 32 -- -- F --
258.mgt1 hpcuser workq STDIN 210895 2 16 -- -- F 00:19
260.mgt1 hpcuser workq perl_job 211412 1 8 -- 00:30 F 00:00
261.mgt1 hpcuser workq perl_job 211520 1 8 -- 00:30 F 00:00
262.mgt1 hpcuser workq STDIN -- 4 32 -- -- F --
263.mgt1 hpcuser workq STDIN 211712 1 8 -- -- F 00:01
264.mgt1 hpcuser workq perl_job 211846 1 8 -- 00:30 F 00:00
265.mgt1 hpcuser workq perl_job 211915 1 8 -- 00:30 F 00:00
266.mgt1 hpcuser workq perl_job 212000 1 8 -- 00:30 F 00:03
267.mgt1 hpcuser workq perl_job 212329 2 16 -- 00:30 F 00:00
268.mgt1 hpcuser workq perl_job 212416 2 16 -- 00:30 F 00:00
269.mgt1 hpcuser workq perl_job -- 2 16 -- 00:30 F --
272.mgt1 hpcuser workq perl_job -- 2 16 -- 00:30 F --
273.mgt1 hpcuser workq HelloJob 26688 1 3 1gb -- F 00:00
274.mgt1 hpcuser workq HelloJob 26840 1 3 1gb -- F 00:00
275.mgt1 hpcuser workq perl_job -- 2 16 -- 00:30 F --
276.mgt1 hpcuser workq perl -- 2 4 -- 00:01 F --
277.mgt1 hpcuser workq perl 27709 1 2 -- 00:01 F 00:00
278.mgt1 hpcuser workq perl 27891 1 2 -- 00:01 F 00:00
279.mgt1 hpcuser workq perl -- 2 4 -- 00:01 F --
280.mgt1 hpcuser workq perl_job 74128 1 8 -- 00:30 F 00:00
281.mgt1 hpcuser workq perl_job 74253 1 8 -- 00:30 F 00:00
282.mgt1 hpcuser workq perl_job -- 1 8 -- 00:30 F --
283.mgt1 hpcuser workq perl_job -- 1 8 -- 00:30 F --
284.mgt1 hpcuser workq perl_job -- 1 8 -- 00:30 F --
285.mgt1 hpcuser workq perl_job 74495 1 6 -- 00:30 F 00:00
287.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
288.mgt1 hpcuser workq perl_job 14898 2 12 -- 00:30 F 00:00
289.mgt1 hpcuser workq perl_job 33051 2 12 -- 00:30 F 00:00
290.mgt1 hpcuser workq perl_job 33120 2 12 -- 00:30 F 00:00
291.mgt1 hpcuser workq perl_job 33206 2 12 -- 00:30 F 00:00
292.mgt1 hpcuser workq perl_job 33390 2 12 -- 00:30 F 00:00
296.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
297.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
298.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
299.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
300.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
301.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
302.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
305.mgt1 hpcuser workq example-jo -- 1 1 -- -- F --
306.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
307.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
308.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
309.mgt1 hpcuser workq perl_job -- 2 12 -- 00:30 F --
314.mgt1 hpcuser workq STDIN -- 2 16 -- -- F --
315.mgt1 hpcuser workq STDIN -- 2 16 -- -- F --
316.mgt1 hpcuser workq STDIN 17071 2 8 -- -- F 00:01
(base) [root@mgt1 pbs]# qstat -a

mgt1:
                                                            Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
293.mgt1 hpcuser workq perl_job 33503 2 12 -- 00:30 E 00:00
294.mgt1 hpcuser workq perl_job 33572 2 12 -- 00:30 E 00:00
295.mgt1 hpcuser workq perl_job 33641 2 12 -- 00:30 E 00:00
310.mgt1 hpcuser workq perl_job 16303 2 12 -- 00:30 E 00:00
311.mgt1 hpcuser workq perl_job 16382 2 12 -- 00:30 E 00:00
312.mgt1 hpcuser workq perl_job 16505 2 12 -- -- E 00:00
313.mgt1 hpcuser workq perl_job 16953 2 12 -- -- E 00:00

```

Regards,  
Aniesh

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [September 15, 2022, 10:27am UTC](https://community.openpbs.org/t/nodes-resources-not-getting-released-to-the-resource-pool-after-the-job-is-run-and-exited/3280/3 "2022-09-15T10:27:30Z")

</div>

> [@anieshmathew](#):
>
> As I mentioned earlier the status of the job is E.

If the job state remain in E state (exiting state) for a long time, then there is an issue with your password-less scp (for that job owner) from server to node and node to server. Please check the mom logs (on the compute node that this job was running) against he job id

> [@anieshmathew](#):
>
> When I delete a job, it changes to state F and the resources become available for job submission.

E - Exiting state  
F - finished state

The problem seems to be with the SSH/SCP issue with the user. Make sure StrictHostkeychecking is set to no in the /etc/ssh/ssh\_config across the cluster.

---

<div class="post-metadata">

**Author:** ![anieshmathew](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@anieshmathew](https://community.openpbs.org/u/anieshmathew)\
**Post date:** [September 16, 2022, 1:07pm UTC](https://community.openpbs.org/t/nodes-resources-not-getting-released-to-the-resource-pool-after-the-job-is-run-and-exited/3280/4 "2022-09-16T13:07:17Z")

</div>

> [@adarsh](#):
>
> StrictHostkeychecking

Hi Adarsh,

Thank you for the response. The issue is passwordless ssh is not configured in our cluster.

We have shared the home directory of users on an NFS mount from the head node. We are facing issues in configuring passwordless ssh for users on NFS mounted home directories. I assume only host-based authentication is only configurable in such scenario.

Regards,  
Aniesh Mathew

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [September 16, 2022, 10:11pm UTC](https://community.openpbs.org/t/nodes-resources-not-getting-released-to-the-resource-pool-after-the-job-is-run-and-exited/3280/5 "2022-09-16T22:11:35Z")

</div>

HI Aniesh,

Both host-based /user key based (as the home directory is common across all the nodes) passwordless ssh can be supported on NFS mounted home directories. However, hostbased is the way to go.

Thank you