Jobs always in pend status suddenly

Users/Site Administrators
1

Hi Team,

the jobs in PBS cluster suddenly are not dispatched and always in pend status.
the comment shows below content. could you please share any suggestion?

Not Running: Insufficient amount of resource: mem (R: 3096mb A: 0
kb T: 76098994172kb)

pbs version is v18.1.2

Thanks.

2

  1. Please check the resources_available.mem and resources_assigned.mem on our compute nodes by running pbsnodes -av or pbsnodes -aSj

  2. R == Requested memory
    A == Available memory
    T == Total memory

  3. Your job that is in the “Q” state has requested more memory than what is available on any of the compute node(s) of your PBS Complex. Hence your job is in the queued state

  4. Try submitting the below command and check it it works

qsub --  /bin/sleep 1000
qstat -answ1
ps -ef | grep sleep ( on the compute node on which above job is running )

Hope it works out for you.

3

Thanks Adarsh.

  1. there is available memory on the hosts.
  2. if I submit job without mem=xx , the job will start quickly.

any further suggestion.
Thanks.

4

Could you please share

  1. pbsnodes -av output when the job is showing the above R A T message in its comment (like above) 2. tracejob
  2. qstat -fx
5

user@host:~$ tracejob 135652

Job: 135652.server

10/28/2019 03:00:03 S enqueuing into queueA, state 1 hop 1
10/28/2019 03:00:03 S Job Queued at request of nagios@server, owner = nagios@server, job name = STDIN, queue =
queue
10/28/2019 03:00:03 S Job Modified at request of Scheduler@server
10/28/2019 03:00:03 A queue=queue
10/28/2019 05:38:40 L Considering job to run
10/28/2019 05:38:40 L Insufficient amount of resource: mem (R: 40000mb A: 0kb T: 76098994172kb)

pbsnodes -aSj:
ip-172-19-133-1 free 4 4 0 89gb/158gb 6/24 0/0 0/0 119826,135241,133093,133343
ip-172-19-133-1 job-busy 2 2 0 50gb/158gb 0/24 0/0 0/0 130707,131254
ip-172-19-133-1 free 15 15 0 50gb/158gb 6/24 0/0 0/0 120651,124265,135526,128912,129028,130100,130920,130951,130987,130990,132289,134059,134147,134224,134240
ip-172-19-133-1 free 10 10 0 28gb/158gb 4/24 0/0 0/0 123546,124773,127780,129965,131380,132352,134568,134463,134589,134688
ip-172-19-133-1 free 11 11 0 72gb/158gb 6/24 0/0 0/0 135138,135140,126055,127143,131163,131333,131553,131709,132245,132992,133589
ip-172-19-133-1 free 7 7 0 66gb/158gb 10/24 0/0 0/0 124751,124957,125678,128288,130860,131577,133214
ip-172-19-133-1 free 14 14 0 24gb/158gb 4/24 0/0 0/0 127054,127834,134488,134594,134510,134552,134433,132139,132540,133366,133461,133499,133844,133943

output of qstat:
comment = Not Running: Insufficient amount of resource: mem (R: 40000mb A:
0kb T: 76098994172kb)

Thanks.

6

Request pbsnodes -av and qstat -fx 135652 output as well