After restarting PBSPro (2022.1) it appears that the scheduler, server, and comm are down. When I submit job arrays it is unable to allocate resources to each job even though no jobs are running.
All nodes are free:
pbsnodes -aSj
mem ncpus nmics ngpus
vnode state njobs run susp f/t f/t f/t f/t jobs
kvn01 free 0 0 0 252gb/252gb 64/64 0/0 0/0 –
kvn02 free 0 0 0 252gb/252gb 64/64 0/0 0/0 –
kvn03 free 0 0 0 252gb/252gb 64/64 0/0 0/0 –
kvn04 free 0 0 0 252gb/252gb 64/64 0/0 0/0 –
kvn05 free 0 0 0 252gb/252gb 64/64 0/0 0/0 –
kvn06 free 0 0 0 252gb/252gb 64/64 0/0 0/0 –
kvn07 free 0 0 0 252gb/252gb 64/64 0/0 0/0 –
kvn08 free 0 0 0 252gb/252gb 64/64 0/0 0/0 –
But when i submit a job array I get “comment = Not Running: No available resources on nodes” and it gets put in the Q state indefinitely
qstat -f
Job Id: 17158.kvh.cm.cluster
Job_Name = test.sh
Job_Owner = scurti15@kvh.cm.cluster
job_state = Q
queue = workq
server = kvh.cm.cluster
Checkpoint = u
ctime = Mon May 20 13:42:26 2024
Error_Path = kvh.cm.cluster:/data/home/scurti15/scripts/pbs/
Hold_Types = n
Join_Path = n
Keep_Files = n
Mail_Points = a
mtime = Mon May 20 13:42:26 2024
Output_Path = kvh.cm.cluster:/data/home/scurti15/scripts/pbs/
Priority = 0
qtime = Mon May 20 13:42:26 2024
Rerunable = True
Resource_List.ncpus = 1
Resource_List.nodect = 1
Resource_List.place = pack
Resource_List.select = 1:ncpus=1
substate = 10
Variable_List = PBS_O_HOME=/data/home/scurti15,PBS_O_LANG=en_US.UTF-8,
PBS_O_LOGNAME=scurti15,
PBS_O_PATH=/data/home/scurti15/anaconda3/bin:/data/home/scurti15/anaco
nda3/condabin:/cm/local/apps/gcc/11.2.0/bin:/data/home/scurti15/.local/
bin:/data/home/scurti15/bin:/cm/local/apps/environment-modules/4.5.3/bi
n:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin:/cm
/local/apps/environment-modules/4.5.3/bin:/opt/pbs/bin:/data/home/scurt
i15/anaconda3/lib:/data/home/scurti15/anaconda3/bin:/data/home/scurti15
/tools:/data/home/scurti15/tools/samtools:/data/home/scurti15/tools/bed
tools:/data/home/scurti15/tools/fastqc:/data/home/scurti15/tools/gatk:/
data/home/scurti15/tools/samtools-1.17:/data/home/scurti15/tools/htslib
-1.17:/data/home/scurti15/tools/ucsc:/usr/include,
PBS_O_MAIL=/var/spool/mail/scurti15,PBS_O_SHELL=/bin/bash,
PBS_O_HOST=kvh.cm.cluster,
PBS_O_WORKDIR=/data/home/scurti15/scripts/fragmentomics,
PBS_O_SYSTEM=Linux,PBS_O_QUEUE=workq
comment = Not Running: No available resources on nodes
etime = Mon May 20 13:42:26 2024
Submit_arguments = gc_content_wgs.sh
array = True
array_state_count = Queued:90 Running:0 Exiting:0 Expired:0
array_indices_submitted = 1-90
array_indices_remaining = 1-90
project = _pbs_project_default
Submit_Host = kvh.cm.cluster
max_run_subjobs = 45
It seems like the server, scheduler, and comm are down:
/etc/init.d/pbs status
pbs_server is not running
pbs_sched is not running
pbs_comm is not running
The server logs show the following:
05/20/2024 16:18:10;0006;Server@kvh;Svr;Server@kvh;PBSProNodes feature not found, checking for PBSProSockets
05/20/2024 16:18:10;0006;Server@kvh;Svr;Server@kvh;NetlibToCCheckout error. Error Code: 9
Error String: Feature: PBSProSockets
Error Code: 9
Error Description:
[NETWORK] 6200@kvh.cm.cluster - (Err: 9) Feature not found
License Path: 6200@kvh.cm.cluster
: (null)
05/20/2024 16:18:11;0100;Server@kvh;Req;;Type 0 request received from scurti15@kvh.cm.cluster, sock=10
05/20/2024 16:18:11;0100;Server@kvh;Req;;Type 95 request received from scurti15@kvh.cm.cluster, sock=22
05/20/2024 16:18:11;0100;Server@kvh;Req;;Type 21 request received from scurti15@kvh.cm.cluster, sock=10
05/20/2024 16:18:11;0100;Server@kvh;Req;;Type 21 request processed from scurti15@kvh.cm.cluster, sock=10
05/20/2024 16:18:11;0100;Server@kvh;Req;;Type 19 request received from scurti15@kvh.cm.cluster, sock=10
05/20/2024 16:18:11;0100;Server@kvh;Req;;Type 19 request processed from scurti15@kvh.cm.cluster, sock=10
SELinux is disabled:
SELinux status: disabled
The server is resolvable:
pbs_hostn -v kvh
primary name: kvh.cm.cluster (from gethostbyname())
aliases: kvh
aliases: master.cm.cluster
aliases: master
aliases: localmaster.cm.cluster
aliases: localmaster
aliases: ldapserver.cm.cluster
aliases: ldapserver
address length: 4 bytes
address: 10.141.255.254 (4278160650 dec) name: kvh.cm.cluster
When I try to get the status of PBS using systemctl status pbs it prints nothing
I am able to ping the host
ping kvh
PING kvh.cm.cluster (10.141.255.254) 56(84) bytes of data.
64 bytes from kvh.cm.cluster (10.141.255.254): icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from kvh.cm.cluster (10.141.255.254): icmp_seq=2 ttl=64 time=0.047 ms
64 bytes from kvh.cm.cluster (10.141.255.254): icmp_seq=3 ttl=64 time=0.047 ms
64 bytes from kvh.cm.cluster (10.141.255.254): icmp_seq=4 ttl=64 time=0.021 ms
^C
— kvh.cm.cluster ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3060ms
rtt min/avg/max/mdev = 0.021/0.041/0.050/0.012 ms
nslookup kvh
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: kvh.cm.cluster
Address: 10.141.255.254
Could use any help to get the nodes up and running. Thank you!