I solved the problem before and successfully connected to server, which is also a client.
I typed
qsub data.sh
and it returned
No Permission.
qsub: cannot connect to server songyi719-ThinkPad-X1-Extreme-2nd (errno=15007)
Firewall and port is opened, and I also tried running on sudo, but error message remains same. What can be the problem?
Please share the contents of
/etc/hosts
/etc/pbs.conf
hostname and hostname -A` commands output
ping output
ps -ef | grep pbs_ output
content of /etc/hosts is
127.0.0.1 localhost
110.76.77.137 songyi719-ThinkPad-X1-Extreme-2nd
#127.0.1.1 songyi719-ThinkPad-X1-Extreme-2nd
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
contents of /etc/pbs.conf is
PBS_SERVER=songyi719-ThinkPad-X1-Extreme-2nd
PBS_START_SERVER=1
PBS_START_SCHED=1
PBS_START_COMM=1
PBS_START_MOM=1
PBS_EXEC=/opt/pbs
PBS_HOME=/var/spool/pbs
PBS_CORE_LIMIT=unlimited
PBS_SCP=/usr/bin/scp
hostname is âsongyi719-ThinkPad-X1-Extreme-2ndâ
hostname -A is also âsongyi719-ThinkPad-X1-Extreme-2ndâ
ping output is
PING songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137) 56(84) bytes of data.
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=3 ttl=64 time=0.043 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=4 ttl=64 time=0.053 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=5 ttl=64 time=0.049 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=6 ttl=64 time=0.033 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=7 ttl=64 time=0.042 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=8 ttl=64 time=0.046 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=9 ttl=64 time=0.047 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=10 ttl=64 time=0.068 ms
64 bytes from songyi719-ThinkPad-X1-Extreme-2nd (110.76.77.137): icmp_seq=11 ttl=64 time=0.057 ms
^C
â songyi719-ThinkPad-X1-Extreme-2nd ping statistics â
11 packets transmitted, 11 received, 0% packet loss, time 10229ms
rtt min/avg/max/mdev = 0.033/0.048/0.068/0.008 ms
ps -ef | grep pbs_output is
songyi7+ 67540 67391 0 18:26 pts/0 00:00:00 grep --color=auto pbs_output
127.0.0.1 localhost
110.76.77.137 songyi719-ThinkPad-X1-Extreme-2nd
#127.0.1.1 songyi719-ThinkPad-X1-Extreme-2nd
[/quote]
/etc/init.d/pbs stop
Could you please update the line to this
110.76.77.137 songyi719-ThinkPad-X1-Extreme-2nd pbs
PBS_SERVER=pbs
/etc/init.d/pbs start
/etc/init.d/pbs status #check and share the output
Check all the services are up and running and make sure ports 15001 to 15009 and 17001 are open for communication and not blocked.
I am sorry the command should have been as below
ps -ef | grep pbs_
Look in the server log to see what it reports. Set log_events to 2047 to make sure you see all messages first. Do you see Type XX requests being received by the server?
I assume qstat -Bf does work? If not, itâs fruitless to try qsub.
Check whether pbs_iff has a setuid bit set. If not, then âresvportâ authentication (which is the default) will fail. Also: root by default is not allowed to run jobs (thatâs a safety feature).
âFirewall and port is openedâ
If itâs a NAT firewall, that is not enough. resvport authentication is incompatible with a NAT firewall.
Itâs unclear where youâre trying qsub. Iâd try to see if it works on the server itself first (where hopefully local traffic to PBS_SERVER does not go through a firewall).
As Adarsh rightly said, itâs customary to add a mnemonic alias to /etc/hosts and use that as PBS_SERVER in case your hostnames are that long. You usually donât want your fully specified job IDs to be 123.songyi719-ThinkPad-X1-Extreme-2nd.
after #5 pbs status, it returned
pbs_server is pid 2405
pbs_mom is pid 1688
pbs_sched is pid 1703
pbs_comm is 1668
I also tried ps -ef | grep pbs_ too
root 1668 1 0 12:22 ? 00:00:00 /opt/pbs/sbin/pbs_comm
root 1688 1 0 12:22 ? 00:00:00 /opt/pbs/sbin/pbs_mom
root 1703 1 0 12:22 ? 00:00:00 /opt/pbs/sbin/pbs_sched
root 1914 1 0 12:22 ? 00:00:04 /opt/pbs/sbin/pbs_ds_monitor monitor
postgres 2355 1950 0 12:22 ? 00:00:00 postgres: postgres pbs_datastore 110.76.77.137(46022) idle
root 2405 1 0 12:22 ? 00:00:00 /opt/pbs/sbin/pbs_server.bin
songyi7+ 18382 17808 0 20:16 pts/0 00:00:00 grep --color=auto pbs_
After #5, it seems like pbs started to run, but it still denies my order
Please refer to Alexis suggestions above,
qmgr : set server log_events = 2047
Run the commands that are failing with 15007 and check the $PBS_HOME/server_logs/20210127 for the relevant log bits that are populated when you run the commands (probably you might do a tail of the logs and in parallel on another terminal window run the commands) . Share us those logs when you encounter error for 15007
is your qsub , qstat commands run from a terminal on or connected to songyi719-ThinkPad-X1-Extreme-2nd
#1 I found /pbs/server_logs, and one of the recent lines are written as
01/27/2021 21:22:43;0001;Server@songyi719-thinkpad-x1-extreme-2nd;Svr;Server@songyi719-thinkpad-x1-extreme-2nd;is_request, bad attempt to connect from 110.76.77.137:15003
01/27/2021 21:22:47;0001;Server@songyi719-thinkpad-x1-extreme-2nd;Svr;Server@songyi719-thinkpad-x1-extreme-2nd;is_request, bad attempt to connect from 110.76.77.137:15003
01/27/2021 20:14:47;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;?;Req Header bad, errno 0, dis error 7
01/27/2021 20:14:47;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;req_reject;Reject reply code=15056, aux=0, type=0, from @songyi719-thinkpad-x1-extreme-2nd
01/27/2021 20:14:50;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;?;Req Header bad, errno 0, dis error 7
01/27/2021 20:14:50;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;req_reject;Reject reply code=15056, aux=0, type=0, from @songyi719-thinkpad-x1-extreme-2nd
01/27/2021 20:15:14;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;?;Req Header bad, errno 0, dis error 7
01/27/2021 20:15:14;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;req_reject;Reject reply code=15056, aux=0, type=0, from @songyi719-thinkpad-x1-extreme-2nd
#2 Yes qstat and every pbs related orders arenât working
#3 pbs_iff is setted as -rwsr-xr-x. I believe this means pbs_iff has setuid bit set.
#4 Iâm not a network expert, but after bit of googling, I think my internet is presumably not using NAT firewall
#5 Well this PC is both server and client at same time now, And that didnât worked pretty much well if I understood correct. Maybe I should try again
#6 Maybe I should have named my pc shorter when I first installed Ubuntu. I was totally noob about linux back then. Well in fact not very different even today
#1 It doesnât seem to work well⌠It returned
Unknown Host.
qmgr: cannot connect to server :
Unknown Host.
qmgr: cannot connect to server set
Unknown Host.
qmgr: cannot connect to server server
Unknown Host.
qmgr: cannot connect to server log_event
Unknown Host.
qmgr: cannot connect to server =
#2 Also, I restarted pbs and status changed a bit
pbs_server is pid 20051
pbs_mom is pid 19948
pbs_sched is pid 19959
pbs_comm is 19938
01/27/2021 21:22:43;0001;Server@songyi719-thinkpad-x1-extreme-2nd;Svr;Server@songyi719-thinkpad-x1-extreme-2nd;is_request, bad attempt to connect from 110.76.77.137:15003
01/27/2021 21:22:47;0001;Server@songyi719-thinkpad-x1-extreme-2nd;Svr;Server@songyi719-thinkpad-x1-extreme-2nd;is_request, bad attempt to connect from 110.76.77.137:15003
01/27/2021 20:14:47;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;?;Req Header bad, errno 0, dis error 7
This suggests that the client is not (new) OpenPBS. Thatâs what youâd expect if e.g. a Torque client was trying to talk to an OpenPBS server.
As for your qmgr comand, type in qmgr then type in the command. If you want it on one line, use
qmgr -c âset server log_events=2047â (not forgetting the quotes).
Mein Gott⌠maybe Iâve installed Torque client long ago
I tried the command, but it just says
No Permission.
qmgr: cannot connect to server
Should I just delete Torque client?
Yes, it is good to have one active WLM on the system.
You can remove it using apt-get remove
Also, you can source /etc/profile.d/pbs.sh and use qmgr/qsub/qstat commands.
Always check by check âwhich qmgrâ , so it is pointing to the correct PBS Pro binary
or else you can source /etc/pbs.conf;$PBS_EXEC/bin/qstat , source /etc/pbs.conf;$PBS_EXEC/bin/qmgr , source /etc/pbs.conf;$PBS_EXEC/bin/qsub
Iâve installed torque via make install, so it was impossible to use apt-get command. So I just removed core file and directory that is assumed to be part of torque/slurm
Now I get
No default server name.
qsub: cannot connect to server (null) (errno=15034)
Maybe I didnât delete it enough. Server is running fine, so Iâm sure that I at least didnât delete wrong file. Regretting why I installed the torqueâŚ
Update : I used checkinstall and probably removed torque properly.
Now it shows
qsub: No default queue specified
to my order âqsub data.shâ
Is this positive signal?
Yes.
That simply means you have to define a queue and set the default queue to it.
Normally the default (which you can see using qmgr -c âprint serverâ) includes:
create queue workq
set queue workq queue_type = Execution
set queue workq resources_default.mem = 123mb
set queue workq resources_default.ncpus = 1
set queue workq resources_default.node_type = A
set queue workq enabled = True
set queue workq started = True
[âŚ]
set server default_queue = workq
I donât know how youâve installed it; normally if the PBS_HOME doesnât exist, the service startup script calls pbs_habitat (from the libexec directory) and that gives you a default queue config. Chances are that the presence of Torque threw a spanner in the works there and made it abort before it could define a default queue and set it.
Yes PBS_HOME doesnât exist and pbs_habitat exists.
qmgr -c âprint serverâ returned
#
# Set server attributes.
#
set server scheduling = True
set server log_events = 511
set server mailer = /usr/sbin/sendmail
set server mail_from = adm
set server query_other_jobs = True
set server resources_default.ncpus = 1
set server default_chunk.ncpus = 1
set server scheduler_iteration = 600
set server resv_enable = True
set server node_fail_requeue = 310
set server max_array_size = 10000
set server pbs_license_min = 0
set server pbs_license_max = 2147483647
set server pbs_license_linger_time = 31536000
set server eligible_time_enable = False
set server max_concurrent_provision = 5
set server max_job_sequence_id = 9999999
When I searched internet, it says that this command should show default queue, but it didnât. However, when I tried to set queue using
qmgr -c âset server default_queue=testqâ
it returned
qmgr obj= svr=default: Unauthorized Request
qmgr: Error (15007) returned from server
I guess initial problem remains stillâŚ
Update : When I tried to set default queue, it gave me error 15007, so I checked server log, and it is written
01/28/2021 14:34:37;0100;Server@songyi719-thinkpad-x1-extreme-2nd;Req;;Type 0 request received from songyi719@songyi719-thinkpad-x1-extreme-2nd, sock=18
01/28/2021 14:34:37;0100;Server@songyi719-thinkpad-x1-extreme-2nd;Req;;Type 95 request received from songyi719@songyi719-thinkpad-x1-extreme-2nd, sock=19
01/28/2021 14:34:43;0100;Server@songyi719-thinkpad-x1-extreme-2nd;Req;;Type 82 request received from songyi719@songyi719-thinkpad-x1-extreme-2nd, sock=18
01/28/2021 14:34:43;0100;Server@songyi719-thinkpad-x1-extreme-2nd;Req;;Type 20 request received from songyi719@songyi719-thinkpad-x1-extreme-2nd, sock=18
01/28/2021 14:34:43;0100;Server@songyi719-thinkpad-x1-extreme-2nd;Req;;Type 21 request received from songyi719@songyi719-thinkpad-x1-extreme-2nd, sock=18
01/28/2021 14:45:46;0100;Server@songyi719-thinkpad-x1-extreme-2nd;Req;;Type 9 request received from songyi719@songyi719-thinkpad-x1-extreme-2nd, sock=18
01/28/2021 14:45:46;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;req_reject;Reject reply code=15007, aux=0, type=9, from songyi719@songyi719-thinkpad-x1-extreme-2nd
01/28/2021 14:46:10;0100;Server@songyi719-thinkpad-x1-extreme-2nd;Req;;Type 9 request received from songyi719@songyi719-thinkpad-x1-extreme-2nd, sock=18
01/28/2021 14:46:10;0080;Server@songyi719-thinkpad-x1-extreme-2nd;Req;req_reject;Reject reply code=15007, aux=0, type=9, from songyi719@songyi719-thinkpad-x1-extreme-2nd
I donât know well about what this means, but I could at least figure out that error message has changed from yesterdayâŚ
Perhaps reading the documentation might prove useful. By default, only root is a PBS manager and is allowed to manage the serverâŚ
[BTW: root by default cannot run jobs.]
Apart from that: your client is now correctly talking to the server, as the logs indicate.
Thanks for big effort on helping me. I will search on documentation, and if It doesnât work, I would re-install pbs using .deb rather than âmake installââŚ