Qmgr Unauthorized Request

Users/Site Administrators
1

I have installed pbspro on a brand new cluster where Centos 7.4 is installed.

I built the rpms and installed the server one on the frontend and the execution one on a node.

Now I am trying to configure a first test queue, but any command I execute in qmgr ends with;

qmgr obj=<—something—> svr=default: Unauthorized Request
qmgr: Error (15007) returned from server

I am launching qmgr as root; but the root user should be a manager by default, correct ? Am I missing something ?

Thank you for any assistance,
Paolo

2

Paolo,

  1. are you running the command on the server itself ? or from the PBS clients commands from a remote host ?
  2. Make sure the /etc/hosts is populated correctly and represents the PBS_SERVER name mentioned in the /etc/pbs.conf
  3. Also, please make sure the PBS_SERVER name is resolvable (DNS)

Thank you

3

I am running the command on the server itself

I have two entries in /etc/host for the server, the “internal” and the “external” one;

192.168.33.1 werewolf.werewolf werewolf
90.147.33.20 werewolf.lambrate.inaf.it

I have tried setting PBS_SERVER to each of these values but the error still occurs

Yes, werewolf is resolvable.

Thank you,
Paolo

4

Please try below:

  1. Comment the second entry in the /etc/hosts
    90.147.33.20 werewolf.lambrate.inaf.it

  2. update the entry against PBS_SERVER in the /etc/pbs.conf to werewolf

  3. try the the qmgr commands that you were trying to run

Thank you,

5

Same error :disappointed_relieved:

6

For some reason , the PBS Server thinks, the commands originating from itself is coming from a different host. Hence, you are seeing this message qmgr: Error (15007) returned from server

7

If you open your server logs, you can see what the server thinks its hostname is:

08/27/2019 08:57:34.688203;0040;Server@shecil;Svr;shecil;Scheduler sent command 3

In this case, the server thinks it’s running on shecil (Server@shecil)
If you run a command that connects to the server, the server longs will print the hostname of the incoming request:

[vstumpf@shecil ~]$ qstat
[vstumpf@shecil ~]$ tail -n 4 /var/spool/pbs/server_logs/20190827 
08/27/2019 09:10:09.693158;0100;Server@shecil;Req;;Type 0 request received from vstumpf@shecil, sock=15

In this case, the request is coming from vstumpf@shecil. The server will do a string comparison of the two hosts.

8

If I issue a “qstat” command or I try any qmgr command nothing is written in the server log. Do I need to enable the logging somewhere ?

Moreover, digging into documentation I found this troubleshooting suggestion

Additionally, try running pbs_iff by typing 
pbs_iff server_host 15001
If pbs_iff fails to print the garbage and/or exits with a nonzero status, either the Server is not running or it was installed with a different encryption system from that used for pbs_iff.

The command exists with zero status, but nothing is printed; could this be the problem ?

9

Logging is enabled by default, so no configuration needs to be done.
qmgr -c "set server log_events = 511 "
0. please check all your PBS Pro OSS commands are the same version as the PBS Server
0. please check chmod 4755 $PBS_EXEC/sbin/pbs_iff

  1. Do you have any jobs running in your PBS setup ?
  2. try deleting a node and adding a node
  3. create a queue and delete a queue

[root@admin ~]# pbs_iff -t admin 15001 ( i do not get anything printed out after running this command )
[root@admin ~]# echo $?
0

Thank you

10

It seems that the

chmod 4755 $PBS_EXEC/sbin/pbs_iff

did the job!

Even if the setuid permission on this file seemed to me already setup before… anyway, now it works.

Thank you very much for the assistance,
Paolo

1 Like