I have a legacy software that use qsub/qstat/qdel (it submit jobs programmatically). I don’t want to pollute the system installing a lots of legacy dependencies.
My idea is to create a container (in my case i use singularity). In the container i have installed al dependencies excluding pbs. Than i have bindend pbs installation dir from host to container.
I have the following error:
$ singularity shell --bind /opt/pbs --bind /etc/pbs.conf /work/XXXX/legacysofware.sif
Singularity> /opt/pbs/bin/qstat
pbs_iff: cannot connect to host
pbs_iff: cannot connect to host
auth: error returned: -1
auth: Unable to authenticate connection (XXXXX:15001)
Invalid argument
qstat: cannot connect to server XXXXX (errno=-1)
I think that the problem is related with the setuid bit of pbs_iff. For security reason is disabled on singularity.
Do you have any suggestion ? There is a clean way to solve this problem?
Why not just install pbspro-client in the container and treat that container as a submission node? You then do not need to bind /opt/pbs or anything else.
auth: Unable to authenticate connection (XXXXX:15001)
The port is blocked for some reason. 15001 is the port that the PBS server on the head node listens on.
Check your firewall is not blocking the singularity node on the server.
Check you do not have a firewall running in the singularity.
Check root on the singularity is the same as root on the server. Maybe try starting the singularity as root to see if you get the same problem.
Check your running the same version of PBS in the container as on the head node.
Can you ping or ssh to the head node from the singularity and vice versa.
Mike
Same problem here, in the end I created a wrapper for qsub that executes a qsub with ssh on a node that is not running singularity. Not the cleanest way to do this. I would like to have a pbs client install without the need for suid.
You need to use munge or enable the container to have an IP address on the host interface.
For Munge see:
If you don’t use munge, then the authentication method is “resvport”. That cannot work with a NAT in between. “resvport” means that pbs_iff will get a reserved port, connect with it to the server, and then vouch for the earlier established client connection by telling it what IP address and port are OK to listen to, but if what the server sees are different ports on the host side of a NAT it won’t work.
With Munge you don’t need a second TCP connection from pbs_iff that will use the port if the client in its message – authentication is in-band.
If you container has an IP alias on a host interface then you also won’t have a NAT and then resvport will also work.