How to submit PBS jobs from a Singularity container?

Users/Site Administrators
1

I would like to submit a PBS job from inside a Singularity container and have it communicate with the PBS server outside the container and run on nodes that are also outside the container. I’m using PBS Pro 2020. Does anyone know how to do this?

Here’s what I’ve done so far. I tried binding various system directories from outside the container to equivalent mount points inside the container (/var, /etc, etc). I also duplicated inside the container whatever looked like important entries in the /etc/group and /etc/passwd file. I When I try to run ‘qstat’ inside the container, I get this error:

> qstat
pbs_iff: cannot connect to host
pbs_iff: cannot connect to host
auth: error returned: -1
auth: Unable to authenticate connection (pbsserver:15001)
qstat: cannot connect to server pbsserver (errno=-1)

Does anyone know how I can solve this, or at least make progress?

2

Just a wild guess, since I also had problems with pbs_iff, which handles authentication with the server (see man pbs_iff): Check that Singularity does not mangle/remap source TCP ports.
OpenPBS relies on port numbers to verify “valid” or “trustworthy” connections. If a middle-layer remaps those ports, the authentication will not succeed. An alternative seems to be to use MUNGE, but I have not tried this yet.

3

We were already using MUNGE authentication within our cluster before we started experimenting with singularity. I can say that job submissions (along with other PBS commands) from inside the container does work, but the trick was to make sure the container bind-mounts the socket file (In my case /var/run/munge/munge.socket.2) that munged creates so that the processes inside the container are communicating with the munge daemon running outside the container.

4

Thank you for your help. My system has no directory named /var/run/munge. I assume that means my installation of PBS Pro is not configured to use MUNGE. I’m not the system admin for this machine and I won’t be able to change to MUNGE if that’s required. Will I be able to submit PBS jobs from inside the container without MUNGE?

Also, for experimentation, is it possible to build a local version of OpenPBS in my home directory and have it submit jobs to a server running PBS Pro? If that is true, I can conduct a lot more experiments. But if there’s no chance OpenPBS clients (qsub and qstat) can ever interact with a server running PBS Pro, then I won’t start down this path. Does anyone know?

5

I have done this many times. So long as your private versions of the cient commands use the same pbs_iff as the system version, there should be no problem.