Did anyone tried to run Docker containers using PBS

Users/Site Administrators
1

Did anyone run Docker containers using PBS Professional? I know there are couple of open source projects like Apache Mesos etc used to schedule docker containers on clusters.

2

We’re trying - no luck so far, though. Our main issue is passing through proper credentials so we can mount home drives… It does spin up the container, though.

3

I’m trying to set up the ability to run within containers on our NVIDIA box running Ubuntu. Using PBS 17.1.0 and following the guidance in “PBS Pro with Docker Integration” (in the PBS Resource Library), I modified the execjob_begin hook slightly to allow interactive jobs. My problem is getting the job inside the created container to run as the user (not root). Within the hook we run “docker exec -it -u <uid#> /bin/bash”, but it gives the error “unable to find user <uid#>: no matching entries in passwd file”, although we’ve mounted /etc/passwd inside the container. Interestingly, we can manually create a container and run that same command interactively on the box and it works.

If anyone else has seen this issue, I’d sure like to hear about any solutions.

4

Hi we have a pretty good integration of PBS + docker via our hooks mechanism. This is being made available in the commercial version of PBS Professional at the start of the next year.

This plugin takes care of multi-node, multi-container jobs, while plugging the problems of security that comes with using docker.

To answer your question, to run the job inside the container as the user, please start the container itself with the --user (or -u) option and specify the user-id. Of course the image you instantiate must be able to recognize the user inside it (in case the application needs it), otherwise it will appear only as a uid and not a username, and if the application executes a “get password entry” call then it could fail (like pbsdsh would).

5

In the upcoming implementation that you mention, doesn’t the user submitting the job need to be in the docker group (thereby giving them root privileges)?

6

Hi @pcebull,

No. In our upcoming implementation, the user will not need to be part of the docker group (or have root privileges). We use a PBS supplied setgid wrapper program to communicate to the docker daemon.

Thanks,
Subhasis