# Any updates on GPU support since 2010?

**URL:** <https://community.openpbs.org/t/any-updates-on-gpu-support-since-2010/162>\
**Category:** Users/Site Administrators\
**Created:** [July 15, 2016, 12:49am UTC](https://community.openpbs.org/t/any-updates-on-gpu-support-since-2010/162 "2016-07-15T00:49:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![hi-watanabe](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hi-watanabe](https://community.openpbs.org/u/hi-watanabe)\
**Post date:** [July 15, 2016, 12:49am UTC](https://community.openpbs.org/t/any-updates-on-gpu-support-since-2010/162/1 "2016-07-15T00:49:14Z")

</div>

Hi,

I’m testing PBS Pro Open Source version on CentOS 6. It works fine 🙂

According to a Altair’s pdf “Scheduling Jobs onto NVIDIA Tesla GPU Computing Processors using PBS Professional” which was written in October 2010, it says “Note also that PBS Professional allocates GPUs, but doesn’t bind jobs to any particular GPU; the application itself (or the CUDA library) is responsible for the actual binding.”

Is this behavior (about actual binding jobs to a particular GPU) same on the latest PBS Pro Open Source version? In case of multiple GPUs in a PC, how can we get exclusive use of GPU among multiple jobs ? Is there way to automatically set CUDA\_VISIBLE\_DEVICES by PBS ?

Best regards,  
WATANABE

---

<div class="post-metadata">

**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\
**Post date:** [July 15, 2016, 1:32am UTC](https://community.openpbs.org/t/any-updates-on-gpu-support-since-2010/162/2 "2016-07-15T01:32:15Z")

</div>

Watanabe,

Thanks for trying out PBS Professional. In the 13.0 commercial release we provided the ability to set the CUDA\_VISIBLE\_DEVICES variable in our limited availability cgroups hook and pbs\_attach command. I am currently working on contributing the cgroups hook to the open source community. If all goes well it will be available in the master branch by the end of the month. This should provide you with the ability to set CUDA\_VISIBLE\_DEVICES on single node jobs. However, due to how it was implemented we decided not to release the updated pbs\_attach command to the community until we can rework it to conform to our architectural standards. We hope to address the issues with pbs\_attach in our next major release of PBS Professional in Q1 of next year. The ticket for this work is PP-278 if you are interested in watching or working on the ticket.

I hope this helps.

Jon

---

<div class="post-metadata">

**Author:** ![alexis.cousein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/alexis.cousein/32/265_2.png) [@alexis.cousein](https://community.openpbs.org/u/alexis.cousein)\
**Post date:** [July 15, 2016, 8:23am UTC](https://community.openpbs.org/t/any-updates-on-gpu-support-since-2010/162/3 "2016-07-15T08:23:08Z")

</div>

Note that even without a pbs\_attach rework you can still use CUDA integration: either if you don’t use multihost jobs or if you use pbs\_tmrsh rather than pbs\_attach. It’s only when you spawn remote processes directly using ssh that you need pbs\_attach (but some libraries e.g. Intel MPI let you use your own remote process launcher that follows rsh semantics, and then you can use pbs\_tmrsh and let MoM spawn remote processes directly).

If course this doesn’t change the fact you need more than _just_ Open Source PBSPro; you still need some type of hook to manage which resources are bound to which job.

OTOH, it’s possible to use something simpler than the cgroup hook (if you roll your own); you can just create extra vnodes in v2 configuration files that own specific GPUs and then parse the job’s exec\_vnode to find out what devices to use.

But if you have more than one socket and GPUs on both sockets, you’ll want processes contained on particular cores and then if you want it all to “automagically” work you’ll indeed need something just as complex as the cgroup hooks, unless you are prepared to also use numactl in /etc/profile.d scripts or your job scripts to figure out what cpus and memory to use from the GPUs assigned to the job (which is not _that_ hard in practice; it’s a lot easier to write a set of hooks that works for your particular use case than to write the _general_ cgroup hook with NUMA and devices support.)

---

<div class="post-metadata">

**Author:** ![hi-watanabe](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hi-watanabe](https://community.openpbs.org/u/hi-watanabe)\
**Post date:** [July 17, 2016, 12:58am UTC](https://community.openpbs.org/t/any-updates-on-gpu-support-since-2010/162/4 "2016-07-17T00:58:03Z")

</div>

Hi Jon,

Thank you for sharing information.  
I look forward to the new version.

Thanks,  
WATANABE

---

<div class="post-metadata">

**Author:** ![hi-watanabe](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hi-watanabe](https://community.openpbs.org/u/hi-watanabe)\
**Post date:** [July 17, 2016, 1:03am UTC](https://community.openpbs.org/t/any-updates-on-gpu-support-since-2010/162/5 "2016-07-17T01:03:06Z")

</div>

Hi alxis.cousein,

Thank you very much for your detailed notes and practical advise.  
I’ll check the cgroup hook and other techniques you wrote.

Thanks,  
WATANABE