MIG - open PBS support

Users/Site Administrators
1

Hi,

Does MIG - Multi instances GPU supported in open-pbs?
This link suggests it is supported without any change to configuration files etc, but I succeed to work on GPU and fail to work on MIG (I get an allocation of only GPU - not MIG although they are defined and works statically).

Thanks at advance

2

Based on the documentation, MIG support was added in V2021.1.1. You also need to have the cgroups hook enabled. If you do that, PBS should automatically recognize the instances. For instance, if you were on a node with (4) A100s and MIG mode was not on, the cgroups hook would set ngpus=4. If you then put all four A100s into MIG mode and configured them to each have 7 instances, then the cgroups hook would report ngpus=28. Note that PBS does not differentiate the size of the instance. If you configured one A100 into 7 instances, one into two instances, one into 3 instances, and one into only one instance, the cgroups hook would report that as ngpus=13 and you could randomly get an instance that is 1/7th of the A100 or the full A100.

I hope that helps.

Bill

1 Like
3

Hi Bill,

Thank you very much for your response.

  • We have ‘pbs_version = 20.0.1’ and I think it is the updated version you refer to

  • We work with CGroups hooks

  • We accept that MIG sizes are not distinguished in PBS

We have servers with regular GPU which works fine, but when we use MIG it keeps recognizing and allocates only the GPU (not their MIGs).

Do you sure no change in the configuration file should be made? Do you familiar with an actual system that works with MIG in PBS? could you kindly share its configuration file?

Thank you,
Eviatar

4

I think I understand now the version situation. The last version (20.0.1) is from 07/2020 before MIG support so I do not know any way to have the MIG support included.

Do you know if there exists a beta version for tests until the next release?

Thank you

5

I don’t understand why you need a beta? The support is available now in the most recent version(2021.1.2). If you install the most recent version MIG support should be there.

The one thing I am not certain about is whether it is available in OpenPBS or just PBSPro. We contribute to OpenPBS, but use PBSPro.

Bill

6

Yes, we are using openPBS (last version dated 2020).
Thank you Bill for helping to clarify the situation.