# Detecting GPU cores separately

**URL:** <https://community.openpbs.org/t/detecting-gpu-cores-separately/4002>\
**Category:** Users/Site Administrators\
**Created:** [October 1, 2024, 1:01pm UTC](https://community.openpbs.org/t/detecting-gpu-cores-separately/4002 "2024-10-01T13:01:51Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![boboshaq](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@boboshaq](https://community.openpbs.org/u/boboshaq)\
**Post date:** [October 1, 2024, 1:01pm UTC](https://community.openpbs.org/t/detecting-gpu-cores-separately/4002/1 "2024-10-01T13:01:51Z")

</div>

Hi,  
I have configured cgroups with the device subsystem enabled for GPU detection on my new node with 3 GPU cards. PBS only detects 3 ngpus on that node. Is this correct? Is there any way I can set PBS to detect all GPU cores from a graphics card separately? For example, if one graphics card has 3000 GPUs, then with three cards, PBS detects 9000 ngpus available for use?

> **cgroup config file**
>
> ```auto
> {
> "cgroup_prefix" : "pbs_jobs",
> "exclude_hosts" : [],
> "exclude_vntypes" : ["no_cgroups"],
> "run_only_on_hosts" : [],
> "periodic_resc_update" : true,
> "vnode_per_numa_node" : "vntype in: gpu_node*",
> "online_offlined_nodes" : true,
> "use_hyperthreads" : true,
> "ncpus_are_cores" : false,
> "discover_gpus" : true,
> "manage_rlimit_as" : true,
> "cgroup" : {
> "cpuacct" : {
> "enabled" : true,
> "exclude_hosts" : [],
> "exclude_vntypes" : []
> },
> "cpuset" : {
> "enabled" : true,
> "exclude_cpus" : [],
> "exclude_hosts" : [],
> "exclude_vntypes" : [],
> "mem_fences" : false,
> "mem_hardwall" : false,
> "memory_spread_page" : false
> },
> "devices" : {
> "enabled" : "vntype in: gpu_node*",
> "exclude_hosts" : [],
> "exclude_vntypes" : [],
> "allow" : [
> "b *:* rwm",
> "c *:* rwm",
> ["nvidiactl", "rwm", "*"]
> ]
> },
> "memory" : {
> "enabled" : true,
> "exclude_hosts" : [],
> "exclude_vntypes" : [],
> "soft_limit" : false,
> "enforce_default" : true,
> "exclhost_ignore_default" : false,
> "default" : "256MB",
> "reserve_percent" : 0,
> "reserve_amount" : "1GB"
> },
> "memsw" : {
> "enabled" : false,
> "exclude_hosts" : [],
> "exclude_vntypes" : [],
> "enforce_default" : true,
> "exclhost_ignore_default" : false,
> "default" : "0B",
> "reserve_percent" : 0,
> "reserve_amount" : "0B"
> }
> }
> }
> 
> ```

> **pbsnodes -aSjv**
>
> ```auto
> mem ncpus nmics ngpus
> vnode state njobs run susp f/t f/t f/t f/t jobs
> --------------- --------------- ------ ----- ------ ----------- ------- ------- ------- -------
> node1 free 0 0 0 0 b/0 b 0/0 0/0 0/0 --
> node1[0] free 0 0 0 100gb/100gb 32/32 0/0 2/2 --
> node1[1] free 0 0 0 100gb/100gb 32/32 0/0 1/1 --
> 
> ```

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [October 2, 2024, 5:39pm UTC](https://community.openpbs.org/t/detecting-gpu-cores-separately/4002/2 "2024-10-02T17:39:06Z")

</div>

> [@boboshaq](#):
>
> PBS only detects 3 ngpus on that node. Is this correct?

Yes correct

> [@boboshaq](#):
>
> Is there any way I can set PBS to detect all GPU cores from a graphics card separately?

No, there isn’t a way.

> [@boboshaq](#):
>
> For example, if one graphics card has 3000 GPUs, then with three cards, PBS detects 9000 ngpus available for use?

If there is a application that can use gpu cores of a graphics card , then it might be possible to define a customer resourc gpucores [I have not found any such use cases or applications using gpucores (like cpu cores), so this is based on assumption] .

It would be interesting to know if there is any such solution.