# The memory used by multiple nodes is not displayed

**URL:** <https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169>\
**Category:** Users/Site Administrators\
**Created:** [August 1, 2025, 6:47am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169 "2025-08-01T06:47:27Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 1, 2025, 6:47am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/1 "2025-08-01T06:47:27Z")

</div>

After a job is executed on multiple nodes, the memory used by the job is not displayed when viewing it using `qstat -xf`.

qstat -xf 518:  
Job Id: 518.e004-2  
Job\_Name = 1066  
Job\_0wner = job\_user@e004-2  
resources\_used.cpupercent = 0  
resources\_used.cput = 00:00:00  
resources\_used.mem = 0kb  
resources\_used.ncpus = 36  
resources\_used.vmem = 0kb  
resources\_used.walltime = 00:00:00  
job\_state = F  
queue = q200  
server = e004-2  
exec\_host = e005/0\*\*18+e006/0\*\*18  
exec\_vnode = (e005:ncpus=18)+(e006:ncpus=18)  
……

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [August 1, 2025, 7:00am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/2 "2025-08-01T07:00:44Z")

</div>

You would need tight integration. if in case you are using MPI Flavours (Intel MPI, OpenMPI).  
Otherwise, you would have to use cgroups for proper accounting of resources.

> [@PSS instead of RSS as mem](https://community.openpbs.org/t/pss-instead-of-rss-as-mem/2735):
>
> Hi, Would it be possible to report (and enforce) PSS (proportional set size) instead of RSS? Yes, that’s specific to Linux. Evgeny

---

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 1, 2025, 8:52am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/3 "2025-08-01T08:52:17Z")

</div>

The problem is, I have used cgroups。

[root@e004-2 ~]# qmgr -c ‘l h’  
Hook pbs\_cgroups  
type = site  
enabled = true  
event = execjob\_begin,execjob\_epilogue,execjob\_end,execjob\_launch,execjob\_attach,execjob\_resize,execjob\_abort,execjob\_postsuspend,execjob\_preresume,exechost\_periodic, exechost\_startup  
user = pbsadmin  
alarm = 90  
freq = 120  
order = 100  
debug = false  
fail\_action = offline\_vnodes

**pbs\_cgroups:**  
{  
“cgroup\_prefix”: “pbs\_jobs”,  
“exclude\_hosts”: ,  
“exclude\_vntypes”: [“no\_cgroups”],  
“run\_only\_on\_hosts”: ,  
“periodic\_resc\_update”: true,  
“vnode\_per\_numa\_node”: false,  
“online\_offlined\_nodes”: true,  
“use\_hyperthreads”: false,  
“ncpus\_are\_cores”: false,  
“discover\_gpus”: true,  
“manage\_rlimit\_as”: true,  
“cgroup” : {  
“cpuacct”:{  
“enabled”: true,  
“exclude\_hosts”: ,  
“exclude\_vntypes”:   
},  
”cpuset" : {  
“enabled”: true,  
“exclude\_cpus”: ,  
“exclude\_hosts”: ,  
“exclude\_vntypes”: ,  
”allow\_zero\_cpus": true,  
“mem\_fences”: false,  
“mem\_hardwall”: false,  
“memory\_spread\_page”: false  
},  
”devices": {  
“enabled”: true,  
“exclude\_hosts”: ,  
“exclude\_vntypes”: ,  
“allow”: [  
“b \*:\* rwm”,  
“b 7:\* rwm”,  
“c \*:\* rwm”,  
“c 195:\* m”,  
“c 136:\* rwm”,  
[“infiniband/rdma\_cm”,“rwm”],  
[“fuse”,“rwm”],  
[“net/tun”,“rwm”],  
[“tty”,“rwm”],  
[“ptmx”,“rwm”],  
[“console”,“rwm”],  
[“null”,“rwm”],  
[“zero”,“rwm”].  
[“full”,“rwm”],  
[“random”,“rwm”].  
[“urandom”,“rwm”],  
[“cpu/0/cpuid”,“rwm”,“\*”],  
[“nvidia-modeset”, “rwm”],  
[“nvidia-uvm”,“rwm”],  
[“nvidia-uvm-tools”“rwm”],  
[“nvidiactl”, “rwm”]  
]  
},  
“memory”: {  
“enabled”: false,  
“exclude\_hosts”: :  
“exclude\_vntypes”: ,  
“soft\_limit”: false,  
“enforce\_default”: false,  
“exclhost\_ignore\_default”: false,  
“default”: “256MB”,  
“reserve\_percent”: 0,  
“reserve\_amount”: “0GB”  
},  
“memsw” : {  
“enabled”: false,  
“exclude\_hosts”: ',  
“exclude\_vntypes”: ,  
“enforce\_default”: true,  
“exclhost\_ignore\_default” : false,  
“default”: “0B”,  
“reserve\_percent”: 0,  
“reserve\_amount”: “64MB”,  
“manage\_cgswap”: false  
},  
“hugetlb” : {  
“enabled”: false,  
“exclude\_hosts”: ,  
“exclude\_vntypes”: ,  
“enforce\_default”: true,  
“exclhost\_ignore\_default”: false,  
“default”: “0B”,  
“reserve\_percent”: 0,  
“reserve\_amount”: “0B”  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [August 1, 2025, 8:57am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/4 "2025-08-01T08:57:03Z")

</div>

Please check whether your memory subsystem is enabled in cgroups.  
Also, please share the characteristics of your job and script used to run the job.

---

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 1, 2025, 9:11am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/5 "2025-08-01T09:11:20Z")

</div>

The memory subsystem, `“enabled”: false`:  
“memory”: {  
“enabled”: false,  
“exclude\_hosts”: :  
“exclude\_vntypes”: ,  
“soft\_limit”: false,  
“enforce\_default”: false,  
“exclhost\_ignore\_default”: false,  
“default”: “256MB”,  
“reserve\_percent”: 0,  
“reserve\_amount”: “0GB”  
}  
If I `enabled:true`, when I submit a job without explicitly setting memory usage, I should receive the default memory limit. However, I want to use the full memory of the node. How should I handle this?

**\*.pbs:**  
#!/bin/bash  
#PBS -N 1069  
#PBS -k oed  
#PBS -l select=1:ncpus=18:mpiprocs=18+1:ncpus=18:mpiprocs=18  
#PBS -q q200  
#PBS -o ./front\_m.out  
#PBS -e ./front\_m.err  
cd $PBS\_O\_WORKDIR  
/opt/software/abaqus/Commands/abq6144 job=front\_m cpus=36 int

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [August 1, 2025, 11:13am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/6 "2025-08-01T11:13:21Z")

</div>

> [@wakaka](#):
>
> I want to use the full memory of the node. How should I handle this?

Please update this line

#PBS -l select=1:ncpus=18:mpiprocs=18+1:ncpus=18:mpiprocs=1

to

#PBS -l select=1:ncpus=18\*\*:mem=100gb\*\*:mpiprocs=18+1:ncpus=18: **mem=100gb** :mpiprocs=18

or with the below two lines

#PBS -l select=2:ncpus=18: **mem=100gb** :mpiprocs=18  
#PBS -l place=scatter

---

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 4, 2025, 1:17am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/7 "2025-08-04T01:17:35Z")

</div>

However, I don’t have a node with more than 100GB. If I submit the job, there will be no available resources and it will be always in a queued state. And if I need to enable the memory subsystem to true?

Now I find a new problem. When I enable the cgroups, I select a number of cpus to a job, but it just use one cpu in execution node to run this job. Is there something wrong in my settings?

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [August 4, 2025, 9:13am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/8 "2025-08-04T09:13:51Z")

</div>

> [@wakaka](#):
>
> However, I don’t have a node with more than 100GB. If I submit the job, there will be no available resources and it will be always in a queued state. And if I need to enable the memory subsystem to true?

Sorry, that was an example of memory request. Please choose the memory suitable for your compute node configuration. Yes, please enable the memory subsystem to true.

> [@wakaka](#):
>
> Now I find a new problem. When I enable the cgroups, I select a number of cpus to a job, but it just use one cpu in execution node to run this job. Is there something wrong in my settings?

the cgroups should give the exact resource that is requested in your qsub statement. I do not see any issue with your configuration.

---

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 7, 2025, 3:29am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/9 "2025-08-07T03:29:49Z")

</div>

Unfortunately, the fact is that my program starts many processes, but they are all limited to running on one CPU. They are all competing for this one CPU. Can I let the process monopolize one CPU?

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/2006124622f6020a79cf8c1cee562438b2a233bc.jpeg)

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [August 11, 2025, 10:05am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/10 "2025-08-11T10:05:33Z")

</div>

Please check whether this is application related. This might not have anything to do with the scheduler. Try to run the application batch command line with the pre-defined hosts list and check whether the cpu cores are evenly used or context switching happens and they always use the same cores.

---

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 12, 2025, 2:23am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/11 "2025-08-12T02:23:14Z")

</div>

I found that this should be a problem with the software version. I changed the version and executed the job without this problem. In addition, even if I turned on the memory subsystem, after multiple nodes completed a job, I still could not see the cpu and memory usage statistics using `qstat -xf job_id`

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/23280e57fdb40a46b05d20e93e6d36e82c6e40d2.jpeg)

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/ab3e4f6543144c940a9e71cd5d78a8263782af0f.png)

---

<div class="post-metadata">

**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\
**Post date:** [August 12, 2025, 8:23pm UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/12 "2025-08-12T20:23:10Z")

</div>

IIRC, remote resource usage is collected only every two minutes or so. Try running a longer job and see if you get reasonable memory use numbers.

---

<div class="post-metadata">

**Author:** ![wakaka](https://avatars.discourse-cdn.com/v4/letter/w/d07c76/32.png) [@wakaka](https://community.openpbs.org/u/wakaka)\
**Post date:** [August 13, 2025, 1:05am UTC](https://community.openpbs.org/t/the-memory-used-by-multiple-nodes-is-not-displayed/4169/13 "2025-08-13T01:05:09Z")

</div>

Well, the job finished after running for more than 15 hours. This problem does not occur when I run the job on a single node.