# Memory usage across nodes

**URL:** <https://community.openpbs.org/t/memory-usage-across-nodes/2597>\
**Category:** Users/Site Administrators\
**Created:** [April 25, 2021, 10:12am UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597 "2021-04-25T10:12:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Manan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/manan/32/409_2.png) [@Manan](https://community.openpbs.org/u/Manan)\
**Post date:** [April 25, 2021, 10:12am UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597/1 "2021-04-25T10:12:44Z")

</div>

I am a new user to PBS

I have a 12 node cluster with 376 GB of memory on each node but I want to run a parallel job that requires 1TB of memory. So is there a way that PBS can give a argument such that I can Use the memory of other nodes parallely and run my job of 1TB.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 25, 2021, 7:18pm UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597/2 "2021-04-25T19:18:31Z")

</div>

Please try this  
`qsub -l select=3:ncpus=16:mem=350gb:mpiprocs=16 -l place=scatter -- /bin/sleep 1000`

---

<div class="post-metadata">

**Author:** ![Manan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/manan/32/409_2.png) [@Manan](https://community.openpbs.org/u/Manan)\
**Post date:** [April 26, 2021, 7:44am UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597/3 "2021-04-26T07:44:38Z")

</div>

Thanks, Adarsh for such a quick reply.

I have tried qsub -l select=3:ncpus=16:mem=350gb:mpiprocs=16 -l place=scatter – /bin/sleep 1000  
the job goes in queue state and then gets terminated stating insufficient resources.  
I want to run a job that requires 1TB but our each node has 376 GB of memory so I was want a way such that such I can run a parallel job using the memory from different nodes.

As we have 12 nodes with 376 GB of memory each so could you please give us a solution to it

Thanks in advance

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 27, 2021, 7:37am UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597/4 "2021-04-27T07:37:53Z")

</div>

> [@Manan](#):
>
> I have tried qsub -l select=3:ncpus=16:mem=350gb:mpiprocs=16 -l place=scatter – /bin/sleep 1000  
> the job goes in queue state and then gets terminated stating insufficient resources.

Please note : if the requested resources to run the job are not available, then the job would be in the queued state until the resources are available, they would not be deleted. if the job gets deleted/terminated, then there must be some other issue.

> [@Manan](#):
>
> I want to run a job that requires 1TB but our each node has 376 GB of memory so I was want a way such that such I can run a parallel job using the memory from different nodes.

The parallel job would use 350GB from each of the compute nodes (3 in this case) and the total memory requested for this job would be 1.x TB.

If your job has terminated, then please share us the output of

1. source /etc/pbs.conf ; $PBS\_EXEC/unsupported/pbs\_dtj \< job id \>  
eg. source /etc/pbs.conf ; $PBS\_EXEC/unsupported/pbs\_dtj 111

---

<div class="post-metadata">

**Author:** ![Manan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/manan/32/409_2.png) [@Manan](https://community.openpbs.org/u/Manan)\
**Post date:** [April 27, 2021, 1:06pm UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597/5 "2021-04-27T13:06:38Z")

</div>

thanks, Adarsh for your tremendous support.

I was requesting resources more than available, issue solved.

thank you

---

<div class="post-metadata">

**Author:** ![edumendaya](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/edumendaya/32/782_2.png) [@edumendaya](https://community.openpbs.org/u/edumendaya)\
**Post date:** [November 28, 2024, 5:39am UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597/6 "2024-11-28T05:39:22Z")

</div>

I am just involved in a similar issue too.  
Manan: you mentioned you were ‘requesting resources more than available’. How did you fixed the problem? I mean, which part of the line  
‘qsub -l select=3:ncpus=16:mem=350gb:mpiprocs=16 -l place=scatter – /bin/sleep 1000’ did you adjusted to not overpass your cluster’s resources?  
Thanks in advance

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [November 29, 2024, 5:13pm UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597/7 "2024-11-29T17:13:19Z")

</div>

> [@edumendaya](#):
>
> ‘qsub -l select=3:ncpus=16:mem=350gb:mpiprocs=16 -l place=scatter – /bin/sleep 1000’

This means you are asking for 3 compute nodes , each node with 16 cores, 350GB RAM.  
So now you need to check whether

- each of your compute nodes have 16 cores and 350GB of memory
- you can share the obfuscated pbsnodes -av output if in case node names are classified.

---

<div class="post-metadata">

**Author:** ![edumendaya](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/edumendaya/32/782_2.png) [@edumendaya](https://community.openpbs.org/u/edumendaya)\
**Post date:** [November 29, 2024, 7:43pm UTC](https://community.openpbs.org/t/memory-usage-across-nodes/2597/8 "2024-11-29T19:43:29Z")

</div>

Adarsh: thank you very much!

I’m going to try considering this.
