Only Jobs running under PBS (pbs_mom) are limited to 2097152 bytes

[mmotsko]

We recently performed an in place upgrade from Debian 8 to Debian 9 on our cluster. Torque was not upgraded. Since the upgrade User limits were restricted. I have been able to raise the limits for everything but the jobs running under the pbs_mom. Users directly logging in to a node receive unlimited memory limits. The pbs_mom is running with no memory limits. Only running jobs are affected and receive these limits:

 core file size          (blocks, -c) unlimited
► data seg size           (kbytes, -d) 2097152  ◄
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 128351
max     locked memory       (kbytes, -l) unlimited
► max memory size         (kbytes, -m) 2097152  ◄
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 128351
virtual memory          (kbytes, -v) 16777216
file locks                      (-x) unlimited

Limits assigned to the running pbs_mom process:

Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            unlimited            unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             128347               128347               processes 
Max open files            65536                65536                files     
Max locked memory         unlimited            unlimited            bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       128347               128347               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

Important information:
PBS Torque MOM version 5.1.3

Kernel:
Linux polyp14 4.9.0-14-amd64 #1 SMP Debian 4.9.246-2 (2020-12-17) x86_64 GNU/Linux

To change the limits these files have been created/modified:

/etc/security/limits.conf
/etc/systemd/user.conf
/etc/systemd/system.conf
/etc/init./pbs_mom
/etc/systemd/system/pbs_mom.service.d/override.conf

I do not believe cgroups are active.

The only task that receives reduced limits is a job running under PBS. Thank you for any help or guidance you can provide .

[adarsh]

Thank you @mmotsko for sharing this on the forum

@mmotsko

To change the limits these files have been created/modified:

Please try insert ulimit settings in /etc/init.d/pbs_mom

Note: in openpbs it would be $PBS_EXEC/lib/init.d/limits.pbs_mom

This is forum for openpbs (PBS Pro OSS) .

PBS Default number of nodes and number of cpus

So a little history for your knowledge. PBS was originally developed for NASA under a contract project. Altair Engineering acquired the rights to all the PBS technology and intellectual property from Veridian in 2003. Altair Engineering currently owns and maintains the intellectual property associated with PBS, and also employs the original development team from NASA. The following versions of PBS are currently available: OpenPBS — original open source version released by MRJ in 1998 (not ac…

Please find the documentation for openpbs at
https://www.altair.com/pdfs/pbsworks/PBSInstallGuide2021.1.pdf
https://www.altair.com/pdfs/pbsworks/PBSAdminGuide2021.1.pdf

Hope this helps

[mmotsko]

Thank you @adarsh for you assistance. I tried the recommendations you suggested prior to posting to this forum. I am a long ago administrator of PBS Pro and Torque and I had the misconception that PBS Torque was now OpenPBS. Thank you again.