We recently performed an in place upgrade from Debian 8 to Debian 9 on our cluster. Torque was not upgraded. Since the upgrade User limits were restricted. I have been able to raise the limits for everything but the jobs running under the pbs_mom. Users directly logging in to a node receive unlimited memory limits. The pbs_mom is running with no memory limits. Only running jobs are affected and receive these limits:
core file size (blocks, -c) unlimited ► data seg size (kbytes, -d) 2097152 ◄ scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 128351 max locked memory (kbytes, -l) unlimited ► max memory size (kbytes, -m) 2097152 ◄ open files (-n) 65536 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) 128351 virtual memory (kbytes, -v) 16777216 file locks (-x) unlimited
Limits assigned to the running pbs_mom process:
Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size unlimited unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 128347 128347 processes Max open files 65536 65536 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 128347 128347 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
PBS Torque MOM version 5.1.3
Linux polyp14 4.9.0-14-amd64 #1 SMP Debian 4.9.246-2 (2020-12-17) x86_64 GNU/Linux
To change the limits these files have been created/modified:
/etc/security/limits.conf /etc/systemd/user.conf /etc/systemd/system.conf /etc/init./pbs_mom /etc/systemd/system/pbs_mom.service.d/override.conf
I do not believe cgroups are active.
The only task that receives reduced limits is a job running under PBS. Thank you for any help or guidance you can provide .