# Increasing the open files limit across all nodes **URL:** \ **Category:** Users/Site Administrators\ **Created:** [December 18, 2018, 10:47pm UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361 "2018-12-18T22:47:35Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![speleolinux](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@speleolinux](https://community.openpbs.org/u/speleolinux)\ **Post date:** [December 18, 2018, 10:47pm UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361/1 "2018-12-18T22:47:35Z")
Hi I need to up the ulimit for open files from 1024 to maybe 10000. I have added that limit for a user in `/etc/security/limits.conf` on all nodes. I have added “`session required pam_limits.so`” to `/etc/pam.d/login` With a normal login that larger limit now works but not for PBS interactive logins nor PBS jobs. I have seen this post where they needed to up the stack-size. > [@How to set up stack-size to unlimited in PBS pro](http://community.openpbs.org/t/how-to-set-up-stack-size-to-unlimited-in-pbs-pro/235/2): > > I believe that you can do what you are looking for by changing the file PBS\_EXEC/lib/init.d/limits.pbs\_mom on each mom I typically just remove the if (line 4) and the fi (last line) lines. Then I set the desired ulimit parameters and values and then restart the mom. Jon They needed to edit PBS\_EXEC/lib/init.d/limits.pbs\_mom on each mom. I have therefore edited `/usr/pbs/lib/init.d/limits.pbs_mom` (on one not busy node) and added at the end of the file: ``` if [-f /etc/sgi-release -o -f /etc/sgi-compute-node-release] ; then MEMLOCKLIM=`ulimit -l` NOFILESLIM=`ulimit -n` STACKLIM=`ulimit -s` ulimit -l unlimited ulimit -n 16384 ulimit -s unlimited fi NOFILESLIM=10000 ``` then `/etc/init.d/pbs restart` Using a PBS interactive login job to login to that node I get the old limit still: ``` mynode~$ ulimit -Sn 1024 ``` If I login directly without going through PBS I get the larger limit set in limits.conf as expected. So there is something else I need to do to get PBS to set a ulimit. (Note: I have not changed limits.pbs\_mom on the head node.) I’m using PBSPro 14.2. There is no mention of ulimit or NOFILESLIM in the Admin or Reference Guides. Mike ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [December 19, 2018, 11:34am UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361/2 "2018-12-19T11:34:23Z")
Please edit $PBS\_EXEC/lib/init.d/limits.pbs\_mom on each mom. with the below contents (remove any other contents) and restart the mom services ulimit -H -s unlimited ulimit -S -s 65536 ulimit -l unlimited ulimit -n 10000 Otherwise: - try by setting it in sysctl.conf and then sysctl -p - /etc/security/limits.d/nofile.conf , setting hard and soft limits in this file ---
**Author:** ![speleolinux](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@speleolinux](https://community.openpbs.org/u/speleolinux)\ **Post date:** [December 19, 2018, 12:51pm UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361/3 "2018-12-19T12:51:33Z")
Hi adarsh I added those 4 ulimit commands to limits.pbs\_mom just one one non-busy node. Then /etc/init.d/pbs restart. Now an interactive PBS does show the higher ulimit 🙂 So I should not have used “NOFILESLIM=10000” (it looked to me like NOFILESLIM was a PBS env variable) and I should use the actual ulimit commands in limits.pbs\_mom Do I need to do a “init.d/pbs restart” or would a HUP of the mom process be sufficient? i.e. ``` $ ps ax | grep pbs 16461 ? Ssl 0:00 /usr/pbs/sbin/pbs_mom $ sudo kill -HUP 16461 ``` A HUP would not affect running jobs - is that correct? A restart of the mom would affect running jobs ? I’ll check more tomorrow when at work. Mike ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [December 19, 2018, 3:14pm UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361/4 "2018-12-19T15:14:23Z")
Hi Mike , > [@speleolinux](#): > > So I should not have used “NOFILESLIM=10000” (it looked to me like NOFILESLIM was a PBS env variable) and I should use the actual ulimit commands in limits.pbs\_mom Please use ulimit commands in limits.pbs\_mom > [@speleolinux](#): > > Do I need to do a “init.d/pbs restart” or would a HUP of the mom process be sufficient? i.e. /etc/init.d/pbs restart or systemctl restart pbs is required Thank you ---
**Author:** ![pguyan](https://avatars.discourse-cdn.com/v4/letter/p/b5ac83/32.png) [@pguyan](https://community.openpbs.org/u/pguyan)\ **Post date:** [December 19, 2018, 4:38pm UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361/5 "2018-12-19T16:38:51Z")
I’ve been having similar issues, my solution is the following: In /etc/systemd/system.conf append “DefaultTasksMax=65536” set hardnofile in /etc/security/limits.conf to “\* hard nofile 131072” Hope this helps, Pete ---
**Author:** ![speleolinux](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@speleolinux](https://community.openpbs.org/u/speleolinux)\ **Post date:** [December 20, 2018, 11:13pm UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361/6 "2018-12-20T23:13:50Z")
Hi all Thanks for this help. I have rolled out those changes to several nodes that had no jobs on them, installing the new limits.pbs\_mom and a full restart of PBS mom. From my testing that looks like it has worked fine. Thanks Mike ---
**Author:** ![vincent718](https://avatars.discourse-cdn.com/v4/letter/v/f9ae1b/32.png) [@vincent718](https://community.openpbs.org/u/vincent718)\ **Post date:** [January 13, 2021, 1:52pm UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361/7 "2021-01-13T13:52:16Z")
Hi @speleolinux . When you say full restart of PBS mom do you restart on master node or compute node? I also want increase the file limits on our HPC and have been struggling for months now. Kindly indicate the exact steps you used for me . Thanks ---
**Author:** ![speleolinux](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@speleolinux](https://community.openpbs.org/u/speleolinux)\ **Post date:** [January 18, 2021, 5:58am UTC](https://community.openpbs.org/t/increasing-the-open-files-limit-across-all-nodes/1361/8 "2021-01-18T05:58:21Z")
Hi Vincent Check if you already have a file /opt/pbs/lib/init.d/limits.pbs\_mom on your execution nodes. If so make a backup of it. Create your new version and copy it into place. On each exec node restart PBS: /etc/init.d/pbs restart You can do this without affecting running jobs. Though best still to do this on a test node first. Mike