# $TMPDIR not created on all execution hosts? **URL:** \ **Category:** Users/Site Administrators\ **Created:** [July 10, 2018, 8:49pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072 "2018-07-10T20:49:47Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![coreyferrier](https://avatars.discourse-cdn.com/v4/letter/c/f07891/32.png) [@coreyferrier](https://community.openpbs.org/u/coreyferrier)\ **Post date:** [July 10, 2018, 8:49pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/1 "2018-07-10T20:49:47Z")
I thought $TMPDIR was supposed to be automatically created on all execution hosts in addition to the lead node, but it isn’t. We have $TMPDIR set to /local\_scratch in the MOM config file. Do we need to create it via a prologue script? ---
**Author:** ![gabe](https://avatars.discourse-cdn.com/v4/letter/g/5daacb/32.png) [@gabe](https://community.openpbs.org/u/gabe)\ **Post date:** [July 10, 2018, 9:16pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/2 "2018-07-10T21:16:11Z")
The way I interpret the documentation (section 9.1.15 in the Admin Guide), you set $tmpdir in the MOM config on each node to be the root of your temp location (/local\_scratch in your case). If you’ve done that, then I would expect the MOM to create a unique directory in /local\_scratch for the job; probably /local\_scratch/$PBS\_JOBID. HTH, Gabe ---
**Author:** ![coreyferrier](https://avatars.discourse-cdn.com/v4/letter/c/f07891/32.png) [@coreyferrier](https://community.openpbs.org/u/coreyferrier)\ **Post date:** [July 10, 2018, 9:25pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/3 "2018-07-10T21:25:31Z")
Yes, I understood it that way too. However, the TMPDIR is being created on the lead node only. ---
**Author:** ![gabe](https://avatars.discourse-cdn.com/v4/letter/g/5daacb/32.png) [@gabe](https://community.openpbs.org/u/gabe)\ **Post date:** [July 10, 2018, 9:43pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/5 "2018-07-10T21:43:03Z")
If you are setting $tmpdir in the MOM config on each host, but the job-specific temp dir is not being created on each host, then it could be a bug/regression. If you’ve just recently added $tmpdir to each MOM config, be sure that you restarted each of the MOMs. ---
**Author:** ![coreyferrier](https://avatars.discourse-cdn.com/v4/letter/c/f07891/32.png) [@coreyferrier](https://community.openpbs.org/u/coreyferrier)\ **Post date:** [July 10, 2018, 10:05pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/6 "2018-07-10T22:05:29Z")
We’ve had tmpdir set to /local\_scratch for a long time now. However I’ve never checked to see if tmpdir was being created on all execution hosts. We are running open source 17.1.0. Don’t have the opportunity to update in the near future. Is creating tmpdir manually via a prologue acceptable? ---
**Author:** ![gabe](https://avatars.discourse-cdn.com/v4/letter/g/5daacb/32.png) [@gabe](https://community.openpbs.org/u/gabe)\ **Post date:** [July 10, 2018, 10:34pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/7 "2018-07-10T22:34:28Z")
I can confirm in PBS Pro 14.2.6 that when I set $tmpdir in the MOM config on several hosts and then run a job on those hosts, the temporary directory is only created by the Mother Superior and _not_ by the sister MOMs. This is contrary to the documentation (section 10.14.3 in the Administrator Guide for 14.2.1). Additionally, PBS\_TMPDIR is not defined in my job environment, neither by the Mother Superior nor the sister MOMs, also contrary to the documentation. @scc, do you know if there have been any bugs or regressions filed about this? Either there is a problem with PBS Pro or there is a problem with the documentation. ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [July 11, 2018, 4:58pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/8 "2018-07-11T16:58:54Z")
Hi @gabe and @coreyferrier! The tmpdir on the secondary execution hosts is not created until pbs\_mom actually spawns a task for the job on the node (using pbs\_attach to attach an existing process to the node does not result in the creation of the job specific temp directory, since it would be too late to get $TMPDIR set in the processes’ environment to use it anyway). ``` [user1@centos7-2 ~]$ qsub -lselect=2:ncpus=1 -lplace=scatter -I qsub: waiting for job 30.centos7-2 to start qsub: job 30.centos7-2 ready [user1@centos7-2 ~]$ cat $PBS_NODEFILE centos7-2.prog.altair.com centos7.prog.altair.com [root@centos7 ~]# ls -ld /var/tmp/pbs.* ls: cannot access /var/tmp/pbs.*: No such file or directory [user1@centos7-2 ~]$ pbs_tmrsh centos7.prog.altair.com /bin/true [user1@centos7-2 ~]$ [root@centos7 ~]# ls -ld /var/tmp/pbs.* drwx------. 2 user1 user1 6 Jul 11 12:57 /var/tmp/pbs.30.centos7-2 [root@centos7 ~]# ``` I hope this helps! ---
**Author:** ![coreyferrier](https://avatars.discourse-cdn.com/v4/letter/c/f07891/32.png) [@coreyferrier](https://community.openpbs.org/u/coreyferrier)\ **Post date:** [July 12, 2018, 10:37am UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/9 "2018-07-12T10:37:27Z")
Thanks Scott! I’ve confirmed you are correct. In my earlier test I had not spawned a task on the other nodes as you state. ---
**Author:** ![berlin2123](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/berlin2123/32/448_2.png) [@berlin2123](https://community.openpbs.org/u/berlin2123)\ **Post date:** [October 16, 2025, 12:59pm UTC](https://community.openpbs.org/t/tmpdir-not-created-on-all-execution-hosts/1072/10 "2025-10-16T12:59:53Z")
So, what is the way to solve this? My software need to write in TMPDIR. The software will stop if without TMPDIR. I am using `qsub test.pbs` to run jobs. So, write a for-loop in the test.pbs file to run “pbs\_tmrsh“ ? fine, write the fllowing lines into the pbs file, solve it. ```auto cat $PBS_NODEFILE | uniq | while read -r line; do pbs_tmrsh $line /bin/true done ```