# Job error: 'Exit\_status = -2' **URL:** \ **Category:** Users/Site Administrators\ **Created:** [August 14, 2018, 4:21pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140 "2018-08-14T16:21:54Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 14, 2018, 4:21pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/1 "2018-08-14T16:21:54Z")
Hello, We are encountering “Exit\_status = -2” on a particular node and we are not sure why… There are no mom\_logs and server\_logs don’t help either. We did notice that submitting a job after a node reboot results in a hanging experience for the user: > $ qsub -I -l walltime=08:00:00 -l host=node0047 -q def-devel > > qsub: waiting for job 38178.bright01-thx to start > \<…it just HANGS…so I quit…\> > ^CDo you wish to terminate the job and exit (y|[n])? y > Job 38178.bright01-thx is being deleted Even though qstat -f 38178 says: ``` comment = Job run at Tue Aug 14 at 11:01 on (node0047:ncpus=1:mem=1048576kb :mic_cores=0:ngpus=0) ``` And then the next job submission attempt just quits immediately with the ‘Exit\_status = -2’ error: > $ qsub -I -l walltime=08:00:00 -l host=node0047 -q def-devel > > qsub: waiting for job 38179.bright01-thx to start > > qsub: job 38179.bright01-thx ready > > qsub: job 38179.bright01-thx completed Any ideas how we can uncover the root cause of this issue? Thanks, Siji ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 15, 2018, 7:48am UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/2 "2018-08-15T07:48:57Z")
Please increase the mom logging by adding the below line to $PBS\_HOME/mom\_priv/config and restart the mom service on node0047 and submit a new job ``` $logevent 0xfffffff ``` Exit\_Code -2 means - Job execution failed, after files, no retry Probably tracejob and mom logs would help. Might be related to the ports / firewall. Thank you ---
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 16, 2018, 2:46pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/3 "2018-08-16T14:46:36Z")
@adarsh After increasing logging, there still isn’t any mom\_log to review. In fact, the $PBS\_MOM\_HOME/mom\_logs is empty. > # tracejob 38822 > > Job: 38822.bright01-thx > > 08/16/2018 09:37:40 L Considering job to run > 08/16/2018 09:37:40 S Job Queued at request of saula@login-0002, owner = saula@login-0002, job name = STDIN, queue = def-devel > 08/16/2018 09:37:40 S Job Run at request of Scheduler@bright01-thx.thunder.ccast on exec\_vnode (node0047:ncpus=1:mem=1048576kb:mic\_cores=0:ngpus=0) > 08/16/2018 09:37:40 S Job Modified at request of Scheduler@bright01-thx.thunder.ccast > 08/16/2018 09:37:40 L Job run > 08/16/2018 09:37:40 S enqueuing into def-devel, state 1 hop 1 > 08/16/2018 09:37:40 A queue=def-devel > 08/16/2018 09:37:40 A user=saula group=saula\_g project=\_pbs\_project\_default jobname=STDIN queue=def-devel ctime=1534430260 qtime=1534430260 etime=1534430260 > start=1534430260 exec\_host=node0047/0 exec\_vnode=(node0047:ncpus=1:mem=1048576kb:mic\_cores=0:ngpus=0) Resource\_List.host=node0047 > Resource\_List.mem=1gb Resource\_List.mic\_cores=0 Resource\_List.ncpus=1 Resource\_List.ngpus=0 Resource\_List.nodect=1 Resource\_List.place=pack > Resource\_List.select=1:host=node0047:ncpus=1 Resource\_List.walltime=08:00:00 resource\_assigned.mem=1048576kb resource\_assigned.ncpus=1 > resource\_assigned.ngpus=0 resource\_assigned.mic\_cores=0 > 08/16/2018 09:39:02 S Obit received momhop:1 serverhop:1 state:4 substate:41 > 08/16/2018 09:39:06 S Exit\_status=-2 resources\_used.cpupercent=0 resources\_used.cput=00:00:00 resources\_used.mem=0kb resources\_used.ncpus=1 resources\_used.vmem=0kb > resources\_used.walltime=00:00:00 > 08/16/2018 09:39:06 A user=saula group=saula\_g project=\_pbs\_project\_default jobname=STDIN queue=def-devel ctime=1534430260 qtime=1534430260 etime=1534430260 > start=1534430260 exec\_host=node0047/0 exec\_vnode=(node0047:ncpus=1:mem=1048576kb:mic\_cores=0:ngpus=0) Resource\_List.host=node0047 > Resource\_List.mem=1gb Resource\_List.mic\_cores=0 Resource\_List.ncpus=1 Resource\_List.ngpus=0 Resource\_List.nodect=1 Resource\_List.place=pack > Resource\_List.select=1:host=node0047:ncpus=1 Resource\_List.walltime=08:00:00 session=0 end=1534430346 Exit\_status=-2 resources\_used.cpupercent=0 > resources\_used.cput=00:00:00 resources\_used.mem=0kb resources\_used.ncpus=1 resources\_used.vmem=0kb resources\_used.walltime=00:00:00 run\_count=1 Also, what does the “…,after files,…” mean in your interpretation of Exit\_Code -2? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 16, 2018, 3:21pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/4 "2018-08-16T15:21:12Z")
> [@sijisaula](#): > > after files means after staging of the data ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 16, 2018, 3:23pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/5 "2018-08-16T15:23:10Z")
> [@sijisaula](#): > > After increasing logging, there still isn’t any mom\_log to review. In fact, the $PBS\_MOM\_HOME/mom\_logs is empty. Please check pbs\_mom service is up and running, please share your /etc/pbs.conf contents. If mom services are up and running then there should be $PBS\_HOME/mom\_logs/YYYYMMDD or $PBS\_MOM\_HOME/mom\_logs/YYYYMMDD ---
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 16, 2018, 4:22pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/6 "2018-08-16T16:22:23Z")
Mom is up per systemctl and our pbs.conf is shown beneath: > [node0047 ~]# systemctl status pbs > > **●** pbs.service - LSB: The Portable Batch System (PBS) is a flexible workload > > Loaded: loaded (/etc/rc.d/init.d/pbs; bad; vendor preset: disabled) > > Active: **active (running)** since Wed 2018-08-15 09:33:44 CDT; 1 day 1h ago > > Docs: man:systemd-sysv-generator(8) > > Process: 72288 ExecStop=/etc/rc.d/init.d/pbs stop (code=exited, status=0/SUCCESS) > > Process: 72335 ExecStart=/etc/rc.d/init.d/pbs start (code=exited, status=0/SUCCESS) > > CGroup: /system.slice/pbs.service > > └─72405 /cm/shared/apps/pbspro-ce/current/sbin/pbs\_mom > > Aug 15 09:33:44 node0047 systemd[1]: Starting LSB: The Portable Batch System (PBS) is a flexible workload… > > Aug 15 09:33:44 node0047 pbs[72335]: Starting PBS > > Aug 15 09:33:44 node0047 pbs[72335]: PBS mom > > Aug 15 09:33:44 node0047 systemd[1]: Started LSB: The Portable Batch System (PBS) is a flexible workload. > [node0047 ~]# cat /etc/pbs.conf > > PBS\_EXEC=/cm/shared/apps/pbspro-ce/current > > PBS\_HOME=/cm/local/apps/pbspro-ce/var/spool > > PBS\_START\_SERVER=0 > > PBS\_START\_SCHED=0 > > PBS\_MOM\_HOME=/cm/local/apps/pbspro-ce/var/spool > > PBS\_START\_MOM=1 > > PBS\_START\_COMM=0 > > PBS\_SERVER=bright01-thx > > PBS\_SCP=/usr/bin/scp > > PBS\_RSHCOMMAND=/usr/bin/ssh > > PBS\_CORE\_LIMIT=unlimited ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 16, 2018, 4:43pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/7 "2018-08-16T16:43:26Z")
Thank you , Could you please search **20180816** file on the mom node ? find / -name “20180816” -print ---
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 16, 2018, 6:15pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/8 "2018-08-16T18:15:47Z")
Still nothing… [node0047 ~]# find / -name “20180816” -print [node0047 ~]# I’m doing some other digging as well, but let me know if you have other thoughts. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 16, 2018, 7:17pm UTC](https://community.openpbs.org/t/job-error-exit-status-2/1140/9 "2018-08-16T19:17:58Z")
1. Try running this command $PBS\_EXEC/unsupported/pbs\_dtj 38822 2. re-install PBS MOM on that node , check whether the logs are written or check whether logs are redirected to syslogs in your exsiting setup or other 3. Looks like you are using cluster management system (by the looks of your /etc/pbs.conf ) - make sure hostname resolution is correct / firewall / seliinux etc - [Interactive Job errors out with 'apparently deleted'](http://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/13?u=adarsh) - kill the PBS MOM on the compute node and start it with strace -o /tmp/mom.txt $PBS\_EXEC/sbin/pbs\_mom - strace qsub -I -l walltime=08:00:00 -l host=node0047 -q def-devel - then try to look into strace information and analyze