# Interactive Job errors out with 'apparently deleted' **URL:** \ **Category:** Users/Site Administrators\ **Created:** [August 1, 2018, 8:05pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112 "2018-08-01T20:05:19Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 1, 2018, 8:05pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/1 "2018-08-01T20:05:19Z")
Hello, We are seeing this issue following some recent change to queue priorities and preemptive scheduling: $ qsub -I -l walltime=08:00:00 -l select=1:ncpus=4 -q def-devel qsub: waiting for job 29679.bright01-thx to start qsub: job 29679.bright01-thx apparently deleted What could be the source of the issue? Here’s what our queue configuration looks like: create queue def-devel set queue def-devel queue\_type = Execution set queue def-devel Priority = 100 set queue def-devel acl\_host\_enable = False set queue def-devel resources\_max.walltime = 08:00:00 set queue def-devel resources\_min.walltime = 00:00:00 set queue def-devel resources\_available.ngpus = 0 set queue def-devel max\_run\_res.condo = [o:PBS\_ALL=38] set queue def-devel max\_run\_res.gpuHost = [o:PBS\_ALL=2] set queue def-devel max\_run\_res.ncpus = [o:PBS\_ALL=257] set queue def-devel enabled = True set queue def-devel started = True Many thanks, Siji ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 1, 2018, 8:17pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/2 "2018-08-01T20:17:51Z")
Please check or share the _tracejob 29679_ output and _mom logs_ of the system on which this job was running. ---
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 1, 2018, 9:08pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/3 "2018-08-01T21:08:35Z")
> [@adarsh](#): > > _tracejob 29679_ Sure here it is: # tracejob 29679 Job: 29679.bright01-thx 08/01/2018 14:56:26 S enqueuing into def-devel, state 1 hop 1 08/01/2018 14:56:26 A queue=def-devel 08/01/2018 14:56:27 S Job Queued at request of saula@login2, owner = saula@login2, job name = STDIN, queue = def-devel 08/01/2018 14:56:28 L Considering job to run 08/01/2018 14:56:28 S Job Run at request of Scheduler@bright01-thx.thunder.ccast on exec\_vnode (node0115:ncpus=4:mem=1048576kb:mic\_cores=0:ngpus=0) 08/01/2018 14:56:28 L Job run 08/01/2018 14:56:29 S Job Modified at request of Scheduler@bright01-thx.thunder.ccast 08/01/2018 14:56:29 S Obit received momhop:1 serverhop:1 state:4 substate:41 08/01/2018 14:56:29 A user=saula group=saula\_g project=\_pbs\_project\_default jobname=STDIN queue=def-devel ctime=1533153386 qtime=1533153386 etime=1533153386 start=1533153389 exec\_host=node0115/0_4 exec\_vnode=(node0115:ncpus=4:mem=1048576kb:mic\_cores=0:ngpus=0) Resource\_List.mem=1gb Resource\_List.mic\_cores=0 Resource\_List.ncpus=4 Resource\_List.ngpus=0 Resource\_List.nodect=1 Resource\_List.place=free Resource\_List.select=1:ncpus=4 Resource\_List.walltime=08:00:00 resource\_assigned.mem=1048576kb resource\_assigned.ncpus=4 resource\_assigned.ngpus=0 resource\_assigned.mic\_cores=0 08/01/2018 14:56:30 S Exit\_status=-1 resources\_used.cpupercent=0 resources\_used.cput=00:00:00 resources\_used.mem=0kb resources\_used.ncpus=4 resources\_used.vmem=0kb resources\_used.walltime=00:00:00 08/01/2018 14:56:30 A user=saula group=saula\_g project=\_pbs\_project\_default jobname=STDIN queue=def-devel ctime=1533153386 qtime=1533153386 etime=1533153386 start=1533153389 exec\_host=node0115/0_4 exec\_vnode=(node0115:ncpus=4:mem=1048576kb:mic\_cores=0:ngpus=0) Resource\_List.mem=1gb Resource\_List.mic\_cores=0 Resource\_List.ncpus=4 Resource\_List.ngpus=0 Resource\_List.nodect=1 Resource\_List.place=free Resource\_List.select=1:ncpus=4 Resource\_List.walltime=08:00:00 session=0 end=1533153390 Exit\_status=-1 resources\_used.cpupercent=0 resources\_used.cput=00:00:00 resources\_used.mem=0kb resources\_used.ncpus=4 resources\_used.vmem=0kb resources\_used.walltime=00:00:00 run\_count=1 ---
**Author:** ![bremanandjk](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bremanandjk/32/144_2.png) [@bremanandjk](https://community.openpbs.org/u/bremanandjk)\ **Post date:** [August 1, 2018, 9:35pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/4 "2018-08-01T21:35:22Z")
One reason would be, - If pbs\_mom could not resolve the FQDN of the pbs\_server, this issue will happen. In that case, we can see below pbs\_mom logs. 08/02/2018 02:50:34;0100;pbs\_mom;Job;6.BLRLAP796;allowed brema to access window station and desktop, User brema passworded 08/02/2018 02:50:34;0001;pbs\_mom;Svr;pbs\_mom;No error (0) in finish\_exec, cannot open qsub sock for 6.BLRLAP796 08/02/2018 02:50:34;0008;pbs\_mom;Job;6.BLRLAP796;cannot open qsub sock for 6.BLRLAP796 08/02/2018 02:50:35;0100;pbs\_mom;Job;6.BLRLAP796;task 00000001 cput= 0:00:00 08/02/2018 02:50:35;0008;pbs\_mom;Job;6.BLRLAP796;kill\_job 08/02/2018 02:50:35;0100;pbs\_mom;Job;6.BLRLAP796;spark cput= 0:00:00 mem=0kb 08/02/2018 02:50:35;0100;pbs\_mom;Job;6.BLRLAP796;Obit sent 08/02/2018 02:50:35;0100;pbs\_mom;Req;;Type 6 request received from brema@192.168.10.1:15001, sock=1 08/02/2018 02:50:35;0080;pbs\_mom;Job;6.BLRLAP796;delete job request received 08/02/2018 02:50:35;0008;pbs\_mom;Job;6.BLRLAP796;kill\_job ---
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 1, 2018, 9:49pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/5 "2018-08-01T21:49:28Z")
Hmm, that sounds really strange since we have batch jobs running on those nodes without incident. Just interactive jobs are acting this way. I believe the issue here is specific to either the queue or the job type… ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 1, 2018, 9:53pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/6 "2018-08-01T21:53:21Z")
> [@sijisaula](#): > > 08/01/2018 14:56:30 S Exit\_status=-1 resources\_used.cpupercent=0 resources\_used.cput=00:00:00 resources\_used.mem=0kb resources\_used.ncpus=4 resources\_used.vmem=0kb > resources\_used.walltime=00:00:00 Exit\_Code = -1 Job execution failed, before files, no retry Could you please paste the mom logs for this job id 29679 from node0115 ? ---
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 1, 2018, 10:06pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/7 "2018-08-01T22:06:30Z")
Interesting there isn’t a job directory or file related to this job on node0115: [node0115 ~]# ls -ltr /cm/local/apps/pbspro-ce/var/spool/mom\_priv/jobs/_29679_ ls: cannot access /cm/local/apps/pbspro-ce/var/spool/mom\_priv/jobs/_29679_: No such file or directory ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 1, 2018, 10:12pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/8 "2018-08-01T22:12:07Z")
Please check as below on node0115 for PBS Mom logs : - source /etc/pbs.conf - cd $PBS\_HOME/mom\_logs/ - cat 20180801 | grep or do vi 20180801 and check ---
**Author:** ![speleolinux](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@speleolinux](https://community.openpbs.org/u/speleolinux)\ **Post date:** [August 2, 2018, 3:04am UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/9 "2018-08-02T03:04:06Z")
Hi bremanandjk suggested “If pbs\_mom could not resolve the FQDN of the pbs\_server, this issue will happen.” I had the same error with only interactive jobs failing and it turned out I had the wrong ip address for the login node in the /etc/hosts file on the head node/PBS server. Hence the FQDN lookup was incorrect. Mike ---
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 2, 2018, 8:25am UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/10 "2018-08-02T08:25:58Z")
> [@sijisaula](#): > > 29679 Here’s what we have: [node0115 mom\_logs]# cat 20180801 | grep 29679 08/01/2018 14:56:29;0001;pbs\_mom;Svr;pbs\_mom;Operation now in progress (115) in finish\_exec, cannot open qsub sock for 29679.bright01-thx 08/01/2018 14:56:29;0001;pbs\_mom;Job;29679.bright01-thx;job not started, Failure -1 08/01/2018 14:56:29;0100;pbs\_mom;Job;29679.bright01-thx;task 00000001 cput= 0:00:00 08/01/2018 14:56:29;0008;pbs\_mom;Job;29679.bright01-thx;kill\_job 08/01/2018 14:56:29;0100;pbs\_mom;Job;29679.bright01-thx;node0115 cput= 0:00:00 mem=0kb 08/01/2018 14:56:29;0008;pbs\_mom;Job;29679.bright01-thx;no active tasks 08/01/2018 14:56:29;0100;pbs\_mom;Job;29679.bright01-thx;Obit sent 08/01/2018 14:56:30;0008;pbs\_mom;Job;29679.bright01-thx;no active tasks 08/01/2018 14:56:30;0080;pbs\_mom;Job;29679.bright01-thx;delete job request received 08/01/2018 14:56:30;0008;pbs\_mom;Job;29679.bright01-thx;kill\_job ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 2, 2018, 8:28am UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/11 "2018-08-02T08:28:46Z")
> [@sijisaula](#): > > 08/01/2018 14:56:29;0001;pbs\_mom;Svr;pbs\_mom;Operation now in progress (115) in finish\_exec, **cannot open qsub sock for 29679.bright01-thx** As rightly mentioned by @bremanandjk , it is related to DNS issues > [@speleolinux](#): > > “If pbs\_mom could not resolve the FQDN of the pbs\_server, this issue will happen.” ---
**Author:** ![sijisaula](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sijisaula](https://community.openpbs.org/u/sijisaula)\ **Post date:** [August 2, 2018, 9:15am UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/12 "2018-08-02T09:15:34Z")
@bremanandjk, @speleolinux, @adarsh You might be onto something here… I did find our login-node’s ip in /etc/hosts to be inconsistent with that of the ip listed in the PBS server’s /etc/host and the ip on the compute node’s /etc/host file. I went ahead and corrected this error in the login node’s /etc/host file but still received the ‘apparently deleted error’ Maybe the pbs\_mom on this compute node needs to be restarted? I’ll try that next and check over my /etc/hosts afresh… ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [August 2, 2018, 2:52pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/13 "2018-08-02T14:52:29Z")
If that does not work I’d suggest trying the following (which mimics essentially what PBS is doing to make the connection in the interactive job): Submit an interactive job that we know will not run (give it a really high ncpus request so that it will remain queued). While that job is queued, look at it in qstat -f and note the full value of PBS\_O\_HOST in the Variable\_List attribute. On the same host that qsub -I is waiting for the job to start, run netstat -anp | grep qsub | grep LISTEN, note the port number (after the “0.0.0.0:”). Now log into node0115 as root and issue the command “telnet X Y”, where X is the PBS\_O\_HOST value from qstat -f and Y is the port number from netstat. Do you get “Escape character is ‘^]’.”, or something else? ---
**Author:** ![ndusek](https://avatars.discourse-cdn.com/v4/letter/n/779978/32.png) [@ndusek](https://community.openpbs.org/u/ndusek)\ **Post date:** [August 7, 2018, 4:58pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/14 "2018-08-07T16:58:19Z")
I work with @sijisaula and am assisting with this as well. I followed your instructions and did not receive the standard telnet response of “Escape character is ‘^]’.” Instead, I got the following (leaving out the actual address): Trying [$PBS\_O\_HOST address] … telnet: connect to address [$PBS\_O\_HOST address]: No route to host However, the same $PBS\_O\_HOST can be pinged successfully from node0115 (and other compute nodes). Does this narrow it down at all? ---
**Author:** ![ndusek](https://avatars.discourse-cdn.com/v4/letter/n/779978/32.png) [@ndusek](https://community.openpbs.org/u/ndusek)\ **Post date:** [August 7, 2018, 5:28pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/15 "2018-08-07T17:28:18Z")
I think I see where you’re going with this. The port that the qsub process listens on for each interactive job has to be open on the firewall? I opened that specific port while the job was still queued and telnet now returns the expected response. However, it looks like for each new interactive job, qsub listens on a different random high-numbered port. If I were to create a general firewall rule, what range of ports should I include to guarantee that interactive qsub processes are always accepted? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [August 8, 2018, 7:07am UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/16 "2018-08-08T07:07:17Z")
PBS Pro ephemeral ports for interactive job submission: Ports from 32768 to 61000 should be open on compute nodes ---
**Author:** ![ndusek](https://avatars.discourse-cdn.com/v4/letter/n/779978/32.png) [@ndusek](https://community.openpbs.org/u/ndusek)\ **Post date:** [August 8, 2018, 12:50pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/17 "2018-08-08T12:50:23Z")
Great, thank you. I can confirm that this problem has been resolved. Thanks everyone for your help! ---
**Author:** ![brownwrap](https://avatars.discourse-cdn.com/v4/letter/b/94ad74/32.png) [@brownwrap](https://community.openpbs.org/u/brownwrap)\ **Post date:** [November 18, 2019, 11:49pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/18 "2019-11-18T23:49:38Z")
> [@adarsh](#): > > 32768 This problem seems to be the same issue I currently have. The recommendation was to make sure a range of ports was open. Nothing has changed on our setup, other than a reboot, but interactive jobs stopped working. ---
**Author:** ![boboshaq](https://avatars.discourse-cdn.com/v4/letter/b/bc8723/32.png) [@boboshaq](https://community.openpbs.org/u/boboshaq)\ **Post date:** [July 14, 2023, 12:15pm UTC](https://community.openpbs.org/t/interactive-job-errors-out-with-apparently-deleted/1112/19 "2023-07-14T12:15:53Z")
> [@adarsh](#): > > Ports from 32768 to 61000 should be open on compute nodes Hi, in my case it was sufficiently to open ports just on submit node (headnode, PBS\_O\_HOST). Port range on Centos 8 is 32768-60999 . You may check it in Your Linux system by command: sudo sysctl net.ipv4.ip\_local\_port\_range . Regards!