I execute an application through PBS every five minutes(scheduled on crontab), utilizing a Docker environment for its execution. Generally, this workflow operates seamlessly. However, on rare occasions, the application fails to execute. In such instances, no output is generated, and the underlying job is not considered for execution.
For clarity, I have attached logs of both, abnormal as well as normal execution of job.
Thanks in advance for advice/help.
Logs for every abnormal instance :
05/30/2024 21:40:27;0100;pbs_python;Hook;pbs_python;create_job: Creating directory /sys/fs/cgroup/cpuset/pbspro.slice/pbspro-6015500.Prophet.slice/
05/30/2024 21:40:27;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:28;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:32;0100;pbs_python;Hook;pbs_cgroups;env orig: PATH=/bin:/usr/bin,PBS_O_SYSTEM=Linux,PBS_O_SHELL=/bin/sh,PBS_O_HOME=/home/seradmin,PBS_O_LOGNAME=seradmin,PBS_O_WORKDIR=/home/seradmin,PBS_O_LANG=en_US.UTF-8,PBS_O_PATH=/usr/bin:/bin,CONTAINER_IMAGE=nil_22_r_v7,PBS_O_QUEUE=Hercules,PBS_O_HOST=apprentice,HOME=/home/seradmin,LOGNAME=seradmin,PBS_JOBNAME=realtime-forecast,PBS_JOBID=6015500.Prophet,PBS_QUEUE=Hercules,SHELL=/bin/bash,USER=seradmin,PBS_JOBCOOKIE=48C6378423040DA02FBF49D24E6A1B95,PBS_NODENUM=0,PBS_TASKNUM=1,PBS_MOMPORT=15003,OMP_NUM_THREADS=8,NCPUS=8,PBS_NODEFILE=/var/spool/pbs/aux/6015500.Prophet,TMPDIR=/var/tmp/pbs.6015500.Prophet,PBS_JOBDIR=/home/seradmin,PBS_ENVIRONMENT=PBS_BATCH,ENVIRONMENT=BATCH
05/30/2024 21:40:32;0100;pbs_python;Hook;pbs_cgroups;env new: PBS_O_SYSTEM=Linux,PBS_JOBCOOKIE=48C6378423040DA02FBF49D24E6A1B95,PBS_JOBNAME=realtime-forecast,PBS_O_HOME=/home/seradmin,PBS_O_HOST=apprentice,PBS_NODENUM=0,PBS_O_LOGNAME=seradmin,PBS_O_SHELL=/bin/sh,PBS_O_LANG=en_US.UTF-8,USER=seradmin,PATH=/bin:/usr/bin,PBS_MOMPORT=15003,HOME=/home/seradmin,TMPDIR=/var/tmp/pbs.6015500.Prophet,ENVIRONMENT=BATCH,PBS_NODEFILE=/var/spool/pbs/aux/6015500.Prophet,SHELL=/bin/bash,PBS_ENVIRONMENT=PBS_BATCH,OMP_NUM_THREADS=8,NCPUS=8,PBS_JOBDIR=/home/seradmin,PBS_O_QUEUE=Hercules,PBS_JOBID=6015500.Prophet,PBS_O_WORKDIR=/home/seradmin,PBS_O_PATH=/usr/bin:/bin,CUDA_VISIBLE_DEVICES=,CONTAINER_IMAGE=nil_22_r_v7,LOGNAME=seradmin,PBS_TASKNUM=1,PBS_QUEUE=Hercules
05/30/2024 21:40:33;0100;pbs_python;Hook;PBS_hpc_container;argv new: /opt/pbs/sbin/pbs_container 6015500.Prophet 1 /opt/pbs exec 6015500.Prophet /bin/bash -c /var/spool/pbs/mom_priv/jobs/6015500.Prophet.SC
05/30/2024 21:40:34;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/systemd/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:34;0100;pbs_python;Hook;pbs_python;_delete_cgroup_children: Removing directory /sys/fs/cgroup/systemd/pbspro.slice/pbspro-6015500.Prophet.orphan.slice/9fe96b8f6e55b73f4b107dfc554c6cc54ab76f1d2502de077063dcdd9ce95530
05/30/2024 21:40:34;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/systemd/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:34;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/memory/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/memory/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/blkio/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/blkio/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/cpu,cpuacct/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/cpu,cpuacct/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/freezer/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/freezer/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/cpuset/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/cpuset/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/hugetlb/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/hugetlb/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/devices/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/devices/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;cleanup_orphans: Removing orphaned cgroup: /sys/fs/cgroup/pids/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:35;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/pids/pbspro.slice/pbspro-6015500.Prophet.orphan.slice
05/30/2024 21:40:36;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:36;0008;pbs_mom;Job;6015500.Prophet;no active process for task 00000001
05/30/2024 21:40:36;0008;pbs_mom;Job;6015500.Prophet;Started, pid = 104698
05/30/2024 21:40:36;0008;pbs_mom;Job;6015500.Prophet;Terminated
05/30/2024 21:40:36;0100;pbs_mom;Job;6015500.Prophet;task 00000001 cput= 0:00:00
05/30/2024 21:40:36;0008;pbs_mom;Job;6015500.Prophet;kill_job
05/30/2024 21:40:36;0100;pbs_mom;Job;6015500.Prophet;Prophet cput= 0:00:00 mem=0kb
05/30/2024 21:40:36;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:36;0100;pbs_mom;Job;6015500.Prophet;Obit sent
05/30/2024 21:40:36;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:37;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:37;0080;pbs_mom;Job;6015500.Prophet;copy file request received
05/30/2024 21:40:38;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:38;0100;pbs_mom;Job;6015500.Prophet;staged 2 items out over 0:00:01
05/30/2024 21:40:39;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:39;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:39;0008;pbs_mom;Job;6015500.Prophet;no active tasks
05/30/2024 21:40:39;0080;pbs_mom;Job;6015500.Prophet;delete job request received
05/30/2024 21:40:39;0080;pbs_python;Hook;pbs_python;/sys/fs/cgroup/net_cls,net_prio/pbspro.slice/pbspro-6015500.Prophet.slice: Path removed successfully
05/30/2024 21:40:39;0080;pbs_python;Hook;pbs_python;/sys/fs/cgroup/perf_event/pbspro.slice/pbspro-6015500.Prophet.slice: Path removed successfully
05/30/2024 21:40:40;0008;pbs_mom;Job;6015500.Prophet;kill_job
Logs of normal execution :
05/30/2024 21:45:25;0100;pbs_python;Hook;pbs_python;create_job: Creating directory /sys/fs/cgroup/cpuset/pbspro.slice/pbspro-6015523.Prophet.slice/
05/30/2024 21:45:26;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:26;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:26;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:28;0100;pbs_python;Hook;pbs_cgroups;env orig: PATH=/bin:/usr/bin,PBS_O_SYSTEM=Linux,PBS_O_SHELL=/bin/sh,PBS_O_HOME=/home/seradmin,PBS_O_LOGNAME=seradmin,PBS_O_WORKDIR=/home/seradmin,PBS_O_LANG=en_US.UTF-8,PBS_O_PATH=/usr/bin:/bin,CONTAINER_IMAGE=nil_22_r_v7,PBS_O_QUEUE=Hercules,PBS_O_HOST=apprentice,HOME=/home/seradmin,LOGNAME=seradmin,PBS_JOBNAME=realtime-forecast,PBS_JOBID=6015523.Prophet,PBS_QUEUE=Hercules,SHELL=/bin/bash,USER=seradmin,PBS_JOBCOOKIE=123CE72F4B76820E12B225352A0DE076,PBS_NODENUM=0,PBS_TASKNUM=1,PBS_MOMPORT=15003,OMP_NUM_THREADS=8,NCPUS=8,PBS_NODEFILE=/var/spool/pbs/aux/6015523.Prophet,TMPDIR=/var/tmp/pbs.6015523.Prophet,PBS_JOBDIR=/home/seradmin,PBS_ENVIRONMENT=PBS_BATCH,ENVIRONMENT=BATCH
05/30/2024 21:45:28;0100;pbs_python;Hook;pbs_cgroups;env new: PBS_O_SYSTEM=Linux,PBS_JOBCOOKIE=123CE72F4B76820E12B225352A0DE076,PBS_JOBNAME=realtime-forecast,PBS_O_HOME=/home/seradmin,PBS_O_HOST=apprentice,PBS_NODENUM=0,PBS_O_LOGNAME=seradmin,PBS_O_SHELL=/bin/sh,PBS_O_LANG=en_US.UTF-8,USER=seradmin,PATH=/bin:/usr/bin,PBS_MOMPORT=15003,HOME=/home/seradmin,TMPDIR=/var/tmp/pbs.6015523.Prophet,ENVIRONMENT=BATCH,PBS_NODEFILE=/var/spool/pbs/aux/6015523.Prophet,SHELL=/bin/bash,PBS_ENVIRONMENT=PBS_BATCH,OMP_NUM_THREADS=8,NCPUS=8,PBS_JOBDIR=/home/seradmin,PBS_O_QUEUE=Hercules,PBS_JOBID=6015523.Prophet,PBS_O_WORKDIR=/home/seradmin,PBS_O_PATH=/usr/bin:/bin,CUDA_VISIBLE_DEVICES=,CONTAINER_IMAGE=nil_22_r_v7,LOGNAME=seradmin,PBS_TASKNUM=1,PBS_QUEUE=Hercules
05/30/2024 21:45:29;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:29;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:29;0100;pbs_python;Hook;PBS_hpc_container;argv new: /opt/pbs/sbin/pbs_container 6015523.Prophet 1 /opt/pbs exec 6015523.Prophet /bin/bash -c /var/spool/pbs/mom_priv/jobs/6015523.Prophet.SC
05/30/2024 21:45:29;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:29;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:30;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:30;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:45:30;0008;pbs_mom;Job;6015523.Prophet;Started, pid = 111971
05/30/2024 21:45:32;0008;pbs_python;Job;6015523.Prophet;_execjob_attach_handler: Attaching PID 112279
05/30/2024 21:45:32;0008;pbs_mom;Job;6015523.Prophet;pid 112279 sid 112279 cmd pbs_sleep attached as task 00000002
05/30/2024 21:45:33;0008;pbs_python;Job;6015523.Prophet;_execjob_attach_handler: Attaching PID 112468
05/30/2024 21:45:33;0008;pbs_mom;Job;6015523.Prophet;pid 112468 sid 112468 cmd 6015523.Pro attached as task 00000003
05/30/2024 21:45:33;0008;pbs_python;Job;6015523.Prophet;_execjob_attach_handler: Attaching PID 112486
05/30/2024 21:45:33;0001;pbs_mom;Job;6015523.Prophet;tm_attach: sid 112468 already attached
05/30/2024 21:46:04;0008;pbs_mom;Job;6015523.Prophet;no active process for task 00000003
05/30/2024 21:46:04;0080;pbs_mom;Job;6015523.Prophet;task 00000001 terminated
05/30/2024 21:46:04;0008;pbs_mom;Job;6015523.Prophet;Terminated
05/30/2024 21:46:04;0100;pbs_mom;Job;6015523.Prophet;task 00000001 cput= 0:00:02
05/30/2024 21:46:04;0100;pbs_mom;Job;6015523.Prophet;task 00000002 cput= 0:00:00
05/30/2024 21:46:04;0100;pbs_mom;Job;6015523.Prophet;task 00000003 cput= 0:00:29
05/30/2024 21:46:04;0008;pbs_mom;Job;6015523.Prophet;kill_job
05/30/2024 21:46:04;0080;pbs_mom;Job;6015523.Prophet;task 00000002 force exited
05/30/2024 21:46:04;0100;pbs_mom;Job;6015523.Prophet;Prophet cput= 0:00:31 mem=481672kb
05/30/2024 21:46:10;0008;pbs_mom;Job;6015523.Prophet;no active process for task 00000002
05/30/2024 21:46:10;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/freezer/pbspro.slice/pbspro-6015523.Prophet.slice
05/30/2024 21:46:10;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/cpuset/pbspro.slice/pbspro-6015523.Prophet.slice
05/30/2024 21:46:10;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/hugetlb/pbspro.slice/pbspro-6015523.Prophet.slice
05/30/2024 21:46:15;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:46:15;0100;pbs_mom;Job;6015523.Prophet;Obit sent
05/30/2024 21:46:20;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:46:20;0080;pbs_mom;Job;6015523.Prophet;copy file request received
05/30/2024 21:46:20;0100;pbs_mom;Job;6015523.Prophet;staged 2 items out over 0:00:00
05/30/2024 21:46:21;0008;pbs_mom;Job;6015523.Prophet;no active tasks
05/30/2024 21:46:21;0080;pbs_mom;Job;6015523.Prophet;delete job request received
05/30/2024 21:46:23;0080;pbs_python;Hook;pbs_python;/sys/fs/cgroup/net_cls,net_prio/pbspro.slice/pbspro-6015523.Prophet.slice: Path removed successfully
05/30/2024 21:46:23;0080;pbs_python;Hook;pbs_python;/sys/fs/cgroup/perf_event/pbspro.slice/pbspro-6015523.Prophet.slice: Path removed successfully
05/30/2024 21:46:23;0008;pbs_mom;Job;6015523.Prophet;kill_job