# Job completed but no output

**URL:** <https://community.openpbs.org/t/job-completed-but-no-output/3775>\
**Category:** Users/Site Administrators\
**Created:** [February 6, 2024, 11:16am UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775 "2024-02-06T11:16:55Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 11:16am UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/1 "2024-02-06T11:16:55Z")

</div>

Hi all,  
After running the following job script, no `output file` or `error file` created.

```auto
#!/bin/bash

### Thejob name

#PBS -N moose_test

### The number of node and processors per node

#PBS -l nodes=node03:ppn=56

### The maximum time for job running

#PBS -l walltime=48:00:00

### The standard output and error

#PBS -j oe

### The queue for job running

#conda activate moose

#

cd $PBS_O_WORKDIR

NSLOTS=`cat ${PBS_NODEFILE} | wc -l`

echo "This jobs is "$PBS_JOBID@$PBS_QUEUE

uniq -c $PBS_NODEFILE |awk '{print $2":"$1}'

#

date

time mpiexec -n 56 ./*opt -i ./test/tests/kernels/simple_diffusion/simple_diffusion.i

date

exit 0

```

When I run ` ssh node03` , I don’t need password.

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 11:53am UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/2 "2024-02-06T11:53:03Z")

</div>

Does the job run, i.e. what’s the output of qstat -a and qstat -xf, also tracejob jobid ?

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 12:13pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/3 "2024-02-06T12:13:45Z")

</div>

> [@CGully](#):
>
> qstat -a

Hi, here lists the info of the command you mentioned:

qstat -a

```auto
node01: 
                                                                                  Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
----------------------- ----------- -------- ---------------- ------ ----- ------ --------- --------- - ---------
867.node01 DengChaoQun batch moose_test 29802 1 56 -- 48:00:00 C --

```

qstat -xf

```auto
<?xml version="1.0"?>
<Data><Job><Job_Id>867.node01</Job_Id><Job_Name>moose_test</Job_Name><Job_Owner>DengChaoQun@node01</Job_Owner><resources_used><cput>00:02:32</cput><vmem>0kb</vmem><walltime>00:00:08</walltime><mem>0kb</mem><energy_used>0</energy_used></resources_used><job_state>C</job_state><queue>batch</queue><server>node01</server><Checkpoint>u</Checkpoint><ctime>1707221317</ctime><Error_Path>node01:/home/DengChaoQun/projects/bees2024/moose_test.e867</Error_Path><exec_host>node03/0-55</exec_host><Hold_Types>n</Hold_Types><Join_Path>oe</Join_Path><Keep_Files>n</Keep_Files><Mail_Points>a</Mail_Points><mtime>1707221329</mtime><Output_Path>node01:/home/DengChaoQun/projects/bees2024/moose_test.o867</Output_Path><Priority>0</Priority><qtime>1707221317</qtime><Rerunable>True</Rerunable><Resource_List><nodes>node03:ppn=56</nodes><walltime>48:00:00</walltime><nodect>1</nodect></Resource_List><session_id>29802</session_id><Variable_List>PBS_O_QUEUE=batch,PBS_O_HOME=/home/DengChaoQun,PBS_O_LOGNAME=DengChaoQun,PBS_O_PATH=/home/DengChaoQun/mpich-4.0.2/install/bin:/home/DengChaoQun/gcc-13.1.0/gcc-install/bin:/opt/torque/bin:/opt/torque/sbin:/home/DengChaoQun/.vscode-server/bin/8b3775030ed1a69b13e4f4c628c612102e30a681/bin/remote-cli:/home/DengChaoQun/mpich-4.0.2/install/bin:/home/DengChaoQun/gcc-13.1.0/gcc-install/bin:/opt/rh/devtoolset-9/root/usr/bin:/opt/torque/bin:/opt/torque/sbin/opt/intel/oneapi/vtune/2022.2.0/bin64:/opt/intel/oneapi/vpl/2022.1.0/bin:/opt/intel/oneapi/mpi/2021.6.0/libfabric/bin:/opt/intel/oneapi/mpi/2021.6.0/bin:/opt/intel/oneapi/mkl/2022.1.0/bin/intel64:/opt/intel/oneapi/itac/2021.6.0/bin:/opt/intel/oneapi/inspector/2022.1.0/bin64:/opt/intel/oneapi/dpcpp-ct/2022.1.0/bin:/opt/intel/oneapi/dev-utilities/2021.6.0/bin:/opt/intel/oneapi/debugger/2021.6.0/gdb/intel64/bin:/opt/intel/oneapi/compiler/2022.1.0/linux/lib/oclfpga/bin:/opt/intel/oneapi/compiler/2022.1.0/linux/bin/intel64:/opt/intel/oneapi/compiler/2022.1.0/linux/bin:/opt/intel/oneapi/clck/2021.6.0/bin/intel64:/opt/intel/oneapi/advisor/2022.1.0/bin64:/opt/torque/bin:/opt/torque/sbin:/home/DengChaoQun/mpich-4.0.2/install/bin:/home/DengChaoQun/gcc-13.1.0/gcc-install/bin:/home/DengChaoQun/miniforge/envs/moose/bin:/home/DengChaoQun/miniforge/condabin:/opt/torque/bin:/opt/torque/sbin:/usr/lib64/qt-3.3/bin:/home/DengChaoQun/perl5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/DengChaoQun/.local/bin:/home/DengChaoQun/bin:/home/DengChaoQun/miniforge/envs/moose/wasp/bin,PBS_O_MAIL=/var/spool/mail/DengChaoQun,PBS_O_SHELL=/bin/bash,PBS_O_LANG=en_US.UTF-8,PBS_O_WORKDIR=/home/DengChaoQun/projects/bees2024,PBS_O_HOST=node01,PBS_O_SERVER=node01</Variable_List><euser>DengChaoQun</euser><egroup>DengChaoQun</egroup><queue_type>E</queue_type><sched_hint>Unable to copy files back - please see the mother superior&apos;s log for exact details.</sched_hint><comment>Job started on Tue Feb 06 at 20:08</comment><etime>1707221317</etime><exit_status>0</exit_status><submit_args>run_moose.pbs</submit_args><start_time>1707221317</start_time><start_count>1</start_count><fault_tolerant>False</fault_tolerant><comp_time>1707221329</comp_time><job_radix>0</job_radix><total_runtime>12.143389</total_runtime><submit_host>node01</submit_host><init_work_dir>/home/DengChaoQun/projects/bees2024</init_work_dir><request_version>1</request_version></Job></Data>

```

tracejob 867

```auto
/var/spool/torque/server_priv/accounting/20240206: Permission denied
/var/spool/torque/mom_logs/20240206: No matching job records located

Job: 867.node01

02/06/2024 20:08:37.607 S enqueuing into batch, state 1 hop 1
02/06/2024 20:08:37.788 S Job Modified at request of root@node01
02/06/2024 20:08:37.821 L Job Run
02/06/2024 20:08:37.788 S Job Run at request of root@node01
02/06/2024 20:08:37.821 S Not sending email: User does not want mail of this type.
02/06/2024 20:08:49.931 S Not sending email: User does not want mail of this type.
02/06/2024 20:08:49.932 S Exit_status=0 resources_used.cput=152 resources_used.vmem=0kb resources_used.walltime=00:00:08 resources_used.mem=0kb resources_used.energy_used=0

```

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 12:23pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/4 "2024-02-06T12:23:25Z")

</div>

Think this could be why you are not seeing the logs

> [@echo1115](#):
>
> `>Unable to copy files back - please see the mother superior&apos;s log for exact details.</sched_hint><comment>Job started on Tue Feb 06 at 20:`

Are you able to view the mom\_logs on node03, may show the reason why it cannot copy the files back.

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 12:37pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/5 "2024-02-06T12:37:52Z")

</div>

here are somo info from the mom\_logs file:

```auto
02/06/2024 20:10:06.995;01; pbs_mom.3769;Job;TMomFinalizeJob3;job 867.node01 started, pid = 29802
02/06/2024 20:10:14.878;128; pbs_mom.3769;Job;867.node01;scan_for_terminated: job 867.node01 task 1 terminated, sid=29802
02/06/2024 20:10:14.878;08; pbs_mom.3769;Job;867.node01;job was terminated
02/06/2024 20:10:14.878;128; pbs_mom.3769;Svr;preobit_preparation;top
02/06/2024 20:10:19.102;128; pbs_mom.3769;Job;867.node01;obit sent to server
02/06/2024 20:10:19.107;128; pbs_mom.3811;Job;867.node01;removed job script
02/06/2024 20:11:32.391;02; pbs_mom.3769;Svr;pbs_mom;Torque Mom Version = 6.1.1.1, loglevel = 0
02/06/2024 20:16:32.864;02; pbs_mom.3769;Svr;pbs_mom;Torque Mom Version = 6.1.1.1, loglevel = 0
02/06/2024 20:21:37.863;02; pbs_mom.3769;Svr;pbs_mom;Torque Mom Version = 6.1.1.1, loglevel = 0
02/06/2024 20:26:42.933;02; pbs_mom.3769;Svr;pbs_mom;Torque Mom Version = 6.1.1.1, loglevel = 0
02/06/2024 20:31:47.887;02; pbs_mom.3769;Svr;pbs_mom;Torque Mom Version = 6.1.1.1, loglevel = 0

```

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 1:53pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/6 "2024-02-06T13:53:32Z")

</div>

On node03 do you see anything in /var/spool/pbs/undelivered/

On node03 can you touch a file in the $PBS\_O\_WORKDIR?  
On node03 is scp installed?

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 2:22pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/7 "2024-02-06T14:22:07Z")

</div>

Thanks for your reply!

> [@CGully](#):
>
> On node03 do you see anything in /var/spool/pbs/undelivered/

On node03, I see the the output file `867.node01.OU` in /var/spool/pbs/undelivered/ , but no error file.

> [@CGully](#):
>
> On node03 is scp installed?

Since other people can get output file by using the same script, I think the scp is installed on node03.

> [@CGully](#):
>
> On node03 can you touch a file in the $PBS\_O\_WORKDIR?

- when I run `echo $PBS_O_WORKDIR` on node03, it returns nothing.
- and when I run `touch $PBS_O_WORKDIR/test_file.txt` , it shows:  
`touch: cannot touch ‘/test_file.txt’: Permission denied`

So maybe this causes the problem.

I am a beginner in PBS scripting. Can you help me solve this problem？ Thanks a lot!

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 2:26pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/8 "2024-02-06T14:26:51Z")

</div>

As you have specified the -j oe 867.node01.OU will contain standard error.

oe Standard error and standard output are merged into standard output.

Is there anything useful in this file?

You will have to know already the location of $PBS\_O\_WORKDIR, i.e. the directory that you are submitting from on the server.

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 2:51pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/9 "2024-02-06T14:51:36Z")

</div>

> [@CGully](#):
>
> Is there anything useful in this file?

it contains the output info of my running case, but nothing useful to solve my problem

> [@echo1115](#):
>
> and when I run `touch $PBS_O_WORKDIR/test_file.txt` , it shows:  
> `touch: cannot touch ‘/test_file.txt’: Permission denied`

Maybe I make a mistake here and caused a misunderstanding, on node03, when I run `touch test_file.txt` instead of `touch $PBS_O_WORKDIR/test_file.txt` , it can create a file named `test_file.txt` .

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 2:58pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/10 "2024-02-06T14:58:29Z")

</div>

Is that in the same directory that you submitted from i.e. /home/username/projects/bees2024/ ?

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 3:00pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/11 "2024-02-06T15:00:01Z")

</div>

yes, the same directory and on node03 now

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 3:12pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/12 "2024-02-06T15:12:08Z")

</div>

If you remove the line :

> [@echo1115](#):
>
> ```auto
> time mpiexec -n 56 ./*opt -i ./test/tests/kernels/simple_diffusion/simple_diffusion.i
> 
> ```

Does it then output properly?  
What are others running?

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 3:21pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/13 "2024-02-06T15:21:41Z")

</div>

> [@CGully](#):
>
> If you remove the line :
> 
> > [@echo1115](#):
> >
> > ```auto
> > time mpiexec -n 56 ./*opt -i ./test/tests/kernels/simple_diffusion/simple_diffusion.i
> > 
> > ```
> 
> Does it then output properly?

I removed this line but the new output file is still in the directory /var/spool/pbs/undelivered/

Other people used the same script to run the same case, but his output file would be in the directory that he submitted from.

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 3:43pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/14 "2024-02-06T15:43:36Z")

</div>

You could try qsub -koed -o (output\_path) -e (error\_path) job.sh

Which should directly output standard out and error to the final destination.  
Any errors should be in /var/spool/torque/mom\_logs/ on node03.

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 3:54pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/15 "2024-02-06T15:54:31Z")

</div>

> [@CGully](#):
>
> You could try qsub -koed -o (output\_path) -e (error\_path) job.sh

Can I run this command on login node? or have to run on node03?

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 3:55pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/16 "2024-02-06T15:55:43Z")

</div>

On login node, where you normally qsub from.

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 4:18pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/17 "2024-02-06T16:18:19Z")

</div>

- I create a directory `for_out` in /home/username/projects/bees2024/, and run `qsub -koe -o ./for_out/out.OU -e ./for_out/error.ER run_moose.pbs`, but still no output files

- The job id is 873, but this time I can’t find 873 related files in the directory /var/spool/pbs/undelivered/

- Here is the info in /var/spool/torque/mom\_logs/ :

```auto
02/07/2024 00:12:10.903;01; pbs_mom.3769;Job;TMomFinalizeJob3;job 873.node01 started, pid = 40049
02/07/2024 00:12:18.662;128; pbs_mom.3769;Job;873.node01;scan_for_terminated: job 873.node01 task 1 terminated, sid=40049
02/07/2024 00:12:18.662;08; pbs_mom.3769;Job;873.node01;job was terminated
02/07/2024 00:12:18.662;128; pbs_mom.3769;Svr;preobit_preparation;top
02/07/2024 00:12:27.052;128; pbs_mom.3769;Job;873.node01;obit sent to server
02/07/2024 00:12:27.055;128; pbs_mom.3825;Job;873.node01;removed job script
02/07/2024 00:14:35.506;02; pbs_mom.3769;Svr;pbs_mom;Torque Mom Version = 6.1.1.1, loglevel = 0

```

---

<div class="post-metadata">

**Author:** ![CGully](https://avatars.discourse-cdn.com/v4/letter/c/e5b9ba/32.png) [@CGully](https://community.openpbs.org/u/CGully)\
**Post date:** [February 6, 2024, 4:27pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/18 "2024-02-06T16:27:31Z")

</div>

Could you try and use the full path rather than relative?  
i.e. -o /home/DengChaoQun/projects/bees2022/out.OU -e /home/DengChaoQun/projects/bees2022/err.ER

---

<div class="post-metadata">

**Author:** ![echo1115](https://avatars.discourse-cdn.com/v4/letter/e/7993a0/32.png) [@echo1115](https://community.openpbs.org/u/echo1115)\
**Post date:** [February 6, 2024, 4:36pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/19 "2024-02-06T16:36:18Z")

</div>

I tried the full path, but the result is the same.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [February 6, 2024, 7:02pm UTC](https://community.openpbs.org/t/job-completed-but-no-output/3775/20 "2024-02-06T19:02:55Z")

</div>

Could you please check whether you are using OpenPBSPro or torque.  
Could you please share the output of the below commands  
qstat --version  
qstat -Bf

[Next page](https://community.openpbs.org/t/job-completed-but-no-output/3775.md?page=2)