# Parallel job not starting correctly

**URL:** <https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062>\
**Category:** Users/Site Administrators\
**Created:** [April 14, 2022, 3:07pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062 "2022-04-14T15:07:59Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 14, 2022, 3:07pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/1 "2022-04-14T15:07:59Z")

</div>

Dear community,

I am facing an issue with OpenPBS v20.0 that I have issue to understand, so I will describe symptoms.

We are using a commercial code, ANSYS CFX which offers the possibility of performing distributed and shared memory parallel calculations. This tool officially supports PBS.

I setup a PBS server with 3 nodes. I verified that on each node ANSYS CFX is capable of running correctly with it shared memory option. It works perfectly on each node.

When I try to launch exactly the same case via PBS, the calculation starts correctly up until it reaches the moment when the parallel solver should be initiated. At that moment, the code freezes without any particular output. PBS waits that the walltime is exceeded to just kill the process…

No particular message can be found in is not giving any particular message:  
mom\_logs:  
04/14/2022 16:46:13;0100;pbs\_mom;Req;;Type 1 request received from root@xxx.xxx.xxx.xxx:15001, sock=10  
04/14/2022 16:46:13;0100;pbs\_mom;Req;;Type 3 request received from root@xxx.xxx.xxx.xxx:15001, sock=10  
04/14/2022 16:46:13;0100;pbs\_mom;Req;;Type 5 request received from root@xxx.xxx.xxx.xxx:15001, sock=10  
04/14/2022 16:46:13;0008;pbs\_mom;Job;194.Server1;Started, pid = 1342717  
04/14/2022 16:57:41;0008;pbs\_mom;Job;194.Server1;walltime 688 exceeded limit 600  
04/14/2022 16:57:41;0008;pbs\_mom;Job;194.Server1;kill\_job  
04/14/2022 16:57:41;0080;pbs\_mom;Job;194.Server1;task 00000001 terminated  
04/14/2022 16:57:51;0008;pbs\_mom;Job;194.Server1;kill\_job  
04/14/2022 16:57:51;0080;pbs\_mom;Job;194.Server1;task 00000001 force exited  
04/14/2022 16:57:51;0008;pbs\_mom;Job;194.Server1;Terminated  
04/14/2022 16:57:51;0100;pbs\_mom;Job;194.Server1;task 00000001 cput=00:01:21  
04/14/2022 16:57:51;0008;pbs\_mom;Job;194.Server1;kill\_job  
04/14/2022 16:57:51;0100;pbs\_mom;Job;194.Server1;Server2 cput=00:01:21 mem=3028896kb  
04/14/2022 16:57:51;0100;pbs\_mom;Job;194.Server1:Obit sent

Other tools are perfecly working… I facing this issue only with ANSYS CFX.

Have you already observe such behavior? Can you help me to understand the issue?

Thank you in advance.  
Best regards.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 15, 2022, 4:35pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/2 "2022-04-15T16:35:27Z")

</div>

> [@NomisTuo](#):
>
> 04/14/2022 16:57:41;0008;pbs\_mom;Job;194.Server1;walltime 688 exceeded limit 600

Please check this line, your job was killed due to insufficient walltime request for the job.  
It seems job has requested a walltime of 600 seconds. Please submit the job with more walltime or no walltime (if walltime is not defined , then it is 5 years)

---

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 16, 2022, 6:19pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/3 "2022-04-16T18:19:21Z")

</div>

Dean Adarsh,

Thank you for your reply, but as written in my original post, my job freezes way before reaching my walltime value. My job is finally terminated by PBS thanks to the specified walltime value…

Running exactly the same case (requesting the same amount of cores,…) on a node without passing by PBS takes about half of the requested walltime.

Do you have any other idea?

Best regards.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 17, 2022, 1:32pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/4 "2022-04-17T13:32:24Z")

</div>

Thank you @NomisTuo .

1. Please check and share your job script
2. Can you run your script without using PBS and whether it runs or hangs ?
3. can you check the stdout and stderr file of the job that is deleted by the walltime
4. when the job is scheduled on the compute node , do you see the processes related to this job

---

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 20, 2022, 6:23am UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/5 "2022-04-20T06:23:05Z")

</div>

Dear Ararsh,

Here are the requested info:

1. My job script:  
#!/bin/bash -l

#PBS -l walltime=00:10:00  
#PBS -l nodes=1:ppn=36  
#PBS -q TH  
#PBS -o PBS.out  
#PBS -e PBS.err

export LD\_LIBRARY\_PATH=/nfs/soft/ansys\_inc/v211/licensingclient/linx64/lib/usr/lib64:/opt/pbs/lib

cd /nfs/users/test/IEP\_P3-karnak2-72over96cpus

/nfs/soft/ansys\_inc/v211/CFX/bin/cfx5solve -batch -def “CFX\_25.def” -fullname “IEP\_P3\_PBS” -size 1.2 -parallel -part 36 -start-method “Open MPI Local Parallel”

1. The script works perfectly when it is used “interactively” on each node.

2. The \*out and \*err files are obtained from my “home” folder. The error file contains this message:  
=\>\> PBS: job killed: walltime 688 exceeded limit 600  
A fatal error has occurred in cfx5solve:

cfx5solve was killed by the user.

1. The job starts normally on the computational node. It freezes at the moment when it should initiate its parallel computation. (The first part of the tool is purely scalar).

I hope those information will help you to understand the issue… I also contacted the software vendors. They are suspecting that an “environment variable” is interferring.

Thank you in advance.  
Best regards

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 20, 2022, 4:03pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/6 "2022-04-20T16:03:15Z")

</div>

> [@NomisTuo](#):
>
> 1. #!/bin/bash -l
> 
> #PBS -l walltime=00:10:00  
> #PBS -l nodes=1:ppn=36  
> #PBS -q TH  
> #PBS -o PBS.out  
> #PBS -e PBS.err
> 
> export LD\_LIBRARY\_PATH=/nfs/soft/ansys\_inc/v211/licensingclient/linx64/lib/usr/lib64:/opt/pbs/lib
> 
> cd /nfs/users/test/IEP\_P3-karnak2-72over96cpus
> 
> /nfs/soft/ansys\_inc/v211/CFX/bin/cfx5solve -batch -def “CFX\_25.def” -fullname “IEP\_P3\_PBS” -size 1.2 -parallel -part 36 -start-method “Open MPI Local Parallel”

Can you try the below script and update the license paths, make sure the inputfile and this pbsscript is in a shared location accessible by all the nodes

> #PBS -N pbs-cfx-script  
> #PBS -l select=1:ncpus=36:mpiprocs=36  
> #PBS -q TH  
> #PBS -o PBS.out  
> #PBS -e PBS.err
> 
> cd $PBS\_O\_WORKDIR  
> env  
> echo “======================”  
> input="CFX\_25.def”  
> export LD\_LIBRARY\_PATH=/nfs/soft/ansys\_inc/v211/licensingclient/linx64/lib/usr/lib64:/opt/pbs/lib  
> export PATH=/nfs/soft/ansys\_inc/v211/CFX/bin  
> export SERVER=7252@license-server  
> export ANSYSLI\_SERVERS=7253@license-server
> 
> hostlist=`cat $PBS_NODEFILE | awk 'BEGIN {getline; printf "g"$1}; {printf ",g"$1}; END {printf "\n"}'`
> 
> nParts=$(cat $PBS\_NODEFILE|wc -w)  
> hostlist=$(cat $PBS\_NODEFILE|tr ‘\n’ ‘,’)  
> time /nfs/soft/ansys\_inc/v211/CFX/bin/cfx5solve -batch -def ${input} -part $nParts -par-dist ${hostlist} -start-method “HP MPI Distributed Parallel”  
> status=$?  
> exit $status

Thank you for sharing the details, please try the above and check the stdout and stderr file.

---

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 25, 2022, 7:40am UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/7 "2022-04-25T07:40:15Z")

</div>

Dear adarsh,

Thank you for this submission script. I had to update just one line in order to get things “working”:

```auto
export PATH=/nfs/soft/ansys_inc/v211/CFX/bin

```

changed into

```auto
export PATH=/nfs/soft/ansys_inc/v211/CFX/bin:$PATH

```

In order to have all conventionnal tools easily accessible (awk, cat, …). I also add a walltime limit in order to avoid any “forever” job issue.

Unfortunately, the behaviour of the job is the same as before. The CFX run starts normally untill it reaches the point where the parellel processes need to start. At that moment CFX just freezes…

Here are the content of the files:  
stdout:

```auto
PBS_ENVIRONMENT=PBS_BATCH
LD_LIBRARY_PATH=/nfs/soft/ansys_inc/v211/licensingclient/linx64/lib/usr/lib64:/opt/pbs/lib
PBS_O_LANG=en_US.UTF-8
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
HOSTNAME=SERVER2
OLDPWD=/nfs/users/verdebs
PBS_O_HOME=/nfs/users/verdebs
PBS_JOBID=207.SERVER1
ENVIRONMENT=BATCH
PBS_JOBNAME=pbs-cfx-script
NCPUS=36
PBS_O_PATH=/usr/share/Modules/bin:.:/opt/pbs/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/nfs/users/verdebs/.local/bin:/nfs/users/verdebs/bin
which_declare=declare -f
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
PBS_O_WORKDIR=/nfs/users/verdebs/IEP_P3-72over96cpus
USER=verdebs
AWP_ROOT211=/nfs/soft/ansys_inc/v211/
PBS_NODEFILE=/var/spool/pbs/aux/207.SERVER1
PBS_TASKNUM=1
PWD=/nfs/users/verdebs/IEP_P3-72over96cpus
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
HOME=/nfs/users/verdebs
PBS_MOMPORT=15003
XDG_DATA_DIRS=/nfs/users/verdebs/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
PBS_JOBCOOKIE=7E4DDBBD716F47971EB1A94072A4AD35
PBS_O_SHELL=/bin/bash
TMPDIR=/var/tmp/pbs.207.SERVER1
LOADEDMODULES=
PBS_O_QUEUE=TH
MAIL=/var/spool/mail/verdebs
SHELL=/bin/bash
SHLVL=2
PBS_O_HOST=server1.d10.tes.local
PBS_O_SYSTEM=Linux
MANPATH=::/opt/pbs/share/man
PBS_O_LOGNAME=verdebs
PBS_NODENUM=0
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles
PBS_JOBDIR=/nfs/users/verdebs
LOGNAME=verdebs
MODULEPATH_modshare=/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1
PATH=/usr/share/Modules/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/nfs/users/verdebs/.local/bin:/nfs/users/verdebs/bin
PBS_QUEUE=TH
MODULESHOME=/usr/share/Modules
HISTSIZE=1000
PBS_O_MAIL=/var/spool/mail/verdebs
OMP_NUM_THREADS=36
LESSOPEN=||/usr/bin/lesspipe.sh %s
BASH_FUNC_which%%=() { ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
BASH_FUNC_module%%=() { unset _mlshdbg;
 if ["${MODULES_SILENT_SHELL_DEBUG:-0}" = '1']; then
 case "$-" in
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 unset _mlre _mlIFS;
 if [-n "${IFS+x}"]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if ["${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [-n "`eval 'echo ${'$_mlv'+x}'`"]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [-n "${_mlre:-}"]; then
 eval `eval ${_mlre} /usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash '"$@"'`;
 else
 eval `/usr/bin/tclsh /usr/share/Modules/libexec/modulecmd.tcl bash "$@"`;
 fi;
 _mlstatus=$?;
 if [-n "${_mlIFS+x}"]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 unset _mlre _mlv _mlrv _mlIFS;
 if [-n "${_mlshdbg:-}"]; then
 set -$_mlshdbg;
 fi;
 unset _mlshdbg;
 return $_mlstatus
}
BASH_FUNC_switchml%%=() { typeset swfound=1;
 if ["${MODULES_USE_COMPAT_VERSION:-0}" = '1']; then
 typeset swname='main';
 if [-e /usr/share/Modules/libexec/modulecmd.tcl]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [-e /usr/share/Modules/libexec/modulecmd-compat]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [$swfound -eq 0]; then
 echo "Switching to Modules $swname version";
 source /usr/share/Modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}
BASH_FUNC_ml%%=() { module ml "$@"
}
_=/bin/env
======================
g

```

And the stderr file:

```auto
/var/spool/pbs/mom_priv/jobs/207.SERVER1.SC: line 16: /var/spool/pbs/aux/207.SERVER1: Permission denied
A fatal error has occurred in cfx5solve:

cfx5solve was killed by the user.

```

I hope it will help you to understand the issue.  
Thanks a lot for your efforts!

Have a nice day.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 25, 2022, 11:06am UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/8 "2022-04-25T11:06:01Z")

</div>

Thank you @NomisTuo for this information and analysis.  
It seems the issue is more related to CFX then the workload manager.  
It would better if you can contact the vendor support with the above information and test cfx batch command line without using PBS Pro ( do not user interactive here, as the GUI would setup the enviroment variables automatically, which might have been missed while running a batch job). They might be able to guide you through this easily.

> [@NomisTuo](#):
>
> ```auto
> /var/spool/pbs/mom_priv/jobs/207.SERVER1.SC: line 16: /var/spool/pbs/aux/207.SERVER1: Permission denied
> A fatal error has occurred in cfx5solve:
> 
> ```

application launch issue, might have crashed soon after it was launched for batch mode

---

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 25, 2022, 11:11am UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/9 "2022-04-25T11:11:10Z")

</div>

Hello adarsh,

Thank you for your reply, but CFX runs perfectly when I try to launch it interactively on each node…

How it could come that CFX works perfectly without using PBS and it stops behaving correctly when it is initiated by PBS?

Best regards.

---

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 25, 2022, 2:13pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/10 "2022-04-25T14:13:32Z")

</div>

Dear adarsh,

I keep on trying to understand the issue I am facing… I just notice using a “Interactive” job that the $PBS\_NODEFILE variable shows exactly the same value on each line…

```auto
[verdebs@SERVER2 ~]$ cat $PBS_NODEFILE
SERVER2
SERVER2

```

I would have more expect something like

```auto
[verdebs@SERVER2 ~]$ cat $PBS_NODEFILE
SERVER2*01
SERVER2*02

```

This seems odd to me but maybe I am wrong?

Best regards.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 25, 2022, 2:37pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/11 "2022-04-25T14:37:41Z")

</div>

```auto
> qsub -l select=2:ncpus=2 -l place=free -I 
> cat $PBS_NODEFILE
> server2
> server2
> 
> qsub -l select=2:ncpus=2:mpiprocs=2 -l place=free -I 
> cat $PBS_NODEFILE
> server2
> server2
> server2
> server2
> 
> #if in case you have two nodes server1 and server2
> qsub -l select=2:ncpus=2:mpiprocs=2 -l place=scatter -I 
> cat $PBS_NODEFILE
> server1
> server1
> server2
> server2

```

> [@NomisTuo](#):
>
> This seems odd to me but maybe I am wrong?

```auto
> 
> You can re-write the hostfile to your requirment and use it in the script
> total_cores=`cat $PBS_NODEFILE | wc -l `
> total_chunks=`cat $PBS_NODEFILE | uniq | wc -l `
> proc_per_host=$(expr $total_cores / $total_chunks)
> for i in `cat $PBS_NODEFILE | uniq ` ; do echo $i:$pro_per_host >> hosts.txt ; done 
> #Point your -host argument to the hosts.txt instead of $PBS_NODEFILE

```

---

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 26, 2022, 6:37am UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/12 "2022-04-26T06:37:03Z")

</div>

Dear adarsh,

Thank you for your reply.

Than I think that the PBS\_NODEFILE is adequately populated by the system. Just one last question regarding this PBS\_NODEFILE and my PBS configuration.

My server is composed of 3 nodes presenting the definition archetype (where X=1, 2 or 3):  
Mom = SERVERX.domain.localdomain  
ntype = PBS  
state = free  
pcpus = 96  
resources\_available.arch = linux  
resources\_available.host = serverX  
resources\_available.mem = 394561752kb  
resources\_available.ncpus = 96  
resources\_available.vnode = SERVERX  
resources\_assigned.accelerator\_memory = 0kb  
resources\_assigned.hbmem = 0kb  
resources\_assigned.mem = 0kb  
resources\_assigned.naccelerators = 0  
resources\_assigned.ncpus = 0  
resources\_assigned.vmem = 0kb  
resv\_enable = True  
sharing = default\_shared  
license = l  
last\_state\_change\_time = Mon Mar 28 17:55:34 2022  
last\_used\_time = Thu Apr 14 16:26:34 2022

Within my PBS\_NODEFILE, I get the list containing SERVERX.domain.localdomain repeated the adequate amont of time. Is this normal or should I get either the “host”, ie. serverX, or the “vnode”, ie. SERVERX repeated?

I am just wondering if CFX is experiencing problem due to CFX or due to my PBS setup… Is there some typical sanity check that should be performed on a newly setup PBS installation?

Thank you in advance.  
Best regards.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 26, 2022, 7:17am UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/13 "2022-04-26T07:17:00Z")

</div>

> [@NomisTuo](#):
>
> Within my PBS\_NODEFILE, I get the list containing SERVERX.domain.localdomain repeated the adequate amont of time. Is this normal or should I get either the “host”, ie. serverX, or the “vnode”, ie. SERVERX repeated?

If your /etc/hosts and DNS resolves serverx.domain.localdomain correctly to a static IP address , which is the IP of the compute node(s), then it is not a problem. Otherwise, you can re-write PBS\_NODEFILE with shorthostname to a hosts.txt file and use host.txt.

`for i in $PBS_NODE_FILE;do echo i$i | cut -d'.' -f1 >> hosts.txt ; done`

> [@NomisTuo](#):
>
> I am just wondering if CFX is experiencing problem due to CFX or due to my PBS setup… Is there some typical sanity check that should be performed on a newly setup PBS installation?

It is a common issue,"PBS is a messenger and do not shoot the messenger " stuck between applicaiton and system issues. We always recommend to test the batch command line along with the enviroment variables required to run the applicaiton without using PBS First. If it runs successfully on the compute nodes, then integrate that application in a PBS Script.

Any issues with the PBS Pro: you can increase the log level on the server logs, scheduler logs and mom logs and find out the issue in these log files. Any system related check in the /var/log/messages of the respective hosts.

---

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 26, 2022, 8:07am UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/14 "2022-04-26T08:07:30Z")

</div>

> [@adarsh](#):
>
> It is a common issue,"PBS is a messenger and do not shoot the messenger " stuck between applicaiton and system issues. We always recommend to test the batch command line along with the enviroment variables required to run the applicaiton without using PBS First. If it runs successfully on the compute nodes, then integrate that application in a PBS Script.

In this case, CFX runs perfectly using batch command line. The issue is appearing when PBS is used to submit the job…

> [@adarsh](#):
>
> Any issues with the PBS Pro: you can increase the log level on the server logs, scheduler logs and mom logs and find out the issue in these log files. Any system related check in the /var/log/messages of the respective hosts.

How may I increase the log level of the different players (server, scheduler and mom)?

Thank you in advance for your help

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 26, 2022, 2:44pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/15 "2022-04-26T14:44:28Z")

</div>

> [@NomisTuo](#):
>
> In this case, CFX runs perfectly using batch command line. The issue is appearing when PBS is used to submit the job…

Please share us the .o and .e file that is created after the job has failed, share the application log file  
Please share us the batch command line and environment variables used to run the cfx job successful without using pbs.

> [@NomisTuo](#):
>
> How may I increase the log level of the different players (server, scheduler and mom)?

server logs: qmgr: set server log\_events=2047 # default is 511  
sched logs: qmgr : set schedlog\_events\*\* = 4095 # default is 767  
mom logs : edit $PBS\_HOME/mom\_priv/config and add $logevents 0xffffffff ( kill -HUP \<pid of pbs\_mom)

Please note you have to revert back to default log levels, otherwise the harddisk space would be consumed in short amount of time.

---

<div class="post-metadata">

**Author:** ![NomisTuo](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nomistuo/32/516_2.png) [@NomisTuo](https://community.openpbs.org/u/NomisTuo)\
**Post date:** [April 27, 2022, 7:46am UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/16 "2022-04-27T07:46:00Z")

</div>

Dear adarsh,

Here is the .e file created by the job (as mentionned before, it is killed due to the exceeded walltime):

```auto
=>> PBS: job killed: walltime 151 exceeded limit 120
A fatal error has occurred in cfx5solve:

cfx5solve was killed by the user.

```

The application log file (in this case, the CFX out file) is completely normal. It is just that like the process stops when it reaches the “parallel” phase of the process:

```auto

 +--------------------------------------------------------------------+
 | CPU Time Requirements of Partitioner |
 +--------------------------------------------------------------------+

 Preparations 1.21E-02 2.7 %
 Low-level Mesh Partitioning 1.65E-03 0.4 %
 File Reading 1.21E-01 27.3 %
 Partition Smoothing 6.82E-03 1.5 %
 Topology - Domain Interface 9.00E-06 0.0 %
 Topology - Global 4.25E-04 0.1 %
 Topology - Element/Face/Patch 1.16E-03 0.3 %
 Topology - Vertex 1.03E-04 0.0 %
 Data Compression 7.60E-05 0.0 %
 Variable Updates 2.05E-03 0.5 %
 File Writing 3.66E-03 0.8 %
 Miscellaneous 2.93E-01 66.3 %
                                --------
 Total 4.42E-01

 +--------------------------------------------------------------------+
 | Job Information at End of Run |
 +--------------------------------------------------------------------+

 Host computer: SERVER2 (PID:2175371)

 Job finished: Wed Apr 27 08:55:20 2022

 Total wall clock time: 8.980E-01 seconds
             or: ( 0: 0: 0: 0.898 )
                 ( Days: Hours: Minutes: Seconds )

 +--------------------------------------------------------------------+
 | |
 | Solver |
 | |
 +--------------------------------------------------------------------+

 +--------------------------------------------------------------------+
 | A fatal error has occurred in cfx5solve: |
 | |
 | cfx5solve was killed by the user. |
 +--------------------------------------------------------------------+

```

Here is the environment variables used in batch

```auto
LD_LIBRARY_PATH=/nfs/soft/ansys_inc/v211/licensingclient/linx64/lib/usr/lib64:/opt/pbs/lib
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:
SSH_CONNECTION=10.42.10.163 33084 10.42.10.164 22
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
HOSTNAME=SERVER2
OLDPWD=/nfs/users/verdebs
which_declare=declare -f
XDG_SESSION_ID=473
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
USER=verdebs
AWP_ROOT211=/nfs/soft/ansys_inc/v211/
PWD=/nfs/users/verdebs/CFXtst
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
HOME=/nfs/users/verdebs
SSH_CLIENT=10.42.10.163 33084 22
XDG_DATA_DIRS=/nfs/users/verdebs/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
LOADEDMODULES=
SSH_TTY=/dev/pts/0
MAIL=/var/spool/mail/verdebs
TERM=xterm
SHELL=/bin/bash
SHLVL=1
MANPATH=::/opt/pbs/share/man
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles
LOGNAME=verdebs
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/2000/bus
XDG_RUNTIME_DIR=/run/user/2000
MODULEPATH_modshare=/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1
PATH=/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/nfs/users/verdebs/.local/bin:/nfs/users/verdebs/bin
MODULESHOME=/usr/share/Modules
HISTSIZE=1000
LESSOPEN=||/usr/bin/lesspipe.sh %s
_=/usr/bin/env

```

Here is the command used

```auto
cfx5solve -example StaticMixer -parallel -part 2 -par-local -start-method "Open MPI Local Parallel"

```

If it can help, here is the environment variables during the PBS run:

```auto
PBS_ENVIRONMENT=PBS_INTERACTIVE
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:
LD_LIBRARY_PATH=/nfs/soft/ansys_inc/v211/licensingclient/linx64/lib/usr/lib64:/opt/pbs/lib
PBS_O_LANG=en_US.UTF-8
SSH_CONNECTION=10.42.47.245 65371 10.42.10.163 22
MODULES_RUN_QUARANTINE=LD_LIBRARY_PATH LD_PRELOAD
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
DISPLAY=localhost:11.0
HOSTNAME=SERVER2
OLDPWD=/nfs/users/verdebs
PBS_O_HOME=/nfs/users/verdebs
PBS_JOBID=239.SERVER1
PBS_JOBNAME=STDIN
NCPUS=2
PBS_O_PATH=/usr/share/Modules/bin:.:/opt/pbs/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/nfs/users/verdebs/.local/bin:/nfs/users/verdebs/bin
which_declare=declare -f
XDG_SESSION_ID=3070
MODULES_CMD=/usr/share/Modules/libexec/modulecmd.tcl
PBS_O_WORKDIR=/nfs/users/verdebs/CFXtst
USER=verdebs
AWP_ROOT211=/nfs/soft/ansys_inc/v211/
PBS_NODEFILE=/var/spool/pbs/aux/239.SERVER1
PBS_TASKNUM=1
PWD=/nfs/users/verdebs/CFXtst
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
HOME=/nfs/users/verdebs
SSH_CLIENT=10.42.47.245 65371 22
PBS_MOMPORT=15003
XDG_DATA_DIRS=/nfs/users/verdebs/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
PBS_JOBCOOKIE=418777613DB694897DEB82F50D9D6593
PBS_O_SHELL=/bin/bash
TMPDIR=/var/tmp/pbs.239.SERVER1
LOADEDMODULES=
SSH_TTY=/dev/pts/1
PBS_O_QUEUE=TH
MAIL=/var/spool/mail/verdebs
SHELL=/bin/bash
TERM=xterm
SHLVL=2
PBS_O_HOST=server1.tes.local
PBS_O_SYSTEM=Linux
MANPATH=::/opt/pbs/share/man:/opt/pbs/share/man
PBS_O_LOGNAME=verdebs
PBS_NODENUM=0
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/share/modulefiles
GDK_BACKEND=x11
PBS_JOBDIR=/nfs/users/verdebs
LOGNAME=verdebs
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-1sdDq9P79D,guid=fcb6389845d24d7ebaf0350462678f96
XDG_RUNTIME_DIR=/run/user/2000
MODULEPATH_modshare=/usr/share/modulefiles:1:/usr/share/Modules/modulefiles:1:/etc/modulefiles:1
PATH=/usr/share/Modules/bin:.:/opt/pbs/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/nfs/users/verdebs/.local/bin:/nfs/users/verdebs/bin:/opt/pbs/bin:/nfs/users/verdebs/.local/bin:/nfs/users/verdebs/bin
PBS_QUEUE=TH
MODULESHOME=/usr/share/Modules
HISTSIZE=1000
OMP_NUM_THREADS=2
LESSOPEN=||/usr/bin/lesspipe.sh %s
PBS_O_MAIL=/var/spool/mail/verdebs
_=/usr/bin/env

```

Once the job starts, I do see on SERVER2 the process named 239.SERVER1.SC. I also sees the usual ANSYS subprocesses kicked in. Once the “parallel” solving process should appear there is only 1 mpirun process starting instead of the 2 requested…

I do not know if this can be helpful, but we running PBS on a CentOS Stream 8. Could it be that the system prohibits PBS to initiate some process?

I will try to increase the log\_events values as suggested and look at the variaous log files trying to detect something unsual…

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [April 27, 2022, 3:08pm UTC](https://community.openpbs.org/t/parallel-job-not-starting-correctly/3062/17 "2022-04-27T15:08:21Z")

</div>

> [@NomisTuo](#):
>
> Once the job starts, I do see on SERVER2 the process named 239.SERVER1.SC. I also sees the usual ANSYS subprocesses kicked in. Once the “parallel” solving process should appear there is only 1 mpirun process starting instead of the 2 requested…

So you run the below command opening a terminal on the compute node (server2) and it works

```auto
cfx5solve -example StaticMixer -parallel -part 2 -par-local -start-method "Open MPI Local Parallel"

```

> [@NomisTuo](#):
>
> I do not know if this can be helpful, but we running PBS on a CentOS Stream 8. Could it be that the system prohibits PBS to initiate some process?

As you can see the PBS Pro executes the batch command lines that are mentioned in the script submitted. Finally, it is the cfx that is going to solve the problem and PBS is just waiting for its exit status ( once cfx batch is executed it is the cfx that controls and spawns the processes) and PBS does not come into picture . PBS Provides the chunks/cores that is allocated to this job via $PBS\_NODEFILE for the cfx command to use it.

There are many using cfx with PBS Pro , example: [https://www.usq.edu.au/-/media/usq/current-students/academic/research/conducting-research/eresearch-services/hpc/pbs-ansys-examples.ashx?la=en&hash=8C8DE80D7F89273F3DB0160ACBE1354B](https://www.usq.edu.au/-/media/usq/current-students/academic/research/conducting-research/eresearch-services/hpc/pbs-ansys-examples.ashx?la=en&hash=8C8DE80D7F89273F3DB0160ACBE1354B)

You would need to contact CFX support team and get their help.