PBS interpreting server_dyn_res script output as 0

Users/Site Administrators
1

I’ve followed the instructions in 5.14.6.3.ii of the PBS Pro Guide (20.0.1).

My flexlm script looks like this:

#!/bin/bash
# some comments...
raw=($(lmutil lmstat -c $2 -f $1 | grep Total | awk '{ print $0 }'))
avail=$((${raw[5]}-${raw[10]}))
total=${raw[5]}
echo $((avail))
resv=$(printf %.0f $(echo "$total*((100-$3)*0.01)" | bc)) #Percent to reserve for non-PBS jobs.

if [ "$avail" -le "$resv" ];then
        echo $avail
else
        echo $resv
fi

My qsub statement (which should pass and submit for execution) looks like this:

qsub -l msimhdlsim=1 ./scripts/echo_host.sh

Looking at sched_log i get this:

08/15/2024 15:16:29;0040;pbs_sched;Job;73.pbsserv;Insufficient amount of server resource: msimhdlsim (R: 1 A: 0 T: 0)
08/15/2024 15:16:29;0040;pbs_sched;Job;73.pbsserv;Job will never run with the resources currently configured in the complex
08/15/2024 15:16:29;0080;pbs_sched;Job;74.pbsserv;Considering job to run

sched_config server_dyn_res_line:

server_dyn_res: "msimhdlsim !/opt/pbs/server_dyn_res_scripts/lmquery msimhdlsim 8888@mylicserv 10"

sched_config resources:

resources: "ncpus, mem, arch, host, vnode, aoe, eoe, qlist, msimhdlsim"

resourcedef:

qlist type=string_array flag=h
msimhdlsim type=long

I’m not quite sure what i’m missing here. When i run the script with the arguments as entered in sched_config i get the expected output:

[root@pbsserv pbs]# /opt/pbs/server_dyn_res_scripts/lmquery msimhdlsim 8888@mylicserv 10
16

Any help is appreciated

2

Hi!

Why do you have the line echo $((avail)) ? Isn’t it a debug line?

I recommend adding #!/bin/bash on the first line of your flexlm script. Otherwise, the server_dyn_res query uses the/bin/sh interpreter.

Check the sched_log for server_dyn_res lines at the beginning of the scheduler iteration.

3

The manual says: “The script must return the value via stdout in a single line ending with a newline”. FWIW i’ve tried doing

$avail

echo $avail

with no difference.

I didn’t include the first few lines but I already have. I’ll edit the excerpt above to reflect that.

Nothing that i can see:

08/16/2024 14:37:26;0080;pbs_sched;Req;;Starting Scheduling Cycle
08/16/2024 14:37:26;0040;pbs_sched;Node;pbscompute1;Unknown Node State: initializing on add operation
08/16/2024 14:37:26;0040;pbs_sched;Node;pbscompute1;Unknown Node State: initializing
08/16/2024 14:37:26;0080;pbs_sched;Job;76.pbsserv;Considering job to run
08/16/2024 14:37:26;0040;pbs_sched;Job;76.pbsserv;Insufficient amount of server resource: msimhdlsim (R: 50 A: 0 T: 0)
08/16/2024 14:37:26;0040;pbs_sched;Job;76.pbsserv;Job will never run with the resources currently configured in the complex
08/16/2024 14:37:26;0080;pbs_sched;Job;93.pbsserv;Considering job to run
08/16/2024 14:37:26;0040;pbs_sched;Job;93.pbsserv;Insufficient amount of server resource: msimhdlsim (R: 1 A: 0 T: 0)
08/16/2024 14:37:26;0040;pbs_sched;Job;93.pbsserv;Job will never run with the resources currently configured in the complex
08/16/2024 14:37:26;0080;pbs_sched;Job;94.pbsserv;Considering job to run
08/16/2024 14:37:26;0040;pbs_sched;Job;94.pbsserv;Insufficient amount of server resource: msimhdlsim (R: 1 A: 0 T: 0)
08/16/2024 14:37:26;0040;pbs_sched;Job;94.pbsserv;Job will never run with the resources currently configured in the complex
08/16/2024 14:37:26;0080;pbs_sched;Req;;Leaving Scheduling Cycle
4

Could you please do this test

server_dyn_res: "msimhdlsim !/opt/pbs/server_dyn_res_scripts/lmquery msimhdlsim 8888@mylicserv 10"

Comment the above line in $PBS_HOME/sched_priv/sched_config

Add the below line

server_dyn_res: "msimhdlsim !/opt/pbs/server_dyn_res_scripts/check.sh

cat /opt/pbs/server_dyn_res_scripts/check.sh
#!/bin/bash
echo "100"

chmod +x /opt/pbs/server_dyn_res_scripts/check.sh
.//opt/pbs/server_dyn_res_scripts/check.sh
100

kill -HUP <pid of the scheduler>

Submit and check the scheduler message 

qsub -l msimhdlsim=101 ./scripts/echo_host.sh
qstat -answ1
5

Looks like the test worked. I submitted two jobs:

qsub -l msimhdlsim=101 ./scripts/echo_host.sh 
107.pbsserv
qsub -l msimhdlsim=99 ./scripts/echo_host.sh
108.pbsserv

08/19/2024 15:28:14;0080;pbs_sched;Job;107.pbsserv;Considering job to run
08/19/2024 15:28:14;0040;pbs_sched;Job;107.pbsserv;Insufficient amount of server resource: msimhdlsim (R: 101 A: 100 T: 100)
08/19/2024 15:28:14;0040;pbs_sched;Job;107.pbsserv;Job will never run with the resources currently configured in the complex
08/19/2024 15:28:14;0080;pbs_sched;Job;108.pbsserv;Considering job to run
08/19/2024 15:28:14;0040;pbs_sched;Job;108.pbsserv;Job run

For confirmation, my check.sh:

#!/bin/bash
echo "100"

FWIW, i added echo "100" and echo "50" to lmquery in place of the previous echo statements:

#!/bin/bash
# some comments...
###########################################################
raw=($(lmutil lmstat -c $2 -f $1 | grep Total | awk '{ print $0 }')) #'{avail=($6-$11); total=$6}'
avail=$((${raw[5]}-${raw[10]}))
total=${raw[5]}
resv=$(printf %.0f $(echo "$total*((100-$3)*0.01)" | bc))

#echo "available: $avail reserve: $resv"
if [ "$avail" -le "$resv" ];then
        echo "100"
        #echo $avail
else
        echo "50"
        #echo $resv
fi

And this behaves as expected:

qsub -l msimhdlsim=101 ./scripts/echo_host.sh 
114.pbsserv

qstat -answ1

pbsserv: 
                                                                                                   Req'd  Req'd   Elap
Job ID                         Username        Queue           Jobname         SessID   NDS  TSK   Memory Time  S Time
------------------------------ --------------- --------------- --------------- -------- ---- ----- ------ ----- - -----
114.pbsserv                  pbs_user        qmed            echo_host.sh         --     1     2  400mb   --  Q   --   -- 
   Can Never Run: Insufficient amount of server resource: msimhdlsim (R: 101 A: 50 T: 50)
1 Like
6

Hi all,

I solved this issue by using the absolute path to lmutil:

#!/bin/bash
# some comments...
raw=($(/usr/local/sbin/lmutil lmstat -c $2 -f $1 | grep Total | awk '{ print $0 }'))
avail=$((${raw[5]}-${raw[10]}))
.....

Even though root has lmutil in its path, it appears the shell PBS runs the dyn_res scripts does not take the full root env.