CentOS 7
PBSpro 14.1.0-26
I am encountering a strange issue with email notifications. If I submit a regular job and use the #PBS -m abe and #PBS -M email@address.tld, I get notified of the job start and end. If I do the same with an array job there are no emails sent out. The only time I see emails sent out for array jobs is when the job is cancelled by the root user.
Am I missing an option to have array jobs send out emails?
Thank you,
SS
Looks like a known bug. From the 14.2.1 (commercial) User Guide subjobs should send email:
The mail points argument is a string which consists of either the single character “n”, or one or more of the characters “a”, “b”, and “e”.
a Send mail when job or subjob is aborted by batch system
b Send mail when job or subjob begins execution
e Send mail when job or subjob ends execution
But there is an open bug (see PP-773):
Array jobs do not send any emails when specified with -m abe. The situation is reproducible.
The below does not send any email -
qsub -l select=1:ncpus=1 -N “TESTING” -m abe -J 1-10 -M @ – /bin/sleep 10
Whereas, the below sends emails -
qsub -l select=1:ncpus=1 -N “TESTING_NONARRAY” -m abe -M @ – /bin/sleep 10
The above command is a non-array job it sends email.
smgoosen thank you for info.
HI,
would the commercial version has this bug also?
I have tested job array function with open source pbspro. there are two issues identified,
a testing environment has one server and two compute nodes each having 12 cores.
I submitted an array job with 30 subjobs. only 7 subjobs rather than 12 subjobs ran on each nodes. why? resources were not fully utilized.
I also tested preemption with array job. by setting sched_preempt_enforce_resumption as true in shecduler’s configuration file, I expect that the scheduler would treat preempted subjobs as top jobs so that when the preempor job completes, the suspended subjobs would resume to run immediately. However, the other queued subjobs started running while preempted subjobs were still suspended until there were free cores.
running array jobs is getting popular on many HPC facilities now. if these issues existed, many systems would stick to torque and maui.
Thanks,
Sue
Thanks for sharing what you are seeing. I would recommend that you start a new post since the two issues that you pointed out do not have much to do with the title of this post.
My guess is something is not configured correctly on your nodes. Can you provide the output of pbsnodes -a on the forum? Also, can you provide the output of qmgr -c “p s”?
It is possible that there is a bug here. Please review these bugs (https://pbspro.atlassian.net/browse/PP-310?jql=text%20~%20"resumption") and see if one of them matches what you are seeing. If not, please file a bug at pbspro.attlassian.net. If you are able to also provide the steps you took to encounter that issue that would be helpful for us to address the issue.
thanks, Jon.
“pbsnodes -a” give this infor,
node01
Mom = node01.physics.usyd.edu.au
ntype = PBS
state = free
pcpus = 24
resources_available.arch = linux
resources_available.host = node01
resources_available.mem = 32935272kb
resources_available.ncpus = 12
resources_available.Qlist = physics,hippocrates
resources_available.vnode = node01
resources_assigned.accelerator_memory = 0kb
resources_assigned.mem = 0kb
resources_assigned.naccelerators = 0
resources_assigned.ncpus = 0
resources_assigned.vmem = 0kb
resv_enable = True
sharing = default_shared
license = l
node03
Mom = node03.physics.usyd.edu.au
ntype = PBS
state = free
pcpus = 24
resources_available.arch = linux
resources_available.host = node03
resources_available.mem = 32935272kb
resources_available.ncpus = 12
resources_available.Qlist = physics,hippocrates
resources_available.vnode = node03
resources_assigned.accelerator_memory = 0kb
resources_assigned.mem = 0kb
resources_assigned.naccelerators = 0
resources_assigned.ncpus = 0
resources_assigned.vmem = 0kb
resv_enable = True
sharing = default_shared
license = l
qmgr -c “p s” shows,
create resource Qlist
set resource Qlist type = string_array
set resource Qlist flag = h
create queue workq
set queue workq queue_type = Execution
set queue workq enabled = True
set queue workq started = True
create queue physics
set queue physics queue_type = Execution
set queue physics Priority = 10
set queue physics default_chunk.Qlist = physics
set queue physics enabled = True
set queue physics started = True
create queue cmt
set queue cmt default_chunk.Qlist = cmt
create queue yossarian
set queue yossarian queue_type = Execution
set queue yossarian default_chunk.Qlist = yossarian
set queue yossarian enabled = True
set queue yossarian started = True
create queue hippocrates
set queue hippocrates queue_type = Execution
set queue hippocrates Priority = 200
set queue hippocrates default_chunk.Qlist = hippocrates
set queue hippocrates enabled = True
set queue hippocrates started = True
set server scheduling = True
set server default_queue = physics
set server log_events = 511
set server mail_from = adm
set server query_other_jobs = True
set server resources_default.ncpus = 1
set server default_chunk.ncpus = 1
set server scheduler_iteration = 600
set server resv_enable = True
set server node_fail_requeue = 310
set server max_array_size = 10000
set server pbs_license_min = 0
set server pbs_license_max = 2147483647
set server pbs_license_linger_time = 31536000
set server eligible_time_enable = False
set server max_concurrent_provision = 5
Sue
At first glance, I don’t see anything wrong with your configuration. Can you please share the command for how you submitted the job array? (i.e. qsub -J 1-30 …)
Hi Sue,
My only idea is that your jobs were very very short. The scheduler started a cycle and started running subjobs. Before the scheduler had finished its cycle, some subjobs had ended. By the time the cycle had ended, only 7 subjobs were running. It had run 10, but only 7 were remaining(indices 3-10). What was the array job’s array_indices_remaining attribute after the cycle?
sched_preempt_enforce_resumption is a sched object attribute set via qmgr. It isn’t set via the sched_config file. If set via the sched_config file, you’d get a error message in your sched log, but it’s kind of hidden in the startup messages. It’s really easy to miss.
Bhroam
these are PBS directives in the job script,
#!/bin/csh -x
#PBS -N UCMHflux
#PBS -M xue.yang@sydney.edu.au
#PBS -V
#PBS -q physics
#PBS -l select=1:ncpus=1:mem=4GB
#PBS -l walltime=24:00:00
#PBS -J 1-30
#PBS -m abe
/usr/physics/pbspro/bin/qstat -1nt
headnode:
Req’d Req’d Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
79.headnode test physics UCMHflux – 1 1 4gb 24:00 B – –
79[1].headnode test physics UCMHflux 28977 1 1 4gb 24:00 R 00:06 node01/0
79[2].headnode test physics UCMHflux 28978 1 1 4gb 24:00 R 00:06 node01/1
79[3].headnode test physics UCMHflux 28979 1 1 4gb 24:00 R 00:06 node01/2
79[4].headnode test physics UCMHflux 28980 1 1 4gb 24:00 R 00:06 node01/3
79[5].headnode test physics UCMHflux 28981 1 1 4gb 24:00 R 00:06 node01/4
79[6].headnode test physics UCMHflux 28982 1 1 4gb 24:00 R 00:06 node01/5
79[7].headnode test physics UCMHflux 28983 1 1 4gb 24:00 R 00:06 node01/6
79[8].headnode test physics UCMHflux 10973 1 1 4gb 24:00 R 00:06 node03/0
79[9].headnode test physics UCMHflux 10974 1 1 4gb 24:00 R 00:06 node03/1
79[10].headnode test physics UCMHflux 10975 1 1 4gb 24:00 R 00:06 node03/2
79[11].headnode test physics UCMHflux 10976 1 1 4gb 24:00 R 00:06 node03/3
79[12].headnode test physics UCMHflux 10977 1 1 4gb 24:00 R 00:06 node03/4
79[13].headnode test physics UCMHflux 10978 1 1 4gb 24:00 R 00:06 node03/5
79[14].headnode test physics UCMHflux 10979 1 1 4gb 24:00 R 00:06 node03/6
79[15].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[16].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[17].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[18].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[19].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[20].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[21].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[22].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[23].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[24].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[25].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[26].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[27].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[28].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[29].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
79[30].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –
thanks,
Sue
It’s the memory on the nodes that’s hindering running more subjobs. My back of the envelope calculations show that you have 31gb per node. 7 subjobs require 28gb. There isn’t another 4gb to run another.
Bhroam
thank you, Bhroam, for pointing out this miss. my ignorance.
Sue
Hi Bhroam,
I removed sched_preempt_enforce_resumption setting in sched configuration file and ran this,
qmgr -c ‘set sched sched_preempt_enforce_resumption=True’
qmgr -c ‘list sched’
Sched scheduler@headnode
sched_cycle_length = 00:20:00
sched_preempt_enforce_resumption = True
but the second issue retains. perhaps we have to wait for v15.0.0 to release and to fix it?
Sue
Hey Sue,
If you can set sched_preempt_enforce_resumption then the feature should work. You don’t need to wait for a future release.
How are jobs being preempted? Express queue? Soft limits? Are they suspended/requeued?
The preempted jobs should be top jobs in subsequent cycles after they are preempted. You should see a log message saying they are top jobs. It’s something like, "Job is a top job and will run at "
They are pretty high on the priority list. Any express queue jobs will come before them, but that’s it.
Bhroam
Hi Bhroam,
subjobs were pre-empted as suspended by a job through express queue. after I deleted the express job, other queued subjobs got to run while the pre-empted subjobs are still in suspension.
/usr/physics/pbspro/bin/qstat -1nt | more
headnode:
Req’d Req’d Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
154.headnode sxy physics pbstest – 1 1 2gb 05:00 B – –
154[1].headnode sxy physics pbstest 26574 1 1 2gb 05:00 S 00:01 node01/0
154[2].headnode sxy physics pbstest 26575 1 1 2gb 05:00 S 00:01 node01/1
154[3].headnode sxy physics pbstest 26576 1 1 2gb 05:00 S 00:01 node01/2
154[4].headnode sxy physics pbstest 26577 1 1 2gb 05:00 S 00:01 node01/3
154[5].headnode sxy physics pbstest 26578 1 1 2gb 05:00 S 00:01 node01/4
154[6].headnode sxy physics pbstest 26579 1 1 2gb 05:00 S 00:01 node01/5
154[7].headnode sxy physics pbstest 26580 1 1 2gb 05:00 S 00:01 node01/6
154[8].headnode sxy physics pbstest 26581 1 1 2gb 05:00 S 00:01 node01/7
154[9].headnode sxy physics pbstest 26582 1 1 2gb 05:00 S 00:01 node01/8
154[10].headnod sxy physics pbstest 26583 1 1 2gb 05:00 S 00:01 node01/9
154[11].headnod sxy physics pbstest 26584 1 1 2gb 05:00 S 00:01 node01/10
154[12].headnod sxy physics pbstest 26585 1 1 2gb 05:00 S 00:01 node01/11
154[13].headnod sxy physics pbstest 6662 1 1 2gb 05:00 S 00:01 node03/0
154[14].headnod sxy physics pbstest 6663 1 1 2gb 05:00 S 00:01 node03/1
154[15].headnod sxy physics pbstest 6664 1 1 2gb 05:00 S 00:01 node03/2
154[16].headnod sxy physics pbstest 6665 1 1 2gb 05:00 S 00:01 node03/3
154[17].headnod sxy physics pbstest 6666 1 1 2gb 05:00 S 00:01 node03/4
154[18].headnod sxy physics pbstest 6667 1 1 2gb 05:00 S 00:01 node03/5
154[19].headnod sxy physics pbstest 6668 1 1 2gb 05:00 S 00:01 node03/6
154[20].headnod sxy physics pbstest 6669 1 1 2gb 05:00 S 00:01 node03/7
154[21].headnod sxy physics pbstest 6670 1 1 2gb 05:00 S 00:01 node03/8
154[22].headnod sxy physics pbstest 6671 1 1 2gb 05:00 S 00:01 node03/9
154[23].headnod sxy physics pbstest 6672 1 1 2gb 05:00 S 00:01 node03/10
154[24].headnod sxy physics pbstest 6673 1 1 2gb 05:00 S 00:01 node03/11
154[25].headnod sxy physics pbstest 28040 1 1 2gb 05:00 R 00:00 node01/0
154[26].headnod sxy physics pbstest 28041 1 1 2gb 05:00 R 00:00 node01/1
154[27].headnod sxy physics pbstest 28042 1 1 2gb 05:00 R 00:00 node01/2
154[28].headnod sxy physics pbstest 28043 1 1 2gb 05:00 R 00:00 node01/3
154[29].headnod sxy physics pbstest 28044 1 1 2gb 05:00 R 00:00 node01/4
154[30].headnod sxy physics pbstest 28045 1 1 2gb 05:00 R 00:00 node01/5
154[31].headnod sxy physics pbstest 28046 1 1 2gb 05:00 R 00:00 node01/6
154[32].headnod sxy physics pbstest 28047 1 1 2gb 05:00 R 00:00 node01/7
154[33].headnod sxy physics pbstest 28048 1 1 2gb 05:00 R 00:00 node01/8
154[34].headnod sxy physics pbstest 28049 1 1 2gb 05:00 R 00:00 node01/9
154[35].headnod sxy physics pbstest 28050 1 1 2gb 05:00 R 00:00 node01/10
154[36].headnod sxy physics pbstest 28051 1 1 2gb 05:00 R 00:00 node01/11
154[37].headnod sxy physics pbstest 8025 1 1 2gb 05:00 R 00:00 node03/0
154[38].headnod sxy physics pbstest 8026 1 1 2gb 05:00 R 00:00 node03/1
154[39].headnod sxy physics pbstest 8027 1 1 2gb 05:00 R 00:00 node03/2
154[40].headnod sxy physics pbstest 8028 1 1 2gb 05:00 R 00:00 node03/3
154[41].headnod sxy physics pbstest 8029 1 1 2gb 05:00 R 00:00 node03/4
154[42].headnod sxy physics pbstest 8030 1 1 2gb 05:00 R 00:00 node03/5
154[43].headnod sxy physics pbstest 8031 1 1 2gb 05:00 R 00:00 node03/6
154[44].headnod sxy physics pbstest 8032 1 1 2gb 05:00 R 00:00 node03/7
154[45].headnod sxy physics pbstest 8033 1 1 2gb 05:00 R 00:00 node03/8
154[46].headnod sxy physics pbstest 8034 1 1 2gb 05:00 R 00:00 node03/9
154[47].headnod sxy physics pbstest 8035 1 1 2gb 05:00 R 00:00 node03/10
154[48].headnod sxy physics pbstest 8036 1 1 2gb 05:00 R 00:00 node03/11
154[49].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[50].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[51].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[52].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[53].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[54].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[55].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[56].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[57].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[58].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[59].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
154[60].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –
Sue
additional infor,
qmgr -c ‘list sched’
Sched scheduler@headnode
sched_cycle_length = 00:20:00
sched_preempt_enforce_resumption = True
Sue
Sue,
I just found the issue in the code. Since subjobs don’t exist prior to being run, the scheduler makes sure it doesn’t alter them. It does this by checking if it is a subjob. Well after the job is run, it is still a subjob.
We mark a job as preempted by altering a boolean on the job.
This means the subjob never gets marked as preempted. Since it never gets marked as preempted, it doesn’t get higher priority in subsequent cycles.
Please file a bug on pbspro.atlassian.net
Bhroam