# Email notification from array jobs

**URL:** <https://community.openpbs.org/t/email-notification-from-array-jobs/548>\
**Category:** Users/Site Administrators\
**Created:** [May 18, 2017, 4:19pm UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548 "2017-05-18T16:19:10Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![sscluster](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@sscluster](https://community.openpbs.org/u/sscluster)\
**Post date:** [May 18, 2017, 4:19pm UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/1 "2017-05-18T16:19:10Z")

</div>

CentOS 7  
PBSpro 14.1.0-26

I am encountering a strange issue with email notifications. If I submit a regular job and use the #PBS -m abe and #PBS -M email@address.tld, I get notified of the job start and end. If I do the same with an array job there are no emails sent out. The only time I see emails sent out for array jobs is when the job is cancelled by the root user.

Am I missing an option to have array jobs send out emails?

Thank you,

SS

---

<div class="post-metadata">

**Author:** ![smgoosen](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@smgoosen](https://community.openpbs.org/u/smgoosen)\
**Post date:** [May 19, 2017, 5:47pm UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/2 "2017-05-19T17:47:48Z")

</div>

Looks like a known bug. From the 14.2.1 (commercial) User Guide subjobs should send email:

The mail points argument is a string which consists of either the single character “n”, or one or more of the characters “a”, “b”, and “e”.  
a Send mail when job or subjob is aborted by batch system  
b Send mail when job or subjob begins execution  
e Send mail when job or subjob ends execution

But there is an open bug (see PP-773):

Array jobs do not send any emails when specified with -m abe. The situation is reproducible.  
The below does not send any email -

qsub -l select=1:ncpus=1 -N “TESTING” -m abe -J 1-10 -M @ – /bin/sleep 10

Whereas, the below sends emails -

qsub -l select=1:ncpus=1 -N “TESTING\_NONARRAY” -m abe -M @ – /bin/sleep 10

The above command is a non-array job it sends email.

---

<div class="post-metadata">

**Author:** ![sscluster](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@sscluster](https://community.openpbs.org/u/sscluster)\
**Post date:** [May 19, 2017, 5:58pm UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/3 "2017-05-19T17:58:08Z")

</div>

smgoosen thank you for info.

---

<div class="post-metadata">

**Author:** ![sxy](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sxy](https://community.openpbs.org/u/sxy)\
**Post date:** [May 31, 2017, 2:02am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/4 "2017-05-31T02:02:56Z")

</div>

HI,

would the commercial version has this bug also?

I have tested job array function with open source pbspro. there are two issues identified,

1. a testing environment has one server and two compute nodes each having 12 cores.  
I submitted an array job with 30 subjobs. only 7 subjobs rather than 12 subjobs ran on each nodes. why? resources were not fully utilized.

2. I also tested preemption with array job. by setting sched\_preempt\_enforce\_resumption as true in shecduler’s configuration file, I expect that the scheduler would treat preempted subjobs as top jobs so that when the preempor job completes, the suspended subjobs would resume to run immediately. However, the other queued subjobs started running while preempted subjobs were still suspended until there were free cores.

running array jobs is getting popular on many HPC facilities now. if these issues existed, many systems would stick to torque and maui.

Thanks,

Sue

---

<div class="post-metadata">

**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\
**Post date:** [May 31, 2017, 3:26am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/5 "2017-05-31T03:26:08Z")

</div>

Thanks for sharing what you are seeing. I would recommend that you start a new post since the two issues that you pointed out do not have much to do with the title of this post.

> [@sxy](#):
>
> a testing environment has one server and two compute nodes each having 12 cores.  
> I submitted an array job with 30 subjobs. only 7 subjobs rather than 12 subjobs ran on each nodes. why? resources were not fully utilized.

My guess is something is not configured correctly on your nodes. Can you provide the output of pbsnodes -a on the forum? Also, can you provide the output of qmgr -c “p s”?

> [@sxy](#):
>
> I also tested preemption with array job. by setting sched\_preempt\_enforce\_resumption as true in shecduler’s configuration file, I expect that the scheduler would treat preempted subjobs as top jobs so that when the preempor job completes, the suspended subjobs would resume to run immediately. However, the other queued subjobs started running while preempted subjobs were still suspended until there were free cores.

It is possible that there is a bug here. Please review these bugs ([https://pbspro.atlassian.net/browse/PP-310?jql=text%20~%20"resumption"](https://pbspro.atlassian.net/browse/PP-310?jql=text%20~%20%22resumption%22)) and see if one of them matches what you are seeing. If not, please file a bug at [pbspro.attlassian.net](http://pbspro.attlassian.net). If you are able to also provide the steps you took to encounter that issue that would be helpful for us to address the issue.

---

<div class="post-metadata">

**Author:** ![sxy](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sxy](https://community.openpbs.org/u/sxy)\
**Post date:** [May 31, 2017, 4:36am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/6 "2017-05-31T04:36:38Z")

</div>

thanks, Jon.

“pbsnodes -a” give this infor,

node01  
Mom = [node01.physics.usyd.edu.au](http://node01.physics.usyd.edu.au)  
ntype = PBS  
state = free  
pcpus = 24  
resources\_available.arch = linux  
resources\_available.host = node01  
resources\_available.mem = 32935272kb  
resources\_available.ncpus = 12  
resources\_available.Qlist = physics,hippocrates  
resources\_available.vnode = node01  
resources\_assigned.accelerator\_memory = 0kb  
resources\_assigned.mem = 0kb  
resources\_assigned.naccelerators = 0  
resources\_assigned.ncpus = 0  
resources\_assigned.vmem = 0kb  
resv\_enable = True  
sharing = default\_shared  
license = l

node03  
Mom = [node03.physics.usyd.edu.au](http://node03.physics.usyd.edu.au)  
ntype = PBS  
state = free  
pcpus = 24  
resources\_available.arch = linux  
resources\_available.host = node03  
resources\_available.mem = 32935272kb  
resources\_available.ncpus = 12  
resources\_available.Qlist = physics,hippocrates  
resources\_available.vnode = node03  
resources\_assigned.accelerator\_memory = 0kb  
resources\_assigned.mem = 0kb  
resources\_assigned.naccelerators = 0  
resources\_assigned.ncpus = 0  
resources\_assigned.vmem = 0kb  
resv\_enable = True  
sharing = default\_shared  
license = l

qmgr -c “p s” shows,

# 

# Create resources and set their properties.

# 

# 

# Create and define resource Qlist

# 

create resource Qlist  
set resource Qlist type = string\_array  
set resource Qlist flag = h

# 

# Create queues and set their attributes.

# 

# 

# Create and define queue workq

# 

create queue workq  
set queue workq queue\_type = Execution  
set queue workq enabled = True  
set queue workq started = True

# 

# Create and define queue physics

# 

create queue physics  
set queue physics queue\_type = Execution  
set queue physics Priority = 10  
set queue physics default\_chunk.Qlist = physics  
set queue physics enabled = True  
set queue physics started = True

# 

# Create and define queue cmt

# 

create queue cmt  
set queue cmt default\_chunk.Qlist = cmt

# 

# Create and define queue yossarian

# 

create queue yossarian  
set queue yossarian queue\_type = Execution  
set queue yossarian default\_chunk.Qlist = yossarian  
set queue yossarian enabled = True  
set queue yossarian started = True

# 

# Create and define queue hippocrates

# 

create queue hippocrates  
set queue hippocrates queue\_type = Execution  
set queue hippocrates Priority = 200  
set queue hippocrates default\_chunk.Qlist = hippocrates  
set queue hippocrates enabled = True  
set queue hippocrates started = True

# 

# Set server attributes.

# 

set server scheduling = True  
set server default\_queue = physics  
set server log\_events = 511  
set server mail\_from = adm  
set server query\_other\_jobs = True  
set server resources\_default.ncpus = 1  
set server default\_chunk.ncpus = 1  
set server scheduler\_iteration = 600  
set server resv\_enable = True  
set server node\_fail\_requeue = 310  
set server max\_array\_size = 10000  
set server pbs\_license\_min = 0  
set server pbs\_license\_max = 2147483647  
set server pbs\_license\_linger\_time = 31536000  
set server eligible\_time\_enable = False  
set server max\_concurrent\_provision = 5

Sue

---

<div class="post-metadata">

**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\
**Post date:** [May 31, 2017, 3:00pm UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/7 "2017-05-31T15:00:30Z")

</div>

At first glance, I don’t see anything wrong with your configuration. Can you please share the command for how you submitted the job array? (i.e. qsub -J 1-30 …)

---

<div class="post-metadata">

**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\
**Post date:** [May 31, 2017, 6:25pm UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/8 "2017-05-31T18:25:36Z")

</div>

Hi Sue,

> [@sxy](#):
>
> a testing environment has one server and two compute nodes each having 12 cores.  
> I submitted an array job with 30 subjobs. only 7 subjobs rather than 12 subjobs ran on each nodes. why? resources were not fully utilized.

My only idea is that your jobs were very very short. The scheduler started a cycle and started running subjobs. Before the scheduler had finished its cycle, some subjobs had ended. By the time the cycle had ended, only 7 subjobs were running. It had run 10, but only 7 were remaining(indices 3-10). What was the array job’s array\_indices\_remaining attribute after the cycle?

> [@sxy](#):
>
> I also tested preemption with array job. by setting sched\_preempt\_enforce\_resumption as true in shecduler’s configuration file, I expect that the scheduler would treat preempted subjobs as top jobs so that when the preempor job completes, the suspended subjobs would resume to run immediately. However, the other queued subjobs started running while preempted subjobs were still suspended until there were free cores.

sched\_preempt\_enforce\_resumption is a sched object attribute set via qmgr. It isn’t set via the sched\_config file. If set via the sched\_config file, you’d get a error message in your sched log, but it’s kind of hidden in the startup messages. It’s really easy to miss.

Bhroam

---

<div class="post-metadata">

**Author:** ![sxy](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sxy](https://community.openpbs.org/u/sxy)\
**Post date:** [June 1, 2017, 12:47am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/9 "2017-06-01T00:47:58Z")

</div>

these are PBS directives in the job script,

#!/bin/csh -x  
#PBS -N UCMHflux  
#PBS -M xue.yang@sydney.edu.au  
#PBS -V  
#PBS -q physics  
#PBS -l select=1:ncpus=1:mem=4GB  
#PBS -l walltime=24:00:00  
#PBS -J 1-30  
#PBS -m abe

* * *

> /usr/physics/pbspro/bin/qstat -1nt

headnode:  
Req’d Req’d Elap  
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

* * *

79.headnode test physics UCMHflux – 1 1 4gb 24:00 B – –  
79[1].headnode test physics UCMHflux 28977 1 1 4gb 24:00 R 00:06 node01/0  
79[2].headnode test physics UCMHflux 28978 1 1 4gb 24:00 R 00:06 node01/1  
79[3].headnode test physics UCMHflux 28979 1 1 4gb 24:00 R 00:06 node01/2  
79[4].headnode test physics UCMHflux 28980 1 1 4gb 24:00 R 00:06 node01/3  
79[5].headnode test physics UCMHflux 28981 1 1 4gb 24:00 R 00:06 node01/4  
79[6].headnode test physics UCMHflux 28982 1 1 4gb 24:00 R 00:06 node01/5  
79[7].headnode test physics UCMHflux 28983 1 1 4gb 24:00 R 00:06 node01/6  
79[8].headnode test physics UCMHflux 10973 1 1 4gb 24:00 R 00:06 node03/0  
79[9].headnode test physics UCMHflux 10974 1 1 4gb 24:00 R 00:06 node03/1  
79[10].headnode test physics UCMHflux 10975 1 1 4gb 24:00 R 00:06 node03/2  
79[11].headnode test physics UCMHflux 10976 1 1 4gb 24:00 R 00:06 node03/3  
79[12].headnode test physics UCMHflux 10977 1 1 4gb 24:00 R 00:06 node03/4  
79[13].headnode test physics UCMHflux 10978 1 1 4gb 24:00 R 00:06 node03/5  
79[14].headnode test physics UCMHflux 10979 1 1 4gb 24:00 R 00:06 node03/6  
79[15].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[16].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[17].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[18].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[19].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[20].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[21].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[22].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[23].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[24].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[25].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[26].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[27].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[28].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[29].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –  
79[30].headnode test physics UCMHflux – 1 1 4gb 24:00 Q – –

thanks,

Sue

---

<div class="post-metadata">

**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\
**Post date:** [June 1, 2017, 12:54am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/10 "2017-06-01T00:54:06Z")

</div>

> [@sxy](#):
>
> PBS -l select=1:ncpus=1:mem=4GB

> [@sxy](#):
>
> resources\_available.mem = 32935272kb

It’s the memory on the nodes that’s hindering running more subjobs. My back of the envelope calculations show that you have 31gb per node. 7 subjobs require 28gb. There isn’t another 4gb to run another.

Bhroam

---

<div class="post-metadata">

**Author:** ![sxy](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sxy](https://community.openpbs.org/u/sxy)\
**Post date:** [June 1, 2017, 1:02am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/11 "2017-06-01T01:02:28Z")

</div>

thank you, Bhroam, for pointing out this miss. my ignorance.

Sue

---

<div class="post-metadata">

**Author:** ![sxy](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sxy](https://community.openpbs.org/u/sxy)\
**Post date:** [June 1, 2017, 3:08am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/12 "2017-06-01T03:08:33Z")

</div>

Hi Bhroam,

> [@bhroam](#):
>
> sched\_preempt\_enforce\_resumption is a sched object attribute set via qmgr. It isn’t set via the sched\_config file. If set via the sched\_config file, you’d get a error message in your sched log, but it’s kind of hidden in the startup messages. It’s really easy to miss.

I removed sched\_preempt\_enforce\_resumption setting in sched configuration file and ran this,

> qmgr -c ‘set sched sched\_preempt\_enforce\_resumption=True’  
> qmgr -c ‘list sched’  
> Sched scheduler@headnode  
> sched\_cycle\_length = 00:20:00  
> sched\_preempt\_enforce\_resumption = True

but the second issue retains. perhaps we have to wait for v15.0.0 to release and to fix it?

Sue

---

<div class="post-metadata">

**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\
**Post date:** [June 1, 2017, 8:28pm UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/13 "2017-06-01T20:28:11Z")

</div>

Hey Sue,  
If you can set sched\_preempt\_enforce\_resumption then the feature should work. You don’t need to wait for a future release.

How are jobs being preempted? Express queue? Soft limits? Are they suspended/requeued?

The preempted jobs should be top jobs in subsequent cycles after they are preempted. You should see a log message saying they are top jobs. It’s something like, "Job is a top job and will run at "

They are pretty high on the priority list. Any express queue jobs will come before them, but that’s it.

Bhroam

---

<div class="post-metadata">

**Author:** ![sxy](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sxy](https://community.openpbs.org/u/sxy)\
**Post date:** [June 2, 2017, 12:16am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/14 "2017-06-02T00:16:43Z")

</div>

Hi Bhroam,

> [@bhroam](#):
>
> How are jobs being preempted? Express queue? Soft limits? Are they suspended/requeued?

subjobs were pre-empted as suspended by a job through express queue. after I deleted the express job, other queued subjobs got to run while the pre-empted subjobs are still in suspension.

> /usr/physics/pbspro/bin/qstat -1nt | more

headnode:  
Req’d Req’d Elap  
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time

* * *

154.headnode sxy physics pbstest – 1 1 2gb 05:00 B – –  
154[1].headnode sxy physics pbstest 26574 1 1 2gb 05:00 S 00:01 node01/0  
154[2].headnode sxy physics pbstest 26575 1 1 2gb 05:00 S 00:01 node01/1  
154[3].headnode sxy physics pbstest 26576 1 1 2gb 05:00 S 00:01 node01/2  
154[4].headnode sxy physics pbstest 26577 1 1 2gb 05:00 S 00:01 node01/3  
154[5].headnode sxy physics pbstest 26578 1 1 2gb 05:00 S 00:01 node01/4  
154[6].headnode sxy physics pbstest 26579 1 1 2gb 05:00 S 00:01 node01/5  
154[7].headnode sxy physics pbstest 26580 1 1 2gb 05:00 S 00:01 node01/6  
154[8].headnode sxy physics pbstest 26581 1 1 2gb 05:00 S 00:01 node01/7  
154[9].headnode sxy physics pbstest 26582 1 1 2gb 05:00 S 00:01 node01/8  
154[10].headnod sxy physics pbstest 26583 1 1 2gb 05:00 S 00:01 node01/9  
154[11].headnod sxy physics pbstest 26584 1 1 2gb 05:00 S 00:01 node01/10  
154[12].headnod sxy physics pbstest 26585 1 1 2gb 05:00 S 00:01 node01/11  
154[13].headnod sxy physics pbstest 6662 1 1 2gb 05:00 S 00:01 node03/0  
154[14].headnod sxy physics pbstest 6663 1 1 2gb 05:00 S 00:01 node03/1  
154[15].headnod sxy physics pbstest 6664 1 1 2gb 05:00 S 00:01 node03/2  
154[16].headnod sxy physics pbstest 6665 1 1 2gb 05:00 S 00:01 node03/3  
154[17].headnod sxy physics pbstest 6666 1 1 2gb 05:00 S 00:01 node03/4  
154[18].headnod sxy physics pbstest 6667 1 1 2gb 05:00 S 00:01 node03/5  
154[19].headnod sxy physics pbstest 6668 1 1 2gb 05:00 S 00:01 node03/6  
154[20].headnod sxy physics pbstest 6669 1 1 2gb 05:00 S 00:01 node03/7  
154[21].headnod sxy physics pbstest 6670 1 1 2gb 05:00 S 00:01 node03/8  
154[22].headnod sxy physics pbstest 6671 1 1 2gb 05:00 S 00:01 node03/9  
154[23].headnod sxy physics pbstest 6672 1 1 2gb 05:00 S 00:01 node03/10  
154[24].headnod sxy physics pbstest 6673 1 1 2gb 05:00 S 00:01 node03/11  
154[25].headnod sxy physics pbstest 28040 1 1 2gb 05:00 R 00:00 node01/0  
154[26].headnod sxy physics pbstest 28041 1 1 2gb 05:00 R 00:00 node01/1  
154[27].headnod sxy physics pbstest 28042 1 1 2gb 05:00 R 00:00 node01/2  
154[28].headnod sxy physics pbstest 28043 1 1 2gb 05:00 R 00:00 node01/3  
154[29].headnod sxy physics pbstest 28044 1 1 2gb 05:00 R 00:00 node01/4  
154[30].headnod sxy physics pbstest 28045 1 1 2gb 05:00 R 00:00 node01/5  
154[31].headnod sxy physics pbstest 28046 1 1 2gb 05:00 R 00:00 node01/6  
154[32].headnod sxy physics pbstest 28047 1 1 2gb 05:00 R 00:00 node01/7  
154[33].headnod sxy physics pbstest 28048 1 1 2gb 05:00 R 00:00 node01/8  
154[34].headnod sxy physics pbstest 28049 1 1 2gb 05:00 R 00:00 node01/9  
154[35].headnod sxy physics pbstest 28050 1 1 2gb 05:00 R 00:00 node01/10  
154[36].headnod sxy physics pbstest 28051 1 1 2gb 05:00 R 00:00 node01/11  
154[37].headnod sxy physics pbstest 8025 1 1 2gb 05:00 R 00:00 node03/0  
154[38].headnod sxy physics pbstest 8026 1 1 2gb 05:00 R 00:00 node03/1  
154[39].headnod sxy physics pbstest 8027 1 1 2gb 05:00 R 00:00 node03/2  
154[40].headnod sxy physics pbstest 8028 1 1 2gb 05:00 R 00:00 node03/3  
154[41].headnod sxy physics pbstest 8029 1 1 2gb 05:00 R 00:00 node03/4  
154[42].headnod sxy physics pbstest 8030 1 1 2gb 05:00 R 00:00 node03/5  
154[43].headnod sxy physics pbstest 8031 1 1 2gb 05:00 R 00:00 node03/6  
154[44].headnod sxy physics pbstest 8032 1 1 2gb 05:00 R 00:00 node03/7  
154[45].headnod sxy physics pbstest 8033 1 1 2gb 05:00 R 00:00 node03/8  
154[46].headnod sxy physics pbstest 8034 1 1 2gb 05:00 R 00:00 node03/9  
154[47].headnod sxy physics pbstest 8035 1 1 2gb 05:00 R 00:00 node03/10  
154[48].headnod sxy physics pbstest 8036 1 1 2gb 05:00 R 00:00 node03/11  
154[49].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[50].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[51].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[52].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[53].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[54].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[55].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[56].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[57].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[58].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[59].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –  
154[60].headnod sxy physics pbstest – 1 1 2gb 05:00 Q – –

Sue

---

<div class="post-metadata">

**Author:** ![sxy](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sxy](https://community.openpbs.org/u/sxy)\
**Post date:** [June 2, 2017, 12:19am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/15 "2017-06-02T00:19:16Z")

</div>

additional infor,

> qmgr -c ‘list sched’  
> Sched scheduler@headnode  
> sched\_cycle\_length = 00:20:00  
> sched\_preempt\_enforce\_resumption = True

Sue

---

<div class="post-metadata">

**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\
**Post date:** [June 2, 2017, 12:25am UTC](https://community.openpbs.org/t/email-notification-from-array-jobs/548/16 "2017-06-02T00:25:04Z")

</div>

Sue,  
I just found the issue in the code. Since subjobs don’t exist prior to being run, the scheduler makes sure it doesn’t alter them. It does this by checking if it is a subjob. Well after the job is run, it is still a subjob.

We mark a job as preempted by altering a boolean on the job.

This means the subjob never gets marked as preempted. Since it never gets marked as preempted, it doesn’t get higher priority in subsequent cycles.

Please file a bug on [pbspro.atlassian.net](http://pbspro.atlassian.net)

Bhroam