Ignoring finished dependencies

Developers
1

If I run a job with a dependency like such:

-W depend=afterok:job1

it will wait for job1 to succeed before the job is run. If, at the time of submission, job1 has already finished successfully, however, the new job will be held. Is there anyway to configure PBSPro such that the job will be run in this situation?

Thanks,
Jan

2

Place a user hold on the first job when you submit it, then submit the second job with the dependency, then release the first job.

$ qsub -h jobscript1
123.foo
$ qsub -W depend=afterok:123.foo jobscript2
124.foo
$ qrls 123.foo
1 Like
3

That sounds like a good enhancement to make in PBS.

4

I agree, is there any use in setting an after dependency on a job that doesn’t exist? If not, we could probably just remove the dependency on submission.

5

I’d go the route of rejecting a job that is trying to depend on a job that doesn’t exist. Unless job history is on, we can’t evaluate what the afterok/afternotok nature of the dependency. We wouldn’t want to just strip the dependency. If the user says only run the dependent job if the parent job ran successfully, we wouldn’t want to run it if the job crapped out.

Bhroam

3 Likes
6

if job history is turned off, that’s kind of what we do today, the job is aborted by the server. I do like the thought of keeping things consistent, reject always if parent job doesn’t exist.

7

I’ve only just started looking into PBSPro as a pipeline runner behind a scientific web application I’m developing. So users can flexibly submit jobs that depend on previous jobs they’ve submitted without very defined pipelines.

To me it’s intuitive that a job fails if it depends on a job for which there is no history. However, if the job is in the history and has completed successfully I find it counter-intuitive that the job is held. Given, I might be using PBSPro in a slightly different manner than it is designed for.

8

I agree with you, the job getting Held seems odd, although we do document it so that’s how it was designed to be. We are talking about a design change now, so depending on what the community agrees on, we can either reject a job always if the parent job doesn’t exist, or handle it differently if history is enabled, possibly accepting it if the dependency was resolved correctly.

9

No sure if it’s still of interest but if the jobs are submitted via bash script it’s actually possible to use this pattern to only hold the second job if the “job1” is still running:

if qstat -s job1; then
    qsub -W depend=afterok:job1 -- sleep 5
else
   echo "job1 finished no need to hold"
   qsub -- sleep 5
fi