PP-506,PP-507: Add support for requesting resources with logical 'or' and conditional operators

[arungrover]

@billnitzberg

What issue is this is attempting to address? Unless there is a strong understanding of a known issue, it would be better to start by treating each job as a “regular job”: no caveats except the “only run one” behavior. (More caveats means more complexity means less resilience and less adoption – simpler is almost always better.) I would suggest dropping this (for now), and see what early adopters find as the real issues (ideally, during a Beta). Then, if there is an issue, fix it, and fix it right.

I just wanted to minimize the overhead of having more than one job from a job_set on the calendar. If scheduler adds more than one job then scheduler will reserve those resources and likely not run some other job which it could have. I get your point too, I will take this off the document

@billnitzberg

If there is not a compelling use case for handling requeues, an alternative to this would be to change the semantics from “run only one to completion” to “start only one”, and once one job in a set is started, delete the rest. This would make it easier to define what happens for some operations (e.g., how to handle qmove to another server aka peer scheduling), and would also likely reduce implementation and test effort. Again, as in the above, one would want to adjust based on early adopter feedback.

I’ll make this change in the document. It would really make things easy if we delete jobs as soon as one starts. I’ll keep this part of the change as “Experimental” since it can change based on feedback.

@billnitzberg

Interface 1: New Job attribute called “job_set” - qsub option “-s”

Do we really need a single character option? Why not just use -Wjob_set= as the only interface?
If 103.mycluster.co is a job_set and 104, 105, and 106 are members, can I submit a job that has -Wjob_set=104?

I’ll change the option to -Wjob_set that seems more readable. Subhasis had a similar question like you have of submitting a job where the job-id pointed by “job_set” option isn’t a job_set leader. I forgot to write this up, but, I think PBS server can just reject this job request. Users must know the job_set they are submitting to. What do you think?
While I am writing this, I think we need a command to just list down all the job_sets if we want users to submit to the right job_set. Otherwise, PBS server should just accept it and move the job under the right job_set (which in your example would be 103)

@billnitzberg

When a job is requested with multiple select specifications, PBS server will honor the queued limits set on the server/queue and run job submission hook on each of the resource specification. If one of the resource specification is found to be exceeding the limits then that resource specification will be ignored.

What is the output of such a command? Is it one job ID or many job IDs?
Not sure about ignoring an rejected request – shouldn’t the behavior be the same as if I submitted a job with qsub -Wjob_set= …, and wouldn’t that be to throw an error? What happens if all the requests are ignored?

Output of such a command will be one job id which will be the ID of the job_set leader. Regarding rejecting the request, internally server will throw an error but qsub will ignore the reject and move on to the next resource request. It can also print a message on stderr about why a resource request could not be submitted but that might break backward compatibility. Server, on the other hand, will surely log the reason of rejecting a job submission.

@billnitzberg

Interface 3: Extend PBS to allow users to submit jobs with a node-filter (nfilter) resource

Suggest “filter” instead of “nfilter”.

I wanted to keep it as nfilter because it signifies what it is going to filter. If we extend this filter mechanism to replace limits or queues, server we can then call it as jfilter. It’s because based on the prefix “n” or “j” the whole input that is going to be passed to the filter can be easily interpreted.

@billnitzberg

To access a specific resource out of resources_available, resources_assigned inputs, users must enclose each resource name within square brackets “” like this - “resources_available[‘ncpus’]”

Q: Is this the same syntax used in PBS hooks and the Scheduler’s job_sort_formula? (Ideally, we should have only one syntax.) Sorry, I just can’t recall this one… If it is the same syntax, I suggest making that statement explicit.

It isn’t same as job_sort_formula syntax. The reason is that formula just works on the resources requested by the job, so if the formula is like this “ncpus + 2 * mem” it is safe to assume that user is talking about resources requested by the job. In this case, we are exposing resources_available and resources_assigned on the nodes to the users. Both of these can have the same resource name, so we need a way to distinguish them.
Implementation wise this way of specifying filter can be easily interpreted in python if we expose two dicts (resources_assigned and resources_available) to it.

@billnitzberg

Accounting logs…

It would be good to capture the whole workload trace data in the PBS accounting logs (while also minimizing the impact on existing accounting post processing tools). At least some way to represent that a job is part of a job set, and some way to capture that a non-run member of a set has been removed from the queue. (This probably requires more discussion.)

Well, my opinion is why to take a different direction in accounting too. We can probably log job_set information is a job is part of a job_set but other than that it will just look like a bunch of jobs were submitted, one of them ran and others were deleted. This could happen in any normal day-to-day accounting logs too.
exposing job_set information in accounting record will give post processing tools a way to correlate things and make sense out of it.
What do you think?