I hope you are all well. I am looking for advice to improve my project planning and management of resources methods.
I currently manage a mid sized hpc group that can handle a wide range of calculating activities; from brie;, single node tasks to extended; multi node models. Although I am successfully setting up the OpenPBS environment and can submit and monitor jobs, I am facing a few issues that I hope experienced users can help me with.
I want to be sure that essential jobs are given preference without postponing fewer important tasks. I have heard about constructing job groups that have different preferences; but I am not sure how to balance it successfully. Can you share your job scheduling strategies and arrangements?
Usually I find services that are unemployed and fully booked. What are some great methods to use to create spending methods which advertise maximum use? Are there any ideas for setting networks and CPUs to avoid resource wasting and ensure proper sharing?
According to my understanding; it may greatly improve productivity by putting smaller jobs into inactive spaces left by more complex tasks. Yet; there is not much guidance on how to put this up. Could somebody provide me with a detailed example of how to set up and complete OpenPBS?
In a multiple users setting maintaining equal opportunity is essential for making sure no one user controls resources. What are some effective OpenPBS policies and methods for ensuring user equality and how do you monitor and change them over time?
I am searching for advice on the best tools and methods for monitoring work performance and resolving difficulties with OpenPBS. Are there certain records; programs, and third party tools you suggest to maintain effortless activity?
Please find my answers, also based on the demand usage trend, urgent jobs, hpc politics/circumstances you would have many options to tune the cluster to satisfy the business requirments.
Make sure walltime is requested with each and every job. Jobs (using hooks and with a reject message mentioning why the job is rejected and what should be included in the job request) are rejected when there is no walltime requested or users requested more than what is available on the cluster in total. Please check the job_sort_fomula + strict ordering + backfill from the admin guide and its examples. If all the jobs have walltime request (near estimated or near accurate, then scheduler prediction/estimation would be correct).
job-sort_formula, strict ordering, backfill, walltime request , -l place=free,pack, , select statement would help. Also, it is more open question, some create queues (qlist) for singlecorejobs, multinode jobs, interactive jobs and urgent jobs, with restriction of walltime on respective queues. Some use reservations, to reserve resources for emergency jobs.
As a consequence of different HPC clusters/groups having different stakeholders, it is unusual for clusters to have the same exact policy.
That being said, an approach to arrive to the currently preferred policy for a given HPC resource/set of clusters, work as follows:
list down the list of stakeholders or stakeholder groups
assume all possible sources of workload i.e. job tasks arrive at an infinite rate from each and every possible source, for several weeks or more.
draw your assumptions about system stability/perturbation (f.i. nodes failure rates)
then define who will gets a chunk from the (limited) resources available and at what percentage (or partition) and who gets queued under these most extreme conditions. Read: some of the stakeholders are usually paying the infrastructure, you would rather keep them in good balance or, you get queued in a next funding round.
N.B. The way to satisfy the business requirements may change from one scheduling system to the next and sometimes even among versions of the same PBS implementation. Consulting the scheduling manual is essential at this stage to validate the proposed solution.
Assuming a day arrives that the first group of tasks is serviced, iteratively go down the list of workload sources until it is exhausted.
At this time you might have a plan which is defensible; not near the end yet: communicate the plan to stakeholders and set up a process/path for revision - needs and circumstances will change over time.
Good luck, you will need some of it.
Having said that, I am the original author of the qtop tool:
It was put together to solve exactly this problem and it was first used within several LHC/LCG clusters running several HEP projects in coopetition! If you attempt to use the tool, prefer to try out the very latest branches against more recent versions of PBS and if, need be, reach out to me via linkedin for whatever extra help needed. Furthermore, qtop may support other queueing systems, it is designed to be very modular.
Also, if you are a python volunteer in seek of an interesting project in need of a decent CI/CD push, you are more than welcome to tag along. I have the gitlab expertise to help it but cannot own the testing effort during this period.
to improve our cluster job scheduling and resource management processes.