The case where I worry that busy vs. used might be confusing is that the new “last busy time” attribute will be updated even in cases where the node was never actually “busy” (nor job-busy", etc.). I still feel that used is less confusing.
Thanks for updating the format of the attribute, what you have is great.
As for making it a resource, I don’t actually think it SHOULD be a resource, an attribute is much more fitting, but it would be a good addition to node_sort_key (one which we have implemented using hooks and custom resources at customer sites in the past, though it does not scale well on large systems with lost of job turnover as the hook needed to contact the server to update the nodes resource). What about special casing it for node_sort_key just like we currently special case the node attribute “Priority” to be exposed as “sort_priority” in job_sort_key (see find_node_amount() in src/scheduler/sort.c and under “/* resource names for sorting special cases */” in src/scheduler/config.h)?
I am not purposefully trying to introduce scope creep here
, but I think it is worth discussing at the same time that we are introducing this new interface.
When a new vnode is added and has not yet run any jobs or reservations, what will the value of this time stamp be? I only mean when a vnode newly appears in the node list since I assume that this time stamp data is persistent between mom and server restarts. The obvious choice to me is the time at which the pbs_mom (or “first pbs_mom” in the case of a Cray system not using vnode_pool) first starts reporting the vnode to the server as “free”. I think the behavior should be explicitly stated in the EDD.
Finally, on this same interface, we should make sure that this new time stamp interface gets updated when vnodes are released early from a running job (recently implemented PP-339 and PP-647). The current wording is “end of any job”, which could be interpreted in 2 ways since now a job can “end” in an important sense on a sister node well before the job actually “ends” completely.
Questions / Comments on other parts of the EDD aside from interface 5:
Interface 8: the summary leads me to believe that this is to introduce an entirely new PBS_ hook, not adding a new event trigger to an existing hook. Even though I can understand that the fact that it is using the same existing script (with added scripting, of course) may be am implementation detail, I still find it confusing. What about “Interface 8: Server periodic action added to pbshook PBS_Power” for a summary?
Interface 9: I think this interface is trying to do 2 things: introduce a new hook event (power_provisioning) AND say that the existing PBS_power hook will be modified to utilize this new event. If that understanding is correct, then it needs to be broken up 2 interfaces, one similar to interface 8 that says “power_provisioning action added to pbshook PBS_Power”, and another that talks only about the new hook event.
On the topic of the above 2 points: is it proper to actually describe things in this way in the EDD? Isn’t it an implementation detail (internal design) that ramping the nodes up and down is accomplished with the PBS_power hook by adding new periodic actions to the hook? Saying something like “at most $SVR_ATR_max_ramprate_limit nodes will be ramped down every $freq seconds)” and “at most $SVR_ATR_max_ramprate_limit nodes will be ramped up in anticipation of use” (with more detail, of course) more accurately describes the external behavior that can be relied on.
Interface 9: The interface for the adding the new hook event itself needs to include more detail on what various PBS attributes/objects are available within the hook event, caveats of use, when it is triggered, etc. (like in the “PP-425 to PP-434” EDD for example).
Interface 10: line 3 in the table should say “Nodes are being ramped up”, not “down”.