# PP-718: Add fairshare usage values to the job\_sort\_formula **URL:** \ **Category:** Developers\ **Created:** [March 21, 2017, 1:39am UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471 "2017-03-21T01:39:48Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [March 21, 2017, 1:39am UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/1 "2017-03-21T01:39:48Z")
I’ve created a design document for [PP-718](https://pbspro.atlassian.net/browse/PP-718): Add fairshare usage values to the formula. If people could give it a read over and give me comments, that’d be great. [Design Doc](https://pbspro.atlassian.net/wiki/display/PD/PP-718%3A+Add+fairshare+usage+values+to+the+job_sort_formula) Thanks, Bhroam ---
**Author:** ![billnitzberg](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/billnitzberg/32/6_2.png) [@billnitzberg](https://community.openpbs.org/u/billnitzberg)\ **Post date:** [April 13, 2017, 5:32pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/2 "2017-04-13T17:32:41Z")
This will be a great addition to the PBS Pro formula – thanks @bhroam! I suggest dropping “tree” from “fairshare\_tree\_usage”, so it is shorter and is more parallel with the name “fairshare\_perc”. (I think “usage” might also be confusing, as it connotes “amount used” – if I think of a better word, I’ll share it.) ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [April 13, 2017, 7:04pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/3 "2017-04-13T19:04:43Z")
@billnitzberg thanks for your comments. I chose “fairshare\_tree\_usage” instead of “fairshare\_usage” because the two are kind of different. We refer to an entity’s fairshare usage as their actual usage. This takes the entire tree into account, so I added tree in there. Usage does mean amount used. This is the usage in a normalized form (number between 0 and 1). The real usage number is the accumulated amount of usage calculated by the fairshare formula (with decay taken into account). If the word really bothers you, we could change it, but I think it’s OK. Bhroam ---
**Author:** ![mkaro](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/mkaro/32/85_2.png) [@mkaro](https://community.openpbs.org/u/mkaro)\ **Post date:** [April 13, 2017, 7:06pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/4 "2017-04-13T19:06:59Z")
Perhaps normalized\_usage would be a better choice? ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [April 18, 2017, 2:39pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/5 "2017-04-18T14:39:01Z")
@mkaro, with the suggested “normalized\_usage” is not obvious that it is directly tied to fairshare. “fairshare\_normalized\_usage” is a bit long for my taste, but it does convey the meaning more clearly, I think. @bhroam, in the section providing an example way to do a direct comparison between entities, the “entity’s target” term is really “fairshare\_perc” (as it correctly appears the second time), and so “2^-(fairshare\_tree\_usage / entity’s target)” should be updated to use consistent terminology. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 2, 2017, 10:53pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/6 "2017-06-02T22:53:34Z")
I’ve updated the EDD slightly. I’ve added a 3rd interface which is changes to pbsfs. The arbitrary usage will show up in both the default output and the pbsfs -g output. Bhroam ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [June 5, 2017, 1:29pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/7 "2017-06-05T13:29:50Z")
Hi Bhroam, I like the addition, but I am concerned that this change to default output may cause problems for people who have built scripts around the original pbsfs output format. I am concerned about this more for pbsfs than for pbsfs -g. Should we add a new option (-a?) to produce this output? Also, I am pretty sure the answer is “no”, but is there a way to obtain fairshare usage data through the API that we’d also need to consider with this change? ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 5, 2017, 7:55pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/8 "2017-06-05T19:55:27Z")
> [@scc](#): > > I like the addition, but I am concerned that this change to default output may cause problems for people who have built scripts around the original pbsfs output format. I am concerned about this more for pbsfs than for pbsfs -g. Should we add a new option (-a?) to produce this output? I hear your concern and agree that it could be an issue. I’m not sure adding a new option is the answer though. We’d then have two ways to print pretty much the exact same data. I doubt the new option would be used. I removed the new column from interface 3. You can still obtain the arbitrary usage via pbsfs -g for an entity. > [@scc](#): > > Also, I am pretty sure the answer is “no”, but is there a way to obtain fairshare usage data through the API that we’d also need to consider with this change? The IFL API talks to the server. The fairshare usage is in the control of the scheduler (at least for now). The pbsfs command directly talks to the usage file. So no, we don’t need to worry about the API. PTL is currently broken when it tries to reset the fairshare usage in revert\_to\_defaults(). It tries to remove the usage file. The file gets rewritten on the next scheduling cycle. To fix this, I have added interface 4. Right now when the scheduler receives a HUP, it will write out its view of the fairshare usage before rereading it again. This ways needed back when we didn’t write out the fairshare usage every scheduling cycle. I will modify this behavior so that it won’t write out the usage on a HUP. It will just read the current usage file. If changes were made via pbsfs, the scheduler will see them. PTL reverts to defaults by removing the fairshare usage file. I’ll modify the PTL library to HUP the scheduler after it does so. Bhroam ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 5, 2017, 10:39pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/9 "2017-06-05T22:39:13Z")
In a discussion with @billnitzberg, he pointed out that Interface 4 shouldn’t be part of this issue. It is a pbsfs/PTL issue and doesn’t have anything to do with fairshare in the formula. I happen to need to fix it to write PTL tests, but it should be tracked separately. I have filed PP-795 for this issue. ---
**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\ **Post date:** [June 20, 2017, 7:56pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/10 "2017-06-20T19:56:39Z")
I’m sure it is very late to comment on the design proposal now but just have 2 small observations - - IMO we should make is the value of formula (pow(2, -(fairshare\_tree\_usage/fairshare\_perc))) a pre-computed factor or at least give it a name and replace it while evaluating job\_sort\_formula. I’m not saying that we do not expose fairshare\_tree\_usage and fairshare\_perc. Doing this will make it simple for admins to set fairshare formula and also allow them to use a different formula is they want to. - Example of setting job\_sort\_formula using qmgr needs a change, it is missing “-(”. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [June 21, 2017, 6:27pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/11 "2017-06-21T18:27:31Z")
Thank you for your comments @arungrover. I have updated the EDD. I’ve replaced the term arbitrary usage with effective usage. The number is derived in a deterministic way. It isn’t arbitrary. I’ve reordered the interfaces to add interface 2. There is a new formula keyword ‘fairshare\_factor’. This is a shorthand keyword to make users of fairshare in the formula easier. If they want to use our suggested formula, they can use fairshare\_factor. If they want to use their own, they can use fairshare\_tree\_usage in their own formula. Lastly I have an example on how to add fairshare to the formula using math. The pow() function uses a comma which is a special character to qmgr. This requires extra quoting. Take a look. Bhroam ---
**Author:** ![scc](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/scc/32/234_2.png) [@scc](https://community.openpbs.org/u/scc)\ **Post date:** [June 22, 2017, 6:06pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/12 "2017-06-22T18:06:36Z")
Changes look good to me, thanks! ---
**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\ **Post date:** [June 26, 2017, 9:01pm UTC](https://community.openpbs.org/t/pp-718-add-fairshare-usage-values-to-the-job-sort-formula/471/13 "2017-06-26T21:01:53Z")
@bhroam your change looks good!