PP-484: PP-485: Enhance output of qstat to make it more admin and script friendly

Developers
1

Hi All,

I have posted an EDD ( EDD for enhancing qstat output ) for the proposed changes to enhance the output of qstat to make it more admin and script friendly. Please review and provide feedback.

Thanks,
Nithin.

2

Hello Nithin, thanks! I have a few comments:

  1. Jon ended up performing much of this work previously for ticket PP-483, see https://pbspro.atlassian.net/wiki/pages/viewpage.action?pageId=31621169. That should be either be used as the starting point or allowed to stan on its own, depending on what we decide in comment 2) below. I am sorry if I didnā€™t mention the existence of this EDD previously.

  2. I believe that filing PP-508 as a user story was unnecessary. I had previously filed these tickets surrounding the planned qstat changes:

PP-483
As a user or admin, I would like qstat to return a full job attribute list with 1 line per job attribute, so that I can more easily use tools like grep to find the information I am looking for regarding my jobs

PP-484
As an admin or user, I would like qstat to be able to output full job attribute information in JSON format, so that I can more easily write scripts consuming this information

PP-485
As an admin or user, I would like qstat to be able to output full job attribute information in delimeter separated value format, so that I can more easily write scripts consuming this information

We can either change PP-508 to be an epic that encompasses those three tickets, or close it altogether. Before I take any action like that, was one of those options your original intention? And if so, was your plan to add the JSON and DSV interface options to this same EDD page?

  1. About Interface 1 handling Resource_List.select: I think that breaking up the select statement like this distorts and misrepresents the resource value and we should just print it in normal select statement format like we do in the present behavior, we should just not break it up into multiple lines if it is longer than 78 characters if this new option is specified.

  2. Same comment as 3) except pertaining to schedselect attribute.

  3. Similarly, I believe that we should leave Variable_List as all one line as it is today, but simply not wrap it if it exceeds 78 characters.

  4. The actual syntax used needs to be ā€œqstat -f1ā€ , rather than ā€œqstat -f 1ā€ (no space vs. space). The former can mean ā€œfull output with one line per attributeā€, but the later already means ā€œfull output with line breaks for job 1.servernameā€.

3

Thanks Scott for the update.

I understand that this EDD is redundant in nature. So you can close the ticket-PP-508.
You can ignore this EDD until there is a new update.
Sorry for the inconvenience.

4

IMHO, PP-484 is the only one that matters for now. We might consider implementing the others in the future. We may also find that the others are unnecessary once PP-484 is complete.

5

Iā€™ve updated the EDD based on the comments received. Please review.

6

Hello Nithin, thanks for the update. There is one problem I see with the json example though. The ā€œ{ā€ here is incorrect:

 "pbs_server":"vbox", 
 { 
      "Jobs"

And once that is removed a ā€œ}ā€ needs to be removed from the end.

FYI, this is a handy tool when mocking up json: http://jsonlint.com/

7

Hi,

I wanted to ask about the design for the option arguments (-Fjson and -Fdsv), and make sure the design follows standard option argument practice, as itā€™s not explicit in the design. Since the option arguments do not use ā€œā€“ā€, Iā€™m assuming the option arguments are not ā€œFjsonā€ and ā€œFdsvā€, but ā€œFā€ alone with a required optarg.

Just want to verify that it is OK to have a space after the -F and before the optarg, e.g., the following are the same: ā€œqstat -Fjson ā€¦ā€ and ā€œqstat -F json ā€¦ā€.

Thanks!

8

Hi Bill, good question, and yes, things like this work with pbsnodes today, so the expectation would be that qstat handles the arguments in the same way:

pbsnodes -av -F json

pbsnodes -avF json

pbsnodes -av -F dsv -D ,

pbsnodes -avF dsv -D ,

1 Like
9

Thanks, Bill and Scott.

Iā€™ve updated the EDD with the following -

  1. Removed JSON object opening before ā€œjobsā€ and the corresponding closing tag.
  2. Introduced a space between -F and optarg.
10

Looks good to me, thanks! I sign off.

Thanks,
Scott Campbell

11

Looks good to me , You have my sign-off too.

Thanks,
Brem

12

Iā€™ve updated the page with a ā€œNOTE:ā€ confirming that json and dsv will work only with the full set of attributes (while using ā€œ-fā€ option). Please share your thoughts.

13

Hi Nithin,

EDD looks good. Below are few comments:

  1. on -F dsv - What happens if a user give delimiter ā€˜,ā€™ and an value of attribute contains it ? It will be escaped ?
  2. on -F json - its better to provide all values in string format and requester will caste it accordingly. e.g., timestamp, Priority etc.
14

And another thought. Its better to have it(json and dsv output) for below options also:

qstat -Bf - for server information
qstat -Qf - for Queue information
pbs_rstat -f for Reservation information

Thanks,

15

Good suggestion. We can print with escape character if the delimiter itself is appearing on the value.[quote=ā€œvishwaks, post:13, topic:344ā€]
on -F json - its better to provide all values in string format and requester will caste it accordingly. e.g., timestamp, Priority etc.
[/quote]

The pbsnodes output in JSON format considers numeric values as a number (http://www.json.org/). The same is followed in qstat.

16

This will be handled in the present enhancement. All the options of qstat with ā€œ-fā€.

17

Good to know buddy. This will be helpful for PTL as well :+1:

18

On the EDD page, could you please add a link to the community discussion.
Also, for the ā€œconflicting errorsā€, would you be adding a new message or an existing one will be reused?

19

qstat: conflicting options. is an existing error message. Iā€™ve updated the EDD to clarify this. Also provided a link to community discussion.
Thanks.

20

Sorry I have not replied sooner. I am not used to looking for EDD in user/site area. I would expect them to be in the developer area. Anyways, one concern that I have with doing json with qstat is that user environment variables can have characters that can break the json output. Are you doing anything special to ensure that this does not happen? If so, what?