Hi All,
I had posted my analysis and Proposed Design for PP-464 yesterday.
Request inputs/comments/signoff.
Thanks,
Prakash
These accounting_logs changes may caused problems with a feature that will be checked-in soon to OSS. It’s PP-388 and here’s the EDD:
In the EDD, the string value of accumulated string resources will be in JSON-format, which will have the quotes in them. Please take some of the examples and show how it would look under PP-464. For example,
resources_used.foo_str=’{“a”: 1, “b”: 2, “c”:1,“d”: 4}’
Also in the EDD, I see the following:
While reading the values, each escaped quote will become part of the value.
While reading the values, each occurrence of “\”, will be replaced with a ‘’.
While reading the values, the enclosing quotes (if any) will be dropped.
-> Who is reading the accounting_logs value? It can be many different utilities not under the control of PBS. Is this something that we document and tell accounting logs utility writer to follow?
Internally, tracejob tool reads the accounting files.
I believe, PBSA too might be reading them.
Yes, this is something that we would need to document for any possible external accounting logs utility writers to follow.
Regarding the conflict with that this change might have with the feature that you are working on, I am of the belief that these changes will be using the “values” and insert escape characters while storing them in the accounting logs. I do not think the format of the value will have any effect on these changes.
OR Am I missing anything obvious?
Thanks,
Prakash
Please change the title of this posting so that it matches all the existing design discussions. Design discussions should have a title that exactly matches the JIRA ticket, e.g.: PP-###:
Please remove the “EDD for” at the beginning of this one to be consistent.
Thanks!
@smgoosen’s questions
Hi Prakash,
I was thinking each step would have an example.
Questions I still have:
If there are no non-alphanumeric cahracters in the value then there are no quotes around the value in the accounting log?
When you say “While reading…” what do you mean? You specifically mean tracejob, right? Customer’s tools might have to be modified?
Thanks,
Sam Goosen
Could you please clarify, what you mean by each step.
correct. we need the quotes only if we have non alphanumeric characters in the value.
any tool/utility that reads the accounting logs file, would need to be
modified. This is what I had said when the discussions were happening
earlier, but the field wanted this solution to be implemented.
@smgoosen, I have updated the Proposed Design with examples for every statement. Could you please provide your comments/sign-off.
Thanks,
Prakash
The EDD looks good, thanks for adding the examples
1 Like
How will this work with PP-409? It’s EDD can be found at https://pbspro.atlassian.net/wiki/display/PD/PP-388%3A+Allow+mom+hooks+to+accumulate+resources_used+values.
Jon,
Until I am missing anything, PP-464 works on the values and how they are written and read from the accounting logs, so PP-409 and PP-464 should not interfere.
Thanks,
Prakash
@prakashcv13: Will PP-464 also modify the value recorded in resources_used? Then that could conflict with PP-409, which will have JSON values like:
esources_used.foo_str=’{“nine”: 9, “ten”: 10}’ resources_used.foo_assn2=’{“vn1”: 1, “vn2”: 2 ,“vn3”: 3 ,“vn4”: 4, “vn5”: 5, “vn6”: 6}’
@bayucan, PP-464 does not modify the value stored in resources_used.{resource_name}. It does change how these values get recorded in the accounting logs.
I went ahead and tested the PP-409 code with the code in PP-464 and I’m definitely seeing a difference in accounting_logs output…
For example, before:
resources_used.foo_str="’{“eight”: 8, “seven”: 7, “nine”: 9}’“
with PP-464, it’s now:
resources_used.foo_str=”’{“eight”: 8, “seven”: 7, “nine”: 9}’"
yes, but the value within PBS doesn’t change. It is only how these values get stored in the accounting logs.
I am not sure, if we are on the same page about how we perceive “conflict”. Could you please make me understand.
This is a problem. The point of making the output json was so that admins could easily read it in with a json reader. I took the output and tried to read it back in and it failed.
a = “‘{"eight": 8, "seven": 7, "nine": 9}’”
a
‘'{“eight”: 8, “seven”: 7, “nine”: 9}'’
import json
b = json.loads(a)
Traceback (most recent call last):
File “”, line 1, in
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py”, line 338, in loads
return _default_decoder.decode(s)
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py”, line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py”, line 384, in raw_decode
raise ValueError(“No JSON object could be decoded”)
ValueError: No JSON object could be decoded
When I brought this up months ago to Prakash and Sam on the forum I thought I was assured that it would not be a problem. Recommendations on a fix? Do we need to back out PBS-13646?
The issue is that a json formatted string should be able to read directly in. However, this fix breaks it. Now this would work if we removed the single quotes “\’{}\’”. But with the leading and the trailing double quotes it makes the string unreadable to the json module. We don’t want to have make any python/perl/etc script to have to do something special to read in a json string. That would defeat the purpose. One fix would be to have your code replace the single quotes if you are going to put leading double quotes. The other option is to not add leading and trailing double quotes if it starts and ends with single quotes.