# PP-464: PBS Accounting Logs - Incorrect Escaping, Proposed Design posted **URL:** \ **Category:** Developers\ **Created:** [October 19, 2016, 11:42am UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293 "2016-10-19T11:42:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [October 19, 2016, 11:42am UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/1 "2016-10-19T11:42:58Z")
Hi All, I had posted my [analysis](https://pbspro.atlassian.net/wiki/display/PD/Analysis+for+PP-464) and [Proposed Design](https://pbspro.atlassian.net/wiki/display/PD/EDD+for+PP-464) for [PP-464](https://pbspro.atlassian.net/browse/PP-464) yesterday. Request inputs/comments/signoff. Thanks, Prakash ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [October 19, 2016, 9:14pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/2 "2016-10-19T21:14:57Z")
These accounting\_logs changes may caused problems with a feature that will be checked-in soon to OSS. It’s PP-388 and here’s the EDD: [https://pbspro.atlassian.net/wiki/display/PD/PP-388%3A+Allow+mom+hooks+to+accumulate+resources\_used+values](https://pbspro.atlassian.net/wiki/display/PD/PP-388%3A+Allow+mom+hooks+to+accumulate+resources_used+values) * * * In the EDD, the string value of accumulated string resources will be in JSON-format, which will have the quotes in them. Please take some of the examples and show how it would look under PP-464. For example, resources\_used.foo\_str=’{“a”: 1, “b”: 2, “c”:1,“d”: 4}’ ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [October 19, 2016, 9:20pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/3 "2016-10-19T21:20:01Z")
Also in the EDD, I see the following: While reading the values, each escaped quote will become part of the value. While reading the values, each occurrence of “\”, will be replaced with a ‘’. While reading the values, the enclosing quotes (if any) will be dropped. -\> Who is reading the accounting\_logs value? It can be many different utilities not under the control of PBS. Is this something that we document and tell accounting logs utility writer to follow? ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [October 19, 2016, 9:38pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/4 "2016-10-19T21:38:20Z")
Internally, tracejob tool reads the accounting files. I believe, PBSA too might be reading them. Yes, this is something that we would need to document for any possible external accounting logs utility writers to follow. ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [October 19, 2016, 9:41pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/5 "2016-10-19T21:41:41Z")
Regarding the conflict with that this change might have with the feature that you are working on, I am of the belief that these changes will be using the “values” and insert escape characters while storing them in the accounting logs. I do not think the format of the value will have any effect on these changes. OR Am I missing anything obvious? Thanks, Prakash ---
**Author:** ![billnitzberg](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/billnitzberg/32/6_2.png) [@billnitzberg](https://community.openpbs.org/u/billnitzberg)\ **Post date:** [October 20, 2016, 3:52pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/6 "2016-10-20T15:52:46Z")
Please change the title of this posting so that it matches all the existing design discussions. Design discussions should have a title that exactly matches the JIRA ticket, e.g.: PP-###: #. Please remove the “EDD for” at the beginning of this one to be consistent. Thanks! ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [October 20, 2016, 5:22pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/7 "2016-10-20T17:22:39Z")
@billnitzberg, made the change, is it ok now? Thanks, Prakash ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [October 26, 2016, 7:45am UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/8 "2016-10-26T07:45:12Z")
@smgoosen’s questions Hi Prakash, I was thinking each step would have an example. Questions I still have: If there are no non-alphanumeric cahracters in the value then there are no quotes around the value in the accounting log? When you say “While reading…” what do you mean? You specifically mean tracejob, right? Customer’s tools might have to be modified? Thanks, Sam Goosen ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [October 26, 2016, 7:46am UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/9 "2016-10-26T07:46:58Z")
> [@prakashcv13](#): > > I was thinking each step would have an example. Could you please clarify, what you mean by each step. > [@prakashcv13](#): > > If there are no non-alphanumeric cahracters in the value then there are no quotes around the value in the accounting log? correct. we need the quotes only if we have non alphanumeric characters in the value. > [@prakashcv13](#): > > When you say “While reading…” what do you mean? You specifically mean tracejob, right? Customer’s tools might have to be modified? any tool/utility that reads the accounting logs file, would need to be modified. This is what I had said when the discussions were happening earlier, but the field wanted this solution to be implemented. ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [November 11, 2016, 12:35pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/10 "2016-11-11T12:35:19Z")
@smgoosen, I have updated the [Proposed Design](https://pbspro.atlassian.net/wiki/display/PD/EDD+for+PP-464) with examples for every statement. Could you please provide your comments/sign-off. Thanks, Prakash ---
**Author:** ![smgoosen](https://avatars.discourse-cdn.com/v4/letter/s/53a042/32.png) [@smgoosen](https://community.openpbs.org/u/smgoosen)\ **Post date:** [November 18, 2016, 8:05pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/11 "2016-11-18T20:05:25Z")
The EDD looks good, thanks for adding the examples ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [November 28, 2016, 9:27pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/12 "2016-11-28T21:27:53Z")
How will this work with PP-409? It’s EDD can be found at [https://pbspro.atlassian.net/wiki/display/PD/PP-388%3A+Allow+mom+hooks+to+accumulate+resources\_used+values](https://pbspro.atlassian.net/wiki/display/PD/PP-388%3A+Allow+mom+hooks+to+accumulate+resources_used+values). ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [November 29, 2016, 4:28am UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/13 "2016-11-29T04:28:38Z")
Jon, Until I am missing anything, PP-464 works on the values and how they are written and read from the accounting logs, so PP-409 and PP-464 should not interfere. Thanks, Prakash ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [January 18, 2017, 7:58pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/14 "2017-01-18T19:58:22Z")
@prakashcv13: Will PP-464 also modify the value recorded in resources\_used? Then that could conflict with PP-409, which will have JSON values like: esources\_used.foo\_str=’{“nine”: 9, “ten”: 10}’ resources\_used.foo\_assn2=’{“vn1”: 1, “vn2”: 2 ,“vn3”: 3 ,“vn4”: 4, “vn5”: 5, “vn6”: 6}’ ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [January 18, 2017, 8:45pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/15 "2017-01-18T20:45:30Z")
@bayucan, PP-464 does not modify the value stored in resources\_used.{resource\_name}. It does change how these values get recorded in the accounting logs. ---
**Author:** ![bayucan](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bayucan/32/24_2.png) [@bayucan](https://community.openpbs.org/u/bayucan)\ **Post date:** [January 18, 2017, 8:50pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/16 "2017-01-18T20:50:51Z")
I went ahead and tested the PP-409 code with the code in PP-464 and I’m definitely seeing a difference in accounting\_logs output… For example, before: resources\_used.foo\_str="’{“eight”: 8, “seven”: 7, “nine”: 9}’“ with PP-464, it’s now: resources\_used.foo\_str=”’{“eight”: 8, “seven”: 7, “nine”: 9}’" ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [January 18, 2017, 8:54pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/17 "2017-01-18T20:54:40Z")
yes, but the value within PBS doesn’t change. It is only how these values get stored in the accounting logs. I am not sure, if we are on the same page about how we perceive “conflict”. Could you please make me understand. ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [January 18, 2017, 9:22pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/18 "2017-01-18T21:22:13Z")
This is a problem. The point of making the output json was so that admins could easily read it in with a json reader. I took the output and tried to read it back in and it failed. > > > a = “‘{"eight": 8, "seven": 7, "nine": 9}’” > > > a > > > ‘'{“eight”: 8, “seven”: 7, “nine”: 9}'’ > > > import json > > > b = json.loads(a) > > > Traceback (most recent call last): > > > File “”, line 1, in > > > File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/ **init**.py”, line 338, in loads > > > return \_default\_decoder.decode(s) > > > File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py”, line 366, in decode > > > obj, end = self.raw\_decode(s, idx=\_w(s, 0).end()) > > > File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py”, line 384, in raw\_decode > > > raise ValueError(“No JSON object could be decoded”) > > > ValueError: No JSON object could be decoded When I brought this up months ago to Prakash and Sam on the forum I thought I was assured that it would not be a problem. Recommendations on a fix? Do we need to back out PBS-13646? ---
**Author:** ![jon](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/jon/32/23_2.png) [@jon](https://community.openpbs.org/u/jon)\ **Post date:** [January 18, 2017, 9:22pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/19 "2017-01-18T21:22:35Z")
The issue is that a json formatted string should be able to read directly in. However, this fix breaks it. Now this would work if we removed the single quotes “\’{}\’”. But with the leading and the trailing double quotes it makes the string unreadable to the json module. We don’t want to have make any python/perl/etc script to have to do something special to read in a json string. That would defeat the purpose. One fix would be to have your code replace the single quotes if you are going to put leading double quotes. The other option is to not add leading and trailing double quotes if it starts and ends with single quotes. ---
**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\ **Post date:** [March 19, 2017, 4:01pm UTC](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293/20 "2017-03-19T16:01:23Z")
@jon, @smgoosen, I have updated the EDD, request you to go through the same and let me know if any changes are needed. [Next page](https://community.openpbs.org/t/pp-464-pbs-accounting-logs-incorrect-escaping-proposed-design-posted/293.md?page=2)