# Design for PTL test execution report in JSON format **URL:** \ **Category:** Developers\ **Created:** [August 27, 2018, 8:49am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157 "2018-08-27T08:49:04Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [August 27, 2018, 8:49am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/1 "2018-08-27T08:49:04Z")
Hi All, Please review the new design for “PTL test execution report in JSON format” posted at: [https://pbspro.atlassian.net/wiki/spaces/PD/pages/629014541/PTL+test+execution+report+in+JSON+format](https://pbspro.atlassian.net/wiki/spaces/PD/pages/629014541/PTL+test+execution+report+in+JSON+format) Please let me know of any comments or suggestions. Thanks, Sarita ---
**Author:** ![kjakkali](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/kjakkali/32/138_2.png) [@kjakkali](https://community.openpbs.org/u/kjakkali)\ **Post date:** [August 27, 2018, 9:23am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/2 "2018-08-27T09:23:25Z")
Thanks Sarita, What will be contents of test\_conf ? Can you add an example? ---
**Author:** ![sujatapatnaik52](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@sujatapatnaik52](https://community.openpbs.org/u/sujatapatnaik52)\ **Post date:** [August 27, 2018, 9:33am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/3 "2018-08-27T09:33:38Z")
Hi Sarita, It seems the key value pairs are constant in the json. Just a quick question, If I want to add any additional info about a test case in some string format or so, which of the keys mentioned in the json format I can use? ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [August 30, 2018, 8:56am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/4 "2018-08-30T08:56:03Z")
Thanks for your inputs @kjakkali and @sujatapatnaik52, I have updated the document with your comments incorporated. ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [August 30, 2018, 8:58am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/5 "2018-08-30T08:58:50Z")
@All, Please review the updated design document. Below are the changes done; based on comments received along with some additional ones: 1. Updated the terms with string specifics and angular braces for values (\<\>) 2. Added explanation of string specifics of json file 3. Added multiple test cases & test suites to template 4. Added example file 5. testparam was a repeated as test\_conf, hence removed testparam 6. Total test summary subsection and its fields added 7. ‘measurements’ replaced with ‘test\_run\_info’ since it was a misnomer for use cases where the test gave out a string format of information of test execution ---
**Author:** ![kjakkali](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/kjakkali/32/138_2.png) [@kjakkali](https://community.openpbs.org/u/kjakkali)\ **Post date:** [August 30, 2018, 9:06am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/6 "2018-08-30T09:06:44Z")
@saritakh : Thanks. Updated design looks good to me. ---
**Author:** ![sujatapatnaik52](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@sujatapatnaik52](https://community.openpbs.org/u/sujatapatnaik52)\ **Post date:** [August 30, 2018, 9:12am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/7 "2018-08-30T09:12:01Z")
Thanks @saritakh. The changes looks good. ---
**Author:** ![vishwaks](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/vishwaks/32/74_2.png) [@vishwaks](https://community.openpbs.org/u/vishwaks)\ **Post date:** [August 30, 2018, 11:39am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/8 "2018-08-30T11:39:44Z")
Hi Sarita, Inside test summary, we have result summary, I suggest to have total\_run, succeeded, failed, errored etc as separate keys instead of a string. And I have few questions 1. “tests with failures” means failed, errored and timedout tests or only failed tests. 2. In “test\_run\_info”, can a testwriter append with custom values per testcase for e.g., def test\_t1(self): test-steps self.test\_run\_info.append({custom:value}) ? 3. How the requirements info will be fetched per testcase? ---
**Author:** ![anamika](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@anamika](https://community.openpbs.org/u/anamika)\ **Post date:** [August 31, 2018, 7:15am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/9 "2018-08-31T07:15:13Z")
Hi @saritakh, Thanks for working on the design. It looks good to me. Have few minor comments. -typo “dictinary” at few places -tags can be specified at test suite level as well, right? I have seen a few tests like that. how will they be represented in JSON format? - I agree with Vishwa having separate keys for total\_run, Pass, failed, error, skipped and timeout. - Can you add examples for test\_run\_info? I think I am not completely clear on how to use that? Also JSON file will not contain the actual output of the test but only test info, correct? ---
**Author:** ![suresht](https://avatars.discourse-cdn.com/v4/letter/s/ec9cab/32.png) [@suresht](https://community.openpbs.org/u/suresht)\ **Post date:** [August 31, 2018, 10:42am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/10 "2018-08-31T10:42:56Z")
@saritakh Thanks for posting the design document. Follow are my comments. 1. Can we make the epoch time in readable format for run\_id like 083120181546\_run something like this. We can also have one more field like total\_time\_in\_epoch to purely store overall time needed in seconds. 2. Instead of having status as string can we have a sub dictionary for it something like “test\_results\_by\_type”: { “run”: 5, “succeeded”: 3, “failed”: 1, … } also can this summary be created per test suite level as well? 3. In test\_summary dictionary can we have test\_start\_duration and test\_end\_duration instead of just having the overall test\_duration. This way it is more helpful. Otherwise we need to parse each and every test case results individual dictionaries to achieve the same. 4. It would be great to provide an example for test\_run\_info dictionary. Also please provide an example for requirements dictionary. Thanks, Suresh ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [August 31, 2018, 10:44am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/11 "2018-08-31T10:44:25Z")
@All, I have updated the design document based on review comments and new ideas, please review the same and let me know of any comments. Below are the updates done: 1. Updated the field explanations in the template instead of a separate list 2. Added a dictionary for result\_summary for count of tests 3. Updated tests\_with\_failures explanation as tests that failed and errored out 4. Replaced “test\_run\_info” back with “measurements”, since a new method is defined to add additional information to the json report 5. Updated new method to set the “measurements” list of dictionary 6. Added new method to add additional information of test execution result as a dictionary ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [August 31, 2018, 10:45am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/12 "2018-08-31T10:45:08Z")
Thanks @vishwaks for your inputs, please find my replies below: 1. Updated the field name explanation for tests\_with\_failures accordingly 2. Below would be a sample of usage of set\_test\_measurements() method def test\_t1(self): test-steps self.set\_test\_measurements({custom\_fieldname1:value1,custom\_fieldname2:value2}) 3. Requirements information i.e. cluster information would be fetched from the @requiremnts decorator specified at the test case. Refer to this link for more details: [https://pbspro.atlassian.net/wiki/spaces/PD/pages/458883073/PP-1281+New+decorator+in+PTL+using+which+user+can+provide+cluster+information+required+for+a+test](https://pbspro.atlassian.net/wiki/spaces/PD/pages/458883073/PP-1281+New+decorator+in+PTL+using+which+user+can+provide+cluster+information+required+for+a+test). Please let me know of any clarifications ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [August 31, 2018, 10:45am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/13 "2018-08-31T10:45:39Z")
Thanks @anamika for your inputs, please find my replies: - updated the typos - Ultimately the tags end up getting applied to the test case when specified at test suite level. Here all the tests will have the same value for “tags” field - updated methods should clarify how the data gets added to the json report. Please let me know of any clarifications. ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [August 31, 2018, 10:46am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/14 "2018-08-31T10:46:49Z")
@sujatapatnaik52, The new method PBSTestSuite.add\_to\_test\_run\_data() satisfies the requirement mentioned by you. So now, one can create a dictionary in test code keeping in mind the hierarchy of the json report template; with whatever data to be filled; that is needed to be added to json report in string formats and passed to this method. Please let me know of any clarifications ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [August 31, 2018, 10:50am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/15 "2018-08-31T10:50:29Z")
Thanks @suresht for your comments, I just now updated the document and missed your points by few minutes. As I see, the new changes satisfies your second comment. Regarding comments 1, 3 & 4; I will get back to you on them as early as possible. ---
**Author:** ![vishwaks](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/vishwaks/32/74_2.png) [@vishwaks](https://community.openpbs.org/u/vishwaks)\ **Post date:** [August 31, 2018, 11:27am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/16 "2018-08-31T11:27:40Z")
Thanks for the information. So in requirements section data will be populated by requirements decorator, in that case I have an question, in future requirements decorator might get more parameters and multiple logical operators support(=, !=, \<,\> etc), in requirements section in json info can logical operators will be populated? If not we have to I suggest to add it. ---
**Author:** ![anamika](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@anamika](https://community.openpbs.org/u/anamika)\ **Post date:** [August 31, 2018, 4:47pm UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/17 "2018-08-31T16:47:22Z")
Thanks for making the changes Sarita. ### **Interface: PBSTestSuite.set\_test\_measurements()** –\> what attributes it is going to set? Do you have the list or it takes as an argument? A test writer might want to access that data inside the test as well. Do we need another helper method like get\_test\_measurement? ### Interface: PBSTestSuite.add\_to\_test\_run\_data() –\> examples please. Also specify if this takes any arguments. ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [September 3, 2018, 9:47am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/18 "2018-09-03T09:47:48Z")
@All, I have updated the design document with below, please review and let me know of any comments: - Updated method 3 name as “add\_additional\_data\_to\_report()” instead of “add\_test\_run\_info()” - Added sample test code and updated the sample json report accordingly ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [September 3, 2018, 9:50am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/19 "2018-09-03T09:50:45Z")
Thanks @suresht for your inputs, please find my replies below and let me know of your views: 1. I think epoch time in readable format is not necessary since epoch time is just taken as a run identifier number in order to differentiate between 2 test run reports. Otherwise I do not think this data is needed anywhere else. I prefer to keep it as is. 2. Already updated as sub dictionary. I dont think it would be necessary to create summary at test suite level; also if added it would become duplicate data; since each test case gives out its result separately. Of course in the cases where the failure numbers are high in a test run, the test suites should be run individually in order to debug. 3. I think test start time and test end time belong more to be part of test logs than part of test report. test duration is specified for the whole test run and test start & end time is given for individual test cases in order to search the test logs in case of failed test case. 4. I have added an example for both the methods being called in test case code. ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [September 3, 2018, 9:52am UTC](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157/20 "2018-09-03T09:52:30Z")
Thanks @anamika, please find my replies below: #### Interface: PBSTestSuite.set\_test\_measurements() –\> what attributes it is going to set? Do you have the list or it takes as an argument? A test writer might want to access that data inside the test as well. Do we need another helper method like get\_test\_measurement? **Sarita ==\>** Updated the details that it accepts a dictionary. The dictionary can contain any key value pairs and is completely upto the test writer to fill it up with any type of these pairs. I dont think that we need a helper method to get this data since it would already be part of the array and dictionary, which is locally accessible. #### Interface: PBSTestSuite.add\_to\_test\_run\_data() –\> examples please. Also specify if this takes any arguments. **Sarita ==\>** Updated the method name as add\_additional\_data\_to\_report() along with details that it accepts a dictionary I have added an example of the method usage int est code. [Next page](https://community.openpbs.org/t/design-for-ptl-test-execution-report-in-json-format/1157.md?page=2)