Test Framework should allow testers to run a test for multiple times

Developers
1

Hi,
I have posted a design document to run a test for multiple times on test framework.
Please review and provide suggestions.
Design Doc:
https://pbspro.atlassian.net/wiki/spaces/PD/pages/1386938446/Test+Framework+should+allow+testers+to+run+a+test+for+multiple+times

2

@birbalsain I suggest to add a new option (like -r and/or --repeat-tests) to pbs_benchpress instead adding new key-value in -p.

3

Thanks @birbalsain for suggesting this feature.

I’m also agree with @hirenvadalia for the new option in pbs_benchpress.
And also can you add few details about the output of the repeated tests.

  1. Will there be run instance number associated in the results file(for e.g., ptl_test_results.json) ?
  2. If a test fails during run then PTL will stop or continue with next run ?
4

@birbalsain Does this work for single test case or we can pass multiple tests ?

5

It will work for all.

6

Anybody have any idea where from should i restart the test? In PTLTestRunner class, I am calling startTest method from _set_test_end_data method but test is not restarted.

7

@vishwaks , @hirenvadalia , @shilpakodli please review the updated design document.

8

@birbalsain In Details section, third point says " If repetition is 3 and test fails at 2nd run then it mark that as fail and will go to next run after the given test repetition delay" , which means, if the third run passes then overall test result shows as test has passed ?

9

@shilpakodli After each repetition we get summary of that repetition and at the end of all repetitions of all test-cases, We gets summary of all test-case but consider last repetition of each test-case.

11

Hi, we will get one log message after each repetition “test: test-case-name repeated 1 times out of required 3 times”

12

In updated design, we get summary(passed/failed/skipped) of all repetition except last repetition, just after the all repetition of that test-case but Last repetition summary we get at the end.

13

@kjakkali @hirenvadalia @vishwaks @visheshh @shilpakodli Please review the updated design.

14

@birbalsain: Below are my comments:

  1. In the example you have shown, after each test completing it’s repeat cycle shows summary 2 times. Why ?
  2. Can you add an example where when I gave repeat as 3 and test failed for first 2 times and passed for 3rd time, what is the summary we get after repeat cycle completes and after all tests done ?
  3. Why do we need delay between repeat cycle for a test ?
  4. I think it’s better if we update our overall summary(after completion of all tests) to include repeat cycle summary looks good, otherwise it is very hard to go through ptl logs when the test logs are big.

4th point is my suggestion, if it’s complex to do now and others are okay with showing repeat cycle summary in the logs after each cycle(repeat), then you can ignore that comment.

15

1.repeat_tests_delay is configurable, by default it is 5 sec.
2.In PR, I have attached log for all case skipped/failed/passed.
3. If we want overall summary at the end then it will consider repetition as a test count, Means if 3 test repeat 3 time then we get summary out of 9.

16

@birbalsain: thanks for the quick answers.

  1. still why do we need delay of default 5 ? can we keep default delay 0 ?
  2. Added comments on PR for this.
17

@vishwaks yes, we can make default delay as 0. For overall count i discussed with @kjakkali, he suggested me to don’t include repetition in final count.

18

@birbalsain: In case of repeat-test > 1 then show the test summary after each run and if all repeat instances are passed then only it should be counted as passed in overall test results, otherwise it should be failed/error/timedout/skipped based on repeat instance status. This will make users to check each instance result, otherwise (incase of all passed at last instance) there are chance they might ignore previous instance results.

Or else I suggest stop the test if any repeat instance hit other than passed, then users will see this instance result at the end.

Let me know what you think, we can discuss it offline if required.

19

@hirenvadalia @vishwaks @visheshh @kjakkali Please review the design.Now we get summary at the end of all test. like currently we are getting. Final counts includes all the repetition.

20

Thanks @birbalsain. EDD looks good to me.

21

@hirenvadalia ok, updated design is using the new option “–repeat-tests” to repeat test.