# Tunable test parameters **URL:** \ **Category:** Developers\ **Created:** [April 2, 2019, 11:05am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535 "2019-04-02T11:05:02Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![visheshh](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/visheshh/32/209_2.png) [@visheshh](https://community.openpbs.org/u/visheshh)\ **Post date:** [April 2, 2019, 11:05am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/1 "2019-04-02T11:05:02Z")
Hi, I have posted an Design Document for adding a decorator in PTL to have modifiable test parameters specific to test cases. Please review and provide suggestions. Design Doc: [https://pbspro.atlassian.net/wiki/spaces/PD/pages/1198981123/Tunable+test+parameters](https://pbspro.atlassian.net/wiki/spaces/PD/pages/1198981123/Tunable+test+parameters) ---
**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\ **Post date:** [April 3, 2019, 8:56pm UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/2 "2019-04-03T20:56:55Z")
@visheshh how will it work if you wish to pass two different values in two different tests but the parameter name is same? If it does not work that way then do you recommend using unique parameter names to avoid collision? ---
**Author:** ![visheshh](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/visheshh/32/209_2.png) [@visheshh](https://community.openpbs.org/u/visheshh)\ **Post date:** [April 4, 2019, 4:48am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/3 "2019-04-04T04:48:05Z")
Hi @arungrover, In that case i would recommend test developer should make it unique to avoid collision . if they are not unique it will only be effective if only one test case is run at a time. I will make a note of it in Design doc. ---
**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\ **Post date:** [April 4, 2019, 9:36pm UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/4 "2019-04-04T21:36:45Z")
Now that I think more about it, this would be a wise recommendation to give but test writers may not know what parameters other tests are relying on (especially when the name collision happens across testsuites). How about we specify keys mangled with the class names in the benchpress command. Like a “test t1” in “Testsuite class A” has testparam decorator set to “@testparam(param1:p1, param2:p2)” and in the benchpress command users will specify “-p A.param1=x1:A.param2=x2” This way we can just handle it internally and not worry about collisions. ---
**Author:** ![visheshh](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/visheshh/32/209_2.png) [@visheshh](https://community.openpbs.org/u/visheshh)\ **Post date:** [April 5, 2019, 4:47am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/5 "2019-04-05T04:47:32Z")
@arungrover The idea sounds good. I had not thought about the collision across testsuites as i was focusing on Performance/Load tests using this decorator felt the testsuite’s would be always be run individually. If people use tags or run multiple testsuites, at that level i don’t think the test runner would try to modify the test specific parameters rather run with default values. But that is a possibility to be considered but i do not think it would be used much frequently. The end users for modifying the test params i beleive are the one’s who run these tests on their dev builds or while testing we want to scale up on cloud/big machines or scale down on dev vm’s. ---
**Author:** ![kjakkali](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/kjakkali/32/138_2.png) [@kjakkali](https://community.openpbs.org/u/kjakkali)\ **Post date:** [April 5, 2019, 8:41am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/6 "2019-04-05T08:41:12Z")
@visheshh : Instead of decorator and passing those parameter values from -p option. Can we create a separate config file (json) with same name as testsuite name. Ex: pbs\_smoktest.py pbs\_smoketest.json The config will have all the test parameters with default values. The test suite should load the config file at the start. When user wants to run test suite with different values , he can just edit the config file and run pbs\_benchpress. With this approach user don’t need to remember or type the parameters at command line. This will also take care name space collision and user can run multiple test suites or tags. Let me know your thoughts. ---
**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\ **Post date:** [April 5, 2019, 9:20am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/7 "2019-04-05T09:20:18Z")
I like this idea: using json config file ---
**Author:** ![visheshh](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/visheshh/32/209_2.png) [@visheshh](https://community.openpbs.org/u/visheshh)\ **Post date:** [April 5, 2019, 10:39am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/8 "2019-04-05T10:39:56Z")
@kjakkali Idea of having an optional JSON Conf file for the tests makes sense. We can store the values for testsuite and for each test cases in a hierarchical level. Then we could have same names for the variables and there would be no collision. ---
**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\ **Post date:** [April 5, 2019, 5:59pm UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/9 "2019-04-05T17:59:01Z")
I thought about a config file too, but I think the Idea here was to dynamically change the value of parameters when we invoke the benchpress command. If to change a parameter we have to maintain/modify a config file then why not have that configuration as part of the testsuite itself? Why do we have to create and maintain a separate configuration file for it? I guess, @visheshh can answer what is the motivation behind this proposal. I’d say if it is acceptable to change a file specify different test parameters then we should just have the test configurations inside the testsuite itself. ---
**Author:** ![visheshh](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/visheshh/32/209_2.png) [@visheshh](https://community.openpbs.org/u/visheshh)\ **Post date:** [April 8, 2019, 4:51am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/10 "2019-04-08T04:51:13Z")
@arungrover Motivation behind the proposal is to have test parameters modifiable without changing the test file . I know both test file and JSON file are files . But having it in a JSON file makes it easily machine readable and will help test management tools to provide an option to read it and it can be made modifiable at execution time. Saying that I agree having it in -p is quickly accessible and i feel it is most user friendly method for running single tests. I still feel most of the time when we try to scale up/down we run a single test case or test suite at the most. But if we plan to run more test suites and tags and when running through a test management tool having conf in JSON file would be more helpful to avoid collisions. ---
**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\ **Post date:** [April 8, 2019, 9:23pm UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/11 "2019-04-08T21:23:54Z")
I feel while using json file as input would be beneficial but this idea is not very well defined. It will raises questions on where will we keep these files, will it contain one testsuite or multiple, who will own these files, what will be its structure etc. Can you flush out this approach more before we decide which way to go? ---
**Author:** ![visheshh](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/visheshh/32/209_2.png) [@visheshh](https://community.openpbs.org/u/visheshh)\ **Post date:** [April 9, 2019, 10:18am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/12 "2019-04-09T10:18:24Z")
Hi @arungrover @kjakkali My idea is to have one json file per testsuite at the same location where test file exists. pbs\_smoketest.py will have a pbs\_smoketest.json Structure should be something like : { “TestSmoketest”: { “test\_t1”: { “x”: “y”, “a”: “b” }, “test\_t2”: { “x”: “z”, “a”: “c” } } } ---
**Author:** ![kjakkali](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/kjakkali/32/138_2.png) [@kjakkali](https://community.openpbs.org/u/kjakkali)\ **Post date:** [April 9, 2019, 5:28pm UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/13 "2019-04-09T17:28:51Z")
@visheshh : This looks good to me. ---
**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\ **Post date:** [April 9, 2019, 9:47pm UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/14 "2019-04-09T21:47:17Z")
I still feel that it is better not not to give input in a file. Test maintenance becomes difficult with additional configuration file. You will have to log results of the tests and their config json files. You will have to make sure in the test that the type and value of parameters is what the test expects (this is probably true whether you give json or not). I feel the configuration is better contained inside the testsuite itself or as a command line option. Another option is to read it out of the environment but doing that will also make debugging ambiguous because it will be hard to tell what test read out of the environment. ---
**Author:** ![vstumpf](https://avatars.discourse-cdn.com/v4/letter/v/dbc845/32.png) [@vstumpf](https://community.openpbs.org/u/vstumpf)\ **Post date:** [April 10, 2019, 12:06am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/15 "2019-04-10T00:06:06Z")
The configs can be added to the ptl\_test\_results.json file. Merge all of the configurations and add it to the object stored in that file. Or, just add the configuration to each testcase’s object in the json file. (i think this one is better) ---
**Author:** ![kjakkali](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/kjakkali/32/138_2.png) [@kjakkali](https://community.openpbs.org/u/kjakkali)\ **Post date:** [April 10, 2019, 3:51am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/16 "2019-04-10T03:51:48Z")
@arungrover : I am fine with both approaches. The only advantage i see in JSON is easy to use.The user don’t need to remember tunable parameters and no typos while mentioning in command line options. We don’t need any extra command line options. When user creates a new test suite - user will add a json file with default values. - test should read json file at the start - once test finishes it should record the tunable parameters in ptl\_test\_resutls.json at test case level If user wants to run test case with different values he just need to update json file locally and run the test case without any changes in command line. ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [April 10, 2019, 9:11am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/17 "2019-04-10T09:11:35Z")
As of now, the test configuration parameters passed in ‘-p’ or ‘–param-file’ are saved at pbs\_benchpress run level hierarchy; not at per test suite / test case level in the ptl\_test\_results.json file. ---
**Author:** ![saritakh](https://avatars.discourse-cdn.com/v4/letter/s/c68b51/32.png) [@saritakh](https://community.openpbs.org/u/saritakh)\ **Post date:** [April 10, 2019, 9:25am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/18 "2019-04-10T09:25:41Z")
In my opinion; cases where we tune test parameters in single run are not as common as default run. If we tune, we would be doing in multiple runs. Even if we tune and run, it is much more uncommon to run multiple tuned tests together in one run. Also, a json file per test suite becomes costly for maintenance. One workaround for that would be a template json file where the user fills in test suite name, test case name, key value pairs etc. We already have paramfile option in case of long list of parameters. I would prefer command line option. ---
**Author:** ![visheshh](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/visheshh/32/209_2.png) [@visheshh](https://community.openpbs.org/u/visheshh)\ **Post date:** [April 10, 2019, 10:46am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/19 "2019-04-10T10:46:20Z")
@vstumpf @saritakh, I was planning to have it per test case inside measurements as test\_config . As the test config is stored in self.conf . It can be added under measurements in teardown. ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [April 10, 2019, 10:55am UTC](https://community.openpbs.org/t/tunable-test-parameters/1535/20 "2019-04-10T10:55:03Z")
I don’t really understand what you mean by “test parameters”. The document doesn’t mention motivation, can you please edit the doc and mention what problem(s) this change will solve and give a few examples explaining the same? [Next page](https://community.openpbs.org/t/tunable-test-parameters/1535.md?page=2)