Enhancement of pbs_benchpress to run test case from non-tests directory

Developers
1

Hi All,

Please review the design document for pbs_benchpress to run from non-tests directory

https://pbspro.atlassian.net/wiki/spaces/PD/pages/1319829507/Enhancement+of+pbs+benchpress+to+run+from+non-tests+directory

Objective:
After PTL rpm Installation, the user has to cd to tests dir to execute test cases. We have -f option in pbs_benchpress but it supports only file. But we cannot use it to run a whole bunch of different tests.

when user runs pbs_benchpress -t testcase, it should do a path search ā€“ search current directory, next search test case in installed location.

Please let me know if any comments or suggestions.

Thanks,
Shilpa

2

Hi @shilpakodli,

Thank you for sharing the design doc.
A few comments -

  1. Design page should have a link to the forum discussion.
  2. There are many typos - like dircetory, pbs_benhcpress, Serached etc.
  3. The statement ā€œpbs_benhcpress -i option is also now can be runā€¦ā€ needs rephrasing.
  4. Examples would help in better understanding.

Thanks,
Prakash

3

Hi @prakashcv13, Thanks for the review comments.
I have updated EDD with examples and corrected typos.

Please have a look at it.

4

@shilpakodli I would suggest to rename variable name from PTLTESTS to PTL_TESTS_DIR.

5

How about PTL_TESTS_PATH?

6

PTL_TESTS_PATH seems to be more reasonable since path represents how to reach a filesystem object.

7

both are alright, but usually ā€œPATHā€ would support more than one directories, but here we are supporting only one. I would go for PTL_TESTS_DIR.
If we plan to expand the support to multiple directories in the future, then PTL_TESTS_PATH would be better.

8

I agree with @prakashcv13.

9

I agree with @prakashcv13. As we are supporting one test dir so, I prefer PTL_TESTS_DIR over PTL_TESTS_PATH

10

Thanks everyone. I will rename PTLTESTS to PTL_TESTS_DIR as we are supporting single tests directory as of now.

11

Iā€™m not sure I understand. You mean PATH can be set to path1:path2:path3 ? if yes then thatā€™s true for any environment variable right?

12

@shilpakodli, In scenario2
We are considering ā€˜PTL_TESTS_DIRā€™ dir as last option, since user has set it we should consider it before 'installed testā€™dir.

13

@agrawalravi90 - I am assuming, this question was for me. I will try to be more clear this time -
Usually PATH would mean we support path1:path2:path3.
As of now we are not supporting this, but if we plan to support this in the future, we should use PTL_TESTS_PATH or for that matter PTL_USER_TESTS_PATH.
If we do not plan to support multiple directories even in future, we should use PTL_TESTS_DIR or PTL_USER_TESTS_DIR.

I am leaned towards including ā€œUSERSā€ in the name as we already have implicit paths that pbs_benchpress would search.

@shilpakodli - could you please also add some detail on the motivation behind adding this support. Based on that we could decide/discuss on the order in which pbs_benchpress should search for the tests.

14

@shilpakodli - I have a question. What would happen if the user sets the environment variable to have more than one directory?

15

Ok, so you are saying that there would be an expectation from the users that PTL_TESTS_PATH will accept multiple cascaded paths. I see your point, but I donā€™t think itā€™ll be an issue, we just need to document that it accepts only 1 directory. But if you strongly feel about it, then Iā€™m ok with PTL_TESTS_DIR.
I donā€™t think we need to add the word USERS in there, we donā€™t do that for any of the other PBS environment variables. Letā€™s keep it short, otherwise itā€™ll be a pain to use.

16

@prakashcv13 - If we give multiple paths in PTL_TESTS_DIR then it wonā€™t be able to search test case since, I am passing entire value of PTL_TESTS_DIR as tests directory path.

17

@shilpakodli - thank you - how about adding this as scenario 4?

18

@agrawalravi90 - I agree that adding ā€œUSERā€ to the name would make it longer. The design is clear about the order in which we would be looking for tests, so ā€œUSERā€ is implicit.
@shilpakodli - I agree with @kjakkali wrt the order. There is a possibility that a user writes a testsuite with a name identical to one in PTL installation directory, the user would expect his version of the testsuite to run.

19

Sure, we can have one scenario of having multiple paths in PTL_TESTS_DIR where it will throw error by saying ā€œUnknown testsuite/testcaseā€

20

@shilpakodli : The latest EDD looks good. I sign off.