# PP-302: Implement save of PBS data for post-run analysis

**URL:** <https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380>\
**Category:** Developers\
**Created:** [December 8, 2016, 8:25am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380 "2016-12-08T08:25:58Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\
**Post date:** [December 8, 2016, 8:25am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/1 "2016-12-08T08:25:58Z")

</div>

Hi @developers,

There is new design document is available for review:  
[https://pbspro.atlassian.net/wiki/display/PD/PP-302%3A+Implement+save+of+PBS+data+for+post-run+analysis](https://pbspro.atlassian.net/wiki/display/PD/PP-302%3A+Implement+save+of+PBS+data+for+post-run+analysis)

Please have look and provide your comments/suggestion to community.

Thanks,  
Hiren

---

<div class="post-metadata">

**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\
**Post date:** [December 9, 2016, 5:33am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/2 "2016-12-09T05:33:21Z")

</div>

Hiren,

Having the core file for post analysis may also help.

Thanks,  
Prakash

PS: Please add a link on the design page to the community discussion.

---

<div class="post-metadata">

**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\
**Post date:** [December 9, 2016, 5:43am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/3 "2016-12-09T05:43:43Z")

</div>

@prakashcv13 AFAIK core file will be created in PBS\_HOME/\*\_priv directory and PBS\_HOME is already in post analysis data.  
I think I had already add link to design page in this discussion see first post from me.

---

<div class="post-metadata">

**Author:** ![prakashcv13](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/prakashcv13/32/65_2.png) [@prakashcv13](https://community.openpbs.org/u/prakashcv13)\
**Post date:** [December 9, 2016, 5:46am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/4 "2016-12-09T05:46:41Z")

</div>

The request is for a link from the design page to this page and not vice versa 🙂 .  
And you are right about the PBS\_HOME having the core file.

Thanks,  
Prakash

---

<div class="post-metadata">

**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\
**Post date:** [December 9, 2016, 6:00am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/5 "2016-12-09T06:00:33Z")

</div>

@prakashcv13 Done. I have added link for this discussion in design doc.

---

<div class="post-metadata">

**Author:** ![crjayadev](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/crjayadev/32/119_2.png) [@crjayadev](https://community.openpbs.org/u/crjayadev)\
**Post date:** [December 13, 2016, 2:45am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/6 "2016-12-13T02:45:30Z")

</div>

Looks good. This data will be useful.

---

<div class="post-metadata">

**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\
**Post date:** [December 13, 2016, 7:41pm UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/7 "2016-12-13T19:41:38Z")

</div>

@hirenvadalia If PTL does not reset the log files present in PBS\_HOME for each test case failure then wouldn’t it be sufficient to just have one copy of “`PBS_<hostname>.tar.gz`” for each test suite?

---

<div class="post-metadata">

**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\
**Post date:** [December 14, 2016, 1:56am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/8 "2016-12-14T01:56:31Z")

</div>

@arungrover Yes, PTL does not reset logs file and initially I had same thought of coping PBS\_HOME tarball at test suite level but then I realize that almost every test case make changes to PBS configuration (ex. one test case test enables RR in sched config where another test case disables RR in sched config). So, in this case how we will know what was change/content in sched config from post analysis data if data is stored at test suite level. Also in test suite may contain single node as well as multi-node test cases so in this case single node test case will have only one tarball while multi-node test case will have multiple tarball based on number of nodes. So, after considering above cases I choose coping PBS\_HOME tarball at test case level instead of test suite level.

---

<div class="post-metadata">

**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\
**Post date:** [December 14, 2016, 3:19am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/9 "2016-12-14T03:19:05Z")

</div>

Well change is content of files like sched\_config can be easily judged by looking at the failing test case and checking what modification it made to the config file.  
Regarding taking backup for multi-node test cases… why don’t we consider the worst case scenario and store one backup from all the machines where these test cases might have run. If we do this, we can still related the failing test case with the logs.

What do you think?

---

<div class="post-metadata">

**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\
**Post date:** [December 14, 2016, 3:35am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/10 "2016-12-14T03:35:15Z")

</div>

@arungrover Well we can judge by looking at failing test cases but not always and not for all files, and specifically for database directory inside PBS\_HOME.

Why we want to store data from machine where failing test case has not run?

I would prefer to test case level data saving instead test suite level data saving.

---

<div class="post-metadata">

**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\
**Post date:** [December 14, 2016, 4:12am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/11 "2016-12-14T04:12:49Z")

</div>

Well since database and other files that are there, which ideally do not change from case to case in a suite, I feel it will have a lot of redundant data.

Now if we can design something which can smartly collect PBS\_HOME from only machines where a test case has failed then that would be great. It will be ironical if we agree to make a copy of home for every failed test case but disagree on taking a copy of home from every machine if a test suite encounters any failure.

---

<div class="post-metadata">

**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\
**Post date:** [December 14, 2016, 5:15am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/12 "2016-12-14T05:15:48Z")

</div>

@arungrover Yes database will be definitely different from case to case as every test case has different config like jobs, resvs and nodes etc. so considering database directory I don’t see any redundant data.

AFAIK currently PTL does not have any interface which can tell exactly on which machine case is failed (I’m not sure but may be its possible to implement this but that will be like totally refactoring PTL as far as I have knowledge about PTL). If you have any idea on how to implement without refactoring PTL please do suggest I will love to implement that.

---

<div class="post-metadata">

**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\
**Post date:** [December 14, 2016, 6:37am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/13 "2016-12-14T06:37:08Z")

</div>

Well the state of jobs/nodes is already what you store using pbsnodes, qmgr, qstat so db is not going to have anything different.  
One example of redundancy will be if there is a product bug that makes a binary dump core. Now consider a test suite with 10 test cases. On failure of each test case we will store PBS\_HOME and at the end of the test suite we will end up with tarballs consisting a total of 55 core files.

About the way to check for failure, I’d assume that it is the test case that fails. So when you know a test case has failed you store PBS\_HOME from all the machines it was running on. If it is hard to identify then just collect logs from all the moms when the test suite finishes.

---

<div class="post-metadata">

**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\
**Post date:** [December 14, 2016, 6:39am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/14 "2016-12-14T06:39:12Z")

</div>

while writing my previous comment I realized that we don’t collect anything for reservations. I think “pbs\_rstat -f” should be sufficient to collect all reservation specific information.

---

<div class="post-metadata">

**Author:** ![ashwathraop](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/ashwathraop/32/79_2.png) [@ashwathraop](https://community.openpbs.org/u/ashwathraop)\
**Post date:** [December 14, 2016, 6:55am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/15 "2016-12-14T06:55:59Z")

</div>

I agree with Arun here on taking datastore backup for every test case may not be worth it as you already have configurations, nodes, jobs and reservation details taken using commands. Datastore will have same data. Also when there are logs and configurations readily available I would prefer to refer them before getting into DB and start running queries.

---

<div class="post-metadata">

**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\
**Post date:** [December 14, 2016, 7:35am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/16 "2016-12-14T07:35:54Z")

</div>

@arungrover good catch I will update design document for “pbs\_rstat -f” as well as for " qmgr -c ‘p h’ and qmgr -c ‘p pbshook’ " (Thanks to @ashwathraop for catching this)

---

<div class="post-metadata">

**Author:** ![hirenvadalia](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/hirenvadalia/32/19_2.png) [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)\
**Post date:** [December 14, 2016, 7:39am UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/17 "2016-12-14T07:39:01Z")

</div>

I have updated design document as per my last comment.

---

<div class="post-metadata">

**Author:** ![kjakkali](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/kjakkali/32/138_2.png) [@kjakkali](https://community.openpbs.org/u/kjakkali)\
**Post date:** [December 14, 2016, 4:18pm UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/18 "2016-12-14T16:18:06Z")

</div>

@hirenvadalia : Overall all design looks good.

- PBS logs are the one which eats up disk space, so I would prefer to save logs only once per test suite instead of saving at test case level.

- For cpuset systems we need to collect cpuset info (cpuset -s / -r) in case of failure.

- In some cases we need job’s error file which exist in user’s home dir.

- Instead of running list of pbs commands , can we make use of pbs\_diag command?

- If the test case fails at setup/teardown level do you save the post analysis data?

---

<div class="post-metadata">

**Author:** ![kjakkali](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/kjakkali/32/138_2.png) [@kjakkali](https://community.openpbs.org/u/kjakkali)\
**Post date:** [December 14, 2016, 4:23pm UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/19 "2016-12-14T16:23:19Z")

</div>

@arungrover : We can make out configuration changes by logs and test script file but I feel it is an overhead. If we save it will make it easy to debug failure.

---

<div class="post-metadata">

**Author:** ![arungrover](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/arungrover/32/40_2.png) [@arungrover](https://community.openpbs.org/u/arungrover)\
**Post date:** [December 14, 2016, 6:29pm UTC](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380/20 "2016-12-14T18:29:14Z")

</div>

Well I think people who will be working on these issues reported by PTL can easily look at test case. But, still if you think it makes it easy then just copy config files in each test case and then have one single tar of PBS\_HOME per testsuite

[Next page](https://community.openpbs.org/t/pp-302-implement-save-of-pbs-data-for-post-run-analysis/380.md?page=2)
