# Update file format and file name for data saved using save\_configuration() method in PTL
**URL:** https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516
**Category:** Developers
**Created:** [March 20, 2019, 6:25am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516 "2019-03-20T06:25:10Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@RKORavi](https://community.openpbs.org/u/RKORavi)
#### Post date: [March 20, 2019, 6:25am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/1 "2019-03-20T06:25:10Z")
Hi all,
I have created an EDD for interface change to save\_configuration() & load\_configuration() in PTL.
EDD link: [https://pbspro.atlassian.net/wiki/spaces/PD/pages/1156284419/Update+file+format+and+file+name+for+data+saved+using+save+configuration+method+in+PTL](https://pbspro.atlassian.net/wiki/spaces/PD/pages/1156284419/Update+file+format+and+file+name+for+data+saved+using+save+configuration+method+in+PTL)
Please review & share your comments.
Thanks & Regards,
Ravi Kant
---
### Author:  [@kjakkali](https://community.openpbs.org/u/kjakkali)
#### Post date: [March 20, 2019, 2:00pm UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/2 "2019-03-20T14:00:08Z")
@RKORavi : Overall design looks good.
I have few questions.
- How are you handling multinode cluster pbs configuration?
- Can you add an example of json file?
---
### Author:  [@RKORavi](https://community.openpbs.org/u/RKORavi)
#### Post date: [March 21, 2019, 7:19am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/3 "2019-03-21T07:19:30Z")
Hi Kumar,
I have updated the EDD to address above two questions. Please review.
Regards,
Ravi
---
### Author:  [@vccardenas](https://community.openpbs.org/u/vccardenas)
#### Post date: [March 22, 2019, 6:33pm UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/4 "2019-03-22T18:33:23Z")
@ [RKORavi](http://community.openpbs.org/u/RKORavi), a correction:
**Interface:** load\_configuration()
**Synopsis:** Apply configuration saved by load\_configuration --\> should be save\_configuration
Also the statement: “A Single file to be used to save all the configurations” seems to apply per host.
Will this file include all the server, scheduler and mom configurations in case all PBS daemons were on one host?
After loading the saved configuration do we need to HUP or restart any/all of the daemons?
---
### Author:  [@RKORavi](https://community.openpbs.org/u/RKORavi)
#### Post date: [March 25, 2019, 3:24am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/5 "2019-03-25T03:24:13Z")
@vccardenas thanks for review. I have updated the EDD, please check.
When all PBS daemons are running on a single host, all configuration will be saved to a single file. Idea is to keep different file for each host.
Yes, we have restart or HUP daemons based on configuration that is loaded. I have added this in EDD.
Regards,
Ravi
---
### Author:  [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)
#### Post date: [March 27, 2019, 3:49am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/6 "2019-03-27T03:49:27Z")
@RKORavi Here are my comments on this design doc:
- Add details about what will be saved by save\_configuration() (You can refer [here](https://pbspro.atlassian.net/wiki/spaces/PD/pages/557088789/Design+for+a+supported+way+to+change+default+setup+in+PTL))
- 1st point (and 3rd point) says single file per host: why per host file? can’t we include all data from all host into a single file (even if it is a multinode cluster)? (Means every call to save\_configuration will generate only one new file)
- 2nd point says, file contents will be saved in base64 encoded but the example shows qmgr output, also if I remember correctly then when we discussed offline this we discussed only encoding all PBS configuration and hooks related files not qmgr or pbsnodes output… Also if you encode qmgr output and while loading that how will you compare that? For example, let’s say after you saved qmgr -c “print server” output user has added in a resource, now when you compare base64 of this you will find out that content is different but how will you find out what to delete? because saved configuration doesn’t have any logic to delete a newly added resource
- About 5th point: Do we really need to say that? I mean we know that after changing sched config we need to HUP sched to apply that config right? so why we need to explicitly say that? I think we should remove 5th point so that it won’t create any confusion
- What will happen if tempdir already has ‘pbsptl\_host1\_1.json’ when the user invokes pbs\_benchpress?
- I think we should clean up tempdir for these JSON files when the user invokes pbs\_benchpress
---
### Author:  [@RKORavi](https://community.openpbs.org/u/RKORavi)
#### Post date: [March 27, 2019, 5:49am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/7 "2019-03-27T05:49:47Z")
Hi Hiren,
Thankyou for reviewing the EDD.
- I have updated EDD on contents to be saved by save\_configuration.
- If we maintain single file per host then it would be easier to send mom\_config & sched\_config file(in multi sched setup) to different host. Are you suggesting that we keep single file to save all the configuration for multinode setup, but then we have to parse file contents to be send to different host.
- I did not think of removing any custom resource. You are right we should encode only file contents and not print server. I have updated EDD regarding this.
- I have removed fifth point.
- Yes, tempdir has to be cleaned up in case a file with same name is available, added this as point 3 in EDD.
Regards,
Ravi
---
### Author:  [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)
#### Post date: [March 27, 2019, 6:58am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/8 "2019-03-27T06:58:12Z")
> [@RKORavi](#):
>
> If we maintain single file per host then it would be easier to send mom\_config & sched\_config file(in multi sched setup) to different host. Are you suggesting that we keep single file to save all the configuration for multinode setup, but then we have to parse file contents to be send to different host.
Yes, it will be good to keep single file for all hosts even if it multinode setup
---
### Author:  [@neha.padole](https://community.openpbs.org/u/neha.padole)
#### Post date: [March 27, 2019, 8:43am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/9 "2019-03-27T08:43:28Z")
In the section “Different aspects to be covered by this EDD”, in #1 ‘pbsptl\_conf’ is specified as the prefix of the saved json file. In #5, the example is “pbsptl\_host1\_1.json”.
---
### Author:  [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)
#### Post date: [March 27, 2019, 8:52am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/10 "2019-03-27T08:52:04Z")
Thanks for working on this Ravi. PFB some comments on the design:
> **save\_configuration()** will save the following from the PBS cluster:
> Commands output:
> qmgr print server
> qmgr print sched
> qmgr print hook
> qmgr export hook \ application/x-python default
> qmgr export hook \ application/x-config default
> qmgr print resource
> qmgr list pbshook
> qmgr export pbshook \ application/x-config default
> pbsnodes
> Configuration files:
> sched\_priv/sched\_config
> sched\_priv/holidays
> sched\_priv/resource\_group
> sched\_priv/dedicated\_time
> multiple schedulers directories if any
> mom\_priv/config (files of each hosts listed in pbsnodes)
> mom\_priv/config.d/ (saves vnode definition files listed with “pbs\_mom -s list”)
> /etc/pbs.conf (all nodes)
Was save\_configuration already capturing all of this before? Is yes then please mention that save\_configuration will continue capturing all of this. If you are changing it to capture something new then please highlight that in the doc.
Also, it will be very helpful to know what information will be captured for each daemon. So, can you please break this down into what info will be captured for each daemon?
> A Single file per host to be used to save all the configurations,
So, for a multi-node configuration, we will capture multiple files, where will those files go? is there going to be a parent directory inside which they will be stored? Also, on which host will they be accumulated?
> File name with the prefix as “pbsptl\_conf” and suffix as “. json”.
Where does the hostname go in the filename? I think in later examples you mention pbsptl\_\\_conf.json, so please correct it here.
> File contents can be encoded in base64 & that base64 is added to json file.
So will there be an argument to save\_configuration() which decides whether to encrypt the data or not?
> Different file to be used for each host, to handle multinode cluster pbs configuration. Each file to have a prefix of hostname to identify between configuration of different hosts.
> e.g, pbsptl\_host1\_versionNumber.json, pbsptl\_host2\_versionNumber,json
You already talk about file name in bullet 1, so I suggest that you merge the two
> Different versions of files to be maintained, every time save\_configuration() is called.
then why do you need to do this:
> When user invokes pbs\_benchpress, cleanup of tempdir is performed in case file with same name is present.
---
### Author:  [@RKORavi](https://community.openpbs.org/u/RKORavi)
#### Post date: [March 28, 2019, 4:11am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/11 "2019-03-28T04:11:17Z")
Hi Ravi,
Thank you for reviewing the EDD.
Changes i have made to EDD after your comment:
1. Save configuration will continue to save contents that it was doing it before. I am not adding anything new to save configuration.
2. Divided save\_configuration contents as per each daemons.
3. Single file to be used to save all the hosts configurations. Updated json file example accoding to this.
4. Merged point 1 & point 4.
- Any config file that is to be saved is encoded in base64 format. So, I am not adding an argument to save\_configuration.
Regards,
Ravi
---
### Author:  [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)
#### Post date: [March 28, 2019, 7:10am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/12 "2019-03-28T07:10:07Z")
> [@RKORavi](#):
>
> Any config file that is to be saved is encoded in base64 format. So, I am not adding an argument to save\_configuration.
I think you should. It can be useful for somebody to just look at the data dump itself to infer some information instead of going through the effort of loading it via pbs\_config. Your second bullet also seems to suggest that the encoding is optional: " 2. File format: File contents can be encoded in base64 & that base64 is added to json file."
> [@RKORavi](#):
>
> Even though we maintain different version of file for each iteration of save\_configuration, it is just a precautionary measure to check if there is any file name with same name already present in temp directory.
I’m not sure I understand why though. What’s the harm if you don’t remove the already existing file?
---
### Author:  [@RKORavi](https://community.openpbs.org/u/RKORavi)
#### Post date: [March 29, 2019, 2:36am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/13 "2019-03-29T02:36:53Z")
Hi Ravi,
If any config file is not encoded to base64 format then user has to take care of line like:
resources: “ncpus, mem, arch, host, vnode, aoe, eoe” because it can’t be stored as it is in json file. He might have to parse config file contents to take care of adding “” in these lines. But when we do not give an option whether to encode config file contents or not then all user has to do is decode the contents to look at data dump. your thoughts?
Regards,
Ravi
---
### Author:  [@RKORavi](https://community.openpbs.org/u/RKORavi)
#### Post date: [March 29, 2019, 3:12am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/14 "2019-03-29T03:12:16Z")
Hi all,
I would like to have some suggestions on how to store hook files, I am thinking of storing all hooks .PY & .CF file in a tar ball or should i store it in same json file. But storing hook files content in same json file will increase its size drastically.
Any suggestion on this will be helpful, before i update EDD.
Regards,
Ravi
---
### Author:  [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)
#### Post date: [March 29, 2019, 5:20am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/15 "2019-03-29T05:20:54Z")
@RKORavi I will vote for storing any information not matter what it is in json file only so that we have to take care of one file only… Keeping multiple file + different format will cause maintenance + complexity.
---
### Author:  [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)
#### Post date: [March 29, 2019, 5:32am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/16 "2019-03-29T05:32:11Z")
Ok, if it’s complicated to generate json for non-encrypted data then I’m ok with not providing an option to dump non-encrypted data, if a use case for it emerges later then we can add it then.
---
### Author:  [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)
#### Post date: [March 29, 2019, 5:33am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/17 "2019-03-29T05:33:24Z")
Well you either increase the size of one file or the whole package that includes multiple files, it doesn’t make much difference. If you are worried about the dump getting too big then just compress the one file.
---
### Author:  [@RKORavi](https://community.openpbs.org/u/RKORavi)
#### Post date: [March 29, 2019, 5:49am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/18 "2019-03-29T05:49:03Z")
Thank you all for review and suggestions. I will go ahead with maintaining a single json file. If there are no further comments then please go ahead and sign-off EDD so that i could start making code changes.
Regards,
Ravi
---
### Author:  [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)
#### Post date: [March 29, 2019, 5:56am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/19 "2019-03-29T05:56:30Z")
Please remove the following bullet, it suggests that encrypting in base64 is optional:
“2. File format: File contents can be encoded in base64 & that base64 is added to json file.”
Also, can you add the bit that we just discussed about hooks?
---
### Author:  [@hirenvadalia](https://community.openpbs.org/u/hirenvadalia)
#### Post date: [March 29, 2019, 6:00am UTC](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516/20 "2019-03-29T06:00:29Z")
@RKORavi Design doc looks good to me. Thanks for making changes.
[Next page](https://community.openpbs.org/t/update-file-format-and-file-name-for-data-saved-using-save-configuration-method-in-ptl/1516.md?page=2)