PP-677: Add YAML config files (.yml) to the supported list of config files for PBS hooks

Developers
1

This note is to inform the community of work being done to add support for .yml files as pbs hook config files

Support YAML formatted file as PBS hook config file

Please feel free to provide feedback on this topic

2

What is the motivation for supporting YAML?
What version(s) of YAML will be supported?
Will PBS Pro be linked against a YAML library?

3

It would be nice to give an example of a YAML configuration file, and how it might get invoked and processed in a hook script.

1 Like
4

YAML is a superset of json and allows the users to add comments

1.2

I would like to use the PyYAML library and install it locally on my machine. Then when the configuration file is imported, the python interpreter would check to see that it is correct YAML syntax. If so, the file is imported. If not, an error message is returned

5

Al,

Below is an example of the cgroups configuration file in YAML

# Cgroups config file
cgroup_prefix         : pbspro
exclude_hosts         :
        - node001
        - node002
# Exclude certain vnode types from running cgroups
exclude_vntypes       :
        - disable_cgroups
        - login_node
run_only_on_hosts     : []
periodic_resc_update  : true
vnode_per_numa_node   : false
online_offlined_nodes : true
use_hyperthreads      : false
cgroup :
    cpuacct :
        enabled         : true
        exclude_hosts   :
            - node003
        exclude_vntypes :
            - red_node
    cpuset :
        enabled         : true
        exclude_hosts   :
            - node004
        exclude_vntypes :
            - green_node
    devices :
        enabled         : false
        exclude_hosts   : []
        exclude_vntypes : []
        allow           :
            - b *:* rwm
            - c *:* rwm
            - ["mic/scif", "rwm"]
            - ["nvidiactl", "rwm", "*"]
            - ["nvidia-uvm", "rwm"]

    hugetlb :
        enabled         : false
        exclude_hosts   : []
        exclude_vntypes : []
        default         : 0MB
        reserve_percent : 0
        reserve_amount  : 0MB
    memory :
        enabled         : true
        exclude_hosts   : []
        exclude_vntypes :
            - blue_node
        soft_limit      : false
        default         : 256MB
        reserve_percent : 0
        reserve_amount  : 1GB
    memsw :
        enabled         : true
        exclude_hosts   : []
        exclude_vntypes :
            - grey_node
        default         : 256MB
        reserve_percent : 0
        reserve_amount  : 1GB

To use this config in a hook do the following

import yaml
with open(‘config.yaml’, ‘r’) as f:
cfg = yaml.safe_load(f)

Below are a couple of useful links to better understand YAML

1 Like
6

There has been discussions about moving all configuration to PBS. Current design of hooks configuration files are not in line with that intended direction for PBS. Given that, why would we continue to enhance such configuration files.

If we were to support configuration files, then yes, yaml files are truely intended for that - not json files. However, given the direction we intend to go (i.e., move all configuration to qmgr), I think we should avoid doing these enhancements, and instead expend efforts to move away from the configuration files.

7

When and who made this decision? Where can I find it on the community page?

8

Not yet. Some of us are working towards putting that up on the community pages soon. We will update the community soon about the same. I will circle back and provide a link to that in this discussion thread as soon as we can put that up. That design guidance itself, obviously, will also be under community review.