Cgroup error causing suspended jobs

hellmersjl · October 3, 2018, 4:44pm

I have a situation where I’m getting an error in the pbs_cgroups.PY that is causing jobs to be held. The MOM log shows the following error (I’ve expanded some of the messages for readability).

10/03/2018 08:14:34;0080;
pbs_python;
Hook;
pbs_python;[‘Traceback (most recent call last):’,
’ File “”, line 4597, in main’,
’ File “”, line 724, in invoke_handler’,
’ File “”, line 754, in _execjob_begin_handler’,
’ File “”, line 3802, in configure_job’,
’ File “”, line 3249, in assign_job’, ‘KeyError: 1’]

10/03/2018 08:14:34;0001;
pbs_python;
Hook;
pbs_python;
Unexpected error in pbs_cgroups handling execjob_begin event for job 1234.ds-rnd-gpu (suspended): KeyError (1,)

10/03/2018 08:14:34;0080;pbs_python;Hook;pbs_python;Elapsed time: 0.2100
10/03/2018 08:14:34;0100;pbs_mom;Hook;pbs_cgroups;execjob_begin request rejected by ‘pbs_cgroups’
10/03/2018 08:14:34;0008;pbs_mom;Job;1234.ds-rnd-gpu;Unexpected error in pbs_cgroups handling execjob_begin event for job 1234.ds-rnd-gpu (suspended): KeyError (1,)
10/03/2018 08:14:34;0100;pbs_mom;Req;;Type 3 request received from root@10.0.0.100:15001, sock=1
10/03/2018 08:14:34;0080;pbs_mom;Req;req_reject;Reject reply code=15004, aux=0, type=3, from root@10.0.0.100:15001
10/03/2018 08:14:34;0100;pbs_mom;Req;;Type 5 request received from root@10.0.0.100:15001, sock=1
10/03/2018 08:14:34;0080;pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=5, from root@10.0.0.100:15001

It seems that it’s getting some sort of error when assign cpus. I’m using

-l npcus=4

When submitting jobs. I can submit a few jobs like this and they run but when the resources get used up and the job should get queued, instead it gets held.

I’m also getting a message every couple of minutes

10/03/2018 09:14:07;0100;pbs_python;Hook;pbs_python;_get_vnode_type: Could not determine vntype
10/03/2018 09:14:07;0080;pbs_python;Hook;pbs_python;Elapsed time: 0.1922
I don’t know if this is any sort of problem or not, but it makes me wonder if I don’t have some sort of systematic problem.

My JSON configuration for the hook is

{
“cgroup_prefix” : “pbspro”,
“exclude_hosts” : [“ds-rnd-gpu”],
“exclude_vntypes” : ,
“run_only_on_hosts” : ,
“periodic_resc_update” : true,
“vnode_per_numa_node” : false,
“online_offlined_nodes” : true,
“use_hyperthreads” : false,
“ncpus_are_cores” : false,
“cgroup” : {
“cpuacct” : {
“enabled” : true,
“exclude_hosts” : ,
“exclude_vntypes” :
},
“cpuset” : {
“enabled” : true,
“exclude_cpus” : ,
“exclude_hosts” : ,
“exclude_vntypes” : ,
“mem_fences” : true,
“mem_hardwall” : false,
“memory_spread_page” : false
},
“devices” : {
“enabled” : true,
“exclude_hosts” : [“ds-rnd-gpu”],
“exclude_vntypes” : ,
“allow” : [
“b : rwm”,
“c : rwm”,
[“nvidiactl”, “rwm”, “*”],
[“nvidia-uvm”, “rwm”]
]
},
“hugetlb” : {
“enabled” : false,
“exclude_hosts” : ,
“exclude_vntypes” : ,
“default” : “0MB”,
“reserve_percent” : “0”,
“reserve_amount” : “0MB”
},
“memory” : {
“enabled” : true,
“exclude_hosts” : ,
“exclude_vntypes” : ,
“soft_limit” : false,
“default” : “256MB”,
“reserve_percent” : “0”,
“reserve_amount” : “64MB”
},
“memsw” : {
“enabled” : false,
“exclude_hosts” : ,
“exclude_vntypes” : ,
“default” : “256MB”,
“reserve_percent” : “0”,
“reserve_amount” : “64MB”
}
}
}

mkaro · October 3, 2018, 6:54pm

Hello @hellmersjl,

The handler for the execjob_begin hook is the most complex of the cgroup handlers. In order to figure out what went wrong, I’ll ask that you enable verbose logging in MoM. Just add the “$logevent 0xffff” line to mom_priv/config and restart MoM. Then repeat your job submission and see what the logs have to offer. It looks as though one of the dictionary keys was formatted incorrectly, but it’s impossible to tell which one without more detailed information.

Regarding the vnode type… the message you are seeing is harmless. If you want to set a vnode type, you may do so by creating a string resource called resources_available.vntype and assign it a value via qmgr. Alternatively, you can place a file named vntype in the mom_priv directory that contains the string you want to use.

Looking at your cgroup hook configuration file, the only thing that stands out is the fact that the memory subsystem is enabled, but memsw is not. If a job exceeds the memory limit set for the cgroup, it will begin to swap. Since memsw is disabled, there is no limit to the amount of swap space it may consume. In general, you want to either enable or disable both subsystems. There are some cases (diskless nodes with no swap) where you may want memory enabled and memsw disabled, but those are less common. The rest of the file looks fine.

Thanks,

Mike

mkaro · October 3, 2018, 7:05pm

Looking a little closer at the traceback you provided, it looks like the problem occurred when the hook was assigning cpuset resources for the job. If your compute nodes have only one NUMA node, you may disable the cpuset subsystem in the hook configuration file. That should get you around the problem for now, but I would appreciate your help to actually fix the underlying cause.

Could you also provide all of the parameters passed to qsub and any #PBS directives in your job script?

Thanks,

Mike

jon · October 4, 2018, 4:29am

@hellmersjl can you see what happens if instead of submitting -l ncpus=4 can you instead do -l select=1:ncpus=4:mem=8gb:mpiprocs=4 and see if the error goes away? My guess is the hook is struggling with the -l ncpus request since that is old syntax.

hellmersjl · October 4, 2018, 8:06pm

I did the following

changed memsw to be enabled in the cgroup json file
used the resource options as specified
Still getting jobs set to held. Here’s the log.

10/04/2018 12:52:18;0100;pbs_mom;Req;;Type 1 request received from root@10.0.0.100:15001, sock=1
10/04/2018 12:52:18;0400;pbs_mom;Hook;pbs_cgroups;started
10/04/2018 12:52:18;0400;pbs_mom;Node;ds-rnd-gpu2[1];implicitly added host to vmap
10/04/2018 12:52:18;0400;pbs_mom;Hook;pbs_cgroups;execve /opt/pbs/bin/pbs_python --hook -i /var/spool/pbs/mom_priv/hooks/tmp/hook_execjob_begin_pbs_cgroups_18422.in -o /var/spool/pbs/mom_priv/hooks/tmp/hook_execjob_begin_pbs_cgroups_18422.out -L /var/spool/pbs/mom_logs -e 65535 -r /var/spool/pbs/mom_priv/hooks/resourcedef /var/spool/pbs/mom_priv/hooks/pbs_cgroups.PY runas_jobuser=0 in child pid=18422
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;–> Python Interpreter started, compiled with version:‘2.7.1’ ←
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;–> Python module path is now: [‘/opt/pbs/python/lib/python27.zip’, ‘/opt/pbs/python/lib/python2.7/’, ‘/opt/pbs/python/lib/python2.7/plat-linux2’, ‘/opt/pbs/python/lib/python2.7/lib-tk’, ‘/opt/pbs/python/lib/python2.7/lib-old’, ‘/opt/pbs/python/lib/python2.7/lib-dynload’, ‘/opt/pbs/lib/python/altair’] ←
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;sys.modules= {‘pbs.v1._svr_types’: <module ‘pbs.v1._svr_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_svr_types.pyo’>, ‘copy_reg’: <module ‘copy_reg’ from ‘/opt/pbs/python/lib/python2.7/copy_reg.pyo’>, ‘pbs.v1._pbs_v1’: None, ‘genericpath’: <module ‘genericpath’ from ‘/opt/pbs/python/lib/python2.7/genericpath.pyo’>, ‘main’: <module ‘main’ (built-in)>, ‘imp’: <module ‘imp’ (built-in)>, ‘builtin’: <module ‘builtin’ (built-in)>, ‘_pbs_v1.svr_types’: <module ‘_pbs_v1.svr_types’ (built-in)>, ‘encodings’: <module ‘encodings’ from ‘/opt/pbs/python/lib/python2.7/encodings/init.pyo’>, ‘os.path’: <module ‘posixpath’ from ‘/opt/pbs/python/lib/python2.7/posixpath.pyo’>, ‘encodings.encodings’: None, ‘pbs.v1.imp’: None, ‘abc’: <module ‘abc’ from ‘/opt/pbs/python/lib/python2.7/abc.pyo’>, ‘pbs._pbs_v1’: None, ‘posixpath’: <module ‘posixpath’ from ‘/opt/pbs/python/lib/python2.7/posixpath.pyo’>, ‘_weakrefset’: <module ‘_weakrefset’ from ‘/opt/pbs/python/lib/python2.7/_weakrefset.pyo’>, ‘pbs.v1’: <module ‘pbs.v1’ from ‘/opt/pbs/lib/python/altair/pbs/v1/init.pyo’>, ‘errno’: <module ‘errno’ (built-in)>, ‘encodings.codecs’: None, ‘pbs.v1._export_types’: <module ‘pbs.v1._export_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_export_types.pyo’>, ‘pbs.v1.os’: None, ‘_abcoll’: <module ‘_abcoll’ from ‘/opt/pbs/python/lib/python2.7/_abcoll.pyo’>, ‘types’: <module ‘types’ from ‘/opt/pbs/python/lib/python2.7/types.pyo’>, ‘UserDict’: <module ‘UserDict’ from ‘/opt/pbs/python/lib/python2.7/UserDict.pyo’>, ‘_warnings’: <module ‘_warnings’ (built-in)>, ‘pbs.v1.pbs_ifl’: <module ‘pbs.v1.pbs_ifl’ from ‘/opt/pbs/lib/python/altair/pbs/v1/pbs_ifl.pyo’>, ‘stat’: <module ‘stat’ from ‘/opt/pbs/python/lib/python2.7/stat.pyo’>, ‘zipimport’: <module ‘zipimport’ (built-in)>, ‘encodings.builtin’: None, ‘pbs.pbs’: None, ‘warnings’: <module ‘warnings’ from ‘/opt/pbs/python/lib/python2.7/warnings.pyo’>, ‘_codecs’: <module ‘_codecs’ (built-in)>, ‘encodings.ascii’: <module ‘encodings.ascii’ from ‘/opt/pbs/python/lib/python2.7/encodings/ascii.pyo’>, ‘swig_runtime_data4’: <module ‘swig_runtime_data4’ (built-in)>, ‘sys’: <module ‘sys’ (built-in)>, ‘pbs.v1._base_types’: <module ‘pbs.v1._base_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_base_types.pyo’>, ‘pbs.v1.sys’: None, ‘codecs’: <module ‘codecs’ from ‘/opt/pbs/python/lib/python2.7/codecs.pyo’>, ‘_pbs_ifl’: <module ‘_pbs_ifl’ (built-in)>, ‘pbs.v1._pbs_ifl’: None, ‘pbs’: <module ‘pbs’ from ‘/opt/pbs/lib/python/altair/pbs/init.pyo’>, ‘os’: <module ‘os’ from ‘/opt/pbs/python/lib/python2.7/os.pyo’>, ‘signal’: <module ‘signal’ (built-in)>, ‘linecache’: <module ‘linecache’ from ‘/opt/pbs/python/lib/python2.7/linecache.pyo’>, ‘_pbs_v1’: <module ‘_pbs_v1’ (built-in)>, ‘posix’: <module ‘posix’ (built-in)>, ‘encodings.aliases’: <module ‘encodings.aliases’ from ‘/opt/pbs/python/lib/python2.7/encodings/aliases.pyo’>, ‘exceptions’: <module ‘exceptions’ (built-in)>, ‘pbs.v1._exc_types’: <module ‘pbs.v1._exc_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_exc_types.pyo’>, ‘_weakref’: <module ‘_weakref’ (built-in)>}
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;BEGIN setting up all resource attributes
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;DONE setting up all resource attributes, number set <53>
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;BEGIN setting up all queue attributes
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;DONE setting up all queue attributes, number set <54>
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;BEGIN setting up all job attributes
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;DONE setting up all job attributes, number set <94>
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;BEGIN setting up all server attributes
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;DONE setting up all server attributes, number set <88>
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;BEGIN setting up all reservation attributes
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;DONE setting up all reservation attributes, number set <43>
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;BEGIN setting up all vnode attributes
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;DONE setting up all vnode attributes, number set <27>
10/04/2018 12:52:18;0400;pbs_python;Hook;_pbs_python_event_set;python_restart_max_hooks is now 100
10/04/2018 12:52:18;0400;pbs_python;Hook;_pbs_python_event_set;python_restart_max_objects is now 1000
10/04/2018 12:52:18;0400;pbs_python;Hook;_pbs_python_event_set;python_restart_min_interval is now 30
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;pbs_python_populate_python_class_from_svrattrl==>
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=id al_resc=null al_value=1250.ds-rnd-gpu al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Job_Name al_resc=null al_value=sleeplong.sh al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Job_Owner al_resc=null al_value=jhs4@ds-rnd-gpu al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=queue al_resc=null al_value=workq al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=server al_resc=null al_value=ds-rnd-gpu al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Checkpoint al_resc=null al_value=u al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Error_Path al_resc=null al_value=ds-rnd-gpu:/data/jhs4/prod/sleeplong.sh.e1250 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=exec_host2 al_resc=null al_value=ds-rnd-gpu2:15002/14 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=exec_vnode al_resc=null al_value=(ds-rnd-gpu2[1]:ncpus=4:mem=8388608kb) al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Join_Path al_resc=null al_value=n al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Keep_Files al_resc=null al_value=n al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=mtime al_resc=null al_value=1538682738 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Output_Path al_resc=null al_value=ds-rnd-gpu:/data/jhs4/prod/sleeplong.sh.o1250 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Resource_List al_resc=mem al_value=8gb al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Resource_List al_resc=ncpus al_value=4 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Resource_List al_resc=place al_value=free al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=schedselect al_resc=null al_value=1:ncpus=4:mem=8gb:mpiprocs=4:Qlist=workq al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Variable_List al_resc=null al_value=PBS_O_HOME=/users/JHS4,PBS_O_LANG=en_US.UTF-8,PBS_O_LOGNAME=jhs4,PBS_O_PATH=/opt/pbs/bin:/usr/lib64/qt-3.3/bin:/users/JHS4/perl5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/users/JHS4/bin,PBS_O_MAIL=/var/spool/mail/jhs4,PBS_O_SHELL=/bin/bash,PBS_O_WORKDIR=/data/jhs4/prod,PBS_O_SYSTEM=Linux,PBS_O_QUEUE=workq,PBS_O_HOST=ds-rnd-gpu al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=euser al_resc=null al_value=jhs4 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=egroup al_resc=null al_value=users al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=hashname al_resc=null al_value=1250.ds-rnd-gpu al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=run_count al_resc=null al_value=1 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=job_kill_delay al_resc=null al_value=10 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=project al_resc=null al_value=_pbs_project_default al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=run_version al_resc=null al_value=1 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=_msmom al_resc=null al_value=True al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=_stdout_file al_resc=null al_value= al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=_stderr_file al_resc=null al_value= al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;pbs_python_populate_python_class_from_svrattrl==>
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=resources_assigned al_resc=ncpus al_value=16 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=resources_assigned al_resc=mem al_value=33554432kb al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;Compiling script file: </var/spool/pbs/mom_priv/hooks/pbs_cgroups.PY>
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;main: Function called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;main: Host is ds-rnd-gpu2
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;main: Hook name is pbs_cgroups
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;event_name: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;main: Event type is execjob_begin
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;main: Hook utility class instantiated
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;hashandler: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: vnhost pattern: ds-rnd-gpu2[[\d+]]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: Job exec_vnode list: (ds-rnd-gpu2[1]:ncpus=4:mem=8388608kb)
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: Vnodes on ds-rnd-gpu2: [‘ds-rnd-gpu2[1]’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: ds-rnd-gpu2[1]:ncpus missing
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: ds-rnd-gpu2[1]:mem missing
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: Chunk ds-rnd-gpu2[1] resources: {‘vnodes’: {‘ds-rnd-gpu2[1]’: {‘mem’: 0b, ‘ncpus’: 0L}}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: resources[ds-rnd-gpu2][ncpus] is now 4
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: resources[ds-rnd-gpu2][mem] is now 8388608kb
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_job_resources: Resources for ds-rnd-gpu2: {‘mem’: 8388608kb, ‘ncpus’: 4L, ‘vnodes’: {‘ds-rnd-gpu2[1]’: {‘mem’: 8388608kb, ‘ncpus’: 4L}}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;main: Job information class instantiated
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;parse_config_file: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;parse_config_file: Config file is /var/spool/pbs/mom_priv/hooks/pbs_cgroups.CF
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;parse_config_file: cgroup hook configuration: {‘nvidia-smi’: ‘/usr/bin/nvidia-smi’, ‘cgroup_prefix’: ‘pbspro’, ‘use_hyperthreads’: False, ‘vnode_per_numa_node’: False, ‘online_offlined_nodes’: True, ‘exclude_vntypes’: [], ‘kill_timeout’: 10, ‘placement_type’: ‘load_balanced’, ‘periodic_resc_update’: True, ‘run_only_on_hosts’: [], ‘cgroup’: {‘memsw’: {‘default’: ‘256MB’, ‘exclude_vntypes’: [], ‘enabled’: True, ‘reserve_amount’: ‘64MB’, ‘reserve_percent’: ‘0’, ‘exclude_hosts’: []}, ‘net_prio’: {‘enabled’: False}, ‘net_cls’: {‘enabled’: False}, ‘blkio’: {‘enabled’: False}, ‘cpuacct’: {‘exclude_vntypes’: [], ‘enabled’: True, ‘exclude_hosts’: []}, ‘devices’: {‘exclude_vntypes’: [], ‘enabled’: True, ‘allow’: [‘b : rwm’, ‘c : rwm’, [‘nvidiactl’, ‘rwm’, ''], [‘nvidia-uvm’, ‘rwm’]], ‘exclude_hosts’: [‘ds-rnd-gpu’]}, ‘cpuset’: {‘exclude_cpus’: , ‘memory_spread_page’: False, ‘exclude_vntypes’: , ‘enabled’: True, ‘mem_hardwall’: False, ‘mem_fences’: True, ‘exclude_hosts’: }, ‘hugetlb’: {‘default’: ‘0MB’, ‘exclude_vntypes’: , ‘enabled’: False, ‘reserve_amount’: ‘0MB’, ‘reserve_percent’: ‘0’, ‘exclude_hosts’: }, ‘perf_event’: {‘enabled’: False}, ‘memory’: {‘default’: ‘256MB’, ‘exclude_vntypes’: , ‘enabled’: True, ‘reserve_amount’: ‘64MB’, ‘reserve_percent’: ‘0’, ‘soft_limit’: False, ‘exclude_hosts’: }, ‘pids’: {‘enabled’: False}}, ‘exclude_hosts’: [‘ds-rnd-gpu’], ‘ncpus_are_cores’: False, ‘cgroup_lock_file’: ‘/var/spool/pbs/mom_priv/cgroups.lock’}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;/var/spool/pbs/mom_priv/cgroups.lock file lock acquired by main
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_check_os: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_check_os: Detected Linux kernel version 3.10.0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_systemd_version: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_paths: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_assemble_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_vnode_type: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;resc_vntype:
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_vnode_type: Failed to read vntype file /var/spool/pbs/mom_priv/vntype
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;file_vntype:
10/04/2018 12:52:18;0100;pbs_python;Hook;pbs_python;_get_vnode_type: Could not determine vntype
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_target_subsystems: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called

hellmersjl · October 4, 2018, 8:08pm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_target_subsystems: Enabled subsystems: [‘memsw’, ‘cpuacct’, ‘devices’, ‘cpuset’, ‘memory’, ‘systemd’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Devices path is /sys/fs/cgroup/devices/pbspro.slice/pbspro-1249.ds\x2drnd\x2dgpu.slice/devices.list
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Appending b : rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Appending c : rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Appending c 195:* rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Appending c 244:0 rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Returning {‘1249.ds-rnd-gpu’: {‘cpuset’: {‘cpus’: [0, 1, 2, 3], ‘mems’: [0]}, ‘memsw’: {‘limit_in_bytes’: 8589934592}, ‘devices’: {‘list’: [‘b : rwm\n’, ‘c : rwm\n’, ‘c 195:* rwm\n’, ‘c 244:0 rwm\n’]}, ‘memory’: {‘limit_in_bytes’: 8589934592, ‘soft_limit_in_bytes’: 9223372036854771712}}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;main: Cgroup utility class instantiated
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;invoke_handler: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;invoke_handler: UID: real=0, effective=0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;invoke_handler: GID: real=0, effective=0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;hashandler: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_execjob_begin_handler: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_discover_cpuinfo: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Mapping hyperthreads to cores
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;HT cores: [8, 9, 10, 11, 12, 13, 14, 15]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_discover_cpuinfo returning: {‘hyperthreads_per_core’: 2, ‘logical_cpus’: 16, ‘cpu’: {0: {‘cpu cores’: 8, ‘bogomips’: ‘6385.62’, ‘core id’: 0, ‘apicid’: 0, ‘fpu_exception’: ‘yes’, ‘stepping’: 1, ‘cache_alignment’: 64, ‘clflush size’: 64, ‘microcode’: ‘0xb000021’, ‘cache size’: ‘25600 KB’, ‘cpuid level’: 20, ‘fpu’: ‘yes’, ‘model name’: ‘Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz’, ‘siblings’: 16, ‘physical id’: 0, ‘address sizes’: ‘46 bits physical, 48 bits virtual’, ‘cpu family’: 6, ‘vendor_id’: ‘GenuineIntel’, ‘threads’: [8], ‘wp’: ‘yes’, ‘power management’: ‘’, ‘flags’: [‘fpu’, ‘vme’, ‘de’, ‘pse’, ‘tsc’, ‘msr’, ‘pae’, ‘mce’, ‘cx8’, ‘apic’, ‘sep’, ‘mtrr’, ‘pge’, ‘mca’, ‘cmov’, ‘pat’, ‘pse36’, ‘clflush’, ‘dts’, ‘acpi’, ‘mmx’, ‘fxsr’, ‘sse’, ‘sse2’, ‘ss’, ‘ht’, ‘tm’, ‘pbe’, ‘syscall’, ‘nx’, ‘pdpe1gb’, ‘rdtscp’, ‘lm’, ‘constant_tsc’, ‘arch_perfmon’, ‘pebs’, ‘bts’, ‘rep_good’, ‘nopl’, ‘xtopology’, ‘nonstop_tsc’, ‘aperfmperf’, ‘eagerfpu’, ‘pni’, ‘pclmulqdq’, ‘dtes64’, ‘monitor’, ‘ds_cpl’, ‘vmx’, ‘smx’, ‘est’, ‘tm2’, ‘ssse3’, ‘fma’, ‘cx16’, ‘xtpr’, ‘pdcm’, ‘pcid’, ‘dca’, ‘sse4_1’, ‘sse4_2’, ‘x2apic’, ‘movbe’, ‘popcnt’, ‘tsc_deadline_timer’, ‘aes’, ‘xsave’, ‘avx’, ‘f16c’, ‘rdrand’,
etc.

hellmersjl · October 4, 2018, 8:09pm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_discover_meminfo: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Discover meminfo: {‘SwapTotal’: ‘25006076k’, ‘MemTotal’: ‘65729260k’, ‘HugePages_Rsvd’: 0, ‘Hugepagesize’: ‘2048k’, ‘HugePages_Total’: 0}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_discover_numa_nodes: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_discover_numa_nodes: {0: {‘MemTotal’: ‘67028780k’, ‘cpus’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘devices’: , ‘HugePages_Total’: 0}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_discover_devices: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/nvidia-uvm-tools, Major: 244, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/nvidia-uvm, Major: 244, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/nvidia1, Major: 195, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/nvidia0, Major: 195, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/nvidiactl, Major: 195, Minor: 255, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/pps1, Major: 248, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ptp2, Major: 247, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcsa5, Major: 7, Minor: 133, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcs5, Major: 7, Minor: 5, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcsa4, Major: 7, Minor: 132, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcs4, Major: 7, Minor: 4, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcsa3, Major: 7, Minor: 131, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcs3, Major: 7, Minor: 3, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcsa2, Major: 7, Minor: 130, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcs2, Major: 7, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcsa6, Major: 7, Minor: 134, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcs6, Major: 7, Minor: 6, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/dm-2, Major: 253, Minor: 2, Type: b
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/kvm, Major: 10, Minor: 232, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/watchdog0, Major: 252, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/watchdog, Major: 10, Minor: 130, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/mei0, Major: 246, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/sg1, Major: 21, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/sg0, Major: 21, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/fuse, Major: 10, Minor: 229, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ppp, Major: 108, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/btrfs-control, Major: 10, Minor: 234, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/uinput, Major: 10, Minor: 223, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vhci, Major: 10, Minor: 137, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/uhid, Major: 10, Minor: 239, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vhost-net, Major: 10, Minor: 238, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/dm-1, Major: 253, Minor: 1, Type: b
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/dm-0, Major: 253, Minor: 0, Type: b
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/sda2, Major: 8, Minor: 2, Type: b
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/sda1, Major: 8, Minor: 1, Type: b
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/sda, Major: 8, Minor: 0, Type: b
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/sr0, Major: 11, Minor: 0, Type: b
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ptp1, Major: 247, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/pps0, Major: 248, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ptp0, Major: 247, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/mpt3ctl, Major: 10, Minor: 222, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/loop-control, Major: 10, Minor: 237, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/network_throughput, Major: 10, Minor: 59, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/network_latency, Major: 10, Minor: 60, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu_dma_latency, Major: 10, Minor: 61, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/mcelog, Major: 10, Minor: 227, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/rtc0, Major: 253, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/usbmon4, Major: 250, Minor: 4, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/usbmon3, Major: 250, Minor: 3, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/usbmon2, Major: 250, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/usbmon1, Major: 250, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/usbmon0, Major: 250, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/crash, Major: 10, Minor: 62, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/nvram, Major: 10, Minor: 144, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/hpet, Major: 10, Minor: 228, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ttyS3, Major: 4, Minor: 67, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ttyS2, Major: 4, Minor: 66, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ttyS1, Major: 4, Minor: 65, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ttyS0, Major: 4, Minor: 64, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/ptmx, Major: 5, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/autofs, Major: 10, Minor: 235, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snapshot, Major: 10, Minor: 231, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty63, Major: 4, Minor: 63, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty62, Major: 4, Minor: 62, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty61, Major: 4, Minor: 61, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty60, Major: 4, Minor: 60, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty59, Major: 4, Minor: 59, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty58, Major: 4, Minor: 58, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty57, Major: 4, Minor: 57, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty56, Major: 4, Minor: 56, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty55, Major: 4, Minor: 55, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty54, Major: 4, Minor: 54, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty53, Major: 4, Minor: 53, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty52, Major: 4, Minor: 52, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty51, Major: 4, Minor: 51, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty50, Major: 4, Minor: 50, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty49, Major: 4, Minor: 49, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty48, Major: 4, Minor: 48, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty47, Major: 4, Minor: 47, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty46, Major: 4, Minor: 46, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty45, Major: 4, Minor: 45, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty44, Major: 4, Minor: 44, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty43, Major: 4, Minor: 43, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty42, Major: 4, Minor: 42, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty41, Major: 4, Minor: 41, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty40, Major: 4, Minor: 40, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty39, Major: 4, Minor: 39, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty38, Major: 4, Minor: 38, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty37, Major: 4, Minor: 37, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty36, Major: 4, Minor: 36, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty35, Major: 4, Minor: 35, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty34, Major: 4, Minor: 34, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty33, Major: 4, Minor: 33, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty32, Major: 4, Minor: 32, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty31, Major: 4, Minor: 31, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty30, Major: 4, Minor: 30, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty29, Major: 4, Minor: 29, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty28, Major: 4, Minor: 28, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty27, Major: 4, Minor: 27, Type: c

hellmersjl · October 4, 2018, 8:10pm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty26, Major: 4, Minor: 26, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty25, Major: 4, Minor: 25, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty24, Major: 4, Minor: 24, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty23, Major: 4, Minor: 23, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty22, Major: 4, Minor: 22, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty21, Major: 4, Minor: 21, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty20, Major: 4, Minor: 20, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty19, Major: 4, Minor: 19, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty18, Major: 4, Minor: 18, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty17, Major: 4, Minor: 17, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty16, Major: 4, Minor: 16, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty15, Major: 4, Minor: 15, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty14, Major: 4, Minor: 14, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty13, Major: 4, Minor: 13, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty12, Major: 4, Minor: 12, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty11, Major: 4, Minor: 11, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty10, Major: 4, Minor: 10, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty9, Major: 4, Minor: 9, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty8, Major: 4, Minor: 8, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty7, Major: 4, Minor: 7, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty6, Major: 4, Minor: 6, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty5, Major: 4, Minor: 5, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty4, Major: 4, Minor: 4, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty3, Major: 4, Minor: 3, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty2, Major: 4, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty1, Major: 4, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcsa1, Major: 7, Minor: 129, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcs1, Major: 7, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcsa, Major: 7, Minor: 128, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vcs, Major: 7, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty0, Major: 4, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/console, Major: 5, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/tty, Major: 5, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/oldmem, Major: 1, Minor: 12, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/kmsg, Major: 1, Minor: 11, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/urandom, Major: 1, Minor: 9, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/random, Major: 1, Minor: 8, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/full, Major: 1, Minor: 7, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/zero, Major: 1, Minor: 5, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/port, Major: 1, Minor: 4, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/null, Major: 1, Minor: 3, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/mem, Major: 1, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vga_arbiter, Major: 10, Minor: 63, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/dri/card1, Major: 226, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/dri/renderD129, Major: 226, Minor: 129, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/dri/card0, Major: 226, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/dri/renderD128, Major: 226, Minor: 128, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/hwC0D0, Major: 116, Minor: 17, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC0D0c, Major: 116, Minor: 16, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC0D0p, Major: 116, Minor: 15, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/controlC0, Major: 116, Minor: 14, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/hwC1D0, Major: 116, Minor: 13, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC1D9p, Major: 116, Minor: 12, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC1D8p, Major: 116, Minor: 11, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC1D7p, Major: 116, Minor: 10, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC1D3p, Major: 116, Minor: 9, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/controlC1, Major: 116, Minor: 8, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/hwC2D0, Major: 116, Minor: 7, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC2D9p, Major: 116, Minor: 6, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC2D8p, Major: 116, Minor: 5, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC2D7p, Major: 116, Minor: 4, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/pcmC2D3p, Major: 116, Minor: 3, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/controlC2, Major: 116, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/seq, Major: 116, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/snd/timer, Major: 116, Minor: 33, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/net/tun, Major: 10, Minor: 200, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/vfio/vfio, Major: 10, Minor: 196, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bsg/1:0:0:0, Major: 251, Minor: 3, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bsg/0:0:0:0, Major: 251, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bsg/end_device-0:0, Major: 251, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bsg/sas_host0, Major: 251, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/mapper/control, Major: 10, Minor: 236, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event15, Major: 13, Minor: 79, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event14, Major: 13, Minor: 78, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event13, Major: 13, Minor: 77, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event12, Major: 13, Minor: 76, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event11, Major: 13, Minor: 75, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event10, Major: 13, Minor: 74, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event9, Major: 13, Minor: 73, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event8, Major: 13, Minor: 72, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event7, Major: 13, Minor: 71, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event6, Major: 13, Minor: 70, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event5, Major: 13, Minor: 69, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event4, Major: 13, Minor: 68, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event3, Major: 13, Minor: 67, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event2, Major: 13, Minor: 66, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event1, Major: 13, Minor: 65, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/event0, Major: 13, Minor: 64, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/input/mice, Major: 13, Minor: 63, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bus/usb/004/001, Major: 189, Minor: 384, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bus/usb/003/001, Major: 189, Minor: 256, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bus/usb/002/002, Major: 189, Minor: 129, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bus/usb/002/001, Major: 189, Minor: 128, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bus/usb/001/002, Major: 189, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/bus/usb/001/001, Major: 189, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/raw/rawctl, Major: 162, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/microcode, Major: 10, Minor: 184, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/15/cpuid, Major: 203, Minor: 15, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/15/msr, Major: 202, Minor: 15, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/14/cpuid, Major: 203, Minor: 14, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/14/msr, Major: 202, Minor: 14, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/13/cpuid, Major: 203, Minor: 13, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/13/msr, Major: 202, Minor: 13, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/12/cpuid, Major: 203, Minor: 12, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/12/msr, Major: 202, Minor: 12, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/11/cpuid, Major: 203, Minor: 11, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/11/msr, Major: 202, Minor: 11, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/10/cpuid, Major: 203, Minor: 10, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/10/msr, Major: 202, Minor: 10, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/9/cpuid, Major: 203, Minor: 9, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/9/msr, Major: 202, Minor: 9, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/8/cpuid, Major: 203, Minor: 8, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/8/msr, Major: 202, Minor: 8, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/7/cpuid, Major: 203, Minor: 7, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/7/msr, Major: 202, Minor: 7, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/6/cpuid, Major: 203, Minor: 6, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/6/msr, Major: 202, Minor: 6, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/5/cpuid, Major: 203, Minor: 5, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/5/msr, Major: 202, Minor: 5, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/4/cpuid, Major: 203, Minor: 4, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/4/msr, Major: 202, Minor: 4, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/3/cpuid, Major: 203, Minor: 3, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/3/msr, Major: 202, Minor: 3, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/2/cpuid, Major: 203, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/2/msr, Major: 202, Minor: 2, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/1/cpuid, Major: 203, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/1/msr, Major: 202, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/0/cpuid, Major: 203, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/cpu/0/msr, Major: 202, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_discover_gpus: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;NVIDIA SMI command: [‘/usr/bin/nvidia-smi’, ‘-q’, ‘-x’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;root.tag: nvidia_smi_log
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;GPUs: {‘nvidia0’: ‘0000:03:00.0’, ‘nvidia1’: ‘0000:04:00.0’}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/nvidia0, Major: 195, Minor: 0, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Path: /dev/nvidia1, Major: 195, Minor: 1, Type: c
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: tty
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: vc
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: rtc
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: misc
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: pps
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: raw
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: thermal
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: ata_port
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: cpuid
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: watchdog
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: ptp
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: scsi_disk
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: powercap
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: hwmon
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: ata_device
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: bdi
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: msr
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: sas_end_device
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: wmi
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: i2c-adapter
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: input
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: net
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: usbmon
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: mdio_bus
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: ata_link
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: mem
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: sas_phy
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: mei
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: sas_device
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: sas_port
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: scsi_device
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: graphics
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: bsg
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: sound
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: vtconsole
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: sas_host
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: dmi
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: scsi_host
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: scsi_generic
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: devices
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: drm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: pci_bus
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_add_device_counts_to_numa_nodes: Device class: block
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;NUMA nodes: {0: {‘MemTotal’: ‘67028780k’, ‘ngpus’: 2, ‘cpus’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘devices’: , ‘HugePages_Total’: 0}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_execjob_begin_handler: NodeConfig class instantiated
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_execjob_begin_handler: Host assigned job resources: {‘mem’: 8388608kb, ‘ncpus’: 4L, ‘vnodes’: {‘ds-rnd-gpu2[1]’: {‘mem’: 8388608kb, ‘ncpus’: 4L}}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;create_paths: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_create_slice: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing 1 to /sys/fs/cgroup/memory/pbspro.slice/memory.use_hierarchy
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_copy_from_parent: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing 0-15 to /sys/fs/cgroup/cpuset/pbspro.slice/cpuset.cpus
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_copy_from_parent: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing 0 to /sys/fs/cgroup/cpuset/pbspro.slice/cpuset.mems
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing 1 to /sys/fs/cgroup/memory/pbspro.slice/memory.use_hierarchy
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;delete: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Devices path is /sys/fs/cgroup/devices/pbspro.slice/pbspro-1249.ds\x2drnd\x2dgpu.slice/devices.list
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Appending b : rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Appending c : rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Appending c 195:* rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Appending c 244:0 rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called

hellmersjl · October 4, 2018, 8:10pm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;enabled: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Examining pbspro-[0-9]*
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_unescape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Job ID is 1249.ds-rnd-gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_get_assigned_cgroup_resources: Returning {‘1249.ds-rnd-gpu’: {‘cpuset’: {‘cpus’: [0, 1, 2, 3], ‘mems’: [0]}, ‘memsw’: {‘limit_in_bytes’: 8589934592}, ‘devices’: {‘list’: [‘b : rwm\n’, ‘c : rwm\n’, ‘c 195:* rwm\n’, ‘c 244:0 rwm\n’]}, ‘memory’: {‘limit_in_bytes’: 8589934592, ‘soft_limit_in_bytes’: 9223372036854771712}}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;create_job: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_create_slice: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_setup_subsys_devices: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Initial devices.list: [‘a : rwm’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing a : rwm to /sys/fs/cgroup/devices/pbspro.slice/pbspro-1250.ds\x2drnd\x2dgpu.slice/devices.deny
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Removing access to the following: [‘195:0’, ‘195:1’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing c 195:0 rwm to /sys/fs/cgroup/devices/pbspro.slice/pbspro-1250.ds\x2drnd\x2dgpu.slice/devices.deny
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing c 195:1 rwm to /sys/fs/cgroup/devices/pbspro.slice/pbspro-1250.ds\x2drnd\x2dgpu.slice/devices.deny
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Allowing access to the following: [‘b : rwm’, ‘c : rwm’, [‘nvidiactl’, ‘rwm’, ‘'], [‘nvidia-uvm’, ‘rwm’]]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;string item: b : rwm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing b : rwm to /sys/fs/cgroup/devices/pbspro.slice/pbspro-1250.ds\x2drnd\x2dgpu.slice/devices.allow
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: c 195:1 rwm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;string item: c : rwm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing c : rwm to /sys/fs/cgroup/devices/pbspro.slice/pbspro-1250.ds\x2drnd\x2dgpu.slice/devices.allow
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: c 195:1 rwm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Device allow: [‘nvidiactl’, ‘rwm’, '’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Stat file: /dev/nvidiactl
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing c 195:* rwm to /sys/fs/cgroup/devices/pbspro.slice/pbspro-1250.ds\x2drnd\x2dgpu.slice/devices.allow
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: c 195:* rwm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Device allow: [‘nvidia-uvm’, ‘rwm’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Stat file: /dev/nvidia-uvm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: writing c 244:0 rwm to /sys/fs/cgroup/devices/pbspro.slice/pbspro-1250.ds\x2drnd\x2dgpu.slice/devices.allow
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;write_value: c 244:0 rwm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Updated devices.list: [‘b : rwm’, ‘c : rwm’, ‘c 195:* rwm’, ‘c 244:0 rwm’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0100;pbs_python;Hook;pbs_python;create_job: Creating directory /sys/fs/cgroup/cpuset/pbspro.slice/pbspro-1250.ds\x2drnd\x2dgpu.slice/
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_cgroup_path: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;_systemd_escape: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;configure_job: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;get_memory_on_node: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;total mem: 67306762240
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;reserved mem: 67108864
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;remaining mem: 67238887424
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;get_memory_on_node: Returning: 65662976k
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;mem_avail 67238887424
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;default: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;get_vmem_on_node: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;total swap: 25606221824
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;get_memory_on_node: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;total mem: 67306762240
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;reserved mem: 67108864
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;remaining mem: 67238887424
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;get_memory_on_node: Returning: 65662976k
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;total mem: 67238887424
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;total vmem: 92845109248
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;reserved vmem: 67108864
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;remaining vmem: 92776955904
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;get_vmem_on_node: Returning: 90602496k
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;vmem_avail 92776955904
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;default: Method called
10/04/2018 12:52:18;0100;pbs_python;Hook;pbs_python;configure_job: vmem not requested, assigning 8388608k to cgroup
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;configure_job: INFO: vmem is enabled in the hook configuration file and should also be listed in the resources line of the scheduler configuration file
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available Keys: {‘devices’: , ‘MemTotal’: ‘67028780k’, ‘cpus’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘ngpus’: 2, ‘HugePages_Total’: 0}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available: {0: {‘devices’: , ‘MemTotal’: ‘67028780k’, ‘cpus’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘ngpus’: 2, ‘HugePages_Total’: 0}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available prior to device add: {0: {‘MemTotal’: ‘67028780k’, ‘ngpus’: 2, ‘cpus’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘devices’: , ‘memory’: 68637470720, ‘HugePages_Total’: 0}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: tty
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: vc
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: rtc
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: misc
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: pps
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: raw
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: thermal
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: ata_port
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: cpuid
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: watchdog
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: ptp
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: scsi_disk
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: powercap
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: hwmon
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: ata_device
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: bdi
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: msr
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: sas_end_device
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: wmi
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: i2c-adapter
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: input
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: net
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: usbmon
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: mdio_bus
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: ata_link
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: mem
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: sas_phy
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: mei
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: gpu
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Devices: {‘nvidia0’: {‘realpath’: ‘/sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0’, ‘major’: 195, ‘type’: ‘c’, ‘numa_node’: 0, ‘device’: ‘/dev/nvidia0’, ‘bus_id’: ‘0000:03:00.0’, ‘minor’: 0}, ‘nvidia1’: {‘realpath’: ‘/sys/devices/pci0000:00/0000:00:03.0/0000:04:00.0’, ‘major’: 195, ‘type’: ‘c’, ‘numa_node’: 0, ‘device’: ‘/dev/nvidia1’, ‘bus_id’: ‘0000:04:00.0’, ‘minor’: 1}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Device: gpu, Socket: 0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Device: gpu, Socket: 0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: sas_device
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: sas_port
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: scsi_device
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: graphics
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: bsg
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: sound
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: vtconsole
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: sas_host
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: dmi
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: scsi_host
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: scsi_generic
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: devices
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: drm
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: pci_bus
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;available_node_resources: Device Names: block
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available: {0: {‘MemTotal’: ‘67028780k’, ‘ngpus’: 2, ‘cpus’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘devices’: [‘nvidia0’, ‘nvidia1’], ‘memory’: 68637470720, ‘HugePages_Total’: 0}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Assigned: {‘1249.ds-rnd-gpu’: {‘cpuset’: {‘cpus’: [0, 1, 2, 3], ‘mems’: [0]}, ‘memsw’: {‘limit_in_bytes’: 8589934592}, ‘devices’: {‘list’: [‘b : rwm\n’, ‘c : rwm\n’, ‘c 195:* rwm\n’, ‘c 244:0 rwm\n’]}, ‘memory’: {‘limit_in_bytes’: 8589934592, ‘soft_limit_in_bytes’: 9223372036854771712}}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Running: [‘/opt/pbs/bin/printjob’, ‘-a’, ‘/var/spool/pbs/mom_priv/jobs/1249.ds-rnd-gpu.JB’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;cpus: [0, 1, 2, 3], sockets: [0], memory limit: 8589934592
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;devices: [‘b : rwm\n’, ‘c : rwm\n’, ‘c 195:* rwm\n’, ‘c 244:0 rwm\n’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Sockets: [0] Available: {0: {‘MemTotal’: ‘67028780k’, ‘ngpus’: 2, ‘cpus’: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘devices’: [‘nvidia0’, ‘nvidia1’], ‘memory’: 68637470720, ‘HugePages_Total’: 0}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Decrementing memory: 68637470720 by 8589934592
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Assigned device to 1249.ds-rnd-gpu: [‘b : rwm\n’, ‘c : rwm\n’, ‘c 195:* rwm\n’, ‘c 244:0 rwm\n’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Check device: b : rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available device: [‘nvidia0’, ‘nvidia1’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Check device: c : rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available device: [‘nvidia0’, ‘nvidia1’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Check device: c 195:* rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available device: [‘nvidia0’, ‘nvidia1’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Check device: c 244:0 rwm

10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available device: [‘nvidia0’, ‘nvidia1’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;get_device_name: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Get device name: major: 244, minor: 0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Possible devices: [‘nvidia0’, ‘nvidia1’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Checking device: nvidia0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Check gpu device: nvidia0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Device major: 244, minor: 0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Checking device: nvidia1
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Check gpu device: nvidia1
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Device major: 244, minor: 0
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;No match found
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Returned device: None
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Available resources: {0: {‘MemTotal’: ‘67028780k’, ‘ngpus’: 2, ‘cpus’: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘devices’: [‘nvidia0’, ‘nvidia1’], ‘memory’: 60047536128, ‘HugePages_Total’: 0}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;assign_job: Method called
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Requested: {‘mem’: 8388608kb, ‘vmem’: 8388608kb, ‘ncpus’: 4L, ‘vnodes’: {‘ds-rnd-gpu2[1]’: {‘mem’: 8388608kb, ‘ncpus’: 4L}}}, Available: {0: {‘MemTotal’: ‘67028780k’, ‘ngpus’: 2, ‘cpus’: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘devices’: [‘nvidia0’, ‘nvidia1’], ‘memory’: 60047536128, ‘HugePages_Total’: 0}}, Numa Nodes: {0: {‘MemTotal’: ‘67028780k’, ‘ngpus’: 2, ‘cpus’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], ‘devices’: , ‘HugePages_Total’: 0}}
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;Current target is vnode ds-rnd-gpu2[1]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;/var/spool/pbs/mom_priv/cgroups.lock file lock released by main
10/04/2018 12:52:18;0080;pbs_python;Hook;pbs_python;[‘Traceback (most recent call last):’, ’ File “”, line 4597, in main’, ’ File “”, line 724, in invoke_handler’, ’ File “”, line 754, in _execjob_begin_handler’, ’ File “”, line 3802, in configure_job’, ’ File “”, line 3249, in assign_job’, ‘KeyError: 1’]
10/04/2018 12:52:18;0800;pbs_python;Hook;pbs_python;event_name: Method called
10/04/2018 12:52:18;0001;pbs_python;Hook;pbs_python;Unexpected error in pbs_cgroups handling execjob_begin event for job 1250.ds-rnd-gpu (suspended): KeyError (1,)
10/04/2018 12:52:18;0080;pbs_python;Hook;pbs_python;Elapsed time: 0.2095
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;pbs_populate_svrattrl_from_python_class==>
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Error_Path al_resc=null al_value=ds-rnd-gpu:/data/jhs4/prod/sleeplong.sh.e1250 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=hashname al_resc=null al_value=1250.ds-rnd-gpu al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=run_version al_resc=null al_value=1 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=schedselect al_resc=null al_value=1:ncpus=4:mem=8gb:mpiprocs=4:Qlist=workq al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=server al_resc=null al_value=ds-rnd-gpu al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=egroup al_resc=null al_value=users al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Variable_List al_resc=null al_value=PBS_O_SYSTEM=Linux,PBS_O_SHELL=/bin/bash,PBS_O_HOME=/users/JHS4,PBS_O_HOST=ds-rnd-gpu,PBS_O_LOGNAME=jhs4,PBS_O_WORKDIR=/data/jhs4/prod,PBS_O_LANG=en_US.UTF-8,PBS_O_PATH=/opt/pbs/bin:/usr/lib64/qt-3.3/bin:/users/JHS4/perl5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/users/JHS4/bin,PBS_O_QUEUE=workq,PBS_O_MAIL=/var/spool/mail/jhs4 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Checkpoint al_resc=null al_value=u al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=exec_vnode al_resc=null al_value=(ds-rnd-gpu2[1]:ncpus=4:mem=8388608kb) al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=exec_host2 al_resc=null al_value=ds-rnd-gpu2:15002/1*4 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=run_count al_resc=null al_value=1 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Job_Name al_resc=null al_value=sleeplong.sh al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=euser al_resc=null al_value=jhs4 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=project al_resc=null al_value=_pbs_project_default al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Output_Path al_resc=null al_value=ds-rnd-gpu:/data/jhs4/prod/sleeplong.sh.o1250 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=job_kill_delay al_resc=null al_value=10 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Hold_Types al_resc=null al_value=s al_flags=64
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Join_Path al_resc=null al_value=n al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=mtime al_resc=null al_value=1538682738 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Resource_List al_resc=ncpus,long al_value=4 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Resource_List al_resc=mem,size al_value=8gb al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Resource_List al_resc=place,string al_value=free al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Keep_Files al_resc=null al_value=n al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=Job_Owner al_resc=null al_value=jhs4@ds-rnd-gpu al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;pbs_populate_svrattrl_from_python_class==>
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=ds-rnd-gpu2[1].resources_assigned al_resc=ncpus,long al_value=16 al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=ds-rnd-gpu2[1].resources_assigned al_resc=mem,size al_value=33554432kb al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;print_svrattrl_list;al_name=ds-rnd-gpu2[1].name al_resc=null al_value=ds-rnd-gpu2[1] al_flags=0
10/04/2018 12:52:18;0400;pbs_python;Hook;fprint_svrattrl_list;pbs.event().job
10/04/2018 12:52:18;0400;pbs_python;Hook;fprint_svrattrl_list;pbs.event().vnode_list
10/04/2018 12:52:18;0400;pbs_python;Svr;pbs_python;–> Stopping Python interpreter ←
10/04/2018 12:52:18;0400;pbs_mom;Hook;pbs_cgroups;finished
10/04/2018 12:52:18;0100;pbs_mom;Hook;pbs_cgroups;execjob_begin request rejected by ‘pbs_cgroups’
10/04/2018 12:52:18;0008;pbs_mom;Job;1250.ds-rnd-gpu;Unexpected error in pbs_cgroups handling execjob_begin event for job 1250.ds-rnd-gpu (suspended): KeyError (1,)
10/04/2018 12:52:18;0100;pbs_mom;Req;;Type 3 request received from root@10.0.0.100:15001, sock=1
10/04/2018 12:52:18;0080;pbs_mom;Req;req_reject;Reject reply code=15004, aux=0, type=3, from root@10.0.0.100:15001
10/04/2018 12:52:18;0100;pbs_mom;Req;;Type 5 request received from root@10.0.0.100:15001, sock=1
10/04/2018 12:52:18;0080;pbs_mom;Req;req_reject;Reject reply code=15001, aux=0, type=5, from root@10.0.0.100:15001
10/04/2018 12:52:18;0800;pbs_mom;n/a;mom_get_sample;nprocs: 303, cantstat: 0, nomem: 0, skipped: 0, cached: 0, max excluded PID: 0

mkaro · October 5, 2018, 3:18pm

Thank you for supplying the logs.

Do you have a vnode definition file in place? If so, it may be interfering with what the server thinks is present and what the cgroup hook detects.

Is the problem happening on one node or all of them?

If it were me, I’d delete the node (and all associated vnodes) and reconfigure it from scratch. Do not create a vnode definition file. Instead, set vnode_per_numa_node to true in your cgroup hook configuration file.

Stop mom on the execution node
Delete the nodes and associated vnodes in the server
Restart the server (otherwise, it will “remember” the old configuration when you restart mom)
Move PBS_HOME (/var/spool/pbs) out of the way (e.g. /var/spool/pbs.old) so it will be created again from scratch
Configure the node accordingly. Don’t create vnodes, let the hook do it for you.

hellmersjl · October 5, 2018, 5:57pm

This is happening on all the execution nodes. I was using a vnode definition file, but as per your recommendation, I have removed that.

After performing your recommended changes the jobs are still being held. I’m getting the message in the mom_log

10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;Job not found: 1253.ds-rnd-gpu
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;Requeuing job 1253.ds-rnd-gpu
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;Run count for job 1253.ds-rnd-gpu: 6
10/05/2018 09:38:52;0001;pbs_python;Hook;pbs_python;Processing error in pbs_cgroups handling execjob_begin event for job 1253.ds-rnd-gpu: CgroupProcessingError (‘Failed to assign resources’,)
10/05/2018 09:38:52;0008;pbs_mom;Job;1253.ds-rnd-gpu;Processing error in pbs_cgroups handling execjob_begin event for job 1253.ds-rnd-gpu: CgroupProcessingError (‘Failed to assign resources’,)
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/systemd/pbspro.slice/pbspro-1253.ds\x2drnd\x2dgpu.slice
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/memory/pbspro.slice/pbspro-1253.ds\x2drnd\x2dgpu.slice
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/blkio/pbspro.slice/pbspro-1253.ds\x2drnd\x2dgpu.slice
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/cpu,cpuacct/pbspro.slice/pbspro-1253.ds\x2drnd\x2dgpu.slice
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/devices/pbspro.slice/pbspro-1253.ds\x2drnd\x2dgpu.slice
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;_remove_cgroup: Removing directory /sys/fs/cgroup/cpuset/pbspro.slice/pbspro-1253.ds\x2drnd\x2dgpu.slice
10/05/2018 09:38:52;0100;pbs_python;Hook;pbs_python;create_job: Creating directory /sys/fs/cgroup/cpuset/pbspro.slice/pbspro-1253.ds\x2drnd\x2dgpu.slice/

So it’s not assigning the resources. (note: log setting is not verbose right now).

Note: that these compute nodes are hyperthreading. Could this be causing a problem?

hellmersjl · October 9, 2018, 3:17pm

I went ahead and turned off hyperthreading at the BIOS level, and that seems to have helped. Unfortunately my NVIDIA drivers want to be reinstalled for some reason.

mkaro · October 9, 2018, 5:19pm

Sorry for not responding sooner, it’s been a very busy week. I don’t want to send you on a wild goose chase for no reason, but you may want to try the following: leave hyperthreading enabled (presumably you want this setting enabled since it was enabled in the BIOS) and set “use_hyperthreads” to true in the cgroup hook config file. That may prevent you from having to reinstall the NVIDIA drivers. No idea why you should have to reinstall them in the first place, but that’s a question for NVIDIA.

hellmersjl · October 9, 2018, 7:15pm

Actually I’ve wanted turn off hyperthreading for awhile now, so that is the setting I prefer.

Changing to not use hyperthreading seems to have really helped.

I’m not sure if it is a PBS issue or not, but the jobs seems to “stall” out at some point and eventually just die without reporting an error. I’m not sure if this is an application issue for us, or a PBS problem.

If I have a job that is using the vnode using all the gpus and only one of the cores, do all of the cores get reserved for the job?

mkaro · October 9, 2018, 8:05pm

Under the covers, the cgroup hook is creating directories and manipulating file contents under /sys/fs/cgroup (or wherever your cgroup filesystems are mounted). If I wanted to know which CPUs PBS Pro reserved for my job I would look at the contents of /sys/fs/cgroup/cpuset/pbspro.slice/[server]-[jobid]/cpuset.cpus where [server] and [jobid] are relative to your environment. If I wanted to know which CPUs are available to PBS Pro I would look in /sys/fs/cgroup/cpuset/pbspro.slice/cpuset.cpus. If the content of the two files match, then all CPUs have been allocated to the job.

You might try some “sleep” jobs to see if they return as expected. While the job is running, you can inspect the contents of the cgroup files. Or you can submit an interactive job (qsub -I {that’s a capital i}) so that PBS Pro gives you a shell that you may use to explore. Can’t say for certain if PBS Pro is the culprit here, but this should help determine whether it is.

hellmersjl · October 9, 2018, 9:27pm

I’ll look at that.

We have another problem. When we run a job with -l ngpus=1, it seems to lookup the whole vnode so that when we submit another job requesting one gpu it will never get scheduled on that node, until the first job is done.
Originally I had defined vnodes with one gpu a piece specifically to resolve this problem, but not it’s back since we are letting pbs_cgroups define the vnodes.

hellmersjl · October 18, 2018, 6:18pm

Hey Mike,

I just wanted to let you know that my issue has been resolved. The last problem we had was that the job was exceeding the allocated memory and that was causing our problem.

Thanks for all the help!!

Joe

mkaro · October 18, 2018, 6:28pm

Glad to hear you got it working! Pesky computers always doing exactly what they’re told.

Topic		Replies	Views
CgroupProcessingError ('Failed to assign resources') Users/Site Administrators	1	1030	December 6, 2019
Jobs killed by pbs_mom due to resource violation Users/Site Administrators	3	758	September 25, 2019
Failed to assign resources to job Users/Site Administrators	9	1413	May 26, 2022
Cgroups not cleaning up after job kill, further jobs unable to assign resources Users/Site Administrators	1	564	July 3, 2023
Cgroups hook write permission error Users/Site Administrators	5	902	July 1, 2022

Cgroup error causing suspended jobs

Related Topics