Error importing hook when installing kubernetes-pbs-pro-connector

Users/Site Administrators
1

Hi everyone, I’m having trouble installing the necessary PBS Pro hooks.

From within qmgr, I receive an error for both of the necessary import hook commands as follows:

import hook pbs-kubernetes application/x-python base64 pbs_kubernetes.PY

Output:

qmgr obj=pbs-kubernetes svr=default: <content-type> must be application/x-python or application/x-config
qmgr: hook error returned from server

import hook pbs-kubernetes application/x-config base64 pbs_kubernetes.CF

Output:

qmgr obj=pbs-kubernetes svr=default: Unable to decode...bad input
qmgr: hook error returned from server

I’m not quite sure what is going on.
OS: CentOS 7
pbs v20.0.0

2

Hi @ wuyadang
Could you post the contents of pbs_kubernetes.CF file and the output of qmgr -c “list hook”

3

Are you sure you want “base64” for the encodings for the hook and its config file? I always used “default”. I think base64 is only for reloading the result of a “print hook” command.

4

Hi @saimanikant, thanks for the response!

My pbs_kubernetes.CF file contents:

{
    "kubelet_config": "/etc/kubernetes/"
}

I’ve also tried pointing to the actual file, for example:

{
    "kubelet_config": "/etc/kubernetes/kubelet-config.yaml"
}

Here is the output of qmgr -c "list hook":

Hook pbs_cgroups
    type = site
    enabled = false
    event = execjob_begin,execjob_epilogue,execjob_end,execjob_launch,
        execjob_attach,
        execjob_resize,
        execjob_abort,
        execjob_postsuspend,
        execjob_preresume,
        exechost_periodic,
        exechost_startup
    user = pbsadmin
    alarm = 90
    freq = 120
    order = 100
    debug = false
    fail_action = offline_vnodes

Hook pbs-kubernetes
    type = site
    enabled = true
    event = execjob_end,execjob_launch
    user = pbsadmin
    alarm = 30
    order = 1
    debug = false
    fail_action = none```
5

Hi @dtalcott, thanks for the response!

TBH, I’m just implementing per the kubernetes-pbspro-connector’s Installation README on github.

I tried what you suggested and used “default”, and the qmgr import command ran without problem! Now, to test the functionality…

I’ve never dealt with pbs hooks before this, so not entirely sure what’s going on, but I’ll post updates.

7

Sorry, accidentally deleted my last reply:
Importing the pbs hook with “default” instead of “base64”, as suggested by @dtalcott:

After starting kubectl proxy , pointing the kubernetes.go file to API proxy, and starting the scheduler, the scheduler runs, but stops every time i run

kubectl apply -f <xxx.yaml>

and returns the message:
exit status 175 message.

8

The custom scheduler submits a PBS job here:

You can change here and give the complete path to qsub executable here.
Also set the path accordingly for qstat commands in kubernetes.go file (line number 254 and 300)

10

hmm. i modified the lines you mentioned above, still getting exit status 175 .

Interestingly, if I remove the resource requests/limits from the pod’s yaml spec, this doesn’t happen, but I receive a strconv.Atoi: parsing "": invalid syntax error.

11

Today I have same problem. Have you already resolved this?

I got the error message below.
qmgr obj=pbs-kubernetes svr=default: Incorrect padding
qmgr: hook error returned from server
qmgr obj=pbs-kubernetes svr=default: Unable to decode…bad input
qmgr: hook error returned from server

I set up the kubernetes environment with Rancher rke.
My Environment is below
OS: RHEL 8.6
PBS: 20.0.1
Kubernetes: RKE (1.23.6)

Best Regards

12

Where can I find kubernetes-pbs-pro-connector now ? I was looking at the Github made available here https://github.com/PBSPro/kubernetes-pbspro-connector but seems not be to be there anymore ?

Best Regards
Carlo