# Can't start PBS server (15011 decode\_attr\_db job\_sort\_formula) **URL:** \ **Category:** Users/Site Administrators\ **Created:** [March 26, 2022, 9:03pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051 "2022-03-26T21:03:30Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![christopher](https://avatars.discourse-cdn.com/v4/letter/c/439d5e/32.png) [@christopher](https://community.openpbs.org/u/christopher)\ **Post date:** [March 26, 2022, 9:03pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/1 "2022-03-26T21:03:30Z")
Have a working version of OpenPBS on a test box. After a reboot it will start all daemons but the server. Error below from server logs. [root@service0 server\_logs]# ps -ef | grep pbs root 56326 1 0 09:28 ? 00:00:00 /opt/pbs/sbin/pbs\_comm root 56341 1 0 09:28 ? 00:00:00 /opt/pbs/sbin/pbs\_sched root 56454 1 0 09:28 ? 00:00:13 /opt/pbs/sbin/pbs\_ds\_monitor monitor postgres 56512 1 0 09:28 ? 00:00:00 /usr/bin/postgres -D /p/home/pbs/server/datastore -p 15007 [root@service0 server\_logs]# more 20220326 03/26/2022 09:28:55;0002;Server@service0;Svr;Log;Log opened 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;pbs\_version=20.0.0 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;pbs\_build=mach=N/A:security=N/A:configure\_args=N/A 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;hostname=service0.ib0.ice-tds.erdc.hpc.mil;pbs\_leaf\_name=N/A;pbs\_mom\_node\_name=N/A 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;ipv4 interface lo: localhost4.localdomain4 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;ipv4 interface ib0: service0.ib0.ice-tds.erdc.hpc.mil 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;ipv4 interface bond0: service0.head.ice-tds.erdc.hpc.mil 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;ipv6 interface lo: localhost6.localdomain6 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;ipv6 interface ib0: service0 03/26/2022 09:28:55;0002;Server@service0;Svr;Server@service0;ipv6 interface bond0: service0 03/26/2022 09:28:55;0006;Server@service0;Fil;Server@service0;Version 20.0.0, started, initialization type = 1 03/26/2022 09:28:56;0002;Server@service0;Svr;Server@service0;pbs\_status\_db exit code 1 03/26/2022 09:28:56;0002;Server@service0;Svr;Server@service0;Starting PBS dataservice 03/26/2022 09:29:00;0002;Server@service0;Svr;Server@service0;connected to PBS dataservice@service0.ib0.ice-tds.erdc.hpc.mil 03/26/2022 09:29:00;0086;Server@service0;Svr;pbs\_python\_ext\_quick\_start\_interpreter;–\> Python Interpreter quick started, compiled with version:'3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]’ ← 03/26/2022 09:29:00;0086;Server@service0;Svr;pbs\_python\_ext\_quick\_start\_interpreter;–\> Inserted Altair PBS Python modules dir ‘/opt/pbs/lib/python/altair’ ‘/opt/pbs/lib/python/altair /pbs/v1’\<– 03/26/2022 09:29:00;0086;Server@service0;Svr;pbs\_python\_ext\_quick\_shutdown\_interpreter;–\> Stopping Python interpreter ← 03/26/2022 09:29:00;0d80;Server@service0;TPP;Server@service0(Main Thread);TPP authentication method = resvport 03/26/2022 09:29:00;0c06;Server@service0;TPP;Server@service0(Main Thread);TPP leaf node names = 10.148.0.41:15001,127.0.0.1:15001,10.148.0.41:15001,172.23.0.22:15001 03/26/2022 09:29:00;0d80;Server@service0;TPP;Server@service0(Main Thread);Initializing TPP transport Layer 03/26/2022 09:29:00;0d80;Server@service0;TPP;Server@service0(Main Thread);Max files allowed = 16384 03/26/2022 09:29:00;0d80;Server@service0;TPP;Server@service0(Main Thread);TPP initialization done 03/26/2022 09:29:00;0d80;Server@service0;TPP;Server@service0(Main Thread);Connecting to pbs\_comm service0:17001 03/26/2022 09:29:00;0c06;Server@service0;TPP;Server@service0(Thread 0);Thread ready 03/26/2022 09:29:00;0c06;Server@service0;TPP;Server@service0(Thread 0);Registering address 10.148.0.41:15001 to pbs\_comm service0:17001 03/26/2022 09:29:00;0c06;Server@service0;TPP;Server@service0(Thread 0);Registering address 172.23.0.22:15001 to pbs\_comm service0:17001 03/26/2022 09:29:00;0c06;Server@service0;TPP;Server@service0(Thread 0);Connected to pbs\_comm service0:17001 03/26/2022 09:29:00;0002;Server@service0;n/a;setup\_env;read environment from /p/home/pbs/server/pbs\_environment 03/26/2022 09:29:00;0000;Server@service0;Svr;Server@service0;Supported authentication method: resvport 03/26/2022 09:29:00;0004;Server@service0;Svr;Server@service0;node\_fail\_requeue value changed to 0 03/26/2022 09:29:00;0001;Server@service0;Svr;Server@service0;PBS server internal error (15011) in decode\_attr\_db, Action function failed for job\_sort\_formula attr, errn 15011 Thanks, Chris ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [March 28, 2022, 6:32am UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/2 "2022-03-28T06:32:36Z")
@christopher Could you please share the output of `qstat -Bf and qmgr -c "p s"` ---
**Author:** ![christopher](https://avatars.discourse-cdn.com/v4/letter/c/439d5e/32.png) [@christopher](https://community.openpbs.org/u/christopher)\ **Post date:** [March 28, 2022, 4:15pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/3 "2022-03-28T16:15:03Z")
The server won’t start, so no ability to run qstat or qmgr. There would be no jobs in the queue as this system is very lightly used. I don’t unfortunately have a current qmgr backup. But its very simpler to our other PBSPro systems. This system was changed to OpenPBS in Jan and has been working since then. ---
**Author:** ![christopher](https://avatars.discourse-cdn.com/v4/letter/c/439d5e/32.png) [@christopher](https://community.openpbs.org/u/christopher)\ **Post date:** [March 28, 2022, 4:20pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/4 "2022-03-28T16:20:24Z")
The formula would look something like: set server job\_sort\_formula = queue\_priority + (0.005 \* ncpus) + (0.000277778 \* eligible\_time) + paws + bias create resource paws set resource paws type = float set resource paws flag = r set server resources\_default.paws = 0 create resource bias set resource bias type = float set resource bias flag = r set server resources\_default.bias = 0 ---
**Author:** ![subhasisb](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/subhasisb/32/12_2.png) [@subhasisb](https://community.openpbs.org/u/subhasisb)\ **Post date:** [March 29, 2022, 4:49am UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/5 "2022-03-29T04:49:06Z")
Hi, You can remove the job\_sort\_formula from the database directly. Keep a copy of PBS home, then run the following steps: 1. run pbs\_ds\_password to set a known password for the database 2. Start postgres by running pbs\_dataservice start 3. Login to the database using psql -p 15007 -d pbs\_datastore -U postgres 4. select \* from pbs.server; 5. update pbs.server set attributes = delete(attributes, ‘job\_sort\_formula’); 6. Stop database using pbs\_dataservice stop 7. Restart pbs services For the specific job\_sort\_formula that is causing the server restart failure, you could submit a bug to be fixed by the community. Regards, Subhasis ---
**Author:** ![christopher](https://avatars.discourse-cdn.com/v4/letter/c/439d5e/32.png) [@christopher](https://community.openpbs.org/u/christopher)\ **Post date:** [March 29, 2022, 2:33pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/6 "2022-03-29T14:33:32Z")
Thank you Subhasis. From “select \*” I see my formula as: “job\_sort\_formula”=\>“11.queue\_priority+(0.005\*ncpus)+(0.000277778\*eligible\_time)+paws+bias” However the delete doesn’t work: pbs\_datastore=# update pbs.server set attributes = delete(attributes, ‘job\_sort\_formula’); ERROR: column “‘job\_sort\_formula’” does not exist LINE 1: …te pbs.server set attributes = delete(attributes, ‘job\_sort\_… ^ pbs\_datastore=# update pbs.server set attributes = delete(attributes, “job\_sort\_formula”); ERROR: column “job\_sort\_formula” does not exist LINE 1: …te pbs.server set attributes = delete(attributes, "job\_sort\_… ^ pbs\_datastore=# update pbs.server set attributes = delete(attributes, job\_sort\_formula); ERROR: column “job\_sort\_formula” does not exist LINE 1: …te pbs.server set attributes = delete(attributes, job\_sort\_f… ^ Chris ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [March 29, 2022, 3:34pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/7 "2022-03-29T15:34:38Z")
> [@christopher](#): > > pbs\_datastore=# update pbs.server set attributes = delete(attributes, ‘job\_sort\_formula’); > ERROR: column “‘job\_sort\_formula’” does not exist This is probably the copy paste error inserting special quotes surrounding the job\_sort\_formula, , if we type it we will see the debug message > [@subhasisb](#): > > update pbs.server set attributes = delete(attributes, ‘job\_sort\_formula’); ERROR: function delete(public.hstore, unknown) does not exist LINE 1: update pbs.server set attributes = delete(attributes,'job\_so… ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts. ---
**Author:** ![christopher](https://avatars.discourse-cdn.com/v4/letter/c/439d5e/32.png) [@christopher](https://community.openpbs.org/u/christopher)\ **Post date:** [March 29, 2022, 3:56pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/8 "2022-03-29T15:56:31Z")
Ahh, requires single quotes but not Microsoft quotes. LOL. Thanks. The server is started now. ---
**Author:** ![christopher](https://avatars.discourse-cdn.com/v4/letter/c/439d5e/32.png) [@christopher](https://community.openpbs.org/u/christopher)\ **Post date:** [March 29, 2022, 4:07pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/9 "2022-03-29T16:07:09Z")
How do I submit a bug report? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [March 29, 2022, 4:50pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/10 "2022-03-29T16:50:53Z")
> **[Issues · openpbs/openpbs](https://github.com/openpbs/openpbs/issues)** > > An HPC workload manager and job scheduler for desktops, clusters, and clouds. - Issues · openpbs/openpbs ---
**Author:** ![weallcock](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/weallcock/32/466_2.png) [@weallcock](https://community.openpbs.org/u/weallcock)\ **Post date:** [April 4, 2022, 4:12pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/11 "2022-04-04T16:12:57Z")
Could this be related? [Crash on server restart if job\_sort\_formula is set · Issue #2475 · openpbs/openpbs · GitHub](https://github.com/openpbs/openpbs/issues/2475) ---
**Author:** ![christopher](https://avatars.discourse-cdn.com/v4/letter/c/439d5e/32.png) [@christopher](https://community.openpbs.org/u/christopher)\ **Post date:** [April 12, 2022, 5:25pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/12 "2022-04-12T17:25:37Z")
I submitted a new bug on this, but #2475 is the same issue. It was opened Aug 2021. How can the formula be broken for almost a year? Thanks, Chris ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [April 26, 2022, 6:31pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/13 "2022-04-26T18:31:56Z")
I played around with this a little. The following hack allows the server to come all the way up, giving you a chance to fix any issues with job\_sort\_formula. It is not a complete fix in that it does not detect a bad formula when reloading from the database (but, if the database is corrupted, you have other issues). I did not find any other problems in my (limited) testing. ```auto diff --git a/src/server/sched_func.c b/src/server/sched_func.c index d6b4f285..2c69c372 100644 --- a/src/server/sched_func.c +++ b/src/server/sched_func.c @@ -162,8 +162,13 @@ validate_job_formula(attribute *pattr, void *pobject, int actmode) return PBSE_SVR_SCHED_JSF_INCOMPAT; } - if (!Py_IsInitialized()) + if (!Py_IsInitialized()) { + if (get_sattr_long(SVR_ATR_State) == SV_STATE_INIT) { + /* Cannot validate during initialization */ + return PBSE_NONE; + } return PBSE_INTERNAL; + } globals1 = malloc(globals_size1); if (globals1 == NULL) { ``` ---
**Author:** ![weallcock](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/weallcock/32/466_2.png) [@weallcock](https://community.openpbs.org/u/weallcock)\ **Post date:** [May 12, 2022, 5:50pm UTC](https://community.openpbs.org/t/cant-start-pbs-server-15011-decode-attr-db-job-sort-formula/3051/14 "2022-05-12T17:50:32Z")
Altair just merged a fix for this: [Fix server not starting with job\_sort\_formula set by bayucan · Pull Request #2533 · openpbs/openpbs · GitHub](https://github.com/openpbs/openpbs/pull/2533)