Upgrade fails from 20.0.1 to 22.05.11

What a rabbit hole! I think recent updates to OpenPBS have broken the database upgrade path.

This is what I think is happening at a high level. The startup script notices the version of PBS has changed, so invokes pbs_habitat to fix things. pbs_habitat asks pbs_db_utility to update the database. pbs_db_utility looks for the directory PBS_HOME/pgsql.forupgrade and gives up with the messages shown near the top of your journalctl log if the directory cannot be found.

It turns out that the pgsql.forupgrade directory is created by pbs_habitat itself, but only after pbs_db_utility is run.

I have a completely untested hack that you could try just to get past this issue:

index 0eecd0e0..9b0b34be 100644
--- a/src/cmds/scripts/pbs_habitat.in
+++ b/src/cmds/scripts/pbs_habitat.in
@@ -311,6 +311,7 @@ if [ "${PBS_START_SERVER:-0}" != 0 ] ; then
        export PBS_ENVIRONMENT
 
        if [ $create_new_svr_data -eq 0 ]; then
+               backup_pgsql # HACK HACK HACK
                # datastore directory already exists
                # do the database upgrade
                ${PBS_EXEC}/libexec/pbs_db_utility ${UPGRADE_DB}

No guarantees. Do not use on a production system, etc.