Datastore and upgrade psql

Users/Site Administrators
1

Hi,
Why, every time I update postgres, the datastore stops working? I then have to recreate the entire cluster configuration manually in qmgr.

    Upgrade       postgresql-private-devel-13.18-1.el9.x86_64          @crb
    Upgraded      postgresql-private-devel-13.16-1.el9.x86_64          @@System
    Upgrade       postgresql-server-devel-13.18-1.el9.x86_64           @crb
    Upgraded      postgresql-server-devel-13.16-1.el9.x86_64           @@System
    Upgrade       postgresql-upgrade-devel-13.18-1.el9.x86_64          @crb
    Upgraded      postgresql-upgrade-devel-13.16-1.el9.x86_64          @@System

sudo /pbs/libexec/pbs_habitat
***
*** Backing up /pbs/pbshome/pgsql.forupgrade to /pbs/pbshome/pgsql.forupgrade.pre.
*** /pbs/pbshome/pgsql.forupgrade.pre. may need to be manually removed if you do not wish to downgrade PBS.
psql: error: FATAL:  the database system is starting up
Cannot upgrade PBS datastore version 
Failed to upgrade PBS Datastore
*** End of/pbs/libexec/pbs_habitat

$ sudo /etc/init.d/pbs start
Starting PBS
PBS comm
/pbs/sbin/pbs_comm ready (pid=60896), Proxy Name:headnode:17001, Threads:4
PBS sched
Notifying Secondary Server that we are taking over
Connecting to PBS dataservice......continuing in background.
PBS server


01/22/2025 13:50:33;0002;Server@headnode;Svr;Server@headnode;pbs_status_db exit code 1
01/22/2025 13:50:33;0002;Server@headnode;Svr;Server@headnode;Starting PBS dataservice
01/22/2025 13:50:48;0002;Server@headnode;Svr;Server@headnode;Starting PBS dataservice
01/22/2025 13:50:52;0002;Server@headnode;Svr;Server@headnode;pbs_status_db exit code 1
01/22/2025 13:50:52;0002;Server@headnode;Svr;Server@headnode;Starting PBS dataservice

pic1
pic11805×137 71.6 KB

Output from db_errfile_status_(…):

pg_ctl: no server running
postgresql log

2025-01-22 11:36:38.302 CET [350880] LOG: received fast shutdown request
2025-01-22 11:36:38.312 CET [350880] LOG: aborting any active transactions
2025-01-22 11:36:38.441 CET [350880] LOG: background worker “logical replication launcher” (PID 350897) exited with exit code 1
2025-01-22 11:36:38.444 CET [350892] LOG: shutting down
2025-01-22 11:36:38.621 CET [350880] LOG: database system is shut down
2025-01-22 11:37:12.951 CET [684012] LOG: starting PostgreSQL 13.16 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 11:37:12.952 CET [684012] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 11:37:12.952 CET [684012] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 11:37:12.966 CET [684012] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 11:37:12.977 CET [684012] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 11:37:13.079 CET [684023] LOG: database system was shut down at 2025-01-22 11:36:38 CET
2025-01-22 11:37:13.168 CET [684012] LOG: database system is ready to accept connections
2025-01-22 11:37:47.975 CET [684012] LOG: received fast shutdown request
2025-01-22 11:37:47.979 CET [684012] LOG: aborting any active transactions
2025-01-22 11:37:47.981 CET [684012] LOG: background worker “logical replication launcher” (PID 684033) exited with exit code 1
2025-01-22 11:37:47.985 CET [684127] FATAL: terminating connection due to administrator command
2025-01-22 11:37:48.017 CET [684028] LOG: shutting down
2025-01-22 11:37:48.144 CET [684012] LOG: database system is shut down
2025-01-22 11:38:00.090 CET [220401] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 11:38:00.091 CET [220401] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 11:38:00.091 CET [220401] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 11:38:00.093 CET [220401] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 11:38:00.108 CET [220401] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 11:38:00.186 CET [220412] LOG: database system was shut down at 2025-01-22 11:37:48 CET
2025-01-22 11:38:00.229 CET [220401] LOG: database system is ready to accept connections
2025-01-22 11:38:24.255 CET [220401] LOG: received smart shutdown request
2025-01-22 11:38:24.255 CET [220505] FATAL: terminating connection due to administrator command
2025-01-22 11:38:24.258 CET [220401] LOG: received SIGHUP, reloading configuration files
2025-01-22 11:38:24.276 CET [220401] LOG: background worker “logical replication launcher” (PID 220418) exited with exit code 1
2025-01-22 11:38:24.313 CET [220413] LOG: shutting down
2025-01-22 11:38:24.405 CET [220401] LOG: database system is shut down
2025-01-22 11:40:34.750 CET [685605] LOG: starting PostgreSQL 13.16 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 11:40:34.750 CET [685605] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 11:40:34.750 CET [685605] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 11:40:34.766 CET [685605] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 11:40:34.782 CET [685605] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 11:40:34.815 CET [685616] LOG: database system was shut down at 2025-01-22 11:38:24 CET
2025-01-22 11:40:34.847 CET [685605] LOG: database system is ready to accept connections
2025-01-22 12:05:25.491 CET [685605] LOG: received fast shutdown request
2025-01-22 12:05:25.493 CET [685605] LOG: aborting any active transactions
2025-01-22 12:05:25.497 CET [685605] LOG: background worker “logical replication launcher” (PID 685622) exited with exit code 1
2025-01-22 12:05:25.497 CET [685617] LOG: shutting down
2025-01-22 12:05:25.560 CET [685605] LOG: database system is shut down
2025-01-22 12:06:03.816 CET [7077] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:06:03.816 CET [7077] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 12:06:03.816 CET [7077] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 12:06:03.819 CET [7077] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 12:06:03.836 CET [7077] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 12:06:03.915 CET [7088] LOG: database system was shut down at 2025-01-22 12:05:25 CET
2025-01-22 12:06:03.978 CET [7077] LOG: database system is ready to accept connections
2025-01-22 12:26:49.815 CET [7077] LOG: received fast shutdown request
2025-01-22 12:26:49.818 CET [7077] LOG: aborting any active transactions
2025-01-22 12:26:49.820 CET [7077] LOG: background worker “logical replication launcher” (PID 7094) exited with exit code 1
2025-01-22 12:26:49.821 CET [7089] LOG: shutting down
2025-01-22 12:26:49.877 CET [7077] LOG: database system is shut down
2025-01-22 12:29:13.555 CET [7823] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:29:13.555 CET [7823] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:29:13.555 CET [7823] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:29:13.555 CET [7823] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:29:13.555 CET [7823] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:29:13.556 CET [7823] WARNING: could not create listen socket for “"
2025-01-22 12:29:13.562 CET [7823] FATAL: could not create any TCP/IP sockets
2025-01-22 12:29:13.621 CET [7823] LOG: database system is shut down
2025-01-22 12:29:33.941 CET [8020] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:29:33.941 CET [8020] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:29:33.941 CET [8020] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:29:33.941 CET [8020] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:29:33.941 CET [8020] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:29:33.941 CET [8020] WARNING: could not create listen socket for "”
2025-01-22 12:29:33.941 CET [8020] FATAL: could not create any TCP/IP sockets
2025-01-22 12:29:33.946 CET [8020] LOG: database system is shut down
2025-01-22 12:29:47.375 CET [8217] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:29:47.376 CET [8217] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:29:47.376 CET [8217] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:29:47.376 CET [8217] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:29:47.376 CET [8217] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:29:47.376 CET [8217] WARNING: could not create listen socket for “"
2025-01-22 12:29:47.376 CET [8217] FATAL: could not create any TCP/IP sockets
2025-01-22 12:29:47.379 CET [8217] LOG: database system is shut down
2025-01-22 12:30:02.817 CET [8531] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:30:02.817 CET [8531] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:30:02.817 CET [8531] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:30:02.817 CET [8531] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:30:02.817 CET [8531] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:30:02.817 CET [8531] WARNING: could not create listen socket for "”
2025-01-22 12:30:02.817 CET [8531] FATAL: could not create any TCP/IP sockets
2025-01-22 12:30:02.820 CET [8531] LOG: database system is shut down
2025-01-22 12:30:13.012 CET [10651] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:30:13.012 CET [10651] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 12:30:13.012 CET [10651] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 12:30:13.015 CET [10651] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 12:30:13.030 CET [10651] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 12:30:13.059 CET [10663] LOG: database system was shut down at 2025-01-22 12:26:49 CET
2025-01-22 12:30:13.083 CET [10651] LOG: database system is ready to accept connections
2025-01-22 12:30:37.799 CET [8981] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:30:37.799 CET [8981] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:30:37.799 CET [8981] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:30:37.799 CET [8981] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:30:37.799 CET [8981] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:30:37.799 CET [8981] WARNING: could not create listen socket for “"
2025-01-22 12:30:37.799 CET [8981] FATAL: could not create any TCP/IP sockets
2025-01-22 12:30:37.803 CET [8981] LOG: database system is shut down
2025-01-22 12:30:52.221 CET [9200] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:30:52.221 CET [9200] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:30:52.221 CET [9200] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:30:52.221 CET [9200] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:30:52.221 CET [9200] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:30:52.221 CET [9200] WARNING: could not create listen socket for "”
2025-01-22 12:30:52.221 CET [9200] FATAL: could not create any TCP/IP sockets
2025-01-22 12:30:52.224 CET [9200] LOG: database system is shut down
2025-01-22 12:31:05.648 CET [9474] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:31:05.649 CET [9474] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:31:05.649 CET [9474] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:31:05.649 CET [9474] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:31:05.649 CET [9474] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:31:05.649 CET [9474] WARNING: could not create listen socket for “"
2025-01-22 12:31:05.649 CET [9474] FATAL: could not create any TCP/IP sockets
2025-01-22 12:31:05.652 CET [9474] LOG: database system is shut down
2025-01-22 12:31:13.177 CET [10651] LOG: could not open file “postmaster.pid”: No such file or directory
2025-01-22 12:31:13.177 CET [10651] LOG: performing immediate shutdown because data directory lock file is invalid
2025-01-22 12:31:13.177 CET [10651] LOG: received immediate shutdown request
2025-01-22 12:31:13.177 CET [10651] LOG: could not open file “postmaster.pid”: No such file or directory
2025-01-22 12:31:13.177 CET [10667] WARNING: terminating connection because of crash of another server process
2025-01-22 12:31:13.177 CET [10667] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2025-01-22 12:31:13.177 CET [10667] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2025-01-22 12:31:13.177 CET [10843] WARNING: terminating connection because of crash of another server process
2025-01-22 12:31:13.177 CET [10843] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2025-01-22 12:31:13.177 CET [10843] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2025-01-22 12:31:13.177 CET [10992] WARNING: terminating connection because of crash of another server process
2025-01-22 12:31:13.177 CET [10992] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2025-01-22 12:31:13.177 CET [10992] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2025-01-22 12:31:13.217 CET [10651] LOG: database system is shut down
2025-01-22 12:31:21.096 CET [9695] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:31:21.096 CET [9695] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:31:21.096 CET [9695] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:31:21.097 CET [9695] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:31:21.097 CET [9695] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:31:21.097 CET [9695] WARNING: could not create listen socket for "”
2025-01-22 12:31:21.097 CET [9695] FATAL: could not create any TCP/IP sockets
2025-01-22 12:31:21.101 CET [9695] LOG: database system is shut down
2025-01-22 12:31:34.518 CET [9915] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:31:34.518 CET [9915] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:31:34.518 CET [9915] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:31:34.518 CET [9915] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:31:34.518 CET [9915] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:31:34.518 CET [9915] WARNING: could not create listen socket for “"
2025-01-22 12:31:34.518 CET [9915] FATAL: could not create any TCP/IP sockets
2025-01-22 12:31:34.523 CET [9915] LOG: database system is shut down
2025-01-22 12:31:51.959 CET [10140] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:31:51.960 CET [10140] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:31:51.960 CET [10140] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:31:51.960 CET [10140] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:31:51.960 CET [10140] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:31:51.960 CET [10140] WARNING: could not create listen socket for "”
2025-01-22 12:31:51.960 CET [10140] FATAL: could not create any TCP/IP sockets
2025-01-22 12:31:51.963 CET [10140] LOG: database system is shut down
2025-01-22 12:32:05.392 CET [10356] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:32:05.392 CET [10356] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:32:05.392 CET [10356] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:32:05.392 CET [10356] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:32:05.392 CET [10356] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:32:05.392 CET [10356] WARNING: could not create listen socket for “"
2025-01-22 12:32:05.392 CET [10356] FATAL: could not create any TCP/IP sockets
2025-01-22 12:32:05.395 CET [10356] LOG: database system is shut down
2025-01-22 12:32:23.829 CET [10600] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 12:32:23.829 CET [10600] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 12:32:23.829 CET [10600] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:32:23.829 CET [10600] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 12:32:23.829 CET [10600] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 12:32:23.829 CET [10600] WARNING: could not create listen socket for "”
2025-01-22 12:32:23.829 CET [10600] FATAL: could not create any TCP/IP sockets
(…)
2025-01-22 13:19:17.893 CET [48146] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 13:19:17.893 CET [48146] LOG: could not bind IPv4 address “0.0.0.0”: Address already in use
2025-01-22 13:19:17.893 CET [48146] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 13:19:17.893 CET [48146] LOG: could not bind IPv6 address “::”: Address already in use
2025-01-22 13:19:17.893 CET [48146] HINT: Is another postmaster already running on port 15007? If not, wait a few seconds and retry.
2025-01-22 13:19:17.893 CET [48146] WARNING: could not create listen socket for “*”
2025-01-22 13:19:17.893 CET [48146] FATAL: could not create any TCP/IP sockets
2025-01-22 13:19:17.897 CET [48146] LOG: database system is shut down
2025-01-22 13:20:15.814 CET [49163] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 13:20:15.814 CET [49163] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 13:20:15.814 CET [49163] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 13:20:15.816 CET [49163] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 13:20:15.831 CET [49163] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 13:20:15.949 CET [49174] LOG: database system was interrupted; last known up at 2025-01-22 12:32:10 CET
2025-01-22 13:20:18.908 CET [49174] LOG: invalid primary checkpoint record
2025-01-22 13:20:18.908 CET [49174] PANIC: could not locate a valid checkpoint record
2025-01-22 13:20:19.003 CET [49163] LOG: startup process (PID 49174) was terminated by signal 6: Aborted
2025-01-22 13:20:19.003 CET [49163] LOG: aborting startup due to startup process failure
2025-01-22 13:20:19.006 CET [49163] LOG: database system is shut down
2025-01-22 13:20:19.007 CET [49286] FATAL: terminating connection due to unexpected postmaster exit
2025-01-22 13:20:21.303 CET [49394] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 13:20:21.304 CET [49394] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 13:20:21.304 CET [49394] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 13:20:21.306 CET [49394] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 13:20:21.342 CET [49394] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 13:20:21.356 CET [49405] LOG: database system was interrupted; last known up at 2025-01-22 12:32:10 CET
2025-01-22 13:20:24.063 CET [49405] LOG: invalid primary checkpoint record
2025-01-22 13:20:24.063 CET [49405] PANIC: could not locate a valid checkpoint record
2025-01-22 13:20:24.140 CET [49394] LOG: startup process (PID 49405) was terminated by signal 6: Aborted
2025-01-22 13:20:24.140 CET [49394] LOG: aborting startup due to startup process failure
2025-01-22 13:20:24.143 CET [49394] LOG: database system is shut down
2025-01-22 13:20:26.720 CET [49621] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 13:20:26.720 CET [49621] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 13:20:26.720 CET [49621] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 13:20:26.722 CET [49621] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 13:20:26.737 CET [49621] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 13:20:26.752 CET [49632] LOG: database system was interrupted; last known up at 2025-01-22 12:32:10 CET
2025-01-22 13:20:29.437 CET [49632] LOG: invalid primary checkpoint record
2025-01-22 13:20:29.437 CET [49632] PANIC: could not locate a valid checkpoint record
2025-01-22 13:20:29.521 CET [49621] LOG: startup process (PID 49632) was terminated by signal 6: Aborted
2025-01-22 13:20:29.521 CET [49621] LOG: aborting startup due to startup process failure
2025-01-22 13:20:29.525 CET [49621] LOG: database system is shut down
2025-01-22 13:20:34.134 CET [49858] LOG: starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 13:20:34.134 CET [49858] LOG: listening on IPv4 address “0.0.0.0”, port 15007
2025-01-22 13:20:34.134 CET [49858] LOG: listening on IPv6 address “::”, port 15007
2025-01-22 13:20:34.137 CET [49858] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.15007”
2025-01-22 13:20:34.154 CET [49858] LOG: listening on Unix socket “/tmp/.s.PGSQL.15007”
2025-01-22 13:20:34.167 CET [49869] LOG: database system was interrupted; last known up at 2025-01-22 12:32:10 CET
2025-01-22 13:20:36.857 CET [49869] LOG: invalid primary checkpoint record
2025-01-22 13:20:36.857 CET [49869] PANIC: could not locate a valid checkpoint record
(…)

Any advice?

2

After I ran the command:

pg_resetwal -f datastore

PANIC error no longer occurs:

2025-01-22 15:06:20.893 CET [110463] LOG:  starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 15:06:20.893 CET [110463] LOG:  listening on IPv4 address "0.0.0.0", port 15007
2025-01-22 15:06:20.893 CET [110463] LOG:  listening on IPv6 address "::", port 15007
2025-01-22 15:06:20.896 CET [110463] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.15007"
2025-01-22 15:06:20.912 CET [110463] LOG:  listening on Unix socket "/tmp/.s.PGSQL.15007"
2025-01-22 15:06:20.926 CET [110474] LOG:  database system was shut down at 2025-01-22 15:05:47 CET
2025-01-22 15:06:21.003 CET [110463] LOG:  database system is ready to accept connections
2025-01-22 15:06:28.404 CET [110463] LOG:  received fast shutdown request
2025-01-22 15:06:28.407 CET [110463] LOG:  aborting any active transactions
2025-01-22 15:06:28.410 CET [110463] LOG:  background worker "logical replication launcher" (PID 110480) exited with exit code 1
2025-01-22 15:06:28.410 CET [110475] LOG:  shutting down
2025-01-22 15:06:28.476 CET [110463] LOG:  database system is shut down
2025-01-22 15:07:48.332 CET [111854] LOG:  starting PostgreSQL 13.18 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2025-01-22 15:07:48.332 CET [111854] LOG:  listening on IPv4 address "0.0.0.0", port 15007
2025-01-22 15:07:48.332 CET [111854] LOG:  listening on IPv6 address "::", port 15007
2025-01-22 15:07:48.335 CET [111854] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.15007"
2025-01-22 15:07:48.352 CET [111854] LOG:  listening on Unix socket "/tmp/.s.PGSQL.15007"
2025-01-22 15:07:48.380 CET [111865] LOG:  database system was shut down at 2025-01-22 15:06:28 CET
2025-01-22 15:07:48.400 CET [111854] LOG:  database system is ready to accept connections
2025-01-22 15:07:54.826 CET [111854] LOG:  received fast shutdown request
2025-01-22 15:07:54.829 CET [111854] LOG:  aborting any active transactions
2025-01-22 15:07:54.832 CET [111854] LOG:  background worker "logical replication launcher" (PID 111871) exited with exit code 1
2025-01-22 15:07:54.832 CET [111866] LOG:  shutting down
2025-01-22 15:07:54.906 CET [111854] LOG:  database system is shut down

However, an error appears:

background worker "logical replication launcher" (PID 111871) exited with exit code 1

Where is the log associated with “logical replication launcher” ?

3

I ran pbs_probe and see that pgsql is missing:

# PBS_EXEC/sbin/pbs_probe


====== System Information =======


sysname=Linux
nodename=headnode
release=5.14.0-547.el9.x86_64
version=#1 SMP PREEMPT_DYNAMIC Mon Dec 30 20:10:38 UTC 2024
machine=x86_64


====== Problems in PBS EXEC Hierarchy =======

Real Path Problems:
PBS_EXEC/bin/pbs_ds_password, No such file or directory

PBS_EXEC/bin/pbs_dataservice, No such file or directory

PBS_EXEC/sbin/pbs-report, No such file or directory

PBS_EXEC/etc/pbs_habitat, No such file or directory

PBS_EXEC/etc/pbs_init.d, No such file or directory

PBS_EXEC/etc/pbs_postinstall, No such file or directory

PBS_EXEC/etc/pbs_db_utility, No such file or directory

PBS_EXEC/etc/pbs_topologyinfo, No such file or directory

PBS_EXEC/lib/pm, No such file or directory

PBS_EXEC/man, No such file or directory

PBS_EXEC/tcltk, No such file or directory

PBS_EXEC/python, No such file or directory

PBS_EXEC/pgsql, No such file or directory

Maybe that’s the root of the problem?