Trying to run OpenPBS on openvpn

BrianLYS · October 15, 2024, 3:24pm

Hi all I’m trying to run one head node and one compute node with openvpn on ubuntu 22.04.

It works normally if I use the private ip address of 172.168… that AWS assigns to my instances but if I use 10.8.0.150 which is my openvpn static ip address then I get this error.

qmgr obj=compute-node-ip svr=default: Unauthorized Request 
qmgr: Error (15007) returned from server

It seems that OpenPBS does not like the static ip address assigned to it by openvpn ? Not sure how to get pass this error.

Head Installation script

##########################################################
## PBS Head Node Installation Script for Ubuntu Systems ##
##########################################################

# 0. Disable SELinux (PBS does not support SELinux, not applicable to Ubuntu)
log "SELinux is not applicable to Ubuntu."

# etc/hosts
log "Updating /etc/hosts..."
echo "10.8.0.151 compute-node-ip" | tee -a /etc/hosts
echo "10.8.0.150 head-node-ip" | tee -a /etc/hosts

# 1. Define the installation and build directories
INSTALL_DIR="/opt/pbs"
BUILD_DIR="/opt/pbs_build"
TARBALL="openpbs-23.06.06.tar.gz"
TARBALL_URL="https://github.com/openpbs/openpbs/archive/refs/tags/v23.06.06.tar.gz"

# 1.1 Create the build directory
log "Creating build directory at $BUILD_DIR..."
mkdir -p $BUILD_DIR
cd $BUILD_DIR || { log "Failed to change directory to $BUILD_DIR"; exit 1; }

# 2. Install the prerequisite packages for building PBS
log "Installing prerequisite packages..."
apt-get install -y gcc make libtool libhwloc-dev libx11-dev \
    libxt-dev libedit-dev libical-dev ncurses-dev perl \
    postgresql-server-dev-all postgresql-contrib python3-dev tcl-dev tk-dev swig \
    libexpat-dev libssl-dev libxext-dev libxft-dev autoconf \
    automake g++ libcjson-dev

# 3. Install the prerequisite packages for running PBS
log "Installing runtime packages..."
apt-get install -y expat libedit2 postgresql python3 postgresql-contrib sendmail-bin \
    sudo tcl tk libical3

# Optionally ensure PostgreSQL is running
log "Starting PostgreSQL service..."
sudo systemctl start postgresql

# 4. Download the PBS tarball to the build directory if not already present
if [ ! -f "$BUILD_DIR/$TARBALL" ]; then
    log "Downloading PBS tarball..."
    wget -O $TARBALL $TARBALL_URL
fi

# 5. Unpack the PBS tarball and navigate to the source directory
log "Extracting PBS tarball..."
tar -xpvf $TARBALL
cd openpbs-23.06.06 || { log "Failed to change directory to openpbs-23.06.06"; exit 1; }

# 6. Generate the configure script and Makefiles
log "Running autogen.sh to generate configure script..."
./autogen.sh

# 7. (Optional) Display available build parameters
log "Displaying available build parameters..."
./configure --help

# 8. Configure the build for your environment with a custom install directory
log "Configuring PBS build with prefix $INSTALL_DIR..."
./configure --prefix=$INSTALL_DIR

# 9. Build PBS
log "Building PBS..."
make

# 10. Install PBS
log "Installing PBS into $INSTALL_DIR..."
make install

# 11. Configure PBS by executing the post-install script
log "Running post-install script..."
$INSTALL_DIR/libexec/pbs_postinstall

# 12. Edit /etc/pbs.conf to configure the PBS services
log "Configuring /etc/pbs.conf for head node..."
sed -i 's/PBS_SERVER=.*$/PBS_SERVER=head-node-ip/g' /etc/pbs.conf
sed -i 's/PBS_START_SERVER=0/PBS_START_SERVER=1/g' /etc/pbs.conf
sed -i 's/PBS_START_SCHED=0/PBS_START_SCHED=1/g' /etc/pbs.conf
sed -i 's/PBS_START_COMM=0/PBS_START_COMM=1/g' /etc/pbs.conf
sed -i 's/PBS_START_MOM=0/PBS_START_MOM=0/g' /etc/pbs.conf

# 13. Modify file permissions to add SUID privilege
log "Modifying file permissions for SUID privilege..."
chmod 4755 $INSTALL_DIR/sbin/pbs_iff $INSTALL_DIR/sbin/pbs_rcp

# 14. Start PBS services
log "Starting PBS services..."
/etc/init.d/pbs start

# 15. Update PATH and MANPATH variables
log "Updating PATH and MANPATH..."
source /etc/profile.d/pbs.sh

# 16. Add the compute node to the PBS configuration
log "Adding compute node to PBS configuration..."
sudo /opt/pbs/bin/qmgr -c "create node compute-node-ip"

# Optionally, make the PATH update persistent by adding to bashrc
log "Making the PATH update persistent by adding to bashrc..."
echo "echo 'source /etc/profile.d/pbs.sh' >> ~/.bashrc"

# 17. PBS head node installation and configuration is complete
log "PBS head node installation complete! You can now configure your compute nodes."

Compute Installation Script

#############################################################
## PBS Compute Node Installation Script for Ubuntu Systems ##
#############################################################

# 0. Disable SELinux (PBS does not support SELinux, not applicable to Ubuntu)
log "SELinux is not applicable to Ubuntu."

# etc/hosts
log "Updating /etc/hosts..."
echo "10.8.0.151 compute-node-ip" | tee -a /etc/hosts
echo "10.8.0.150 head-node-ip" | tee -a /etc/hosts

# 1. Define the installation and build directories
INSTALL_DIR="/opt/pbs"
BUILD_DIR="/opt/pbs_build"
TARBALL="openpbs-23.06.06.tar.gz"
TARBALL_URL="https://github.com/openpbs/openpbs/archive/refs/tags/v23.06.06.tar.gz"

# 1.1 Create the build directory
log "Creating build directory at $BUILD_DIR..."
mkdir -p $BUILD_DIR
cd $BUILD_DIR || { log "Failed to change directory to $BUILD_DIR"; exit 1; }

# 2. Install the prerequisite packages for building PBS
log "Installing prerequisite packages..."
apt-get install -y gcc make libtool libhwloc-dev libx11-dev \
    libxt-dev libedit-dev libical-dev ncurses-dev perl \
    postgresql-server-dev-all postgresql-contrib python3-dev tcl-dev tk-dev swig \
    libexpat-dev libssl-dev libxext-dev libxft-dev autoconf \
    automake g++ libcjson-dev

# 3. Install the prerequisite packages for running PBS
log "Installing runtime packages..."
apt-get install -y expat libedit2 postgresql python3 postgresql-contrib sendmail-bin \
    sudo tcl tk libical3

# Optionally ensure PostgreSQL is running
log "Starting PostgreSQL service..."
sudo systemctl start postgresql

# 4. Download the PBS tarball to the build directory if not already present
if [ ! -f "$BUILD_DIR/$TARBALL" ]; then
    log "Downloading PBS tarball..."
    wget -O $TARBALL $TARBALL_URL
fi

# 5. Unpack the PBS tarball and navigate to the source directory
log "Extracting PBS tarball..."
tar -xpvf $TARBALL
cd openpbs-23.06.06 || { log "Failed to change directory to openpbs-23.06.06"; exit 1; }

# 6. Generate the configure script and Makefiles
log "Running autogen.sh to generate configure script..."
./autogen.sh

# 7. (Optional) Display available build parameters
log "Displaying available build parameters..."
./configure --help

# 8. Configure the build for your environment with a custom install directory
log "Configuring PBS build with prefix $INSTALL_DIR..."
./configure --prefix=$INSTALL_DIR

# 9. Build PBS
log "Building PBS..."
make

# 10. Install PBS
log "Installing PBS into $INSTALL_DIR..."
make install

# 11. Configure PBS by executing the post-install script
log "Running post-install script..."
$INSTALL_DIR/libexec/pbs_postinstall

# 12. Edit /etc/pbs.conf to configure the PBS services for compute node
log "Configuring /etc/pbs.conf for compute node..."
sed -i "s/PBS_SERVER=.*$/PBS_SERVER=head-node-ip/g" /etc/pbs.conf
sed -i 's/PBS_START_MOM=0/PBS_START_MOM=1/g' /etc/pbs.conf
sed -i 's/PBS_START_SERVER=1/PBS_START_SERVER=0/g' /etc/pbs.conf
sed -i 's/PBS_START_SCHED=1/PBS_START_SCHED=0/g' /etc/pbs.conf
sed -i 's/PBS_START_COMM=1/PBS_START_COMM=0/g' /etc/pbs.conf

# 13. Modify file permissions to add SUID privilege
log "Modifying file permissions for SUID privilege..."
chmod 4755 $INSTALL_DIR/sbin/pbs_iff $INSTALL_DIR/sbin/pbs_rcp

# 14. Start the PBS MOM service (compute node)
log "Starting PBS MOM service..."
/etc/init.d/pbs start

# 15. Update PATH and MANPATH variables
log "Updating PATH and MANPATH..."
source /etc/profile.d/pbs.sh

# 16. Start the PBS service
# log "Starting PBS service..."
sudo systemctl start pbs

# Optionally, make the PATH update persistent by adding to bashrc
log "Making the PATH update persistent by adding to bashrc..."
echo "echo 'source /etc/profile.d/pbs.sh' >> ~/.bashrc"

# 17. PBS compute node installation and configuration is complete
log "PBS compute node installation complete! This node is ready to accept jobs from the head node."

speleolinux · October 15, 2024, 10:14pm

Hi
I once got the error message “Error (15007) returned from server” when I did not have permission to update or change PBS settings via qmgr. I needed to add myself to the list of users that can be a server manager. If brian is your login name then you would:
Qmgr: set server managers += brian@*

Check what you have currently with: qmgr -c “p server” | grep managers

Mike

adarsh · October 15, 2024, 10:57pm

Please check this

BrianLYS · October 16, 2024, 2:46am

It’s not a permission error, the thing is if I don’t use the openvpn static ip and I use the private ip address of my instance then the command runs.

Example:
If I use 172.168.8.0 instead of 10.8.0.150 . Then the command works otherwise it fails

dtalcott · October 17, 2024, 8:44pm

Have you looked in the server logs for clues, /var/spool/pbs/server_logs/ ?

What is your route table?

ip route show

How do packets flow from server to node? On the server, run

 traceroute 10.8.0.151
 traceroute compute-node-ip
 traceroute head-node-ip

Similarly, on the compute node run

 traceroute 10.8.0.150
 traceroute head-node-ip
 traceroute compute-node-ip

vchlum · October 31, 2024, 8:38pm

I have no experience with PBS on VPN but I suppose you could route the PBS traffic via the tun0 interface e.g. like this:

On the node, it could look like this:

mark the traffic to pbs server

iptables -t mangle -A OUTPUT -d <pbs-server-IP> -j MARK --set-mark 0x20

route the marked traffic via tun0

echo "123 PBS" >> /etc/iproute2/rt_tables
ip rule add from all fwmark 0x20 lookup PBS
ip route add default via <vpn_gateway> dev tun0 table PBS

…and maybe set the destination of the traffic:

iptables -t nat -A OUTPUT -d <pbs-server-IP> -j DNAT --to <VPN-pbs-server-IP>

…and maybe set something similar on the PBS server to route the traffic from the server back to the nodes via tun interface…

Topic		Replies	Views
How can I add compute node (ubuntu) on openpbs server Users/Site Administrators	1	546	March 26, 2023
Problems with installing OpenPBS on only one machine Users/Site Administrators	1	441	October 26, 2020
PBS daemons are running but connection refused Users/Site Administrators	3	1910	April 10, 2021
Pbsnodes: Server has no node list Users/Site Administrators	1	2582	February 28, 2021
Setting up pbspro on one computer Users/Site Administrators	4	1321	June 12, 2019

Trying to run OpenPBS on openvpn

Head Installation script

Compute Installation Script

Related topics