pbs mom is randomly crashing on compute nodes. Below is mom_logs
05/04/2021 21:22:18;0080;pbs_mom;Job;4372.master1;copy file request received
05/04/2021 21:22:18;0100;pbs_mom;Job;4372.master1;staged 1 items out over 0:00:00
05/04/2021 21:22:18;0008;pbs_mom;Job;4372.master1;no active tasks
05/04/2021 21:22:18;0100;pbs_mom;Req;;Type 6 request received from root@192.168.1.254:15001, sock=1
05/04/2021 21:22:18;0080;pbs_mom;Job;4372.master1;delete job request received
05/04/2021 21:22:18;0008;pbs_mom;Job;4372.master1;kill_job
05/04/2021 21:22:18;0100;pbs_mom;Req;;Type 1 request received from root@192.168.1.254:15001, sock=1
05/04/2021 21:22:18;0100;pbs_mom;Req;;Type 3 request received from root@192.168.1.254:15001, sock=1
05/04/2021 21:22:18;0100;pbs_mom;Req;;Type 5 request received from root@192.168.1.254:15001, sock=1
05/04/2021 21:22:18;0008;pbs_mom;Job;4368.master1;Started, pid = 112521
05/04/2021 21:35:10;0080;pbs_mom;Job;4371.master1;task 00000001 terminated
05/04/2021 21:35:10;0008;pbs_mom;Job;4371.master1;Terminated
05/04/2021 21:35:10;0100;pbs_mom;Job;4371.master1;task 00000001 cput=1272:18:12
05/04/2021 21:35:10;0008;pbs_mom;Job;4371.master1;kill_job
05/04/2021 21:35:10;0100;pbs_mom;Job;4371.master1;n001 cput=1272:18:12 mem=1478928kb
05/04/2021 21:35:10;0008;pbs_mom;Job;4371.master1;no active tasks
05/04/2021 21:35:10;0100;pbs_mom;Job;4371.master1;Obit sent
05/04/2021 21:35:10;0100;pbs_mom;Req;;Type 54 request received from root@192.168.1.254:15001, sock=1
05/04/2021 21:35:10;0080;pbs_mom;Job;4371.master1;copy file request received
05/04/2021 21:35:11;0100;pbs_mom;Job;4371.master1;staged 1 items out over 0:00:01
05/04/2021 21:35:11;0008;pbs_mom;Job;4371.master1;no active tasks
05/04/2021 21:35:11;0100;pbs_mom;Req;;Type 6 request received from root@192.168.1.254:15001, sock=1
05/04/2021 21:35:11;0080;pbs_mom;Job;4371.master1;delete job request received
05/04/2021 21:35:11;0008;pbs_mom;Job;4371.master1;kill_job
05/04/2021 21:35:11;0100;pbs_mom;Req;;Type 1 request received from root@192.168.1.254:15001, sock=1
05/04/2021 21:35:11;0100;pbs_mom;Req;;Type 3 request received from root@192.168.1.254:15001, sock=1
05/04/2021 21:35:11;0100;pbs_mom;Req;;Type 57 request received from root@192.168.1.254:15001, sock=1
snaps of /var/log/messages
May 4 21:35:11 n001 kernel: pbs_mom[2271]: segfault at 8 ip 000000000045656a sp 00007fff419670b0 error 4 in pbs_mom[400000+c6000]