Adarsh,
Thank you very much for the further inputs.
The preempt message is a little bit mis-leading. I belive it’'s due to the qrun.
I did su to root and force the job to run with qrun and tried to see if the job could be dispatched to the ‘free’ nodes
Managed to get the reason, mom nodes were in an ‘idle’ status and didn’t work as expected, the mom log was zero, though systemctl status showed it’s active and running, pbsnodes also showed everything is OK.
Fix should be very simple, just reboot the node or restart the mom service.
Wondering how to dig out the root cause or what’re possible reasons. I didn’t get any error message or warning message for those ‘idling nodes’ from server log and scheduler log. Looked like server still thought all nodes are ‘good’.