# Revalidating nodes **URL:** \ **Category:** Users/Site Administrators\ **Created:** [October 9, 2023, 4:35pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682 "2023-10-09T16:35:41Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [October 9, 2023, 4:35pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/1 "2023-10-09T16:35:41Z")
we have a complex that creates/uses/destroys nodes very frequently. this runs smoothly aside from occasionally jobs wont start on the nodes issue. pbsnodes shows the node as up and free, the resources associated with the node(s) nicely match the resource requested in the job but its always “job cannot run”. restarting the PBS server process gets everything back in sync and the jobs will start running again. The thought occurred to me that maybe the server is getting its in memory view of what the nodes look like messed up and it takes a restart to re-validate everything again. This then got me wondering if there was a easier way to get the server to re-validate besides a server restart. as that takes a long(ish) time and qstat/qsub/etc commands fail while its being restarted. thanks s ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [October 12, 2023, 8:23am UTC](https://community.openpbs.org/t/revalidating-nodes/3682/2 "2023-10-12T08:23:02Z")
Please try to initiate a scheduling cycle manually after creating node, to see it helps to avoid service restart. `qmgr -c "set server scheduling=t"` ---
**Author:** ![alexis.cousein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/alexis.cousein/32/265_2.png) [@alexis.cousein](https://community.openpbs.org/u/alexis.cousein)\ **Post date:** [October 15, 2023, 11:27pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/3 "2023-10-15T23:27:29Z")
Same here: it’s not _the server_ that is not in sync (by definition pbsnodes reflects the server view of the world) but the scheduler; if your scheduling cycles are long the scheduler will try to place jobs using the now stale state of the world that it fetched at the start of the cycle. If you create nodes and want jobs to use them, set server scheduling to false using qmgr, wait for the scheduler to end its cycle (or kill and restart _it_ i.e. pbs\_sched, and not pbs\_server; it’s stateless so it does no harm) and then set scheduling to true again. ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [October 16, 2023, 4:02am UTC](https://community.openpbs.org/t/revalidating-nodes/3682/4 "2023-10-16T04:02:52Z")
this makes a lot of sense yes. I will try these solutions and see what happens the next time it randomly gets confused. thanks ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [November 11, 2023, 10:23pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/5 "2023-11-11T22:23:15Z")
finally got around to being able to test this. (blame the lack of a real govt budget) for whatever reason restarting the scheduler does not unwedge things. it takes a restart of the server. I had hopes for the restarting schedule thing. it made sense. could it be mom related? a server restart when it re-connects to the moms on the nodes might be re-validating something that finally allows life to go on. thanks s ---
**Author:** ![HeyHey](https://avatars.discourse-cdn.com/v4/letter/h/cdc98d/32.png) [@HeyHey](https://community.openpbs.org/u/HeyHey)\ **Post date:** [November 15, 2023, 2:54pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/6 "2023-11-15T14:54:29Z")
You said create and destroy the nodes frequently? Could something on that side be off (bad IP, comm, mis-started service), something that didn’t quite tick off right until the server was rebooted? ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [March 16, 2025, 9:50pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/7 "2025-03-16T21:50:45Z")
Im thinking the same solution from > [@Some jobs stay queued for extended periods of time despite availability of hosts](https://community.openpbs.org/t/some-jobs-stay-queued-for-extended-periods-of-time-despite-availability-of-hosts/4095): > > Hi, We’re using the 23.06 release of OpenPBS and facing an issue where most jobs run on-time successfully but a small number of jobs stay queued for extended periods of time despite nodes being available. Investigation: Lets take PBS Job ID 1094 as an example. Here, the job gets submitted at 02/19/2025 21:00:08 and doesn’t actually run until 02/19/2025 21:53:24 despite nodes being available. This job was submitted to the custom “medium” queue. The logs for this look like: Line 3: Line … might be the issue Im seeing from this. testing to know for sure. thanks s ---
**Author:** ![osk](https://avatars.discourse-cdn.com/v4/letter/o/ed8c4c/32.png) [@osk](https://community.openpbs.org/u/osk)\ **Post date:** [May 19, 2025, 12:04pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/8 "2025-05-19T12:04:51Z")
Just following up to ask if you ever found a root cause for this issue, seems we’re both running into the same or a similar issue. ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [May 19, 2025, 2:12pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/9 "2025-05-19T14:12:33Z")
still having the issue. but in re-reading the topic just now I had not understood adarsh’s/alexis’ response fully and probably did the wrong thing to fix.or at least the proper sequence of things. will try again ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [May 28, 2025, 5:20pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/10 "2025-05-28T17:20:24Z")
still needing to restart the server. basic process in case something leaps out at someone: loop forever: - new job shows up: - launch new machines in the cloud to match the resources selected in job - create vnodes for new machines - if jobs exists: - look for any that are still queued: - is the ‘state’ for the nodes matched up to this job marked as ‘free’: - has the job been waiting to start long (\> 2minutes / 6 scheduler cycles): - option 1: restart server process - option 2: - set scheduling to False in qmgr - stop scheduler (ptl\_testlib.Scheduler().stop()) - wait a couple seconds - initialize scheduler (ptl\_testlib.Scheduler().initialise()) - start scheduler (ptl\_testlib.Scheduler().start()) - set scheduling to True in qmgr option 1 always gets the scheduler to run the job on the nodes at the start of the next scheduling cycle. option 2 doesnt even after many scheduling cycles ---
**Author:** ![osk](https://avatars.discourse-cdn.com/v4/letter/o/ed8c4c/32.png) [@osk](https://community.openpbs.org/u/osk)\ **Post date:** [June 3, 2025, 8:55am UTC](https://community.openpbs.org/t/revalidating-nodes/3682/11 "2025-06-03T08:55:37Z")
Thanks for the detailed workaround ! We went another way since our workload cannot accept a scheduler restart. One of our findings was that this issue is triggered when a cloud machine leaves the pool of compute nodes, the remaining ones get stuck for 33 minutes. But if you add a new machine to the pool, the remaining machines become unstuck. So our workaround has been to monitor the state of queued jobs and if there are any that are queued for more than 10 minutes we spin up a new machine, which makes the remaining nodes available again. While this has minor cost implications, it’s better than jobs not getting run on time. We still don’t have a root cause identified for the issue, but our evaluation of PBS Pro showed that this issue is not reproducible in the latest release, so it definitely is a bug in OpenPBS, we just don’t know which bug it is. ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [June 3, 2025, 1:48pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/12 "2025-06-03T13:48:37Z")
interesting solution, I think I will steal that idea. Our users complain when qstat fails and my solution has it not operating for a minute. But Im wondering if it has to be a whole new new machine or if an existing machine could have a new vnode added to it. (Our PBS server has a few vnodes assigned to it to do certain daily internal tasks that are done via batch jobs outside user control, maybe adding a useless vnode and removing once jobs start flowing again also might work. No cost implications.) thanks for the new solutions. s ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [June 3, 2025, 2:17pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/13 "2025-06-03T14:17:51Z")
update on the useless vnode concept: in my case its not going to be any better. The vnodes on our PBS server are set up via a config file which looks to require a PBS service restart during the update process. If our vnodes were set up dynamically on the PBS server it has a chance of working. I have to look into going that route. Or maybe just using your original idea of a new small/cheap instance starting up for a few minutes to kick scheduling is ok. thanks s (but maybe the docs are a little sloppy and when they say to restart PBS it really only means restarting the mom on the node, as having vnodes on the PBS server really isnt normal…) ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [June 3, 2025, 5:25pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/14 "2025-06-03T17:25:13Z")
update: for me starting up a new job that launched a new node for a short job let the “hung” jobs start. But then the next job also hung which required a short job to kick things off and so on. A new vnode on an existing node didnt help. So in the end I will be stuck with restarting the PBS server process occasionally. and hope that the next version of openPBS happens to fix the issue on purpose or accidentally. thanks s ---
**Author:** ![osk](https://avatars.discourse-cdn.com/v4/letter/o/ed8c4c/32.png) [@osk](https://community.openpbs.org/u/osk)\ **Post date:** [June 13, 2025, 9:45am UTC](https://community.openpbs.org/t/revalidating-nodes/3682/15 "2025-06-13T09:45:10Z")
An update on our own testing of the ‘add/remove nodes to get unstuck’ strategy shows that this also fails to work sometimes. So it’s not a foolproof solution. Looks like we’ll have to go with a restart based solution like you. Through my parsing of the docs it seems like using **qterm -t quick** to shutdown (ensuring running jobs don’t get killed) and then **pbs\_server -t warm** to restart is the way to go. Is that the process you ended up for safely restarting the server ? ---
**Author:** ![steveheistand](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@steveheistand](https://community.openpbs.org/u/steveheistand)\ **Post date:** [June 13, 2025, 2:26pm UTC](https://community.openpbs.org/t/revalidating-nodes/3682/16 "2025-06-13T14:26:34Z")
Im doing everything inside python so I use the ‘ptl’ package to interact with the PBS services. import ptl.lib.pbs\_testlib as ptl\_testlib ptl\_testlib.Server(server\_name).restart() will just restart the pbs\_server process leaving mom/sched/jobs/etc alone to continue on as normal.