@bhroam and @billnitzberg I have another proposal: make the job attr updates from sched to server asynchronous, similar to what we are discussing for asynrunjob here: Scheduler can spend 94% of its time waiting for job run ACK
When I make the job attr updates from sched asynchronous, the sched cycle time comes down to 8.5 mins, most of it was just time taken to run the jobs. If I combine the changes for asyrunjob as well, the sched cycle time goes down to just 1m50s, compared to 25 minutes before