I wanted to start a discussion about the jobid namespace problem that comes along with multi-server and how we should solve it. We have broadly two directions to go in:
- keep the strictly sequential nature of jobids that exists today in PBS, i.e - if you submit 100 jobs, you’ll get jobids from 0.server_name to 99.server_name, in increasing order.
- Sacrifice the strictly sequential nature of jobids, so if one so if you submit 100 jobs, you are not guaranteed to get job ids from 0.server_name to 100.server_name, they can be anything.
I thought of a few approaches that we can choose for each option:
Approaches which sacrifice sequential jobids:
a) Add a prefix to job ids of each server, something like: jobs starting with id 0 belong to server 0, 1 to server 1 and so on … the con is obviously that we can only have 10 multi-servers max, but maybe 10 is large enough? another con is that jobids won’t be sequential. pro is that this is really easy to implement and understand and will not add any performance degradation
b) Job id ranges: here we will ask admins to configure job id ranges for each server. So, server 1 will get ids from 0 to 1 million, server 2 will gets 1 to 2 million and so on. pro is that this doesn’t add any performance degradation, also easy to implement, but suffers from the problem of jobids not being sequential. Also, I’m not clear on what we will do when the ranges run out.
c) Servers generate ids with an offset: if there are 3 servers then server 1 generates ids 0, 3, 6 etc., server 2 generates ids 1, 4, 7 etc. and server 3 generates 2, 5, 8 etc. This has the same advantages of the first two approaches, plus the jobids generated by each server are closer to each other. It does have the disadvantage that the logic is highly dependent on the number of servers, so it might involve a complex remapping of ids to servers if the number of servers is modified.
Approaches which preserve sequential jobids:
a) Have a job id generating service: this one will be a single point of contact for all servers to get unique, sequential job ids. It will allow job ids to still be sequential. Cons: possible degradation in performance as it serializes job id creation, and it will also mean that clients (except scheduler) will have to contact this service to know which server a jobid maps to, again causing loss in performance, but maybe not that much. Also, we will have to think about when to clean up its cache of jobid:server as jobs leave the system.
b) Use a “load balancer” which will take “all” requests and route them to the different servers depending on which server they should go to. For qsub requests, this service can generate a unique jobid, and choose one of the servers to send it to. Instead of storing a huge map of jobids to server like we have to in bullet a), we can use a technique like consistent hashing to decide which server a jobid should go to so we might not need to maintain a cache if we do this. Con again is possible loss in performance because we’ve introduced a single entity through which all client calls go (scheduler should still be able to contact each server directly, so this would apply to front-end clients only)
Can you guys think of other approaches?
I think we need to first decide if sequential id generation is important and then discuss which approach is better for a balance between performance and ease of use in case of change in number of servers. Please provide feedback!
Design in review: https://openpbs.atlassian.net/wiki/spaces/PD/pages/2320924673/jobid+namespace+redesign+for+multi-server