# Multi-server switches and sharding algorithm design **URL:** \ **Category:** Developers\ **Created:** [March 11, 2020, 2:57am UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045 "2020-03-11T02:57:32Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![bremanandjk](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bremanandjk/32/144_2.png) [@bremanandjk](https://community.openpbs.org/u/bremanandjk)\ **Post date:** [March 11, 2020, 2:57am UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/1 "2020-03-11T02:57:32Z")
Hello All, We have started our work on the PBSPro multi-server project. As part of multi-server architecture, the pbs.conf file switches used to control the configuration and the sharding algorithm to choose the right server is designed and explained in detail at [https://pbspro.atlassian.net/wiki/spaces/PD/pages/1585545237/Sharding+algorithm+design](https://pbspro.atlassian.net/wiki/spaces/PD/pages/1585545237/Sharding+algorithm+design) We are expecting a sensible feedback. Thanks in advance! ---
**Author:** ![nithinj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nithinj/32/45_2.png) [@nithinj](https://community.openpbs.org/u/nithinj)\ **Post date:** [March 11, 2020, 5:49pm UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/2 "2020-03-11T17:49:24Z")
Thanks @bremanandjk for sharing. A few questions… How does PBS\_SERVER\_INSTANCES work for a mom on a remote machine? Does it has to specify server\_name:port\_number or will it guess from the already existing parameters such as PBS\_SERVER? If the conf file contains server\_name:port\_number, do we have to give PBS\_SERVER parameter explicitly? If it makes use of the already existing PBS\_SERVER field, does it has to appear before PBS\_SERVER\_INSTANCES? After modifying PBS\_SERVER\_INSTANCES, what are the steps to be followed for this change to be reflected in the whole cluster? Can an instance be removed from this list? Does the order in which server instances are listed matters? Do all clients and moms have to follow the same PBS\_SERVER\_INSTANCES at a given point in time? **pbs\_shard\_init** () takes struct for **server\_instance** whereas **pbs\_shard\_get\_server\_byindex** expects int for **inactive\_server\_indexes**. Can we maintain consistency? ---
**Author:** ![bremanandjk](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bremanandjk/32/144_2.png) [@bremanandjk](https://community.openpbs.org/u/bremanandjk)\ **Post date:** [March 11, 2020, 9:17pm UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/3 "2020-03-11T21:17:27Z")
Thanks for your questions @nithinj > do we have to give PBS\_SERVER parameter explicitly? As we know, pbs\_mom can’t run alone without knowing the location of pbs\_server. So YES, it is a must thing to specify the PBS\_SERVER parameter in the pbs.conf file. It is not only for the Mom case, even if the scheduler runs in the standalone host, the PBS\_SERVER parameter should exist in the conf file. > After modifying PBS\_SERVER\_INSTANCES, what are the steps to be followed for this change to be reflected in the whole cluster? Can an instance be removed from this list? In the first implementation of multi-server, if the admin updates this parameter, it is outlined that admin should restart all the PBS moms or need to HUP them to re-read the changes in configuration. > Does the order in which server instances are listed matters? Yes, the order matters. In ideal HPC clusters, conf files are managed by distributing the same file across all machines using the distribution (dsh) tools. It would not be complex to retain the same order. > Do all clients and moms have to follow the same PBS\_SERVER\_INSTANCES at a given point in time? As mentioned earlier, the configuration file should be consistent across all machines in the PBS Pro cluster. > **pbs\_shard\_init** () takes struct for **server\_instance** whereas **pbs\_shard\_get\_server\_byindex** expects int for **inactive\_server\_indexes**. Can we maintain consistency? In the sharding policy definition, we have endorsed that sharding is a joint understanding between the application and the library. Since we already pass on the array of struct to **pbs\_shard\_init()**, the library just needs to know the inactive servers, passing index should be convenient in terms of computation. There is no benefit of passing in the struct form again. ---
**Author:** ![suresht](https://avatars.discourse-cdn.com/v4/letter/s/ec9cab/32.png) [@suresht](https://community.openpbs.org/u/suresht)\ **Post date:** [March 12, 2020, 10:48am UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/4 "2020-03-12T10:48:11Z")
**PBS\_MAX\_SERVERS** : Specify the maximum number of servers that this cluster will be allowed to start without bringing down everything in the cluster. However, while adding any new servers, all the moms are needed to hup for re-reading the updated pbs.conf file. **PBS\_SERVER\_INSTANCES: [[host]:port,][[host]:port,]…** Are we planning to run multiple servers in the same machine only in phase-1? If not where is the database located and how Servers connect to it? As we know we have pbs conf variables like PBS\_DATA\_SERVICE\_HOST, PBS\_DATA\_SERVICE\_PORT to connect to external database host. If we are planning to use them it is good to mention all these details also in the document. ---
**Author:** ![bremanandjk](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bremanandjk/32/144_2.png) [@bremanandjk](https://community.openpbs.org/u/bremanandjk)\ **Post date:** [March 12, 2020, 1:44pm UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/5 "2020-03-12T13:44:52Z")
Thanks @suresht for your feedback, We are planning to run multi-servers on remote machine also along with data service host. I have updated the document with data service related switches. ---
**Author:** ![nithinj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nithinj/32/45_2.png) [@nithinj](https://community.openpbs.org/u/nithinj)\ **Post date:** [March 12, 2020, 5:25pm UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/6 "2020-03-12T17:25:50Z")
> [@bremanandjk](#): > > So YES, it is a must thing to specify the PBS\_SERVER parameter in the pbs.conf file. I see from the doc that you can use host:port for PBS\_SERVER\_INSTANCES. Do they still have to specify PBS\_SERVER? Isn’t it redundant? Also thanks for adding more details on PBS\_SERVER in the doc. > [@bremanandjk](#): > > t is outlined that admin should restart all the PBS moms or need to HUP them to re-read the changes in configuration. Can you mention it under the PBS\_SERVER\_INSTANCES? Only moms are mentioned in the document? How about scheduler and other servers? Can an instance be removed from PBS\_SERVER\_INSTANCES list? > [@bremanandjk](#): > > Yes, the order matters. Thanks for confirming and updating the docs. > [@bremanandjk](#): > > he configuration file should be consistent across all machines in the PBS Pro cluster. I hope you are talking about the two multi-server switches mentioned in the document. Can you specify it in the document. > [@bremanandjk](#): > > Since we already pass on the array of struct to **pbs\_shard\_init()** , the library just needs to know the inactive servers, passing index should be convenient in terms of computation. The library is not returning anything to the caller as part of init(). If you are assuming that the caller maintains the same structure as the library, why would the init() is expecting the strucure when int is efficient? ---
**Author:** ![bremanandjk](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bremanandjk/32/144_2.png) [@bremanandjk](https://community.openpbs.org/u/bremanandjk)\ **Post date:** [March 12, 2020, 5:47pm UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/7 "2020-03-12T17:47:02Z")
> Can you mention it under the PBS\_SERVER\_INSTANCES? Only moms are mentioned in the document? How about scheduler and other servers? I have changed moms to all the services. > [@nithinj](#): > > I see from the doc that you can use host:port for PBS\_SERVER\_INSTANCES. Do they still have to specify PBS\_SERVER? Isn’t it redundant? Also thanks for adding more details on PBS\_SERVER in the doc. In doc, We are not mandating PBS\_SERVER parameter, we are just saying that admin can ignore only if PBS\_SERVER exists prior to PBS\_SERVER\_INSTANCES. It’s an optional thing. > [@nithinj](#): > > The library is not returning anything to the caller as part of init(). If you are assuming that the caller maintains the same structure as the library, why would the init() is expecting the strucure when int is efficient? The reason is very simple, we are passing additional information to the library so that it will be feasible to implement another version of the library that sets the server id based on host:port combination. And, even in the case of “Service locators” like etcd, zookeper and, consul…, it would be needed to know the list of configured servers for validation purposes. ---
**Author:** ![bremanandjk](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bremanandjk/32/144_2.png) [@bremanandjk](https://community.openpbs.org/u/bremanandjk)\ **Post date:** [March 12, 2020, 6:40pm UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/8 "2020-03-12T18:40:53Z")
> [@nithinj](#): > > If you are assuming that the caller maintains the same structure as the library, Here I am not assuming. The design is caller should maintain the array of struct and pass on to init(). ---
**Author:** ![nithinj](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/nithinj/32/45_2.png) [@nithinj](https://community.openpbs.org/u/nithinj)\ **Post date:** [March 13, 2020, 5:18pm UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/9 "2020-03-13T17:18:03Z")
Thanks for updating the document. Also repeating a question which you have missed in the last attempts: > [@nithinj](#): > > Can an instance be removed from PBS\_SERVER\_INSTANCES list? > [@bremanandjk](#): > > we are passing additional information to the library so that it will be feasible to implement another version of the library that sets the server id based on host:port combination. I see. So the int array are the indexes of struct which the caller has already passed for init(). It will be more clear if you can specify it in the document. As PBS\_SERVER\_INSTANCES takes more than one server as input, do we still acknowledge PBS\_PRIMARY and PBS\_SECONDARY variables which were used for failover? ---
**Author:** ![bremanandjk](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bremanandjk/32/144_2.png) [@bremanandjk](https://community.openpbs.org/u/bremanandjk)\ **Post date:** [March 13, 2020, 6:20pm UTC](https://community.openpbs.org/t/multi-server-switches-and-sharding-algorithm-design/2045/10 "2020-03-13T18:20:53Z")
> [@nithinj](#): > > Also repeating a question which you have missed in the last attempts: That question has already been explained in the doc. I will re-iterate to you here. "However, while adding/removing any servers, all the services are needed to hup for re-reading the updated pbs.conf file. " > [@nithinj](#): > > I see. So the int array are the indexes of struct which the caller has already passed for init(). It will be more clear if you can specify it in the document. PLEASE refer these lines in the document, The logic will return the next active server instance by referring to the array of configured server instances and an array of inactive server instances. However, the caller should maintain the same series of server instances received from the last call. On success, returns one of the index value of server\_instance array; on error, it returns -1 > [@nithinj](#): > > As PBS\_SERVER\_INSTANCES takes more than one server as input, do we still acknowledge PBS\_PRIMARY and PBS\_SECONDARY variables which were used for failover? Here, this design page is cautiously designed to talk about the sharding algorithm and switches related to sharding, Just implementation of this module alone will not be sufficient to achieve multi-server architecture. Ideally, you should expect the topics other than the “Sharding algorithm” under this tree [https://pbspro.atlassian.net/wiki/spaces/PD/pages/1507164161/PBS+Pro+multi-server+project+architecture+and+design](https://pbspro.atlassian.net/wiki/spaces/PD/pages/1507164161/PBS+Pro+multi-server+project+architecture+and+design). Said that the failover related work is not handled as part of this work. I abstain from adding any additional information.