# Placement sets with hypercube network topology **URL:** \ **Category:** Users/Site Administrators\ **Created:** [February 29, 2020, 8:35pm UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036 "2020-02-29T20:35:04Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![craig\_robinson](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@craig\_robinson](https://community.openpbs.org/u/craig_robinson)\ **Post date:** [February 29, 2020, 8:35pm UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/1 "2020-02-29T20:35:04Z")
Hi all, I’m experimenting with placement sets using 16 x VMs to simulate a small, 3D hypercube environment: - Vertex 1 (switch 1) - nodes n101 & n102 - Vertex 2 (switch 2) - nodes n201 & n202 - Vertex 3 (switch 3) - nodes n301 & n302 - Vertex 4 (switch 4) - nodes n401 & n402 - Vertex 5 (switch 5) - nodes n501 & n502 - Vertex 6 (switch 6) - nodes n601 & n602 - Vertex 7 (switch 7) - nodes n701 & n702 - Vertex 8 (switch 8) - nodes n801 & n802 ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/2d929b7130f040bfce1c09bd6ee2b0d342c9ea5b.png) The results so far are pleasing, with jobs being placed on nodes as expected (minimising the number of ‘hops’ between nodes assigned to the same job). The thing is, although it’s working fine, I’m not sure that I’ve gone about configuring PBS the best way. The resulting config is quite large/complex for a simple, 16-node cluster. If I used the same methodology for a large cluster, it’d be far too complex to manage. I’m probably over-thinking it by trying to cover every (optimal) node combination. I’d really appreciate some guidance from anyone who has experience with a similar (real-world) configuration. TIA, Craig ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/9129ca902104b79511f317c2b313ba04e4b74ccc.png) ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/155714cbe8ac526e47454caf9d994250fa12d65c.png) ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/9829034ad9d528c54ba509838ef6c22630ffde16.png) ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [March 1, 2020, 1:59am UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/2 "2020-03-01T01:59:00Z")
If I understand your use case correctly then I think you should only need to create 1 host level resource, say “switch”, then just set resources\_available.switch on the nodes to their respective switch number, say “s1”, “s2”, … “s8” … and set node\_group\_key to “switch”, this should make the scheduler group the nodes by each switch’s value and place the jobs on nodes with the same switch value ---
**Author:** ![craig\_robinson](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@craig\_robinson](https://community.openpbs.org/u/craig_robinson)\ **Post date:** [March 2, 2020, 8:35am UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/3 "2020-03-02T08:35:46Z")
Thanks very much for your response agrawalravi90, I’ll give that a try and let you know how it works out. In the meantime, I thought I’d add an example of how the current config is working. If I take two nodes offline (n201 & n302)… ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/b6f60432995db89d688ac493dbd208eed05a74dd.png) …then submit a job that requires four nodes… ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/3f2b7aa06d443064ab81e36865bb4f1ef8bdfdd6.png) …PBS chooses two nodes from two adjacent vertices/switches (see original diagram above). This is the exact behaviour that I want; minimum hops between nodes. Regards, Craig ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [March 2, 2020, 7:51pm UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/4 "2020-03-02T19:51:16Z")
Thanks for the clarification, I think I was mistaken, I thought that you didn’t want jobs span across switches. Ya, I can’t think of a simpler way of doing this, I’ll let those more knowledgeable comment. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [March 3, 2020, 1:36am UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/5 "2020-03-03T01:36:44Z")
For a larger system, you need to be less precise. If you provide all the placement sets for perfect placement, the scheduler will get bogged down and will be too slow. While the exact hypercube is nice, it probably doesn’t make much difference to have perfect placement. Slightly imperfect placement is probably good enough. Basically strip out some of the permutations of the larger sets. Bhroam ---
**Author:** ![craig\_robinson](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@craig\_robinson](https://community.openpbs.org/u/craig_robinson)\ **Post date:** [March 4, 2020, 7:54pm UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/6 "2020-03-04T19:54:31Z")
Thanks Ravi, Yeah, I tried this config… ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/b9c90ad53a74d93249ba4657bb2a367fbe9c3e71.png) …and it worked fine for jobs with 1-2 nodes. Jobs with \>2 nodes remained in the queue (due to “set sched do\_not\_span\_psets = true” and/or “set sched only\_explicit\_psets = true”). If I changed either of these to “false”, the job ran, but not on the desired nodes. Regards, Chris ---
**Author:** ![agrawalravi90](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@agrawalravi90](https://community.openpbs.org/u/agrawalravi90)\ **Post date:** [March 4, 2020, 10:04pm UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/7 "2020-03-04T22:04:16Z")
ya that’s what I’d expect, I had misunderstood your use case. ---
**Author:** ![bhroam](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/bhroam/32/20_2.png) [@bhroam](https://community.openpbs.org/u/bhroam)\ **Post date:** [March 4, 2020, 11:27pm UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/8 "2020-03-04T23:27:32Z")
when do\_not\_span\_psets is true, a job must fit within a placement set. Since the largest placement set you have set up is 2 nodes, the largest job can only request 2 nodes. You need to create larger placement sets for larger jobs. Set up 2 node placement sets for nodes 0 hops away Set up larger placement sets for nodes 1 hop away etc This way larger jobs will use the larger placement sets. These larger placement sets are the ones I was talking about before where you do not create all permutations of them. You need to create all of the smaller ones, and less and less of the larger ones. Bhroam ---
**Author:** ![craig\_robinson](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@craig\_robinson](https://community.openpbs.org/u/craig_robinson)\ **Post date:** [March 5, 2020, 10:07am UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/9 "2020-03-05T10:07:20Z")
Thanks very much Bhroam, I took your advice and went for a much simpler config… ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/806a9e4a6c44436126ebaa3bb99672a195562f95.png) Sure, it’s not perfect, but it works how I want it to in a majority of cases. Thanks again. Regards, Craig ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [March 6, 2020, 1:47am UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/10 "2020-03-06T01:47:18Z")
FWIW, We have a 12-D Infiniband hypercube system with 9 nodes at each vertex (SGI ICE-X). In our experience with real codes, the number of hops between nodes has less effect than does congestion on those links. And, except for a very few codes, the effect is negligible. Simply ordering the nodes so the dimension 1 connections are adjacent, then those 1-D hypercubes are grouped into adjacent 2-D cubes, etc, gives an arrangement where grabbing a sequential list of nodes **tends** to get a fairly compact set in terms of highest dimension needed to encompass all nodes. If the job owns all the nodes in a low-dimension hypercube, there is little contention for the links from other jobs. ---
**Author:** ![craig\_robinson](https://avatars.discourse-cdn.com/v4/letter/c/46a35a/32.png) [@craig\_robinson](https://community.openpbs.org/u/craig_robinson)\ **Post date:** [March 10, 2020, 11:42am UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/11 "2020-03-10T11:42:33Z")
Thanks very much Dale, I take your point re number of hops and congestion. Wouldn’t minimising the number of hops be a good way of reducing the likelihood of congestion? Re your second paragraph, could you explain that in a little more detail please? I take it to mean something like… nodes 101-109 on vertex 1 nodes 201-209 on vertex 2 … … nodes 701-709 on vertex 7 nodes 801-809 on vertex 8 …with the vertices arranged so that the next sequential vertex is adjacent… ![image](https://canada1.discourse-cdn.com/flex030/uploads/pbspro/original/1X/1376a9b128fe3b7ac22588dc6c60cd1677f06522.png) Regards, Craig ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [March 10, 2020, 5:03pm UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/12 "2020-03-10T17:03:13Z")
Yes, minimizing the number of hops tends to reduce the congestion, but this depends on how your network is routed. On the ICE-X systems, routing is done first by using minhop to find the list of shortest routes between vertices, then by selecting from those the path that traverses the network in dimension number order. (On the ICE-X the dimensions are ordered based on switch port number. E.g., if the inter-switch links use ports 10 - 19, then port 10 is dimension 1, port 11 is dimension 2, etc. [port numbers need not be successive]) Given this routing, if I have a job that occupies 2^N vertices and those vertices use only dimensions 1 - N links to talk to each other, then no other job will need to use any of my inter-switch links. So, any congestion is my own doing. Now, this job’s placement also satisfies the goal of minimal total hops. In fact, I think, but cannot prove, that any placement on 2^N vertices that uses only links from N different dimensions will also be free from congestion from other jobs. As to the numbering: Your numbering looks good to me, and is better than what is used on the ICE-X. Effectively, on the ICE-X, the vertices are numbered, starting at 0, such that directly connected vertices differ in their number by a power of 2. Thus, vertex 0 connects to 1, 2, 4, 8, …; 1 connects to 0, 3, 5, 9, … . This results in cases where selecting nodes sequentially from a list of free nodes can have more total hops than with your numbering scheme. ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [March 11, 2020, 1:14am UTC](https://community.openpbs.org/t/placement-sets-with-hypercube-network-topology/2036/13 "2020-03-11T01:14:55Z")
I did more thinking and am pretty sure I was wrong about the “any placement” part. To show it’s wrong, take a job that occupies opposite corners of the cube. If the cube is big enough, each of the two routes used between those vertices will pass through at least two connected vertices that are not part of the job. Those two vertices form a 2^1 cube, but their link includes traffic from outside the job.