# How to aggregate heterogeneous entire clusters into “one big cluster” for cross-cluster job submission **URL:** \ **Category:** Users/Site Administrators\ **Created:** [September 26, 2018, 12:07pm UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217 "2018-09-26T12:07:26Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![smilezy](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@smilezy](https://community.openpbs.org/u/smilezy)\ **Post date:** [September 26, 2018, 12:07pm UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/1 "2018-09-26T12:07:26Z")
how to aggregate heterogeneous entire clusters into “one big cluster” for cross-cluster job submission，as mentioned in [PBS Works](https://pbsgridworks.de/PBSProduct.aspx?n=PBS-Professional&c=Features) Thanks for your attention ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [September 26, 2018, 12:15pm UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/2 "2018-09-26T12:15:28Z")
Please check 1. Peer Scheduling if you would like to do it at the PBS Level using PBS Pro OSS 4.9.31 Peer Scheduling section from [https://www.pbsworks.com/pdfs/PBS18.2\_BigBook.pdf](https://www.pbsworks.com/pdfs/PBS18.2_BigBook.pdf) 2. PBS Access Suite (Commercial product) portal to submit jobs across multiple PBS Pro Clusters 3. Connect all the compute nodes from the respective to one PBS Pro server and manage the jobs with queues and policies. Note: you cannot mix and match Compute Nodes running Windows and Linux under on PBS Server hosted on Linux or Windows system. Linux PBS Server will **serve** Linux compute nodes and the same for Windows. If my answers does not answer your queries, please explain a bit in detail ---
**Author:** ![smilezy](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@smilezy](https://community.openpbs.org/u/smilezy)\ **Post date:** [September 27, 2018, 2:06am UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/3 "2018-09-27T02:06:27Z")
Hi，adarsh Thank you for your reply and attention, this is the answer I want, I will try, thank you very much. ---
**Author:** ![smilezy](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@smilezy](https://community.openpbs.org/u/smilezy)\ **Post date:** [September 27, 2018, 3:01am UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/4 "2018-09-27T03:01:59Z")
Hello, adarsh I have another question about PBS database，PBS Pro’s task submission records, scheduling status and other information are stored in the postgresql database.However, I found that the database does not immediately synchronize the job information each time the job is submitted, and sometimes the job information is not saved in the job\_attr table. I want to develop a cluster management system, including job management, resource management, etc.How do I get the job information in real time during the development process? Thank you for your attention. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [September 27, 2018, 7:47am UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/5 "2018-09-27T07:47:35Z")
The information disposed by qmgr -c "p s " , pbsnodes -av , qstat -fx , qstat -anws1 are in the memory. Any other attributes you change on any object is stored in database immediately - like form qmgr Node states are not updated in the database, since they are transient in nature, it is not stored in the database. 1. you can get the real time job information by running qstat -fx , tracejob , pbs\_dtj 2. You can use the PBS Libraries and develop code which suits your requirement , plese follow the guide [High-performance Computing (HPC) and Cloud Solutions | Altair](https://www.pbsworks.com/pdfs/PBSProgramGuide18.2.pdf) > [@smilezy](#): > > I want to develop a cluster management system, including job management, resource management, etc.How do I get the job information in real time during the development process? Most of the Cluster management system have PBS Pro OSS integrated. Could you please explain what you would like to achieve, so that other experienced community members might contribute their suggestions. ---
**Author:** ![smilezy](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@smilezy](https://community.openpbs.org/u/smilezy)\ **Post date:** [September 28, 2018, 2:35am UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/6 "2018-09-28T02:35:19Z")
Thank you for your reply ,We wanted to develop a job management system based on PBS, where monitoring functions are similar to Compute Manager, and we used Java to implement back-end logic, but we didn’t know how to get job information. For example, job ID, name, state, and so on Except get the real time job information by running qstat -fx, tracejob, pbs\_dtj, and then parse the query results We are currently using to get job information from PBS postgresql database, but found that all assignments submitted will not be saved to the database, the table “job”, “job\_attr”, “job\_scr”, especially in most case, it will not save job attributes, so we can’t get the job from PBS database of complete information, under what conditions operation attributes are saved in the “job\_attr”? If the job is not saved to the PBS postgresql database, how does the job not saved to the database be saved to memory? And there is a delay in saving to the database, why do jobs submitted through the command line not immediately save to the database? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [September 28, 2018, 7:50am UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/7 "2018-09-28T07:50:46Z")
Thank you for the information. The goal of the PBS datastore is not to store the entirety of the information of the cluster ( historical and real time information). Also, it is not good idea to connect to the PBS datastore to get the information of the cluster and the purpose of PBS datastore had no intention to support such an activity (integrate with cluster management system). It is not recommended to connect to PBS datastore with active PBS Server service , as it might have unknown consequences (integrity of the datastore). > [@smilezy](#): > > We wanted to develop a job management system based on PBS, where monitoring functions are similar to Compute Manager, and we used Java to implement back-end logic, but we didn’t know how to get job information. For example, job ID, name, state, and so on - use the PBS Pro API call to achieve this in an optimised way. - use PBS libraries to interact with the system from within Cluster Manager ( using C , python, java) ---
**Author:** ![smilezy](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@smilezy](https://community.openpbs.org/u/smilezy)\ **Post date:** [September 29, 2018, 1:58am UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/8 "2018-09-29T01:58:35Z")
Thank you very much, if we want to count all the execution hosts load information, not the vnode, and show it in the pie chart on the management system homepage, is there any advice? Does PBS provide a way to get the execution hosts load information? Thanks again ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [September 29, 2018, 7:27am UTC](https://community.openpbs.org/t/how-to-aggregate-heterogeneous-entire-clusters-into-one-big-cluster-for-cross-cluster-job-submission/1217/9 "2018-09-29T07:27:46Z")
> [@smilezy](#): > > if we want to count all the execution hosts load information, not the vnode, and show it in the pie chart on the management system homepage Then you can use a execution host periodic hook, that collects the load information of the compute nodes at regular intervals. Similar , all health check scripts can be implemented in the same way. > [@Host utilisation](http://community.openpbs.org/t/host-utilisation/960/2): > > PBS does not track this information. This particular information is more node specific and not job specific. If you would like this information then you can create a exechost\_periodic hook that collects these values and then assigns them to custom resources that get assigned to the nodes. For example you could create 3 custom resources in qmgr qmgr -c “create resource r15s r1m r15m type=float,flag=h” then create a periodic mom hook like this. import pbs import sys import os load = os.getlo… > [@smilezy](#): > > Does PBS provide a way to get the execution hosts load information? Thanks again exechost\_periodic hook provides the framework within which any kind of information can be collected periodically Other option: 1. Ganglia Thank you for these queries