# How to query the number of available cores to your job

**URL:** <https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185>\
**Category:** Users/Site Administrators\
**Created:** [September 10, 2018, 2:09pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185 "2018-09-10T14:09:45Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [September 10, 2018, 2:09pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/1 "2018-09-10T14:09:45Z")

</div>

Dear folks I was wondering if it would be possible to let me know how to query the number of available jobs to your batch job after you’ve set up your nodes with something like `#PBS -l nodes=3,cores=24`.

I know there’s `pbsnodes -a` but I’m not sure if there anything like `pbscores` I’m expecting to return the number of nodes that I’ve requested for this particular job which is 24.

Thanks!

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [September 11, 2018, 7:26am UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/2 "2018-09-11T07:26:56Z")

</div>

Please note the syntax: nodes and cores are old syntax, please update your scripts to use select,ncpus,mem,mpiprocs etc.

#PBS -l select=3:ncpus=24  
Which means you are requested 1 chunk with 24 cores .  
In total you are requesting 72 cores for this job.

Please follow the quick start guide : [https://www.pbsworks.com/pdfs/PBSQuickStartGuide18.2.pdf](https://www.pbsworks.com/pdfs/PBSQuickStartGuide18.2.pdf)

The below commands might also help you to get the information you need:

1. pbsnodes -aSj
2. qstat -fx jobid
3. qstat -answ1

---

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [September 11, 2018, 8:27am UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/3 "2018-09-11T08:27:35Z")

</div>

Thank you so much for the response and the link. I’ll definitely have a good look at them. I was wondering if you could point me out to a similar guide or material which shows how to use spark with PBS?

---

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [September 11, 2018, 10:31am UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/4 "2018-09-11T10:31:35Z")

</div>

`pbsnodes -aSj` seems to be an invalid commnad

-bash-4.1$ pbsnodes -aSj  
pbsnodes: invalid option – ‘S’  
pbsnodes: invalid option – ‘j’  
usage: pbsnodes [-{c|d|l|o|p|r}] [-s server] [-n] [-N “note”] [-A “append note”] [-q] [-m standby|suspend|sleep|hibernate|shutdown] node …  
pbsnodes [-{a|x}] [-s server] [-q] [node]

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [September 11, 2018, 3:55pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/5 "2018-09-11T15:55:33Z")

</div>

Please note the command is valid for PBS Pro OSS . Please share the version of the PBS Pro you are running.

This is what i see on my system :

```
> root@pbsserver ~]# pbsnodes -aSj
> mem ncpus nmics ngpus
> vnode state njobs run susp f/t f/t f/t f/t jobs
> --------------- --------------- ------ ----- ------ ------------ ------- ------- ------- -------
> cn1 state-unknown 0 0 0 0kb/0kb 0/0 0/0 0/0 --
> cn2 state-unknown 0 0 0 0kb/0kb 0/0 0/0 0/0 --
> cn3 free 0 0 0 23gb/23gb 4/4 0/0 2/2 --
> 
> [root@pbsserver ~]# pbsnodes --version
> pbs_version = 18.1.0
```

---

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [October 10, 2018, 12:05am UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/6 "2018-10-10T00:05:18Z")

</div>

@adarsh Hi and thank you so much for your help. My apologies for the late reply.  
Here’s the version of pbs running on the cluster

```auto
-bash-4.1$ pbsnodes --version
Version: 6.0.0.1
Commit: 21cc3d8b18424c77871cd95265148deb97993d14

```

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [October 10, 2018, 7:12am UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/7 "2018-10-10T07:12:51Z")

</div>

Please note , the flavour and version of the scheduler is not PBS Pro OSS.  
The PBS Pro OSS version history 13.x.x , 14.x.x and the latest is 18.1.2  
Could you please check whether you have downloaded the software from the below link

[https://www.pbspro.org/Download.aspx#download](https://www.pbspro.org/Download.aspx#download)

or

> **[PBSPro/pbspro](https://github.com/pbspro/pbspro)**
>
> An HPC workload manager and job scheduler for desktops, clusters, and clouds. - PBSPro/pbspro

---

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [October 10, 2018, 1:09pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/8 "2018-10-10T13:09:22Z")

</div>

Hi @adarsh thank you for your clarifications. I don’t have any admin access on this system, I haven’t installed anything. This is a shared environment which is using SGE and pbs but to be honest I don’t know if it’s PBSPro. The other question that I have is whether PBSPro and PBS are different pieces of software?

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [October 10, 2018, 1:51pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/9 "2018-10-10T13:51:04Z")

</div>

> [@kirk](#):
>
> The other question that I have is whether PBSPro and PBS are different pieces of software?

Both are the same piece of software with respect scheduling, workload management,job management, but :  
**PBS Professional Commercial** - licensed - [www.pbsworks.com](http://www.pbsworks.com)  
**PBS Professional OSS** – open source - [www.pbspro.org](http://www.pbspro.org)

Please run these commands on the headnode and check the output:  
rpm -qa | grep pbs  
rpm -qa | grep sge  
rpm -qa | grep torque

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [October 10, 2018, 1:59pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/10 "2018-10-10T13:59:14Z")

</div>

> [@kirk](#):
>
> usage: pbsnodes [-{c|d|l|o|p|r}] [-s server] [-n] [-N “note”]

I should have noticed this, your cluster is not running PBS Pro OSS but it is running torque.  
Please use the above snippet and search on google.

Refer these discussions:

> [@PBS Default number of nodes and number of cpus](http://community.openpbs.org/t/pbs-default-number-of-nodes-and-number-of-cpus/834/4):
>
> So a little history for your knowledge. PBS was originally developed for NASA under a contract project. Altair Engineering acquired the rights to all the PBS technology and intellectual property from Veridian in 2003. Altair Engineering currently owns and maintains the intellectual property associated with PBS, and also employs the original development team from NASA. The following versions of PBS are currently available: OpenPBS — original open source version released by MRJ in 1998 (not ac…

> [@Migrating from Torque/maui, slurm vs pbs pro](http://community.openpbs.org/t/migrating-from-torque-maui-slurm-vs-pbs-pro/740):
>
> We have a system set up with Torque-Maui and would like to get a new workload manager. Our requirements include support scheduling on Windows, being able to scale with 300-500 systems and 30000-50000 jobs at least What is a better scheduler to switch to between PBS Pro and SLURM? Our observation is that Maui crashes once the number of jobs lined up in the queues exceed 33-34k. Does this happen in PBS Pro too? What are the steps to switch from the existing system to PBS Pro?

---

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [October 10, 2018, 2:07pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/11 "2018-10-10T14:07:33Z")

</div>

@adarsh thanks so much. The commands don’t output anything. I suppose that `torque` is commercial since it’s from adaptivecomputing company. Torque is installed in:

```auto
 bash-4.1$ which pbsnodes
/opt/torque/torque/bin/pbsnodes

```

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [October 10, 2018, 2:48pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/12 "2018-10-10T14:48:12Z")

</div>

Thank you @kirk for this information. This answers everything related to information disposed by pbsnodes

---

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [October 10, 2018, 3:53pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/13 "2018-10-10T15:53:06Z")

</div>

@adarsh thank you for your patience with all my stupid questions. If you don’t mind me asking one last stupid question.  
I have the following scenario where I want to run a spark job across N nodes.  
In order to do that I have to login to node 0 and set that as the master by executing a particular spark command and finally login the rest N-1 nodes and make them the workers by executing another spark command.

The way that I’ve found is using `pbsdsh`.

1. Is there any other way to login to nodes selectively, doing simple ssh doesn’t work since it requires a password and apparently is different than the password that I use to ssh into the login node of the cluster.

2. The main issue that I have with `pbsdsh` is that it doesn’t forward any environment variables into the nodes that you try to execute a command, is there any way around this?

Minimal working example:

```auto
#!/bin/bash -l 

#PBS -l nodes=1:ppn=12

# set the walltime in hours:minites:seconds
#PBS -l walltime=00:30:00

#PBS -N TestJob

# join the standard output and error files
#PBS -j oe

# send me email when job begins (b), ends (e) and/or aborts (a)
#PBS -m bea

# inherit the current environment
#PBS -V

#load a module file 
module purge
module load Miniconda
source activate conda-environment

cd ${PBS_O_WORKDIR}

pbsdsh -n 0 -- /bin/bash -c "module load Miniconda; source activate conda-environment; which python;"

exit 0

```

Output from `pbsdsh` command:

```auto
module: command not found
activate: No such file or directory
/usr/bin/python

```

What am I missing here? Am I doing something wrong?

Many thanks in advance?

---

<div class="post-metadata">

**Author:** ![mkaro](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/mkaro/32/85_2.png) [@mkaro](https://community.openpbs.org/u/mkaro)\
**Post date:** [October 10, 2018, 3:56pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/14 "2018-10-10T15:56:04Z")

</div>

> **[Environment Modules (software)](https://en.wikipedia.org/wiki/Environment_Modules_(software))**
>
> The Environment Modules system is a tool to help users manage their Unix or Linux shell environment, by allowing groups of related environment-variable settings to be made or removed dynamically.
> Modules has been around since the early 1990s and is used at some of the largest computer centers to deploy multiple versions of different software tools to users. The National Energy Research Scientific Computing Center (NERSC) reports that they use Environment Modules to manage nearly all software. ...

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [October 10, 2018, 7:01pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/15 "2018-10-10T19:01:30Z")

</div>

+1 @mkaro

No problems @kirk , please keep them coming. It would good to see PBS Pro OSS ( in the near future ) on your cluster 🙂

> [@kirk](#):
>
> What am I missing here? Am I doing something wrong?

Please try the below

1. Save the above script in a common location accessible by all the compute nodes (main\_pbs.sh)

> [@kirk](#):
>
> The main issue that I have with `pbsdsh` is that it doesn’t forward any environment variables into the nodes that you try to execute a command, is there any way around this?

1. Save the list of commands to be called by pbsdsh in another script and save it in the same location  
(client\_pbs.sh)  
cat pbs.sh  
#!/bin/bash -L  
module load Miniconda  
source activate conda-environment  
which python  
hostname

3.Add the below line (lines in case of multinode )  
pbsdsh -n 0 – /bin/bash -c “/shared/path/to/client\_pbs.sh” # this will execute on the mother superior

#pbsdsh -w node2,node3 “/shared/path/to/somescript” node2,node3 is the list of sister nodes which  
excludes mother superior, you need to create this from $PBS\_NODEFILE

> [@kirk](#):
>
> Is there any other way to login to nodes selectively, doing simple ssh doesn’t work since it requires a password and apparently is different than the password that I use to ssh into the login node of the cluster.

Password-less SSH need to be configured for all the users , it should work between server to node, node to server and node to node ( stricthostkeychecking should be turned off ) . This is a basic requirement.

---

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [October 11, 2018, 3:09pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/16 "2018-10-11T15:09:52Z")

</div>

@adarsh Thank you! Your suggestion helped me a lot. I also found that it is possible to pass environment variables while you’re submitting the job using `qsub -V BASH_ENV=~/.bash_profile` “your job”;

I managed to achieve what I wanted by this this simple for loop after your suggestions:

```auto
i=0
for node in $UNIQ_NODES; do
    if [$i == 0]; then
        pbsdsh -u -h $node "$PBS_O_WORKDIR/master.sh"
    else
        pbsdsh -u -h $node "$PBS_O_WORKDIR/worker.sh"
    fi
    i=$((i + 1))
done

```

After the above snippet there’s a simple python script to verify if things have been setup properly

```auto
python verify.py

```

There’s one thing that still puzzles me. It seems that the for loop get executed properly and things are setup as they should, but the python script is never executed. The job get’s cancelled since it reaches the requested wall time and after debugging the messages it seems everything to be set up properly except from the fact that the python script is never executed.

Any ides on why that might happen?

---

<div class="post-metadata">

**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\
**Post date:** [October 11, 2018, 3:25pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/17 "2018-10-11T15:25:49Z")

</div>

Nice one ! I like your script 🙂  
I hope you got these $UNIQ\_NODES from $PBS\_NODEFILE

quote=“kirk, post:16, topic:1185”]  
qsub -V  
[/quote]  
FYI:  
Using qsub -V is not a best practice. If the environment variable list set on the system is too huge, then it is too much of information to store in the pbs datastore. Better to use qsub -v VAR1=a,VAR2=b,VAR3=c or have specific variable set in the script.

> [@kirk](#):
>
> After the above snippet there’s a simple python script to verify if things have been setup properly

Please try : verify is in the location of master.sh  
/usr/bin/python $PBS\_O\_WORKDIR/verify.py

Note: Make sure any script that you would like to execute is accessible by all the nodes and is available in the same working director.

If you submit a job without a walltime , then the default walltime set on the job(s) is 5 years

---

<div class="post-metadata">

**Author:** ![kirk](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kirk](https://community.openpbs.org/u/kirk)\
**Post date:** [October 11, 2018, 3:43pm UTC](https://community.openpbs.org/t/how-to-query-the-number-of-available-cores-to-your-job/1185/18 "2018-10-11T15:43:22Z")

</div>

Thanks @adarsh

> [@adarsh](#):
>
> I hope you got these $UNIQ\_NODES from $PBS\_NODEFILE

exactly after piping into `uniq`.

> [@adarsh](#):
>
> Better to use qsub -v VAR1=a,VAR2=b,VAR3=c or have specific variable set in the script.

That is good to know. Thank you!

> [@adarsh](#):
>
> Please try : verify is in the location of master.sh  
> /usr/bin/python $PBS\_O\_WORKDIR/verify.py

Indeed `verify.py` is in the same folder as `master.sh` . Only thing that I can see is that I forgot to prepend `$PBS_O_WORKDIR`. Would that explain though that is not being executed?

> [@adarsh](#):
>
> Note: Make sure any script that you would like to execute is accessible by all the nodes and is available in the same working director.

Pretty much all the scripts are in the same directory, a.k.a `$PBS_O_WORKDIR` from which I launch the job.

> [@adarsh](#):
>
> If you submit a job without a walltime , then the default walltime set on the job(s) is 5 years

I’d only wish, the max we get at the uni. cluster is 15 days.

Let me give it a try with `$PBS_O_WORKDIR` prepended and get back in case of any updates.