# How to set all cpu core's num for a job directly, no care host's num **URL:** \ **Category:** Users/Site Administrators\ **Created:** [December 22, 2021, 10:01am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911 "2021-12-22T10:01:48Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![zhangyi2012](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zhangyi2012/32/456_2.png) [@zhangyi2012](https://community.openpbs.org/u/zhangyi2012)\ **Post date:** [December 22, 2021, 10:01am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/1 "2021-12-22T10:01:48Z")
I know the general resource req. For example, three resource chunks are defined in the script, and 32 core per chunk|node, eg: #PBS -l select=3:ncpus=32 But, normal user may be want just input total cpu core’s num, no care how many chunk or node will be use. How to do that? eg: how to directly define the total number of CPU cores required to be 96, regardless of num of resource chunk or vnodes are required? #PBS -l ncpus=96 will conver to #PBS -l select=1:ncpus=96 #PBS -l place=pack, and one host has only 32 cores, then job faild as “Insufficient amount of resource: ncpus” ---
**Author:** ![zhangyi2012](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zhangyi2012/32/456_2.png) [@zhangyi2012](https://community.openpbs.org/u/zhangyi2012)\ **Post date:** [December 22, 2021, 10:17am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/2 "2021-12-22T10:17:04Z")
**The follow is failed job info:** Job Id: 41.poc-self-master Job\_Name = pbs-test-job Job\_Owner = hpcuser@poc-self-login job\_state = Q queue = workq server = poc-self-master Checkpoint = u ctime = Wed Dec 22 17:21:41 2021 Error\_Path = poc-self-login:/home/nfs/pbs-test-job.e41 Hold\_Types = n Join\_Path = n Keep\_Files = n Mail\_Points = a mtime = Wed Dec 22 17:21:41 2021 Output\_Path = poc-self-login:/home/nfs/pbs-test-job.o41 Priority = 0 qtime = Wed Dec 22 17:21:41 2021 Rerunable = True Resource\_List.ncpus = 96 Resource\_List.nodect = 1 Resource\_List.place = pack Resource\_List.select = 1:ncpus=96 substate = 10 Variable\_List = PBS\_O\_HOME=/home/hpcuser,PBS\_O\_LANG=en\_US.UTF-8, PBS\_O\_LOGNAME=hpcuser, PBS\_O\_PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/ bin:/home/hpcuser/.local/bin:/home/hpcuser/bin, PBS\_O\_MAIL=/var/spool/mail/hpcuser,PBS\_O\_SHELL=/bin/bash, PBS\_O\_WORKDIR=/home/nfs,PBS\_O\_SYSTEM=Linux,PBS\_O\_QUEUE=workq, PBS\_O\_HOST=poc-self-login comment = Not Running: Insufficient amount of resource: ncpus etime = Wed Dec 22 17:21:41 2021 Submit\_arguments = pbs\_mpi\_test\_script project = \_pbs\_project\_default ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [December 23, 2021, 7:15pm UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/3 "2021-12-23T19:15:33Z")
You could try `-l select=96:ncpus=1 -l place=free`. This would find CPUs for the job whereever they were available. It might give poor performance if the CPUs of the job need to talk to each other a lot (e.g., with MPI). ---
**Author:** ![zhangyi2012](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zhangyi2012/32/456_2.png) [@zhangyi2012](https://community.openpbs.org/u/zhangyi2012)\ **Post date:** [December 24, 2021, 3:35am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/4 "2021-12-24T03:35:11Z")
I got it, thank you for your reply. I still have a new question, sorry. There are 2 hosts, poc-self-master and poc-self-com001, 32 cores and 64 threads(HT) per host case1: #PBS -l select=2:ncpus=32:mpiprocs=32 #PBS -l place=free Because free, total 64 cpus of 2 chunks all come from poc-self-master(HT, threads=64), output: resources\_used.cpupercent = 4549 resources\_used.cput = 00:01:31 resources\_used.mem = 0kb resources\_used.ncpus = 64 resources\_used.vmem = 0kb resources\_used.walltime = 00:00:02 job\_state = F queue = workq server = poc-self-master Checkpoint = u ctime = Fri Dec 24 11:13:51 2021 Error\_Path = poc-self-login:/home/nfs/pbs-test-job.e86 exec\_host = poc-self-master/0_32+poc-self-master/1_32 exec\_vnode = (poc-self-master:ncpus=32)+(poc-self-master:ncpus=32) case2: #PBS -l select=2:ncpus=32:mpiprocs=32 #PBS -l place=scatter Because scatter, one chunk comes from master, the other one comes from com002, output: resources\_used.cpupercent = 100 resources\_used.cput = 00:00:01 resources\_used.mem = 428kb resources\_used.ncpus = 64 resources\_used.vmem = 12908kb resources\_used.walltime = 00:00:00 job\_state = F queue = workq server = poc-self-master Checkpoint = u ctime = Fri Dec 24 11:19:12 2021 Error\_Path = poc-self-login:/home/nfs/pbs-test-job.e87 exec\_host = poc-self-master/0_32+poc-self-com001/0_32 exec\_vnode = (poc-self-master:ncpus=32)+(poc-self-com001:ncpus=32) Question: case1(use 64 HT threads in master): resources\_used.cpupercent = 4549 resources\_used.cput = 00:01:31 case2(32 core in master,32 core in com001): resources\_used.cpupercent = 100 resources\_used.cput = 00:00:01 I knows something about hyper thread(HT), but I still confused. the cpupercent and cput of the 2 case are very different, why??? I even wonder if I have a wrong understanding of cputs and cpupercent or not. ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [December 27, 2021, 12:58am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/5 "2021-12-27T00:58:24Z")
Your test case ran too quickly. Try a test that runs for several minutes and see if you get more reasonable results. (Also, are you sure the two-node case ran correctly? The walltime is so short it looks like it might have failed on startup.) It is interesting that cramming all the processes onto one node via hyperthreading was so slow. Where I used to work, we seldom used hyperthreading, and this is a good example of why. ---
**Author:** ![zhangyi2012](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zhangyi2012/32/456_2.png) [@zhangyi2012](https://community.openpbs.org/u/zhangyi2012)\ **Post date:** [December 27, 2021, 1:56am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/6 "2021-12-27T01:56:51Z")
Thank you for your reply, I think the two cases ran correcttly, which are hello example like this,which only MPI\_Init MPI\_Finalize, and that is why case run so quickly: int main(int argc, char\* argv[]) { int rank, size, len, name\_len;; char version[MPI\_MAX\_LIBRARY\_VERSION\_STRING]; char processor\_name[MPI\_MAX\_PROCESSOR\_NAME]; ``` MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Get_processor_name(processor_name, &name_len); MPI_Get_library_version(version, &len); printf("Hello, world, I am %d of %d on %s, (%s, %d)\n", rank, size, processor_name, version, len); MPI_Finalize(); return 0; ``` } I think HT should be disabled directly, or not be disabled is also ok when I only run 32 MPI processes on one host which has 32 cores(64 HT threads), because it will just only use the 32 natural cores automatically not HT threads Am I right ? Thank you! ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [December 27, 2021, 10:45am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/7 "2021-12-27T10:45:31Z")
Usually HT is disabled in the BIOS for most of the HPC clusters , unless there is a reason for the speedup (gained by enabling HT) by some of the applications. Mostly this would be application specific. ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [December 27, 2021, 4:13pm UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/8 "2021-12-27T16:13:58Z")
> [@zhangyi2012](#): > > I think HT should be disabled directly, or not be disabled is also ok when I only run 32 MPI processes on one host which has 32 cores(64 HT threads), because it will just only use the 32 natural cores automatically not HT threads > Am I right ? You are mostly correct. Modern Linux kernels do a reasonable job of spreading CPU load across HT resources. So, it’s okay to leave HT enabled, provided you don’t have more active threads than the number of real cores. I believe this will be only slightly slower than disabling HT in the BIOS, depending on the processor hardware version. [My memory is not positive on the next parts.] This is what we did at the site where I worked. We configured the MoMs so that they reported ncpus as the number of cores, so jobs would not use hyperthreading by accident. If a job wanted hyperthreading, it requested ncpus of half what it really wanted and then just used all the threads. (Nodes were exclusive.) When we wanted to run benchmarks for the Top 500 ranking, we turned the secondary threads off after boot by writing 0 to /sys/devices/system/cpu/cpuX/online for each HT thread. This was almost as good as disabling HT in the BIOS. ---
**Author:** ![zhangyi2012](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zhangyi2012/32/456_2.png) [@zhangyi2012](https://community.openpbs.org/u/zhangyi2012)\ **Post date:** [December 28, 2021, 2:45am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/9 "2021-12-28T02:45:10Z")
Thank you for your reply, I will disable HT in BIOS at this stage. I think starccm+ should just use real cores. Thank you! ---
**Author:** ![zhangyi2012](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zhangyi2012/32/456_2.png) [@zhangyi2012](https://community.openpbs.org/u/zhangyi2012)\ **Post date:** [December 28, 2021, 2:53am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/10 "2021-12-28T02:53:32Z")
Thank you for your reply. 1.The first part of your reply is very usefull for me, which verified my point of view, OS kernel can use (real) cores reasonably as long as I don’t have more active threads|processes than real cores 2.I will disable HT in BIOS directly, and I’m not sure if pbs mom reports real cores by default or not, I will check that soon ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [December 29, 2021, 8:04pm UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/11 "2021-12-29T20:04:36Z")
Just to make things more complicated: Before you commit to turning HT off in the BIOS, you might run some benchmarks first. That is, there are many processes running on a typical Linux install other than your application. Some of these will need the CPU from time to time (e.g., kernel threads, device drivers, including the one for your interconnect, the PBS MoM herself, timers, …). If you run with HT off and run as many application threads per host as you have cores, then every time one of these other threads needs the CPU, the application thread on that CPU gets completely suspended. On the other hand, with HT enabled, the application thread just runs a little slower for a time while the system process uses the other thread for that core. Which way is better is very application dependent. So, to tune your system for StarCCM+, you could come up with a representative run that takes about 10 minutes (adjust the number of iteration steps to get that time). Then run that job three or four times under each of the following conditions: - HT enabled in the BIOS, ncpus/mpiprocs = # of cores. - HT disabled in the BIOS, ncpus/mpiprocs = # of cores. - HT disabled in the BIOS, ncpus/mpiprocs = (# of cores - 1) The last one sets aside a core for systemy threads. I’m assuming you run the nodes in exclusive mode, so the benchmark of interest is job walltime. ---
**Author:** ![zhangyi2012](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zhangyi2012/32/456_2.png) [@zhangyi2012](https://community.openpbs.org/u/zhangyi2012)\ **Post date:** [December 30, 2021, 6:05am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/12 "2021-12-30T06:05:08Z")
Thank you very much! I learned a lot from you. In my case, there is only one job at a time, which is equal to exclusive mode I think. The “# of cores” of the 3 cases above are all num of real cores (32), not HT cores(64), is it right? ---
**Author:** ![dtalcott](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/dtalcott/32/410_2.png) [@dtalcott](https://community.openpbs.org/u/dtalcott)\ **Post date:** [December 30, 2021, 5:14pm UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/14 "2021-12-30T17:14:38Z")
> [@zhangyi2012](#): > > In my case, there is only one job at a time, which is equal to exclusive mode I think. > The “# of cores” of the 3 cases above are all num of real cores (32), not HT cores(64), is it right? Correct. I usually distinguish between a core and the (usually) two execution threads that can run simultaneously on the core. Note that in the third test case, you might need to tell StarCCM+ that there are fewer ranks than in the other cases. I don’t know if it figures that out on its own. ---
**Author:** ![zhangyi2012](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zhangyi2012/32/456_2.png) [@zhangyi2012](https://community.openpbs.org/u/zhangyi2012)\ **Post date:** [December 31, 2021, 1:28am UTC](https://community.openpbs.org/t/how-to-set-all-cpu-cores-num-for-a-job-directly-no-care-hosts-num/2911/15 "2021-12-31T01:28:13Z")
Thank you for your reply. I think starccm can’t figure out ranks on its own, what I need to do is set ncpus and mpiprocess in pbs script. I will share the test result to you when the test done. Thank you!