# Unable to submit the job on compute node(node01) **URL:** \ **Category:** Users/Site Administrators\ **Created:** [March 4, 2020, 9:21am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039 "2020-03-04T09:21:06Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![zainul1114](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zainul1114/32/263_2.png) [@zainul1114](https://community.openpbs.org/u/zainul1114)\ **Post date:** [March 4, 2020, 9:21am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/1 "2020-03-04T09:21:06Z")
Hi, I have installed Pbspro 19.1.1 on Master(pbs server) and node01 In master i can able to submit the job and i am getting STDIN.o3 STDIN.e3 files from pbsdata user(pbsdata user is in master machine) qsub -l select=1:ncpus=1:mem=100mb:host=master – /bin/sleep 10 And, When i submit the job to node01 qsub -l select=1:ncpus=1:mem=100mb:host=node01 – /bin/sleep 10 i am getting [pbsdata@master ~]$ qstat -ans master: Req’d Req’d Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time * * * 7.master pbsdata workq STDIN – 1 1 100mb – H – – job held, too many failed attempts to run my logs: cat /var/spool/pbs/mom\_logs/20200304 |grep 7.master 03/04/2020 12:57:44;0028;pbs\_mom;Job;7.master;No Password Entry for User pbsdata 03/04/2020 12:57:44;0008;pbs\_mom;Job;7.master;kill\_job 03/04/2020 12:57:44;0100;pbs\_mom;Job;7.master;node01 cput= 0:00:00 mem=0kb 03/04/2020 12:57:44;0008;pbs\_mom;Job;7.master;no active tasks 03/04/2020 12:57:44;0100;pbs\_mom;Job;7.master;Obit sent 03/04/2020 12:57:44;0080;pbs\_mom;Job;7.master;delete job request received 03/04/2020 12:57:44;0008;pbs\_mom;Job;7.master;kill\_job [pbsdata@master ~]$ pbsnodes -a master Mom = master ntype = PBS state = free pcpus = 2 resources\_available.arch = linux resources\_available.host = master resources\_available.mem = 2046864kb resources\_available.ncpus = 2 resources\_available.vnode = master resources\_assigned.accelerator\_memory = 0kb resources\_assigned.hbmem = 0kb resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared last\_state\_change\_time = Wed Mar 4 13:10:28 2020 last\_used\_time = Wed Mar 4 14:24:20 2020 node01 Mom = node01 ntype = PBS state = free pcpus = 2 resources\_available.arch = linux resources\_available.host = node01 resources\_available.mem = 2046864kb resources\_available.ncpus = 2 resources\_available.vnode = node01 resources\_assigned.accelerator\_memory = 0kb resources\_assigned.hbmem = 0kb resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared last\_state\_change\_time = Wed Mar 4 13:10:28 2020 last\_used\_time = Wed Mar 4 13:21:27 2020 [pbsdata@master ~]$ [pbsdata@master ~]$ cat /etc/pbs.conf PBS\_EXEC=/share/apps/platform/pbs PBS\_HOME=/var/spool/pbs PBS\_SERVER=master PBS\_START\_SERVER=1 PBS\_START\_SCHED=1 PBS\_START\_COMM=1 PBS\_START\_MOM=1 PBS\_CORE\_LIMIT=unlimited PBS\_RCP=/bin/false PBS\_SCP=/usr/bin/scp PBS\_RSHCOMMAND=/usr/bin/ssh [root@node01 ~]# cat /etc/pbs.conf PBS\_EXEC=/opt/pbs PBS\_HOME=/var/spool/pbs PBS\_SERVER=master PBS\_START\_SERVER=0 PBS\_START\_SCHED=0 PBS\_START\_COMM=0 PBS\_START\_MOM=1 PBS\_CORE\_LIMIT=unlimited PBS\_RCP=/bin/false PBS\_SCP=/usr/bin/scp PBS\_RSHCOMMAND=/usr/bin/ssh Please help me to submit the job successfully from pbsdata user(on master machine) to node01. Regards, Zain ---
**Author:** ![wgy](https://avatars.discourse-cdn.com/v4/letter/w/e47c2d/32.png) [@wgy](https://community.openpbs.org/u/wgy)\ **Post date:** [March 4, 2020, 2:08pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/2 "2020-03-04T14:08:00Z")
You got to have pbsdata user on node01. Henry Wu|吴光宇 ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [March 4, 2020, 4:28pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/3 "2020-03-04T16:28:08Z")
+1 @wgy > [@zainul1114](#): > > 03/04/2020 12:57:44;0028;pbs\_mom;Job;7.master;No Password Entry for User pbsdata can you ssh login into node01 as “pbsdata” ? and does the home directory for pbsdata exists ? ---
**Author:** ![cherryppju](https://avatars.discourse-cdn.com/v4/letter/c/b5a626/32.png) [@cherryppju](https://community.openpbs.org/u/cherryppju)\ **Post date:** [March 4, 2020, 8:28pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/4 "2020-03-04T20:28:17Z")
+1 @wgy @adarsh By default, the job owner is pbsdata but you can specify -u option to use different user on compute nodes. > ``` > -u user_list > > ``` ``` List of usernames. Job is run under a username from this list. Sets job's User_List attribute to user_list. Default: job owner (username on submit host.) Format of user_list: user[@host][,user@host ...] ``` ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [March 4, 2020, 10:15pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/5 "2020-03-04T22:15:16Z")
Please follow this link > [@Submit job as different user?](http://community.openpbs.org/t/submit-job-as-different-user/1630): > > I’m working on a server to allow users to submit jobs to the cluster without using qsub directly (we authenticate the request, and if valid use qsub to submit the job to run). From the qsub docs, it looks like the -u option can be used to submit a job as a different user than the currently logged in user. I’m struggling a bit to get this to work though in practice - it’s not clear to me what configuration is needed. I’ve gotten things working by adding serverhost serveruser to the ~/.rhosts fil… ~/.rhosts should be populated with host(s) and otherusername(s) , if userA wants to submit job(s) as otheruser(s) ---
**Author:** ![zainul1114](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zainul1114/32/263_2.png) [@zainul1114](https://community.openpbs.org/u/zainul1114)\ **Post date:** [March 5, 2020, 4:40am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/6 "2020-03-05T04:40:18Z")
> [@zainul1114](#): > > No Password Entry for User Thank you your response Adarsh, pbsdata user home directory not exists on node01 and i have created home dir and passwordless ssh between pbsdata users of master and node01. Now, i am able to submit the jobs to node01. finally, can you explain me, that the PBS pro cluster needs to have the same user on all the VMs in this cluster? For Ex: Master (PBS Server/ Head node/login node) Node01 (computenode) Node02 (computenode) Here all the nodes should have test users with passwordless ssh b/w test users. Regards, Zain ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [March 5, 2020, 11:53am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/7 "2020-03-05T11:53:08Z")
In any cluster environment, the user needs to have seamless SSH access with HostKeyChecking disabled or approved. across the nodes of the cluster. Basically, the no password should be asked when ssh’ing - master / headnode to compute node(s) - compute node(s) to headnode - compute node(s) to compute node(s) Usually, in a cluster environment the USER HOME DIRECTORY would be common (mounted across) across all the compute nodes. Also , NIS / PBIS / others might have been used for storing client/server/user details. ---
**Author:** ![zainul1114](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zainul1114/32/263_2.png) [@zainul1114](https://community.openpbs.org/u/zainul1114)\ **Post date:** [March 5, 2020, 1:26pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/8 "2020-03-05T13:26:28Z")
Thanks for clarifying ssh across the nodes. Please guide me to restrict the cores based on user/group else all users when they submitting the job. For example: Master - 24c Node01 - 24c total 48c IT group use only 10cores Bio group use only 20cores Chem group use only 10cores how can i restrict with their default queues, this scenarios?? if incase i want to use (top priority) 40c then how can submit the job? Please guide me to configure this scenarious. Regards, Zain ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [March 6, 2020, 9:02am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/9 "2020-03-06T09:02:23Z")
Please refer this documentation: > **[PBS19.2.3\_BigBook.pdf](https://www.altair.com/pdfs/pbsworks/PBS19.2.3_BigBook.pdf)** > > 22.77 MB and this section **5.15.1.9.ii Examples of Setting Server and Queue Limits** This will cover all your use cases . Are these ( IT, Bio, Chem) Linux groups ? ---
**Author:** ![zainul1114](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/zainul1114/32/263_2.png) [@zainul1114](https://community.openpbs.org/u/zainul1114)\ **Post date:** [March 6, 2020, 9:16am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/10 "2020-03-06T09:16:04Z")
Thanks for sharing link with session Adarsh. I will check and get back to you. Regards, Zain ---
**Author:** ![vincent718](https://avatars.discourse-cdn.com/v4/letter/v/f9ae1b/32.png) [@vincent718](https://community.openpbs.org/u/vincent718)\ **Post date:** [February 11, 2021, 2:59pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/11 "2021-02-11T14:59:11Z")
Hi @adarsh I reinstalled our compute node and installed PBS. After which now all submitted jobs are held . qstat -answ and qstat -f all says job held, too many failed attempts to run I probed deeper and below are some lines of the logs 02/11/2021 15:26:02;0008;pbs\_mom;Job;111852.master1.local;kill\_job 02/11/2021 15:26:02;0100;pbs\_mom;Job;111852.master1.local;compute-0-1 cput= 0:00:00 mem=0kb 02/11/2021 15:26:02;0008;pbs\_mom;Job;111852.master1.local;no active tasks 02/11/2021 15:26:02;0100;pbs\_mom;Job;111852.master1.local;Obit sent 02/11/2021 15:26:02;0080;pbs\_mom;Job;111852.master1.local;delete job request received 02/11/2021 15:26:02;0008;pbs\_mom;Job;111852.master1.local;kill\_job 02/11/2021 15:26:02;0028;pbs\_mom;Job;111852.master1.local;No Password Entry for User = jkzabee 02/11/2021 15:26:02;0008;pbs\_mom;Job;111852.master1.local;kill\_job 02/11/2021 15:26:02;0100;pbs\_mom;Job;111852.master1.local;compute-0-1 cput= 0:00:00 mem=0kb 02/11/2021 15:26:02;0008;pbs\_mom;Job;111852.master1.local;no active tasks 02/11/2021 15:26:02;0100;pbs\_mom;Job;111852.master1.local;Obit sent 02/11/2021 15:26:02;0080;pbs\_mom;Job;111852.master1.local;delete job request received 02/11/2021 15:26:02;0008;pbs\_mom;Job;111852.master1.local;kill\_job Please advice Thanks ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [February 11, 2021, 7:08pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/12 "2021-02-11T19:08:38Z")
> [@vincent718](#): > > 02/11/2021 15:26:02;0028;pbs\_mom;Job;111852.master1.local;No Password Entry for User = jkzabee Please check the compute for 1. whether user home direct exists 2. can you login as that user on that compute node without any issues (passwordless ssh is working both ways server to node , node to server) 3. check whether you can login to that compute node as that user with a password These hints would help to resolve. Thank you for sharing ---
**Author:** ![vincent718](https://avatars.discourse-cdn.com/v4/letter/v/f9ae1b/32.png) [@vincent718](https://community.openpbs.org/u/vincent718)\ **Post date:** [February 11, 2021, 9:57pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/13 "2021-02-11T21:57:10Z")
Hi @adarsh. When I ssh to compute node it asks for a password. I also realized that the compute node has no records of the users. Can you advice how to update the compute node with the users account. For the home directory we use a shared directory which is mounted on all nodes ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [February 12, 2021, 8:40am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/14 "2021-02-12T08:40:54Z")
> [@vincent718](#): > > I also realized that the compute node has no records of the users. Can you advice who to update the compute node with the users. > For the home directory we use a shared directory which is mounted on all nodes 1. manually create users with the same password on the compute nodes and reference to the mounted home directory in the /etc/passwd file – after doing this make sure you can login and get the home directory at the prompt 2. Using NIS or PBIS or LDAP ---
**Author:** ![vincent718](https://avatars.discourse-cdn.com/v4/letter/v/f9ae1b/32.png) [@vincent718](https://community.openpbs.org/u/vincent718)\ **Post date:** [February 12, 2021, 9:56am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/15 "2021-02-12T09:56:33Z")
Hi @adarsh. I have created both the user account on both login and compute. Now i can ssh to the compute-node with out issues. But the job being in the held state persists. It first gets queued and qstat -f says Not Running: Insufficient amount of resource: host [quote=“adarsh, post:14, topic:2039”] Using NIS or PBIS or LDAP [/quote] Hi We are not using any of those three we create users with the conventional linux useradd ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.openpbs.org/u/adarsh)\ **Post date:** [February 12, 2021, 4:41pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/16 "2021-02-12T16:41:31Z")
Please share the qstat -fx \< of that job id\> and pbsnodes -av output. The job request could not be satisfied by any of the available nodes, hence the scheduler message. ---
**Author:** ![vincent718](https://avatars.discourse-cdn.com/v4/letter/v/f9ae1b/32.png) [@vincent718](https://community.openpbs.org/u/vincent718)\ **Post date:** [February 12, 2021, 10:25pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/17 "2021-02-12T22:25:52Z")
Hello Adarsh, Thanks for the help. I was able to finally get PBS to run the jobs. What I found out was that the user accounts did not exist on compute nodes. Manually creating the same user on the compute nodes did not work. What worked was to copy /etc/passwd , from the masternode to all the compute-nodes. I also reinstalled PBS on the compute nodes as well. Thanks once again ---
**Author:** ![watzinki](https://avatars.discourse-cdn.com/v4/letter/w/f04885/32.png) [@watzinki](https://community.openpbs.org/u/watzinki)\ **Post date:** [March 14, 2022, 7:50pm UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/18 "2022-03-14T19:50:35Z")
Dear @vincent718 , I am having a similar issue.So you copy passwd file in head note to slave nodes and replaced the passwd file with the ones in the slaves nodes? ---
**Author:** ![vincent718](https://avatars.discourse-cdn.com/v4/letter/v/f9ae1b/32.png) [@vincent718](https://community.openpbs.org/u/vincent718)\ **Post date:** [March 18, 2022, 9:42am UTC](https://community.openpbs.org/t/unable-to-submit-the-job-on-compute-node-node01/2039/19 "2022-03-18T09:42:08Z")
Yes. That is what I did. You should also copy the group file