# VMEM is not enforced **URL:** \ **Category:** Users/Site Administrators\ **Created:** [January 21, 2019, 8:25am UTC](https://community.openpbs.org/t/vmem-is-not-enforced/1413 "2019-01-21T08:25:47Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![tomoki](https://avatars.discourse-cdn.com/v4/letter/t/50afbb/32.png) [@tomoki](https://community.openpbs.org/u/tomoki)\ **Post date:** [January 21, 2019, 8:25am UTC](https://community.openpbs.org/t/vmem-is-not-enforced/1413/1 "2019-01-21T08:25:47Z")
Hi, I am testing the the vmem limit enforcement refer to 5.14.3.3 “Configuring Per-job Resource Limit Enforcement at Vnodes”. But, vmem is not killed forcibly. I had expected that the following message would be output. =\>\> PBS: job killed: vmem xxxxxxxx exceeded limit yyyyyy. However, the job ended normally. Why is the job not killed due to vmem limitation? 1. Running program (Allocate 1gb memory). 2. Running script (5bg per process). [test@sl02-sms ~]$ cat mallc.sh #!/bin/bash #PBS -N test #PBS -j oe 3. Running job. [test@sl02-sms ~]$ qsub -l pvmem=2gb,vmem=4gb mallc.sh 1443.sl02-sms [test@sl02-sms ~]$ !!! vmem was not killed by job limit. !!! [test@sl02-sms ~]$ cat test.o1443 resource limit information core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63668 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 16384 cpu time (seconds, -t) unlimited max user processes (-u) 63668 virtual memory (kbytes, -v) 2097152 file locks (-x) unlimited 1 process runnning 2 process runnning 3 process runnning 4 process runnning 5 process runnning memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 memory allocated size=268439536 [test@sl02-sms ~]$ qstat -f Job Id: 1443.sl02-sms Job\_Name = test Job\_Owner = test@sl02-sms resources\_used.cpupercent = 0 resources\_used.cput = 00:00:00 resources\_used.mem = 5124kb resources\_used.ncpus = 1 resources\_used.vmem = 249844kb resources\_used.walltime = 00:00:12 job\_state = R queue = workq server = sl02-sms Checkpoint = u ctime = Fri Jan 18 11:31:05 2019 Error\_Path = sl02-sms:/home/test/test.e1443 exec\_host = sl02-c001/0 exec\_vnode = (sl02-c001:ncpus=1) Hold\_Types = n Join\_Path = oe Keep\_Files = n Mail\_Points = a mtime = Fri Jan 18 11:31:05 2019 Output\_Path = sl02-sms:/home/test/test.o1443 Priority = 0 qtime = Fri Jan 18 11:31:05 2019 Rerunable = True Resource\_List.ncpus = 1 Resource\_List.nodect = 1 Resource\_List.place = pack Resource\_List.pvmem = 2gb Resource\_List.select = 1:ncpus=1:vmem=4gb Resource\_List.vmem = 4gb [root@sl02-c001 ~]# ps -aef | grep mallc |grep -v grep test 25586 25585 0 11:25 ? 00:00:00 ./mallc test 25587 25585 0 11:25 ? 00:00:00 ./mallc test 25588 25585 0 11:25 ? 00:00:00 ./mallc test 25589 25585 0 11:25 ? 00:00:00 ./mallc test 25590 25585 0 11:25 ? 00:00:00 ./mallc [root@sl02-c001 ~]# My environment is below. [test@sl02-sms ~]$ qsub --version pbs\_version = 14.1.2 [root@sl02-sms ~]# grep ^resources /var/spool/pbs/sched\_priv/sched\_config resources: “ncpus, mem, arch, host, vnode, netwins, aoe” [root@sl02-sms ~]# [test@sl02-sms ~]$ pbsnodes -av sl02-c001 Mom = sl02-c001.localdomain ntype = PBS state = free pcpus = 12 resources\_available.arch = linux resources\_available.host = sl02-c001 resources\_available.mem = 16363572kb resources\_available.ncpus = 12 resources\_available.vnode = sl02-c001 resources\_assigned.accelerator\_memory = 0kb resources\_assigned.dyna-license = 0 resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.netwins = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared sl02-c002 Mom = sl02-c002.localdomain ntype = PBS state = free pcpus = 12 resources\_available.arch = linux resources\_available.host = sl02-c002 resources\_available.mem = 16363572kb resources\_available.ncpus = 12 resources\_available.vnode = sl02-c002 resources\_assigned.accelerator\_memory = 0kb resources\_assigned.dyna-license = 0 resources\_assigned.mem = 0kb resources\_assigned.naccelerators = 0 resources\_assigned.ncpus = 0 resources\_assigned.netwins = 0 resources\_assigned.vmem = 0kb resv\_enable = True sharing = default\_shared [test@sl02-sms ~]$ [root@sl02-c001 ~]# cat /var/spool/pbs/mom\_priv/config $clienthost sl02-sms $usecp \*:/home /home $restrict\_user\_maxsysid 999 $enforce mem ---
**Author:** ![alexis.cousein](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.openpbs.org/alexis.cousein/32/265_2.png) [@alexis.cousein](https://community.openpbs.org/u/alexis.cousein)\ **Post date:** [February 22, 2022, 8:51am UTC](https://community.openpbs.org/t/vmem-is-not-enforced/1413/2 "2022-02-22T08:51:37Z")
Where in the documentation did you see that “vmem” would be enforced? vmem is _the sum of_ the virtual size of all processes. This is not something “to be enforced”, since each process has its own 64-bit address space (and we do not know how much is shared with other processes, which is often a lot in multiprocessing libraries like OpenMP or MPI). It is _measured_, that is all. The code assumes that the user wil know how to interpret this (after knowing how the application allocates things and maps them into virtual space). By default (and in some contexts it’s a bug, so there is a bug open on that) MoM will set RLIMIT\_AS to the vmem limit, so if a SINGLE process tries to map more, it will USUALLY not succeed, but really, you want to use pvmem for that. And you do, but that neuters what vmem does by default. > [@tomoki](#): > > Running script (5bg per process). That is not “5gb per process” – that is 1gb per process with 5 processes. But there is no mechanism to limit the sum of the virtual size of _all_ processes because that is usually meaningless. What exactly is the use case? When using the cgroup hook the semantics of “vmem” changes completely, and is used to denote physical memory plus swap usage of the job (including any files mapped into memory by the job) on the host the chunks are mapped onto. And then the kernel will indeed enforce it (first by squeezing out any files mapped in by the program out of the page cache, and finally by refusing to allocate memory and signaling to the hook the job should be killed).