If the memory is not requested via qsub, then there will be default memory request set to the job by the server attribute or by the cgroup hook. If you are using cgroup hook, then the job would run within the boundaries of what has been requested for the job.
Refer: What would happen to a job if its memory usage exceeds its requested size? - #3 by sxy