NFS mounted drive for nodes

Users/Site Administrators
1

I have installed OpenPBS version = 20.0.0 on Ubuntu 18.04, with 3 GPU servers configured as nodes
when a user submits a job to PBS it assigns the job to any one of the 3 GPU servers. Same set of users are present in all the 3 nodes. Users need data (usually 10GB or more) in their respective home directories to run a model training job. The problem seems to be keeping the same set of data in each of the 3 nodes so that the job can be executed by PBS, in any node.

Do you suggest to have a single NFS mounted drive in all the 3 nodes, so that the data needs to be copied only once and the same data is available in all the 3 nodes ? or is there a solution when we use only local hard disks specific to each node ?

2
  • if the all these 3 nodes needs to access the same data , then NFS would be slow down the acces ( with increasing network traffic)
  • it is better to host the data locally on those nodes
  • locally compute on the node and then only copy back the results to the NFS.

If data access is frequent and enormous, then NFS would be slow. However, it is true that you can host the data on the headnode and NFS mount it across the compute nodes.

This works even well, you can target respective nodes based on the type of the learning data avaialble on that compute node. Or can be used with generic learning data.