Mount Home Directory in AWS ParallelCluster 🗂

External filesystems can be mounted and used as home directories in AWS ParallelCluster. This has several advantages over the default, which is an EBS volume on the head node /home shared via NFSv4 to the compute nodes.

• Home directories can be persisted after cluster deletion, saving data and allowing users to reproduce the same environment
• Home directories can be mounted on multiple clusters, allowing users to have the same filesystem between different clusters
• Reduce dependency on HeadNode, this allows you to size down the HeadNode since it’s no longer serving critical traffic i.e. ~/.ssh/ directory
• Use a filesystem such as EFS that can dynamically expand

See #2441.

Setup

1. In this guide, I’ll assume you already have AWS ParallelCluster Manager setup, if you don’t follow the instructions on hpcworkshops.com to get started.

2. Setup a cluster with an external filesystem mount /shared, Note this can’t interfere with the default home directory, /home, which is required for parallelcluster.

   HeadNode:
   InstanceType: t2.micro
   Ssh:
       KeyName: keypair
   Networking:
       SubnetId: subnet-123456789
   LocalStorage:
       RootVolume:
       VolumeType: gp3
   Iam:
       AdditionalIamPolicies:
       - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
   Scheduling:
   Scheduler: slurm
   SlurmQueues:
       - Name: queue0
       ComputeResources:
           - Name: queue0-t2-micro
           MinCount: 0
           MaxCount: 4
           InstanceType: t2.micro
       Networking:
           SubnetIds:
           - subnet-123456789
       ComputeSettings:
           LocalStorage:
           RootVolume:
               VolumeType: gp3
   Region: us-east-2
   Image:
   Os: alinux2
   SharedStorage:
   - Name: Efs0
       StorageType: Efs
       MountDir: /shared
       EfsSettings:
       ThroughputMode: bursting
   

3. After the cluster is created, connect to the cluster with SSM:

   

   Then run the following commands to switch ec2-user to /shared/ec2-user:

   exit # go back to ssm-user
   sudo su # switch to root
   usermod -d /shared/ec2-user -m ec2-user
   

Test

Now that we’ve switched the home directory, we can log out and connect again via SSM and test that it worked:

You’ll see the new home directory is /shared/ec2-user, which contains the contents of /home/ec2-user.

Multi-User (Posix)

If you’re using a multi-user environment, make sure to specify the home directory upon user creation:

useradd -m -d /shared/$USER $USER

Since /shared/ec2-user is already mounted to all the compute nodes, you simply need to create the user on the compute node and point it at the right home directory:

useradd -d /shared/$USERNAME -u $USERID $USERNAME

See Multi-User Setup for more details.

Multi-User (Active Directory)

If you’ve setup your cluster with active directory, it’s even easier. You can provide a flag in the DirectoryServices/AdditionalSssdConfigs section of your config to specify the default home directory, for example:

AdditionalSssdConfigs:
  override_homedir = /shared/%u

In ParallelCluster Manager:

This sets the user’s home directory to /shared/<username>. Read more about it here.