How To Create Directory In Hdfs

In this article , We will learn how to create home directory in HDFS for a new user.

Every user should have home directory in HDFS if he/she wants to access HDFS. Some hadoop jobs use user's home directory to store intermediate/temporary data . Jobs will fail if no home directory for user.
On Local file system , user's home directory is created under /home directory and On HDFS, User's home directory is created under /user folder.

1) Create a user on local file system

First we need to create a user on local file system (i.e. Operating System) using useradd command. And user should be created on all nodes in the cluster.

The picture below shows new user nirupam is created and nirupam's home directory in local file system is /home/nirupam.

By default , user does not have a password , You can set password using passwd command if you want.

2) Create a directory in HDFS for new user.

We need to create a directory under /user in HDFS for new user. This directory needs to be created using hdfs user as hdfs user is super user for hadoop cluster.

The picture below shows a new directory is created for nirupam user under /user directory in HDFS.

3) Check the owner

As new directory is created by hdfs user, hdfs user will be the owner of the directory. We need to change the owner of this directory to new user.

The picture below shows owner of the /user/nirupam directory in HDFS.

4) Change the owner

Change the owner of new directory created in HDFS to new user created in local file system. chown command can be used to change the owner.

The picture below shows changing the owner of HDFS directory /user/nirupam to nirupam user from hdfs user.

5) Change the permissions

We need to change the permissions of this newly created so that no other users can have read,write and execute permissions except owner.

The picture below modifies permissions of /user/nirupam directory to 700 so that only own can have read, write and execute permissions.

6) Test the user's HDFS home directory.

We have successfully created home directory in HDFS for new user. We need to test it now.

We will try to upload a new file to HDFS without specifying destination directory. File will be uploaded to user's home directory if no destination is specified.

The picture below shows new file is uploaded to nirupam's user home directory as destination directory is not specified.