How to Provide Elasticity to the DataNode Storage by Integrating through the LVM and Hadoop💥

Hello Enthusiasts….!!😃

🌟In this article, We are going to discuss how we can integrate LVM with Hadoop in order to provide elasticity to the DataNode Storage. Here, Elasticity means whenever we have a need to change the storage provided by DataNode to the Hadoop-Cluster, we can achieve it, without shutting down or restarting, or stopping the service of DataNode, we can directly increase or decrease the size of DataNode by using some commands. It leads to less consumption of time and provides high speed.

✨Task Description —

7.1: Elasticity Task

◆Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

◆Increase or Decrease the Size of Static Partition in Linux

✍Prerequisite :
●Basic knowledge of Hadoop
●Knowledge of Linux Partition and LVM

Logical Volume Management(LVM):

LVM is a tool that includes allocating disks, striping, mirroring, and resizing the logical volume. In this, a hard drive (or disk) is first allocated to the physical volume then, this physical volume can be placed on other block devices i.e. volume group which might span one or more drives. At last, physical volumes are combined into logical volumes.

Hadoop :
Hadoop is an open-source framework that allows to storage and process of big data in a distributed storage environment across a cluster of computers. It provides massive storage for any kind of data, enormous processing power, and the ability to handle virtually limitless concurrent tasks or jobs.

What is Elasticity?

Elasticity is the concept which we can use to increase or decrease the volume of Hadoop Data Node. Hadoop data nodes shared storage can’t be static so LVM is used to make it dynamic.

Now let’s begin our practice with a simple Hadoop cluster of one data node only.

Step-1) Create a Hadoop Cluster.

• Configure NameNode -

Install Java and Hadoop in NameNode System then configure the hdfs-site.xml file-

Configure the core-site.xml file-

Now format the NameNode and start the Service-

• Configure Data Node -

Install Java and Hadoop in DataNode System then configure the hdfs-site.xml file-

Configure the core-site.xml file-

Now start the Service of DataNode

hadoop dfsadmin -report

Here, we can see that DataNode is providing all 50GB storage to NameNode.

Step-2) Now Create Partition: Attach Volume to DataNode System -

Create a volume of 10GB-

Here, I have attached both the volume to my DataNode.

• Now Check that volume (Hard Disk) is attached or not by using the ‘fdisk -l’ command-

fdisk -l

Here two-volume of 10GB and 9GB have attached.

Step-3) Now create Physical Volume for both Volumes-

• Physical volume for the first volume-

• Physical volume for the second volume-

Step-4) Now create a Volume Group for both the physical volumes.

Here Volume Group of 18.99Gib is created.

Step-5) Create a Logical Volume of 5GB from the above volume group.

l

• Now format the Logical Volume-

• Now check that logical volume is mounted or not-

Step-6) Now update this directory to a hdfs-site.xml file of DataNode.

• Now Start the Service of DataNode.

• Now check the report of Hadoop cluster in NameNode-

Here, Now DataNode is only Contributing 5GB to NameNode.

Increase the Logical Volume:

• I am increasing the size of the logical volume by 2GB-

lvextend --size +2G /dev/V_Group/lv1

• Now We have to format 2Gb extra part, to format the Non-partition portion from the complete LV.

resize2fs /dev/V_Group/lv1

lvdisplay V_Group/lv1

• Now check the Hadoop cluster report in NameNode-

Here, we have extended the logical volume successfully.

• In the same way, we can also reduce the LV size.

Now check the size of logical volume-

lvdisplay V_Group/lv1

Here, LV size is reduced from 7GB to 3GB.

🌟Finally, DataNode Storage has been exceeded elastically through the LVM concept.

🎗️ TASK COMPLETED SUCCESSFULLY !!🎗️

THANKYOU FOR READING