Integrating LVM with Hadoop

Hello everyone here is my new article on Integration of Logical

Volume management with the Storage of data node in Hadoop.

What is Logical Volume Management(LVM)?

In Linux, Logical Volume Manager is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume.

LVM Architecture

• To create an LVM logical volume, the physical volumes are combined into a volume group (VG). This creates a pool of disk space out of which LVM logical volumes (LVs) can be allocated. This process is analogous to the way in which disks are divided into partitions. A logical volume is used by file systems and applications (such as databases).

✍️✍️Task Description✍️✍️

🌀 7.1: Elasticity Task

🔅Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

• Let’s Attach a volume to your instance.
• Now, let’s check if the volume is attached or not :

• So we attached two volumes of 10 Gib and 20 Gib
• Now let’s create the Physical volumes(PV) of both volumes we attached.

First one :

pvcreate /dev/sdb
pvdisplay /dev/sdb

Second one :

pvcreate /dev/sdc
pvdisplay /dev/sdc

• We have To create a Volume Group(VG) of both the physical volumes

vgcreate iiecvg/dev/sdb /dev/sdc
vgdisplay iiecvg

• Let’s create a logical volume(LV) of 15Gib from the above volume group.

lvcreate --size 15G--name mylv1 iiecvg
lvdisplay /dev/iiecvg/mylv1

After creating the Logical volume we can use it as storage for the Datanode.

• To use it as a Storage for DataNode follow the following steps.

Step 1:Create a folder in the root folder.

mkdir /datanode

Step 2: Mount the above folder to the logical volume.

mount /dev/iiecvg/mylv1  /datanode

Successfully Mounted

After mounting the folder configure it in hdfs-site.xml file.

🔅Increase or Decrease the Size of Static Partition in Linux.

• As we configured with 30 GiB of the logical volume.

Let’s increase the size of the logical volume :

Step 1: Extend it

lvextend --size +10G /dev/iiecvg/mylv1

Step 2: Resize it

resize2fs /dev/iiecvg/mylv1

The storage is Extended successfully on the fly.

🔅!! Automating LVM Partition using Python-Script !!

press 1

press 5

Press 6

Press 2

Press 3

Press 7

Press 4

Press 8

Press 4