Integrating LVM with Hadoop

Hello everyone here is my new article on Integration of Logical

Volume management with the Storage of data node in Hadoop.

What is Logical Volume Management(LVM)?

In Linux, Logical Volume Manager is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume.

LVM Architecture

  • To create an LVM logical volume, the physical volumes are combined into a volume group (VG). This creates a pool of disk space out of which LVM logical volumes (LVs) can be allocated. This process is analogous to the way in which disks are divided into partitions. A logical volume is used by file systems and applications (such as databases).

✍️✍️Task Description✍️✍️

🌀 7.1: Elasticity Task

🔅Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

  • Let’s Attach a volume to your instance.
  • Now, let’s check if the volume is attached or not :
  • So we attached two volumes of 10 Gib and 20 Gib
  • Now let’s create the Physical volumes(PV) of both volumes we attached.

First one :

pvcreate /dev/sdbpvdisplay /dev/sdb

Second one :

pvcreate /dev/sdcpvdisplay /dev/sdc
  • We have To create a Volume Group(VG) of both the physical volumes
vgcreate iiecvg/dev/sdb /dev/sdcvgdisplay iiecvg
  • Let’s create a logical volume(LV) of 15Gib from the above volume group.
lvcreate --size 15G--name mylv1 iiecvglvdisplay /dev/iiecvg/mylv1

After creating the Logical volume we can use it as storage for the Datanode.

  • To use it as a Storage for DataNode follow the following steps.

Step 1:Create a folder in the root folder.

mkdir /datanode

Step 2: Mount the above folder to the logical volume.

mount /dev/iiecvg/mylv1  /datanode

Successfully Mounted

After mounting the folder configure it in hdfs-site.xml file.

🔅Increase or Decrease the Size of Static Partition in Linux.

  • As we configured with 30 GiB of the logical volume.

Let’s increase the size of the logical volume :

Step 1: Extend it

lvextend --size +10G /dev/iiecvg/mylv1

Step 2: Resize it

resize2fs /dev/iiecvg/mylv1

🔅!! Automating LVM Partition using Python-Script !!

press 1

press 5

Press 6

Press 2

Press 3

Press 7

Press 4

Press 8

Press 4




Love podcasts or audiobooks? Learn on the go with our new app.

Getting Started With Kai

How to pass the Snowflake SnowPro Core Certification?

Basic web app with Django Rest Framework and JQuery-ajax

Creating Video Chat App using Python

Longitudinal & Lateral Control for Autonomous Vehicles — CARLA Simulator

Mah mind’s telling me noooo…

Learn to Program Better with The Weekly Spot Drill

The smallest, fastest, most secure Python ever

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Himanshi Kabra

Himanshi Kabra

More from Medium

CS371p Spring 2022 Relena Lai

Writing Active Record Database Validation Methods

Series Preview: @ Beloit (5/17–5/22)

Handling Dynamic Objects in CrateDB