how to correctly 'extend' a linux device mapper partition?

Question

I am missing some concept with linux disk management, I have free space on a single physical volume that I want to to extend an ext4 file-system with, which is a logical volume.

I look at fdisk -l - it is a GPT (warnings)

Disk /dev/sda: 299.4 GB, 299439751168 bytes
255 heads, 63 sectors/track, 36404 cylinders, total 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1   584843263   292421631+  ee  GPT

I look at the physical volumes:

root@node-29:/home# pvs
  PV         VG      Fmt  Attr PSize  PFree
  /dev/sda4  os      lvm2 a--  62.00g 4.00m
  /dev/sda5  logs    lvm2 a--  10.00g 4.00m
  /dev/sda6  mysql   lvm2 a--  20.00g 4.00m
  /dev/sda7  narcine lvm2 a--  11.00g 4.00m

this adds up to 93GB, the is a swap partition and a few others but, I have lots of free space

I look at parted, since they are GPT partitions - I want to see the free space available:

root@node-29:/home# parted /dev/sda print free
Model: DELL PERC H710 (scsi)
Disk /dev/sda: 299GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
        17.4kB  1049kB  1031kB  Free Space
 1      1049kB  26.2MB  25.2MB               primary  bios_grub
        26.2MB  27.3MB  1049kB  Free Space
 2      27.3MB  237MB   210MB                primary
        237MB   238MB   1049kB  Free Space
 3      238MB   448MB   210MB   ext2         primary
        448MB   449MB   1049kB  Free Space
 4      449MB   67.1GB  66.6GB               primary
        67.1GB  67.1GB  1049kB  Free Space
 5      67.1GB  77.9GB  10.8GB               primary
        77.9GB  77.9GB  1049kB  Free Space
 6      77.9GB  99.4GB  21.5GB               primary
        99.4GB  99.4GB  1049kB  Free Space
 7      99.4GB  111GB   11.9GB               primary
        111GB   111GB   1049kB  Free Space
 8      111GB   111GB   21.0MB  ext2         primary
        111GB   299GB   188GB   Free Space

there's a little free space on all the volumes but there is the space I want on the last line

Number  Start   End     Size    File system  Name     Flags
        111GB   299GB   188GB   Free Space

In LVM I see the devices are in /dev/mapper, I look at dmsetup

root@node-29:/home# dmsetup info
Name:              narcine-nartemp
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      252, 0
Number of targets: 1
UUID: LVM-eDmr02vsptbjvAdvX9c7VZGuI3drAkuMqYAIwCAy6EEZ2GTAopLlD96o6CmOtszP

Name:              os-swap
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        2
Event number:      0
Major, minor:      252, 4
Number of targets: 1
UUID: LVM-hMIdqpNc1W6paxT044lNpBcPUfWGA2kESL4f6dB9OJu14mKzLvnOzKMeNM6zV4SK

Name:              os-root
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      252, 3
Number of targets: 1
UUID: LVM-hMIdqpNc1W6paxT044lNpBcPUfWGA2kEcIBDlsTce6uqieEfh6ehzYfLxJwaIoEe

Name:              mysql-root
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      252, 1
Number of targets: 1
UUID: LVM-rYCHA1YXEPa6jDTS8NIHRvTXllf7jeNhT5d7pulT4efAq4TMll6ndWVuyiDYgGbs

Name:              logs-log
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      252, 2
Number of targets: 1
UUID: LVM-npxUm7C9dQX7fIrLeAGyfJxrDWKzmb9rVfJS8FEspobIPK8bnuOnDzLNdQtsY4jE

how should I proceed with extending? what is the order (parted, lvm, dmsetup?) - parted tells me the device partition is loop - what does that mean?

Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/os-root: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  53.7GB  53.7GB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/os-swap: 12.9GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system     Flags
 1      0.00B  12.9GB  12.9GB  linux-swap(v1)


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/logs-log: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  10.7GB  10.7GB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mysql-root: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  21.5GB  21.5GB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/narcine-nartmp: 11.8GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  11.8GB  11.8GB  xfs

I expected pvsan -vv to show me free PEfree on /dev/sda - why doesn't it?

what's throwing me is the command dmcreate it looks like with this, there is no need to create "traditional" partitions - is that correct?

thanks.

Solution

How to extend the LVM

cfdisk /dev/sda                        # create new partition, using all free space
pvcreate /dev/sdaX                     # initialize partition for use with LVM
vgdisplay                              # to find VG name
vgextend /dev/vgname /dev/sdaX         # this extends the volume group
lvextend -l +100%FREE /dev/vgname/root # this extends the LVM
resize2fs /dev/vgname/root             # this extends the filesystem

Explanation of LVM

LVM doesn't care about partitions. A LVM has the following hierachy:

1. Filesystem
2. Logical Volumes
3. Volume Groups
4. Physical Volumes
5. (Partitions)
6. Hardware

Lets go from the bottom up.

At the bottom you have the hardware. Big surprise. On top of that you have PVs. Now here is where it becomes confusing. You can either have a PV be the drive itself or a partition. LVM does not need partitions. You can add raw block devices as PVs. However, many people create partitions anyways. There are many reasons for this. Backwards compatibility with tools or people that expect partitions, for example. If a sysadmin does not know the layout and sees an 'empty' disk, he might think the disk is empty, although it is a PV! So thats the reason why you sometimes use partitions as PVs.

This is what you see in your example, whoever set the server up created one partition per VG, apparently.

Next up are the volume groups. A VG is one or multiple PVs. This is the container that holds all the stuff that comes afterwards. Since PVs can be disks, virtual disks from RAID controllers, partitions, etc., VGs can span any number of these things.

On top of VGs you have LVs. This is what you actually put your filesystems on top of. You can look at a LV kind of like a partition. You can find them here:

/dev/VGName/LVname

So a LV always belongs to one VG, but you can have many LVs per VG.

On top of the LV, finally, you put your filesystem.

Why loop

The loop conundrum: there is no loop device! Parted cannot find a partition table on the LVM (as it should be), so it just displays 'loop' instead.

Comments

[Sum1sAdmin]

thanks mzhaase,I am trying to understand the logical disk layout a bit more, cfdisk won't list the current partitions as they are GPT, I suspect that LVM is creating the device mapper entries (?), but the device mapper commands and the output of parted (above) are confusing - for example fdisk, cfdisk tell me to use parted as the partitions are GPT, and parted lists them as loop devices & linux device mapper model

[mzhaase]

@Sum1sAdmin ok, added a short LVM explanation, hope that clears things up.

[Sum1sAdmin]

thanks again for your input, I understand lvm - physical, logical, volume groups etc, - but it seems that these volumes are loop devices - such as an ISO would be, LVM can't do that. 'Model: Linux device-mapper (linear) (dm) Disk /dev/mapper/narcine-nartmp: 11.8GB Sector size (logical/physical): 512B/512B Partition Table: loop'

[mzhaase]

@Sum1sAdmin Ah, now I understand. The LVM does not have a partition table, hence parted is showing 'loop'. It does that when it cannot find a partition table. See: bugzilla.redhat.com/show_bug.cgi?id=741593

[Sum1sAdmin]

that's a 5 year old bug report for rhel, I am interested in an answer that explains how these devices where created, why there are loop devices and how to extend (I understand that LVM will be finally used to extend the volume)

[Tero Kilkanen]

Partition table type loop is the description Parted uses when there is no partition table on a device. And this is the way it should be on Logical Volumes, they simply have filesystems. It does not refer to loop devices in any way. The partition table types (labels) used in Parted are described in gnu.org/software/parted/manual/html_chapter/parted_2.html . Otherwise mzhaase's procedure for the volume extension looks correct. However, performance will not be optimal, because there will be a gap between original and added space.

[Joshua Huber]

Another really good reason to put your PV on a partition instead of a raw device: you may eventually need to replace disks, especially in an array, and the exact same disk model might not be available in 5 years, manufacturers play number tricks the with meanings of "Giga" and "Tera", so one manufacturer's 3 TB disk might be a few MB larger (or more importantly smaller) than your original. If you partition the disk with a single GPT partition, starting at 1MB and leave 100MB free at end, you'll be aligned for sure and will have a small margin in case a replacement drive is trivially smaller.

[mzhaase]

@Sum1sAdmin as mentioned in that bug report, just because parted is reporting loop devices does not mean there are loop devices. Parted also reports loop when it cannot find a partition table, as is the case with lvm.

[Naveed Abbas]

@JoshuaHuber Both your and mzhaase's arguments only apply to physical HDDs, not to vmWare/HyperV/virtual disks. With VM, I don't see any benefit of PV-on-a-partition versus a simple raw PV.

[Joshua Huber]

@kubanczyk, one edge case (and I'm really stretching here), is in case of Junior Admin blunder, your virtual disk gets accidentally attached to another VM which happens to be running Windows. If that happens, the Windows user would get a prompt something akin to "hey you just put in an unpartitioned disk.. would you like me to partition & NTFS format it?" If the PV were inside a partition, you could still zap it, but the steps would be more deliberate. (Again, this is a stretch.) I've done similar countermeasures on thumbdrives, a small FAT32 partition up front to deter a Windows snafu.