Friday, October 10, 2014

Migration old system from single disk to mirrored LVM volume

Last time I create procedure how to convert standalone system (install on the single disk without any replication) to LVM volume with replication on two hard drive. I can't found in Internet how do this, so I decided published this on my blog.

Scenario
I have old server installed on one hard drive, without hardware/software RAID and without LVM - summarizing without any protection for system files. I would like to protect this system from disk hardware damages. How? I want implement LVM volume, exactly mirror volume. Now I have one hard drive for system, I need a second hard drive. With two hard drive we can create mirrored LVM logical volume.

Convertion single disk to LVM mirrored volume.

Of course I would like to do this migration in the most possible easy way and which not taking too long. All procedure which it I presented here I doing in the test installation system on virtual environment (KVM) using Debian distribution.

Main problem
The biggest problem in this migration process is the low capabilities between GRUB bootloader and LVM volumes format. Of course GRUB have "lvm" module and LVM developers work to LVM more compatible with GRUB, but today GRUB can't start from e.g. LVM raid1 volume, so we must use mirrored volume.

Our main goals is to creating replication system from standalone (single disk) system.

LVM migration procedure
So we have system installed on single disk 2GB and this installation using only one partition for all system points e.g. /home, /var.

root@host1:~# df -h
Filesystem                                              Size  Used Avail Use% Mounted on
rootfs                                                  2.0G  593M  1.3G  31% /
udev                                                     10M     0   10M   0% /dev
tmpfs                                                    49M  180K   49M   1% /run
/dev/disk/by-uuid/993c1a89-40b6-456a-8478-d1c2915782fd  2.0G  593M  1.3G  31% /
tmpfs                                                   5.0M     0  5.0M   0% /run/lock
tmpfs                                                    97M     0   97M   0% /run/shm

System is installed on vda, second disk (clear by writes zero) is vdb. Disk vda has DOS partition table and one primary partitions vda1. GRUB bootloader is installed on vda in MBR.

root@host1:~# fdisk /dev/vda

Command (m for help): p

Disk /dev/vda: 2147 MB, 2147483648 bytes
244 heads, 47 sectors/track, 365 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b7ab3

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     4192255     2095104   83  Linux

Command (m for help): q

Before we start move the old system (vda) to the new second disk (vdb) we must reconstruct structure of original disk (vda). Intention, on this is only creates the same type and count partitions, but not size of specific partitions.

In this particular examples we have 2GB partition, but we can create more or less capacity partition. Because I want continue to use old disk (vda) with new bigger disk (vdb has 3GB) in mirror, I can't create greater partition in vdb than has vda, because in vda can fit only <= 2GB mirror images (both images of mirror must by the same size). Furthermore LVM to create mirrored volume require small amount of free space.

So, for this reason we create smallest partitions that has vda < 2GB, exactly 1990MB.

root@host1:~# fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xbb09b322.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/vdb: 3221 MB, 3221225472 bytes
16 heads, 63 sectors/track, 6241 cylinders, total 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xbb09b322

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-6291455, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-6291455, default 6291455): +1990M

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): L

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris        
 1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx         
 5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data    
 6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility   
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt         
 9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access     
 a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O        
 b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor      
 c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi eb  BeOS fs        
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         ee  GPT            
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f1  SpeedStor      
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f4  SpeedStor      
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f2  DOS secondary  
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      fb  VMware VMFS    
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE 
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep        
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT            
1e  Hidden W95 FAT1 80  Old Minix  
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/vdb: 3221 MB, 3221225472 bytes
16 heads, 63 sectors/track, 6241 cylinders, total 6291456 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xbb09b322

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1            2048     4077567     2037760   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

In contrast to vda, vdb has different type of partition - "Linux LVM". Before we can copy system to the new disk with LVM volume we must create PV, VG and LV on vdb1 (if you don't know what it is you must read the LVM documentation).

Now we can create a normal LVM volume, but before this first we must install LVM.

root@host1:~# aptitude install lvm2

root@host1:~# pvcreate /dev/vdb1
  Writing physical volume data to disk "/dev/vdb1"
  Physical volume "/dev/vdb1" successfully created
root@host1:~# vgcreate system /dev/vdb1
  Volume group "system" successfully created
root@host1:~# lvcreate -n rootfs -L 1900MB system
  Logical volume "rootfs" created
root@host1:~# lvs -a
  LV     VG     Attr     LSize Pool Origin Data%  Move Log Copy%  Convert
  rootfs system -wi-a--- 1.86g                                           

Now we got LVM volume and we can create the file system on this volume.

root@host1:~# mkfs.ext4 /dev/system/rootfs 

In this point we have all we need to start copy system files to the new partition. But as you know you can't copy from the mounted system, so we must run server from another system. For this goal we use SystemRescueCd. SystemRescueCd has the advantage of it has built in LVM supports. 

But before we start copy system files, first we do other things which minimize counts of server restart. Now we create the new GRUB menu entry which allows boot our system from vdb1 (VG system, LV rootfs). Please note that boot the bootloadrer installed in vda will boot system storage in vdb (LVM volume). We don't have there system yet, but first we creates this entry in GRUB menu.

root@host1:~# blkid /dev/system/rootfs
/dev/system/rootfs: UUID="7fa359c8-6bde-4ecd-835b-2f753fd16a49" TYPE="ext4"
root@host1:~# vim /boot/grub/grub.cfg 

Add new menu entry to GRUB
You must copy the first menu entry and changes the four things:

  1. add load LVM module ('insmod lvm') before load DOS partition
  2. change 'set root' value to LVM style: 'system-rootfs'
  3. update UUID in command 'search' on '--set' parameter
  4. change is in 'linux' command, the 'root' parameter - also must write to LVM style: '/dev/mapper/system-rootfs'.

Of course this value depends of how you name your LVM groups nad volumes, in this cases it is 'system' and 'rootfs'. The last thing you need to do now is change the name of menu entry e.g. append " LVM".

Now we can back to the copy system files process.

First we must boot our server from CD/DVD and choose default boot options. SystemRescueCd automation scans available drive to LVM metadata, but I do this manually to show that system detect our VG.
Next step we mount both partitions vda1 and vdb1, and copy all system files from it to new LVM volume.

Run server from SystemRescueCd and work on LVM volume. 
After we copy system files to the new volume we can boot our system from it. So reboot Live system and this time select from menu options 'Run from first hard drive' then we see our new entry in GRUB menu.

Boot system from vdb using bootloader on vda.
Now we boot our system from the new copy and you can see different root system file mount point.

root@host1:~# df -hT
Filesystem                Type      Size  Used Avail Use% Mounted on
rootfs                    rootfs    1.9G  596M  1.2G  34% /
udev                      devtmpfs   10M     0   10M   0% /dev
tmpfs                     tmpfs      49M  188K   49M   1% /run
/dev/mapper/system-rootfs ext4      1.9G  596M  1.2G  34% /
tmpfs                     tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                     tmpfs      97M     0   97M   0% /run/shm

root@host1:~# lvs -a -o +devices
  LV     VG     Attr     LSize Pool Origin Data%  Move Log Copy%  Convert Devices 
  rootfs system -wi-ao-- 1.86g                                            /dev/vdb1(0)

To have a complete system on our new hard drive with LVM volume yet we miss only bootloader on this disk. So now we install GRUB on the new disk (vdb).

root@host1:~# grub-install /dev/vdb
/usr/sbin/grub-probe: error: no such disk.
Auto-detection of a filesystem of /dev/mapper/system-rootfs failed.
Try with --recheck.
If the problem persists please report this together with the output of "/usr/sbin/grub-probe --device-map="/boot/grub/device.map" --target=fs -v /boot/grub" to <bug-grub@gnu.org>
root@host1:~# grub-install --recheck /dev/vdb
Installation finished. No error reported.

I have done yet command 'update-grub' but this is not necessary, even unnecessary!

root@host1:~# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
Found Debian GNU/Linux (7.4) on /dev/vda1
done

Ok, from this moment we have all completed system with bootloader in new disk, totally independent from original hard drive. In this moment we finished the migration process to LVM. Now we start new process - convert normal LVM volume to mirrored LVM volume.
But first, it is the good moment to check if all service in system work properly, in particular if this is a productions server. You can now reboot system from new disk and try boot them from new disk (vdb).

Boot from new disk.
Bootloadre in new disk.

If all work properly on new disk, then you can go to the next step witch is destroy original disk - yes we need second disk to create LVM mirror volume. For this goal we delete existing data and partition on vda.

root@host1:~# fdisk /dev/vda

Command (m for help): p

Disk /dev/vda: 2147 MB, 2147483648 bytes
244 heads, 47 sectors/track, 365 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b7ab3

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     4192255     2095104   83  Linux

Command (m for help): d
Selected partition 1

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
root@host1:~# fdisk /dev/vda

Command (m for help): p

Disk /dev/vda: 2147 MB, 2147483648 bytes
16 heads, 63 sectors/track, 4161 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b7ab3

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
Using default value 1
First sector (2048-4194303, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4194303, default 4194303):
Using default value 4194303

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/vda: 2147 MB, 2147483648 bytes
16 heads, 63 sectors/track, 4161 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b7ab3

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1            2048     4194303     2096128   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Next we create identical LVM structure on vda like we have on vdb - first create the new PV, extend VG (system) for new PV (vda1) and last convert exist LV volume (rootfs) to the mirrored volume. LVM automatically use new free PV (vda1) from 'system' VG.

root@host1:~# pvcreate /dev/vda1
  Writing physical volume data to disk "/dev/vda1"
  Physical volume "/dev/vda1" successfully created
root@host1:~# vgextend system /dev/vda1
  Volume group "system" successfully extended

root@host1:~# vgs -a -o +devices
  VG     #PV #LV #SN Attr   VSize VFree Devices    
  system   2   1   0 wz--n- 3.94g 2.08g /dev/vdb1(0)
root@host1:~# pvs -a
  PV         VG     Fmt  Attr PSize PFree
  /dev/root              ---     0      0
  /dev/vda1  system lvm2 a--  2.00g  2.00g
  /dev/vdb1  system lvm2 a--  1.94g 88.00m
  /dev/vdc               ---     0      0
  /dev/vdd               ---     0      0

root@host1:~# lvconvert -m1 system/rootfs
  system/rootfs: Converted: 0.6%
  system/rootfs: Converted: 74.7%
  system/rootfs: Converted: 100.0%
root@host1:~# lvs -a -o +devices
  LV                VG     Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert Devices                              
  rootfs            system mwi-aom- 1.86g                         rootfs_mlog 100.00         rootfs_mimage_0(0),rootfs_mimage_1(0)
  [rootfs_mimage_0] system iwi-aom- 1.86g                                                    /dev/vdb1(0)                         
  [rootfs_mimage_1] system iwi-aom- 1.86g                                                    /dev/vda1(0)                         
  [rootfs_mlog]     system lwi-aom- 4.00m                                                    /dev/vdb1(475)                    

Synchronization two volume.
In this moment we have the same data in two disk (vda1 and vdb1), to fully happy we miss only bootloader on vda.

root@host1:~# grub-install /dev/vda
Installation finished. No error reported.

Now we can reboot our system and boot it from original disk (vda).

root@host1:~# reboot

Broadcast message from root@host1 (pts/0) (Mon Oct  6 18:44:54 2014):

The system is going down for reboot NOW!
root@host1:~# Connection to 192.168.122.248 closed by remote host.
Connection to 192.168.122.248 closed.
grzesiek@probook:~$ ssh root@192.168.122.248
root@192.168.122.248's password:
Linux host1 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Oct  6 18:34:20 2014 from 192.168.122.1
root@host1:~# df -hT
Filesystem                Type      Size  Used Avail Use% Mounted on
rootfs                    rootfs    1.9G  596M  1.2G  34% /
udev                      devtmpfs   10M     0   10M   0% /dev
tmpfs                     tmpfs      49M  200K   49M   1% /run
/dev/mapper/system-rootfs ext4      1.9G  596M  1.2G  34% /
tmpfs                     tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                     tmpfs      97M     0   97M   0% /run/shm
root@host1:~# lvs -a -o +devices
  LV                VG     Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert Devices                             
  rootfs            system mwi-aom- 1.86g                         rootfs_mlog 100.00         rootfs_mimage_0(0),rootfs_mimage_1(0)
  [rootfs_mimage_0] system iwi-aom- 1.86g                                                    /dev/vdb1(0)                        
  [rootfs_mimage_1] system iwi-aom- 1.86g                                                    /dev/vda1(0)                        
  [rootfs_mlog]     system lwi-aom- 4.00m                                                    /dev/vdb1(475)                      
root@host1:~# grub-install /dev/vdb
Installation finished. No error reported.
root@host1:~#

The last, what we must do, it is install GRUB on vdb, this operaions overwrite existing menu entry witch adds command 'update-grub'.
So now we have identical two disk with LVM mirrored volume. Also we have on each fo them installed bootloader so we can boot from any of them. When one of it is crashed we can boot server from SystemRescueCd and convert missing pair mirrored volume to normal LVM volume from one good disk, and we have our system.
Remember, each time you update kernel or GRUB you must also install GRUB on vdb to be able boot your system from vbd in future. I test this and is working properly :)