CentOS & moving /var to a new disk/partition

Question

Slightly older system, roughly 2 years old. Running CentOS

I'm noticing I have /var using a lot of space, and it's all good stuff...so deleting space isn't the answer.

Is there any easy/safe way to move /var to a newly-mounted disk without toasting the OS? I know how to add disks and format them properly, just never really tried moving /var to a new destination before.

I've read articles on Server Fault and other websites, but they're vague in response as to exactly how to go about this. They generally suggest stopping processes, etc. But don't go into detail about how to do this safely, nor how to ensure that you've stopped everything correctly before moving data and mounting everything.

Checked this one already:

Move /var to new set of disks

Didn't want to do something that would severely screw MySQL over. Thanks!

UPDATE:

File system information:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_centos6-centos6_root
                       47G   41G  4.0G  92% /
tmpfs                 3.9G  500K  3.9G   1% /dev/shm
/dev/vda1             276M  199M   64M  76% /boot
/dev/vdb1             197G  119G   68G  64% /home
/dev/vdd1             197G   95G   93G  51% /home2
/dev/vde1              99G  8.1G   86G   9% /home3
/usr/tmpDSK           2.0G   40M  1.8G   3% /tmp

Detailed mounts:

# cat /etc/fstab
/dev/vg_centos6/centos6_root /                  ext4    usrjquota=quota.user,jqfmt=vfsv0        1 1
/dev/vda1                  /boot                ext3    defaults        1 2
/dev/vdb1   /home   ext4    usrjquota=quota.user,jqfmt=vfsv0    0   0
/dev/vdd1       /home2   ext4    usrjquota=quota.user,jqfmt=vfsv0        0       0
/dev/vde1       /home3   ext4    usrjquota=quota.user,jqfmt=vfsv0        0       0
tmpfs                      /dev/shm             tmpfs   defaults        0 0
devpts                     /dev/pts             devpts  gid=5,mode=620  0 0
sysfs                      /sys                 sysfs   defaults        0 0
proc                       /proc                proc    defaults        0 0
/dev/mapper/vg_centos6-lv_swap swap swap defaults 0 0
/usr/tmpDSK             /tmp                    ext3    defaults,noauto        0 0

From what I can see, /var is definitely on the root partition. I'm hoping to change that.

Solution

A good thing is that your system is probably a VM as can be known by the device file node names (e.g. /dev/vdb), which will simplify our procedure by excluding the need to use any specialized tool (e.g. iscsi-initiator-utils) if it came from, say iSCSI storage.

CentOS offers a minimal installation media that includes the most important tool for our purpose, rsync. If you are on CentOS 6 and use a 64-bit system then you can download the minimal CD ISO image by selecting the mirror of your choice, for instance, this. Then, you could download the minimal ISO from here that has a size under 400 MB. Guidelines on the additional steps are as follows:

• Prepare spare disk. Attach the new disk to the VM and format it as usual. You have already written that you know how to do this.

• Boot into rescue mode. Gracefully shutdown the VM and boot the minimal ISO. One of the installation options offered is Rescue installed system. Selecting it will allow us to boot from the CD without any processes writing to your /var directory. This will provide us with a safe environment to transfer data from the old disk to the new one. There are few menus that you will have to navigate before being thrown into a shell prompt. Items in brackets are those that make most sense to me. Please choose ones that you find suitable.

  • Choose language. [English]
  • Keyboard type [us]
  • Do you want to start the network interfaces on this system? [No]. We can do this because you are on a VM and the new disk is going to be available without network connectivity.
  • The rescue environment... [Skip]. This is a long dialogue. Simply choose skip.
  • Start shell

• Transfer data. Let us say the device node on your new disk, which will store the new /var data is /dev/vdc1 and your old root partition, holding the /var directory is /dev/vda2. Create temporary mount points and copy the data:

  mkdir /mnt/var /mnt/var_old
  mount /dev/vda2 /mnt/var_old
  mount /dev/vdc1 /mnt/var
  rsync -a /mnt/var_old/ /mnt/var/
  

• Move old /var. Move the old /var directory away and create a new empty mount-point:

  mv /mnt/var_old/var /mnt/var_old/var.old
  mkdir /mnt/var_old/var
  

• Edit fstab. Setup /etc/fstab so that the filesystem on the new disk gets mounted on /var. Given the above device nodes, add an entry such as the following to /mnt/var_old/etc/fstab:

  /dev/vdc1 /var    ext4    defaults        1 2
  

• Reboot. Type exit on the shell prompt and select to reboot the VM. Then boot into your regular OS installation and not the minimal CD.

If everything went fine then your old /var data would be present in /var.old and /var would contain all that data and be ready for use. You might, if you so desire, remove /var.old after a few days of normal operation.

Comments

[Damiano Verzulli]

Are you using LVM? (....please post the output of "df"). Which filesystem are you using for /var? ext3|4? (...please post the output of "mount"). I'm asking 'cause if your /var is ext3 over LVM, chances are high you can enlarge it, without any "down" to be planned.

[Nameless]

Thank you for your comment. I've posted updated information about the system in question. Hopefully I've provided enough info. Thanks!

[Damiano Verzulli]

I'm tempted to answer your question telling how to "enlarge" (on-line, without any down) your / (root) partition. As such, the additional space could be used for your /var needs. Anyway.... it should be an off-topic answer. So I'll answer about moving /var (on-topic with your question) but... some services will need to be stopped (all the services using /var, like mysql, postgres, apache, syslog, etc.). Ok? I'm asking 'cause... I'd prefer to avoid spending time in useless answers :-)

[Nameless]

Thanks Damiano. I'd considered the resize myself as well, but the host won't allow me to go larger than a certain size for the primary volume attached to this cloud instance. Looking at this again, I suppose I could double what I have (they allow up to 100GB for a primary disk, and we're using about half of that right now), but allowing /var access to its own volume made initial sense to me, because I'd be able to scale it a bit better that way once moved...alleviating me of the primary volume restrictions that the host imposes on me.

[Damiano Verzulli]

Please don't forget that, with this approach, you run the risk to loose some data (written to OLD var, after rsync finished, and before the umount run by the reboot) or, worse, to corrupt data (if you are running a postgres engine with DBs under /var, and it's running during the rsync, I bet rsync-ed DB will be corrupted). If you want to follow the rsync-way, I strongly suggest to run through it with NO process running with open-files under /var.

[c4f4t0r]

@Damiano Verzulli If you have DB is good idea you use a dedicate partition for store you data, the questions was how to migrate the /var.

[Damiano Verzulli]

Sorry but.... I guessed (wrongly?) that a sysadmin hosting its own DB datafiles in an ad-hoc filesystem surely knows how to properly resize a (underused) /var :-) . So I bet OP /var is... DB un-friendly :-)

[Nameless]

Thank you. I figured running "no processes" while doing this was the key. Just wasn't sure how to be absolutely sure that all processes related to /var are actually stopped so we can proceed safely. Yes, I have backups of MySQL data. However, I would prefer not to restore a bunch of corrupted databases if it can be avoided. I've inherited this older system, and it's not to my typical specs that I would use when creating a new server for a client. Trying to make lemonade, here.