LXD moving containers/snapshots to another drive

Question

The default path for LXD is /var/lib/lxd, so containers are in /var/lib/lxd/containers and snapshots are placed in /var/lib/lxd/snapshots.

The /var/ partition is getting full, so I'm planning to use another partition for containers and snapshots which has much more space.

My current options are:

• Create a symlink to the new directory (in the other drive)
• Bind (mount) the new directory into the current one
• Change some lxd setting (if exists) which points to a directory in the other drive

Not sure which method is easier and if there are any other ways to achieve this easily.

I'm worried about having problems with apparmor, the container's permissions or causing issues in lxd/lxc.

Which would be the correct (or best) way to move them?

(April 2017) UPDATE ******************

LXD 2.9+ supports multiple storage pools.

Solution 1

The big problem with LXD is that (currently) it only seems to support just one storage pool. In particular, this becomes somewhat of an issue if you would like to run part of your containers in one disk volume and part in another. Say, have some "fast" containers in an SSD volume and some "slow" containers in an HDD volume.

Should you wish to only run your containers in a single volume, the solution is simple as hell:

• stop your containers
• stop LXD (service lxd stop)
• move your whole /var/lib/lxd directory to your new storage pool
• create a symlink to your new storage pool
• start LXD (service lxd start)
• you're done

For instance, if you have your new storage mounted under /mnt/largepool in a subdirectory named lxd, then create the link like this: ln -s /mnt/largepool/lxd /var/lib/lxd

This way, you will have your containers on your new storage volume.

Please beware that if you are using BTRFS or ZFS as storage backends, you might wish to create the necessary subvolumes on your new storage first, so your containers happen to be located in their own subvolumes. For instance, if you have btrfs and have containers named c1 and c2 and have the directory /mnt/largepool/lxd/containers already in place, before the actual moving of files, create subvolumes: btrfs su create /mnt/largepool/lxd/containers/c1 btrfs su create /mnt/largepool/lxd/containers/c2

This would make it easy to create container snapshots afterwards.

I hope this information was helpful.

Solution 2

LXD is based on a sqlite db so to change pool location just change the db:

install sqlite3 client if you don't have one

echo "UPDATE config SET value='new_pool_lxd' WHERE \ key='storage.zfs_pool_name';"|sqlite3 /var/lib/lxd/lxd.db

assuming your lxd db in default location

then check

echo "SELECT * FROM config;"|sqlite3 /var/lib/lxd/lxd.db

1|storage.zfs_pool_name|new_pool_lxd

hope this helps

Comments

[lepe]

So there is no lxd setting to change the pool location? Thanks for the extra advice about btrfs and zfs, I will keep that in mind.

[Alvils Berzins]

I think it is possible to change the location if you bother to recompile LXD. And yet, a symlink is the simplest solution and it will "just work".

[lepe]

for some reason the table config is empty in my setup. Is that normal?

[Alex Barchiesi]

no it's not. Try a dpkg-reconfigure lxd and see what happens

[lepe]

I reconfigured but still the same. echo "SELECT * FROM config;"|sqlite3 /var/lib/lxd/lxd.db shows nothing (using a clean installation in a test server -- Ubuntu 16.04 --).

[Alex Barchiesi]

what if you create a LXD container ? Where is the container rootfs located ?

[lepe]

the rootfs is located at /var/lib/lxd/containers/*****/rootfs/.

[Alex Barchiesi]

then should really appear in the db, which user are you querying with ?

[code_dredd]

I know I'm late, but you can change the mountpoints using ZFS directly. I had to do this kind of thing when upgrading from the apt-based lxd version 3.0.3 to the snap-based version 3.17 after running /snap/bin/lxd.migrate. Basically, in my case, it was snap stop lxd && zfs unmount tank/lxd && rm -r /var/snap/lxd/common/lxd && zfs set mountpoint=/var/snap/lxd/common/lxd tank/lxd && zfs mount tank/lxd && reboot, making sure to use the correct paths for your own case.