How to replicate a Xen VM using BTRFS as the root filesystem.

How to replicate a Xen VM using BTRFS as the root filesystem.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 3423 bytes --]

I would put this on the wiki in the stuff about use cases, but I don't 
have a wiki account and don't really have the time or interest right now 
in getting one, so I'm posting it here instead.

This is a rather interesting use case for send/receive that I've never 
seen discussed anywhere else.  In essence, it lets you take a working 
Xen user domain which uses BTRFS as the root filesystem, and use that as 
a template for new VM's without having to deal with seed devices (which 
means you can do it with the source domain online with only minimal 
service degradation).

These steps assume you have a minimal working knowledge of Xen and 
BTRFS, as well as access to both Domain-0 and the user domain you are 
copying (refereed to below as the 'source domain'.  I've only used this 
myself with Gentoo, but it should work for almost any Linux distro. 
I've also only done this with fully paravirtualized domains, it's a lot 
more involved to do it with hardware virtualized domains, although PVH 
domains should work with exactly the same steps as regular PV domains. 
Your source domain's root also needs to be in a subvolume on the root 
filesystem, this won't work if it isn't.

1. Create your backend storage devices for the new domain.
2. Attach the storage device (or devices) that will hold the root 
filesystem for the new domain to the source domain.  Using xl, this 
translates to something similar to: 'xl block-attach source 
"/dev/vg/target-disk,raw,xvdz"'
3. From the source domain, use mkfs.btrfs to create a BTRFS filesystem 
on the target disk (be careful, if you specify the wrong disk you may 
cause serious issues).
4. On the source domain, mount both the root filesystem (don't bind 
mount it, and make sure that you are mounting the top-level, not a 
subvolume) and the target filesystem.
5. Create a read-only snapshot of the root subvolume from the source 
filesystem.
6. Use btrfs send piped to btrfs receive to copy the snapshot from the 
source filesystem to the target filesystem.  This will likely take quite 
some time (on the low-end server equivalent hardware I have, it takes 
about 2o-25 minutes for a somewhat minimalistic Gentoo installation).
7. While btrfs send/receive is running, prepare the configuration file 
for the target domain (I usually just copy the config from the source 
domain, and then change only what I need to).
8. Once the send/receive operation is complete, use btrfs property set 
to change the snapshot on the target to be writable, then rename it to 
whatever you want the root subvolume to be called on the target system.
9. In the newly created root subvolume, change any system specific 
configuration to what is needed for the new system (at a bare minimum, 
you probably need to change the hostname and networking configuration, 
and should verify that /etc/fstab and /etc/localtime are correct for the 
target system).
10. Unmount the target filesystem in the source domain.
11. Use 'xl block-detach' to detach the target device from the source 
domain.
12. Use your regular tools to start your new domain, log in, and preform 
any final configuration needed.

Using this methodology, I can have a new Gentoo PV domain running in 
about half an hour, whereas it takes me at least two and a half hours 
(and often much longer than that) when using the regular install process 
for Gentoo.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: How to replicate a Xen VM using BTRFS as the root filesystem.

[Russell Coker]

On Wed, 28 Oct 2015 11:07:20 PM Austin S Hemmelgarn wrote:
> Using this methodology, I can have a new Gentoo PV domain running in 
> about half an hour, whereas it takes me at least two and a half hours 
> (and often much longer than that) when using the regular install process 
> for Gentoo.

On my virtual servers I have a BTRFS subvol /xenstore for the block devices of 
virtual machines.  When I want to duplicate a VM I run
"cp -a --reflink=aways /xenstore/A /xenstore/B" which takes a few seconds.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: How to replicate a Xen VM using BTRFS as the root filesystem.

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 952 bytes --]

On 2015-10-28 22:39, Russell Coker wrote:
> On Wed, 28 Oct 2015 11:07:20 PM Austin S Hemmelgarn wrote:
>> Using this methodology, I can have a new Gentoo PV domain running in
>> about half an hour, whereas it takes me at least two and a half hours
>> (and often much longer than that) when using the regular install process
>> for Gentoo.
>
> On my virtual servers I have a BTRFS subvol /xenstore for the block devices of
> virtual machines.  When I want to duplicate a VM I run
> "cp -a --reflink=aways /xenstore/A /xenstore/B" which takes a few seconds.
Well yes, that works quickly too, but is kind of hard to do when using 
LVM or other non-file-backed block storage, and the usual recommendation 
when not using blktap for storage is to use normal block storage devices 
(that, and I've had some horrible experience WRT performance with 
running VM's with BTRFS filesystems on files on a BTRFS filesystem on 
the host system).



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]