"Instant clone" with Qemu?

"Instant clone" with Qemu?

[Philipp Hahn]

Hello,

by accident I stumbled over "VMware Instant Clone" ¹, which allows 
cloning of running VMs by copy-on-write-sharing the disk images and 
memory content; the network MAC address gets changed (or a different 
bridge is used?).
I wonder if something similar can also be done with Qemu? My current 
solution would be to:
- start and install the VM
- create a live-snapshot into the qcow2 file
- clone the disk image, e.g. put a qcow2 overlay on it per clone
- start and restore the clones from that live-snapshot
- put the clones in individual bridges and let the host do some network 
address translation (NAT) to give each clone a unique external IP address.

Has someone done something similar or is there even a better alternative?

Background: our test suite currently provisions a set of multiple VMs, 
which are dependent on each other. Provisioning them takes sometimes 
many hours. After that the test suite runs inside these VMs and again 
takes many hours.
I'd like to speed that up by parallelizing these tests, e.g.
1. setup the VM environment once
2. clone the VM environments as the resources allow
3. distribute tests over these environments to run in parallel and to 
allow running flaky tests multiple times from a clean clone again

¹<https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-853B1E2B-76CE-4240-A654-3806912820EB.html>
-- 
Philipp Hahn
Open Source Software Engineer

Univention GmbH
Mary-Somerville-Str. 1
28359 Bremen
Germany | Deutschland
Phone: +49 (0)421 22232-0 | E-Mail: info@univention.de

https://www.univention.de | https://www.univention.com

Managing Directors: Peter H. Ganten, Stefan Gohmann
Local court: Amtsgericht Bremen
HRB 20755 | Ust-ID: DE220051310

Re: "Instant clone" with Qemu?

[Stefan Hajnoczi]

On Tue, 5 Dec 2023 at 04:53, Philipp Hahn <hahn@univention.de> wrote:
>
> Hello,
>
> by accident I stumbled over "VMware Instant Clone" ¹, which allows
> cloning of running VMs by copy-on-write-sharing the disk images and
> memory content; the network MAC address gets changed (or a different
> bridge is used?).
> I wonder if something similar can also be done with Qemu? My current
> solution would be to:
> - start and install the VM
> - create a live-snapshot into the qcow2 file
> - clone the disk image, e.g. put a qcow2 overlay on it per clone
> - start and restore the clones from that live-snapshot
> - put the clones in individual bridges and let the host do some network
> address translation (NAT) to give each clone a unique external IP address.
>
> Has someone done something similar or is there even a better alternative?
>
> Background: our test suite currently provisions a set of multiple VMs,
> which are dependent on each other. Provisioning them takes sometimes
> many hours. After that the test suite runs inside these VMs and again
> takes many hours.
> I'd like to speed that up by parallelizing these tests, e.g.
> 1. setup the VM environment once
> 2. clone the VM environments as the resources allow
> 3. distribute tests over these environments to run in parallel and to
> allow running flaky tests multiple times from a clean clone again

It would be simplest to use qcow2 backing files and boot each new
instance from scratch. This involves setting up a master image and
then "qemu-img create -f qcow2 -b master.img vm001.qcow2" to create
the instance image. You may be able to use systemd or your distro's
"first boot" functionality to recreate unique IDs and cryptographic
keys when the new instance boots.

If you really want to use a RAM snapshot then I suggest creating a
qcow2 master image with the savevm command and using "cp
--reflink=always master.qcow2 vm001.qcow2" to create an efficient copy
of the qcow2 file. You'll need some custom scripts to recreate unique
IDs and cryptographic keys inside the new instance after loadvm.

Stefan

>
> ¹<https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-853B1E2B-76CE-4240-A654-3806912820EB.html>
> --
> Philipp Hahn
> Open Source Software Engineer
>
> Univention GmbH
> Mary-Somerville-Str. 1
> 28359 Bremen
> Germany | Deutschland
> Phone: +49 (0)421 22232-0 | E-Mail: info@univention.de
>
> https://www.univention.de | https://www.univention.com
>
> Managing Directors: Peter H. Ganten, Stefan Gohmann
> Local court: Amtsgericht Bremen
> HRB 20755 | Ust-ID: DE220051310
>

Re: "Instant clone" with Qemu?

[Philipp Hahn]

Hello Stefan,

thank you for your kind reply.

Am 05.12.23 um 15:44 schrieb Stefan Hajnoczi:
> On Tue, 5 Dec 2023 at 04:53, Philipp Hahn <hahn@univention.de> wrote:
 >
>> by accident I stumbled over "VMware Instant Clone" ¹, which allows
>> cloning of running VMs by copy-on-write-sharing the disk images and
>> memory content; the network MAC address gets changed (or a different
>> bridge is used?).
>> I wonder if something similar can also be done with Qemu? My current
>> solution would be to:
>> - start and install the VM
>> - create a live-snapshot into the qcow2 file
>> - clone the disk image, e.g. put a qcow2 overlay on it per clone
>> - start and restore the clones from that live-snapshot
>> - put the clones in individual bridges and let the host do some network
>> address translation (NAT) to give each clone a unique external IP address.
>>
>> Has someone done something similar or is there even a better alternative?
>>
>> Background: our test suite currently provisions a set of multiple VMs,
>> which are dependent on each other. Provisioning them takes sometimes
>> many hours. After that the test suite runs inside these VMs and again
>> takes many hours.
>> I'd like to speed that up by parallelizing these tests, e.g.
>> 1. setup the VM environment once
>> 2. clone the VM environments as the resources allow
>> 3. distribute tests over these environments to run in parallel and to
>> allow running flaky tests multiple times from a clean clone again
> 
> It would be simplest to use qcow2 backing files and boot each new
> instance from scratch. This involves setting up a master image and
> then "qemu-img create -f qcow2 -b master.img vm001.qcow2" to create
> the instance image. You may be able to use systemd or your distro's
> "first boot" functionality to recreate unique IDs and cryptographic
> keys when the new instance boots.

Actually I do not want to modify the clones at all: While the machine ID 
is probably less interesting to others, I can even live with re-using 
the SSH keys as this is only for *internal* testing: I can tell `ssh` to 
not check the keys as I can control all the networking, so security is 
of little concern here.

> If you really want to use a RAM snapshot then I suggest creating a
> qcow2 master image with the savevm command and using "cp
> --reflink=always master.qcow2 vm001.qcow2" to create an efficient copy
> of the qcow2 file. You'll need some custom scripts to recreate unique
> IDs and cryptographic keys inside the new instance after loadvm.

Is there a major difference between doing a "savevm" to an external file 
and doing a live snapshot, which stores the "savevm" inside the qcow2 
file itself. The later has the benefit for me, that I only have to 
handle one file; I could even store it for later use if needed.


My main problem currently is cloning the MAC address: As our product is 
an operating system the MAC addresses of the involved systems is stored 
in some databases; while in most cases they are not required, I do not 
want to hunt for these in all kind of different locations and change 
them to some cloned MAC address.
I already had a look at "Virtual Routing and Forwarding"², which allows 
me to resue the same MAC addresses in different network bridge 
interfaces, but what I did not yet get to work is the "routing" between 
them. I found some very nice articles³⁴ on how to do NAT with VRF, but 
it is not yet working.

Philipp

¹<https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-853B1E2B-76CE-4240-A654-3806912820EB.html>
²<https://www.kernel.org/doc/html/latest/networking/vrf.html>
³<https://blog.oddbit.com/post/2023-02-19-vrf-and-nat/>
⁴<https://www.linuxquestions.org/questions/linux-newbie-8/iptables-nat-for-vrf-4175721876/>

Re: "Instant clone" with Qemu?

[Stefan Hajnoczi]

On Fri, 15 Dec 2023 at 06:01, Philipp Hahn <hahn@univention.de> wrote:
>
> Hello Stefan,
>
> thank you for your kind reply.
>
> Am 05.12.23 um 15:44 schrieb Stefan Hajnoczi:
> > On Tue, 5 Dec 2023 at 04:53, Philipp Hahn <hahn@univention.de> wrote:
>  >
> >> by accident I stumbled over "VMware Instant Clone" ¹, which allows
> >> cloning of running VMs by copy-on-write-sharing the disk images and
> >> memory content; the network MAC address gets changed (or a different
> >> bridge is used?).
> >> I wonder if something similar can also be done with Qemu? My current
> >> solution would be to:
> >> - start and install the VM
> >> - create a live-snapshot into the qcow2 file
> >> - clone the disk image, e.g. put a qcow2 overlay on it per clone
> >> - start and restore the clones from that live-snapshot
> >> - put the clones in individual bridges and let the host do some network
> >> address translation (NAT) to give each clone a unique external IP address.
> >>
> >> Has someone done something similar or is there even a better alternative?
> >>
> >> Background: our test suite currently provisions a set of multiple VMs,
> >> which are dependent on each other. Provisioning them takes sometimes
> >> many hours. After that the test suite runs inside these VMs and again
> >> takes many hours.
> >> I'd like to speed that up by parallelizing these tests, e.g.
> >> 1. setup the VM environment once
> >> 2. clone the VM environments as the resources allow
> >> 3. distribute tests over these environments to run in parallel and to
> >> allow running flaky tests multiple times from a clean clone again
> >
> > It would be simplest to use qcow2 backing files and boot each new
> > instance from scratch. This involves setting up a master image and
> > then "qemu-img create -f qcow2 -b master.img vm001.qcow2" to create
> > the instance image. You may be able to use systemd or your distro's
> > "first boot" functionality to recreate unique IDs and cryptographic
> > keys when the new instance boots.
>
> Actually I do not want to modify the clones at all: While the machine ID
> is probably less interesting to others, I can even live with re-using
> the SSH keys as this is only for *internal* testing: I can tell `ssh` to
> not check the keys as I can control all the networking, so security is
> of little concern here.
>
> > If you really want to use a RAM snapshot then I suggest creating a
> > qcow2 master image with the savevm command and using "cp
> > --reflink=always master.qcow2 vm001.qcow2" to create an efficient copy
> > of the qcow2 file. You'll need some custom scripts to recreate unique
> > IDs and cryptographic keys inside the new instance after loadvm.
>
> Is there a major difference between doing a "savevm" to an external file
> and doing a live snapshot, which stores the "savevm" inside the qcow2
> file itself. The later has the benefit for me, that I only have to
> handle one file; I could even store it for later use if needed.

With the reflink approach you still snapshot the VM into the original
qcow2 file (with the "savevm" command), not into an external file. The
reflink creates an efficient copy of the file for each instance of the
VM that you with to clone. But since you've said you don't want to
modify the clones at all, maybe this approach is overkill because you
have to manage these new qcow2 files.

Have you tried the -snapshot command-line option? It creates a
temporary qcow2 overlay that is discarded when QEMU exits. That allows
the guest to write to the disk but those changes won't be permanent
and you can run as many guests simultaneously as you want (each has
its own temporary qcow2 overlay that is managed by QEMU behind the
scenes).

>
>
> My main problem currently is cloning the MAC address: As our product is
> an operating system the MAC addresses of the involved systems is stored
> in some databases; while in most cases they are not required, I do not
> want to hunt for these in all kind of different locations and change
> them to some cloned MAC address.
> I already had a look at "Virtual Routing and Forwarding"², which allows
> me to resue the same MAC addresses in different network bridge
> interfaces, but what I did not yet get to work is the "routing" between
> them. I found some very nice articles³⁴ on how to do NAT with VRF, but
> it is not yet working.

I'm not knowledgeable about VRF. You could also use -netdev
user,hostfwd=tcp::$VM_SSH_NAT_PORT-:22 where VM_SSH_NAT_PORT is a
unique port assigned by the script that launches the guest. That way
each guest can have the same MAC address and IP address but receive
incoming SSH connections.

Stefan

Re: "Instant clone" with Qemu?

[Philipp Hahn]

Hello Stefan,

Am 15.12.23 um 16:21 schrieb Stefan Hajnoczi:
>> Am 05.12.23 um 15:44 schrieb Stefan Hajnoczi:
>>> On Tue, 5 Dec 2023 at 04:53, Philipp Hahn <hahn@univention.de> wrote:
>>   >
>>>> by accident I stumbled over "VMware Instant Clone" ¹, which allows
>>>> cloning of running VMs by copy-on-write-sharing the disk images and
>>>> memory content; the network MAC address gets changed (or a different
>>>> bridge is used?).
>>>> I wonder if something similar can also be done with Qemu? My current
>>>> solution would be to:
>>>> - start and install the VM
>>>> - create a live-snapshot into the qcow2 file
>>>> - clone the disk image, e.g. put a qcow2 overlay on it per clone
>>>> - start and restore the clones from that live-snapshot
>>>> - put the clones in individual bridges and let the host do some network
>>>> address translation (NAT) to give each clone a unique external IP address.
>>>>
>>>> Has someone done something similar or is there even a better alternative?
>>>>
>>>> Background: our test suite currently provisions a set of multiple VMs,
>>>> which are dependent on each other. Provisioning them takes sometimes
>>>> many hours. After that the test suite runs inside these VMs and again
>>>> takes many hours.
>>>> I'd like to speed that up by parallelizing these tests, e.g.
>>>> 1. setup the VM environment once
>>>> 2. clone the VM environments as the resources allow
>>>> 3. distribute tests over these environments to run in parallel and to
>>>> allow running flaky tests multiple times from a clean clone again
>>>
>>> It would be simplest to use qcow2 backing files and boot each new
>>> instance from scratch. This involves setting up a master image and
>>> then "qemu-img create -f qcow2 -b master.img vm001.qcow2" to create
>>> the instance image. You may be able to use systemd or your distro's
>>> "first boot" functionality to recreate unique IDs and cryptographic
>>> keys when the new instance boots.
>>
>> Actually I do not want to modify the clones at all: While the machine ID
>> is probably less interesting to others, I can even live with re-using
>> the SSH keys as this is only for *internal* testing: I can tell `ssh` to
>> not check the keys as I can control all the networking, so security is
>> of little concern here.
>>
>>> If you really want to use a RAM snapshot then I suggest creating a
>>> qcow2 master image with the savevm command and using "cp
>>> --reflink=always master.qcow2 vm001.qcow2" to create an efficient copy
>>> of the qcow2 file. You'll need some custom scripts to recreate unique
>>> IDs and cryptographic keys inside the new instance after loadvm.
>>
>> Is there a major difference between doing a "savevm" to an external file
>> and doing a live snapshot, which stores the "savevm" inside the qcow2
>> file itself. The later has the benefit for me, that I only have to
>> handle one file; I could even store it for later use if needed.
> 
> With the reflink approach you still snapshot the VM into the original
> qcow2 file (with the "savevm" command), not into an external file. The
> reflink creates an efficient copy of the file for each instance of the
> VM that you with to clone. But since you've said you don't want to
> modify the clones at all, maybe this approach is overkill because you
> have to manage these new qcow2 files.
> 
> Have you tried the -snapshot command-line option?

Thank you for the reminder, I already used it in other contexts but 
forgot it here. I'll add it to my toolshed.

Regarding reflink: that requires FS support, which AFAIKS is only 
supported by BTRFS and XFS (and OCFS2). So using `-snapshot` or an 
explicit `qemu-img create -f qcow2 -b $base.qcow2 $overlay.qcow2` works 
on any FS.

>> My main problem currently is cloning the MAC address: As our product is
>> an operating system the MAC addresses of the involved systems is stored
>> in some databases; while in most cases they are not required, I do not
>> want to hunt for these in all kind of different locations and change
>> them to some cloned MAC address.
>> I already had a look at "Virtual Routing and Forwarding"², which allows
>> me to resue the same MAC addresses in different network bridge
>> interfaces, but what I did not yet get to work is the "routing" between
>> them. I found some very nice articles³⁴ on how to do NAT with VRF, but
>> it is not yet working.
> 
> I'm not knowledgeable about VRF. You could also use -netdev
> user,hostfwd=tcp::$VM_SSH_NAT_PORT-:22 where VM_SSH_NAT_PORT is a
> unique port assigned by the script that launches the guest. That way
> each guest can have the same MAC address and IP address but receive
> incoming SSH connections.

Good to know, thank you. My problem is that I have to clone multiple VMs 
belonging together, they must be able to communicate with each other 
using their unmodified IP addresses; I only need to connect to one of 
them from the outside; something like a "jump host".

Thanks again
Philipp

Re: "Instant clone" with Qemu?

[Stefan Hajnoczi]

On Mon, 18 Dec 2023 at 02:36, Philipp Hahn <hahn@univention.de> wrote:
> Am 15.12.23 um 16:21 schrieb Stefan Hajnoczi:
> >> Am 05.12.23 um 15:44 schrieb Stefan Hajnoczi:
> >>> On Tue, 5 Dec 2023 at 04:53, Philipp Hahn <hahn@univention.de> wrote:
> >> My main problem currently is cloning the MAC address: As our product is
> >> an operating system the MAC addresses of the involved systems is stored
> >> in some databases; while in most cases they are not required, I do not
> >> want to hunt for these in all kind of different locations and change
> >> them to some cloned MAC address.
> >> I already had a look at "Virtual Routing and Forwarding"², which allows
> >> me to resue the same MAC addresses in different network bridge
> >> interfaces, but what I did not yet get to work is the "routing" between
> >> them. I found some very nice articles³⁴ on how to do NAT with VRF, but
> >> it is not yet working.
> >
> > I'm not knowledgeable about VRF. You could also use -netdev
> > user,hostfwd=tcp::$VM_SSH_NAT_PORT-:22 where VM_SSH_NAT_PORT is a
> > unique port assigned by the script that launches the guest. That way
> > each guest can have the same MAC address and IP address but receive
> > incoming SSH connections.
>
> Good to know, thank you. My problem is that I have to clone multiple VMs
> belonging together, they must be able to communicate with each other
> using their unmodified IP addresses; I only need to connect to one of
> them from the outside; something like a "jump host".

--netdev user,guestfwd= redirects outgoing connections from a guest.
That might allow you to remap the IP/port of each VM as necessary.

Stefan