From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:34090)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VYiUp-0001XM-TZ
	for qemu-devel@nongnu.org; Tue, 22 Oct 2013 16:24:13 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VYiUj-0005JU-Uo
	for qemu-devel@nongnu.org; Tue, 22 Oct 2013 16:24:07 -0400
Received: from arlo.cse.psu.edu ([130.203.14.76]:48194)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VYiUj-0005JF-QM
	for qemu-devel@nongnu.org; Tue, 22 Oct 2013 16:24:01 -0400
Received: from mail-vb0-f49.google.com ([209.85.212.49])
	by arlo.cse.psu.edu with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.63)
	(envelope-from <xxg113@cse.psu.edu>) id 1VYiUe-0006Kw-Qv
	for qemu-devel@nongnu.org; Tue, 22 Oct 2013 16:23:57 -0400
Received: by mail-vb0-f49.google.com with SMTP id w16so5275058vbb.22
	for <qemu-devel@nongnu.org>; Tue, 22 Oct 2013 13:23:59 -0700 (PDT)
MIME-Version: 1.0
From: Xinyang Ge <xxg113@cse.psu.edu>
Date: Tue, 22 Oct 2013 16:23:39 -0400
Message-ID: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Subject: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: qemu-devel@nongnu.org

Dear QEMU developers,

I am a Ph.D. student in Penn State. And we are currently working on a
project that needs to fork multiple instances of a same VM instance
with exactly same state (e.g., memory layout, registers, etc.) in a
very efficient way. Snapshot is too heavy for us because it needs to
dump the memory state to the filesystem so that reverting is possible
sometime later. Our project does not need to revert a VM to a previous
snapshot but lively clone (or fork) multiple instances and make them
run at the same time. Do you happen to know if it's possible to do
this? What we are envisioning is copy-on-write would happen both on
disks (e.g., qcow2) and memory state (e.g., physical pages).

Thanks,
Xinyang

-- 
Xinyang GE
Department of Computer Science & Engineering
The Pennsylvania State University
Homepage: http://www.cse.psu.edu/~xxg113/

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:51097)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VYk9m-0005u9-R5
	for qemu-devel@nongnu.org; Tue, 22 Oct 2013 18:10:35 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VYk9h-0001l2-SS
	for qemu-devel@nongnu.org; Tue, 22 Oct 2013 18:10:30 -0400
Received: from mx1.redhat.com ([209.132.183.28]:15862)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VYk9h-0001iA-Hq
	for qemu-devel@nongnu.org; Tue, 22 Oct 2013 18:10:25 -0400
Message-ID: <5266F7C4.6080700@redhat.com>
Date: Tue, 22 Oct 2013 23:10:12 +0100
From: Eric Blake <eblake@redhat.com>
MIME-Version: 1.0
References: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
In-Reply-To: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="dbTg0OT7rWwVtv3007daghIcqn9Kkj3BR"
Subject: Re: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Xinyang Ge <xxg113@cse.psu.edu>, qemu-devel@nongnu.org

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--dbTg0OT7rWwVtv3007daghIcqn9Kkj3BR
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 10/22/2013 09:23 PM, Xinyang Ge wrote:
> Dear QEMU developers,
>=20
> I am a Ph.D. student in Penn State. And we are currently working on a
> project that needs to fork multiple instances of a same VM instance
> with exactly same state (e.g., memory layout, registers, etc.) in a
> very efficient way. Snapshot is too heavy for us because it needs to
> dump the memory state to the filesystem so that reverting is possible
> sometime later. Our project does not need to revert a VM to a previous
> snapshot but lively clone (or fork) multiple instances and make them
> run at the same time. Do you happen to know if it's possible to do
> this? What we are envisioning is copy-on-write would happen both on
> disks (e.g., qcow2) and memory state (e.g., physical pages).

Live cloning is a disaster waiting to happen if not done in a very
carefully controlled environment (I could maybe see it useful across two
private networks for forensic analysis or running "what-if" scenarios,
but never for provisioning enterprise-quality public-facing servers).
Remember, if you ever expose both forks of a live clone to the same
network at the same time, you have a security vulnerability if you did
not manage to scrube the random pool of the two guests to be different,
where the crypto behavior of the second guest can be guessed by
observing the behavior of the first.  But scrubbing memory correctly
requires knowing EXACTLY where in memory the random pool is stored,
which is highly guest-dependent, and may be spread across multiple guest
locations.  With offline disk images, the set of information to scrub is
a bit easier, and in fact, 'virt-sysprep' from the libguestfs tools can
do it for a number of guests, but virt-sysprep (rightfully) refuses to
try to scrub a live image.  Do your forked guests really have to run in
parallel, or is it sufficient to serialize the running of one variation
followed by the other variation?

As far as I know, the only way to run two guests that diverge from the
same live state is to take a snapshot and then run two qemu instances
that both point to that common state as their starting point, and I
would personally never attempt it in parallel.  Meanwhile, although you
complained that snapshots are too heavyweight, it's really the only way
I know to even begin to attempt live cloning with current qemu.  Of
course, being open source, you're welcome to submit a patch to add
features to qemu to do a faster live clone.  But be prepared for an
uphill battle if you cannot prove that such a patch does not introduce
security implications running improperly scrubbed forks in parallel.

--=20
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


--dbTg0OT7rWwVtv3007daghIcqn9Kkj3BR
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJSZvfEAAoJEKeha0olJ0Nq3K0H/2cEApXV9iAqXhOuu61pk7GH
PUQU6BJf9CaON4zwohj0EZZRAlLChtHlEqxY4WCuYPQjM/CPMvmrtkNKgmRTg2wf
R5VUMbf3NVrFZNsvkAfmrUvnfwI9baZYpzrOC6+0KqD58g49CaX1YMAdMZch3CPm
p/umk1WB0IGEaX9HVpVdEZFF3qw752eCkmWNtQakQZ7g0Lk3onimWNvQ7kRACLAP
0QrpPcfSURVHfeON2Izgvr+ov4EhAVbN29M4FZYqQcoI7NnQkDfMLWgp096fVMc6
n0682ZiG3gaJKGdKfRKFDA4WNAU+yKtxULMxwUgV7ol6jXAE3udDZUOSl+PoxeA=
=ut4B
-----END PGP SIGNATURE-----

--dbTg0OT7rWwVtv3007daghIcqn9Kkj3BR--

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:34872)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VYzYw-0000yt-0p
	for qemu-devel@nongnu.org; Wed, 23 Oct 2013 10:37:35 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VYzYp-00046p-87
	for qemu-devel@nongnu.org; Wed, 23 Oct 2013 10:37:29 -0400
Received: from arlo.cse.psu.edu ([130.203.14.76]:55346)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VYzYp-00046b-2K
	for qemu-devel@nongnu.org; Wed, 23 Oct 2013 10:37:23 -0400
Received: from mail-vb0-f52.google.com ([209.85.212.52])
	by arlo.cse.psu.edu with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.63)
	(envelope-from <xxg113@cse.psu.edu>) id 1VYzYi-0000k1-0f
	for qemu-devel@nongnu.org; Wed, 23 Oct 2013 10:37:16 -0400
Received: by mail-vb0-f52.google.com with SMTP id f13so564744vbg.11
	for <qemu-devel@nongnu.org>; Wed, 23 Oct 2013 07:37:19 -0700 (PDT)
MIME-Version: 1.0
In-Reply-To: <5266F7C4.6080700@redhat.com>
References: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
	<5266F7C4.6080700@redhat.com>
From: Xinyang Ge <xxg113@cse.psu.edu>
Date: Wed, 23 Oct 2013 10:36:59 -0400
Message-ID: <CACY857+pzTQX3UPJZLhFWaw_h7dSS8toEvmC9HfPdLEunggy9Q@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Subject: Re: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Eric Blake <eblake@redhat.com>
Cc: Hayawardh Vijayakumar <hvijay@cse.psu.edu>, qemu-devel@nongnu.org

> Live cloning is a disaster waiting to happen if not done in a very
> carefully controlled environment (I could maybe see it useful across two
> private networks for forensic analysis or running "what-if" scenarios,
> but never for provisioning enterprise-quality public-facing servers).
> Remember, if you ever expose both forks of a live clone to the same
> network at the same time, you have a security vulnerability if you did
> not manage to scrube the random pool of the two guests to be different,
> where the crypto behavior of the second guest can be guessed by
> observing the behavior of the first. But scrubbing memory correctly
> requires knowing EXACTLY where in memory the random pool is stored,
> which is highly guest-dependent, and may be spread across multiple guest
> locations.  With offline disk images, the set of information to scrub is
> a bit easier, and in fact, 'virt-sysprep' from the libguestfs tools can
> do it for a number of guests, but virt-sysprep (rightfully) refuses to
> try to scrub a live image.  Do your forked guests really have to run in
> parallel, or is it sufficient to serialize the running of one variation
> followed by the other variation?

It's better to have them run in parallel since our project doesn't
have any network stuff. However, running each variation sequentially
is also sufficient for us. What we are concerned the most is whether
we can get a snapshot in milliseconds because we don't really need to
save the memory state to disk for future reversion. Could you let me
know if it's possible for qemu or qemu-kvm with minor changes?

> As far as I know, the only way to run two guests that diverge from the
> same live state is to take a snapshot and then run two qemu instances
> that both point to that common state as their starting point, and I
> would personally never attempt it in parallel.  Meanwhile, although you
> complained that snapshots are too heavyweight, it's really the only way
> I know to even begin to attempt live cloning with current qemu.  Of
> course, being open source, you're welcome to submit a patch to add
> features to qemu to do a faster live clone.  But be prepared for an
> uphill battle if you cannot prove that such a patch does not introduce
> security implications running improperly scrubbed forks in parallel.

Thanks for letting me know about it. Yes, if only there's
communication between the guest and outside world, live cloning can
bring a bunch of security issues (e.g., IP address spoofing). But
since in our scenario VM doesn't have any network stuff, we would be
happy if only there's a quick-and-dirty way to implement it.


Xinyang

--
Xinyang GE
Department of Computer Science & Engineering
The Pennsylvania State University
Homepage: http://www.cse.psu.edu/~xxg113/

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:40615)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VZQJL-0005b4-RH
	for qemu-devel@nongnu.org; Thu, 24 Oct 2013 15:11:17 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VZQJD-0006OA-1z
	for qemu-devel@nongnu.org; Thu, 24 Oct 2013 15:11:11 -0400
Received: from arlo.cse.psu.edu ([130.203.14.76]:58472)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VZQJC-0006Nj-U7
	for qemu-devel@nongnu.org; Thu, 24 Oct 2013 15:11:02 -0400
Received: from mail-ve0-f175.google.com ([209.85.128.175])
	by arlo.cse.psu.edu with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.63)
	(envelope-from <xxg113@cse.psu.edu>) id 1VZQIz-0004re-O3
	for qemu-devel@nongnu.org; Thu, 24 Oct 2013 15:10:54 -0400
Received: by mail-ve0-f175.google.com with SMTP id jz11so1585467veb.6
	for <qemu-devel@nongnu.org>; Thu, 24 Oct 2013 12:10:54 -0700 (PDT)
MIME-Version: 1.0
In-Reply-To: <5266F7C4.6080700@redhat.com>
References: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
	<5266F7C4.6080700@redhat.com>
From: Xinyang Ge <xxg113@cse.psu.edu>
Date: Thu, 24 Oct 2013 15:10:34 -0400
Message-ID: <CACY857K_BXr3AuAe1=GCFSgUt40Lv_ZZxDPq3XKQiSDG5UfVxw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Subject: Re: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Eric Blake <eblake@redhat.com>
Cc: qemu-devel@nongnu.org

> Live cloning is a disaster waiting to happen if not done in a very
> carefully controlled environment (I could maybe see it useful across two
> private networks for forensic analysis or running "what-if" scenarios,
> but never for provisioning enterprise-quality public-facing servers).
> Remember, if you ever expose both forks of a live clone to the same
> network at the same time, you have a security vulnerability if you did
> not manage to scrube the random pool of the two guests to be different,
> where the crypto behavior of the second guest can be guessed by
> observing the behavior of the first. But scrubbing memory correctly
> requires knowing EXACTLY where in memory the random pool is stored,
> which is highly guest-dependent, and may be spread across multiple guest
> locations.  With offline disk images, the set of information to scrub is
> a bit easier, and in fact, 'virt-sysprep' from the libguestfs tools can
> do it for a number of guests, but virt-sysprep (rightfully) refuses to
> try to scrub a live image.  Do your forked guests really have to run in
> parallel, or is it sufficient to serialize the running of one variation
> followed by the other variation?

It's better to have them run in parallel since our project doesn't
have any network stuff. However, running each variation sequentially
is also sufficient for us. What we are concerned the most is whether
we can get a snapshot in milliseconds because we don't really need to
save the memory state to disk for future reversion. Could you let me
know if it's possible for qemu or qemu-kvm with minor changes?

> As far as I know, the only way to run two guests that diverge from the
> same live state is to take a snapshot and then run two qemu instances
> that both point to that common state as their starting point, and I
> would personally never attempt it in parallel.  Meanwhile, although you
> complained that snapshots are too heavyweight, it's really the only way
> I know to even begin to attempt live cloning with current qemu.  Of
> course, being open source, you're welcome to submit a patch to add
> features to qemu to do a faster live clone.  But be prepared for an
> uphill battle if you cannot prove that such a patch does not introduce
> security implications running improperly scrubbed forks in parallel.

Thanks for letting me know about it. Yes, if only there's
communication between the guest and outside world, live cloning can
bring a bunch of security issues (e.g., IP address spoofing). But
since in our scenario VM doesn't have any network stuff, we would be
happy if only there's a quick-and-dirty way to implement it.

Xinyang

-- 
Xinyang GE
Department of Computer Science & Engineering
The Pennsylvania State University
Homepage: http://www.cse.psu.edu/~xxg113/

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:53192)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VZvHf-0001MC-RC
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 00:15:36 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VZvHa-0002Rz-TD
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 00:15:31 -0400
Received: from mx1.redhat.com ([209.132.183.28]:33639)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VZvHa-0002Rv-Fp
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 00:15:26 -0400
Message-ID: <526B41D0.90506@redhat.com>
Date: Sat, 26 Oct 2013 05:15:12 +0100
From: Eric Blake <eblake@redhat.com>
MIME-Version: 1.0
References: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
	<5266F7C4.6080700@redhat.com>
	<CACY857+pzTQX3UPJZLhFWaw_h7dSS8toEvmC9HfPdLEunggy9Q@mail.gmail.com>
In-Reply-To: <CACY857+pzTQX3UPJZLhFWaw_h7dSS8toEvmC9HfPdLEunggy9Q@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="8sNK6FbPg90O6mE8U8LpekLoIJtwpmJXW"
Subject: Re: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Xinyang Ge <xxg113@cse.psu.edu>
Cc: Hayawardh Vijayakumar <hvijay@cse.psu.edu>, qemu-devel@nongnu.org

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--8sNK6FbPg90O6mE8U8LpekLoIJtwpmJXW
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 10/23/2013 03:36 PM, Xinyang Ge wrote:
>> Live cloning is a disaster waiting to happen if not done in a very
>> carefully controlled environment (I could maybe see it useful across t=
wo
>> private networks for forensic analysis or running "what-if" scenarios,=

>> but never for provisioning enterprise-quality public-facing servers).
>> Remember, if you ever expose both forks of a live clone to the same
>> network at the same time, you have a security vulnerability if you did=

>> not manage to scrube the random pool of the two guests to be different=
,
>> where the crypto behavior of the second guest can be guessed by
>> observing the behavior of the first. But scrubbing memory correctly
>> requires knowing EXACTLY where in memory the random pool is stored,
>> which is highly guest-dependent, and may be spread across multiple gue=
st
>> locations.  With offline disk images, the set of information to scrub =
is
>> a bit easier, and in fact, 'virt-sysprep' from the libguestfs tools ca=
n
>> do it for a number of guests, but virt-sysprep (rightfully) refuses to=

>> try to scrub a live image.  Do your forked guests really have to run i=
n
>> parallel, or is it sufficient to serialize the running of one variatio=
n
>> followed by the other variation?
>=20
> It's better to have them run in parallel since our project doesn't
> have any network stuff.

Good, then it sounds like you are being careful about avoiding the worst
aspect of live cloning (as long as two guests are never visible to the
same network, then you aren't exposing security risks over that network).=


> However, running each variation sequentially
> is also sufficient for us. What we are concerned the most is whether
> we can get a snapshot in milliseconds because we don't really need to
> save the memory state to disk for future reversion. Could you let me
> know if it's possible for qemu or qemu-kvm with minor changes?

External snapshots (via the blockdev-snapshot-sync QMP command) can be
taken in a matter of milliseconds if you only care about disk state.
Furthermore, if you want to take a snapshot of both memory and disk
state, such that the clone can be resumed from the same time, you can do
that with a guest downtime that only lasts as long as the
blockdev-snapshot-sync, by first doing a migrate to file then doing the
disk snapshot when the VM pauses at the end of migration.  Resuming the
original guest is fast; resuming from the migration file is a bit
longer, but it is still the fastest way possible to resume from a
memory+disk snapshot.  If you need anything faster, then yes, you would
have to write patches to qemu to attempt cloning via fork() that makes
sure to modify the active disk in use by the fork child so as not to
interfere with the fork parent.

--=20
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


--8sNK6FbPg90O6mE8U8LpekLoIJtwpmJXW
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJSa0HQAAoJEKeha0olJ0NqUTQH/j3hbqgtulBJHtuFOf62KtBV
cD7aSMoHvRjtHeakJrRrDUUmiUMlWTsX9sUSLWIxalNcOiZe8jBqbafaRvvasNrN
uQMvn2m3kY/2ob5LQ29qNtiPNSTLCauV7ezdCk7lOx8Zran7rg/wJYfI+8SVj9q9
JBPPkTVUSvy+ldDCKaEw4mgO/DUSUHNe640bz+8p293NOo6ed/4YkhV3+pjKFl1+
oRPrZ6YMikXPscXVkzOLNCufA+LCTAtgavizjXby3Snt1a5rHqqjW0ekkr1jH/5w
1baLeO+reIGdNmeh8+wcW2pqJ1mHiYhRGBUVAm8W77yzOHDEykMm8voPKoK2FdE=
=aJrl
-----END PGP SIGNATURE-----

--8sNK6FbPg90O6mE8U8LpekLoIJtwpmJXW--

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:58328)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1Va7oN-0000Z0-Lr
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 13:38:13 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1Va7oH-0005EY-M5
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 13:38:07 -0400
Received: from arlo.cse.psu.edu ([130.203.14.76]:59075)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1Va7oH-0005ES-HQ
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 13:38:01 -0400
Received: from mail-vb0-f41.google.com ([209.85.212.41])
	by arlo.cse.psu.edu with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.63)
	(envelope-from <xxg113@cse.psu.edu>) id 1Va7o4-0003US-JD
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 13:37:48 -0400
Received: by mail-vb0-f41.google.com with SMTP id w8so3549179vbj.28
	for <qemu-devel@nongnu.org>; Sat, 26 Oct 2013 10:37:58 -0700 (PDT)
MIME-Version: 1.0
In-Reply-To: <526B41D0.90506@redhat.com>
References: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
	<5266F7C4.6080700@redhat.com>
	<CACY857+pzTQX3UPJZLhFWaw_h7dSS8toEvmC9HfPdLEunggy9Q@mail.gmail.com>
	<526B41D0.90506@redhat.com>
From: Xinyang Ge <xxg113@cse.psu.edu>
Date: Sat, 26 Oct 2013 13:37:38 -0400
Message-ID: <CACY857K6z+n1yqTSz8YZKsPf8-NnSnZ0ngqX17xKhn9o6QSwng@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Subject: Re: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Eric Blake <eblake@redhat.com>
Cc: Hayawardh Vijayakumar <hvijay@cse.psu.edu>, qemu-devel@nongnu.org

> External snapshots (via the blockdev-snapshot-sync QMP command) can be
> taken in a matter of milliseconds if you only care about disk state.
> Furthermore, if you want to take a snapshot of both memory and disk
> state, such that the clone can be resumed from the same time, you can do
> that with a guest downtime that only lasts as long as the
> blockdev-snapshot-sync, by first doing a migrate to file then doing the
> disk snapshot when the VM pauses at the end of migration.  Resuming the
> original guest is fast; resuming from the migration file is a bit
> longer, but it is still the fastest way possible to resume from a
> memory+disk snapshot.  If you need anything faster, then yes, you would
> have to write patches to qemu to attempt cloning via fork() that makes
> sure to modify the active disk in use by the fork child so as not to
> interfere with the fork parent.

I think migrating memory to file then doing external disk snapshot is
exactly what we want. Since we are using libvirt to manage different
VMs, could you give us some specific guides (or references) that how
we could migrate memory state to file using virsh interfaces and do
external snapshots?

Thanks,
Xinyang

-- 
Xinyang GE
Department of Computer Science & Engineering
The Pennsylvania State University
Homepage: http://www.cse.psu.edu/~xxg113/

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:49187)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1Va9Hw-0005KI-Q3
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 15:12:50 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1Va9Hp-00059x-Ec
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 15:12:44 -0400
Received: from arlo.cse.psu.edu ([130.203.14.76]:54065)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1Va9Hp-00059r-A2
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 15:12:37 -0400
Received: from mail-vb0-f42.google.com ([209.85.212.42])
	by arlo.cse.psu.edu with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.63)
	(envelope-from <xxg113@cse.psu.edu>) id 1Va9Hc-0005Ny-9J
	for qemu-devel@nongnu.org; Sat, 26 Oct 2013 15:12:25 -0400
Received: by mail-vb0-f42.google.com with SMTP id p14so2097659vbm.29
	for <qemu-devel@nongnu.org>; Sat, 26 Oct 2013 12:12:33 -0700 (PDT)
MIME-Version: 1.0
In-Reply-To: <526B41D0.90506@redhat.com>
References: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
	<5266F7C4.6080700@redhat.com>
	<CACY857+pzTQX3UPJZLhFWaw_h7dSS8toEvmC9HfPdLEunggy9Q@mail.gmail.com>
	<526B41D0.90506@redhat.com>
From: Xinyang Ge <xxg113@cse.psu.edu>
Date: Sat, 26 Oct 2013 15:12:13 -0400
Message-ID: <CACY857LVjv_TfvQQJxhM9CHMXWr-Z-OCY3Pqe0BaxKHrXiL1_Q@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Subject: Re: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Eric Blake <eblake@redhat.com>
Cc: Hayawardh Vijayakumar <hvijay@cse.psu.edu>, qemu-devel@nongnu.org

> External snapshots (via the blockdev-snapshot-sync QMP command) can be
> taken in a matter of milliseconds if you only care about disk state.
> Furthermore, if you want to take a snapshot of both memory and disk
> state, such that the clone can be resumed from the same time, you can do
> that with a guest downtime that only lasts as long as the
> blockdev-snapshot-sync, by first doing a migrate to file then doing the
> disk snapshot when the VM pauses at the end of migration.  Resuming the
> original guest is fast; resuming from the migration file is a bit
> longer, but it is still the fastest way possible to resume from a
> memory+disk snapshot.  If you need anything faster, then yes, you would
> have to write patches to qemu to attempt cloning via fork() that makes
> sure to modify the active disk in use by the fork child so as not to
> interfere with the fork parent.

I noticed there is a "qemu-monitor-command" option for virsh and maybe
I can use this to issue QMP commands. However, when I run
"blockdev-snapshot-sync", I cannot correctly identify the correct
values for argument "device". Below is part of the VM's configuration
file (libvirt format):

<devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/u12.qcow2'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>

Can you help me find out which is the right value for "device" when I
run "blockdev-snapshot-sync"? Thanks in advance!

Xinyang

-- 
Xinyang GE
Department of Computer Science & Engineering
The Pennsylvania State University
Homepage: http://www.cse.psu.edu/~xxg113/

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:42696)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VantG-0003EO-Dw
	for qemu-devel@nongnu.org; Mon, 28 Oct 2013 10:34:03 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VantB-0002vs-Bc
	for qemu-devel@nongnu.org; Mon, 28 Oct 2013 10:33:58 -0400
Received: from mx1.redhat.com ([209.132.183.28]:49154)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1VantB-0002vi-3P
	for qemu-devel@nongnu.org; Mon, 28 Oct 2013 10:33:53 -0400
Message-ID: <526E75C6.1020307@redhat.com>
Date: Mon, 28 Oct 2013 08:33:42 -0600
From: Eric Blake <eblake@redhat.com>
MIME-Version: 1.0
References: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
	<5266F7C4.6080700@redhat.com>
	<CACY857+pzTQX3UPJZLhFWaw_h7dSS8toEvmC9HfPdLEunggy9Q@mail.gmail.com>
	<526B41D0.90506@redhat.com>
	<CACY857K6z+n1yqTSz8YZKsPf8-NnSnZ0ngqX17xKhn9o6QSwng@mail.gmail.com>
In-Reply-To: <CACY857K6z+n1yqTSz8YZKsPf8-NnSnZ0ngqX17xKhn9o6QSwng@mail.gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="uP6Rwak8liL9rRHXSl6h4GtgjJwBH27ir"
Subject: Re: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Xinyang Ge <xxg113@cse.psu.edu>
Cc: Hayawardh Vijayakumar <hvijay@cse.psu.edu>, qemu-devel@nongnu.org

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--uP6Rwak8liL9rRHXSl6h4GtgjJwBH27ir
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 10/26/2013 11:37 AM, Xinyang Ge wrote:
>> External snapshots (via the blockdev-snapshot-sync QMP command) can be=

>> taken in a matter of milliseconds if you only care about disk state.
>> Furthermore, if you want to take a snapshot of both memory and disk
>> state, such that the clone can be resumed from the same time, you can =
do
>> that with a guest downtime that only lasts as long as the
>> blockdev-snapshot-sync, by first doing a migrate to file then doing th=
e
>> disk snapshot when the VM pauses at the end of migration.  Resuming th=
e
>> original guest is fast; resuming from the migration file is a bit
>> longer, but it is still the fastest way possible to resume from a
>> memory+disk snapshot.  If you need anything faster, then yes, you woul=
d
>> have to write patches to qemu to attempt cloning via fork() that makes=

>> sure to modify the active disk in use by the fork child so as not to
>> interfere with the fork parent.
>=20
> I think migrating memory to file then doing external disk snapshot is
> exactly what we want. Since we are using libvirt to manage different
> VMs, could you give us some specific guides (or references) that how
> we could migrate memory state to file using virsh interfaces and do
> external snapshots?

virsh snapshot-create-as $dom $name --live --memspec /path/to/memoryfile

Libvirt usage questions might be better directed to the libvirt lists.

--=20
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


--uP6Rwak8liL9rRHXSl6h4GtgjJwBH27ir
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJSbnXGAAoJEKeha0olJ0Nq6vQH/12qhCK8Gje7KuywliVp8Yr6
9LZwKvGfc7VZp5Xyvj/bJyMGg+JOgfg70Lk5ADLwQr+ju7awSn1xPc/YwbnZExVd
gAN2p1sNkhuTkZWjxfgSj2bsbV2L8V0ZxEVtkCfZdw9EbmYtaNpSIvpFIFBlBwQp
ZpmyUsGKm3a/TWm5gCA7AJKZU5iRZj2a83RKiF2gi+crHq6WV8Jy+OHvj5BmFGhb
ZbuxC5vAllvQrQzVw2Ym8ojBoDcuHKCKsMwhhbSNyoEGDEBBSxLzAEiLG69gpYAE
XdmnLiIql7XwSAJuDqZU/dQ8YvHVH4AMcZkwlMSIgVuS74XFB51jSb1DYcflcKs=
=y/UT
-----END PGP SIGNATURE-----

--uP6Rwak8liL9rRHXSl6h4GtgjJwBH27ir--

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:52466)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VaqeQ-0001a6-LL
	for qemu-devel@nongnu.org; Mon, 28 Oct 2013 13:30:56 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VaqeF-0007kO-Iz
	for qemu-devel@nongnu.org; Mon, 28 Oct 2013 13:30:50 -0400
Received: from arlo.cse.psu.edu ([130.203.14.76]:38404)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xxg113@cse.psu.edu>) id 1VaqeF-0007k2-EE
	for qemu-devel@nongnu.org; Mon, 28 Oct 2013 13:30:39 -0400
Received: from mail-ve0-f177.google.com ([209.85.128.177])
	by arlo.cse.psu.edu with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.63)
	(envelope-from <xxg113@cse.psu.edu>) id 1VaqeA-0008Le-Tk
	for qemu-devel@nongnu.org; Mon, 28 Oct 2013 13:30:35 -0400
Received: by mail-ve0-f177.google.com with SMTP id oz11so5172784veb.22
	for <qemu-devel@nongnu.org>; Mon, 28 Oct 2013 10:30:36 -0700 (PDT)
MIME-Version: 1.0
In-Reply-To: <526E75C6.1020307@redhat.com>
References: <CACY857Lfzp7BRj=0cKTMYb=CRSU2pHZxLnqLk6S4YvHzFp9Mag@mail.gmail.com>
	<5266F7C4.6080700@redhat.com>
	<CACY857+pzTQX3UPJZLhFWaw_h7dSS8toEvmC9HfPdLEunggy9Q@mail.gmail.com>
	<526B41D0.90506@redhat.com>
	<CACY857K6z+n1yqTSz8YZKsPf8-NnSnZ0ngqX17xKhn9o6QSwng@mail.gmail.com>
	<526E75C6.1020307@redhat.com>
From: Xinyang Ge <xxg113@cse.psu.edu>
Date: Mon, 28 Oct 2013 13:30:16 -0400
Message-ID: <CACY857+5aNmLwrO=+VyLXD9XLQbVttpvVBYfBP8wozUuWw3KMA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Subject: Re: [Qemu-devel] About VM fork in QEMU
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Eric Blake <eblake@redhat.com>
Cc: Hayawardh Vijayakumar <hvijay@cse.psu.edu>, libvirt-users@redhat.com, qemu-devel@nongnu.org

>>> External snapshots (via the blockdev-snapshot-sync QMP command) can be
>>> taken in a matter of milliseconds if you only care about disk state.
>>> Furthermore, if you want to take a snapshot of both memory and disk
>>> state, such that the clone can be resumed from the same time, you can do
>>> that with a guest downtime that only lasts as long as the
>>> blockdev-snapshot-sync, by first doing a migrate to file then doing the
>>> disk snapshot when the VM pauses at the end of migration.  Resuming the
>>> original guest is fast; resuming from the migration file is a bit
>>> longer, but it is still the fastest way possible to resume from a
>>> memory+disk snapshot.  If you need anything faster, then yes, you would
>>> have to write patches to qemu to attempt cloning via fork() that makes
>>> sure to modify the active disk in use by the fork child so as not to
>>> interfere with the fork parent.
>>
>> I think migrating memory to file then doing external disk snapshot is
>> exactly what we want. Since we are using libvirt to manage different
>> VMs, could you give us some specific guides (or references) that how
>> we could migrate memory state to file using virsh interfaces and do
>> external snapshots?
>
> virsh snapshot-create-as $dom $name --live --memspec /path/to/memoryfile

I have tried this command on libvirt v1.1.3 and it returns "error:
invalid argument: qemuDomainSnapshotCreateXML: unsupported flags
(0x100)". Looks like --live is not supported yet. Could you let us
know which version we should of libvirt we should use in order to use
this feature?

Thanks,
Xinyang

-- 
Xinyang GE
Department of Computer Science & Engineering
The Pennsylvania State University
Homepage: http://www.cse.psu.edu/~xxg113/

