* [Qemu-devel] Fix for CVE-2016-5403 causes crash on migration if memory stats are enabled
@ 2016-08-11 7:18 Gaudenz Steinlin
2016-08-12 10:42 ` Stefan Hajnoczi
0 siblings, 1 reply; 5+ messages in thread
From: Gaudenz Steinlin @ 2016-08-11 7:18 UTC (permalink / raw)
To: qemu-devel
[-- Attachment #1: Type: text/plain, Size: 1701 bytes --]
[ Please CC me on replies as I'm not subscribed to this list. ]
Hi
The Fix for CVE-2016-5403 (virtio: error out if guest exceeds virtqueue
size)[1] causes qemu to exit(1) after migration or restart from a saved
state if memory statistics are enabled in libvirt. Qemu exits after
printing "qemu-system-x86_64: Virtqueue size exceeded".
I experienced this problem with the latest security update in Ubuntu
Trusty (14.04) which cherry-picked this fix. If you think that the
latest upstream version is not affected I can try this too. I only
tested with VM started through libvirt. If someone tells me how to
enable memory statistics with plain qemu without libvirt I can test this
too. My guess would be that this does not make a difference.
I discovered this bug because OpenStack Nova enables memory statistics
by default since the Juno release. After the QEMU upgrade to the latest
version in Ubuntu VMs were suddenly shutoff after migration.
Steps to reproduce:
1. Create a VM with libvirt which contains a memory balloon device
defined like this:
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
<stats period='10'/>
</memballoon>
2. Start the VM and let the Linux kernel boot (bug does not appear if
the kernel is not yet booted, eg. while in the PXE boot phase)
3. Issue a managedsave
4. Start the VM again
5. The VM is restored and "crashes" right after it starts running again.
6. You can find the qemu output "qemu-system-x86_64: Virtqueue size
exceeded" in the log at /var/log/libvirt/vmname.log
Gaudenz
[1] https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg06257.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 464 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] Fix for CVE-2016-5403 causes crash on migration if memory stats are enabled
2016-08-11 7:18 [Qemu-devel] Fix for CVE-2016-5403 causes crash on migration if memory stats are enabled Gaudenz Steinlin
@ 2016-08-12 10:42 ` Stefan Hajnoczi
2016-08-14 20:32 ` Gaudenz Steinlin
0 siblings, 1 reply; 5+ messages in thread
From: Stefan Hajnoczi @ 2016-08-12 10:42 UTC (permalink / raw)
To: Gaudenz Steinlin; +Cc: qemu-devel
[-- Attachment #1: Type: text/plain, Size: 1975 bytes --]
On Thu, Aug 11, 2016 at 09:18:12AM +0200, Gaudenz Steinlin wrote:
>
> [ Please CC me on replies as I'm not subscribed to this list. ]
>
> Hi
>
> The Fix for CVE-2016-5403 (virtio: error out if guest exceeds virtqueue
> size)[1] causes qemu to exit(1) after migration or restart from a saved
> state if memory statistics are enabled in libvirt. Qemu exits after
> printing "qemu-system-x86_64: Virtqueue size exceeded".
>
> I experienced this problem with the latest security update in Ubuntu
> Trusty (14.04) which cherry-picked this fix. If you think that the
> latest upstream version is not affected I can try this too. I only
> tested with VM started through libvirt. If someone tells me how to
> enable memory statistics with plain qemu without libvirt I can test this
> too. My guess would be that this does not make a difference.
>
> I discovered this bug because OpenStack Nova enables memory statistics
> by default since the Juno release. After the QEMU upgrade to the latest
> version in Ubuntu VMs were suddenly shutoff after migration.
>
> Steps to reproduce:
> 1. Create a VM with libvirt which contains a memory balloon device
> defined like this:
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
> <stats period='10'/>
> </memballoon>
>
> 2. Start the VM and let the Linux kernel boot (bug does not appear if
> the kernel is not yet booted, eg. while in the PXE boot phase)
> 3. Issue a managedsave
> 4. Start the VM again
> 5. The VM is restored and "crashes" right after it starts running again.
> 6. You can find the qemu output "qemu-system-x86_64: Virtqueue size
> exceeded" in the log at /var/log/libvirt/vmname.log
I couldn't reproduce this with qemu.git/master (28b874429ba) and a RHEL
7.2 guest.
Which guest distro and kernel version are you using?
Are you doing anything that might cause virtio-balloon activity?
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] Fix for CVE-2016-5403 causes crash on migration if memory stats are enabled
2016-08-12 10:42 ` Stefan Hajnoczi
@ 2016-08-14 20:32 ` Gaudenz Steinlin
2016-08-14 20:40 ` Gaudenz Steinlin
0 siblings, 1 reply; 5+ messages in thread
From: Gaudenz Steinlin @ 2016-08-14 20:32 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel
Hi
Stefan Hajnoczi <stefanha@gmail.com> writes:
> [ Unknown signature status ]
> On Thu, Aug 11, 2016 at 09:18:12AM +0200, Gaudenz Steinlin wrote:
>>
>> [ Please CC me on replies as I'm not subscribed to this list. ]
>>
>> Hi
>>
>> The Fix for CVE-2016-5403 (virtio: error out if guest exceeds virtqueue
>> size)[1] causes qemu to exit(1) after migration or restart from a saved
>> state if memory statistics are enabled in libvirt. Qemu exits after
>> printing "qemu-system-x86_64: Virtqueue size exceeded".
>>
>> I experienced this problem with the latest security update in Ubuntu
>> Trusty (14.04) which cherry-picked this fix. If you think that the
>> latest upstream version is not affected I can try this too. I only
>> tested with VM started through libvirt. If someone tells me how to
>> enable memory statistics with plain qemu without libvirt I can test this
>> too. My guess would be that this does not make a difference.
>>
>> I discovered this bug because OpenStack Nova enables memory statistics
>> by default since the Juno release. After the QEMU upgrade to the latest
>> version in Ubuntu VMs were suddenly shutoff after migration.
>>
>> Steps to reproduce:
>> 1. Create a VM with libvirt which contains a memory balloon device
>> defined like this:
>> <memballoon model='virtio'>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>> <stats period='10'/>
>> </memballoon>
>>
>> 2. Start the VM and let the Linux kernel boot (bug does not appear if
>> the kernel is not yet booted, eg. while in the PXE boot phase)
>> 3. Issue a managedsave
>> 4. Start the VM again
>> 5. The VM is restored and "crashes" right after it starts running again.
>> 6. You can find the qemu output "qemu-system-x86_64: Virtqueue size
>> exceeded" in the log at /var/log/libvirt/vmname.log
>
> I couldn't reproduce this with qemu.git/master (28b874429ba) and a RHEL
> 7.2 guest.
>
> Which guest distro and kernel version are you using?
I just retested and ran into the bug with the following guest OSs:
- Ubuntu 16.04 (Linux ubuntu-1604 4.4.0-24-generic #43-Ubuntu SMP Wed Jun 8 19:27:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux)
- Ubuntu 14.04 (Linux ubuntu-1404 3.13.0-88-generic #135-Ubuntu SMP Wed Jun 8 21:10:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux)
- Debian 8.5 (Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linu)
- Centos 7 (Linux centos 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
- Arch 16.07 (Linux arch 4.6.4-1-ARCH #1 SMP PREEMPT Mon Jul 11 19:12:32 CEST 2016 x86_64 GNU/Linux)
- CoreOS 1010.5.0 (Linux coreos.openstacklocal 4.5.0-coreos-r1 #2 SMP Thu May 26 22:21:06 UTC 2016 x86_64 Intel Xeon E312xx (Sandy Bridge) GenuineIntel GNU/Linux)
So it's reproducible with a wide range of Linux OSes and kernel
versions for me. I used the Ubuntu packaged qemu version
2.0.0+dfsg-2ubuntu1.26. The version 2.0.0+dfsg-2ubuntu1.26 which has the
fix for CVE-2016-5403 reversed does not have the bug. So it seems quite
obvious that at least backporting this fix to 2.0.0 is not safe.
If I can get the latest master to compile I will try this too.
> Are you doing anything that might cause virtio-balloon activity?
How can I check that? I do nothing out of the ordinary and the problem
is present just after the guest OS is fully booted but otherwise
completely idle.
Gaudenz
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] Fix for CVE-2016-5403 causes crash on migration if memory stats are enabled
2016-08-14 20:32 ` Gaudenz Steinlin
@ 2016-08-14 20:40 ` Gaudenz Steinlin
2016-08-16 14:29 ` Stefan Hajnoczi
0 siblings, 1 reply; 5+ messages in thread
From: Gaudenz Steinlin @ 2016-08-14 20:40 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: qemu-devel
Gaudenz Steinlin <gaudenz@debian.org> writes:
> Hi
>
> Stefan Hajnoczi <stefanha@gmail.com> writes:
>
>> [ Unknown signature status ]
>> On Thu, Aug 11, 2016 at 09:18:12AM +0200, Gaudenz Steinlin wrote:
>>>
>>> [ Please CC me on replies as I'm not subscribed to this list. ]
>>>
>>> Hi
>>>
>>> The Fix for CVE-2016-5403 (virtio: error out if guest exceeds virtqueue
>>> size)[1] causes qemu to exit(1) after migration or restart from a saved
>>> state if memory statistics are enabled in libvirt. Qemu exits after
>>> printing "qemu-system-x86_64: Virtqueue size exceeded".
>>>
>>> I experienced this problem with the latest security update in Ubuntu
>>> Trusty (14.04) which cherry-picked this fix. If you think that the
>>> latest upstream version is not affected I can try this too. I only
>>> tested with VM started through libvirt. If someone tells me how to
>>> enable memory statistics with plain qemu without libvirt I can test this
>>> too. My guess would be that this does not make a difference.
>>>
>>> I discovered this bug because OpenStack Nova enables memory statistics
>>> by default since the Juno release. After the QEMU upgrade to the latest
>>> version in Ubuntu VMs were suddenly shutoff after migration.
>>>
>>> Steps to reproduce:
>>> 1. Create a VM with libvirt which contains a memory balloon device
>>> defined like this:
>>> <memballoon model='virtio'>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>>> <stats period='10'/>
>>> </memballoon>
>>>
>>> 2. Start the VM and let the Linux kernel boot (bug does not appear if
>>> the kernel is not yet booted, eg. while in the PXE boot phase)
>>> 3. Issue a managedsave
>>> 4. Start the VM again
>>> 5. The VM is restored and "crashes" right after it starts running again.
>>> 6. You can find the qemu output "qemu-system-x86_64: Virtqueue size
>>> exceeded" in the log at /var/log/libvirt/vmname.log
>>
>> I couldn't reproduce this with qemu.git/master (28b874429ba) and a RHEL
>> 7.2 guest.
>>
>> Which guest distro and kernel version are you using?
>
> I just retested and ran into the bug with the following guest OSs:
> - Ubuntu 16.04 (Linux ubuntu-1604 4.4.0-24-generic #43-Ubuntu SMP Wed Jun 8 19:27:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux)
> - Ubuntu 14.04 (Linux ubuntu-1404 3.13.0-88-generic #135-Ubuntu SMP Wed Jun 8 21:10:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux)
> - Debian 8.5 (Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linu)
> - Centos 7 (Linux centos 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> - Arch 16.07 (Linux arch 4.6.4-1-ARCH #1 SMP PREEMPT Mon Jul 11 19:12:32 CEST 2016 x86_64 GNU/Linux)
> - CoreOS 1010.5.0 (Linux coreos.openstacklocal 4.5.0-coreos-r1 #2 SMP Thu May 26 22:21:06 UTC 2016 x86_64 Intel Xeon E312xx (Sandy Bridge) GenuineIntel GNU/Linux)
>
> So it's reproducible with a wide range of Linux OSes and kernel
> versions for me. I used the Ubuntu packaged qemu version
> 2.0.0+dfsg-2ubuntu1.26. The version 2.0.0+dfsg-2ubuntu1.26 which has the
> fix for CVE-2016-5403 reversed does not have the bug. So it seems quite
> obvious that at least backporting this fix to 2.0.0 is not safe.
See also https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1612089
where Marc Deslauriers from Ubuntu reports that he can reproduce this as
well with Qemu 2.0.0 but not with Qemu 2.6 from Ubuntu Yakkety.
I will try the patches you posted later.
Gaudenz
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Qemu-devel] Fix for CVE-2016-5403 causes crash on migration if memory stats are enabled
2016-08-14 20:40 ` Gaudenz Steinlin
@ 2016-08-16 14:29 ` Stefan Hajnoczi
0 siblings, 0 replies; 5+ messages in thread
From: Stefan Hajnoczi @ 2016-08-16 14:29 UTC (permalink / raw)
To: Gaudenz Steinlin; +Cc: qemu-devel
[-- Attachment #1: Type: text/plain, Size: 3849 bytes --]
On Sun, Aug 14, 2016 at 10:40:48PM +0200, Gaudenz Steinlin wrote:
> Gaudenz Steinlin <gaudenz@debian.org> writes:
>
> > Hi
> >
> > Stefan Hajnoczi <stefanha@gmail.com> writes:
> >
> >> [ Unknown signature status ]
> >> On Thu, Aug 11, 2016 at 09:18:12AM +0200, Gaudenz Steinlin wrote:
> >>>
> >>> [ Please CC me on replies as I'm not subscribed to this list. ]
> >>>
> >>> Hi
> >>>
> >>> The Fix for CVE-2016-5403 (virtio: error out if guest exceeds virtqueue
> >>> size)[1] causes qemu to exit(1) after migration or restart from a saved
> >>> state if memory statistics are enabled in libvirt. Qemu exits after
> >>> printing "qemu-system-x86_64: Virtqueue size exceeded".
> >>>
> >>> I experienced this problem with the latest security update in Ubuntu
> >>> Trusty (14.04) which cherry-picked this fix. If you think that the
> >>> latest upstream version is not affected I can try this too. I only
> >>> tested with VM started through libvirt. If someone tells me how to
> >>> enable memory statistics with plain qemu without libvirt I can test this
> >>> too. My guess would be that this does not make a difference.
> >>>
> >>> I discovered this bug because OpenStack Nova enables memory statistics
> >>> by default since the Juno release. After the QEMU upgrade to the latest
> >>> version in Ubuntu VMs were suddenly shutoff after migration.
> >>>
> >>> Steps to reproduce:
> >>> 1. Create a VM with libvirt which contains a memory balloon device
> >>> defined like this:
> >>> <memballoon model='virtio'>
> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
> >>> <stats period='10'/>
> >>> </memballoon>
> >>>
> >>> 2. Start the VM and let the Linux kernel boot (bug does not appear if
> >>> the kernel is not yet booted, eg. while in the PXE boot phase)
> >>> 3. Issue a managedsave
> >>> 4. Start the VM again
> >>> 5. The VM is restored and "crashes" right after it starts running again.
> >>> 6. You can find the qemu output "qemu-system-x86_64: Virtqueue size
> >>> exceeded" in the log at /var/log/libvirt/vmname.log
> >>
> >> I couldn't reproduce this with qemu.git/master (28b874429ba) and a RHEL
> >> 7.2 guest.
> >>
> >> Which guest distro and kernel version are you using?
> >
> > I just retested and ran into the bug with the following guest OSs:
> > - Ubuntu 16.04 (Linux ubuntu-1604 4.4.0-24-generic #43-Ubuntu SMP Wed Jun 8 19:27:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux)
> > - Ubuntu 14.04 (Linux ubuntu-1404 3.13.0-88-generic #135-Ubuntu SMP Wed Jun 8 21:10:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux)
> > - Debian 8.5 (Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linu)
> > - Centos 7 (Linux centos 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> > - Arch 16.07 (Linux arch 4.6.4-1-ARCH #1 SMP PREEMPT Mon Jul 11 19:12:32 CEST 2016 x86_64 GNU/Linux)
> > - CoreOS 1010.5.0 (Linux coreos.openstacklocal 4.5.0-coreos-r1 #2 SMP Thu May 26 22:21:06 UTC 2016 x86_64 Intel Xeon E312xx (Sandy Bridge) GenuineIntel GNU/Linux)
> >
> > So it's reproducible with a wide range of Linux OSes and kernel
> > versions for me. I used the Ubuntu packaged qemu version
> > 2.0.0+dfsg-2ubuntu1.26. The version 2.0.0+dfsg-2ubuntu1.26 which has the
> > fix for CVE-2016-5403 reversed does not have the bug. So it seems quite
> > obvious that at least backporting this fix to 2.0.0 is not safe.
>
> See also https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1612089
> where Marc Deslauriers from Ubuntu reports that he can reproduce this as
> well with Qemu 2.0.0 but not with Qemu 2.6 from Ubuntu Yakkety.
>
> I will try the patches you posted later.
Okay, the Ubuntu package maintainers will have to take a look.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-08-16 14:30 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-08-11 7:18 [Qemu-devel] Fix for CVE-2016-5403 causes crash on migration if memory stats are enabled Gaudenz Steinlin
2016-08-12 10:42 ` Stefan Hajnoczi
2016-08-14 20:32 ` Gaudenz Steinlin
2016-08-14 20:40 ` Gaudenz Steinlin
2016-08-16 14:29 ` Stefan Hajnoczi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).