bad leakage between virtual machines?

bad leakage between virtual machines?

[Tom Horsley]

I just submitted this fedora bug with lots of details:

https://bugzilla.redhat.com/show_bug.cgi?id=555788

It sure seems to me like the contents of the debug
address trap registers are leaking across virtual machines
(at least for opterons on this motherboard :-).

Anyone seen anything like this before? (I've been
having similar problems on this host since it was
running xen 3.1 before switching to kvm).

Re: bad leakage between virtual machines?

[Jan Kiszka]

Tom Horsley wrote:
> I just submitted this fedora bug with lots of details:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=555788
> 
> It sure seems to me like the contents of the debug
> address trap registers are leaking across virtual machines
> (at least for opterons on this motherboard :-).

Are only AMD boxes affected, or did you also test on Intel?

> 
> Anyone seen anything like this before? (I've been
> having similar problems on this host since it was
> running xen 3.1 before switching to kvm).

I still have one debug register oddity on my todo list (that one causes
an unhandled VM exit), but I'm not aware of such an issue nor do I
remember any patch between 2.6.31.x and now at the moment that may have
fixed it. Nevertheless, could you retry with latest kvm kernel or a
recent kvm-kmod release?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: bad leakage between virtual machines?

[Tom Horsley]

On Fri, 15 Jan 2010 18:37:57 +0100
Jan Kiszka wrote:

> Tom Horsley wrote:
> > I just submitted this fedora bug with lots of details:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=555788
> > 
> > It sure seems to me like the contents of the debug
> > address trap registers are leaking across virtual machines
> > (at least for opterons on this motherboard :-).
> 
> Are only AMD boxes affected, or did you also test on Intel?

I haven't explicitly tested on Intel, but we did just see a
compile on a guest on the Intel host also get the same
Breakpoint/Trap error out of the blue, so it looks as if it
does happen on intel as well as amd.

I'm going to try on my home system (intel) some more this
evening when I get home.

Some of the symptoms sort of lead me to think it might be
inheriting random leftover DRn registers at the time a
new virtual machine is created since a VM that has problems
seems to keep having problems, and other VMs zip along
with no problems at all.

> 
> > 
> > Anyone seen anything like this before? (I've been
> > having similar problems on this host since it was
> > running xen 3.1 before switching to kvm).
> 
> I still have one debug register oddity on my todo list (that one causes
> an unhandled VM exit), but I'm not aware of such an issue nor do I
> remember any patch between 2.6.31.x and now at the moment that may have
> fixed it. Nevertheless, could you retry with latest kvm kernel or a
> recent kvm-kmod release?

Not sure how to do that - if it isn't in the fedora repos, I'm in
uncharted territory trying to run special kernels :-).

> 
> Jan
> 

Re: bad leakage between virtual machines?

[Jan Kiszka]

Tom Horsley wrote:
> On Fri, 15 Jan 2010 18:37:57 +0100
> Jan Kiszka wrote:
> 
>> Tom Horsley wrote:
>>> I just submitted this fedora bug with lots of details:
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=555788
>>>
>>> It sure seems to me like the contents of the debug
>>> address trap registers are leaking across virtual machines
>>> (at least for opterons on this motherboard :-).
>> Are only AMD boxes affected, or did you also test on Intel?
> 
> I haven't explicitly tested on Intel, but we did just see a
> compile on a guest on the Intel host also get the same
> Breakpoint/Trap error out of the blue, so it looks as if it
> does happen on intel as well as amd.

OK, will have a look at your test application once time permits.

> 
> I'm going to try on my home system (intel) some more this
> evening when I get home.
> 
> Some of the symptoms sort of lead me to think it might be
> inheriting random leftover DRn registers at the time a
> new virtual machine is created since a VM that has problems
> seems to keep having problems, and other VMs zip along
> with no problems at all.

We have a lazy scheme for switching DR content, but we always clear any
triggering bit in DR7 before returning to Linux (which includes other VMs).

> 
>>> Anyone seen anything like this before? (I've been
>>> having similar problems on this host since it was
>>> running xen 3.1 before switching to kvm).
>> I still have one debug register oddity on my todo list (that one causes
>> an unhandled VM exit), but I'm not aware of such an issue nor do I
>> remember any patch between 2.6.31.x and now at the moment that may have
>> fixed it. Nevertheless, could you retry with latest kvm kernel or a
>> recent kvm-kmod release?
> 
> Not sure how to do that - if it isn't in the fedora repos, I'm in
> uncharted territory trying to run special kernels :-).

kvm-kmod should only be about ./configure && make. You do not even need
to install it (if you want to keep your host "clean"), just 'rmmod
kvm-amd kvm' and insmod the kvm.ko and kvm-amd.ko that is in
kvm-kmod/x86 after the build.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: bad leakage between virtual machines?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

Jan Kiszka wrote:
> Tom Horsley wrote:
>> On Fri, 15 Jan 2010 18:37:57 +0100
>> Jan Kiszka wrote:
>>
>>> Tom Horsley wrote:
>>>> I just submitted this fedora bug with lots of details:
>>>>
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=555788
>>>>
>>>> It sure seems to me like the contents of the debug
>>>> address trap registers are leaking across virtual machines
>>>> (at least for opterons on this motherboard :-).
>>> Are only AMD boxes affected, or did you also test on Intel?
>> I haven't explicitly tested on Intel, but we did just see a
>> compile on a guest on the Intel host also get the same
>> Breakpoint/Trap error out of the blue, so it looks as if it
>> does happen on intel as well as amd.
> 
> OK, will have a look at your test application once time permits.
> 

Just did so with bleeding-edge kvm-kmod and qemu-kvm-0.12.1.2 on an
Intel host. But nothing exciting happened.

Are you sure that test on an Intel host was really an Intel? In your bug
report you write the contrary. Anyway, will have my hands on an AMD box
tomorrow and try to reproduce there.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: bad leakage between virtual machines?

[Jan Kiszka]

Jan Kiszka wrote:
> Jan Kiszka wrote:
>> Tom Horsley wrote:
>>> On Fri, 15 Jan 2010 18:37:57 +0100
>>> Jan Kiszka wrote:
>>>
>>>> Tom Horsley wrote:
>>>>> I just submitted this fedora bug with lots of details:
>>>>>
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=555788
>>>>>
>>>>> It sure seems to me like the contents of the debug
>>>>> address trap registers are leaking across virtual machines
>>>>> (at least for opterons on this motherboard :-).
>>>> Are only AMD boxes affected, or did you also test on Intel?
>>> I haven't explicitly tested on Intel, but we did just see a
>>> compile on a guest on the Intel host also get the same
>>> Breakpoint/Trap error out of the blue, so it looks as if it
>>> does happen on intel as well as amd.
>> OK, will have a look at your test application once time permits.
>>
> 
> Just did so with bleeding-edge kvm-kmod and qemu-kvm-0.12.1.2 on an
> Intel host. But nothing exciting happened.
> 
> Are you sure that test on an Intel host was really an Intel? In your bug
> report you write the contrary. Anyway, will have my hands on an AMD box
> tomorrow and try to reproduce there.

No success reproducing on AMD either. This time I'm using
kvm-kmod-2.6.32.3 (so none of today's debug register patches applied)
and qemu-kvm-0.12.1.2.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux