Questions on the VMentry failure patch

Questions on the VMentry failure patch

[Mohammed Gamal]

Hello,
Guillaume Thouvenin's VMentry failure detection patch was reverted
before because it broke guest rebooting. Although it turned out it
didn't, it was preferred not to re-include it because it was not
reliable enough. Although it is working fine with me, I want to know
why it wasn't re-included? Did it cause any regressions? Is it
preferrable to use an alternative mechanism to detect the VM entry
failure and start real-mode emulation? If yes, What do you think it
should be?

Regards,
Mohammed

Re: Questions on the VMentry failure patch

[Avi Kivity]

Mohammed Gamal wrote:
> Hello,
> Guillaume Thouvenin's VMentry failure detection patch was reverted
> before because it broke guest rebooting. Although it turned out it
> didn't, it was preferred not to re-include it because it was not
> reliable enough. Although it is working fine with me, I want to know
> why it wasn't re-included? Did it cause any regressions? Is it
> preferrable to use an alternative mechanism to detect the VM entry
> failure and start real-mode emulation? If yes, What do you think it
> should be?
>
>   

Anthony Liguori reported problems with Ubuntu IIRC.  I'd like to know 
that they are fixed, or what exactly the issue was.


-- 
error compiling committee.c: too many arguments to function

Re: Questions on the VMentry failure patch

[Mohammed Gamal]

On Mon, Jul 7, 2008 at 5:24 PM, Avi Kivity <avi@qumranet.com> wrote:
> Mohammed Gamal wrote:
>>
>> Hello,
>> Guillaume Thouvenin's VMentry failure detection patch was reverted
>> before because it broke guest rebooting. Although it turned out it
>> didn't, it was preferred not to re-include it because it was not
>> reliable enough. Although it is working fine with me, I want to know
>> why it wasn't re-included? Did it cause any regressions? Is it
>> preferrable to use an alternative mechanism to detect the VM entry
>> failure and start real-mode emulation? If yes, What do you think it
>> should be?
>>
>>
>
> Anthony Liguori reported problems with Ubuntu IIRC.  I'd like to know that
> they are fixed, or what exactly the issue was.
>
Well, Ubuntu and openSuSE live CDs used to crash occasionally after
booting, it didn't happen most of the time though. I've never
experienced this issue after kvm-70 - with the patches the patch
included of course. Did anyone have any issues with the patches
included?

Re: Questions on the VMentry failure patch

[Avi Kivity]

Mohammed Gamal wrote:
> On Mon, Jul 7, 2008 at 5:24 PM, Avi Kivity <avi@qumranet.com> wrote:
>   
>> Mohammed Gamal wrote:
>>     
>>> Hello,
>>> Guillaume Thouvenin's VMentry failure detection patch was reverted
>>> before because it broke guest rebooting. Although it turned out it
>>> didn't, it was preferred not to re-include it because it was not
>>> reliable enough. Although it is working fine with me, I want to know
>>> why it wasn't re-included? Did it cause any regressions? Is it
>>> preferrable to use an alternative mechanism to detect the VM entry
>>> failure and start real-mode emulation? If yes, What do you think it
>>> should be?
>>>
>>>
>>>       
>> Anthony Liguori reported problems with Ubuntu IIRC.  I'd like to know that
>> they are fixed, or what exactly the issue was.
>>
>>     
> Well, Ubuntu and openSuSE live CDs used to crash occasionally after
> booting, it didn't happen most of the time though. I've never
> experienced this issue after kvm-70 - with the patches the patch
> included of course. Did anyone have any issues with the patches
> included?
>   

I think they're fine.  Can you check the vmentry patch now, with 
Anthony's testcase?

-- 
error compiling committee.c: too many arguments to function

Re: Questions on the VMentry failure patch

[Anthony Liguori]

Avi Kivity wrote:
> Mohammed Gamal wrote:
>> On Mon, Jul 7, 2008 at 5:24 PM, Avi Kivity <avi@qumranet.com> wrote:
>>  
>>> Mohammed Gamal wrote:
>>>    
>>>> Hello,
>>>> Guillaume Thouvenin's VMentry failure detection patch was reverted
>>>> before because it broke guest rebooting. Although it turned out it
>>>> didn't, it was preferred not to re-include it because it was not
>>>> reliable enough. Although it is working fine with me, I want to know
>>>> why it wasn't re-included? Did it cause any regressions? Is it
>>>> preferrable to use an alternative mechanism to detect the VM entry
>>>> failure and start real-mode emulation? If yes, What do you think it
>>>> should be?
>>>>
>>>>
>>>>       
>>> Anthony Liguori reported problems with Ubuntu IIRC.  I'd like to 
>>> know that
>>> they are fixed, or what exactly the issue was.
>>>
>>>     
>> Well, Ubuntu and openSuSE live CDs used to crash occasionally after
>> booting, it didn't happen most of the time though. I've never
>> experienced this issue after kvm-70 - with the patches the patch
>> included of course. Did anyone have any issues with the patches
>> included?
>>   
>
> I think they're fine.  Can you check the vmentry patch now, with 
> Anthony's testcase?

If I revert 0eba1b, then use the following command line:

sudo qemu-system-x86_64 -hda ~/images/10gig.img -cdrom 
~/images/isos/ubuntu-7.10-server-amd64.iso -snapshot -boot dc

After about the 7th or so run, I get:

kvm_run: failed entry, reason 5
kvm_run returned -8

So the problem still exists.

Regards,

Anthony Liguori

Re: Questions on the VMentry failure patch

[Anthony Liguori]

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

Avi Kivity wrote:
> Mohammed Gamal wrote:
>
> I think they're fine.  Can you check the vmentry patch now, with 
> Anthony's testcase?

The following patch makes the problem almost 100% reproducible on my 
system.  This strongly suggests a timing issue.

Regards,

Anthony Liguori


[-- Attachment #2: foo.patch --]
[-- Type: application/mbox, Size: 391 bytes --]

Re: Questions on the VMentry failure patch

[Mohammed Gamal]

On Mon, Jul 7, 2008 at 6:29 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Avi Kivity wrote:
>>
>> Mohammed Gamal wrote:
>>
>> I think they're fine.  Can you check the vmentry patch now, with Anthony's
>> testcase?
>
> The following patch makes the problem almost 100% reproducible on my system.
>  This strongly suggests a timing issue.
>
> Regards,
>
> Anthony Liguori
>
>
 It's true indeed, the patch did increase the likelihood of the
problem with me (although it occurs every few runs). I modified
invalid_guest_state() to call kvm_report_emulation_failure() in all
cases and I noticed that whenever the crash happens it happens here:

rip 6e10 66 b8 20 00

It's too late at night here, so I'll not lookup the opcode map now :)
. I'll further look into it later.

Re: Questions on the VMentry failure patch

[Mohammed Gamal]

On Tue, Jul 8, 2008 at 3:52 AM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
> On Mon, Jul 7, 2008 at 6:29 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
>> Avi Kivity wrote:
>>>
>>> Mohammed Gamal wrote:
>>>
>>> I think they're fine.  Can you check the vmentry patch now, with Anthony's
>>> testcase?
>>
>> The following patch makes the problem almost 100% reproducible on my system.
>>  This strongly suggests a timing issue.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>
>  It's true indeed, the patch did increase the likelihood of the
> problem with me (although it occurs every few runs). I modified
> invalid_guest_state() to call kvm_report_emulation_failure() in all
> cases and I noticed that whenever the crash happens it happens here:
>
> rip 6e10 66 b8 20 00
>
> It's too late at night here, so I'll not lookup the opcode map now :)
> . I'll further look into it later.
>

Another thing, I tried -no-kvm-pit switch and it tremendously increase
the likelihood of the crash to almost a 100%.

Re: Questions on the VMentry failure patch

[Mohammed Gamal]

>>  It's true indeed, the patch did increase the likelihood of the
>> problem with me (although it occurs every few runs). I modified
>> invalid_guest_state() to call kvm_report_emulation_failure() in all
>> cases and I noticed that whenever the crash happens it happens here:
>>
>> rip 6e10 66 b8 20 00
>>
>> It's too late at night here, so I'll not lookup the opcode map now :)
>> . I'll further look into it later.
>>
>
> Another thing, I tried -no-kvm-pit switch and it tremendously increase
> the likelihood of the crash to almost a 100%.
>

I updated to the latest kvm-userspace git tree, and now the failure is
happening at completely random instructions whether or not we are
using -no-kvm-pit.

Re: Questions on the VMentry failure patch

[Mohammed Gamal]

On Thu, Jul 10, 2008 at 4:48 PM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>>>  It's true indeed, the patch did increase the likelihood of the
>>> problem with me (although it occurs every few runs). I modified
>>> invalid_guest_state() to call kvm_report_emulation_failure() in all
>>> cases and I noticed that whenever the crash happens it happens here:
>>>
>>> rip 6e10 66 b8 20 00
>>>
>>> It's too late at night here, so I'll not lookup the opcode map now :)
>>> . I'll further look into it later.
>>>
>>
>> Another thing, I tried -no-kvm-pit switch and it tremendously increase
>> the likelihood of the crash to almost a 100%.
>>
>
> I updated to the latest kvm-userspace git tree, and now the failure is
> happening at completely random instructions whether or not we are
> using -no-kvm-pit.
>

I didn't have the gfxboot source code in hand, but now that I've got
it. It clears out that the failure always occurs in the
switch_to_pm_20 routine. However, the failure doesn't happen at one
particular instruction, but either doesn't happen at all or happens at
any instruction between addresses 6e10 and 6e27.

I'm suspecting it might be some kind of a race condition, although I
don't see where in the code - kernel side to specific - that this race
exactly might occur. Maybe the locking changes in the userspace side
helped some underlying issue to come up to the surface just like what
happened with FreeDOS. I'll look further into it, any
pointers/help/suggestions are appreciated.

Re: Questions on the VMentry failure patch

[Avi Kivity]

Mohammed Gamal wrote:
> On Thu, Jul 10, 2008 at 4:48 PM, Mohammed Gamal <m.gamal005@gmail.com> wrote:
>   
>>>>  It's true indeed, the patch did increase the likelihood of the
>>>> problem with me (although it occurs every few runs). I modified
>>>> invalid_guest_state() to call kvm_report_emulation_failure() in all
>>>> cases and I noticed that whenever the crash happens it happens here:
>>>>
>>>> rip 6e10 66 b8 20 00
>>>>
>>>> It's too late at night here, so I'll not lookup the opcode map now :)
>>>> . I'll further look into it later.
>>>>
>>>>         
>>> Another thing, I tried -no-kvm-pit switch and it tremendously increase
>>> the likelihood of the crash to almost a 100%.
>>>
>>>       
>> I updated to the latest kvm-userspace git tree, and now the failure is
>> happening at completely random instructions whether or not we are
>> using -no-kvm-pit.
>>
>>     
>
> I didn't have the gfxboot source code in hand, but now that I've got
> it. It clears out that the failure always occurs in the
> switch_to_pm_20 routine. However, the failure doesn't happen at one
> particular instruction, but either doesn't happen at all or happens at
> any instruction between addresses 6e10 and 6e27.
>
> I'm suspecting it might be some kind of a race condition, although I
> don't see where in the code - kernel side to specific - that this race
> exactly might occur. Maybe the locking changes in the userspace side
> helped some underlying issue to come up to the surface just like what
> happened with FreeDOS. I'll look further into it, any
> pointers/help/suggestions are appreciated.
>   

I suspected an interrupt, which fits the scenario you describe.  
Although Anthony tested this and found out interrupts were not involved, 
IIRC.

-- 
error compiling committee.c: too many arguments to function

Re: Questions on the VMentry failure patch

[Anthony Liguori]

Avi Kivity wrote:
> Mohammed Gamal wrote:
>> On Thu, Jul 10, 2008 at 4:48 PM, Mohammed Gamal 
>> <m.gamal005@gmail.com> wrote:
>>  
>>>>>  It's true indeed, the patch did increase the likelihood of the
>>>>> problem with me (although it occurs every few runs). I modified
>>>>> invalid_guest_state() to call kvm_report_emulation_failure() in all
>>>>> cases and I noticed that whenever the crash happens it happens here:
>>>>>
>>>>> rip 6e10 66 b8 20 00
>>>>>
>>>>> It's too late at night here, so I'll not lookup the opcode map now :)
>>>>> . I'll further look into it later.
>>>>>
>>>>>         
>>>> Another thing, I tried -no-kvm-pit switch and it tremendously increase
>>>> the likelihood of the crash to almost a 100%.
>>>>
>>>>       
>>> I updated to the latest kvm-userspace git tree, and now the failure is
>>> happening at completely random instructions whether or not we are
>>> using -no-kvm-pit.
>>>
>>>     
>>
>> I didn't have the gfxboot source code in hand, but now that I've got
>> it. It clears out that the failure always occurs in the
>> switch_to_pm_20 routine. However, the failure doesn't happen at one
>> particular instruction, but either doesn't happen at all or happens at
>> any instruction between addresses 6e10 and 6e27.
>>
>> I'm suspecting it might be some kind of a race condition, although I
>> don't see where in the code - kernel side to specific - that this race
>> exactly might occur. Maybe the locking changes in the userspace side
>> helped some underlying issue to come up to the surface just like what
>> happened with FreeDOS. I'll look further into it, any
>> pointers/help/suggestions are appreciated.
>>   
>
> I suspected an interrupt, which fits the scenario you describe.  
> Although Anthony tested this and found out interrupts were not 
> involved, IIRC.

It was at least not occurring for me when an interrupt was being 
injected.  That's not to say that it isn't somehow affected by interrupt 
injections but it doesn't appear to be the obvious interrupt occurs 
while doing switch_to_pm_20.

Regards,

Anthony Liguori

Re: Questions on the VMentry failure patch

[Avi Kivity]

Anthony Liguori wrote:
>>
>> I suspected an interrupt, which fits the scenario you describe.  
>> Although Anthony tested this and found out interrupts were not 
>> involved, IIRC.
>
> It was at least not occurring for me when an interrupt was being 
> injected.  That's not to say that it isn't somehow affected by 
> interrupt injections but it doesn't appear to be the obvious interrupt 
> occurs while doing switch_to_pm_20.

A good thing to check is whether guest interrupts are enabled at that 
point.  I suspect they aren't.

It might be a host interrupt, forcing an exit (and an entry) at a random 
point, which then fails.

-- 
error compiling committee.c: too many arguments to function

Re: Questions on the VMentry failure patch

[Mohammed Gamal]

On Mon, Jul 14, 2008 at 7:24 PM, Avi Kivity <avi@qumranet.com> wrote:
> Anthony Liguori wrote:
>>>
>>> I suspected an interrupt, which fits the scenario you describe.  Although
>>> Anthony tested this and found out interrupts were not involved, IIRC.
>>
>> It was at least not occurring for me when an interrupt was being injected.
>>  That's not to say that it isn't somehow affected by interrupt injections
>> but it doesn't appear to be the obvious interrupt occurs while doing
>> switch_to_pm_20.
>
> A good thing to check is whether guest interrupts are enabled at that point.
>  I suspect they aren't.
>

You are right indeed. IF isn't set in the guest's rflags the only bits
set are NT and IOPL=1

Re: Questions on the VMentry failure patch

[Mohammed Gamal]

> It might be a host interrupt, forcing an exit (and an entry) at a random
> point, which then fails.
>
> It might be a host interrupt, forcing an exit (and an entry) at a random
> point, which then fails.
>
I tried to disable vmexits on external interrupts in enter_rmode().
And I get system lockups randomly as well. So I guess you're right
about this too.

Re: Questions on the VMentry failure patch

[Avi Kivity]

Mohammed Gamal wrote:
>> It might be a host interrupt, forcing an exit (and an entry) at a random
>> point, which then fails.
>>
>> It might be a host interrupt, forcing an exit (and an entry) at a random
>> point, which then fails.
>>
>>     
> I tried to disable vmexits on external interrupts in enter_rmode().
> And I get system lockups randomly as well. So I guess you're right
> about this too.
>   

Disabling vmexits on external interrupts will certainly lock up your 
host.  As soon as the guest pages in enough memory to run a loop waiting 
for an interrupt, nothing will cause it to come out.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.