[Qemu-devel] Completing big real mode emulation

[Qemu-devel] Completing big real mode emulation

[Mohammed Gamal]

Hello all,
As some of you might know, I've worked on supporting big real mode
emulation on VMX back in GSoC 2008. Looking at the Qemu GSoC ideas
list for this year, I found it among the possible ideas for a GSoC
project. I'd be interested in driving this feature towards completion,
and I have a few questions about it.

- The kernel-space modifications needed to detect an invalid guest
state on VMX and drive emulation from that point was almost complete.
The part that was missing the most, is that the kvm x86 emulator
wasn't complete and didn't support the entire instruction set. I've
seen that the emulator has been the focus of some recent patches
(namely by Gleb Natapov). Is there anything else required to get big
real mode to work correctly on KVM?

- Do we have other problems supporting big real mode on non-VMX
instruction sets? And do we have problems supporting it on the
userspace side?

- Is there anything I am missing?

Regards,
Mohammed

[Qemu-devel] Re: Completing big real mode emulation

[Avi Kivity]

On 03/19/2010 05:29 PM, Mohammed Gamal wrote:
> Hello all,
> As some of you might know, I've worked on supporting big real mode
> emulation on VMX back in GSoC 2008. Looking at the Qemu GSoC ideas
> list for this year, I found it among the possible ideas for a GSoC
> project. I'd be interested in driving this feature towards completion,
> and I have a few questions about it.
>
> - The kernel-space modifications needed to detect an invalid guest
> state on VMX and drive emulation from that point was almost complete.
> The part that was missing the most, is that the kvm x86 emulator
> wasn't complete and didn't support the entire instruction set. I've
> seen that the emulator has been the focus of some recent patches
> (namely by Gleb Natapov). Is there anything else required to get big
> real mode to work correctly on KVM?
>    

IIRC there are some bugs in invalid guest state detection, so it may 
need some work.  We don't support interrupt injection during invalid 
guest state, that will be tricky since it needs access to memory and 
interrupt injection currently happens from atomic context.  Finally, 
there may still be missing instructions.

> - Do we have other problems supporting big real mode on non-VMX
> instruction sets? And do we have problems supporting it on the
> userspace side?
>    

No.

> - Is there anything I am missing?
>
>    

I think that's pretty much it.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

[Qemu-devel] Re: Completing big real mode emulation

[Alexander Graf]

On 20.03.2010, at 08:49, Avi Kivity wrote:

> On 03/19/2010 05:29 PM, Mohammed Gamal wrote:
>> Hello all,
>> As some of you might know, I've worked on supporting big real mode
>> emulation on VMX back in GSoC 2008. Looking at the Qemu GSoC ideas
>> list for this year, I found it among the possible ideas for a GSoC
>> project. I'd be interested in driving this feature towards completion,
>> and I have a few questions about it.
>> 
>> - The kernel-space modifications needed to detect an invalid guest
>> state on VMX and drive emulation from that point was almost complete.
>> The part that was missing the most, is that the kvm x86 emulator
>> wasn't complete and didn't support the entire instruction set. I've
>> seen that the emulator has been the focus of some recent patches
>> (namely by Gleb Natapov). Is there anything else required to get big
>> real mode to work correctly on KVM?
>>   
> 
> IIRC there are some bugs in invalid guest state detection, so it may need some work.  We don't support interrupt injection during invalid guest state, that will be tricky since it needs access to memory and interrupt injection currently happens from atomic context.  Finally, there may still be missing instructions.
> 
>> - Do we have other problems supporting big real mode on non-VMX
>> instruction sets? And do we have problems supporting it on the
>> userspace side?
>>   
> 
> No.

Interestingly enough Hannes just tried to boot a Windows 98 VM on SVM yesterday and failed, while the same VM worked (mostly) with -no-kvm. So apparently there's more missing to it than just big real mode.

I'd say that a GSoC project would rather focus on making a guest OS work than working on generic big real mode. Having Windows 98 support is way more visible to the users. And hopefully more fun to implement too, as it's a visible goal :-).


Alex

[Qemu-devel] Re: Completing big real mode emulation

[Avi Kivity]

On 03/20/2010 10:34 AM, Alexander Graf wrote:
>
> Interestingly enough Hannes just tried to boot a Windows 98 VM on SVM yesterday and failed, while the same VM worked (mostly) with -no-kvm. So apparently there's more missing to it than just big real mode.
>    

Was there an error message?

> I'd say that a GSoC project would rather focus on making a guest OS work than working on generic big real mode. Having Windows 98 support is way more visible to the users. And hopefully more fun to implement too, as it's a visible goal :-).
>    

Big real mode allows you to boot various OSes, such as that old 
Ubuntu/SuSE boot loader which triggered the whole thing.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

[Qemu-devel] Re: Completing big real mode emulation

[Alexander Graf]

On 20.03.2010, at 09:39, Avi Kivity wrote:

> On 03/20/2010 10:34 AM, Alexander Graf wrote:
>> 
>> Interestingly enough Hannes just tried to boot a Windows 98 VM on SVM yesterday and failed, while the same VM worked (mostly) with -no-kvm. So apparently there's more missing to it than just big real mode.
>>   
> 
> Was there an error message?

He got one. I just tried to boot up my test VM and it just hanged in that empty DOS prompt.

> 
>> I'd say that a GSoC project would rather focus on making a guest OS work than working on generic big real mode. Having Windows 98 support is way more visible to the users. And hopefully more fun to implement too, as it's a visible goal :-).
>>   
> 
> Big real mode allows you to boot various OSes, such as that old Ubuntu/SuSE boot loader which triggered the whole thing.

I thought legacy Windows uses it too?

Either way - then we should make the goal of the project to support those old boot loaders. IMHO it should contain visibility. Doing theoretical stuff is just less fun for all parties. Or does that stuff work already?


Alex

[Qemu-devel] Re: Completing big real mode emulation

[Avi Kivity]

On 03/20/2010 10:55 AM, Alexander Graf wrote:
>>
>>> I'd say that a GSoC project would rather focus on making a guest OS work than working on generic big real mode. Having Windows 98 support is way more visible to the users. And hopefully more fun to implement too, as it's a visible goal :-).
>>>
>>>        
>> Big real mode allows you to boot various OSes, such as that old Ubuntu/SuSE boot loader which triggered the whole thing.
>>      
> I thought legacy Windows uses it too?
>    

IIRC even current Windows (last I checked was XP, but it's probably true 
for newer) invokes big real mode inadvertently.  All it takes is not to 
clear fs and gs while switching to real mode.  It works because the real 
mode code never uses gs and fs (i.e. while we are technically in big 
real mode, the guest never relies on this), and because there are enough 
hacks in vmx.c to make it work (restoring fs and gs after the switch 
back).  IIRC there are other cases of invalid guest state that we hack 
into place during mode switches.

> Either way - then we should make the goal of the project to support those old boot loaders. IMHO it should contain visibility. Doing theoretical stuff is just less fun for all parties. Or does that stuff work already?
>    

Mostly those old guests aged beyond usefulness.  They are still broken, 
but nobody installs new images.  Old images installed via workarounds work.

Goals for this task could include:

  - get those older guests working
  - get emulate_invalid_guest_state=1 to work on all supported guests
  - switch to emulate_invalid_guest_state=1 as the default
  - drop the code supporting emulate_invalid_guest_state=0 eventually

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

[Qemu-devel] Re: Completing big real mode emulation

[Mohammed Gamal]

On Sat, Mar 20, 2010 at 3:18 PM, Avi Kivity <avi@redhat.com> wrote:
> On 03/20/2010 10:55 AM, Alexander Graf wrote:
>>>
>>>> I'd say that a GSoC project would rather focus on making a guest OS work
>>>> than working on generic big real mode. Having Windows 98 support is way more
>>>> visible to the users. And hopefully more fun to implement too, as it's a
>>>> visible goal :-).
>>>>
>>>>
>>>
>>> Big real mode allows you to boot various OSes, such as that old
>>> Ubuntu/SuSE boot loader which triggered the whole thing.
>>>
>>
>> I thought legacy Windows uses it too?
>>
>
> IIRC even current Windows (last I checked was XP, but it's probably true for
> newer) invokes big real mode inadvertently.  All it takes is not to clear fs
> and gs while switching to real mode.  It works because the real mode code
> never uses gs and fs (i.e. while we are technically in big real mode, the
> guest never relies on this), and because there are enough hacks in vmx.c to
> make it work (restoring fs and gs after the switch back).  IIRC there are
> other cases of invalid guest state that we hack into place during mode
> switches.
>
>> Either way - then we should make the goal of the project to support those
>> old boot loaders. IMHO it should contain visibility. Doing theoretical stuff
>> is just less fun for all parties. Or does that stuff work already?
>>
>
> Mostly those old guests aged beyond usefulness.  They are still broken, but
> nobody installs new images.  Old images installed via workarounds work.
>
> Goals for this task could include:
>
>  - get those older guests working
>  - get emulate_invalid_guest_state=1 to work on all supported guests
>  - switch to emulate_invalid_guest_state=1 as the default
>  - drop the code supporting emulate_invalid_guest_state=0 eventually

To this end I guess the next logical step is to compile a list of
guests that are currently not working/work with hacks only, and get
them working. Here are some suggestions:
- MINIX 3.1.6 (developers have been recently filing bug reports
because of boot failures)
- Win XP with emulation enabled
- FreeDOS with memory extenders

Any other guests you'd like to see on this list?

[Qemu-devel] Re: Completing big real mode emulation

[Alexander Graf]

Am 20.03.2010 um 15:02 schrieb Mohammed Gamal <m.gamal005@gmail.com>:

> On Sat, Mar 20, 2010 at 3:18 PM, Avi Kivity <avi@redhat.com> wrote:
>> On 03/20/2010 10:55 AM, Alexander Graf wrote:
>>>>
>>>>> I'd say that a GSoC project would rather focus on making a guest  
>>>>> OS work
>>>>> than working on generic big real mode. Having Windows 98 support  
>>>>> is way more
>>>>> visible to the users. And hopefully more fun to implement too,  
>>>>> as it's a
>>>>> visible goal :-).
>>>>>
>>>>>
>>>>
>>>> Big real mode allows you to boot various OSes, such as that old
>>>> Ubuntu/SuSE boot loader which triggered the whole thing.
>>>>
>>>
>>> I thought legacy Windows uses it too?
>>>
>>
>> IIRC even current Windows (last I checked was XP, but it's probably  
>> true for
>> newer) invokes big real mode inadvertently.  All it takes is not to  
>> clear fs
>> and gs while switching to real mode.  It works because the real  
>> mode code
>> never uses gs and fs (i.e. while we are technically in big real  
>> mode, the
>> guest never relies on this), and because there are enough hacks in  
>> vmx.c to
>> make it work (restoring fs and gs after the switch back).  IIRC  
>> there are
>> other cases of invalid guest state that we hack into place during  
>> mode
>> switches.
>>
>>> Either way - then we should make the goal of the project to  
>>> support those
>>> old boot loaders. IMHO it should contain visibility. Doing  
>>> theoretical stuff
>>> is just less fun for all parties. Or does that stuff work already?
>>>
>>
>> Mostly those old guests aged beyond usefulness.  They are still  
>> broken, but
>> nobody installs new images.  Old images installed via workarounds  
>> work.
>>
>> Goals for this task could include:
>>
>>  - get those older guests working
>>  - get emulate_invalid_guest_state=1 to work on all supported guests
>>  - switch to emulate_invalid_guest_state=1 as the default
>>  - drop the code supporting emulate_invalid_guest_state=0 eventually
>
> To this end I guess the next logical step is to compile a list of
> guests that are currently not working/work with hacks only, and get
> them working. Here are some suggestions:
> - MINIX 3.1.6 (developers have been recently filing bug reports
> because of boot failures)
> - Win XP with emulation enabled
> - FreeDOS with memory extenders
>
> Any other guests you'd like to see on this list?

I remember old openSUSE iso bootloaders had issues. I think it was  
around 10.3, but might have been earlier.

Alex

[Qemu-devel] Re: Completing big real mode emulation

[Sheng Yang]

On Saturday 20 March 2010 23:00:49 Alexander Graf wrote:
> Am 20.03.2010 um 15:02 schrieb Mohammed Gamal <m.gamal005@gmail.com>:
> > On Sat, Mar 20, 2010 at 3:18 PM, Avi Kivity <avi@redhat.com> wrote:
> >> On 03/20/2010 10:55 AM, Alexander Graf wrote:
> >>>>> I'd say that a GSoC project would rather focus on making a guest
> >>>>> OS work
> >>>>> than working on generic big real mode. Having Windows 98 support
> >>>>> is way more
> >>>>> visible to the users. And hopefully more fun to implement too,
> >>>>> as it's a
> >>>>> visible goal :-).
> >>>>
> >>>> Big real mode allows you to boot various OSes, such as that old
> >>>> Ubuntu/SuSE boot loader which triggered the whole thing.
> >>>
> >>> I thought legacy Windows uses it too?
> >>
> >> IIRC even current Windows (last I checked was XP, but it's probably
> >> true for
> >> newer) invokes big real mode inadvertently.  All it takes is not to
> >> clear fs
> >> and gs while switching to real mode.  It works because the real
> >> mode code
> >> never uses gs and fs (i.e. while we are technically in big real
> >> mode, the
> >> guest never relies on this), and because there are enough hacks in
> >> vmx.c to
> >> make it work (restoring fs and gs after the switch back).  IIRC
> >> there are
> >> other cases of invalid guest state that we hack into place during
> >> mode
> >> switches.
> >>
> >>> Either way - then we should make the goal of the project to
> >>> support those
> >>> old boot loaders. IMHO it should contain visibility. Doing
> >>> theoretical stuff
> >>> is just less fun for all parties. Or does that stuff work already?
> >>
> >> Mostly those old guests aged beyond usefulness.  They are still
> >> broken, but
> >> nobody installs new images.  Old images installed via workarounds
> >> work.
> >>
> >> Goals for this task could include:
> >>
> >>  - get those older guests working
> >>  - get emulate_invalid_guest_state=1 to work on all supported guests
> >>  - switch to emulate_invalid_guest_state=1 as the default
> >>  - drop the code supporting emulate_invalid_guest_state=0 eventually
> >
> > To this end I guess the next logical step is to compile a list of
> > guests that are currently not working/work with hacks only, and get
> > them working. Here are some suggestions:
> > - MINIX 3.1.6 (developers have been recently filing bug reports
> > because of boot failures)
> > - Win XP with emulation enabled
> > - FreeDOS with memory extenders
> >
> > Any other guests you'd like to see on this list?
> 
> I remember old openSUSE iso bootloaders had issues. I think it was
> around 10.3, but might have been earlier.
> 
At least 10u2 installer has trouble. I had spent some time on it, finally 
found it's due to ISOLINUX.

The basic issue is it assume that SS selector/base is unchanged when 
enter/exit protect mode. At that time, I've cooked a hack workaround for it, 
but didn't think it's proper to upstream.

-- 
regards
Yang, Sheng

Re: [Qemu-devel] Re: Completing big real mode emulation

[Jamie Lokier]

Avi Kivity wrote:
> >Either way - then we should make the goal of the project to support those 
> >old boot loaders. IMHO it should contain visibility. Doing theoretical 
> >stuff is just less fun for all parties. Or does that stuff work already?
> 
> Mostly those old guests aged beyond usefulness.  They are still broken, 
> but nobody installs new images.  Old images installed via workarounds work.

Hey :) I still install old OSes occasionally, so that I can build and
test code that will run on other people's still-running old machines.

-- Jamie