Ubuntu

Ubuntu

[Benjamin Herrenschmidt]

Hoy !

So it seems to be making fwd progress in fact in the karmic installer,
but it's -extremely- slow. Not sure what's up yet, haven't had a chance
to really instrument it yet.

Let me know if you want to toy with it. Basically, what I do to get
debug is that I take the initrd out of the ISO, and I run with:

 ./qemu-system-ppc [-enable-kvm ... or not] -cdrom <path_to_the_iso>
-boot order=d -m 512 -initrd=<path to their initrd copied to your disk>
-kernel <path to a kernel to test with so you can add debug etc...>
-append "file=/cdrom/preseed/ubuntu.seed debug="

In a few places it looks stuck, but it's just terribly slow. However
it's not using much CPU on the host, seems to be sleeping mostly. Could
be something wrong with decrementer emulation ?

Cheers,
Ben.

Re: Ubuntu

[Alexander Graf]

Am 12.11.2009 um 01:34 schrieb Benjamin Herrenschmidt <benh@kernel.crashing.org 
 >:

> Hoy !
>
> So it seems to be making fwd progress in fact in the karmic installer,
> but it's -extremely- slow. Not sure what's up yet, haven't had a  
> chance
> to really instrument it yet.
>
> Let me know if you want to toy with it. Basically, what I do to get
> debug is that I take the initrd out of the ISO, and I run with:
>
> ./qemu-system-ppc [-enable-kvm ... or not] -cdrom <path_to_the_iso>
> -boot order=d -m 512 -initrd=<path to their initrd copied to your  
> disk>
> -kernel <path to a kernel to test with so you can add debug etc...>
> -append "file=/cdrom/preseed/ubuntu.seed debug="
>
> In a few places it looks stuck, but it's just terribly slow. However
> it's not using much CPU on the host, seems to be sleeping mostly.  
> Could
> be something wrong with decrementer emulation ?

Oh, right. At one point in time something wrt interrupts broke, so  
you'll see that it's waiting for a dma request.

Boy how I hate this whole qdev conversion.

Alex

>
> Cheers,
> Ben.
>
>

Re: Ubuntu

[Benjamin Herrenschmidt]

On Thu, 2009-11-12 at 11:19 +0100, Alexander Graf wrote:
> Am 12.11.2009 um 01:34 schrieb Benjamin Herrenschmidt <benh@kernel.crashing.org 
>  >:
> 
> > Hoy !
> >
> > So it seems to be making fwd progress in fact in the karmic installer,
> > but it's -extremely- slow. Not sure what's up yet, haven't had a  
> > chance
> > to really instrument it yet.
> >
> > Let me know if you want to toy with it. Basically, what I do to get
> > debug is that I take the initrd out of the ISO, and I run with:
> >
> > ./qemu-system-ppc [-enable-kvm ... or not] -cdrom <path_to_the_iso>
> > -boot order=d -m 512 -initrd=<path to their initrd copied to your  
> > disk>
> > -kernel <path to a kernel to test with so you can add debug etc...>
> > -append "file=/cdrom/preseed/ubuntu.seed debug="
> >
> > In a few places it looks stuck, but it's just terribly slow. However
> > it's not using much CPU on the host, seems to be sleeping mostly.  
> > Could
> > be something wrong with decrementer emulation ?
> 
> Oh, right. At one point in time something wrt interrupts broke, so  
> you'll see that it's waiting for a dma request.
> 
> Boy how I hate this whole qdev conversion.

Hehehe ok. Well, without -enable-kvm works :-) Let me know if you find
something, I won't have time to dig until next week. BTW. I've been
toying about some ideas to improve the DEC emulation too, we'll talk
about it later too. But base idea is when DEC is written, you calculate
the "target" timebase value and store that. Makes it easy to re-evaluate
on exit whether you need to trigger a DEC interrupt by comparing with
the current TB...

Cheers,
Ben.

> Alex
> 
> >
> > Cheers,
> > Ben.
> >
> >

Re: Ubuntu

[Alexander Graf]

On 12.11.2009, at 11:28, Benjamin Herrenschmidt wrote:

> On Thu, 2009-11-12 at 11:19 +0100, Alexander Graf wrote:
>> Am 12.11.2009 um 01:34 schrieb Benjamin Herrenschmidt <benh@kernel.crashing.org
>>> :
>>
>>> Hoy !
>>>
>>> So it seems to be making fwd progress in fact in the karmic  
>>> installer,
>>> but it's -extremely- slow. Not sure what's up yet, haven't had a
>>> chance
>>> to really instrument it yet.
>>>
>>> Let me know if you want to toy with it. Basically, what I do to get
>>> debug is that I take the initrd out of the ISO, and I run with:
>>>
>>> ./qemu-system-ppc [-enable-kvm ... or not] -cdrom <path_to_the_iso>
>>> -boot order=d -m 512 -initrd=<path to their initrd copied to your
>>> disk>
>>> -kernel <path to a kernel to test with so you can add debug etc...>
>>> -append "file=/cdrom/preseed/ubuntu.seed debug="
>>>
>>> In a few places it looks stuck, but it's just terribly slow. However
>>> it's not using much CPU on the host, seems to be sleeping mostly.
>>> Could
>>> be something wrong with decrementer emulation ?
>>
>> Oh, right. At one point in time something wrt interrupts broke, so
>> you'll see that it's waiting for a dma request.
>>
>> Boy how I hate this whole qdev conversion.
>
> Hehehe ok. Well, without -enable-kvm works :-) Let me know if you find
> something, I won't have time to dig until next week. BTW. I've been
> toying about some ideas to improve the DEC emulation too, we'll talk
> about it later too. But base idea is when DEC is written, you  
> calculate
> the "target" timebase value and store that. Makes it easy to re- 
> evaluate
> on exit whether you need to trigger a DEC interrupt by comparing with
> the current TB...

Hum, that would defer DEC interrupts to "the next exit". So if your  
guest is sitting in a spin lock for example, it wouldn't get a DEC  
because it doesn't exit from the guest context to actually check if  
TARGET_DEC < TB.

I think the only really good way of doing this is to use the hrtimer  
to trigger the interrupt, not clear the interrupt injection bit, but  
only clear it on mtdec.

That way we get the interrupt to actually interrupt the guest context,  
but instantly trigger the DEC in the guest when it sets IF=1.

I think I had a define for an "alternative" DEC method in book3s.c,  
that basically checks mfdec() against 0 on every exit. Feel free to  
try out of that one helps :-).


Alex

Re: Ubuntu

[Benjamin Herrenschmidt]

> Hum, that would defer DEC interrupts to "the next exit". So if your  
> guest is sitting in a spin lock for example, it wouldn't get a DEC  
> because it doesn't exit from the guest context to actually check if  
> TARGET_DEC < TB.

Ah no, you still set a target..

> I think the only really good way of doing this is to use the hrtimer  
> to trigger the interrupt, not clear the interrupt injection bit, but  
> only clear it on mtdec.
> 
> That way we get the interrupt to actually interrupt the guest context,  
> but instantly trigger the DEC in the guest when it sets IF=1.
> 
> I think I had a define for an "alternative" DEC method in book3s.c,  
> that basically checks mfdec() against 0 on every exit. Feel free to  
> try out of that one helps :-).

I saw that. My idea was more low-level:

 - Guest set DEC, you calculate & cache target
 - On exit to guest, you check target against TB, if passed, then a DEC
   interrupt is pending for guest. If EE is set, you "deliver" it.
 - If not passed, the substraction give you a DEC value (time to next
   DEC expected by guest). You set the DEC to min(DEC,value), ie, you
   make it fire the soonest of the guest DEC target vs. linux host guest
   target. Linux can cope just fine with spurrious DEC interrupts btw.

You have to do some modulo 32 bit etc... but basically, it should work
and it sounds simpler to me and will avoid the churn with timers etc...

Cheers,
Ben.

Re: Ubuntu

[Alexander Graf]

On 12.11.2009, at 11:53, Benjamin Herrenschmidt wrote:

>
>> Hum, that would defer DEC interrupts to "the next exit". So if your
>> guest is sitting in a spin lock for example, it wouldn't get a DEC
>> because it doesn't exit from the guest context to actually check if
>> TARGET_DEC < TB.
>
> Ah no, you still set a target..
>
>> I think the only really good way of doing this is to use the hrtimer
>> to trigger the interrupt, not clear the interrupt injection bit, but
>> only clear it on mtdec.
>>
>> That way we get the interrupt to actually interrupt the guest  
>> context,
>> but instantly trigger the DEC in the guest when it sets IF=1.
>>
>> I think I had a define for an "alternative" DEC method in book3s.c,
>> that basically checks mfdec() against 0 on every exit. Feel free to
>> try out of that one helps :-).
>
> I saw that. My idea was more low-level:
>
> - Guest set DEC, you calculate & cache target
> - On exit to guest, you check target against TB, if passed, then a DEC
>   interrupt is pending for guest. If EE is set, you "deliver" it.
> - If not passed, the substraction give you a DEC value (time to next
>   DEC expected by guest). You set the DEC to min(DEC,value), ie, you
>   make it fire the soonest of the guest DEC target vs. linux host  
> guest
>   target. Linux can cope just fine with spurrious DEC interrupts btw.
>
> You have to do some modulo 32 bit etc... but basically, it should work
> and it sounds simpler to me and will avoid the churn with timers  
> etc...

Oh you mean it will basically avoid us firing off a timer for mtdec 1?

So the only really novel thing I see here is that small mtdecs don't  
require a timer but instead the timer only gets issued on the next  
exit. I'm not sure that's a good idea though :-/. It'd probably be  
better to just define a minimum threshold.

on mtdec:

if (reg < 5)
   fire_off_dec();
else
   fire_dec_on(reg);

Something like that.

I don't want to depend on the host DEC being the clocksource btw. So  
anything munching guest and host DEC together (like pHyp does) sounds  
like bad design to me. Let's better have hrtimers handle the case  
where we don't need a real interrupt.

Alex

Re: Ubuntu

[Benjamin Herrenschmidt]

On Thu, 2009-11-12 at 11:59 +0100, Alexander Graf wrote:
> 
> Oh you mean it will basically avoid us firing off a timer for mtdec 1?

Well, I wasn't specifically thinking about the mtdec 1 case (which we
would get rid off by emulating an MPIC instead of the old pmac pic) but
in general, it would overall simplify the emulation.

So there are two aspects here we are mixing up. Let me clarify:

 - Doing the TB based target thingy simplifies the calculation as to
whether we need to fire off a DEC interrupt to the guest and provides a
better overall emulation of the DEC behaviour since the DEC ticks at the
same rate as the TB.

 - My other idea to modify the CPU DEC instead of firing a hrtimer is a
separate thing. We could stick to a hrtimer if you want, and thus only
set it at mtdec time, though as you said, it's going to be not nice with
the mtdec 1 that linux does with the pmac PIC. But in general, I find it
a lot simpler and -less-code- to just whack the real DEC on exit to be
the min(guest_DEC,host_DEC). No need to go through all of the hrtimer
code etc...

> So the only really novel thing I see here is that small mtdecs don't  
> require a timer but instead the timer only gets issued on the next  
> exit. I'm not sure that's a good idea though :-/. It'd probably be  
> better to just define a minimum threshold.

I wasn't thinking about a minimum threshold. No, we must not have DEC
firing too early. The guest will get confused if it hits before the
timebase reaches the expected value and the interrupt will be "lost"

> on mtdec:
> 
> if (reg < 5)
>    fire_off_dec();
> else
>    fire_dec_on(reg);
> 
> Something like that.
> 
> I don't want to depend on the host DEC being the clocksource btw. So  
> anything munching guest and host DEC together (like pHyp does)
> sounds  
> like bad design to me. Let's better have hrtimers handle the case  
> where we don't need a real interrupt.

Why ? Everybody is asserted to the CPU timebase. That's the real clock
source for both host and guest today and you can't do anything about it
since mftb isn't going to trap. The DEC counts off the TB. So it's going
to be more precise and less code to just muck around with the DEC to
trigger the guest interrupts don't you think ?

Cheers,
Ben.