The virtuailization patches break Voyager.

The virtuailization patches break Voyager.

[Eric W. Biederman]

Guys currently I am horrified by the ease at which I can find
bugs in the pending paravirtualization patches.  I have barely
even looked at arch/i386 in the -mm tree and it feels like
I am tripping over significant bugs left and right.

Because no one has heeded my advice and put in a proper platform
layer on arch/i386 and we are instead doing a half baked job
with paravirt_ops it is still trivially easy to miss the
fact that subarchitectures do something different, and thus
it is easy to miss when you break a sub architecture on
arch/i386.

Not that the paravirtuailzation patches are even safe on the
primary arch/i386.

To some extent I grant with major changes a little goofing up on
pending patches is to be expected, but it would be nice if
things were restructured to make it harder to miss the
subarchitectures on arch/i386.

The patch known as x86_64-mm-use-per-cpu-gdt-immediately-upon-boot
on -mm currently breaks voyager smp support in some very obvious ways.

Making init_gdt a function which is called from voyager_smp.c static
in smpboot.c a file that is not even used on voayger is an obvious
one.

Adding start_pda and not setting it in voyager_smp is another.

Rusty do you think you can address this?

Eric

Re: The virtuailization patches break Voyager.

[Jeremy Fitzhardinge]

Eric W. Biederman wrote:
> Guys currently I am horrified by the ease at which I can find
> bugs in the pending paravirtualization patches.  I have barely
> even looked at arch/i386 in the -mm tree and it feels like
> I am tripping over significant bugs left and right.
>   

Well, I appreciate any and all review comments you have to make.

> Because no one has heeded my advice and put in a proper platform
> layer on arch/i386 and we are instead doing a half baked job
> with paravirt_ops it is still trivially easy to miss the
> fact that subarchitectures do something different, and thus
> it is easy to miss when you break a sub architecture on
> arch/i386.
>   

Yes. Though in many ways paravirt_ops is approaching that platform
layer. In the next round of work, it might be worth renaming it and
actually using it to subsume the subarch mechanism into something that
can deal with any architecture at runtime. While this hasn't been an
explicit goal of the current round of work, I've tried to point things
in that direction where I can (smp_ops and reboot_ops being specific
examples of that).

But you're right; we've been fairly cavalier about Voyager in
particular, with the hope that James' machine will die before he notices
we've broken the build. (Or perhaps it has, and he just keeps building
voyager kernels to torment us.)

> Not that the paravirtuailzation patches are even safe on the
> primary arch/i386.
>   

Are you referring to the PSE pagetable setup problem we've been
discussing, or do you have something else in mind?

> To some extent I grant with major changes a little goofing up on
> pending patches is to be expected, but it would be nice if
> things were restructured to make it harder to miss the
> subarchitectures on arch/i386.
>
> The patch known as x86_64-mm-use-per-cpu-gdt-immediately-upon-boot
> on -mm currently breaks voyager smp support in some very obvious ways.
>
> Making init_gdt a function which is called from voyager_smp.c static
> in smpboot.c a file that is not even used on voayger is an obvious
> one.
>
> Adding start_pda and not setting it in voyager_smp is another.
>
> Rusty do you think you can address this?

Well, it will probably need to be done after the following patches which
get rid of the PDA altogether. But the boot-time setup is now pretty
simple, so it should just be a matter of putting a couple of init_gdts
in the right places. (It is non-static now too.)

J

Re: The virtuailization patches break Voyager.

[Eric W. Biederman]

Jeremy Fitzhardinge <jeremy@goop.org> writes:

> Eric W. Biederman wrote:
>> Guys currently I am horrified by the ease at which I can find
>> bugs in the pending paravirtualization patches.  I have barely
>> even looked at arch/i386 in the -mm tree and it feels like
>> I am tripping over significant bugs left and right.
>>   
>
> Well, I appreciate any and all review comments you have to make.
>
>> Because no one has heeded my advice and put in a proper platform
>> layer on arch/i386 and we are instead doing a half baked job
>> with paravirt_ops it is still trivially easy to miss the
>> fact that subarchitectures do something different, and thus
>> it is easy to miss when you break a sub architecture on
>> arch/i386.
>>   
>
> Yes. Though in many ways paravirt_ops is approaching that platform
> layer. In the next round of work, it might be worth renaming it and
> actually using it to subsume the subarch mechanism into something that
> can deal with any architecture at runtime. While this hasn't been an
> explicit goal of the current round of work, I've tried to point things
> in that direction where I can (smp_ops and reboot_ops being specific
> examples of that).

Good.  We should do that or at the very least cleanup the 
subarchitecture support on arch/i386.


> But you're right; we've been fairly cavalier about Voyager in
> particular, with the hope that James' machine will die before he notices
> we've broken the build. (Or perhaps it has, and he just keeps building
> voyager kernels to torment us.)

Next time I'm in a really tormenting mood I will fire up my 
my ibm ps2 with it's 16Mhz 386 and 6MB and verify that all is working
well there.

I really don't think it is ok to be cavalier about anything
that we actually support.  Usually if we can handle the general
case it makes for better more maintainable code.

So far the paravirt class of machines seems every bit as much a subarch
as voyager and every bit as interesting.

>> Not that the paravirtuailzation patches are even safe on the
>> primary arch/i386.
>>   
>
> Are you referring to the PSE pagetable setup problem we've been
> discussing, or do you have something else in mind?

That is what I was thinking of in particular.  But I don't currently
have much faith in the code review process that has been happening
for paravirt patches.

>> Rusty do you think you can address this?
>
> Well, it will probably need to be done after the following patches which
> get rid of the PDA altogether. But the boot-time setup is now pretty
> simple, so it should just be a matter of putting a couple of init_gdts
> in the right places. (It is non-static now too.)

Well at least in 2.6.21-rc7-mm2 init_gdt was still static, and
still in smpboot.c

Eric

Re: The virtuailization patches break Voyager.

[Jeremy Fitzhardinge]

Eric W. Biederman wrote:
> Next time I'm in a really tormenting mood I will fire up my 
> my ibm ps2 with it's 16Mhz 386 and 6MB and verify that all is working
> well there.
>   

Well, that would be interesting. From a subarch perspective, it would
just be the normal default, and in theory it should work fine. But I
suspect the fpu emulator is probably broken, and non-WP is likely to
have rotted, and lots of other things. Is that an MCA machine?

> I really don't think it is ok to be cavalier about anything
> that we actually support.  Usually if we can handle the general
> case it makes for better more maintainable code.
>
> So far the paravirt class of machines seems every bit as much a subarch
> as voyager and every bit as interesting.
>   
Well, not really. The problem with the subarch mechanism is that it
promotes a lot of copied code with small modifications, and so making
changes is the inherently non-general activity of trying to find all the
various copies, work out what subtle differences they have, and try to
make the appropriate changes in each case. This was one of the major
objections to the original Xen-as-subarch patches, and it is the problem
with Voyager. The mass of preprocessor tricks doesn't help either.

Because paravirt_ops is intended to support both native execution as
well as virtualized, and needs to be able to decide at boot time, we've
been careful to use as much common code as possible, and only make
differences where they are really needed. The executed code path for
booting on native hardware is very similar for the CONFIG_PARAVIRT vs
!PARAVIRT cases (which is why bugs like the PSE pagetable setup have
leaked through), but it also means that bugs are more likely to be
found. Virtualized execution is more different, of course, and has fewer
people actually trying it out - this is bad, of course, but at least any
bugs should be fairly self-contained.

>> Are you referring to the PSE pagetable setup problem we've been
>> discussing, or do you have something else in mind?
>>     
>
> That is what I was thinking of in particular.  But I don't currently
> have much faith in the code review process that has been happening
> for paravirt patches.
>   

Getting reviewer attention has obviously been a problem. We've been
trying to come up with good code, and Andi has been doing a fairly good
job of looking over it all, but we're in complex and subtle parts of the
kernel and the more reviewers the better. So I'm very pleased you're
taking the time to look over this.

> Well at least in 2.6.21-rc7-mm2 init_gdt was still static, and
> still in smpboot.c

Yes, there are newer patches in Andi's queue.

Thanks,
J

Re: The virtuailization patches break Voyager.

[Eric W. Biederman]

Jeremy Fitzhardinge <jeremy@goop.org> writes:

> Eric W. Biederman wrote:
>> Next time I'm in a really tormenting mood I will fire up my 
>> my ibm ps2 with it's 16Mhz 386 and 6MB and verify that all is working
>> well there.
>>   
>
> Well, that would be interesting. From a subarch perspective, it would
> just be the normal default, and in theory it should work fine. But I
> suspect the fpu emulator is probably broken, and non-WP is likely to
> have rotted, and lots of other things. Is that an MCA machine?

Yes it's a MCA machine.  And except for some ps2 keyboard issues it
worked last time I tested it.  So earlier in 2.6 it was fine..

It was kind of fun submitting a bug report that the ps2 keyboard
support did not work on a genuine IBM ps2.

>> I really don't think it is ok to be cavalier about anything
>> that we actually support.  Usually if we can handle the general
>> case it makes for better more maintainable code.
>>
>> So far the paravirt class of machines seems every bit as much a subarch
>> as voyager and every bit as interesting.
>>   
> Well, not really. The problem with the subarch mechanism is that it
> promotes a lot of copied code with small modifications, and so making
> changes is the inherently non-general activity of trying to find all the
> various copies, work out what subtle differences they have, and try to
> make the appropriate changes in each case. This was one of the major
> objections to the original Xen-as-subarch patches, and it is the problem
> with Voyager. The mass of preprocessor tricks doesn't help either.

The subarch support on arch/i386 is silly from a maintenance
perspective.  I have had one or two occasions where I have made a
change to the entire kernel and only overlooked the necessary change
on some i386 subarch.

That said any paravirtualized subarch is more different from stock
x86 then voyager is and in that sense is very much a sub architecture.

I find it distressing that we currently have 2 subarch layers on
arch/i386.

If you look at alpha, or arm, or  ppc or ia64 they all do
subarchitectures much better then arch/i386.

The sane pattern is and seems to has always been.

arch_function()
{
        platform_ops.platform_function();
}

And on the nice architectures is you build for one particular subarch
the abstraction goes away.

At the same time I find it very distressing how many functions named
native_xxx we are accumulating.  Especially when all native refers is
to the default i386 subarch and not to anything particularly native.
Just one particular way something was implemented.

> Because paravirt_ops is intended to support both native execution as
> well as virtualized, and needs to be able to decide at boot time, we've
> been careful to use as much common code as possible, and only make
> differences where they are really needed. The executed code path for
> booting on native hardware is very similar for the CONFIG_PARAVIRT vs
> !PARAVIRT cases (which is why bugs like the PSE pagetable setup have
> leaked through), but it also means that bugs are more likely to be
> found. Virtualized execution is more different, of course, and has fewer
> people actually trying it out - this is bad, of course, but at least any
> bugs should be fairly self-contained.

In general I agree that the paravirt_ops are much saner then what we
have had before but on x86 but they still have a ways to go.

The fact that 2 level or 3 level page tables can't be selected at
runtime seems to be a failing to think of themselves as a generic 
a subarch mechanism.  I can't fault you to much for that one as
that is a little off the beaten path.

>>> Are you referring to the PSE pagetable setup problem we've been
>>> discussing, or do you have something else in mind?
>>>     
>>
>> That is what I was thinking of in particular.  But I don't currently
>> have much faith in the code review process that has been happening
>> for paravirt patches.
>>   
>
> Getting reviewer attention has obviously been a problem. We've been
> trying to come up with good code, and Andi has been doing a fairly good
> job of looking over it all, but we're in complex and subtle parts of the
> kernel and the more reviewers the better. So I'm very pleased you're
> taking the time to look over this.

Thanks I think.  I just wish the thing that I was finding were more
subtle.  Code not building?

Eric

Re: The virtuailization patches break Voyager.

[Andi Kleen]

On Saturday 28 April 2007 09:52:30 Jeremy Fitzhardinge wrote:
> Eric W. Biederman wrote:
> > Next time I'm in a really tormenting mood I will fire up my 
> > my ibm ps2 with it's 16Mhz 386 and 6MB and verify that all is working
> > well there.
> >   
> 
> Well, that would be interesting. From a subarch perspective, it would
> just be the normal default, and in theory it should work fine. But I
> suspect the fpu emulator is probably broken, and non-WP is likely to
> have rotted, 

AFAIK it's been tested occasionally on some embedded 386 systems
(e.g. by Thomas Gleixner). Probably not with the recent changes though. 

> and lots of other things. Is that an MCA machine? 

Yes should be.
 
> > I really don't think it is ok to be cavalier about anything
> > that we actually support.  Usually if we can handle the general
> > case it makes for better more maintainable code.
> >
> > So far the paravirt class of machines seems every bit as much a subarch
> > as voyager and every bit as interesting.
> >   
> Well, not really. The problem with the subarch mechanism is that it
> promotes a lot of copied code with small modifications, and so making
> changes is the inherently non-general activity of trying to find all the
> various copies, work out what subtle differences they have, and try to
> make the appropriate changes in each case. This was one of the major
> objections to the original Xen-as-subarch patches, and it is the problem
> with Voyager. The mass of preprocessor tricks doesn't help either.

Yes I agree. Current i386 subarch is a mess and I hope to slowly phase
it out. mach-{es7000,summit} should just be folded into mach-generic
always (like x86-64) and I'm somewhat hoping that mach-voyager and 
perhaps mach-visws too will just go away at some point.

The future direction are focussed pluggable interfaces like genapic, smp_ops etc.

-Andi

Re: The virtuailization patches break Voyager.

[Jeremy Fitzhardinge]

Eric W. Biederman wrote:
> That said any paravirtualized subarch is more different from stock
> x86 then voyager is and in that sense is very much a sub architecture.
>   

Well, a goal was that there should be no downside to enabling
CONFIG_PARAVIRT all the time, even if you don't compile in any
hypervisor support. I think we're close to that now, at least for plain
i386. And while I don't expect it to ever happen, we could get rid of
CONFIG_PARAVIRT fairly easily, and simplify a pile of headers. Maybe it
could happen if it becomes platform_ops.

> I find it distressing that we currently have 2 subarch layers on
> arch/i386.
>
> If you look at alpha, or arm, or  ppc or ia64 they all do
> subarchitectures much better then arch/i386.
>   

Yes. Given that we're starting on i386 and the existing subarch
mechanism is clearly inappropriate for what we want to do, the result is
that we're effectively building a parallel subarch mechanism and when it
becomes capable enough it can take over from the existing one. Its
another example of the earliest adopter gets the cruddiest technology.

> At the same time I find it very distressing how many functions named
> native_xxx we are accumulating.  Especially when all native refers is
> to the default i386 subarch and not to anything particularly native.
> Just one particular way something was implemented.
>   

The native_* stuff came from the need to have some kind of consistent
naming convention, but I agree it probably isn't the best. i386_* or
x86_* might be better for many of them.

> In general I agree that the paravirt_ops are much saner then what we
> have had before but on x86 but they still have a ways to go.
>   

Yep. Like everything else in the kernel, it's an iterative development
process.

> The fact that 2 level or 3 level page tables can't be selected at
> runtime seems to be a failing to think of themselves as a generic 
> a subarch mechanism.  I can't fault you to much for that one as
> that is a little off the beaten path.
>   

That's something I've been thinking about, because Xen requires the
PAEness of the guest to match the hypervisor, so it would be useful to
switch on the fly (not that non-PAE Xen is really a preferred option
these days). Hm... You might be able to do it now by compiling the
kernel in PAE mode, and then have a set of PAE->non-PAE pagetable
operations in paravirt_ops do the conversion. It nearly works, but you'd
ideally want to abstract all the higher-level pagetable walkers rather
than just the get/set pte operations.

> Thanks I think.  I just wish the thing that I was finding were more
> subtle.  Code not building?

Well, yes, its a bit embarrassing. I just got some more disk so I can
afford to have a few more build trees around.

J

Re: The virtuailization patches break Voyager.

[Thomas Gleixner]

On Sat, 2007-04-28 at 10:42 +0200, Andi Kleen wrote:
> On Saturday 28 April 2007 09:52:30 Jeremy Fitzhardinge wrote:
> > Eric W. Biederman wrote:
> > > Next time I'm in a really tormenting mood I will fire up my 
> > > my ibm ps2 with it's 16Mhz 386 and 6MB and verify that all is working
> > > well there.
> > >   
> > 
> > Well, that would be interesting. From a subarch perspective, it would
> > just be the normal default, and in theory it should work fine. But I
> > suspect the fpu emulator is probably broken, and non-WP is likely to
> > have rotted, 
> 
> AFAIK it's been tested occasionally on some embedded 386 systems
> (e.g. by Thomas Gleixner). Probably not with the recent changes though. 

Last test was 2.6.21-rc7, where I fixed the non-WP thing. The
FP-emulator seems to work.

Which kind of breakage is new ?

	tglx

Re: The virtuailization patches break Voyager.

[Jeremy Fitzhardinge]

Thomas Gleixner wrote:
> Last test was 2.6.21-rc7, where I fixed the non-WP thing. The
> FP-emulator seems to work.
>
> Which kind of breakage is new ?
>   

We were just speculating about what breakage might exist.  If you've
been applying anti-entropy to  real 80386 machines, then I guess it
probably works better than I would have otherwise expected.

    J

Re: The virtuailization patches break Voyager.

[Eric W. Biederman]

Andi Kleen <ak@suse.de> writes:

> On Saturday 28 April 2007 09:52:30 Jeremy Fitzhardinge wrote:
>> Well, not really. The problem with the subarch mechanism is that it
>> promotes a lot of copied code with small modifications, and so making
>> changes is the inherently non-general activity of trying to find all the
>> various copies, work out what subtle differences they have, and try to
>> make the appropriate changes in each case. This was one of the major
>> objections to the original Xen-as-subarch patches, and it is the problem
>> with Voyager. The mass of preprocessor tricks doesn't help either.
>
> Yes I agree. Current i386 subarch is a mess and I hope to slowly phase
> it out. mach-{es7000,summit} should just be folded into mach-generic
> always (like x86-64) and I'm somewhat hoping that mach-voyager and 
> perhaps mach-visws too will just go away at some point.

There is a possibility, that arch/i386 will seen renewed life as an
embedded architecture.  At which point things like mach-visws may
start proliferating.

So I think it makes a lot of sense to see if we can fold mach-visws
and mach-voyager into appropriate pluggable interfaces.

Xen is stranger than anything voyager does.

> The future direction are focussed pluggable interfaces like genapic, smp_ops
> etc.

Sounds good.  Although it may be nice to do the standard platform
trick of having them compile out if you only compile for one
subarchitecture.

Eric

Re: The virtuailization patches break Voyager.

[Andi Kleen]

> 
> The sane pattern is and seems to has always been.
> 
> arch_function()
> {
>         platform_ops.platform_function();
> }

Yes agreed. We'll slowly move there. Patches to accelerate it are
welcome (for post .22) 

But you're flaming the wrong person for this really. Jeremy and other
paravirt implementors have  done a lot of work of moving things into this 
direction.

> At the same time I find it very distressing how many functions named
> native_xxx we are accumulating.  Especially when all native refers is
> to the default i386 subarch and not to anything particularly native.
> Just one particular way something was implemented.

How else would you name and/or implement that?

> The fact that 2 level or 3 level page tables can't be selected at
> runtime seems to be a failing to think of themselves as a generic 
> a subarch mechanism.  I can't fault you to much for that one as
> that is a little off the beaten path.

That would really require generic mm changes to do properly. I know
PA-RISC does it without that, but that wouldn't fly on x86 I think
because PAE and non PAE are more different there.

-Andi

Re: The virtuailization patches break Voyager.

[Andi Kleen]

On Saturday 28 April 2007 11:15:33 Eric W. Biederman wrote:
> Andi Kleen <ak@suse.de> writes:
> 
> > On Saturday 28 April 2007 09:52:30 Jeremy Fitzhardinge wrote:
> >> Well, not really. The problem with the subarch mechanism is that it
> >> promotes a lot of copied code with small modifications, and so making
> >> changes is the inherently non-general activity of trying to find all the
> >> various copies, work out what subtle differences they have, and try to
> >> make the appropriate changes in each case. This was one of the major
> >> objections to the original Xen-as-subarch patches, and it is the problem
> >> with Voyager. The mass of preprocessor tricks doesn't help either.
> >
> > Yes I agree. Current i386 subarch is a mess and I hope to slowly phase
> > it out. mach-{es7000,summit} should just be folded into mach-generic
> > always (like x86-64) and I'm somewhat hoping that mach-voyager and 
> > perhaps mach-visws too will just go away at some point.
> 
> There is a possibility, that arch/i386 will seen renewed life as an
> embedded architecture.  At which point things like mach-visws may
> start proliferating.

Scary thought. But I don't see why people using embedded x86s should suddenly
design new interrupt controllers etc. - after all the main value of using x86s
embedded is some degree of compatibility to PC software.  Ok, we'll see what
happens.
 
> So I think it makes a lot of sense to see if we can fold mach-visws
> and mach-voyager into appropriate pluggable interfaces.

For voyager and NUMAQ i think it's fine to just wait until the last machine dies
(James, how many do you have left? @] iirc the number of NUMAQs still in operation
is also slowly decreasing) 
 
-Andi

Re: The virtuailization patches break Voyager.

[Andi Kleen]

> Last test was 2.6.21-rc7, where I fixed the non-WP thing. The
> FP-emulator seems to work.
> 
> Which kind of breakage is new ?

Nothing known so far, but there were a lot of changes recently. It would be good
if you could retest mainline on that box once the x86 tree is merged up.

-Andi

Re: The virtuailization patches break Voyager.

[Thomas Gleixner]

On Sat, 2007-04-28 at 11:39 +0200, Andi Kleen wrote:
> > Last test was 2.6.21-rc7, where I fixed the non-WP thing. The
> > FP-emulator seems to work.
> > 
> > Which kind of breakage is new ?
> 
> Nothing known so far, but there were a lot of changes recently. It would be good
> if you could retest mainline on that box once the x86 tree is merged up.

I added that box to my auto test farm of bizarre hardware already, so I
will notice breakage quite fast.

	tglx

Re: The virtuailization patches break Voyager.

[Eric W. Biederman]

Andi Kleen <ak@suse.de> writes:

> On Saturday 28 April 2007 11:15:33 Eric W. Biederman wrote:
>
> Scary thought. But I don't see why people using embedded x86s should suddenly
> design new interrupt controllers etc. - after all the main value of using x86s
> embedded is some degree of compatibility to PC software.  Ok, we'll see what
> happens.

Right but visws main difference was that it did not run a x86 BIOS
as I recall.  My memory says all of it's hardware was standard.

>> So I think it makes a lot of sense to see if we can fold mach-visws
>> and mach-voyager into appropriate pluggable interfaces.
>
> For voyager and NUMAQ i think it's fine to just wait until the last machine dies
> (James, how many do you have left? @] iirc the number of NUMAQs still in
> operation
> is also slowly decreasing) 

Maybe.  Again if we could convert them along with everything else to
a modern structure it probably would not matter.

I honestly think it is irresponsible to keep code in tree and not at least
try to keep it working.

Eric

Re: The virtuailization patches break Voyager.

[James Bottomley]

On Sat, 2007-04-28 at 00:40 -0600, Eric W. Biederman wrote:
> Guys currently I am horrified by the ease at which I can find
> bugs in the pending paravirtualization patches.  I have barely
> even looked at arch/i386 in the -mm tree and it feels like
> I am tripping over significant bugs left and right.
> 
> Because no one has heeded my advice and put in a proper platform
> layer on arch/i386 and we are instead doing a half baked job
> with paravirt_ops it is still trivially easy to miss the
> fact that subarchitectures do something different, and thus
> it is easy to miss when you break a sub architecture on
> arch/i386.

I got a bit tired of trying to be proactive.  The last time was for the
%gs per cpu thing, which I saw coming.  The basic problem is that
there's no well organised git tree I can pull from and bisect to trace
problems.

My strategy now is to wait for the merge window to close and then go
around sweeping up the mess and yelling at the offenders ... it's what
all the non-x86 architectures do, and it's definitely an easier process.

> Not that the paravirtuailzation patches are even safe on the
> primary arch/i386.
> 
> To some extent I grant with major changes a little goofing up on
> pending patches is to be expected, but it would be nice if
> things were restructured to make it harder to miss the
> subarchitectures on arch/i386.
> 
> The patch known as x86_64-mm-use-per-cpu-gdt-immediately-upon-boot
> on -mm currently breaks voyager smp support in some very obvious ways.
> 
> Making init_gdt a function which is called from voyager_smp.c static
> in smpboot.c a file that is not even used on voayger is an obvious
> one.
> 
> Adding start_pda and not setting it in voyager_smp is another.
> 
> Rusty do you think you can address this?

James

Re: The virtuailization patches break Voyager.

[James Bottomley]

On Sat, 2007-04-28 at 10:42 +0200, Andi Kleen wrote:
> On Saturday 28 April 2007 09:52:30 Jeremy Fitzhardinge wrote:
> > Eric W. Biederman wrote:
> > > Next time I'm in a really tormenting mood I will fire up my 
> > > my ibm ps2 with it's 16Mhz 386 and 6MB and verify that all is working
> > > well there.
> > >   
> > 
> > Well, that would be interesting. From a subarch perspective, it would
> > just be the normal default, and in theory it should work fine. But I
> > suspect the fpu emulator is probably broken, and non-WP is likely to
> > have rotted, 
> 
> AFAIK it's been tested occasionally on some embedded 386 systems
> (e.g. by Thomas Gleixner). Probably not with the recent changes though. 
> 
> > and lots of other things. Is that an MCA machine? 
> 
> Yes should be.
>  
> > > I really don't think it is ok to be cavalier about anything
> > > that we actually support.  Usually if we can handle the general
> > > case it makes for better more maintainable code.
> > >
> > > So far the paravirt class of machines seems every bit as much a subarch
> > > as voyager and every bit as interesting.
> > >   
> > Well, not really. The problem with the subarch mechanism is that it
> > promotes a lot of copied code with small modifications, and so making
> > changes is the inherently non-general activity of trying to find all the
> > various copies, work out what subtle differences they have, and try to
> > make the appropriate changes in each case. This was one of the major
> > objections to the original Xen-as-subarch patches, and it is the problem
> > with Voyager. The mass of preprocessor tricks doesn't help either.
> 
> Yes I agree. Current i386 subarch is a mess and I hope to slowly phase
> it out. mach-{es7000,summit} should just be folded into mach-generic
> always (like x86-64) and I'm somewhat hoping that mach-voyager and 
> perhaps mach-visws too will just go away at some point.

You wish ...

> The future direction are focussed pluggable interfaces like genapic, smp_ops etc.

Actually, pluggable smp_ops might make voyager look far more standard
(that's pretty much most of what the subarch code does for voyager).
Pluggable genapic would probably be at the wrong level ... voyager is
non-apic based SMP and doesn't fit well into the apic abstraction.

James

Re: The virtuailization patches break Voyager.

[Eric W. Biederman]

James Bottomley <James.Bottomley@HansenPartnership.com> writes:

> I got a bit tired of trying to be proactive.  The last time was for the
> %gs per cpu thing, which I saw coming.  The basic problem is that
> there's no well organised git tree I can pull from and bisect to trace
> problems.
>
> My strategy now is to wait for the merge window to close and then go
> around sweeping up the mess and yelling at the offenders ... it's what
> all the non-x86 architectures do, and it's definitely an easier process.

Reasonable.  I think we would have fewer people to yell at if we
restructured things a bit. 

The fact that we have prominent people deliberately ignoring voyager
during development I find disturbing. 

In this case I just happened to stumble onto the problem while I was
looking at something else, and decided it wasn't worth doing
development on code that was broken to start with.

Eric

Re: The virtuailization patches break Voyager.

[James Bottomley]

On Sat, 2007-04-28 at 11:34 +0200, Andi Kleen wrote:
> > 
> > The sane pattern is and seems to has always been.
> > 
> > arch_function()
> > {
> >         platform_ops.platform_function();
> > }
> 
> Yes agreed. We'll slowly move there. Patches to accelerate it are
> welcome (for post .22) 
> 
> But you're flaming the wrong person for this really. Jeremy and other
> paravirt implementors have  done a lot of work of moving things into this 
> direction.
> 
> > At the same time I find it very distressing how many functions named
> > native_xxx we are accumulating.  Especially when all native refers is
> > to the default i386 subarch and not to anything particularly native.
> > Just one particular way something was implemented.
> 
> How else would you name and/or implement that?
> 
> > The fact that 2 level or 3 level page tables can't be selected at
> > runtime seems to be a failing to think of themselves as a generic 
> > a subarch mechanism.  I can't fault you to much for that one as
> > that is a little off the beaten path.
> 
> That would really require generic mm changes to do properly. I know
> PA-RISC does it without that, but that wouldn't fly on x86 I think
> because PAE and non PAE are more different there.

It's getting embarrassing to find myself responsible for practically
everything in this debate people regard as strange ...

I did this on PA-RISC to save code and indirect lookups in
interruptions.  It's not exactly runtime switchable.  What we do is make
32 bit processes use the two level structure and 64 bit ones the 3 level
structure.  Since almost every user level process is 32 bit (because the
instruction set is the same, like ppc and sparc, but not x86_64) it
really made no sense to waste the page table entries and the lookup
time.

What I did was to alter the page table layout so the first 4GB use a
2-level scheme and anything after that uses 3 levels, so the requested
virtual address does the switch. (it's modelled on the way inode lookups
are done ... you don't begin in the indirect table, you begin in the
direct one).

However, I think we can only do this because we have a software TLB
refill interruption not a HW page table walker.

James

Re: The virtuailization patches break Voyager.

[James Bottomley]

On Sat, 2007-04-28 at 11:37 +0200, Andi Kleen wrote:
> For voyager and NUMAQ i think it's fine to just wait until the last machine dies
> (James, how many do you have left? @] iirc the number of NUMAQs still in operation
> is also slowly decreasing) 

I have 2 L5s and one L4 in my basement.  I have friends in South
Carolina with another 4 L5s, plus I have users in Italy and
Argentina ...

I assume it will break eventually ... fortunately, it is very well
architected mechanically ...

James

Re: The virtuailization patches break Voyager.

[Jeremy Fitzhardinge]

Eric W. Biederman wrote:
> Reasonable.  I think we would have fewer people to yell at if we
> restructured things a bit. 
>
> The fact that we have prominent people deliberately ignoring voyager
> during development I find disturbing. 
>
> In this case I just happened to stumble onto the problem while I was
> looking at something else, and decided it wasn't worth doing
> development on code that was broken to start with.

Well, you've shamed me into fixing it. I'm putting together a
fix-voyager-build patch now.

J

Re: The virtuailization patches break Voyager.

[James Bottomley]

On Sat, 2007-04-28 at 10:02 -0600, Eric W. Biederman wrote:
> Reasonable.  I think we would have fewer people to yell at if we
> restructured things a bit. 

Sure ... propose a restructure and I'll see if I can fit into it.  All
voyager really needs is to subvert the entirey of the SMP layer (not
being APIC based it needs a complete rewrite of most smp_ function)  It
also needs some minor things in setup (like catbus config probes and
memory setup).

> The fact that we have prominent people deliberately ignoring voyager
> during development I find disturbing. 

Heh, I just got used to that ...

> In this case I just happened to stumble onto the problem while I was
> looking at something else, and decided it wasn't worth doing
> development on code that was broken to start with.

James

Re: The virtuailization patches break Voyager.

[H. Peter Anvin]

Jeremy Fitzhardinge wrote:
> Eric W. Biederman wrote:
>> Next time I'm in a really tormenting mood I will fire up my 
>> my ibm ps2 with it's 16Mhz 386 and 6MB and verify that all is working
>> well there.  
> 
> Well, that would be interesting. From a subarch perspective, it would
> just be the normal default, and in theory it should work fine. But I
> suspect the fpu emulator is probably broken, and non-WP is likely to
> have rotted, and lots of other things. Is that an MCA machine?

Heh.  I actually have a Voyager sitting in the garage.  I've never
plugged it in or turned it on, but after this thread I'm wondering if my
rewrite of the setup code will have to entail dragging it in and trying
to boot it (please, no...) :)

Incidentally, the rewrite is looking very promising so far.  Apparently
there is tons of odd or just plain dead code lurking in setup.S.

	-hpa

Re: The virtuailization patches break Voyager.

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Jeremy Fitzhardinge wrote:
>> Eric W. Biederman wrote:
>>> Next time I'm in a really tormenting mood I will fire up my 
>>> my ibm ps2 with it's 16Mhz 386 and 6MB and verify that all is working
>>> well there.  
>> 
>> Well, that would be interesting. From a subarch perspective, it would
>> just be the normal default, and in theory it should work fine. But I
>> suspect the fpu emulator is probably broken, and non-WP is likely to
>> have rotted, and lots of other things. Is that an MCA machine?
>
> Heh.  I actually have a Voyager sitting in the garage.  I've never
> plugged it in or turned it on, but after this thread I'm wondering if my
> rewrite of the setup code will have to entail dragging it in and trying
> to boot it (please, no...) :)
>
> Incidentally, the rewrite is looking very promising so far.  Apparently
> there is tons of odd or just plain dead code lurking in setup.S.

Just send it out for review before we merge it.

We don't want to break things like loadlin support if we don't have to.

If figure worse case we can convert your clean up back into assembly
and have a more readable and maintainable setup.S :)

Eric

Re: The virtuailization patches break Voyager.

[H. Peter Anvin]

Eric W. Biederman wrote:
>>
>> Incidentally, the rewrite is looking very promising so far.  Apparently
>> there is tons of odd or just plain dead code lurking in setup.S.
> 
> Just send it out for review before we merge it.
> 
> We don't want to break things like loadlin support if we don't have to.
> 
> If figure worse case we can convert your clean up back into assembly
> and have a more readable and maintainable setup.S :)
> 

It doesn't look like there should be a problem getting the loadlin hooks
to continue working.  Something else that might be useful to know is if
there are pre-2.02 bootloaders that are still being used (other than
loadlin?)

As the patches I have now I'm breaking support for pre-2.00 bootloaders,
which don't have bzImage support anyway and thus, I think, can be
disregarded as historic.

	-hpa

Re: The virtuailization patches break Voyager.

[Andi Kleen]

> It's getting embarrassing to find myself responsible for practically
> everything in this debate people regard as strange ...

Didn't want to say it was strange, just that it's not as easy on x86
and unclear if feasible there.

Yes I would like to have PAE and non PAE in a single kernel too - 
with my distro hat on having a single kernel binary for everybody
would be great - but i just don't know how to do it nicely.

-Andi

Re: The virtuailization patches break Voyager.

[Andi Kleen]

> Actually, pluggable smp_ops might make voyager look far more standard

Great. Please send patches (but post .22) 

-Andi

Re: The virtuailization patches break Voyager.

[Andi Kleen]

> I got a bit tired of trying to be proactive.  The last time was for the
> %gs per cpu thing, which I saw coming.  The basic problem is that
> there's no well organised git tree I can pull from and bisect to trace
> problems.

There is a quilt tree as documented in MAINTAINERS

ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/patches/

It is fully bisectable (relative to 
ftp://ftp.firstfloor.org/pub/ak/x86_64/quilt/BASEKERNEL)
and everything. 

It is also possible to generate git trees from this using some
simple scripts if you really need git for something (I do this 
for my merges to Linus). 

But the reason -- similar to Andrew -- i don't keep a continuous
git mirror of the quilt tree is that patch ordering etc. might
change anytime and it is hard to express that in a continuous
git tree. Instead I just generate the git branches on demand
from some fixed state for merging.

> My strategy now is to wait for the merge window to close and then go
> around sweeping up the mess and yelling at the offenders ... it's what
> all the non-x86 architectures do, and it's definitely an easier process.

Fine too.  Since you're effectively the only Voyager user anyways
that's a great model.

-Andi

Re: The virtuailization patches break Voyager.

[Andi Kleen]

On Sat, Apr 28, 2007 at 11:20:10AM -0500, James Bottomley wrote:
> On Sat, 2007-04-28 at 10:02 -0600, Eric W. Biederman wrote:
> > Reasonable.  I think we would have fewer people to yell at if we
> > restructured things a bit. 
> 
> Sure ... propose a restructure and I'll see if I can fit into it.  All
> voyager really needs is to subvert the entirey of the SMP layer (not
> being APIC based it needs a complete rewrite of most smp_ function)  It

Xen needs essentially the same and J. did smp_ops for this so in theory
it should work for you.

-Andi