network (irq?) problems in unstable

network (irq?) problems in unstable

[Gerd Knorr]

  Hi,

With xen unstable I have the problem that the network stops
working after a while.  Heavy network traffic seems to make it
more likely, it triggers almost every time I try to untar a
kernel source tree, where the tarball comes from a NFS-mounted
volume.

The network card doesn't receive interrupts any more according
to /proc/interrupts.  It works fine in native linux and also
with xen 2.0.  It's a tulip card.  Anyone has an idea what this
might be or where to look?

  Gerd

/proc/interrupts in native linux:
           CPU0       
  0:     613024          XT-PIC  timer
  1:        673          XT-PIC  i8042
  2:          0          XT-PIC  cascade
  5:       3460          XT-PIC  sym53c8xx
  7:          1          XT-PIC  parport0
  8:          2          XT-PIC  rtc
  9:          0          XT-PIC  acpi, uhci_hcd
 10:          0          XT-PIC  Ensoniq AudioPCI
 11:       1131          XT-PIC  eth0
 12:        446          XT-PIC  i8042
 14:        313          XT-PIC  ide0
 15:       9492          XT-PIC  ide1
NMI:          0 
LOC:          0 
ERR:          0
MIS:          0

/proc/interrupts in domain0, unstable:
           CPU0       
  1:        113        Phys-irq  i8042
  5:       2521        Phys-irq  sym53c8xx
  9:          0        Phys-irq  acpi
 11:        779        Phys-irq  hw-eth0
256:          0     Dynamic-irq  ctrl-if
257:      15736     Dynamic-irq  timer0
258:          0     Dynamic-irq  console
259:          0     Dynamic-irq  net-be-dbg
NMI:          0 
LOC:          0 
ERR:          0
MIS:          0

Re: network (irq?) problems in unstable

[Keir Fraser]

On 30 May 2005, at 13:52, Gerd Knorr wrote:

> With xen unstable I have the problem that the network stops
> working after a while.  Heavy network traffic seems to make it
> more likely, it triggers almost every time I try to untar a
> kernel source tree, where the tarball comes from a NFS-mounted
> volume.
>
> The network card doesn't receive interrupts any more according
> to /proc/interrupts.  It works fine in native linux and also
> with xen 2.0.  It's a tulip card.  Anyone has an idea what this
> might be or where to look?

The new IRQ code in unstable is not so well tested on non-ACPI and 
non-IOAPIC systems (most of our boxes are SMP servers). May be worth 
looking at boot-time output of Xen vs. native Linux. Also, particularly 
if you have a null modem cable attached, add tracing on a debug key to 
see what Xen thinks the status of each IRQ line is (disabled, 
in_progress, etc.). Managing IRQ lines via legacy PIC is pretty 
straightforward so this can't really be that hard a bug to fix I think 
(hope). :-)

  -- Keir

Re: network (irq?) problems in unstable

[Gerd Knorr]

> looking at boot-time output of Xen vs. native Linux. Also, particularly 
> if you have a null modem cable attached, add tracing on a debug key to 
> see what Xen thinks the status of each IRQ line is (disabled, 
> in_progress, etc.). Managing IRQ lines via legacy PIC is pretty 
> straightforward so this can't really be that hard a bug to fix I think 
> (hope). :-)

(XEN) IRQ states:
(XEN)     0:
(XEN)     1: guest
(XEN)     2:
(XEN)     4: inprogress
(XEN)     5: guest
(XEN)     9: guest
(XEN)    11: guest

No change before and after network freeze (#11 is the nic).
Hmm.

  Gerd

-- 
-mm seems unusually stable at present.
	-- akpm about 2.6.12-rc3-mm3

Re: network (irq?) problems in unstable

[Keir Fraser]

On 31 May 2005, at 11:18, Gerd Knorr wrote:

> No change before and after network freeze (#11 is the nic).
> Hmm.

Also interesting, for the guest irqs, is to print out integer:
   ((irq_guest_action_t *)desc->action)->in_flight

And for all irqs, to print out string:
   desc->handler->typename

Maybe the interrupt is getting stuck in flight, and therefore never 
unmasked at the IOAPIC...

  -- Keir

PS. I'm looking at your Xen PAE patches now, so hopefully I'll get them 
checked in later today.

Re: network (irq?) problems in unstable

[Gerd Knorr]

> Also interesting, for the guest irqs, is to print out integer:
>   ((irq_guest_action_t *)desc->action)->in_flight
> 
> And for all irqs, to print out string:
>   desc->handler->typename

(XEN) IRQ states:
(XEN)     0: 45508 XT-PIC
(XEN)     1:  1105 XT-PIC guest(0)
(XEN)     2:     0 XT-PIC
(XEN)     4:     7 XT-PIC inprogress
(XEN)     5:  3140 XT-PIC guest(0)
(XEN)     9:     0 XT-PIC guest(0)
(XEN)    11: 24131 XT-PIC guest(0)
             ^^^^^ irq counter

Nothing unusual, also the irq count is identical with what
linux thinks about it in /proc/interrupts ...

I've also noticed meanwhile that I can make the network work
again by setting the interface down and up again.  That seems
to re-initialize something.

Dones't look like a xen bug to me. I suspect it's a bug in the
tulip driver, probably triggered by the timings being a bit
different when running under xen.

> PS. I'm looking at your Xen PAE patches now, so hopefully I'll get them 
> checked in later today.

Thanks.  At least the "pae-support" patches being merged would
be really nice as it touches many places and breaks all the time
due to some updates in xen.  The hypercall interface changes
probably need some more work in the tools, I'm trying to get
domU's up with the changed hypercall interface.  Well, I would
if I wouldn't debug the network issues at the moment ...

cheers,

  Gerd

-- 
-mm seems unusually stable at present.
	-- akpm about 2.6.12-rc3-mm3

Hypercall interface changes for PAE

[Keir Fraser]

On 31 May 2005, at 12:36, Gerd Knorr wrote:

> Thanks.  At least the "pae-support" patches being merged would
> be really nice as it touches many places and breaks all the time
> due to some updates in xen.  The hypercall interface changes
> probably need some more work in the tools, I'm trying to get
> domU's up with the changed hypercall interface.

Gerd,

Having looked at the PAE patch and thought about the issue some more, 
I'm actually inclined not to split pfn and flags at the hypercall 
interface (e.g., mmu_update_t, do_update_va_mapping()). IIRC I think 
you wanted not to do this in the first place. :-)

I now think that this is a rather arbitrary interface to impose on 
guests. Also it means that we increase the size of mmu_update_t on 
non-PAE, and change the interface unnecessarily for non-PAE. The thing 
that really changed my mind is that the no-execute bit is bit 63 of a 
page-table entry -- so we would need to 'cook' that down into a 32-bit 
flags value to pass across the hypercall interface. Another arbitrary 
and frankly gross thing to have to do.

I think changing val to u64 in mmu_update_t (for 32-bit PAE builds), 
and changing the val to u64 in update_va_mapping(), is probably the way 
to go. Of course, the machine address of the PTE to be modified will 
also need to be a u64. :-)

The downside of this approach is that the C declarations of 
mmu_update_t, update_va_mapping, etc are different on 32-bit PAE 
builds. But only low-level guest code will touch those interfaces 
anyway, and there is unlikely to be code sharing between PAE and 
non-PAE at that level.

What do you think?

  -- Keir

Re: Hypercall interface changes for PAE

[Keir Fraser]

On 31 May 2005, at 19:21, Keir Fraser wrote:

> The downside of this approach is that the C declarations of 
> mmu_update_t, update_va_mapping, etc are different on 32-bit PAE 
> builds. But only low-level guest code will touch those interfaces 
> anyway, and there is unlikely to be code sharing between PAE and 
> non-PAE at that level.
>
> What do you think?

Actually, we could export intpte_t and physaddr_t at the guest 
interface and declare mmu_update_t and friends in terms of those 
typedefs. This would also avoid needing different wrapper 
implementations of those hypercalls within Xen itself. Neat. :-)

  -- Keir

Re: Hypercall interface changes for PAE

[Gerd Knorr]

On Tue, May 31, 2005 at 07:26:51PM +0100, Keir Fraser wrote:
> 
> On 31 May 2005, at 19:21, Keir Fraser wrote:
> 
> >The downside of this approach is that the C declarations of 
> >mmu_update_t, update_va_mapping, etc are different on 32-bit PAE 
> >builds. But only low-level guest code will touch those interfaces 
> >anyway, and there is unlikely to be code sharing between PAE and 
> >non-PAE at that level.
> >
> >What do you think?
> 
> Actually, we could export intpte_t and physaddr_t at the guest 
> interface and declare mmu_update_t and friends in terms of those 
> typedefs. This would also avoid needing different wrapper 
> implementations of those hypercalls within Xen itself. Neat. :-)

That certainly would be the way to go if we want to have
different interfaces for PAE and non-PAE.  I'm not sure it
is a good idea to have different hypercall interfaces for
PAE and non-PAE cases in the first place.

What does this mean for the tools?  Would these also be
either PAE or non-PAE then?

What about the option to maybe run non-PAE guests in PAE-xen
in some translated shadow mode?  That wouldn't work then.
I don't think this would be a big problem though ...

  Gerd

-- 
-mm seems unusually stable at present.
	-- akpm about 2.6.12-rc3-mm3

Re: Hypercall interface changes for PAE

[Keir Fraser]

On 31 May 2005, at 20:02, Gerd Knorr wrote:

> That certainly would be the way to go if we want to have
> different interfaces for PAE and non-PAE.  I'm not sure it
> is a good idea to have different hypercall interfaces for
> PAE and non-PAE cases in the first place.
>
> What does this mean for the tools?  Would these also be
> either PAE or non-PAE then?

At least some parts of the tools (e.g., libxc) will need re-building 
for PAE as they know about the structure of pagetables (2-level vs. 
3-level and so on). Either that or we need to compile both cases into 
the library and auto-switch between implementations of some functions 
at run time. Either way, this problem isn't solved by making the mmu 
hypercalls 'binary compatible' across pae/non-pae.

If we really do care about compatibility across pae/non-pae, I would do 
this by making the pte_val a u64 in all cases rather than splitting pfn 
from flags. Then everyone would just ignore the upper 32 bits on 
non-pae systems. This would waste no more space than 32-bit pfn + 
32-bit flags.

I'm fairly neutral on this: if we're happy to make definitions of 
things like pte's and physaddr's be u64 even on non-pae then that may 
help binary compatibility in future at probably very little cost here 
and now. OTOH we can't be fully binary compatible without shadow 
pagetables anyway, because the pagetable structure differs, so is the 
effort worth it?

> What about the option to maybe run non-PAE guests in PAE-xen
> in some translated shadow mode?  That wouldn't work then.
> I don't think this would be a big problem though ...

I think we can live without this. But even if not then we haven't burnt 
our bridges -- we could redirect those few hypercalls to a very slim 
compatibility layer.

  -- Keir

Re: Hypercall interface changes for PAE

[Gerd Knorr]

On Tue, May 31, 2005 at 08:23:38PM +0100, Keir Fraser wrote:
> 
> On 31 May 2005, at 20:02, Gerd Knorr wrote:
> 
> >That certainly would be the way to go if we want to have
> >different interfaces for PAE and non-PAE.  I'm not sure it
> >is a good idea to have different hypercall interfaces for
> >PAE and non-PAE cases in the first place.
> >
> >What does this mean for the tools?  Would these also be
> >either PAE or non-PAE then?
> 
> At least some parts of the tools (e.g., libxc) will need re-building 
> for PAE as they know about the structure of pagetables (2-level vs. 
> 3-level and so on). Either that or we need to compile both cases into 
> the library and auto-switch between implementations of some functions 
> at run time. Either way, this problem isn't solved by making the mmu 
> hypercalls 'binary compatible' across pae/non-pae.

Disclaimer: Havn't even looked at the tools code yet.

I'd expect with a identical interface we'll only have to add a
PAE-enabled domain builder to boot domU (there are multiple
already anyway, right?), whereas with different interfaces the
split goes down to the shared lib which provides the hypercall
interface to the tools.

Point is I expect it being much easier to switch at runtime
between pae and non-pae in the tools when the hypercall
interface is identical.  I might be wrong on this though.

> If we really do care about compatibility across pae/non-pae, I would do 
> this by making the pte_val a u64 in all cases rather than splitting pfn 
> from flags.

Agreed.

> >What about the option to maybe run non-PAE guests in PAE-xen
> >in some translated shadow mode?  That wouldn't work then.
> >I don't think this would be a big problem though ...
> 
> I think we can live without this. But even if not then we haven't burnt 
> our bridges -- we could redirect those few hypercalls to a very slim 
> compatibility layer.

I'd prefeare 64-bit everythere over a compatibility layer.

What about VT/Pacifica btw?  As far I know the x86_64 guys plan
to allow 32-bit vmx guests, right?  I think to make that work
xen must be able to translate non-PAE page table entries into
x86_64 tables (which are very simliar to PAE).  If this
translation code is present anyway it's probably only a small
step to allow non-PAE guests in PAE mode as well.  If we get
this almost for free we might just implement it, although I see
this more as a "nice-to-have" feature that something really
important.

  Gerd

-- 
-mm seems unusually stable at present.
	-- akpm about 2.6.12-rc3-mm3

Re: Hypercall interface changes for PAE

[Keir Fraser]

On 31 May 2005, at 21:38, Gerd Knorr wrote:

> I'd expect with a identical interface we'll only have to add a
> PAE-enabled domain builder to boot domU (there are multiple
> already anyway, right?), whereas with different interfaces the
> split goes down to the shared lib which provides the hypercall
> interface to the tools.

I don't think we want pae/non-pae builder made visible to higher-level 
tools. We can hide the auto-switch between builders within libxc 
itself. The same goes for domain save/restore code as well (although 
there is potentially scope for more binary-code sharing between 
pae/non-pae in this case).

> Point is I expect it being much easier to switch at runtime
> between pae and non-pae in the tools when the hypercall
> interface is identical.  I might be wrong on this though.

The code that would be affected by an interface difference is precisely 
that code which manipulates page tables, and so is already broken by 
the different pagetable format. The only way unmodified non-pae code 
could possibly be made to work is by using shadow page tables. In that 
case we would hook off the call to e.g., do_mmu_update() very early 
anyway (off into shadow code). Hardly different really from jumping in 
the first place at a different hypercall function with different 
prototype: this latter would arguably be cleaner and less cluttered, as 
well as easily allowing us to support non-pae hypercall interface 
within pae xen.

I really think that creating this interface inconsistency is not 
something to be worried about.

  -- Keir

Re: Hypercall interface changes for PAE

[Keir Fraser]

On 31 May 2005, at 22:18, Keir Fraser wrote:

> The code that would be affected by an interface difference is 
> precisely that code which manipulates page tables, and so is already 
> broken by the different pagetable format. The only way unmodified 
> non-pae code could possibly be made to work is by using shadow page 
> tables. In that case we would hook off the call to e.g., 
> do_mmu_update() very early anyway (off into shadow code). Hardly 
> different really from jumping in the first place at a different 
> hypercall function with different prototype: this latter would 
> arguably be cleaner and less cluttered, as well as easily allowing us 
> to support non-pae hypercall interface within pae xen.
>
> I really think that creating this interface inconsistency is not 
> something to be worried about.

I'll repeat again, though, that I'm neutral on this. A larger 
mmu_update_t and extra register argument to update_va_mapping is not 
going to noticeably slow down non-pae i386.

  -- Keir

Re: Hypercall interface changes for PAE

[David Hopwood]

Keir Fraser wrote:
> On 31 May 2005, at 20:02, Gerd Knorr wrote:
> 
>> That certainly would be the way to go if we want to have
>> different interfaces for PAE and non-PAE.  I'm not sure it
>> is a good idea to have different hypercall interfaces for
>> PAE and non-PAE cases in the first place.
>>
>> What does this mean for the tools?  Would these also be
>> either PAE or non-PAE then?
> 
> At least some parts of the tools (e.g., libxc) will need re-building for 
> PAE as they know about the structure of pagetables (2-level vs. 3-level 
> and so on). Either that or we need to compile both cases into the 
> library and auto-switch between implementations of some functions at run 
> time. Either way, this problem isn't solved by making the mmu hypercalls 
> 'binary compatible' across pae/non-pae.

But it is greatly simplified, IIUC. If the hypercalls are binary compatible
then you have only one set of hypercall interface functions and types, and
switching between two *implementations* of pagetable-related stuff (only
the things that actually need to be different) is quite straightforward.
If you are trying to auto-switch between two hypervisor interfaces that
use different types, then I can think of several ways of doing that, but
they are all significantly more complicated:

a) use preprocessor hackery to declare the interface headers twice with
    different symbol prefixes,
b) use different hypercalls where needed (i.e. not just the same hypercall
    with different types) for PAE and non-PAE
c) compile libxc twice and switch between the versions using dynamic
    linking.

> If we really do care about compatibility across pae/non-pae, I would do 
> this by making the pte_val a u64 in all cases rather than splitting pfn 
> from flags. Then everyone would just ignore the upper 32 bits on non-pae 
> systems. This would waste no more space than 32-bit pfn + 32-bit flags.

Right, that makes sense.

> I'm fairly neutral on this: if we're happy to make definitions of things 
> like pte's and physaddr's be u64 even on non-pae then that may help 
> binary compatibility in future at probably very little cost here and 
> now. OTOH we can't be fully binary compatible without shadow pagetables 
> anyway, because the pagetable structure differs, so is the effort worth it?

Yes, I think so: the code for the tools will end up being much simpler
if PAE and non-PAE are as similar as possible.

-- 
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>

Re: Hypercall interface changes for PAE

[Jonathan S. Shapiro]

On Tue, 2005-05-31 at 19:21 +0100, Keir Fraser wrote:
> The downside of this approach is that the C declarations of 
> mmu_update_t, update_va_mapping, etc are different on 32-bit PAE 
> builds....

The rest of this idea I like, but the statement above seems like the
wrong thing to do. The right thing to do is to preserve a uniform
interface, breaking compatibility. This is a change being made in an
unreleased major version -- an interface change that requires a
recompile is perfectly okay.

A further advantage of this change is that the structure is likely to be
more compatible across different architectures, which will reduce source
code changes in the hypervisor and also (in some cases) in certain
commonly used bits of guest kernel code.

shap

Re: Hypercall interface changes for PAE

[Keir Fraser]

On 31 May 2005, at 19:32, Jonathan S. Shapiro wrote:

> The rest of this idea I like, but the statement above seems like the
> wrong thing to do. The right thing to do is to preserve a uniform
> interface, breaking compatibility. This is a change being made in an
> unreleased major version -- an interface change that requires a
> recompile is perfectly okay.

I think my followup suggestion of hiding the interface difference 
behind typedefs is best (keeps the interface as uniform as possible at 
the source level; and non-PAE will continue to work even without 
recompilation).

  -- Keir

RE: Hypercall interface changes for PAE

[Ian Pratt]

> But it is greatly simplified, IIUC. If the hypercalls are 
> binary compatible then you have only one set of hypercall 
> interface functions and types, and switching between two 
> *implementations* of pagetable-related stuff (only the things 
> that actually need to be different) is quite straightforward.

For the same reason that Linux doesn't support run-time switching
between PAE and non-PAE kernels, doing the same on Xen is going to be an
equivalent pain in the butt.

The only way it can reasonably be done cleanly and with decent
performance is double compilation of the relevant mm functions in Xen
(and libxc too). In which case, having separate hypercall vectors makes
most sense.

So, the correct thing to do is:
 * export the types to the tools as Keir suggests, 32 bit on non-PAE, 64
bit on PAE (and x86_64)
 * use different vectors for the two types of calls
 * for the moment, map them to the same compile-time function.

This gives us good performance, clean code, and future proofing.

Best,
Ian

Re: Hypercall interface changes for PAE

[Gerd Knorr]

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:

> > But it is greatly simplified, IIUC. If the hypercalls are 
> > binary compatible then you have only one set of hypercall 
> > interface functions and types, and switching between two 
> > *implementations* of pagetable-related stuff (only the things 
> > that actually need to be different) is quite straightforward.
> 
> For the same reason that Linux doesn't support run-time switching
> between PAE and non-PAE kernels, doing the same on Xen is going to be an
> equivalent pain in the butt.

Fully agree on the xen kernel side.  The performance hit a runtime
switch would have likely is bigger than simply running PAE all the
time ;)

I don't thing the performance argument is that important for the xen
tools though.  Booting or migrating a domain is a rare event (when
compared to the page table manipulations the xen kernel has to do all
the time).

> The only way it can reasonably be done cleanly and with decent
> performance is double compilation of the relevant mm functions in Xen
> (and libxc too). In which case, having separate hypercall vectors makes
> most sense.

Well, I'd try to get away without double compilation for libxc.

But you guys know that part of the code much better than I do, so if
you think double compilation is the best way to deal with it, lets
take that route.

cheers,

  Gerd

-- 
-mm seems unusually stable at present.
	-- akpm about 2.6.12-rc3-mm3

RE: Hypercall interface changes for PAE

[Ian Pratt]

 
> I don't thing the performance argument is that important for 
> the xen tools though.  Booting or migrating a domain is a 
> rare event (when compared to the page table manipulations the 
> xen kernel has to do all the time).
> 
> > The only way it can reasonably be done cleanly and with decent 
> > performance is double compilation of the relevant mm 
> functions in Xen 
> > (and libxc too). In which case, having separate hypercall vectors 
> > makes most sense.
> 
> Well, I'd try to get away without double compilation for libxc.
> 
> But you guys know that part of the code much better than I 
> do, so if you think double compilation is the best way to 
> deal with it, lets take that route.

The inner most guts of the domain builder where we build the pagetables,
it's probably best to have two totally separate functions as there are
significant differences between the PAE and non PAE initial pagetables.

For the save/restore functions I'd like to share the source code.
However, it would be very ugly indeed to butcher the code such that the
same compiled code can run-time switch. I think the best soloution is
just to run it through the compiler twice with different header files.
[Aside: we need to give the save/restore code the same treatment that
the improved pte typing patch gave to Xen.] 

Ian

Re: Hypercall interface changes for PAE

[Keir Fraser]

On 1 Jun 2005, at 10:30, Ian Pratt wrote:

> The inner most guts of the domain builder where we build the 
> pagetables,
> it's probably best to have two totally separate functions as there are
> significant differences between the PAE and non PAE initial pagetables.
>
> For the save/restore functions I'd like to share the source code.
> However, it would be very ugly indeed to butcher the code such that the
> same compiled code can run-time switch. I think the best soloution is
> just to run it through the compiler twice with different header files.
> [Aside: we need to give the save/restore code the same treatment that
> the improved pte typing patch gave to Xen.]

Restructuring the code so that the minimal amount of stuff has to be 
double-compiled will not be very difficult. Just anything that looks at 
or modifies pagetable entries, pretty much. There really isn't that 
much p.t. code in libxc -- just a few hundred lines total I would say.

But I think just making everything u64 (both ptr and val in 
mmu_update_t, and val in update_va_mapping) is simple, won;t hurt 
performance, and probably makes the tools code a bit simpler to read 
and to build. So let's just do that then and argue about something more 
important. :-)

  -- Keir