1850/2850 hangs under I/O load

1850/2850 hangs under I/O load

[Keir Fraser]

Looking back over the emails on this topic, someone pointed out a
patch for Linux 2.6.10 that disabled software IRQ affinity for
1850/2850 systems.

You can try a similar fix on Xen (either 2.0.x or unstable) by editing
arch/x86/irq.c:pirq_guest_bind(), and remove the following lines:

    if ( desc->handler->set_affinity != NULL )
        desc->handler->set_affinity(<blah>);

If this fixes the I/O hangs for you, it is a nicer fix than
ignorebiostables. I can add a boot parameter to have the same effect,
and also probably have the fix applied automatically for 1850/2850
systems in the unstable tree (just like Linux).

Let me know how it works out.

 -- Keir

Re: 1850/2850 hangs under I/O load

[Brian Hays]

Hi all,

I'm also about to install Xen on a Poweredge 1850. Is this the
recommended set up (patch referenced below) for that hardware? If so,
is there any thing else that may take a hit as far as performance or
reliablity after making the change suggested?

Thank you,
Brian

On 7/13/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> Looking back over the emails on this topic, someone pointed out a
> patch for Linux 2.6.10 that disabled software IRQ affinity for
> 1850/2850 systems.
> 
> You can try a similar fix on Xen (either 2.0.x or unstable) by editing
> arch/x86/irq.c:pirq_guest_bind(), and remove the following lines:
> 
>     if ( desc->handler->set_affinity != NULL )
>         desc->handler->set_affinity(<blah>);
> 
> If this fixes the I/O hangs for you, it is a nicer fix than
> ignorebiostables. I can add a boot parameter to have the same effect,
> and also probably have the fix applied automatically for 1850/2850
> systems in the unstable tree (just like Linux).
> 
> Let me know how it works out.
> 
>  -- Keir
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: 1850/2850 hangs under I/O load

[Keir Fraser]

On 13 Jul 2005, at 17:43, Brian Hays wrote:

> I'm also about to install Xen on a Poweredge 1850. Is this the
> recommended set up (patch referenced below) for that hardware? If so,
> is there any thing else that may take a hit as far as performance or
> reliablity after making the change suggested?

The patch | just posted is a new one I've put up for testing and 
comments.

The usual fixes that people find to work currently are to specify 
'nousb' on domain0's command line, or 'ignorebiostables' on Xen's 
command line.

  -- Keir

Re: 1850/2850 hangs under I/O load

[David H]

I have some supermicro systems with the same chipset/problem as the
1850/2850.  This patch appears to fix the problem for me.  I could
always hang the server by using scp to copy a large file.  I have been
able to scp this file 10 times without a hang!!  However, it looks
like we lose ACPI.  Is that the expected outcome?

cat /proc/interrups before and after:
          CPU0
 1:         10        Phys-irq  i8042
 9:          0        Phys-irq  acpi
 12:        101        Phys-irq  i8042
 15:    4412096        Phys-irq  ide1
 16:      94555        Phys-irq  uhci_hcd, uhci_hcd
 18:    6161124        Phys-irq  uhci_hcd
 19:          0        Phys-irq  uhci_hcd
 48:      84003        Phys-irq  3w-xxxx
 54:    3091939        Phys-irq  eth0
256:          0     Dynamic-irq  ctrl-if
257:   22704931     Dynamic-irq  timer0
258:          0     Dynamic-irq  console
259:          0     Dynamic-irq  net-be-dbg
NMI:          0
LOC:          0
ERR:          0
MIS:          0

          CPU0
 1:         10        Phys-irq  i8042
 12:        101        Phys-irq  i8042
 15:      24398        Phys-irq  ide1
 16:     288202        Phys-irq  uhci_hcd, uhci_hcd
 18:    7522500        Phys-irq  uhci_hcd
 19:          0        Phys-irq  uhci_hcd
 48:      92704        Phys-irq  3w-xxxx
 54:    7764230        Phys-irq  eth0
128:          1     Dynamic-irq  misdirect
129:          0     Dynamic-irq  ctrl-if
130:     239113     Dynamic-irq  timer
131:          0     Dynamic-irq  console
132:          0     Dynamic-irq  net-be-dbg
NMI:          0
ERR:          0

David



On 7/13/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> Looking back over the emails on this topic, someone pointed out a
> patch for Linux 2.6.10 that disabled software IRQ affinity for
> 1850/2850 systems.
> 
> You can try a similar fix on Xen (either 2.0.x or unstable) by editing
> arch/x86/irq.c:pirq_guest_bind(), and remove the following lines:
> 
>     if ( desc->handler->set_affinity != NULL )
>         desc->handler->set_affinity(<blah>);
> 
> If this fixes the I/O hangs for you, it is a nicer fix than
> ignorebiostables. I can add a boot parameter to have the same effect,
> and also probably have the fix applied automatically for 1850/2850
> systems in the unstable tree (just like Linux).
> 
> Let me know how it works out.
> 
>  -- Keir
>

Re: 1850/2850 hangs under I/O load

[David H]

Replying to the list, I forgot to reply all to the last emial:

You are correct of course.  Trying to do too many things at once...

I set some aside some time, got my xen versions all sorted out, and
did a little testing.  It looks like this fixes the problem for my
server in the 2.0.6 and 2.0-testing (not sure if there is any
difference between the the two at this point but I thought in couldn't
hurt to test). However, unstable from last nights tar ball still hangs
although it to take a little longer to do so.  Let me know if there is
anything else I can test.

Thanks you, and sorry for the earlier ACPI "crazy talk". :)

David

On 7/13/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> On 13 Jul 2005, at 20:13, David H wrote:
> 
> > I have some supermicro systems with the same chipset/problem as the
> > 1850/2850.  This patch appears to fix the problem for me.  I could
> > always hang the server by using scp to copy a large file.  I have been
> > able to scp this file 10 times without a hang!!  However, it looks
> > like we lose ACPI.  Is that the expected outcome?
> 
> This is great to hear. Also, the patch cannot possibly have any effect
> at all on use of ACPI -- what makes you think ACPI usage has changed?
> Maybe you are using a different kernel config?
> 
>   -- Keir
> 
>

Re: 1850/2850 hangs under I/O load

[Keir Fraser]

On 14 Jul 2005, at 02:37, David H wrote:

> I set some aside some time, got my xen versions all sorted out, and
> did a little testing.  It looks like this fixes the problem for my
> server in the 2.0.6 and 2.0-testing (not sure if there is any
> difference between the the two at this point but I thought in couldn't
> hurt to test). However, unstable from last nights tar ball still hangs
> although it to take a little longer to do so.  Let me know if there is
> anything else I can test.

Are you sure you fixed up unstable properly, and definitely were 
running the fixed up version? It would be quite surprising if that fix 
worked for xen2 but not for xen3!

I just did some tests myself, and fixing that one use of set_affinity 
in pirq_guest_bind ought to be sufficient to get unstable working on 
your boxes.

  -- Keir

Re: Re: 1850/2850 hangs under I/O load

[Keir Fraser]

On 14 Jul 2005, at 11:29, Keir Fraser wrote:

> Are you sure you fixed up unstable properly, and definitely were 
> running the fixed up version? It would be quite surprising if that fix 
> worked for xen2 but not for xen3!
>
> I just did some tests myself, and fixing that one use of set_affinity 
> in pirq_guest_bind ought to be sufficient to get unstable working on 
> your boxes.

I also just added a new boot parameter 'noirqbalance' to the 
2.0-testing and unstable repositories. If you're using the latest 
version of either of those then just add that to Xen's command line 
instead of manually patching the code.

You can tell if you have a recent enough source tree: 
xen/arch/x86/irq.c will have a new opt_noirqbalance variable declared 
right at the top.

  -- Keir

Re: Re: 1850/2850 hangs under I/O load

[Keir Fraser]

>> Are you sure you fixed up unstable properly, and definitely were 
>> running the fixed up version? It would be quite surprising if that 
>> fix worked for xen2 but not for xen3!
>>
>> I just did some tests myself, and fixing that one use of set_affinity 
>> in pirq_guest_bind ought to be sufficient to get unstable working on 
>> your boxes.
>
> I also just added a new boot parameter 'noirqbalance' to the 
> 2.0-testing and unstable repositories. If you're using the latest 
> version of either of those then just add that to Xen's command line 
> instead of manually patching the code.
>
> You can tell if you have a recent enough source tree: 
> xen/arch/x86/irq.c will have a new opt_noirqbalance variable declared 
> right at the top.

I've also checked in automatic disabling of IRQ balancing/affinity in 
the unstable tree, so with the latest repository you shouldn't even 
have to add an explicit boot parameter.

If you have a serial line attached then you should see Xen print:
  XEN: Platform quirk -- Disabling IRQ balancing/affinity.

at some point during boot of domain0.

  -- Keir

Re: 1850/2850 hangs under I/O load

[David H]

Here is the relevant portion of irq.c, let me know if this is not correct.

        /* Attempt to bind the interrupt target to the correct CPU. */
        cpu_set(v->processor, cpumask);
        /* if ( desc->handler->set_affinity != NULL )
            desc->handler->set_affinity(vector, cpumask); */
 
On 7/14/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
> On 14 Jul 2005, at 02:37, David H wrote:
> 
> > I set some aside some time, got my xen versions all sorted out, and
> > did a little testing.  It looks like this fixes the problem for my
> > server in the 2.0.6 and 2.0-testing (not sure if there is any
> > difference between the the two at this point but I thought in couldn't
> > hurt to test). However, unstable from last nights tar ball still hangs
> > although it to take a little longer to do so.  Let me know if there is
> > anything else I can test.
> 
> Are you sure you fixed up unstable properly, and definitely were
> running the fixed up version? It would be quite surprising if that fix
> worked for xen2 but not for xen3!
> 
> I just did some tests myself, and fixing that one use of set_affinity
> in pirq_guest_bind ought to be sufficient to get unstable working on
> your boxes.
> 
>   -- Keir
> 
>

Re: 1850/2850 hangs under I/O load

[Keir Fraser]

On 14 Jul 2005, at 16:05, David H wrote:

> ere is the relevant portion of irq.c, let me know if this is not 
> correct.
>
>         /* Attempt to bind the interrupt target to the correct CPU. */
>         cpu_set(v->processor, cpumask);
>         /* if ( desc->handler->set_affinity != NULL )
>             desc->handler->set_affinity(vector, cpumask); */

That's the correct bit. Odd it doesn't fix the problem on unstable. I 
checked in the fix anyway, and maybe we can work out what else 
important has changed between 2.0 and unstable.

  -- Keir