Grant Table Network Issues

Grant Table Network Issues

[Michael Vrable]

I've been working on getting networking functioning in translated shadow
mode, and have it to the point where it's almost working--some packets
get through some of the time before the machine crashes.

In an effort to narrow down the problem, I've found that the grant table
network interface code in xen-unstable.hg seems to have some stability
problems as well.  Here's one of them: this is with a mostly unmodified
checkout of xen-unstable.hg (from yesterday evening), patched to produce
more debugging output and also with IP checksumming optimizations
disabled (since I was seeing some trouble with those).  After a very
short while, I get a Dom-0 crash.  This transcript is taken from
Domain-0, pinging an unprivileged domain (no shadow modes enabled):

    potemkin58:~# ping 192.168.1.2
    PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
    (XEN) gnttab_donate: i=0 mfn=000112aa domid=1 gref=000003c1
    (XEN) (file=grant_table.c, line=1146) gnttab_prepare_for_transfer rd(1) ld(0) ref(961).
    (XEN) (file=grant_table.c, line=1225) gnttab_notify_transfer rd(1) ld(0) ref(961).
    (XEN) (file=grant_table.c, line=423) Mapping grant ref (706) for domain (1) with flags (6)
    (XEN) (file=grant_table.c, line=119) activate_grant_ref: mapping=0 granting=1
    (XEN) (file=grant_table.c, line=315) activate_grant_ref: frame=78807
    (XEN) (file=grant_table.c, line=455) map_grant_ref: frame=78807 vaddr=df807000 handle=2
    (XEN) gnttab_donate: i=0 mfn=0007689f domid=1 gref=000003c2
    (XEN) (file=grant_table.c, line=1146) gnttab_prepare_for_transfer rd(1) ld(0) ref(962).
    (XEN) (file=grant_table.c, line=1225) gnttab_notify_transfer rd(1) ld(0) ref(962).
    (XEN) (file=grant_table.c, line=535) Unmapping grant ref (706) for domain (1) with handle (2)
    (XEN) (file=grant_table.c, line=652) unmap_grant_ref: frame=78807
    (XEN) (file=grant_table.c, line=423) Mapping grant ref (706) for domain (1) with flags (6)
    (XEN) (file=grant_table.c, line=119) activate_grant_ref: mapping=0 granting=1
    (XEN) (file=grant_table.c, line=315) activate_grant_ref: frame=7b6b0
    (XEN) (file=grant_table.c, line=455) map_grant_ref: frame=7b6b0 vaddr=df808000 handle=2
    kernel BUG at include/linux/skbuff.h:1148 (kmap_skb_frag)!
     [<c03c2a0b>] skb_checksum+0x27b/0x310
     [<c040228b>] icmp_rcv+0x16b/0x1a0
     [<c03dbebb>] ip_local_deliver+0xdb/0x220
     [<c03dc33e>] ip_rcv+0x33e/0x4b0
     [<c03dc630>] ip_rcv_finish+0x0/0x250
     [<c03c7d64>] netif_receive_skb+0x204/0x270
     [<c03c7e89>] process_backlog+0xb9/0x190
     [<c03c800d>] net_rx_action+0xad/0x1a0
     [<c0123f35>] __do_softirq+0xc5/0xf0
     [<c0123feb>] do_softirq+0x8b/0x90
     [<c01240b5>] irq_exit+0x35/0x40
     [<c010f082>] do_IRQ+0x22/0x30
     [<c0106530>] evtchn_do_upcall+0x70/0xa0
     [<c010a758>] hypervisor_callback+0x2c/0x34
     [<c01064ba>] force_evtchn_callback+0xa/0x10
     [<c014b72f>] __pagevec_lru_add+0x15f/0x1c0
     [<c013fb46>] add_to_page_cache+0x76/0xf0
     [<c018957d>] mpage_readpages+0x18d/0x190
     [<c01cd850>] ext3_get_block+0x0/0xc0
     [<c0147e74>] read_pages+0x124/0x170
     [<c01cd850>] ext3_get_block+0x0/0xc0
     [<c0145573>] __alloc_pages+0x2e3/0x430
     [<c0147fe0>] __do_page_cache_readahead+0x120/0x230
     [<c01415cf>] filemap_nopage+0x2ef/0x410
     [<c01528f8>] do_no_page+0xb8/0x3b0
     [<c01502a3>] pte_alloc_map+0x93/0x210
     [<c0152e36>] handle_mm_fault+0xf6/0x240
     [<c01178ec>] do_page_fault+0x19c/0x5f2
     [<c0114246>] old_mmap+0xd6/0x110
     [<c010a93a>] page_fault+0x2e/0x34
    Kernel panic - not syncing: BUG!
     (XEN) Domain 0 shutdown: rebooting machine.

The line causing trouble is "BUG_ON(in_irq())".  In this example, I had
tcpdump running in both domains; this seems to trigger the problem more
reliably.  I've also seen a similar crash with a TCP connection, but it
takes a few packets before this shows up (the handshake completes, and
the crash happens about the time data packets come back from domain-0;
if checksumming optimizations are enabled, it seems the packets are
dropped so I don't see a crash but I don't get any data either).

I've been having a difficult time tracking this down, so any help is
appreciated.

--Michael Vrable

Re: Grant Table Network Issues

[Keir Fraser]

On 13 Aug 2005, at 19:59, Michael Vrable wrote:

> The line causing trouble is "BUG_ON(in_irq())".  In this example, I had
> tcpdump running in both domains; this seems to trigger the problem more
> reliably.  I've also seen a similar crash with a TCP connection, but it
> takes a few packets before this shows up (the handshake completes, and
> the crash happens about the time data packets come back from domain-0;
> if checksumming optimizations are enabled, it seems the packets are
> dropped so I don't see a crash but I don't get any data either).

On the stack trace, at irq_exit() you definitely have no hardirqs or 
softirqs in progress. But somehow, at kmap_skb_frag(), the hardirq 
section of the preempt mask has become non-zero. You can't have been 
preempted to another cpu during any of this because the preempt mask is 
continuously non-zero throughout original irq handling and subsequent 
softirq handling.

The only code between irq_exit and kmap_skb_frag on the stack trace is 
unmodified Linux code. Assuming that is all correct (and presumably the 
same whether we enable grant tables or not) I might guess another 
interrupt arrives and the handler corrupts things?

  -- Keir

Re: Grant Table Network Issues

[Michael Vrable]

On Sun, Aug 14, 2005 at 09:29:02AM +0100, Keir Fraser wrote:
> On 13 Aug 2005, at 19:59, Michael Vrable wrote:
> 
> >The line causing trouble is "BUG_ON(in_irq())".  In this example, I had
> >tcpdump running in both domains; this seems to trigger the problem more
> >reliably.  I've also seen a similar crash with a TCP connection, but it
> >takes a few packets before this shows up (the handshake completes, and
> >the crash happens about the time data packets come back from domain-0;
> >if checksumming optimizations are enabled, it seems the packets are
> >dropped so I don't see a crash but I don't get any data either).
> 
> On the stack trace, at irq_exit() you definitely have no hardirqs or 
> softirqs in progress. But somehow, at kmap_skb_frag(), the hardirq 
> section of the preempt mask has become non-zero. You can't have been 
> preempted to another cpu during any of this because the preempt mask is 
> continuously non-zero throughout original irq handling and subsequent 
> softirq handling.
> 
> The only code between irq_exit and kmap_skb_frag on the stack trace is 
> unmodified Linux code. Assuming that is all correct (and presumably the 
> same whether we enable grant tables or not) I might guess another 
> interrupt arrives and the handler corrupts things?

I discovered the cause of this and other crashes yesterday: when grant
tables are enabled in the netback driver, net_tx_action() and
net_tx_action_dealloc() in netback.c each allocate large arrays from the
kernel's stack ("gnttab_map_grant_ref_t map_ops[MAX_PENDING_REQS]" and
"gnttab_unmap_grant_ref_t unmap_ops[MAX_PENDING_REQS]").  This results
in a stack overflow.

This in turn causes a number of very spectacular crashes.  One of the
more subtle crashes is the in_irq() bug; the preempt count is stored in
the current thread info, at the bottom of the stack.

Allocating the arrays statically fixes the problem for me.  Steve Hand
says he'll likely be committing a fix soon.

--Michael Vrable

Re: Grant Table Network Issues

[Michael Vrable]

On Sun, Aug 14, 2005 at 08:39:53AM -0700, Michael Vrable wrote:
> Allocating the arrays statically fixes the problem for me.  Steve Hand
> says he'll likely be committing a fix soon.

Never mind; catching up on the changes to xen-unstable.hg, I see that
you've already committed a fix.  Thanks.

--Michael Vrable

RE: Grant Table Network Issues

[Ian Pratt]

> On Sun, Aug 14, 2005 at 08:39:53AM -0700, Michael Vrable wrote:
> > Allocating the arrays statically fixes the problem for me.  
> Steve Hand 
> > says he'll likely be committing a fix soon.
> 
> Never mind; catching up on the changes to xen-unstable.hg, I 
> see that you've already committed a fix.  Thanks.

Michael, if this fix is working fine for you, I minded to enable them in
the default config... 

Ian

Re: Grant Table Network Issues

[Steven Hand]

>
>> On Sun, Aug 14, 2005 at 08:39:53AM -0700, Michael Vrable wrote:
>> > Allocating the arrays statically fixes the problem for me.
>> > Steve Hand says he'll likely be committing a fix soon.
>>
>> Never mind; catching up on the changes to xen-unstable.hg, I
>> see that you've already committed a fix.  Thanks.
>
>Michael, if this fix is working fine for you, I minded to enable them in
>the default config...

Good plan - seems to work fine on driller (the machine which reliably 
reproduced this bug for me) under limited stress testing. 

I have a few more bits to check in (using grants for the shmem frames)
and can enable net grant in the default config then. 


cheers,

S.

Re: Grant Table Network Issues

[Michael Vrable]

On Sun, Aug 14, 2005 at 04:59:53PM +0100, Ian Pratt wrote:
> > On Sun, Aug 14, 2005 at 08:39:53AM -0700, Michael Vrable wrote:
> > > Allocating the arrays statically fixes the problem for me.  
> > Steve Hand 
> > > says he'll likely be committing a fix soon.
> > 
> > Never mind; catching up on the changes to xen-unstable.hg, I 
> > see that you've already committed a fix.  Thanks.
> 
> Michael, if this fix is working fine for you, I minded to enable them in
> the default config... 

I haven't tested this particular commit, since I'm currently using a
testing a version of Xen that is patched to run unprivileged domains in
shadow translated mode, with grant-table based networking.  However, the
equivalent patch in my tree has fixed my networking stability issues.

There are still a few robustness issues to work on.  For example, the
BUG() when grant table donate operations fail in net_rx_action, which
can legitimately happen if the unprivileged domain hasn't set up its
grant tables properly.  I haven't seen this come up in normal use, but
does need to be fixed before the final release, since it will let an
unprivileged domain crash domain-0.  (I have seen it in my work, but
then I'm not doing normal things with unprivileged domains.)

I'd either vote for enabling grant tables now to get some wider testing,
but with a warning that some corner cases still need some work, or do a
check of the places that netback calls BUG() first.

--Michael Vrable

Re: Grant Table Network Issues

[Michael Vrable]

On Sun, Aug 14, 2005 at 09:27:12AM -0700, Michael Vrable wrote:
> I'd either vote for enabling grant tables now to get some wider testing,
> but with a warning that some corner cases still need some work, or do a
> check of the places that netback calls BUG() first.

To follow up: if enabling grant tables, at least the BUG() in the
transmit-to-domU path should be replaced with something less fatal.
Full error handling and recovery (including cleaning up resources) could
be done in parallel with some wider testing.

If Xen 3.0 is being used in some mostly-production environments, we may
want to hold off just a small bit longer.

--Michael Vrable

Re: Grant Table Network Issues

[David Hopwood]

Michael Vrable wrote:
> On Sun, Aug 14, 2005 at 09:29:02AM +0100, Keir Fraser wrote:
>>On 13 Aug 2005, at 19:59, Michael Vrable wrote:
>>
>>>The line causing trouble is "BUG_ON(in_irq())". [...]
>>
>>The only code between irq_exit and kmap_skb_frag on the stack trace is 
>>unmodified Linux code. Assuming that is all correct (and presumably the 
>>same whether we enable grant tables or not) I might guess another 
>>interrupt arrives and the handler corrupts things?
> 
> I discovered the cause of this and other crashes yesterday: when grant
> tables are enabled in the netback driver, net_tx_action() and
> net_tx_action_dealloc() in netback.c each allocate large arrays from the
> kernel's stack ("gnttab_map_grant_ref_t map_ops[MAX_PENDING_REQS]" and
> "gnttab_unmap_grant_ref_t unmap_ops[MAX_PENDING_REQS]").  This results
> in a stack overflow.

Is there no way to make a kernel stack overflow fail fast with an obvious
error, rather than causing other subtle failures?

-- 
David Hopwood <david.nospam.hopwood@blueyonder.co.uk>