Re: Grant Table Network Issues

[Michael Vrable <mvrable@cs.ucsd.edu>]

On Sun, Aug 14, 2005 at 09:29:02AM +0100, Keir Fraser wrote:
> On 13 Aug 2005, at 19:59, Michael Vrable wrote:
> 
> >The line causing trouble is "BUG_ON(in_irq())".  In this example, I had
> >tcpdump running in both domains; this seems to trigger the problem more
> >reliably.  I've also seen a similar crash with a TCP connection, but it
> >takes a few packets before this shows up (the handshake completes, and
> >the crash happens about the time data packets come back from domain-0;
> >if checksumming optimizations are enabled, it seems the packets are
> >dropped so I don't see a crash but I don't get any data either).
> 
> On the stack trace, at irq_exit() you definitely have no hardirqs or 
> softirqs in progress. But somehow, at kmap_skb_frag(), the hardirq 
> section of the preempt mask has become non-zero. You can't have been 
> preempted to another cpu during any of this because the preempt mask is 
> continuously non-zero throughout original irq handling and subsequent 
> softirq handling.
> 
> The only code between irq_exit and kmap_skb_frag on the stack trace is 
> unmodified Linux code. Assuming that is all correct (and presumably the 
> same whether we enable grant tables or not) I might guess another 
> interrupt arrives and the handler corrupts things?

I discovered the cause of this and other crashes yesterday: when grant
tables are enabled in the netback driver, net_tx_action() and
net_tx_action_dealloc() in netback.c each allocate large arrays from the
kernel's stack ("gnttab_map_grant_ref_t map_ops[MAX_PENDING_REQS]" and
"gnttab_unmap_grant_ref_t unmap_ops[MAX_PENDING_REQS]").  This results
in a stack overflow.

This in turn causes a number of very spectacular crashes.  One of the
more subtle crashes is the in_irq() bug; the preempt count is stored in
the current thread info, at the bottom of the stack.

Allocating the arrays statically fixes the problem for me.  Steve Hand
says he'll likely be committing a fix soon.

--Michael Vrable