order:1 failure in ipoib_cm_alloc_rx_skb

order:1 failure in ipoib_cm_alloc_rx_skb

[Venkat Venkatsubra]

Hello All,

Many of our customers have hit this order:1 failure in ipoib_cm_alloc_rx_skb causing softlockup:
....
        skb = dev_alloc_skb(IPOIB_CM_HEAD_SIZE + 12);
....

IPOIB_CM_HEAD_SIZE is defined as
      IPOIB_ENCAP_LEN     = 4,
	IPOIB_CM_MTU	  = 0x10000 - 0x10, /* padding to align header to 16 */
 	IPOIB_CM_BUF_SIZE	  = IPOIB_CM_MTU  + IPOIB_ENCAP_LEN,
	IPOIB_CM_HEAD_SIZE  = IPOIB_CM_BUF_SIZE % PAGE_SIZE,

For a 4kb PAGE_SIZE IPOIB_CM_HEAD_SIZE ends up as
(65536 - 16 + 4) % 4096 = 4084 bytes for a dev_alloc_skb(4084 + 12) or dev_alloc_skb(4096).

Redefining IPOIB_CM_HEAD_SIZE and IPOIB_CM_RX_SG as 
	IPOIB_CM_HEAD_SIZE  = SKB_MAX_HEAD(NET_SKB_PAD + 12),
	IPOIB_CM_RX_SG = ALIGN(IPOIB_CM_BUF_SIZE-IPOIB_CM_HEAD_SIZE,PAGE_SIZE)/PAGE_SIZE + 1,
resolves the problem by avoiding the order:1 allocation.

But in releases where NET_SKB_PAD is defined with max() as
#define NET_SKB_PAD     max(32, L1_CACHE_BYTES)
we get compilation error:
  CC [M]  drivers/infiniband/ulp/ipoib/ipoib_main.o
In file included from drivers/infiniband/ulp/ipoib/ipoib_main.c:35:
drivers/infiniband/ulp/ipoib/ipoib.h:76: error: braced-group within expression allowed only inside a function

Changing it to a #define moves the error to when IPOIB_CM_RX_SG is used.
#define IPOIB_CM_HEAD_SIZE SKB_MAX_HEAD(NET_SKB_PAD + 12)
#define IPOIB_CM_RX_SG  ALIGN(IPOIB_CM_BUF_SIZE-IPOIB_CM_HEAD_SIZE,PAGE_SIZE)/PAGE_SIZE + 1

The same compilation error comes from the "u64 mapping[IPOIB_CM_RX_SG];" line in 
struct ipoib_cm_rx_buf {
        struct sk_buff *skb;
        u64 mapping[IPOIB_CM_RX_SG];
};

How should I fix this compilation problem without having to make too many changes ?

I am including the stack trace of the order:1 failure:
 kswapd0: page allocation failure. order:1, mode:0x20 
 Pid: 273, comm: kswapd0 Not tainted 2.6.32-400.11.1.el5uek #1 
 Call Trace: 
  <IRQ>  [<ffffffff810ddf74>] __alloc_pages_nodemask+0x524/0x595 
  [<ffffffff8110da3f>] kmem_getpages+0x4f/0xf4 
  [<ffffffff8110dc3c>] fallback_alloc+0x158/0x1ce 
  [<ffffffff8110ddd3>] ____cache_alloc_node+0x121/0x134 
  [<ffffffff8110e3f3>] kmem_cache_alloc_node_notrace+0x84/0xb9 
  [<ffffffff8110e46e>] __kmalloc_node+0x46/0x73 
  [<ffffffff813b9aa8>] ? __alloc_skb+0x72/0x13d 
  [<ffffffff813b9aa8>] __alloc_skb+0x72/0x13d 
  [<ffffffff813b9bdb>] alloc_skb+0x13/0x15 
  [<ffffffff813b9f11>] dev_alloc_skb+0x1b/0x38 
  [<ffffffffa029e722>] ipoib_cm_alloc_rx_skb+0x31/0x1de [ib_ipoib] 
  [<ffffffffa029fd04>] ipoib_cm_handle_rx_wc+0x3a1/0x5b8 [ib_ipoib] 
  [<ffffffffa0191bdc>] ? mlx4_ib_free_srq_wqe+0x27/0x54 [mlx4_ib] 
  [<ffffffffa01894d4>] ? mlx4_ib_poll_cq+0x620/0x65e [mlx4_ib] 
  [<ffffffff813b9015>] ? __kfree_skb+0x79/0x7e 
  [<ffffffffa029e9f7>] ? netif_tx_lock+0x44/0x71 [ib_ipoib] 
  [<ffffffffa029ae97>] ipoib_poll+0x87/0x128 [ib_ipoib] 
  [<ffffffff813c4b69>] net_rx_action+0xc6/0x1cd 
  [<ffffffff8105e8cd>] __do_softirq+0xd7/0x19e 
  [<ffffffff810aefdc>] ? handle_IRQ_event+0x66/0x120 
  [<ffffffff81012eec>] call_softirq+0x1c/0x30 
  [<ffffffff81014695>] do_softirq+0x46/0x89 
  [<ffffffff8105e752>] irq_exit+0x3b/0x7a 
  [<ffffffff8145bea1>] do_IRQ+0x99/0xb0 
  [<ffffffff81012713>] ret_from_intr+0x0/0x11 
  <EOI>  [<ffffffff812379a3>] ? radix_tree_delete+0x8f/0x194 
  [<ffffffffa03693b3>] ? __nfs_access_zap_cache+0x75/0xb0 [nfs] 
  [<ffffffff81207d05>] ? ima_inode_free+0x35/0x55 
  [<ffffffff8112fd9c>] ? __destroy_inode+0x26/0x66 
  [<ffffffff8112fdf2>] ? destroy_inode+0x16/0x44 
  [<ffffffff81130074>] ? dispose_list+0xb2/0xe1 
  [<ffffffff81130251>] ? shrink_icache_memory+0x1ae/0x1e0 
  [<ffffffff810e3f83>] ? shrink_slab+0xe1/0x153 
  [<ffffffff810e5063>] ? kswapd+0x3dd/0x516 
  [<ffffffff810e26f3>] ? isolate_pages_global+0x0/0x1ba 
  [<ffffffff810432be>] ? need_resched+0x23/0x2d 
  [<ffffffff81077030>] ? autoremove_wake_function+0x0/0x3d 
  [<ffffffff810e4c86>] ? kswapd+0x0/0x516 
  [<ffffffff81076c87>] ? kthread+0x6e/0x76 
  [<ffffffff81012dea>] ? child_rip+0xa/0x20 
  [<ffffffff81076c19>] ? kthread+0x0/0x76 
  [<ffffffff81012de0>] ? child_rip+0x0/0x20

Thanks.

Venkat

RE: order:1 failure in ipoib_cm_alloc_rx_skb

[David Laight]

From: Venkat Venkatsubra
> Hello All,
> 
> Many of our customers have hit this order:1 failure in ipoib_cm_alloc_rx_skb causing softlockup:
> ....
>         skb = dev_alloc_skb(IPOIB_CM_HEAD_SIZE + 12);
> ....

Notwithstanding any other problems, that allocation could fail even if the size were
small enough that the entire skb would fit in one page.
So the driver needs changing so that the allocation failure doesn't cause a deadlock.

OTOH if you can't allocate 8k of contiguous address space and memory the system
is unlikely to be working very well at all.

	David