All of lore.kernel.org
 help / color / mirror / Atom feed
From: Willy Tarreau <w@1wt.eu>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Sabrina Dubroca <sd@queasysnail.net>,
	Aaron Brown <aaron.f.brown@intel.com>,
	Jeff Kirsher <jeffrey.t.kirsher@intel.com>,
	Ben Hutchings <ben@decadent.org.uk>, Willy Tarreau <w@1wt.eu>
Subject: [PATCH 2.6.32 16/62] e1000: add dummy allocator to fix race condition between mtu change and netpoll
Date: Sun, 13 Sep 2015 00:56:22 +0200	[thread overview]
Message-ID: <20150912225607.258086407@1wt.eu> (raw)
In-Reply-To: <08d3b586eb2e764308c3de9ee398a17c@local>

2.6.32-longterm review patch.  If anyone has any objections, please let me know.

------------------

From: Sabrina Dubroca <sd@queasysnail.net>

commit 08e8331654d1d7b2c58045e549005bc356aa7810 upstream.

There is a race condition between e1000_change_mtu's cleanups and
netpoll, when we change the MTU across jumbo size:

Changing MTU frees all the rx buffers:
    e1000_change_mtu -> e1000_down -> e1000_clean_all_rx_rings ->
        e1000_clean_rx_ring

Then, close to the end of e1000_change_mtu:
    pr_info -> ... -> netpoll_poll_dev -> e1000_clean ->
        e1000_clean_rx_irq -> e1000_alloc_rx_buffers -> e1000_alloc_frag

And when we come back to do the rest of the MTU change:
    e1000_up -> e1000_configure -> e1000_configure_rx ->
        e1000_alloc_jumbo_rx_buffers

alloc_jumbo finds the buffers already != NULL, since data (shared with
page in e1000_rx_buffer->rxbuf) has been re-alloc'd, but it's garbage,
or at least not what is expected when in jumbo state.

This results in an unusable adapter (packets don't get through), and a
NULL pointer dereference on the next call to e1000_clean_rx_ring
(other mtu change, link down, shutdown):

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff81194d6e>] put_compound_page+0x7e/0x330

    [...]

Call Trace:
 [<ffffffff81195445>] put_page+0x55/0x60
 [<ffffffff815d9f44>] e1000_clean_rx_ring+0x134/0x200
 [<ffffffff815da055>] e1000_clean_all_rx_rings+0x45/0x60
 [<ffffffff815df5e0>] e1000_down+0x1c0/0x1d0
 [<ffffffff811e2260>] ? deactivate_slab+0x7f0/0x840
 [<ffffffff815e21bc>] e1000_change_mtu+0xdc/0x170
 [<ffffffff81647050>] dev_set_mtu+0xa0/0x140
 [<ffffffff81664218>] do_setlink+0x218/0xac0
 [<ffffffff814459e9>] ? nla_parse+0xb9/0x120
 [<ffffffff816652d0>] rtnl_newlink+0x6d0/0x890
 [<ffffffff8104f000>] ? kvm_clock_read+0x20/0x40
 [<ffffffff810a2068>] ? sched_clock_cpu+0xa8/0x100
 [<ffffffff81663802>] rtnetlink_rcv_msg+0x92/0x260

By setting the allocator to a dummy version, netpoll can't mess up our
rx buffers.  The allocator is set back to a sane value in
e1000_configure_rx.

Fixes: edbbb3ca1077 ("e1000: implement jumbo receive with partial descriptors")
Signed-off-by: Sabrina Dubroca <sd@queasysnail.net>
Tested-by: Aaron Brown <aaron.f.brown@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
(cherry picked from commit f655adbac3de7ce92b2b0f9ce2d426b55b600e38)
[wt: path is drivers/net/e1000/e1000_main.c in 2.6.32]
Signed-off-by: Willy Tarreau <w@1wt.eu>
---
 drivers/net/e1000/e1000_main.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/drivers/net/e1000/e1000_main.c b/drivers/net/e1000/e1000_main.c
index 4079a33..375d332 100644
--- a/drivers/net/e1000/e1000_main.c
+++ b/drivers/net/e1000/e1000_main.c
@@ -140,6 +140,11 @@ static bool e1000_clean_rx_irq(struct e1000_adapter *adapter,
 static bool e1000_clean_jumbo_rx_irq(struct e1000_adapter *adapter,
 				     struct e1000_rx_ring *rx_ring,
 				     int *work_done, int work_to_do);
+static void e1000_alloc_dummy_rx_buffers(struct e1000_adapter *adapter,
+					 struct e1000_rx_ring *rx_ring,
+					 int cleaned_count)
+{
+}
 static void e1000_alloc_rx_buffers(struct e1000_adapter *adapter,
 				   struct e1000_rx_ring *rx_ring,
 				   int cleaned_count);
@@ -3154,8 +3159,11 @@ static int e1000_change_mtu(struct net_device *netdev, int new_mtu)
 		msleep(1);
 	/* e1000_down has a dependency on max_frame_size */
 	hw->max_frame_size = max_frame;
-	if (netif_running(netdev))
+	if (netif_running(netdev)) {
+		/* prevent buffers from being reallocated */
+		adapter->alloc_rx_buf = e1000_alloc_dummy_rx_buffers;
 		e1000_down(adapter);
+	}
 
 	/* NOTE: netdev_alloc_skb reserves 16 bytes, and typically NET_IP_ALIGN
 	 * means we reserve 2 more, this pushes us to allocate from the next
-- 
1.7.12.2.21.g234cd45.dirty




  parent reply	other threads:[~2015-09-12 23:12 UTC|newest]

Thread overview: 87+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-09-12 22:56 [PATCH 2.6.32 00/62] 2.6.32.68-longterm review Willy Tarreau
2015-09-12 22:56 ` Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 01/62] pipe: iovec: Fix memory corruption when retrying atomic copy as non-atomic Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 02/62] sg_start_req(): make sure that theres not too many elements in iovec Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 03/62] crypto: testmgr - update LZO compression test vectors Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 04/62] TTY: drop driver reference in tty_open fail path Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 05/62] netlink: fix possible spoofing from non-root processes Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 06/62] eCryptfs: Remove buggy and unnecessary write in file name decode routine Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 07/62] HID: fix a couple of off-by-ones Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 08/62] udf: Verify i_size when loading inode Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 09/62] udf: Verify symlink size before loading it Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 11/62] udf: Check path length when reading symlink Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 12/62] udf: Check component length before reading it Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 13/62] Remove repeated loads blocksize Willy Tarreau
2015-09-15  1:42   ` Ben Hutchings
2015-09-15  7:39     ` Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 14/62] udf: Check length of extended attributes and allocation descriptors Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 15/62] udp: fix behavior of wrong checksums Willy Tarreau
2015-09-15  1:44   ` Ben Hutchings
2015-09-15  7:41     ` Willy Tarreau
2015-09-12 22:56 ` Willy Tarreau [this message]
2015-09-12 22:56 ` [PATCH 2.6.32 17/62] powerpc: Fix missing L2 cache size in /sys/devices/system/cpu Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 18/62] s390/hibernate: fix save and restore of kernel text section Willy Tarreau
2015-09-15  2:10   ` Ben Hutchings
2015-09-15  6:09     ` Heiko Carstens
2015-09-15  7:41       ` Willy Tarreau
2015-09-15  7:44         ` Heiko Carstens
2015-09-12 22:56 ` [PATCH 2.6.32 19/62] ptrace: fix race between ptrace_resume() and wait_task_stopped() Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 20/62] memstick: mspro_block: add missing curly braces Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 21/62] md/raid5: dont record new size if resize_stripes fails Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 22/62] powerpc: Align TOC to 256 bytes Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 23/62] jbd2: fix r_count overflows leading to buffer overflow in journal recovery Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 24/62] sd: Disable support for 256 byte/sector disks Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 25/62] lguest: fix out-by-one error in address checking Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 26/62] fs, omfs: add NULL terminator in the end up the token list Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 27/62] x86_64: Fix strnlen_user() to not touch memory after specified maximum Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 28/62] tracing: Have filter check for balanced ops Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 29/62] sctp: Fix race between OOTB responce and route removal Willy Tarreau
2015-09-15  2:26   ` Ben Hutchings
2015-09-15  7:43     ` Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 30/62] x86/reboot: Fix a warning message triggered by stop_other_cpus() Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 31/62] include/linux/sched.h: dont use task->pid/tgid in same_thread_group/has_group_leader_pid Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 32/62] __ptrace_may_access() should not deny sub-threads Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 33/62] MIPS: Octeon: Remove udelay() causing huge IRQ latency Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 34/62] MIPS: Fix race condition in lazy cache flushing Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 35/62] MIPS: Fix cpu_has_mips_r2_exec_hazard Willy Tarreau
2015-09-15  3:02   ` Ben Hutchings
2015-09-15  7:43     ` Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 36/62] MIPS: Octeon: Delete override of cpu_has_mips_r2_exec_hazard Willy Tarreau
2015-09-15 11:37   ` Ben Hutchings
2015-09-16  5:17     ` Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 37/62] hrtimer: Allow concurrent hrtimer_start() for self restarting timers Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 38/62] s5h1420: fix a buffer overflow when checking userspace params Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 39/62] cx24116: " Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 40/62] pktgen: adjust spacing in proc file interface output Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 41/62] staging: vt6655: device_rx_srv check sk_buff is NULL Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 42/62] fixing infinite OPEN loop in 4.0 stateid recovery Willy Tarreau
2015-09-14 23:54   ` Kornievskaia, Olga
2015-09-15  5:37     ` Willy Tarreau
     [not found]       ` <7E69BA76-E3DF-4389-A8C4-F23C9E1FD5CC@netapp.com>
     [not found]         ` <55D3ECC9-EC69-469F-AD46-EE3818F3D138@netapp.com>
2015-09-16  5:33           ` Willy Tarreau
2015-11-15  0:53             ` [stable] Failing to send a CLOSE if file is opened WRONLY and server reboots on a 4.x mount Ben Hutchings
2015-11-15  7:20               ` Willy Tarreau
2015-11-18 23:20               ` Luis Henriques
2015-11-18 23:20                 ` Luis Henriques
2015-09-12 22:56 ` [PATCH 2.6.32 43/62] SUNRPC: Fix a memory leak in the backchannel code Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 44/62] dmaengine: mv_xor: bug fix for racing condition in descriptors cleanup Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 45/62] ext4: fix race between truncate and __ext4_journalled_writepage() Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 46/62] Disable write buffering on Toshiba ToPIC95 Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 47/62] nfs: increase size of EXCHANGE_ID name string buffer Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 48/62] ext4: call sync_blockdev() before invalidate_bdev() in put_super() Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 49/62] NET: ROSE: Dont dereference NULL neighbour pointer Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 50/62] mm: kmemleak: allow safe memory scanning during kmemleak disabling Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 51/62] tracing/filter: Do not WARN on operand count going below zero Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 52/62] tracing/filter: Do not allow infix to exceed end of string Willy Tarreau
2015-09-12 22:56 ` [PATCH 2.6.32 53/62] fuse: initialize fc->release before calling it Willy Tarreau
2015-09-12 22:57 ` [PATCH 2.6.32 54/62] mm: avoid setting up anonymous pages into file mapping Willy Tarreau
2015-09-12 22:57 ` [PATCH 2.6.32 55/62] s390/process: fix sfpc inline assembly Willy Tarreau
2015-09-12 22:57 ` [PATCH 2.6.32 56/62] libata: increase the timeout when setting transfer mode Willy Tarreau
2015-09-12 22:57 ` [PATCH 2.6.32 57/62] x86/xen: Probe target addresses in set_aliased_prot() before the hypercall Willy Tarreau
2015-09-12 22:57 ` Willy Tarreau
2015-09-12 22:57 ` [PATCH 2.6.32 58/62] dccp: fix auto-loading of dccp(_probe) Willy Tarreau
2015-09-12 22:57 ` [PATCH 2.6.32 60/62] dccp: catch failed request_module call in dccp_probe init Willy Tarreau
2015-09-12 22:57 ` [PATCH 2.6.32 61/62] dmaengine: fix missing cnt in ?: in dmatest Willy Tarreau
2015-09-12 22:57 ` [PATCH 2.6.32 62/62] ipv6: Fix return of xfrm6_tunnel_rcv() Willy Tarreau
2015-09-12 23:18 ` [PATCH 2.6.32 00/62] 2.6.32.68-longterm review Willy Tarreau
2015-09-15 12:06 ` Ben Hutchings
2015-09-16  5:23   ` Willy Tarreau
2015-09-16  5:23     ` Willy Tarreau

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20150912225607.258086407@1wt.eu \
    --to=w@1wt.eu \
    --cc=aaron.f.brown@intel.com \
    --cc=ben@decadent.org.uk \
    --cc=jeffrey.t.kirsher@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=sd@queasysnail.net \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.