From: Trond Myklebust <trond.myklebust@primarydata.com>
To: Olga Kornievskaia <aglo@umich.edu>
Cc: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>,
Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: Yet another kernel crash in NFS4 state recovery
Date: Wed, 21 Jan 2015 14:48:07 -0500 [thread overview]
Message-ID: <1421869687.4674.2.camel@primarydata.com> (raw)
In-Reply-To: <CAN-5tyH52i+UbrxfAj=zPfQ1p6uOvE7JfcyLv2Etmm+gC=gdHQ@mail.gmail.com>
On Wed, 2015-01-21 at 14:09 -0500, Olga Kornievskaia wrote:
> On Wed, Jan 21, 2015 at 1:41 PM, Trond Myklebust
> <trond.myklebust@primarydata.com> wrote:
> > On Wed, Jan 21, 2015 at 9:47 AM, Mkrtchyan, Tigran
> > <tigran.mkrtchyan@desy.de> wrote:
> >>
> >>
> >> Now with RHEL7.
> >>
> >> [ 482.016897] BUG: unable to handle kernel NULL pointer dereference at 000000000000001a
> >> [ 482.017023] IP: [<ffffffffa01d7035>] rpc_peeraddr2str+0x5/0x30 [sunrpc]
> >> [ 482.017023] PGD baefe067 PUD baeff067 PMD 0
> >> [ 482.017023] Oops: 0000 [#1] SMP
> >> [ 482.017023] Modules linked in: nfs_layout_nfsv41_files rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables sg ppdev kvm_intel kvm pcspkr serio_raw virtio_balloon i2c_piix4 parport_pc parport mperf nfsd auth_rpcgss nfs_acl lockd sunrpc sr_mod cdrom ata_generic pata_acpi ext4 mbcache jbd2 virtio_blk cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper ttm virtio_net ata_piix drm libata virtio_pci virtio_ring virtio
> >> [ 482.017023] i2c_core floppy
> >> [ 482.017023] CPU: 0 PID: 2834 Comm: xrootd Not tainted 3.10.0-123.13.2.el7.x86_64 #1
> >> [ 482.017023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> >> [ 482.017023] task: ffff8800b188cfa0 ti: ffff880232484000 task.ti: ffff880232484000
> >> [ 482.017023] RIP: 0010:[<ffffffffa01d7035>] [<ffffffffa01d7035>] rpc_peeraddr2str+0x5/0x30 [sunrpc]
> >> [ 482.017023] RSP: 0018:ffff880232485708 EFLAGS: 00010246
> >> [ 482.017023] RAX: 000000000001bcb0 RBX: ffff880233ded800 RCX: 0000000000000000
> >> [ 482.017023] RDX: ffffffffa0494078 RSI: 0000000000000000 RDI: ffffffffffffffea
> >> [ 482.017023] RBP: ffff880232485760 R08: ffff880232485740 R09: 0000000000000000
> >> [ 482.017023] R10: 0000000000000000 R11: fffffffffffffff2 R12: ffff8800bac3e690
> >> [ 482.017023] R13: ffff8800bac3e638 R14: 0000000000000000 R15: 0000000000000000
> >> [ 482.017023] FS: 00007f0d84b79700(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
> >> [ 482.017023] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> >> [ 482.017023] CR2: 000000000000001a CR3: 00000000baefd000 CR4: 00000000000006f0
> >> [ 482.017023] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >> [ 482.017023] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> >> [ 482.017023] Stack:
> >> [ 482.017023] ffffffffa04c79a5 0000000000000000 ffff880232485768 ffffffffa046d858
> >> [ 482.017023] 0000000000000000 ffff8800b188cfa0 ffffffff81086ac0 ffff880232485740
> >> [ 482.017023] ffff880232485740 0000000096605de3 ffff880233ded800 ffff880232485778
> >> [ 482.017023] Call Trace:
> >> [ 482.017023] [<ffffffffa04c79a5>] ? nfs4_schedule_state_manager+0x65/0xf0 [nfsv4]
> >> [ 482.017023] [<ffffffffa046d858>] ? nfs_wait_client_init_complete.part.6+0x98/0xd0 [nfs]
> >> [ 482.017023] [<ffffffff81086ac0>] ? wake_up_bit+0x30/0x30
> >> [ 482.017023] [<ffffffffa04c7a5e>] nfs4_schedule_lease_recovery+0x2e/0x60 [nfsv4]
> >> [ 482.017023] [<ffffffffa04cff64>] nfs41_walk_client_list+0x104/0x340 [nfsv4]
> >> [ 482.017023] [<ffffffffa04c5679>] nfs41_discover_server_trunking+0x39/0x40 [nfsv4]
> >> [ 482.017023] [<ffffffffa04c7ecd>] nfs4_discover_server_trunking+0x7d/0x2e0 [nfsv4]
> >> [ 482.017023] [<ffffffffa04cf944>] nfs4_init_client+0x124/0x2f0 [nfsv4]
> >> [ 482.017023] [<ffffffffa0455eb4>] ? __fscache_acquire_cookie+0x74/0x2a0 [fscache]
> >> [ 482.017023] [<ffffffffa0455eb4>] ? __fscache_acquire_cookie+0x74/0x2a0 [fscache]
> >> [ 482.017023] [<ffffffffa01e62a5>] ? generic_lookup_cred+0x15/0x20 [sunrpc]
> >> [ 482.017023] [<ffffffffa01e2cc1>] ? __rpc_init_priority_wait_queue+0x81/0xc0 [sunrpc]
> >> [ 482.017023] [<ffffffffa01e2d33>] ? rpc_init_wait_queue+0x13/0x20 [sunrpc]
> >> [ 482.017023] [<ffffffffa04cf649>] ? nfs4_alloc_client+0x189/0x1e0 [nfsv4]
> >> [ 482.017023] [<ffffffffa046e6ba>] nfs_get_client+0x26a/0x320 [nfs]
> >> [ 482.017023] [<ffffffffa04cee5e>] nfs4_set_ds_client+0x8e/0xe0 [nfsv4]
> >> [ 482.017023] [<ffffffffa0521779>] nfs4_fl_prepare_ds+0xe9/0x298 [nfs_layout_nfsv41_files]
> >> [ 482.017023] [<ffffffffa051f4c6>] filelayout_read_pagelist+0x56/0x170 [nfs_layout_nfsv41_files]
> >> [ 482.017023] [<ffffffffa04d6b17>] pnfs_generic_pg_readpages+0xe7/0x270 [nfsv4]
> >> [ 482.017023] [<ffffffffa047e1c9>] nfs_pageio_doio+0x19/0x50 [nfs]
> >> [ 482.017023] [<ffffffffa047e534>] nfs_pageio_complete+0x24/0x30 [nfs]
> >> [ 482.017023] [<ffffffffa047fd2a>] nfs_readpages+0x16a/0x1d0 [nfs]
> >> [ 482.017023] [<ffffffff81141a67>] ? __page_cache_alloc+0x87/0xb0
> >> [ 482.017023] [<ffffffff8114da6c>] __do_page_cache_readahead+0x1cc/0x250
> >> [ 482.017023] [<ffffffff8114dc76>] ondemand_readahead+0x126/0x240
> >> [ 482.017023] [<ffffffff8114e051>] page_cache_sync_readahead+0x31/0x50
> >> [ 482.017023] [<ffffffff81142edb>] generic_file_aio_read+0x1ab/0x750
> >> [ 482.017023] [<ffffffffa0474971>] nfs_file_read+0x71/0xf0 [nfs]
> >> [ 482.017023] [<ffffffff811aee9d>] do_sync_read+0x8d/0xd0
> >> [ 482.017023] [<ffffffff811af57c>] vfs_read+0x9c/0x170
> >> [ 482.017023] [<ffffffff811b0242>] SyS_pread64+0x92/0xc0
> >> [ 482.017023] [<ffffffff815f2a19>] system_call_fastpath+0x16/0x1b
> >> [ 482.017023] Code: c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 c7 47 50 40 72 1d a0 48 89 e5 5d c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 <48> 8b 47 30 89 f6 55 48 c7 c2 d8 da 1f a0 48 89 e5 48 8b 84 f0
> >> [ 482.017023] RIP [<ffffffffa01d7035>] rpc_peeraddr2str+0x5/0x30 [sunrpc]
> >> [ 482.017023] RSP <ffff880232485708>
> >> [ 482.017023] CR2: 000000000000001a
> >>
> >>
> >> Looks like clp->cl_rpcclient point to nowhere when nfs4_schedule_state_manager is called.
> >>
> >
> > I'm guessing
> >
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=080af20cc945d110f9912d01cf6b66f94a375b8d
> >
>
> The Oops is seen even with that patch. As I was explained, in the
> commit you pointed at the whole client structure is null. In this case
> it's the rpcclient structure that's invalid.
Ah. You are right... Tigran, how about the following patch?
Cheers
Trond
8<---------------------------------------------------------------------
>From eb8720a31e1d36415c7377f287d5d217540830c3 Mon Sep 17 00:00:00 2001
From: Trond Myklebust <trond.myklebust@primarydata.com>
Date: Wed, 21 Jan 2015 14:37:44 -0500
Subject: [PATCH] NFSv4.1: Fix an Oops in nfs41_walk_client_list
If we start state recovery on a client that failed to initialise correctly,
then we are very likely to Oops.
Reported-by: "Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>
Link: http://lkml.kernel.org/r/130621862.279655.1421851650684.JavaMail.zimbra@desy.de
Cc: stable@vger.kernel.org
Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>
---
fs/nfs/nfs4client.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index 953daa44a282..706ad10b8186 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -639,7 +639,7 @@ int nfs41_walk_client_list(struct nfs_client *new,
prev = pos;
status = nfs_wait_client_init_complete(pos);
- if (status == 0) {
+ if (pos->cl_cons_state == NFS_CS_SESSION_INITING) {
nfs4_schedule_lease_recovery(pos);
status = nfs4_wait_clnt_recover(pos);
}
--
2.1.0
next prev parent reply other threads:[~2015-01-21 19:48 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-01-21 14:47 Yet another kernel crash in NFS4 state recovery Mkrtchyan, Tigran
2015-01-21 18:26 ` Olga Kornievskaia
2015-01-21 18:41 ` Trond Myklebust
2015-01-21 19:09 ` Olga Kornievskaia
2015-01-21 19:48 ` Trond Myklebust [this message]
2015-01-21 20:58 ` Mkrtchyan, Tigran
2015-01-24 21:07 ` Mkrtchyan, Tigran
2015-01-26 9:31 ` Mkrtchyan, Tigran
2015-01-26 12:08 ` Trond Myklebust
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1421869687.4674.2.camel@primarydata.com \
--to=trond.myklebust@primarydata.com \
--cc=aglo@umich.edu \
--cc=linux-nfs@vger.kernel.org \
--cc=tigran.mkrtchyan@desy.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox