From: "J. Bruce Fields" <bfields@fieldses.org>
To: Dave Jones <davej@redhat.com>,
Linux Kernel <linux-kernel@vger.kernel.org>,
linux-nfs@vger.kernel.org,
Weston Andros Adamson <dros@netapp.com>,
Trond Myklebust <Trond.Myklebust@netapp.com>
Subject: Re: nfsd oops on Linus' current tree.
Date: Fri, 21 Dec 2012 13:08:24 -0500 [thread overview]
Message-ID: <20121221180824.GA27729@fieldses.org> (raw)
In-Reply-To: <20121221153348.GA32151@redhat.com>
On Fri, Dec 21, 2012 at 10:33:48AM -0500, Dave Jones wrote:
> Did a mount from a client (also running Linus current), and the
> server spat this out..
>
> [ 6936.306135] ------------[ cut here ]------------
> [ 6936.306154] WARNING: at net/sunrpc/clnt.c:617 rpc_shutdown_client+0x12a/0x1b0 [sunrpc]()
This is a warning added by 168e4b39d1afb79a7e3ea6c3bb246b4c82c6bdb9
"SUNRPC: add WARN_ON_ONCE for potential deadlock", pointing out that
nfsd is calling shutdown_client from a workqueue, which is a problem
because shutdown_client has to wait on rpc tasks that run on a
workqueue.
I don't believe there's any circular dependency among the workqueues
(we're calling shutdown_client from callback_wq, not rpciod_workqueue),
but 168e4b39d1afb.. says that we could get a deadlock if both are
running on the same kworker thread.
I'm not sure what to do about that.
> [ 6936.306156] Hardware name:
> [ 6936.306157] Modules linked in: ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables xfs coretemp iTCO_wdt iTCO_vendor_support snd_emu10k1 microcode snd_util_mem snd_ac97_codec ac97_bus snd_hwdep snd_seq snd_pcm snd_page_alloc snd_timer e1000e snd_rawmidi snd_seq_device snd emu10k1_gp pcspkr i2c_i801 soundcore gameport lpc_ich mfd_core i82975x_edac edac_core vhost_net tun macvtap macvlan kvm_intel kvm binfmt_misc nfsd auth_rpcgss nfs_acl lockd sunrpc btrfs libcrc32c zlib_deflate usb_storage firewire_ohci firewire_core sata_sil crc_itu_t radeon i2c_algo_bit drm_kms_helper ttm drm i2c_core floppy
> [ 6936.306214] Pid: 52, comm: kworker/u:2 Not tainted 3.7.0+ #34
> [ 6936.306216] Call Trace:
> [ 6936.306224] [<ffffffff8106badf>] warn_slowpath_common+0x7f/0xc0
> [ 6936.306227] [<ffffffff8106bb3a>] warn_slowpath_null+0x1a/0x20
> [ 6936.306235] [<ffffffffa02c62ca>] rpc_shutdown_client+0x12a/0x1b0 [sunrpc]
> [ 6936.306240] [<ffffffff81368318>] ? delay_tsc+0x98/0xf0
> [ 6936.306252] [<ffffffffa034a60b>] nfsd4_process_cb_update.isra.16+0x4b/0x230 [nfsd]
> [ 6936.306256] [<ffffffff8109677c>] ? __rcu_read_unlock+0x5c/0xa0
> [ 6936.306260] [<ffffffff81370d46>] ? debug_object_deactivate+0x46/0x130
> [ 6936.306269] [<ffffffffa034a87d>] nfsd4_do_callback_rpc+0x8d/0xa0 [nfsd]
> [ 6936.306272] [<ffffffff810900f7>] process_one_work+0x207/0x760
> [ 6936.306274] [<ffffffff81090087>] ? process_one_work+0x197/0x760
> [ 6936.306277] [<ffffffff81090afe>] ? worker_thread+0x21e/0x440
> [ 6936.306285] [<ffffffffa034a7f0>] ? nfsd4_process_cb_update.isra.16+0x230/0x230 [nfsd]
> [ 6936.306289] [<ffffffff81090a3e>] worker_thread+0x15e/0x440
> [ 6936.306292] [<ffffffff810908e0>] ? rescuer_thread+0x250/0x250
> [ 6936.306295] [<ffffffff8109b16d>] kthread+0xed/0x100
> [ 6936.306299] [<ffffffff810dd86e>] ? put_lock_stats.isra.25+0xe/0x40
> [ 6936.306302] [<ffffffff8109b080>] ? kthread_create_on_node+0x160/0x160
> [ 6936.306307] [<ffffffff81711e2c>] ret_from_fork+0x7c/0xb0
> [ 6936.306310] [<ffffffff8109b080>] ? kthread_create_on_node+0x160/0x160
> [ 6936.306312] ---[ end trace 5bab69e086ae3c6f ]---
> [ 6936.363213] ------------[ cut here ]------------
> [ 6936.363226] WARNING: at fs/nfsd/vfs.c:937 nfsd_vfs_read.isra.13+0x197/0x1b0 [nfsd]()
This warning is unrelated, and is probably just carelessness on my part:
I couldn't see why this condition would happen, and I stuck the warning
in there without looking much harder. Probably we should just revert
79f77bf9a4e3dd5ead006b8f17e7c4ff07d8374e "nfsd: warn on odd reply state
in nfsd_vfs_read" while I go stare at the code.
--b.
> [ 6936.363229] Hardware name:
> [ 6936.363230] Modules linked in: ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables xfs coretemp iTCO_wdt iTCO_vendor_support snd_emu10k1 microcode snd_util_mem snd_ac97_codec ac97_bus snd_hwdep snd_seq snd_pcm snd_page_alloc snd_timer e1000e snd_rawmidi snd_seq_device snd emu10k1_gp pcspkr i2c_i801 soundcore gameport lpc_ich mfd_core i82975x_edac edac_core vhost_net tun macvtap macvlan kvm_intel kvm binfmt_misc nfsd auth_rpcgss nfs_acl lockd sunrpc btrfs libcrc32c zlib_deflate usb_storage firewire_ohci firewire_core sata_sil crc_itu_t radeon i2c_algo_bit drm_kms_helper ttm drm i2c_core floppy
> [ 6936.363284] Pid: 699, comm: nfsd Tainted: G W 3.7.0+ #34
> [ 6936.363286] Call Trace:
> [ 6936.363293] [<ffffffff8106badf>] warn_slowpath_common+0x7f/0xc0
> [ 6936.363296] [<ffffffff8106bb3a>] warn_slowpath_null+0x1a/0x20
> [ 6936.363302] [<ffffffffa031ef77>] nfsd_vfs_read.isra.13+0x197/0x1b0 [nfsd]
> [ 6936.363310] [<ffffffffa0321948>] nfsd_read_file+0x88/0xb0 [nfsd]
> [ 6936.363317] [<ffffffffa0332956>] nfsd4_encode_read+0x186/0x260 [nfsd]
> [ 6936.363325] [<ffffffffa03391cc>] nfsd4_encode_operation+0x5c/0xa0 [nfsd]
> [ 6936.363333] [<ffffffffa032e5a9>] nfsd4_proc_compound+0x289/0x780 [nfsd]
> [ 6936.363339] [<ffffffffa0319e5b>] nfsd_dispatch+0xeb/0x230 [nfsd]
> [ 6936.363355] [<ffffffffa02d3d38>] svc_process_common+0x328/0x6d0 [sunrpc]
> [ 6936.363365] [<ffffffffa02d4433>] svc_process+0x103/0x160 [sunrpc]
> [ 6936.363371] [<ffffffffa031921b>] nfsd+0xdb/0x160 [nfsd]
> [ 6936.363378] [<ffffffffa0319140>] ? nfsd_destroy+0x210/0x210 [nfsd]
> [ 6936.363381] [<ffffffff8109b16d>] kthread+0xed/0x100
> [ 6936.363385] [<ffffffff810dd86e>] ? put_lock_stats.isra.25+0xe/0x40
> [ 6936.363388] [<ffffffff8109b080>] ? kthread_create_on_node+0x160/0x160
> [ 6936.363393] [<ffffffff81711e2c>] ret_from_fork+0x7c/0xb0
> [ 6936.363396] [<ffffffff8109b080>] ? kthread_create_on_node+0x160/0x160
> [ 6936.363398] ---[ end trace 5bab69e086ae3c70 ]---
>
next prev parent reply other threads:[~2012-12-21 18:08 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-12-21 15:33 nfsd oops on Linus' current tree Dave Jones
2012-12-21 18:08 ` J. Bruce Fields [this message]
2012-12-21 18:40 ` Myklebust, Trond
2012-12-21 23:08 ` J. Bruce Fields
2012-12-21 23:15 ` Myklebust, Trond
2012-12-21 23:26 ` J. Bruce Fields
2012-12-21 23:36 ` Myklebust, Trond
2012-12-21 23:45 ` J. Bruce Fields
2013-01-03 16:28 ` Adamson, Dros
2013-01-03 20:11 ` J. Bruce Fields
2013-01-03 20:27 ` Adamson, Dros
2013-01-03 20:52 ` J. Bruce Fields
2013-01-03 22:15 ` Tejun Heo
2013-01-03 22:08 ` Tejun Heo
2013-01-03 22:12 ` Myklebust, Trond
2013-01-03 22:26 ` Tejun Heo
2013-01-03 22:34 ` Tejun Heo
2013-01-03 23:11 ` Myklebust, Trond
[not found] ` <1357254692.55285.33.camel@lade.trondhjem.org>
2013-01-03 23:26 ` Myklebust, Trond
2013-01-04 17:11 ` Adamson, Dros
2013-01-04 18:15 ` [PATCH 1/3] SUNRPC: Ensure that we free the rpc_task after cleanups are done Trond Myklebust
2013-01-04 18:15 ` [PATCH 2/3] NFS: Ensure that we free the rpc_task after read and write " Trond Myklebust
2013-01-04 18:15 ` [PATCH 3/3] SUNRPC: Partial revert of commit 168e4b39d1afb79a7e3ea6c3bb246b4c82c6bdb9 Trond Myklebust
2013-01-04 18:29 ` [PATCH 2/3] NFS: Ensure that we free the rpc_task after read and write cleanups are done Bruce Fields
2013-01-04 18:52 ` Myklebust, Trond
2013-01-04 19:12 ` Bruce Fields
2013-01-07 17:47 ` [PATCH 1/3] SUNRPC: Ensure that we free the rpc_task after " Tejun Heo
2013-01-07 17:54 ` Myklebust, Trond
2013-01-03 22:03 ` nfsd oops on Linus' current tree Tejun Heo
2013-01-03 23:08 ` J. Bruce Fields
2012-12-22 0:48 ` [PATCH] Revert "nfsd: warn on odd reply state in nfsd_vfs_read" J. Bruce Fields
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121221180824.GA27729@fieldses.org \
--to=bfields@fieldses.org \
--cc=Trond.Myklebust@netapp.com \
--cc=davej@redhat.com \
--cc=dros@netapp.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.