Re: [PATCH v5 00/20] nfsd: open file caching

J. Bruce Fields
2015-10-10T13:48:15Z

On Sat, Oct 10, 2015 at 07:19:23AM -0400, Jeff Layton wrote:
> On Thu, 8 Oct 2015 14:04:00 -0400
> "J. Bruce Fields" <bfields@fieldses.org> wrote:
>
> > On Thu, Oct 08, 2015 at 12:55:29PM -0400, Jeff Layton wrote:
> > > My bad...it needs this patch. I'll roll this into the set before the
> > > next posting.
> >
> > Oh, good, thanks.
> >
> > Also, just seen on the server side--not sure what was going on at the
> > time.
> >
> > There were a ton of these:
> >
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: ------------[ cut here ]------------
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: WARNING: CPU: 1 PID: 584 at lib/list_debug.c:59 __list_del_entry+0x9e/0xc0()
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: list_del corruption. prev->next should be ffff88004cb23f80, but was b6a7e8df8948e4eb
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: CPU: 1 PID: 584 Comm: fsnotify_mark Not tainted 4.3.0-rc3-14186-g7619b8e #322
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: ffffffff81f62683 ffff880071af3d50 ffffffff8160540c ffff880071af3d98
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: ffff880071af3d88 ffffffff81077692 ffff88004cb23f80 ffffffff8109c160
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: ffff880071af3e08 ffff880071af3e30 ffff88004cb23f70 ffff880071af3de8
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: Call Trace:
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8160540c>] dump_stack+0x4e/0x82
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81077692>] warn_slowpath_common+0x82/0xc0
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8109c160>] ? sort_range+0x20/0x30
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8107771c>] warn_slowpath_fmt+0x4c/0x50
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8162219e>] __list_del_entry+0x9e/0xc0
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff811ef485>] fsnotify_mark_destroy+0x95/0x140
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff810baa10>] ? wait_woken+0x90/0x90
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff811ef3f0>] ? fsnotify_put_mark+0x30/0x30
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098d6f>] kthread+0xef/0x110
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81a767dc>] ? _raw_spin_unlock_irq+0x2c/0x50
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098c80>] ? kthread_create_on_node+0x200/0x200
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81a7748f>] ret_from_fork+0x3f/0x70
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098c80>] ? kthread_create_on_node+0x200/0x200
> > Oct 08 12:35:07 f21-1.fieldses.org kernel: ---[ end trace 687abd8552e06b32 ]---
> >
>
> Thanks for the bug report! I think I understand the problem now:
>
> It's in the way this patchset embeds a fsnotify_mark inside the
> nfsd_file. The way fsnotify_destroy_mark works sort of requires that it
> be freed separately since it wants to traverse these objects under a
> srcu read lock. The rest of the stack traces are probably collateral
> damage from that mem corruption.
>
> I think I'll have to change the code to allocate the fsnotify_mark objects
> separately. It may also be better to have just one mark per inode and
> have each nfsd_file take a reference to the mark. I'll need to stare at
> the code a bit longer to see what makes the most sense.

OK, thanks!

--b.