Re: [PATCH v5 00/20] nfsd: open file caching

Jeff Layton
2015-10-10T11:19:28Z

On Thu, 8 Oct 2015 14:04:00 -0400
"J. Bruce Fields" <bfields@fieldses.org> wrote:

> On Thu, Oct 08, 2015 at 12:55:29PM -0400, Jeff Layton wrote:
> > My bad...it needs this patch. I'll roll this into the set before the
> > next posting.
>
> Oh, good, thanks.
>
> Also, just seen on the server side--not sure what was going on at the
> time.
>
> There were a ton of these:
>
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ------------[ cut here ]------------
> Oct 08 12:35:07 f21-1.fieldses.org kernel: WARNING: CPU: 1 PID: 584 at lib/list_debug.c:59 __list_del_entry+0x9e/0xc0()
> Oct 08 12:35:07 f21-1.fieldses.org kernel: list_del corruption. prev->next should be ffff88004cb23f80, but was b6a7e8df8948e4eb
> Oct 08 12:35:07 f21-1.fieldses.org kernel: Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc
> Oct 08 12:35:07 f21-1.fieldses.org kernel: CPU: 1 PID: 584 Comm: fsnotify_mark Not tainted 4.3.0-rc3-14186-g7619b8e #322
> Oct 08 12:35:07 f21-1.fieldses.org kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140709_153950- 04/01/2014
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ffffffff81f62683 ffff880071af3d50 ffffffff8160540c ffff880071af3d98
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ffff880071af3d88 ffffffff81077692 ffff88004cb23f80 ffffffff8109c160
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ffff880071af3e08 ffff880071af3e30 ffff88004cb23f70 ffff880071af3de8
> Oct 08 12:35:07 f21-1.fieldses.org kernel: Call Trace:
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8160540c>] dump_stack+0x4e/0x82
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81077692>] warn_slowpath_common+0x82/0xc0
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8109c160>] ? sort_range+0x20/0x30
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8107771c>] warn_slowpath_fmt+0x4c/0x50
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff8162219e>] __list_del_entry+0x9e/0xc0
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff811ef485>] fsnotify_mark_destroy+0x95/0x140
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff810baa10>] ? wait_woken+0x90/0x90
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff811ef3f0>] ? fsnotify_put_mark+0x30/0x30
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098d6f>] kthread+0xef/0x110
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81a767dc>] ? _raw_spin_unlock_irq+0x2c/0x50
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098c80>] ? kthread_create_on_node+0x200/0x200
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81a7748f>] ret_from_fork+0x3f/0x70
> Oct 08 12:35:07 f21-1.fieldses.org kernel: [<ffffffff81098c80>] ? kthread_create_on_node+0x200/0x200
> Oct 08 12:35:07 f21-1.fieldses.org kernel: ---[ end trace 687abd8552e06b32 ]---
>

Thanks for the bug report! I think I understand the problem now:

It's in the way this patchset embeds a fsnotify_mark inside the
nfsd_file. The way fsnotify_destroy_mark works sort of requires that it
be freed separately since it wants to traverse these objects under a
srcu read lock. The rest of the stack traces are probably collateral
damage from that mem corruption.

I think I'll have to change the code to allocate the fsnotify_mark objects
separately. It may also be better to have just one mark per inode and
have each nfsd_file take a reference to the mark. I'll need to stare at
the code a bit longer to see what makes the most sense.

--
Jeff Layton <jlayton@poochiereds.net>