Re: stuck in inotify_release

[Jan Kara <jack@suse.cz>]

Hello,

On Thu 28-03-19 09:26:45, Olivier Chapelliere wrote:
> According to what I read on internet you seem to be the right person to get
> in touch with when one has problems with inotify.

Yes, there's also linux-fsdevel@vger.kernel.org mailing list which we use
(added to CC).

> We are monitoring several directories in python processes through inotify.
> But after few days all processes are stuck in a call to inotify_release.
> Once I detected the problem, I dumped info to dmesg with sysrq-trigger
> (dmesg content attached):
> echo w > /proc/sysrq-trigger

Looking through the stack traces, all of them wait in fput() ->
inotify_release() -> ... -> fsnotify_wait_marks_destroyed() ->
flush_delayed_work(&reaper_work). So they wait for worker process to
destroy all marks for the group. However that worker (kworker/u8:4) is
stuck in:

fsnotify_mark_destroy_workfn() -> synchronize_srcu(&fsnotify_mark_srcu)

So the question is who is holding fsnotify_mark_srcu so that SRCU cannot
declare new grace period. I don't see any such process among the processes
you've shown in the dump (but it should be there) so it's a bit of a
mystery.

> Our production env is ubuntu 18.04 kernel 4.15 fs ext4
> This problem appears on a weekly basis so I will be able to run additional
> commands to track down the issue if needed.

So when this happens again, try grabbing output of sysrq-l and sysrq-t if
we can find the task holding fsnotify_mark_srcu.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR