inotify_add_watch() returning ENOSPC in 2.6.24 [watch descriptor leak?]

inotify_add_watch() returning ENOSPC in 2.6.24 [watch descriptor leak?]

[Clem Taylor]

I'm trying to move a MIPS based embedded system from 2.6.16.16 to
2.6.24. Most things seem to be working, but I'm having troubles with
inotify. The code is using inotify to detect a file written to /tmp
(tmpfs). The writer creates a file with a temporary name and then
rename()s the tmp file over the file I'm monitoring.

With 2.6.16.16, everything works fine, but with 2.6.24, the inotify
process runs for a while (~100 events) and then inotify_add_watch()
returns ENOSPC. Once this happens, I can't add new watches, even if I
kill the process and restart it. fs.inotify.max_user_instances and
fs.inotify.max_user_watches are both 128, so I'd imagine I'm hitting
this limit. For some reason the watches aren't getting cleaned up
(even after the process is killed).

In a loop, the code is doing:
    wd = inotify_add_watch(fd, file, IN_CLOSE_WRITE|IN_DELETE_SELF|IN_ONESHOT);
    blocking read on notify fd

Has something changed in the inotify() API since 2.6.16.16, or could
this be a leak?

                                      --Clem

Re: inotify_add_watch() returning ENOSPC in 2.6.24 [watch descriptor leak?]

[Andrew Morton]

On Tue, 5 Feb 2008 20:49:42 -0500 "Clem Taylor" <clem.taylor@gmail.com> wrote:

> I'm trying to move a MIPS based embedded system from 2.6.16.16 to
> 2.6.24. Most things seem to be working, but I'm having troubles with
> inotify. The code is using inotify to detect a file written to /tmp
> (tmpfs). The writer creates a file with a temporary name and then
> rename()s the tmp file over the file I'm monitoring.
> 
> With 2.6.16.16, everything works fine, but with 2.6.24, the inotify
> process runs for a while (~100 events) and then inotify_add_watch()
> returns ENOSPC. Once this happens, I can't add new watches, even if I
> kill the process and restart it. fs.inotify.max_user_instances and
> fs.inotify.max_user_watches are both 128, so I'd imagine I'm hitting
> this limit. For some reason the watches aren't getting cleaned up
> (even after the process is killed).
> 
> In a loop, the code is doing:
>     wd = inotify_add_watch(fd, file, IN_CLOSE_WRITE|IN_DELETE_SELF|IN_ONESHOT);
>     blocking read on notify fd
> 
> Has something changed in the inotify() API since 2.6.16.16, or could
> this be a leak?
> 

Good bug report, thanks.  That code was significantly altered in June 2006
and perhaps something broke.

It's a bit hard to find people who work on inotify, I'm afraid.  If you had
the time to come up with a script or program which demonstrates the bug,
that would be super-helpful?

Re: inotify_add_watch() returning ENOSPC in 2.6.24 [watch descriptor leak?]

[Clem Taylor]

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

On Feb 6, 2008 4:51 AM, Andrew Morton <akpm@linux-foundation.org> wrote:
> On Tue, 5 Feb 2008 20:49:42 -0500 "Clem Taylor" <clem.taylor@gmail.com> wrote:
> > I'm trying to move a MIPS based embedded system from 2.6.16.16 to
> > 2.6.24. Most things seem to be working, but I'm having troubles with
> > inotify. The code is using inotify to detect a file written to /tmp
> > (tmpfs). The writer creates a file with a temporary name and then
> > rename()s the tmp file over the file I'm monitoring.
> >
> > With 2.6.16.16, everything works fine, but with 2.6.24, the inotify
> > process runs for a while (~100 events) and then inotify_add_watch()
> > returns ENOSPC. Once this happens, I can't add new watches, even if I
> > kill the process and restart it. fs.inotify.max_user_instances and
> > fs.inotify.max_user_watches are both 128, so I'd imagine I'm hitting
> > this limit. For some reason the watches aren't getting cleaned up
> > (even after the process is killed).

> Good bug report, thanks.  That code was significantly altered in June 2006
> and perhaps something broke.

I also tested on a 2.6.20 x86 desktop machine. It took ~8k iterations
to fail, which matched max_user_watches. Once the program fails, it
will fail right away if it is re-run.

> It's a bit hard to find people who work on inotify, I'm afraid.  If you had
> the time to come up with a script or program which demonstrates the bug,
> that would be super-helpful?

Attached is a simple example that shows off the problem. On a system
with a problem, it will only run for about
fs.inotify.max_user_watches iterations. If everything is working, it
should run forever.

                                 Thanks,
                                 Clem

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: inotifyLeak.c --]
[-- Type: text/x-csrc; name=inotifyLeak.c, Size: 3218 bytes --]

/* Inotify IN_ONESHOT leak?
 *
 * This program loops on creating oneshot inotify watches, triggering a close
 * write event and then waiting for the event.  On 2.6.16.16 this works just
 * fine.  When I moved to 2.6.24, this code fails after ~100 events.
 * fs.inotify.max_user_instances and fs.inotify.max_user_watches are both 128,
 * so I'd imagine I am hitting this limit.
 *
 * After killing and restarting the problem, it will fail right away and only
 * a reboot will recover.
 *
 * This also fails on a desktop machine with 2.6.20. It took ~8k iterations
 * to fail, which matches the larger max_user_watches.
 *
 * Compile with:
 * gcc -Wall -o inotifyLeak inotifyLeak.c
 *
 * Worked in 2.6.16.16 [mipsel]
 * Fails in 2.6.20 [Fedora x86]
 * Fails in 2.6.24 [mipsel]
 */
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <sys/inotify.h>
#include <sys/time.h>

/* makeFile(): Create a close write event for inotify to detect. */
int makeFile ( const char *filename )
{
    FILE *file;
    struct timeval tv;

    gettimeofday ( &tv, NULL );

    file = fopen ( filename, "w" );
    if ( file == NULL )
    {
        fprintf ( stderr, "Failed to open \"%s\" for writing: %s\n",
            filename, strerror ( errno ) );
        return -1;
    }

    fprintf ( file, "%u.%06d\n", (unsigned int) tv.tv_sec, (int) tv.tv_usec );
    fclose ( file );

    return 0;
}

int main ( int argc, char *argv[] )
{
    const char filename[] = "/tmp/inotifyLeak.test";
    struct inotify_event event;
    int notifyFD, wd, ret, i;

    if ( ( notifyFD = inotify_init() ) < 0 )
    {
        fprintf ( stderr, "inotify_init() failed: %s\n", strerror ( errno ) );
        return 1;
    }

    /* create initial file */
    makeFile ( filename );

    for ( i = 0 ; ; i++ )
    {
        /* create a one shot event */
        wd = inotify_add_watch ( notifyFD, filename,
            IN_CLOSE_WRITE | IN_DELETE_SELF | IN_ONESHOT );
        if ( wd < 0 )
        {
            /* this is the failure case */
            fprintf ( stderr, "inotify_add_watch() failed: %s [i=%d]\n",
                strerror ( errno ), i );
            return 1;
        }

        /* create an event on the file */
        makeFile ( filename );

        /* blocking read, waiting for event */
        ret = read ( notifyFD, &event, sizeof(event) );
        if ( ret < 0 )
        {
            fprintf ( stderr, "inotify read() failed: %s\n",
                strerror ( errno ) );
            return 1;
        }
        else if ( ret != sizeof(event) )
        {
            fprintf ( stderr, "inotify read() returned %d not %d\n",
                ret, sizeof(event) );
            return 1;
        }
        else if ( event.wd != wd )
        {
            fprintf ( stderr, "Watch mismatch, expected %d, got %d\n",
                wd, event.wd );
            return 1;
        }

        /* if we attempt to call inotify_rm_watch(), here we get EINVAL,
         * which is expected because the watch should have been deleted
         * once the event is triggered.
         */

        /* progress report... */
        fprintf ( stderr, " %d : %d  \r", i, wd );
    }

    return 0;
}

Re: inotify_add_watch() returning ENOSPC in 2.6.24 [watch descriptor leak?]

[Amy Griffis]

Clem Taylor wrote:  [Wed Feb 06 2008, 02:40:58PM EST]
> > Good bug report, thanks.  That code was significantly altered in June 2006
> > and perhaps something broke.
> 
> I also tested on a 2.6.20 x86 desktop machine. It took ~8k iterations
> to fail, which matched max_user_watches. Once the program fails, it
> will fail right away if it is re-run.
> 
> > It's a bit hard to find people who work on inotify, I'm afraid.  If you had
> > the time to come up with a script or program which demonstrates the bug,
> > that would be super-helpful?
> 
> Attached is a simple example that shows off the problem. On a system
> with a problem, it will only run for about
> fs.inotify.max_user_watches iterations. If everything is working, it
> should run forever.

I'll take a look at this. Thanks for providing a reproducer.

Amy

Re: inotify_add_watch() returning ENOSPC in 2.6.24 [watch descriptor leak?]

[Ulisses Furquim]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

Hi,

On Feb 6, 2008 4:40 PM, Clem Taylor <clem.taylor@gmail.com> wrote:
> I also tested on a 2.6.20 x86 desktop machine. It took ~8k iterations
> to fail, which matched max_user_watches. Once the program fails, it
> will fail right away if it is re-run.

Yeah, I had the same results, and it fails afterwards because it
reaches the maximum number of watches per user.

> Attached is a simple example that shows off the problem. On a system
> with a problem, it will only run for about
> fs.inotify.max_user_watches iterations. If everything is working, it
> should run forever.

Ok, I had a go with it and found the problem. We weren't releasing
one-shot watches because the test for them was wrong. We're using the
event's mask to test for one-shot watches when we should've been using
the watch's mask.

Patch against latest Linus git repo attached.

Regards,

-- Ulisses

[-- Attachment #2: patch --]
[-- Type: application/octet-stream, Size: 1033 bytes --]

From 6fcf3bb6227a4949aeee2741dc73eb3c55a3bfef Mon Sep 17 00:00:00 2001
From: Ulisses Furquim <ulissesf@gmail.com>
Date: Thu, 7 Feb 2008 15:19:19 -0300
Subject: [PATCH] inotify: fix check for one-shot watches before destroying them

As the IN_ONESHOT bit is never set when an event is sent we must check it
in the watch's mask and not in the event's mask.

Signed-off-by: Ulisses Furquim <ulissesf@gmail.com>
---
 fs/inotify_user.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fs/inotify_user.c b/fs/inotify_user.c
index a336c97..3ab09a6 100644
--- a/fs/inotify_user.c
+++ b/fs/inotify_user.c
@@ -283,7 +283,7 @@ static void inotify_dev_queue_event(struct inotify_watch *w, u32 wd, u32 mask,
 	/* we can safely put the watch as we don't reference it while
 	 * generating the event
 	 */
-	if (mask & IN_IGNORED || mask & IN_ONESHOT)
+	if (mask & IN_IGNORED || w->mask & IN_ONESHOT)
 		put_inotify_watch(w); /* final put */
 
 	/* coalescing: drop this event if it is a dupe of the previous */
-- 
1.5.4

Re: inotify_add_watch() returning ENOSPC in 2.6.24 [watch descriptor leak?]

[Clem Taylor]

On Feb 7, 2008 1:54 PM, Ulisses Furquim <ulissesf@gmail.com> wrote:
> Ok, I had a go with it and found the problem. We weren't releasing
> one-shot watches because the test for them was wrong. We're using the
> event's mask to test for one-shot watches when we should've been using
> the watch's mask.

Thanks, this patch seems to fix the problem.

                                --Clem

Re: inotify_add_watch() returning ENOSPC in 2.6.24 [watch descriptor leak?]

[Andrew Morton]

On Thu, 7 Feb 2008 16:24:15 -0500
"Clem Taylor" <clem.taylor@gmail.com> wrote:

> On Feb 7, 2008 1:54 PM, Ulisses Furquim <ulissesf@gmail.com> wrote:
> > Ok, I had a go with it and found the problem. We weren't releasing
> > one-shot watches because the test for them was wrong. We're using the
> > event's mask to test for one-shot watches when we should've been using
> > the watch's mask.
> 
> Thanks, this patch seems to fix the problem.
> 

Awesome. Thanks, guys.