PROBLEM: 2.6.35.7 Inotify events missing

PROBLEM: 2.6.35.7 Inotify events missing

[Sylvain Rochet]

[-- Attachment #1.1: Type: text/plain, Size: 4854 bytes --]

Hi,

I write a tool that keep in sync one source (master) to multiple 
targets (slaves) to provide always up to date mirrors in order to 
distribute static content at different locations.

That was working fine on 2.6.26.5 kernel, except bugs that are my 
fault and which are fixed the way the time goes.

A day I upgraded to 2.6.33.5, then 2.6.33.7, finally to 2.6.35.7, and I 
always end up with the same ending, it seems inotify can miss some VFS 
events from time to time.

I wrote a simple tool which is a cleaned code of the first 
explained master process that only display received inotify events and 
with the recursion stuff that keep all directories watched.


First, this is not due to queue limit depth, limits are way over 
requirements. I don't received IN_Q_OVERFLOW event either.

$ cat /proc/sys/fs/inotify/max_{queued_events,user_instances,user_watches}
524288
128
1000000


That may help, files and FS tree are modified over NFS.


Here is what I got:

# stat ./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.xhtml
  File: `./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.xhtml'
  Size: 11696           Blocks: 24         IO Block: 4096   regular file
Device: 90ah/2314d      Inode: 144835214   Links: 1
Access: (0644/-rw-r--r--)  Uid: (20981/ UNKNOWN)   Gid: (26734/ UNKNOWN)
Access: 2010-03-18 14:04:59.000000000 +0000
Modify: 2010-10-14 06:44:27.000000000 +0000
Change: 2010-10-14 06:44:27.000000000 +0000

# stat ./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.i
  File: `./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.i'
  Size: 30575           Blocks: 64         IO Block: 4096   regular file
Device: 90ah/2314d      Inode: 144835393   Links: 1
Access: (0644/-rw-r--r--)  Uid: (20981/ UNKNOWN)   Gid: (26734/ UNKNOWN)
Access: 2010-03-18 14:04:59.000000000 +0000
Modify: 2010-10-14 06:44:26.000000000 +0000
Change: 2010-10-14 06:44:26.000000000 +0000

So, here is two files that were modified at about 2010-10-14 06:44:26.

And now is what I got in the event log of my inotify logger:

# grep db303d42803e58bb5f73467291c9a863 debugevents 
2010/10/12 20:06:48 1286914008:604497: wd=16623 mask=2 cookie=0 len=48 name=db303d42803e58bb5f73467291c9a863.i pathname=./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.i
2010/10/12 20:06:48 1286914008:605215: wd=16623 mask=2 cookie=0 len=48 name=db303d42803e58bb5f73467291c9a863.i pathname=./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.i
2010/10/12 20:06:48 1286914008:686793: wd=16623 mask=2 cookie=0 len=48 name=db303d42803e58bb5f73467291c9a863.xhtml pathname=./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.xhtml
2010/10/12 20:06:48 1286914008:687372: wd=16623 mask=2 cookie=0 len=48 name=db303d42803e58bb5f73467291c9a863.xhtml pathname=./xnlegacies/statique/wiki/cache/d/db303d42803e58bb5f73467291c9a863.xhtml

I did not receive events about the modification of those files at 
2010-10-14 06:44, but we received previous events 2 days ago



Here is another example:

# stat ./synfig/forums/files/thumb_5_ec61d61306649e73f2549775c2ee3e1c   
  File: `./synfig/forums/files/thumb_5_ec61d61306649e73f2549775c2ee3e1c'
  Size: 23886           Blocks: 48         IO Block: 4096   regular file
Device: 90ah/2314d      Inode: 178390723   Links: 1
Access: (0664/-rw-rw-r--)  Uid: (15653/ UNKNOWN)   Gid: (27452/ UNKNOWN)
Access: 2010-10-15 05:28:35.000000000 +0000
Modify: 2010-10-15 05:28:35.000000000 +0000
Change: 2010-10-15 05:28:35.000000000 +0000

Here is a file that was modified at 05:28:35.


# grep thumb_5_ec61d61306649e73f2549775c2ee3e1c debugevents 
2010/10/15 05:28:35 1287120515:058819: wd=16079 mask=100 cookie=0 len=48 name=thumb_5_ec61d61306649e73f2549775c2ee3e1c pathname=./synfig/forums/files/thumb_5_ec61d61306649e73f2549775c2ee3e1c

I received an event that the file was created, at the time of the event 
the file was 0 byte length, the file was then filled, but we did not 
received events about the fact that the file was modified.


If I try to do the same, I get IN_CREATE, then IN_MODIFY,

# echo "test" > hFq0gUrK1EGwoT2pnpCO

# grep hFq0gUrK1EGwoT2pnpCO debugevents 
2010/10/18 19:40:59 1287430859:712722: wd=1 mask=100 cookie=0 len=32 name=hFq0gUrK1EGwoT2pnpCO pathname=./hFq0gUrK1EGwoT2pnpCO
2010/10/18 19:40:59 1287430859:713543: wd=1 mask=2 cookie=0 len=32 name=hFq0gUrK1EGwoT2pnpCO pathname=./hFq0gUrK1EGwoT2pnpCO



I attached my quick-n-dirty inotify logger. Source code of the full 
replication software is available here[1], if it helps.


I can provide as many information as possible and I am willing to do so ;)

Regards,
Sylvain



[1] svn://svn.tuxfamily.org/svnroot/vhffs4/vhffs/trunk/vhffs-fssync/

[-- Attachment #1.2: vhffsfssync_log.c --]
[-- Type: text/x-csrc, Size: 19706 bytes --]

/*
 *  VHFFSFSSYNC: Scalable file system replication over TCP
 *
 *  Copyright 2008  Sylvain Rochet <gradator@gradator.net>
 *
 *  This program is free software; you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License as published by
 *  the Free Software Foundation; either version 3 of the License, or
 *  (at your option) any later version.
 *
 *  This program is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *  GNU General Public License for more details.
 *
 *  You should have received a copy of the GNU General Public License
 *  along with this program.  If not, see http://www.gnu.org/licenses/.
 */

#ifndef __linux__
#error This software is only running on Linux-based OS, bye!
#endif

#define _FILE_OFFSET_BITS 64

#define DEBUG_NET 1
#define DEBUG_INOTIFY 1
#define DEBUG_EVENTS 1
//#define NDEBUG

#include <assert.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <sys/time.h>
#include <fcntl.h>
#include <dirent.h>
#include <string.h>
#include <errno.h>
#include <sys/inotify.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <glib.h>
#include <sys/select.h>
#include <signal.h>
#include <sys/socket.h>
#include <netdb.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <sys/sendfile.h>
#include <getopt.h>


char vhffsfssync_debugtsstr[256];

char *vhffsfssync_debugts() {
	time_t t;
	struct tm *tmp;
	struct timeval tv;

	t = time(NULL);
	tmp = localtime(&t);
	gettimeofday(&tv, NULL);

	strftime(vhffsfssync_debugtsstr, 200, "%Y/%m/%d %H:%M:%S ", tmp);
	sprintf(vhffsfssync_debugtsstr+strlen(vhffsfssync_debugtsstr), "%lld:%.6ld" , (long long)tv.tv_sec, (long)tv.tv_usec );

	return vhffsfssync_debugtsstr;
}


/* -- inotify stuff -- */

#define VHFFSFSSYNC_BUF_LEN 4096
//#define VHFFSFSSYNC_WATCH_MASK IN_ATTRIB|IN_CREATE|IN_DELETE|IN_CLOSE_WRITE|IN_MODIFY|IN_MOVED_FROM|IN_MOVED_TO|IN_DONT_FOLLOW|IN_ONLYDIR
#define VHFFSFSSYNC_WATCH_MASK IN_ATTRIB|IN_CREATE|IN_DELETE|IN_MODIFY|IN_MOVED_FROM|IN_MOVED_TO|IN_DONT_FOLLOW|IN_ONLYDIR
// Not used yet: IN_DELETE_SELF, IN_MOVE_SELF
// Will never be used: IN_ACCESS, IN_OPEN, IN_CLOSE_NOWRITE

// each monitor entry is associated with a path, we need to keep it to compute the path
//char **vhffsfssync_wd_to_watch = NULL;
//int vhffsfssync_wd_to_watch_len = 0;  // number of allocated paths
GHashTable *vhffsfssync_wd_to_watch;


typedef struct vhffsfssync_watch_ {
	int wd;
	char *dirname;
	struct vhffsfssync_watch_ *parent;
} vhffsfssync_watch;

// return a timestamp in ms (it loops for 100000 sec)
/*inline int vhffsfssync_timestamp()  {
	struct timeval tv;
	gettimeofday(&tv, NULL);
 	return (tv.tv_sec%100000)*1000+tv.tv_usec/1000;
}*/

struct vhffsfssync_cookie {
	uint32_t id;
	vhffsfssync_watch *watch;
	char *filename;
	gboolean isdir;
};
static struct vhffsfssync_cookie vhffsfssync_cookie;

// protos
char *vhffsfssync_pathname(vhffsfssync_watch *watch, const char *filename);
vhffsfssync_watch *vhffsfssync_add_watch(int inotifyfd, vhffsfssync_watch *parent, const char *dirname, uint32_t mask);
int vhffsfssync_del_watch(int inotifyfd, vhffsfssync_watch *watch);
vhffsfssync_watch *vhffsfssync_add_watch_recursively(int inotifyfd, vhffsfssync_watch *parent, const char *dirname, uint32_t mask);
int vhffsfssync_manage_event_remove(int inotifyfd, vhffsfssync_watch *watch, char *filename);
int vhffsfssync_manage_event_create(int inotifyfd, vhffsfssync_watch *watch, char *filename);
int vhffsfssync_manage_event(int inotifyfd, struct inotify_event *event);
int vhffsfssync_fake_events_recursively(int inotifyfd, vhffsfssync_watch *watch);


/* -- inotify stuff -- */

char *vhffsfssync_pathname(vhffsfssync_watch *watch, const char *filename)  {

	GString *pathname = g_string_sized_new(256);
	char **dirnames, **curnames, **endnames;
	uint32_t a;

	a = 16;
	dirnames = malloc( a * sizeof(char*) );
	curnames = dirnames;
	endnames = dirnames+a;

	if(filename) {
		*(curnames++) = (char*)filename;
	}

	while(watch) {
		*(curnames++) = watch->dirname;
		watch = watch->parent;
		if(curnames == endnames) {
			a += 16;
			dirnames = realloc( dirnames, a * sizeof(char*) );
			curnames = dirnames+a-16;
			endnames = dirnames+a;
		}
	}

	curnames--;
	g_string_append(pathname, *(curnames--) );
	while( curnames >= dirnames ) {
		g_string_append_c(pathname, '/');
		g_string_append(pathname, *(curnames--)  );
	}
	free(dirnames);

	return g_string_free(pathname, FALSE);
}


vhffsfssync_watch *vhffsfssync_add_watch(int inotifyfd, vhffsfssync_watch *parent, const char *dirname, uint32_t mask)  {

	int wd;
	char *pathname;
	vhffsfssync_watch *watch;

	pathname = vhffsfssync_pathname(parent, dirname);
#if DEBUG_INOTIFY
	printf("%s: t+ %s\n", vhffsfssync_debugts(), pathname);
#endif

	wd = inotify_add_watch(inotifyfd, pathname, mask);
	if(wd < 0) {
		if(errno == ENOSPC)  {
			fprintf(stdout, "%s: Maximum number of watches reached, consider adding more...\n", vhffsfssync_debugts() );
		}
		free(pathname);
		return NULL;
	}

 	if( (watch = g_hash_table_lookup(vhffsfssync_wd_to_watch, &wd)) ) {

		// this was already watched, update name and reattach to the new parent
		free(watch->dirname);
		watch->dirname = g_strdup(dirname);
		watch->parent = parent;

#if DEBUG_INOTIFY
		printf("%s: u+ %d %s\n", vhffsfssync_debugts(), wd, pathname);
#endif
		free(pathname);
		return watch;
	}

	watch = malloc(sizeof(vhffsfssync_watch));
	watch->wd = wd;
	watch->dirname = g_strdup(dirname);
	watch->parent = parent;

//	_wd = g_new(int, 1);
//	*_wd = wd;
	g_hash_table_insert(vhffsfssync_wd_to_watch, &watch->wd, watch);

//	if(wd >= vhffsfssync_wd_to_watch_len)  {
//		vhffsfssync_wd_to_watch_len = ( (wd >>10) +1) <<10;
//		vhffsfssync_wd_to_watch = realloc( vhffsfssync_wd_to_watch, vhffsfssync_wd_to_watch_len * sizeof(void*) );
//	}
//	vhffsfssync_wd_to_watch[wd] = strdup(pathname);
#if DEBUG_INOTIFY
	printf("%s: a+ %d %s\n", vhffsfssync_debugts(), wd, pathname);
#endif
	free(pathname);
	return watch;
}


int vhffsfssync_del_watch(int inotifyfd, vhffsfssync_watch *watch)  {

#if DEBUG_INOTIFY
	char *pathname = vhffsfssync_pathname(watch, NULL);
	printf("%s: - %d %s\n", vhffsfssync_debugts(), watch->wd, pathname);
	free(pathname);
#endif
	g_hash_table_remove(vhffsfssync_wd_to_watch, &watch->wd);
	inotify_rm_watch(inotifyfd, watch->wd);
	free(watch->dirname);
	free(watch);
	return 0;
}


vhffsfssync_watch *vhffsfssync_add_watch_recursively(int inotifyfd, vhffsfssync_watch *parent, const char *dirname, uint32_t mask)  {

	vhffsfssync_watch *watch;
	char *pathname;
	DIR *d;

	watch = vhffsfssync_add_watch(inotifyfd, parent, dirname, mask);
	if(!watch) return NULL;

	pathname = vhffsfssync_pathname(parent, dirname);
	d = opendir(pathname);
	free(pathname);
	if(d) {
		struct dirent *dir;
		while( (dir = readdir(d)) )  {
			if(dir->d_type == DT_DIR && strcmp(dir->d_name, ".") && strcmp(dir->d_name, "..") )  {
				vhffsfssync_add_watch_recursively(inotifyfd, watch, dir->d_name, mask);
			}
		}
		closedir(d);
	}

	return watch;
}


int vhffsfssync_manage_event_remove(int inotifyfd, vhffsfssync_watch *watch, char *filename)  {
	char *pathname;

	pathname = vhffsfssync_pathname(watch, filename);

#if DEBUG_INOTIFY
	printf("%s: ==> REMOVE %s\n", vhffsfssync_debugts(), pathname);
#endif

	free(pathname);
	return 0;
}


int vhffsfssync_manage_event_create(int inotifyfd, vhffsfssync_watch *watch, char *filename)  {
	struct stat st;
	char *pathname;

	pathname = vhffsfssync_pathname(watch, filename);

	if(! lstat(pathname, &st) )  {

		if( S_ISREG(st.st_mode) )  {
#if DEBUG_INOTIFY
			printf("%s: ==> CREATE %s\n", vhffsfssync_debugts(), pathname);
#endif
		}

		else if( S_ISDIR(st.st_mode) )  {
			vhffsfssync_watch *newwatch;
#if DEBUG_INOTIFY
			printf("%s: ==> MKDIR %s\n", vhffsfssync_debugts(), pathname);
#endif
			newwatch = vhffsfssync_add_watch(inotifyfd, watch, filename, VHFFSFSSYNC_WATCH_MASK);
			/* there is a short delay between the mkdir() and the add_watch(),
			   we need to send events about the data which have already been written */
			vhffsfssync_fake_events_recursively( inotifyfd, newwatch );
		}

		else if( S_ISLNK(st.st_mode) )  {
			char *linkto;
			int ret;
			linkto = malloc(st.st_size +1);
			ret = readlink(pathname, linkto, st.st_size);
			if( ret >= 0 )  {
				linkto[st.st_size] = '\0';
#if DEBUG_INOTIFY
				printf("%s: ==> SYMLINK %s -> %s\n", vhffsfssync_debugts(), pathname, linkto);
#endif
			}
			free(linkto);
			if(ret < 0) {
				if(errno == ENOENT) {
					// file already disappeared (common for temporary files)
				} else {
					fprintf(stdout, "%s: cannot readlink() '%s': %s\n", vhffsfssync_debugts(), pathname, strerror(errno));
					free(pathname);
					return -1;
				}
			}

		}
		/* we don't need other file types (chr, block, fifo, socket, ...) */
	}
	else {
		if(errno == ENOENT) {
			// file already disappeared (common for temporary files)
		} else {
			fprintf(stdout, "%s: cannot lstat() '%s': %s\n", vhffsfssync_debugts(), pathname, strerror(errno));
			free(pathname);
			return -1;
		}
	}

	free(pathname);
	return 0;
}


int vhffsfssync_manage_event(int inotifyfd, struct inotify_event *event)  {

	vhffsfssync_watch *watch;
	char *pathname;
#if DEBUG_INOTIFY
	printf("%s: wd=%d mask=%x cookie=%d len=%d", vhffsfssync_debugts(), event->wd, event->mask, event->cookie, event->len);
	if(event->len > 0) printf(" name=%s", event->name);
#endif

	if(event->wd < 0) {
		fprintf(stdout, "\n%s: Maximum number of events reached, some events are lost\n", vhffsfssync_debugts() );
		return -1;
	}

	watch = g_hash_table_lookup(vhffsfssync_wd_to_watch, &event->wd);
	assert( watch != NULL );

	if(event->len > 0)  {
		pathname = vhffsfssync_pathname(watch, event->name);
	} else {
		pathname = vhffsfssync_pathname(watch, NULL);
	}

	printf(" pathname=%s\n", pathname);

	// this event is not waiting for a cookie, delete file if necessary (IN_MOVED_FROM not followed with IN_MOVED_TO)
	if( !(event->mask & IN_MOVED_TO) && vhffsfssync_cookie.id )  {

		vhffsfssync_manage_event_remove(inotifyfd, vhffsfssync_cookie.watch, vhffsfssync_cookie.filename);
		vhffsfssync_cookie.id = 0;
		free(vhffsfssync_cookie.filename);
	}

	// new mtime, mode, owner, group (and also other stuff like atime, but we are not using them)
	if( event->mask & IN_ATTRIB )  {
		struct stat st;
#if DEBUG_INOTIFY
		printf("%s: IN_ATTRIB\n", vhffsfssync_debugts() );
#endif
		if(! lstat(pathname, &st) )  {
		
		}
		else {
			if(errno == ENOENT) {
				// file already disappeared (common for temporary files)
			} else {
				fprintf(stdout, "%s: cannot lstat() '%s': %s\n", vhffsfssync_debugts(), pathname, strerror(errno));
			}
		}

	// new file, directory, or symlink
	} else if( event->mask & IN_CREATE )  {
#if DEBUG_INOTIFY
		printf("%s: IN_CREATE\n", vhffsfssync_debugts() );
#endif
		vhffsfssync_manage_event_create(inotifyfd, watch, event->name);

	// deleted file, directory or symlink
	} else if( event->mask & IN_DELETE )  {
#if DEBUG_INOTIFY
		printf("%s: IN_DELETE\n", vhffsfssync_debugts() );
#endif
		vhffsfssync_manage_event_remove(inotifyfd, watch, event->name);

	// watch deleted, not used
	} else if( event->mask & IN_DELETE_SELF )  {
#if DEBUG_INOTIFY
		printf("%s: IN_DELETE_SELF\n", vhffsfssync_debugts());
#endif
		// We don't send REMOVE here because the dir can be deleted before the
		// event was added, in this case the add_watch failed to monitor this dir
		// and we'll not receive a IN_DELETE_SELF for it
		//
		// Anyway, a IN_IGNORE event will be sent, IN_DELETE_SELF is only
		// useful for monitored files, that is not used here.

	// file modified
	} else if( event->mask & IN_MODIFY )  {
#if DEBUG_INOTIFY
		printf("%s: IN_MODIFY\n", vhffsfssync_debugts() );
		/* we can send the data here */
		printf("%s: ==> SEND %s\n", vhffsfssync_debugts(), pathname);
#endif

	// file modified and closed
	} else if( event->mask & IN_CLOSE_WRITE )  {
#if DEBUG_INOTIFY
		printf("%s: IN_CLOSE_WRITE\n", vhffsfssync_debugts() );
		/* we must send the data here */
		printf("%s: ==> SEND %s\n", vhffsfssync_debugts(), pathname);
#endif

	// watch moved, not used
	} else if( event->mask & IN_MOVE_SELF )  {
#if DEBUG_INOTIFY
		printf("%s: IN_MOVE_SELF\n", vhffsfssync_debugts() );
#endif
		// not needed (we can rely on IN_MOVED_FROM and IN_MOVED_TO)

	// file/symlink/directory moved
	//
	// only from: delete the file/symlink/directory
	// only to: create the file/symlink/directory
	// both: mv the file/symlink/directory
	//
	} else if( event->mask & IN_MOVED_FROM )  {
#if DEBUG_INOTIFY
		printf("%s: IN_MOVED_FROM\n", vhffsfssync_debugts() );
#endif
		// set the cookie
		vhffsfssync_cookie.id = event->cookie;
		vhffsfssync_cookie.watch = watch;
		vhffsfssync_cookie.filename = strdup(event->name);
		vhffsfssync_cookie.isdir = !!( event->mask & IN_ISDIR );

	} else if( event->mask & IN_MOVED_TO )  {
#if DEBUG_INOTIFY
		printf("%s: IN_MOVED_TO\n", vhffsfssync_debugts() );
#endif
		// mv
		if(vhffsfssync_cookie.id == event->cookie)  {
#if DEBUG_INOTIFY
			char *tmp = vhffsfssync_pathname(vhffsfssync_cookie.watch, vhffsfssync_cookie.filename);
			printf("%s: ==> MOVE (%d -> %d) %s -> %s  (used cookie %d)\n", vhffsfssync_debugts(), vhffsfssync_cookie.watch->wd, watch->wd, tmp, pathname, vhffsfssync_cookie.id);
			free(tmp);
#endif
			if( vhffsfssync_cookie.isdir )  {

				char *frompathname = vhffsfssync_pathname(vhffsfssync_cookie.watch, vhffsfssync_cookie.filename);
				vhffsfssync_add_watch(inotifyfd, watch, event->name, VHFFSFSSYNC_WATCH_MASK);
				free(frompathname);
			}
			else {
				vhffsfssync_manage_event_remove(inotifyfd, vhffsfssync_cookie.watch, vhffsfssync_cookie.filename);
				vhffsfssync_manage_event_create(inotifyfd, watch, event->name);
			}

			vhffsfssync_cookie.id = 0;
			free(vhffsfssync_cookie.filename);
		}
		// create
		else  {
			vhffsfssync_manage_event_create(inotifyfd, watch, event->name);
		}

	// watch deleted, clean it
	} else if( event->mask & IN_IGNORED )  {
#if DEBUG_INOTIFY
		printf("%s: IN_IGNORED\n", vhffsfssync_debugts() );
#endif
		vhffsfssync_del_watch(inotifyfd, watch);

	// this event is not handled, this should not happen
	} else {
#if DEBUG_INOTIFY
		printf("%s: OOOOOOOPPPSS!!!!!\n", vhffsfssync_debugts()  );
#endif
	}

	free(pathname);
	return 0;
}


int vhffsfssync_fake_events_recursively(int inotifyfd, vhffsfssync_watch *watch)  {
	DIR *d;
	char *pathname;

	pathname = vhffsfssync_pathname(watch, NULL);
	d = opendir(pathname);
	free(pathname);

	if(d) {
		struct dirent *dir;
		while( (dir = readdir(d)) ) {
			if( strcmp(dir->d_name, ".") && strcmp(dir->d_name, "..") ) {
				// recursivity is done through vhffsfssync_manage_event_create()
				// which calls this function
				vhffsfssync_manage_event_create(inotifyfd, watch, dir->d_name);
			}
		}
		closedir(d);
	}

	return 0;
}


static void usage_exit(int ret_code, char *progname)  {
	printf ("Usage: %s [OPTION]... DIRECTORY\n"
		"Remote synchronous file-copying tool, this is the server (the master)\n\n"
		"  -f, --foreground\tDon't daemonise the server, display errors on the console\n"
		"  -b, --bind=IP\t\tListen to the specified IP address\n"
		"  -p, --port=PORT\tListen to this port\n"
		"      --pidfile=PATH\tWrite the pid to that file\n"
		"  -h, --help\t\tDisplay this help and exit\n"
		"  -v, --version\t\tOutput version information and exit\n",
		progname);
	exit(ret_code);
}


int main(int argc, char *argv[])  {

	int inotifyfd, flags;
	vhffsfssync_watch *watch;

	char *root = NULL;

	struct option long_options[] = {
		{ "help", no_argument, NULL, 'h' },
		{ "version", no_argument, NULL, 'v' },
		{ 0, 0, 0, 0 }
	};

	while(1) {
		int option_index = 0, c;
		c = getopt_long(argc, argv, "hv", long_options, &option_index);
		if(c == -1)
			break;

		switch(c) {
			case 'h':
				usage_exit(0, argv[0]);

			case 'v':
#ifdef VERSION
				fputs("vhffsfssync_master " VERSION "\n", stdout);
#else
				fputs("vhffsfssync_master\n", stdout);
#endif
				exit(0);

			case '?':
				/* `getopt_long' already printed an error message. */
				fprintf(stdout,"Try `%s --help' for more information.\n", argv[0]);
				exit(1);

			default:
				abort();
		}
	}

	if(optind != argc-1)
		usage_exit(1, argv[0]);

	root = argv[optind++];

	/* chdir() to the filesystem to monitor */
	if(!root) return -1;
	if( root[strlen(root)-1] == '/' ) root[strlen(root)-1] = '\0';
#if DEBUG_INOTIFY
	printf("%s: Monitoring %s\n", vhffsfssync_debugts(), root);
#endif
	if( chdir(root) < 0 ) {
		fprintf(stdout, "%s: cannot chdir() to %s: %s\n", vhffsfssync_debugts(), root, strerror(errno));
		return -1;
	}
	if( chroot(".") < 0 ) {
		fprintf(stdout, "%s: cannot chroot() to %s: %s\n", vhffsfssync_debugts(), root, strerror(errno));
		//return -1;
	}
	root = ".";

	/* -- inotify stuff -- */

	vhffsfssync_wd_to_watch = g_hash_table_new_full(g_int_hash, g_int_equal, NULL, NULL);
	vhffsfssync_cookie.watch = NULL;
	vhffsfssync_cookie.id = 0;
	vhffsfssync_cookie.filename = NULL;

	inotifyfd = inotify_init();

	/* set inotifyfd to non-blocking */
	flags = fcntl(inotifyfd, F_GETFL);
	if(flags >= 0) {
		flags |= O_NONBLOCK;
		fcntl(inotifyfd, F_SETFL, flags);
	}

	watch = vhffsfssync_add_watch_recursively(inotifyfd, NULL, root, VHFFSFSSYNC_WATCH_MASK);
	if(!watch)  {
		fprintf(stdout, "%s: Maximum number of watches probably reached, consider adding more or fixing what is being wrong before running me again (strace is your friend)... byebye!\n", vhffsfssync_debugts() );
		return -1;
	}

	printf("%s: Ready\n", vhffsfssync_debugts() );

	/* -- main loop -- */
	while(1)  {
		int max_fd = 0;
		fd_set readfs;
		int ret;

		FD_ZERO(&readfs);

		/* inotify events */
		FD_SET(inotifyfd, &readfs);
		if(inotifyfd > max_fd) max_fd = inotifyfd;

		ret = select(max_fd + 1, &readfs, NULL, NULL, NULL);
		if(ret < 0)  {
			switch(errno)  {
				case EAGAIN:
				case EINTR:
					break;
				default:
					fprintf(stdout, "%s: select() failed: %s\n", vhffsfssync_debugts(), strerror(errno));
			}
		}
		if(ret > 0)  {
			/* inotify events */
			if( FD_ISSET(inotifyfd, &readfs) )  {
				char buf[VHFFSFSSYNC_BUF_LEN];
				ssize_t len;

				len = read(inotifyfd, buf, VHFFSFSSYNC_BUF_LEN);
				if(len < 0)  {
					switch(errno)  {
						case EAGAIN:
						case EINTR:
							break;
						default:
							fprintf(stdout, "%s: read() failed on inotify fd(%d): %s\n", vhffsfssync_debugts(), inotifyfd, strerror(errno));
					}
				}
				else {
					char *cur = buf;
					int n=0;
					while(len > 0)  {
						int register next;
						struct inotify_event *ie;

						ie = (struct inotify_event*)cur;
						vhffsfssync_manage_event( inotifyfd, ie );
						next = sizeof(struct inotify_event);
						next += ie->len;
						len -= next;
						cur += next;
						n++;
					}
					//printf("COIN: %d events read\n", n);;
				}
			}

		}

		fflush(stdout);
//		sleep(10);
	}

	return 0;
}

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 3039 bytes --]

Hi,

On Tue, Oct 19, 2010 at 12:35:40AM +0200, Sylvain Rochet wrote:
> 
> ... upgraded to 2.6.33.5, then 2.6.33.7, finally to 2.6.35.7, and I 
> always end up with the same ending, it seems inotify can miss some VFS 
> events from time to time.

I finally find out why.

The NFS server does not always know the name of the modified file, if 
the modified inode was cleared from the VFS cache fsnotify does not know 
as well the filename then inotify child events on directories are 
silently tossed.

Easy way to reproduce:

Add a few printk debug (here it only works if /data is the NFS export):

--- begin//fs/nfsd/vfs.c        2011-07-22 04:17:23.000000000 +0200
+++ linux-3.0/fs/nfsd/vfs.c     2011-07-30 03:18:17.837560809 +0200
@@ -975,6 +975,8 @@
        inode = dentry->d_inode;
        exp   = fhp->fh_export;
 
+       printk("nfsd write inode=%ld name=%s\n", inode->i_ino, dentry->d_name.name);
+
        /*
         * Request sync writes if
         *  -   the sync export option has been set, or

diff -Nru begin//include/linux/fsnotify.h linux-3.0/include/linux/fsnotify.h
--- begin//include/linux/fsnotify.h     2011-07-22 04:17:23.000000000 +0200
+++ linux-3.0/include/linux/fsnotify.h  2011-07-30 03:07:00.330239062 +0200
@@ -216,8 +232,15 @@
                mask |= FS_ISDIR;
 
        if (!(file->f_mode & FMODE_NONOTIFY)) {
+               if( !strcmp(path->mnt->mnt_mountpoint->d_name.name, "data") )
+                       printk("fsnotify modify inode=%ld name=%s\n", inode->i_ino, file->f_dentry->d_name.name);
                fsnotify_parent(path, NULL, mask);
                fsnotify(inode, mask, path, FSNOTIFY_EVENT_PATH, NULL, 0);
+       } else {
+               if( !strcmp(path->mnt->mnt_mountpoint->d_name.name, "data") )
+                       printk("fsnotify modify-nonotify inode=%ld name=%s\n", inode->i_ino, file->f_dentry->d_name.name);
        }
 }


On the NFS client, open a fd and send some data:

# exec 1> test
# ls -la
# 

On the NFS server, check the kern log:

Aug 20 00:57:44 inotifydebug kernel: nfsd write inode=13 name=test
Aug 20 00:57:44 inotifydebug kernel: fsnotify modify inode=13 name=test

Everything goes well.

Now, clear the VFS cache on the NFS server:

# echo 3 > /proc/sys/vm/drop_caches 

On the NFS client, send some data to the fd:

# ls -la
# 

On the NFS server, check the kern log:

Aug 20 00:58:56 inotifydebug kernel: nfsd write inode=13 name=
Aug 20 00:58:56 inotifydebug kernel: fsnotify modify inode=13 name=

The filename is lost, fsnotify does not know the filename anymore, 
therefore inotify cannot send event about a modified file in a watched 
directory.

End of the story.

I guess this is almost impossible to fix this fsnotify bug, this is due 
by the fact that NFS use inode as file identifiers, so in some case this 
is impossible to know the modified filepath, and therefore impossible to 
match the file event to the directory watch.

Kind regards,
Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Jamie Lokier]

Sylvain Rochet wrote:
> Hi,
> 
> On Tue, Oct 19, 2010 at 12:35:40AM +0200, Sylvain Rochet wrote:
> > 
> > ... upgraded to 2.6.33.5, then 2.6.33.7, finally to 2.6.35.7, and I 
> > always end up with the same ending, it seems inotify can miss some VFS 
> > events from time to time.
> 
> I finally find out why.
> 
> The NFS server does not always know the name of the modified file, if 
> the modified inode was cleared from the VFS cache fsnotify does not know 
> as well the filename then inotify child events on directories are 
> silently tossed.
> 
> Easy way to reproduce:
> 
> Add a few printk debug (here it only works if /data is the NFS export):
> 
> --- begin//fs/nfsd/vfs.c        2011-07-22 04:17:23.000000000 +0200
> +++ linux-3.0/fs/nfsd/vfs.c     2011-07-30 03:18:17.837560809 +0200
> @@ -975,6 +975,8 @@
>         inode = dentry->d_inode;
>         exp   = fhp->fh_export;
>  
> +       printk("nfsd write inode=%ld name=%s\n", inode->i_ino, dentry->d_name.name);
> +
>         /*
>          * Request sync writes if
>          *  -   the sync export option has been set, or
> 
> diff -Nru begin//include/linux/fsnotify.h linux-3.0/include/linux/fsnotify.h
> --- begin//include/linux/fsnotify.h     2011-07-22 04:17:23.000000000 +0200
> +++ linux-3.0/include/linux/fsnotify.h  2011-07-30 03:07:00.330239062 +0200
> @@ -216,8 +232,15 @@
>                 mask |= FS_ISDIR;
>  
>         if (!(file->f_mode & FMODE_NONOTIFY)) {
> +               if( !strcmp(path->mnt->mnt_mountpoint->d_name.name, "data") )
> +                       printk("fsnotify modify inode=%ld name=%s\n", inode->i_ino, file->f_dentry->d_name.name);
>                 fsnotify_parent(path, NULL, mask);
>                 fsnotify(inode, mask, path, FSNOTIFY_EVENT_PATH, NULL, 0);
> +       } else {
> +               if( !strcmp(path->mnt->mnt_mountpoint->d_name.name, "data") )
> +                       printk("fsnotify modify-nonotify inode=%ld name=%s\n", inode->i_ino, file->f_dentry->d_name.name);
>         }
>  }
> 
> 
> On the NFS client, open a fd and send some data:
> 
> # exec 1> test
> # ls -la
> # 
> 
> On the NFS server, check the kern log:
> 
> Aug 20 00:57:44 inotifydebug kernel: nfsd write inode=13 name=test
> Aug 20 00:57:44 inotifydebug kernel: fsnotify modify inode=13 name=test
> 
> Everything goes well.
> 
> Now, clear the VFS cache on the NFS server:
> 
> # echo 3 > /proc/sys/vm/drop_caches 
> 
> On the NFS client, send some data to the fd:
> 
> # ls -la
> # 
> 
> On the NFS server, check the kern log:
> 
> Aug 20 00:58:56 inotifydebug kernel: nfsd write inode=13 name=
> Aug 20 00:58:56 inotifydebug kernel: fsnotify modify inode=13 name=
> 
> The filename is lost, fsnotify does not know the filename anymore, 
> therefore inotify cannot send event about a modified file in a watched 
> directory.
> 
> End of the story.
> 
> I guess this is almost impossible to fix this fsnotify bug, this is due 
> by the fact that NFS use inode as file identifiers, so in some case this 
> is impossible to know the modified filepath, and therefore impossible to 
> match the file event to the directory watch.

Oh dear, that's a security hole, if something is using inotify/dnotify
to watch and assumes that file contents (on the same machine,
i.e. server in this case) do not change if there's no event received.

It also breaks cache applications which make the same assumption.  Is
a solution to open inotify watches on every file individually?  If so
that seems quite severe.

I do quite like the idea of using it to break past fanotify security
restrictions though ;-)

Can it also be bypassed with sys_open_by_handle_at?


Possible solution:

One way to look at this as like NFS having a secret hard link to the
file, which does not show up in st_nlink.

Hard links are already a bit tricky with fsnotify and directory
watches.  You can monitor a directory, but a file in it can change
contents through another path.

However, you can track changes of hard-linked files accurately by
either putting a watch directly on all files whose st_nlink >= 2,
and/or making sure you have watches on enough distinct directories
that they contain st_nlink entries for the same file between them,
because at least one of those directories will get an event.  This is
quite practical: You watch the files directly, until such time as you
have found all its links (if you ever do), then you can remove the
direct file watches.

That gives me an idea to help with the NFS no-name watching:

It looks like when a file is referenced by inode without a path, the
problem is there's no path, so no directory inode to receive the
event?

Then this can be solved, in principle (if there's no better way), by
watching a "virtual directory" that gets all events for when the
access doesn't have a parent directory.  There needs to be some way to
watch it, and some way to get the appropriate file from the event (as
there is no real directory.  Or maybe there could be a virtual
filesystem (like /proc, /sys etc.) containing a magic directory that
receives these inode-only events, such that lookups in that directory
yield the affected file.  Exactly as if the directory contains a hard
link to every file, perhaps a text encoding of the handles passed
through sys_open_by_handle_at.

-- Jamie

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 4246 bytes --]

Hi Jamie,


On Sat, Aug 20, 2011 at 12:37:56AM +0100, Jamie Lokier wrote:
> 
> Oh dear, that's a security hole, if something is using inotify/dnotify
> to watch and assumes that file contents (on the same machine,
> i.e. server in this case) do not change if there's no event received.
> 
> It also breaks cache applications which make the same assumption.
> 
> I do quite like the idea of using it to break past fanotify security
> restrictions though ;-)

It also probably means that fanotify misses some events when a filesystem 
is modified over NFS. If fanotify is used the way it is designed, i.e. 
with an antiviruse software, this may be an interesting way to skip the 
antiviruse check.

Here we go:

NFS server, run the fanotify example tool:

~/fanotify-example# ./fanotify -m /data/

NFS client, open a fd then do some I/O:

# exec 1> test
# ls -la
# 

NFS server log:

/data/test: pid=1235 modify close(writable)

NFS server, cache clearing:

# echo 3 > /proc/sys/vm/drop_caches 

NFS client, more I/O:

# ls -la

NFS server log:

/data: pid=1234 modify close(writable)

We receive an event... which is obviously wrong. This is even worse than 
no event at all, we receive an event about the wrong inode, the parent 
inode of the modified file actually.


> Is a solution to open inotify watches on every file individually?  If 
> so that seems quite severe.

This is what I am going to do, at least temporarily, I only need to 
watch about a million file (and slowly counting).

The startup time to watch an entire filesystem using inotify already 
require a full filesystem walk, watching all files and directories 
instead of directories only will not change much because most of the 
time is spent waiting I/O operations. This may however require a lot 
more memory both on kernel side and userland side.


> Can it also be bypassed with sys_open_by_handle_at?

No clue, this should be checked but there is no evident reason that it 
cannot be bypassed this way as well.


> Possible solution:
> 
> One way to look at this as like NFS having a secret hard link to the
> file, which does not show up in st_nlink.
> 
> Hard links are already a bit tricky with fsnotify and directory
> watches.  You can monitor a directory, but a file in it can change
> contents through another path.
> 
> However, you can track changes of hard-linked files accurately by
> either putting a watch directly on all files whose st_nlink >= 2,
> and/or making sure you have watches on enough distinct directories
> that they contain st_nlink entries for the same file between them,
> because at least one of those directories will get an event.  This is
> quite practical: You watch the files directly, until such time as you
> have found all its links (if you ever do), then you can remove the
> direct file watches.

Yup, I agree.


> That gives me an idea to help with the NFS no-name watching:
> 
> It looks like when a file is referenced by inode without a path, the
> problem is there's no path, so no directory inode to receive the
> event?

No filepath and no filename at all actually. There is no way to find the 
(or "a" if the file is linked to more than one directory) filename other 
than walking among all the directory tree to find where the inode is 
linked. We need a directory entry (dirent) to send an event about a 
modified file inside a watched directory.


> Then this can be solved, in principle (if there's no better way), by
> watching a "virtual directory" that gets all events for when the
> access doesn't have a parent directory.  There needs to be some way to
> watch it, and some way to get the appropriate file from the event (as
> there is no real directory.  Or maybe there could be a virtual
> filesystem (like /proc, /sys etc.) containing a magic directory that
> receives these inode-only events, such that lookups in that directory
> yield the affected file.  Exactly as if the directory contains a hard
> link to every file, perhaps a text encoding of the handles passed
> through sys_open_by_handle_at.

By doing that, we'll only get the inode nb as we cannot fetch the filename.


Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Al Viro]

On Sat, Aug 20, 2011 at 12:37:56AM +0100, Jamie Lokier wrote:

> Possible solution:

> Then this can be solved, in principle (if there's no better way), by
> watching a "virtual directory" that gets all events for when the
> access doesn't have a parent directory.  There needs to be some way to
> watch it, and some way to get the appropriate file from the event (as
> there is no real directory.  Or maybe there could be a virtual
> filesystem (like /proc, /sys etc.) containing a magic directory that
> receives these inode-only events, such that lookups in that directory
> yield the affected file.  Exactly as if the directory contains a hard
> link to every file, perhaps a text encoding of the handles passed
> through sys_open_by_handle_at.

There is a better way - stop using idiotify...  It has always been a
mistake, driven down our throats by filemangler and desktop crowd.
Broken in many, _many_ respects...  Deprecate that crap, remove it
completely in a couple of revisions, let these clowns cope.  Yeah,
yeah, I know...  Not going to happen ;-/  One can dream, though...

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Randy Dunlap]

On Sat, 20 Aug 2011 02:29:43 +0100 Al Viro wrote:

> On Sat, Aug 20, 2011 at 12:37:56AM +0100, Jamie Lokier wrote:
> 
> > Possible solution:
> 
> > Then this can be solved, in principle (if there's no better way), by
> > watching a "virtual directory" that gets all events for when the
> > access doesn't have a parent directory.  There needs to be some way to
> > watch it, and some way to get the appropriate file from the event (as
> > there is no real directory.  Or maybe there could be a virtual
> > filesystem (like /proc, /sys etc.) containing a magic directory that
> > receives these inode-only events, such that lookups in that directory
> > yield the affected file.  Exactly as if the directory contains a hard
> > link to every file, perhaps a text encoding of the handles passed
> > through sys_open_by_handle_at.
> 
> There is a better way - stop using idiotify...  It has always been a
> mistake, driven down our throats by filemangler and desktop crowd.
> Broken in many, _many_ respects...  Deprecate that crap, remove it
> completely in a couple of revisions, let these clowns cope.  Yeah,
> yeah, I know...  Not going to happen ;-/  One can dream, though...

What do you suggest as a functioning alternative to inotify?

---
~Randy

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Al Viro]

On Fri, Aug 19, 2011 at 06:43:11PM -0700, Randy Dunlap wrote:

> What do you suggest as a functioning alternative to inotify?

Depends on what particular (ab)use do you have in mind...

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Jamie Lokier]

Al Viro wrote:
> On Sat, Aug 20, 2011 at 12:37:56AM +0100, Jamie Lokier wrote:
> 
> > Possible solution:
> 
> > Then this can be solved, in principle (if there's no better way), by
> > watching a "virtual directory" that gets all events for when the
> > access doesn't have a parent directory.  There needs to be some way to
> > watch it, and some way to get the appropriate file from the event (as
> > there is no real directory.  Or maybe there could be a virtual
> > filesystem (like /proc, /sys etc.) containing a magic directory that
> > receives these inode-only events, such that lookups in that directory
> > yield the affected file.  Exactly as if the directory contains a hard
> > link to every file, perhaps a text encoding of the handles passed
> > through sys_open_by_handle_at.
> 
> There is a better way - stop using idiotify...  It has always been a
> mistake, driven down our throats by filemangler and desktop crowd.

Well you still have your sense of humour...

I've never understood why you think it's about the file manager /
desktop, or why you so strongly dislike the feature.  It originated
there historically, but that is not it's primary use.

The implementation, sure, but you seem to dislike the very *principle*
of subscribing to changes.

Every interesting use of inotify that I've seen is for some kind of
cache support, to eliminate the majority of stat() calls, to remove
disk I/O (no stat means no inode), to ensure correctness (st_mtime is
coarse and unreliable), and to avoid having to modify every
application which might affect any file from which cached items are
derived to explicitly notify all the applications which might use any
of those files.

You like high performance, reliable and correct behaviour, and high
scalability.  So I have never understood why you dislike the
change-subscription principle so strongly, because it is a natural
ally to those properties.

All the best,
-- Jamie

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Jamie Lokier]

Sylvain Rochet wrote:
> Hi Jamie,
> 
> 
> On Sat, Aug 20, 2011 at 12:37:56AM +0100, Jamie Lokier wrote:
> > 
> > Oh dear, that's a security hole, if something is using inotify/dnotify
> > to watch and assumes that file contents (on the same machine,
> > i.e. server in this case) do not change if there's no event received.
> > 
> > It also breaks cache applications which make the same assumption.
> > 
> > I do quite like the idea of using it to break past fanotify security
> > restrictions though ;-)
> 
> It also probably means that fanotify misses some events when a filesystem 
> is modified over NFS. If fanotify is used the way it is designed, i.e. 
> with an antiviruse software, this may be an interesting way to skip the 
> antiviruse check.
> 
> Here we go:
> 
> NFS server, run the fanotify example tool:
> 
> ~/fanotify-example# ./fanotify -m /data/
> 
> NFS client, open a fd then do some I/O:
> 
> # exec 1> test
> # ls -la
> # 
> 
> NFS server log:
> 
> /data/test: pid=1235 modify close(writable)
> 
> NFS server, cache clearing:
> 
> # echo 3 > /proc/sys/vm/drop_caches 
> 
> NFS client, more I/O:
> 
> # ls -la
> 
> NFS server log:
> 
> /data: pid=1234 modify close(writable)
> 
> We receive an event... which is obviously wrong. This is even worse than 
> no event at all, we receive an event about the wrong inode, the parent 
> inode of the modified file actually.

That sounds like a proper bug, maybe it can be fixed at least?

> > Is a solution to open inotify watches on every file individually?  If 
> > so that seems quite severe.
> 
> This is what I am going to do, at least temporarily, I only need to 
> watch about a million file (and slowly counting).
> 
> The startup time to watch an entire filesystem using inotify already 
> require a full filesystem walk, watching all files and directories 
> instead of directories only will not change much because most of the 
> time is spent waiting I/O operations. This may however require a lot 
> more memory both on kernel side and userland side.

Watching an entire filesystem entails reading all the directories, but
you don't have to fetch the inodes of files.  But still, it's very
slow (takes about 15 minutes on my /home from cold cache, just to read
the million or so directories).

There was some work on propagating events upwards so that efficient
recursive watches could be established, in the context of fanotify but
it would make sense to be available to all fsnotify users.  I wonder
how that went.

> > Then this can be solved, in principle (if there's no better way), by
> > watching a "virtual directory" that gets all events for when the
> > access doesn't have a parent directory.  There needs to be some way to
> > watch it, and some way to get the appropriate file from the event (as
> > there is no real directory.  Or maybe there could be a virtual
> > filesystem (like /proc, /sys etc.) containing a magic directory that
> > receives these inode-only events, such that lookups in that directory
> > yield the affected file.  Exactly as if the directory contains a hard
> > link to every file, perhaps a text encoding of the handles passed
> > through sys_open_by_handle_at.
> 
> By doing that, we'll only get the inode nb as we cannot fetch the filename.

Yes... That's ok if it's one we are tracking inode->multiple-paths in
userspace anyway (for hard links).  But it's quite demanding if we
hoped to avoid fetching and storing that in userspace for
st_nlink == 1 files.

In that case it is still better to get a notification "something
unknown on this FS has changed", rather than no notification.
Userspace would react by flushing all of its cached knowledge of
things under directory watches that don't have direct watches.  But at
least that's reliable and correct behaviour, and if it happens often,
userspace heuristics can react by watching priority inodes more directly.

If that's the common case, then these nameless, pathless events could
just trigger a simple event with catch-all IN_NO_PATH flag set,
referring to the filesytem but no more detail than that.  inotify
would accept that flag when adding a watch, ignore the inode given but
remember the filesystem, and send all events with no path to the
watch(es) created with that flag on that filesystem.  It's a flag
because the event type is still useful.

All the best,
-- Jamie

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[J. Bruce Fields]

On Sat, Aug 20, 2011 at 04:03:35AM +0100, Jamie Lokier wrote:
> Well you still have your sense of humour...
> 
> I've never understood why you think it's about the file manager /
> desktop, or why you so strongly dislike the feature.  It originated
> there historically, but that is not it's primary use.
> 
> The implementation, sure, but you seem to dislike the very *principle*
> of subscribing to changes.
> 
> Every interesting use of inotify that I've seen is for some kind of
> cache support, to eliminate the majority of stat() calls, to remove
> disk I/O (no stat means no inode), to ensure correctness (st_mtime is
> coarse and unreliable),

It seems rather fragile as an mtime replacement unless it's also got
some sort of logging built in at a pretty low level so that you don't
lose events while you're not listening.

And of course events have to be defined very carefully to avoid problems
such as this one.

> and to avoid having to modify every
> application which might affect any file from which cached items are
> derived to explicitly notify all the applications which might use any
> of those files.
> 
> You like high performance, reliable and correct behaviour, and high
> scalability.  So I have never understood why you dislike the
> change-subscription principle so strongly, because it is a natural
> ally to those properties.

I don't think we've seen a design that does all of that yet.

--b.

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Jamie Lokier]

J. Bruce Fields wrote:
> On Sat, Aug 20, 2011 at 04:03:35AM +0100, Jamie Lokier wrote:
> > Well you still have your sense of humour...
> > 
> > I've never understood why you think it's about the file manager /
> > desktop, or why you so strongly dislike the feature.  It originated
> > there historically, but that is not it's primary use.
> > 
> > The implementation, sure, but you seem to dislike the very *principle*
> > of subscribing to changes.
> > 
> > Every interesting use of inotify that I've seen is for some kind of
> > cache support, to eliminate the majority of stat() calls, to remove
> > disk I/O (no stat means no inode), to ensure correctness (st_mtime is
> > coarse and unreliable),
> 
> It seems rather fragile as an mtime replacement unless it's also got
> some sort of logging built in at a pretty low level so that you don't
> lose events while you're not listening.

It mainly serves as an accelerator for existing stat/mtime checks,
though it does improve change detection in the last second or so since
a previous change, which with mtime you have to make pessimistic or
sometimes-incorrect assumptions.

Quite a few programs use inotify now because it saves a little power,
and is a bit more responsive than, say, polling config files with stat().

For reliable filesystem tracking across times when not listening,
especially if you don't trust the clock to have no backward steps (and
you should not), a lazy change count file attribute would do.  It's
been discussed but never implemented.

> And of course events have to be defined very carefully to avoid problems
> such as this one.

This thread has revealed quite a big hole, I agree.  Apps cannot even
use their normal filesystem-type whitelisting to catch this.  This is bad!

It is not the first hole that was found in inotify/dnotify, but it's
the first one I'm aware of that wasn't pointed out long ago and
then quietly ignored :-/

> > and to avoid having to modify every
> > application which might affect any file from which cached items are
> > derived to explicitly notify all the applications which might use any
> > of those files.
> > 
> > You like high performance, reliable and correct behaviour, and high
> > scalability.  So I have never understood why you dislike the
> > change-subscription principle so strongly, because it is a natural
> > ally to those properties.
> 
> I don't think we've seen a design that does all of that yet.

Designs get discussed from time to time, over the decades.

I think one of the reasons it doesn't go further is Al's well-known
objection -- why put the effort in if you know it will be rejected.
And a widespread view that it's just unimportant GUI file manager fluff.
The latter also means dependability issues have tended not to be taken
seriously.

-- Jamie

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[NeilBrown]

On Sat, 20 Aug 2011 01:03:44 +0200 Sylvain Rochet <gradator@gradator.net>
wrote:

> Hi,
> 
> On Tue, Oct 19, 2010 at 12:35:40AM +0200, Sylvain Rochet wrote:
> > 
> > ... upgraded to 2.6.33.5, then 2.6.33.7, finally to 2.6.35.7, and I 
> > always end up with the same ending, it seems inotify can miss some VFS 
> > events from time to time.
> 
> I finally find out why.
> 
> The NFS server does not always know the name of the modified file, if 
> the modified inode was cleared from the VFS cache fsnotify does not know 
> as well the filename then inotify child events on directories are 
> silently tossed.

This may not be helpful but if you export the filesystem with the
"subtree_check" option then the NFS server will always know a name for any
file that it modifies.

This is not without cost though, which is why it is no longer the default.

If the client opens a file, the server moves it to a new directory and then
the server cache is flushed, then the client will get an ESTALE error.  This
is rarely a problem in practice but when it is it can be quite annoying...

NeilBrown



> 
> Easy way to reproduce:
> 
> Add a few printk debug (here it only works if /data is the NFS export):
> 
> --- begin//fs/nfsd/vfs.c        2011-07-22 04:17:23.000000000 +0200
> +++ linux-3.0/fs/nfsd/vfs.c     2011-07-30 03:18:17.837560809 +0200
> @@ -975,6 +975,8 @@
>         inode = dentry->d_inode;
>         exp   = fhp->fh_export;
>  
> +       printk("nfsd write inode=%ld name=%s\n", inode->i_ino, dentry->d_name.name);
> +
>         /*
>          * Request sync writes if
>          *  -   the sync export option has been set, or
> 
> diff -Nru begin//include/linux/fsnotify.h linux-3.0/include/linux/fsnotify.h
> --- begin//include/linux/fsnotify.h     2011-07-22 04:17:23.000000000 +0200
> +++ linux-3.0/include/linux/fsnotify.h  2011-07-30 03:07:00.330239062 +0200
> @@ -216,8 +232,15 @@
>                 mask |= FS_ISDIR;
>  
>         if (!(file->f_mode & FMODE_NONOTIFY)) {
> +               if( !strcmp(path->mnt->mnt_mountpoint->d_name.name, "data") )
> +                       printk("fsnotify modify inode=%ld name=%s\n", inode->i_ino, file->f_dentry->d_name.name);
>                 fsnotify_parent(path, NULL, mask);
>                 fsnotify(inode, mask, path, FSNOTIFY_EVENT_PATH, NULL, 0);
> +       } else {
> +               if( !strcmp(path->mnt->mnt_mountpoint->d_name.name, "data") )
> +                       printk("fsnotify modify-nonotify inode=%ld name=%s\n", inode->i_ino, file->f_dentry->d_name.name);
>         }
>  }
> 
> 
> On the NFS client, open a fd and send some data:
> 
> # exec 1> test
> # ls -la
> # 
> 
> On the NFS server, check the kern log:
> 
> Aug 20 00:57:44 inotifydebug kernel: nfsd write inode=13 name=test
> Aug 20 00:57:44 inotifydebug kernel: fsnotify modify inode=13 name=test
> 
> Everything goes well.
> 
> Now, clear the VFS cache on the NFS server:
> 
> # echo 3 > /proc/sys/vm/drop_caches 
> 
> On the NFS client, send some data to the fd:
> 
> # ls -la
> # 
> 
> On the NFS server, check the kern log:
> 
> Aug 20 00:58:56 inotifydebug kernel: nfsd write inode=13 name=
> Aug 20 00:58:56 inotifydebug kernel: fsnotify modify inode=13 name=
> 
> The filename is lost, fsnotify does not know the filename anymore, 
> therefore inotify cannot send event about a modified file in a watched 
> directory.
> 
> End of the story.
> 
> I guess this is almost impossible to fix this fsnotify bug, this is due 
> by the fact that NFS use inode as file identifiers, so in some case this 
> is impossible to know the modified filepath, and therefore impossible to 
> match the file event to the directory watch.
> 
> Kind regards,
> Sylvain

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[NeilBrown]

On Sun, 21 Aug 2011 21:20:58 +0100 Jamie Lokier <jamie@shareable.org> wrote:

> J. Bruce Fields wrote:
> > On Sat, Aug 20, 2011 at 04:03:35AM +0100, Jamie Lokier wrote:
> > > Well you still have your sense of humour...
> > > 
> > > I've never understood why you think it's about the file manager /
> > > desktop, or why you so strongly dislike the feature.  It originated
> > > there historically, but that is not it's primary use.
> > > 
> > > The implementation, sure, but you seem to dislike the very *principle*
> > > of subscribing to changes.
> > > 
> > > Every interesting use of inotify that I've seen is for some kind of
> > > cache support, to eliminate the majority of stat() calls, to remove
> > > disk I/O (no stat means no inode), to ensure correctness (st_mtime is
> > > coarse and unreliable),
> > 
> > It seems rather fragile as an mtime replacement unless it's also got
> > some sort of logging built in at a pretty low level so that you don't
> > lose events while you're not listening.
> 
> It mainly serves as an accelerator for existing stat/mtime checks,
> though it does improve change detection in the last second or so since
> a previous change, which with mtime you have to make pessimistic or
> sometimes-incorrect assumptions.
> 
> Quite a few programs use inotify now because it saves a little power,
> and is a bit more responsive than, say, polling config files with stat().
> 
> For reliable filesystem tracking across times when not listening,
> especially if you don't trust the clock to have no backward steps (and
> you should not), a lazy change count file attribute would do.  It's
> been discussed but never implemented.
> 
> > And of course events have to be defined very carefully to avoid problems
> > such as this one.
> 
> This thread has revealed quite a big hole, I agree.  Apps cannot even
> use their normal filesystem-type whitelisting to catch this.  This is bad!
> 
> It is not the first hole that was found in inotify/dnotify, but it's
> the first one I'm aware of that wasn't pointed out long ago and
> then quietly ignored :-/
> 
> > > and to avoid having to modify every
> > > application which might affect any file from which cached items are
> > > derived to explicitly notify all the applications which might use any
> > > of those files.
> > > 
> > > You like high performance, reliable and correct behaviour, and high
> > > scalability.  So I have never understood why you dislike the
> > > change-subscription principle so strongly, because it is a natural
> > > ally to those properties.
> > 
> > I don't think we've seen a design that does all of that yet.
> 
> Designs get discussed from time to time, over the decades.
> 
> I think one of the reasons it doesn't go further is Al's well-known
> objection -- why put the effort in if you know it will be rejected.
> And a widespread view that it's just unimportant GUI file manager fluff.
> The latter also means dependability issues have tended not to be taken
> seriously.

I know you weren't asking for design suggestions, but somehow I just couldn't
help myself :-)

The (or "a") problem with {d,i,fa}notify is that it makes a core assumption
that is flawed.  i.e. that a file is in some directory.  It might be nice if
that were a reliable fact but thanks to our founding fathers, it is not.
If a file only ever had one name - never more nor less - and could not have
that name changed while it were open, then quite a lot of things would be a
lot easier.  And probably a lot of things would be a lot harder.  But we don't
live in that world (others do - I think you know where it is).

So we must drop this assumption.

Getting notification on an fd when the opened file changes makes perfect
sense.  Some /proc and /sys files already provide this functionality and we
can expect that more will.  Adding that to regular filesystems may not be out
of the question.  This would be useful, but of limited use.  You could find
out when a given file changed - either an mtime-like change or a ctime-like
change.  By monitoring a directory you could find out when a name was added
or removed.  But to find out when "any file in a directory changes" you would
need to open and monitor every file, which is expensive.

The other ("another") problem is the lack of recursion.  You can find out
when a file in a directory changes, but not a file in a directory tree.  This
significantly reduces the value.   We really want to know about directory
trees.  However a "directory tree" - much like "all the files in a directory"
isn't really a very well defined concept - at least from the perspective of
providing notifications.  You cannot easily answer "is this file in that
tree?" or "which tree(s) is this file in?".

However there are well defined sets of files such that we could reliably
generate notifications if any file in the set were changed, or if a file were
added-to or remove-from the set.  We should be looking for these sorts of
sets and seeing which are useful.

e.g.
 - all files in a given filesystem.   Generating notification for any change
   in a given filesystem is a well defined task.  It might generate too much
   noise, but it would still have a place.

 - all files with a given uid (or gid).

 - all directories.  or all regular files

 - all setuid, setgid, or world-writable files

Each of these are strongly defined and we can map from file to set quite
easily.  We could obviously intersect the sets to, so I could get events when
any directory owned by me on a particular filesystem was changed.  It would
even be reasonable for the events to contain a newly opened fd from which I
can extract dev/inode info and possibly extract a path name.

However this still might not be fine-grained enough.  While a "directory
tree" is not really a well defined concept, it is in my mind.  e.g. it
seems reasonable to want to find out about all changes in $HOME/.config

I can see two approaches to this - though there might be others.  All of
them must in some way create a strong concept of a directory tree.

One is to use bind mounts. i.e. I effectively do
    mount --bind $HOME/.config $HOME/.config
and ask for events from the newly created vfsmnt.
This will not catch changes made through file descriptors that were opened
before I did the mount, or through hard links from some other directory
tree.  But for a particular use-case that might not be a problem.

The other requires support from the filesystem and so cannot be provided
universally.  It could possibly be imposed generically for filesystems that
support extended attributes .... but I feel dirty even suggesting that (Dobby
must now go and iron his hands!)

The filesystem could support the concept of a 'directory tree' much like
BTRFS allows subvolumes which a like independent filesystems within the one
big filesystem.  However for this purpose the 'directory tree' would be a
very light weight concept (it wouldn't need its own inode number space).

For example, each inode could store an extra number which is the inode number
of the root of its directory tree.  This would be inherited from parent
during create. Renaming or linking a file would fail if the target had a
different directory tree number.  (renaming a file with only one link might
succeed and change the directory tree number).   An empty directory could be
told to become a root somehow.

Then you would have a strong concept of a directory tree that could be used
for notifications.
Obviously this approach could not be used to solve any immediate problems.
But if new filesystems started supporting light-weight-directory-trees as a
well defined set of files, then in 5-10 years we might have a nice working
solution.


[of course then you need to layer any design you come up with on NFS ... but
that can probably be done in user-space with libraries and daemons].

NeilBrown

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[J. Bruce Fields]

On Mon, Aug 22, 2011 at 09:07:51AM +1000, NeilBrown wrote:
> One is to use bind mounts. i.e. I effectively do
>     mount --bind $HOME/.config $HOME/.config
> and ask for events from the newly created vfsmnt.
> This will not catch changes made through file descriptors that were opened
> before I did the mount, or through hard links from some other directory
> tree.  But for a particular use-case that might not be a problem.

I'm missing what the extra vfsmount gets you here.  The problems seem
just the same as if you don't have one.

Oh, wait, I see, it's that the file descriptors are associated with
vfsmounts, not just dentries.  Hm.

--b.

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[NeilBrown]

On Mon, 22 Aug 2011 13:22:39 -0400 "J. Bruce Fields" <bfields@fieldses.org>
wrote:

> On Mon, Aug 22, 2011 at 09:07:51AM +1000, NeilBrown wrote:
> > One is to use bind mounts. i.e. I effectively do
> >     mount --bind $HOME/.config $HOME/.config
> > and ask for events from the newly created vfsmnt.
> > This will not catch changes made through file descriptors that were opened
> > before I did the mount, or through hard links from some other directory
> > tree.  But for a particular use-case that might not be a problem.
> 
> I'm missing what the extra vfsmount gets you here.  The problems seem
> just the same as if you don't have one.
> 
> Oh, wait, I see, it's that the file descriptors are associated with
> vfsmounts, not just dentries.  Hm.

"Hm" might be right.  writes have a 'struct file', but mkdir and rename etc
just get an inode.
So to be able to notify based on vfsmnt, mkdirat - for examine - would need
to pass the path to vfs_mkdir rather than path.dentry->d_inode, and vfs_mkdir
would have to pass that path to fsnotify_mkdir.  So more intrusive that I
imagined, but still quite do-able.... if it were thought to be useful.

NeilBrown

Re: PROBLEM: 2.6.35.7 to 3.0 Inotify events missing

[Sylvain Rochet]

[-- Attachment #1.1: Type: text/plain, Size: 371 bytes --]

Hi,

On Sat, Aug 20, 2011 at 12:37:56AM +0100, Jamie Lokier wrote:
> 
> Can it also be bypassed with sys_open_by_handle_at?

Just checked that and... as planned it can be bypassed with 
name_to_handle_at/open_by_handle_at as well.

(I added my quick'n'dirty test code, maybe someone will be interested by 
a working fhandle example on Linux 3.0.x).

Sylvain

[-- Attachment #1.2: fhandle3.c --]
[-- Type: text/x-csrc, Size: 892 bytes --]

#include <stdio.h>
#include <stdlib.h>

#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

struct file_handle {
	int handle_size;
	int handle_type;
	void *f_handle;
};


int main(int argc, char *argv[])
{
	int ret;
	int fd;
	int mntfd;
	struct file_handle fh;
	fh.handle_type = 0;
	fh.f_handle = malloc(100);
	fh.handle_size = 100/sizeof(int);
	errno  = 0;

	printf("%s\n", argv[1]);

	int mntid = 20;

	ret = syscall(303, 0, argv[1], &fh, &mntid, 0);
	if (ret) {
		perror("303 Error:");
		exit(1);
	}
	printf("%d - %d\n", mntid, fh.handle_size);

	printf("sleep\n");
	sleep(60);


	mntfd = open("/data", O_RDONLY);
	printf("dir = %d\n", mntfd );

	fd = syscall(304, mntfd, &fh, O_WRONLY);
	if (fd < 0 ) {
		perror("304 Error:");
		exit(1);
	}
	printf("fd = %d\n", fd);


	write(fd, "toto", 4);


	if( close(fd) ) {
		printf("close failed\n");
	}

	return 0;
}




[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]