inotify limits - thousands (tens of thousands?) of watches

inotify limits - thousands (tens of thousands?) of watches

[Marcin Krol]

Hello everyone,

First, apols for using up bandwidth, but I honestly found no other place 
where I can ask about this (and get meaningful reply).

I'm not a kernel programmer, but I want to develop a program that would 
watch modifications in *all* user directories on a busy server using 
inotify.

This is for high-availability purposes - events would be collected and 
once every several minutes changed dirs would be rsync'ed to failover 
server or smth like that would be done.

As inotify watches particular directory and not its subdirs, I would 
have to watch all directories.

This means I would have to create thousands or even tens of thousands of 
inotify watches.

So my questions are:

1. is it safe? that is, will it not lock the kernel up, or cause 
excessive memory consumption?

2. is it economic in terms of CPU time and RAM? I have no idea how to 
even measure such a thing happening in the kernel..




Here's first test take on inotify watch (runs on Debian):

######################### cut ################################

#include "stdio.h"
#include "stdlib.h"
#include "errno.h"
#include <linux/types.h>
#include <linux/inotify.h>
#include <sys/syscall.h>

#define EVENT_SIZE  ( sizeof (struct inotify_event) )
#define BUF_LEN     ( 1024 * ( EVENT_SIZE + 16 ) )


static inline int inotify_init (void)
{
         return syscall (__NR_inotify_init);
}

static inline int inotify_add_watch (int fd, const char *name, __u32 mask)
{
         return syscall (__NR_inotify_add_watch, fd, name, mask);
}

static inline int inotify_rm_watch (int fd, __u32 wd)
{
         return syscall (__NR_inotify_rm_watch, fd, wd);
}


int main( int argc, char **argv )
{
   int length, i = 0;
   int fd;
   int wd;
   char buffer[BUF_LEN];

   fd = inotify_init();

   if ( fd < 0 ) {
     perror( "inotify_init" );
   }

   wd = inotify_add_watch( fd, argv[1], IN_ALL_EVENTS );
   length = read( fd, buffer, BUF_LEN );

   if ( length < 0 ) {
     perror( "read" );
   }

   while ( i < length ) {
     struct inotify_event *event = ( struct inotify_event * ) &buffer[ i ];
     if ( event->len ) {
       if ( event->mask & IN_CREATE ) {
         if ( event->mask & IN_ISDIR ) {
           printf( "The directory %s was created.\n", event->name );
         }
         else {
           printf( "The file %s was created.\n", event->name );
         }
       }
       else if ( event->mask & IN_DELETE ) {
         if ( event->mask & IN_ISDIR ) {
           printf( "The directory %s was deleted.\n", event->name );
         }
         else {
           printf( "The file %s was deleted.\n", event->name );
         }
       }
       else if ( event->mask & IN_MODIFY ) {
         if ( event->mask & IN_ISDIR ) {
           printf( "The directory %s was modified.\n", event->name );
         }
         else {
           printf( "The file %s was modified.\n", event->name );
         }
       }
       else if ( event->mask & IN_ALL_EVENTS ) {
           printf("Some other event on %s, mask %x.\n", event->name, 
event->mask);
       }
     }
     i += EVENT_SIZE + event->len;
   }

   ( void ) inotify_rm_watch( fd, wd );
   ( void ) close( fd );

   exit( 0 );
}

######################### cut ################################

Regards,
mk

Re: inotify limits - thousands (tens of thousands?) of watches

[Matthias Kaehlcke]

El Wed, May 20, 2009 at 11:26:57AM +0200 Marcin Krol ha dit:

> First, apols for using up bandwidth, but I honestly found no other place  
> where I can ask about this (and get meaningful reply).
>
> I'm not a kernel programmer, but I want to develop a program that would  
> watch modifications in *all* user directories on a busy server using  
> inotify.
>
> This is for high-availability purposes - events would be collected and  
> once every several minutes changed dirs would be rsync'ed to failover  
> server or smth like that would be done.
>
> As inotify watches particular directory and not its subdirs, I would  
> have to watch all directories.
>
> This means I would have to create thousands or even tens of thousands of  
> inotify watches.
>
> So my questions are:
>
> 1. is it safe? that is, will it not lock the kernel up, or cause  
> excessive memory consumption?
>
> 2. is it economic in terms of CPU time and RAM? I have no idea how to  
> even measure such a thing happening in the kernel..

i can't answer your question whether inotify scales or not, but maybe
DRBD (http://en.wikipedia.org/wiki/DRBD) could be an alternative
approach for your problem

-- 
Matthias Kaehlcke
Embedded Linux Engineer
Barcelona

    Me lo contaron y lo olvidé, lo vi y lo entendí, lo hice y lo aprendí
                              (Confucio)
                                                                 .''`.
    using free software / Debian GNU/Linux | http://debian.org  : :'  :
                                                                `. `'`
gpg --keyserver pgp.mit.edu --recv-keys 47D8E5D4                  `-

Re: inotify limits - thousands (tens of thousands?) of watches

[Tvrtko Ursulin]

On Wednesday 20 May 2009 10:26:57 Marcin Krol wrote:
> Hello everyone,
>
> First, apols for using up bandwidth, but I honestly found no other place
> where I can ask about this (and get meaningful reply).
>
> I'm not a kernel programmer, but I want to develop a program that would
> watch modifications in *all* user directories on a busy server using
> inotify.
>
> This is for high-availability purposes - events would be collected and
> once every several minutes changed dirs would be rsync'ed to failover
> server or smth like that would be done.
>
> As inotify watches particular directory and not its subdirs, I would
> have to watch all directories.
>
> This means I would have to create thousands or even tens of thousands of
> inotify watches.
>
> So my questions are:
>
> 1. is it safe? that is, will it not lock the kernel up, or cause
> excessive memory consumption?
>
> 2. is it economic in terms of CPU time and RAM? I have no idea how to
> even measure such a thing happening in the kernel..

Hi,

I don't know the specific answers apart my intuitive feeling, which is that it 
probably would cause excessive memory use and that it is not elegant at all.

What may be interesting to you is the notification "engine" rewrite Eric Paris 
is doing and new fanotify interface which will, hopefully, come soon on top 
of it. With fanotify you will be able to monitor all changes much more easily 
and efficiently.

Have a look at patch descriptions in:
http://people.redhat.com/~eparis/fsnotify/

Tvrtko

Re: inotify limits - thousands (tens of thousands?) of watches

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 2115 bytes --]

Am Mittwoch, 20. Mai 2009 schrieb Marcin Krol:
> Hello everyone,
>
> First, apols for using up bandwidth, but I honestly found no other place
> where I can ask about this (and get meaningful reply).
>
> I'm not a kernel programmer, but I want to develop a program that would
> watch modifications in *all* user directories on a busy server using
> inotify.
>
> This is for high-availability purposes - events would be collected and
> once every several minutes changed dirs would be rsync'ed to failover
> server or smth like that would be done.

Hmmm, I think you could just run a rsync periodically. It might even be faster 
detecting changed files. Unless you are talking really high number of 
directories and files.

> As inotify watches particular directory and not its subdirs, I would
> have to watch all directories.

Yes. Thats cumbersome.

> This means I would have to create thousands or even tens of thousands of
> inotify watches.

I wrote a ruby script using libinotify-ruby which does just that. I only syncs 
on demand tough. I.e. when someplace places a special sync file in a watched 
directory.

> So my questions are:
>
> 1. is it safe? that is, will it not lock the kernel up, or cause
> excessive memory consumption?
>
> 2. is it economic in terms of CPU time and RAM? I have no idea how to
> even measure such a thing happening in the kernel..

That script is running productively for well over a year now.

There have been some problems with it stopping doing work occasionally. It has 
to be restarted. Now there is a monitor process which restarts it 
automatically should it end. Might be a race condition or programming error 
in the script. But the customer is happy with the workaround and didn't want 
me to put any further efforts in finding the real cause.

Regarding the inotify implementation in the kernel I darkly remember that 
there were some improvements. Search inotify or fsnotify in kernel-ml.

Ciao,
-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: inotify limits - thousands (tens of thousands?) of watches

[Marcin Krol]

Martin Steigerwald wrote:
> Hmmm, I think you could just run a rsync periodically. It might even be faster 
> detecting changed files. 

I beg to differ on this: rsync does quite intensive (in terms of disk 
activity and CPU activity) comparisons at the beginning of 
synchronization. It's pretty light later, true, but running rsync every 
few minutes on entire /home is IMO out of question.

> I wrote a ruby script using libinotify-ruby which does just that. I only syncs 
> on demand tough. I.e. when someplace places a special sync file in a watched 
> directory.

> That script is running productively for well over a year now.

Good to know the idea is not totally off the wall.. Thanks.

Anyway, I'll try using fsnotify / fanotify.

My main gripe with it, though, is that it is not in the mainline kernel, 
and thus in all probability it is not tested as widely as inotify.

Are there any chances for its inclusion in the near future?

Regards,
mk

Re: inotify limits - thousands (tens of thousands?) of watches

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 2013 bytes --]

Am Mittwoch, 20. Mai 2009 schrieb Marcin Krol:
> Martin Steigerwald wrote:
> > Hmmm, I think you could just run a rsync periodically. It might even be
> > faster detecting changed files.
>
> I beg to differ on this: rsync does quite intensive (in terms of disk
> activity and CPU activity) comparisons at the beginning of
> synchronization. It's pretty light later, true, but running rsync every
> few minutes on entire /home is IMO out of question.

It depends on the amount of directories being watched and the amount directory 
entries in there. Rsync version 3 starts of synchronizing before building up 
the entire trees on both sides. It uses an incremental approach and this 
appears to make quite a difference.

For syncing /home I wonder whether inotify or fsnotify/fanotify is the right 
approach. A cluster filesystem of some sort comes to my mind instead. But 
question is, which one really works good enough. Didn't do any testing in 
that area yet.

> > I wrote a ruby script using libinotify-ruby which does just that. I only
> > syncs on demand tough. I.e. when someplace places a special sync file in
> > a watched directory.
> >
> > That script is running productively for well over a year now.
>
> Good to know the idea is not totally off the wall.. Thanks.

Main difference is that the sync is only triggered on demand. But the watching 
part appears to work quite nicely. Its not used for /home, but for the static 
html/php/image content for an apache2.2 which handles a lot of domains.

> Anyway, I'll try using fsnotify / fanotify.
>
> My main gripe with it, though, is that it is not in the mainline kernel,
> and thus in all probability it is not tested as widely as inotify.

Well then it will get more testing now I think ;-).

> Are there any chances for its inclusion in the near future?

Indeed would be nice to have.

-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: inotify limits - thousands (tens of thousands?) of watches

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 1646 bytes --]

Am Mittwoch, 20. Mai 2009 schrieb Marcin Krol:
> Martin Steigerwald wrote:
> > Hmmm, I think you could just run a rsync periodically. It might even be
> > faster detecting changed files.
>
> I beg to differ on this: rsync does quite intensive (in terms of disk
> activity and CPU activity) comparisons at the beginning of
> synchronization. It's pretty light later, true, but running rsync every
> few minutes on entire /home is IMO out of question.

Another idea that might be applicable: 

We have a clustered setup - exactly also where that inotify ruby script runs - 
that used LVM and SoftRAID 1 for providing mirroring between both locations.

In each locations there is a RAID array with some hardware RAID, i.e. 
redundant in itself. Each RAID array is connected via FC to both cluster 
servers. Then we layer a SoftRAID 1 on top of it. Both cluster servers see 
the SoftRAID 1 device. One usually only does NFS and one usually only MySQL. 
Thus we made two volume groups. One of them is used by the NFS server only 
and the other one by the MySQL server.

In failover case the remaining server stoniths the failed server and takes 
over the volume group of it.

This way one of the servers could fail and the remaining server will be able 
to access the most recent data. And one of the externel RAID arrays could 
fail as well.

This worked remarkably well for more than a year, too. It won't work when you 
need to access the same volumes on both servers simultaneously, obviously.

-- 
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: inotify limits - thousands (tens of thousands?) of watches

[hooanon05]

Marcin Krol:
> I'm not a kernel programmer, but I want to develop a program that would 
> watch modifications in *all* user directories on a busy server using 
> inotify.
	:::
> 1. is it safe? that is, will it not lock the kernel up, or cause 
> excessive memory consumption?
> 
> 2. is it economic in terms of CPU time and RAM? I have no idea how to 
> even measure such a thing happening in the kernel..

The maximum number of inotify instances per user is limited to
/proc/sys/fs/inotify/max_user_instances which is 128 by default.
And also the number of watches per user is limited by
/proc/sys/fs/inotify/max_user_watches (8192 by defaut).
Theoretically you may be able to monitor 8192 directories, but it
consumes memory.

If periodical rsync is out of question for you, how about kprobe?
While I don't think it is a beautiful solution, to set a hook to
vfs_mkdir, vfs_unlink, etc and to compare the target super_block may
work for you.
See Documentation/kprobe.txt in detail.


J. R. Okajima