Tracing file size changes

Tracing file size changes

[Chris]

I'm trying to think of a good way to figure out where space is going on
somewhat large and busy filesystems.  Often I will be notified by nagios that I
am running out of space somewhere, and it will even tell me what the trend in
usage is.  So now I know that I lost a bunch of space recently.

There are several things I can do at that point:
- Check quota usage.  If quotas are working correctly, this can sometimes
help when one user is obviously using a ton of space.  More often than not,
there is nothing obvious, quotas aren't working or the space is used by system
accounts that provide no help in knowing where the space went.
- Use du.  This is just plain painful and time consuming.  Doing this usually
causes too much load.

Without knowing other ways to find information, I was thinking it would be
really nice to have a way to see which files or directories grew or shrank
since some point in the past.  Would it be possible to get the kernel to tell
me when files were opened for writing and closed so I could record the size
difference?  I know there are various file notify mechanisms, but from what I
can see, they don't handle millions of files very well.

Of course I'm open to other ideas, but this was my curiosity.

Tracing file size changes

[Adam Lee]

On Wed, Aug 10, 2011 at 07:06:29PM -0400, Chris wrote:
> I'm trying to think of a good way to figure out where space is going on
> somewhat large and busy filesystems.  Often I will be notified by nagios that I
> am running out of space somewhere, and it will even tell me what the trend in
> usage is.  So now I know that I lost a bunch of space recently.
> 
> There are several things I can do at that point:
> - Check quota usage.  If quotas are working correctly, this can sometimes
> help when one user is obviously using a ton of space.  More often than not,
> there is nothing obvious, quotas aren't working or the space is used by system
> accounts that provide no help in knowing where the space went.
> - Use du.  This is just plain painful and time consuming.  Doing this usually
> causes too much load.
> 
> Without knowing other ways to find information, I was thinking it would be
> really nice to have a way to see which files or directories grew or shrank
> since some point in the past.  Would it be possible to get the kernel to tell
> me when files were opened for writing and closed so I could record the size
> difference?  I know there are various file notify mechanisms, but from what I
> can see, they don't handle millions of files very well.
> 
> Of course I'm open to other ideas, but this was my curiosity.
> 

Will Inotify help?

http://en.wikipedia.org/wiki/Inotify
http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html

-- 
Regards,
Adam Lee
--------------------------------------------------
E-mail: adam8157 at gmail.com
Website: http://www.adam8157.info
--------------------------------------------------

Tracing file size changes

[Vaughn Clinton]

I like "inotify".  I've become somewhat familiar with the interface and I 
believe this will allow you to do what you want.

There are plenty "C" examples out there.

Cheers,

-----Original Message----- 
From: Adam Lee
Sent: Thursday, August 11, 2011 12:17 PM
To: kernelnewbies at kernelnewbies.org
Subject: Re: Tracing file size changes

On Wed, Aug 10, 2011 at 07:06:29PM -0400, Chris wrote:
> I'm trying to think of a good way to figure out where space is going on
> somewhat large and busy filesystems.  Often I will be notified by nagios 
> that I
> am running out of space somewhere, and it will even tell me what the trend 
> in
> usage is.  So now I know that I lost a bunch of space recently.
>
> There are several things I can do at that point:
> - Check quota usage.  If quotas are working correctly, this can sometimes
> help when one user is obviously using a ton of space.  More often than 
> not,
> there is nothing obvious, quotas aren't working or the space is used by 
> system
> accounts that provide no help in knowing where the space went.
> - Use du.  This is just plain painful and time consuming.  Doing this 
> usually
> causes too much load.
>
> Without knowing other ways to find information, I was thinking it would be
> really nice to have a way to see which files or directories grew or shrank
> since some point in the past.  Would it be possible to get the kernel to 
> tell
> me when files were opened for writing and closed so I could record the 
> size
> difference?  I know there are various file notify mechanisms, but from 
> what I
> can see, they don't handle millions of files very well.
>
> Of course I'm open to other ideas, but this was my curiosity.
>

Will Inotify help?

http://en.wikipedia.org/wiki/Inotify
http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html

-- 
Regards,
Adam Lee
--------------------------------------------------
E-mail: adam8157 at gmail.com
Website: http://www.adam8157.info
--------------------------------------------------

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies 

Tracing file size changes

[Chris]

On Thu, Aug 11, 2011 at 10:47:00AM +0800, Adam Lee wrote:
> > 
> > Without knowing other ways to find information, I was thinking it would be
> > really nice to have a way to see which files or directories grew or shrank
> > since some point in the past.  Would it be possible to get the kernel to tell
> > me when files were opened for writing and closed so I could record the size
> > difference? > 
> Will Inotify help?
> 
> http://en.wikipedia.org/wiki/Inotify
> http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html
> 

It is pretty much what I want to do, the problem with it is that it doesn't
appear to be recursive.  I'd like to just put a watch on root, and get events
for the entire filesystem, not just that directory.  From playing with this for
a couple minutes, it looks like I'd have to walk the entire filesystem setting
watches on every directory.  So while it is very cool for what it does, I don't
think it will help me out much.

Is doing something with ld_preload to get in the way of every open call the
only way to do this?

Tracing file size changes

[Chris]

> > Will Inotify help?
> > 
> > http://en.wikipedia.org/wiki/Inotify
> > http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html
> > 
> 
> It is pretty much what I want to do, the problem with it is that it doesn't
> appear to be recursive.  


I looked again tonight, and it appears that newer kernels with fanotify are
able to do this.  So the feature is available, just too new to be on my
servers.  Oh, well :P

Thanks for the input.  I will probably just end up doing what I can with
ld_preload.

Chris

Tracing file size changes

[Christophe Hauser]

On Fri, Aug 12, 2011 at 08:58:41PM -0400, Chris wrote:
> > > Will Inotify help?
> > > 
> > > http://en.wikipedia.org/wiki/Inotify
> > > http://www.kernel.org/doc/man-pages/online/pages/man7/inotify.7.html
> > > 
> > 
> > It is pretty much what I want to do, the problem with it is that it doesn't
> > appear to be recursive.  
> 
> 
> I looked again tonight, and it appears that newer kernels with fanotify are
> able to do this.  So the feature is available, just too new to be on my
> servers.  Oh, well :P
> 
> Thanks for the input.  I will probably just end up doing what I can with
> ld_preload.
> 
> Chris

Hi Chris,

I'm not sure whether that would help or not, but you could also do it in
kernel space using LSM (if you don't already use another LSM module).

With the file_security hook, you can track any changes on files and
store information in the extended attributes of the filesystem for
example. This might be a bit more flexible than a system wide
ld_preload.

My two cents.

-- 
Christophe