Re: Filesystem access statistics

[Steve Grubb <sgrubb@redhat.com>]

On Wednesday 12 April 2006 11:23, Rudi Chiarito wrote:
> I need to look at a portion of the filesystem namespace and maintain
> aggregate statistics on access patterns. In other words, I have a large
> filesystem and would like to find out which are the hot spots. I don't
> need to keep track of every single file access: since the file count is
> in the order of millions, that would swamp the actual I/O, the
> analysis and the people looking at the final data. It would make sense
> to just group accesses by looking at the top N levels (anything
> accessed at levels N+1, N+2, etc. would be coalesced into the parent
> directory at level N).

I would think that you could write a program to do this via the audit 
dispatcher interface. In auditd.conf, 

dispatcher = /usr/bin/your-program
log_format = nolog

This will tell the audit daemon you don't want the records written to disk and 
that it should pass events to the dispatcher. You can get example code in 
skeleton.c. (rpm -ql audit | grep skel ) The skeleton.c program around line 
123 is where you would add your code to examine the PATH records. You would 
likely want to do

if (hdr.type == AUDIT_PATH) {
	process the path 
}

so that you only look at the right kind of records. If you have too many 
records coming out of the audit system, you might be able to suppress some in 
the kernel by adding rules like this:

-a always,exclude -F 'msgtype<PATH' -F 'msgtype>PATH'

You can then set the audit rules for whatever you want to measure, if all you 
want to measure is the opens,

-a always,exit -S open

You can use devmajor and devminor fields to limit the audit system to 
reporting opens on an exact partition. This is highly recommended. On my 
system, I would do something like:

-a always,exit -S open -F devmajor=3 -F devminor=6

to watch the /tmp directory.

> I think that I can't be the only one with such a need.

Bootchart.org used the audit system to examine the boot order

http://bootchart.org/misc/filemon/filemond

> In my case, the information is going to be used to change the way the tree
> is going to be laid out in the future, as well as determining when parts of
> it can be made read-only (after an inactivity period). I can also see the
> information being useful for selective incremental backups - just look
> at the hot spots - or for smarter ordering during a disaster recovery
> restore (if you're recovering from random access storage, not tape).
> Maybe even locate/slocate/rlocate/mlocate could take advantage of it.

Sounds interesting.

> What would be the best approach to this? 

I think we've laid out an approach above.

1) set custom rules to watch just the syscalls you want on the exact 
partitions you want.
2) put the analysis in a program hanging off of audit event dispatcher and 
turn off audit logging to disk. 

You need audit-1.1 or higher. audit-1.1 has some ABI changes that make it 
incompatible with audit-1.0 systems. Let me know if that is a problem. Also, 
you will need 2.6.17 kernel if you use the '<' or '>' rule operators.

-Steve