From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steve Grubb Subject: Re: Filesystem access statistics Date: Wed, 12 Apr 2006 12:26:29 -0400 Message-ID: <200604121226.29081.sgrubb@redhat.com> References: <20060412152325.GA1399@plain.rackshack.net> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20060412152325.GA1399@plain.rackshack.net> Content-Disposition: inline List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-audit-bounces@redhat.com Errors-To: linux-audit-bounces@redhat.com To: linux-audit@redhat.com List-Id: linux-audit@redhat.com On Wednesday 12 April 2006 11:23, Rudi Chiarito wrote: > I need to look at a portion of the filesystem namespace and maintain > aggregate statistics on access patterns. In other words, I have a large > filesystem and would like to find out which are the hot spots. I don't > need to keep track of every single file access: since the file count is > in the order of millions, that would swamp the actual I/O, the > analysis and the people looking at the final data. It would make sense > to just group accesses by looking at the top N levels (anything > accessed at levels N+1, N+2, etc. would be coalesced into the parent > directory at level N). I would think that you could write a program to do this via the audit dispatcher interface. In auditd.conf, dispatcher = /usr/bin/your-program log_format = nolog This will tell the audit daemon you don't want the records written to disk and that it should pass events to the dispatcher. You can get example code in skeleton.c. (rpm -ql audit | grep skel ) The skeleton.c program around line 123 is where you would add your code to examine the PATH records. You would likely want to do if (hdr.type == AUDIT_PATH) { process the path } so that you only look at the right kind of records. If you have too many records coming out of the audit system, you might be able to suppress some in the kernel by adding rules like this: -a always,exclude -F 'msgtypePATH' You can then set the audit rules for whatever you want to measure, if all you want to measure is the opens, -a always,exit -S open You can use devmajor and devminor fields to limit the audit system to reporting opens on an exact partition. This is highly recommended. On my system, I would do something like: -a always,exit -S open -F devmajor=3 -F devminor=6 to watch the /tmp directory. > I think that I can't be the only one with such a need. Bootchart.org used the audit system to examine the boot order http://bootchart.org/misc/filemon/filemond > In my case, the information is going to be used to change the way the tree > is going to be laid out in the future, as well as determining when parts of > it can be made read-only (after an inactivity period). I can also see the > information being useful for selective incremental backups - just look > at the hot spots - or for smarter ordering during a disaster recovery > restore (if you're recovering from random access storage, not tape). > Maybe even locate/slocate/rlocate/mlocate could take advantage of it. Sounds interesting. > What would be the best approach to this? I think we've laid out an approach above. 1) set custom rules to watch just the syscalls you want on the exact partitions you want. 2) put the analysis in a program hanging off of audit event dispatcher and turn off audit logging to disk. You need audit-1.1 or higher. audit-1.1 has some ABI changes that make it incompatible with audit-1.0 systems. Let me know if that is a problem. Also, you will need 2.6.17 kernel if you use the '<' or '>' rule operators. -Steve