unknown io writes in 2.6.16

unknown io writes in 2.6.16

[Mark Hedges]

Apologies if this is out of place, but I can't find anyone else 
who might know about this.

I'm using 2.6.16 (built from Debian unstable linux-source-2.6.16).
It's a pretty generic system with ext3 partitions.

I stop every non-kernel process except syslogd, klogd and the 
tty's.  Interfaces are down.  It is still in default runlevel.  
But the disk keeps clicking away.

`iostat` reports about 12 bytes written out to my root partition 
every 5 seconds or so.  But `top` and/or `sar -x ALL 1 10` 
report no noticible process activity beyond what they need to 
run, and there is no file on the partition that has its mtime 
modified.

selinux is not compiled in the kernel.

Any idea what's doing these writes?

Is there any way to view disk i/o by process ID?

Mark

Re: unknown io writes in 2.6.16

[David Rees]

On 5/10/06, Mark Hedges <hedges@ucsd.edu> wrote:
> I stop every non-kernel process except syslogd, klogd and the
> tty's.  Interfaces are down.  It is still in default runlevel.
> But the disk keeps clicking away.
>
> Any idea what's doing these writes?

It's most likely atime updates. Mount the partitions with noatime
option and your writes will go away.

-Dave

Re: unknown io writes in 2.6.16

[Mark Hedges]

On Wed, 10 May 2006, David Rees wrote:
> 
> On 5/10/06, Mark Hedges <hedges@ucsd.edu> wrote:
> > I stop every non-kernel process except syslogd, klogd and the
> > tty's.  Interfaces are down.  It is still in default runlevel.
> > But the disk keeps clicking away.
> > 
> > Any idea what's doing these writes?
> 
> It's most likely atime updates. Mount the partitions with noatime
> option and your writes will go away.

Yes, that's it.  Never noticed this much with IDE - this is my 
first SATA system.

Just a wishlist that process IO could be monitored.  I hate to 
say it but ctl-alt-esc in W2K can monitor io by process, and 
that's really useful.  (I will never go back though.)

Or is this something that could be reported with the system call 
auditing infrastructure?

Mark

Re: unknown io writes in 2.6.16

[Giacomo A. Catenazzi]

Mark Hedges wrote:
> Just a wishlist that process IO could be monitored.  I hate to 
> say it but ctl-alt-esc in W2K can monitor io by process, and 
> that's really useful.  (I will never go back though.)

vmstat ?

ciao
	cate

Re: unknown io writes in 2.6.16

[Avi Kivity]

Giacomo A. Catenazzi wrote:
> Mark Hedges wrote:
>> Just a wishlist that process IO could be monitored.  I hate to say it 
>> but ctl-alt-esc in W2K can monitor io by process, and that's really 
>> useful.  (I will never go back though.)
>
> vmstat ?

He's looking for per process stats.

-- 
error compiling committee.c: too many arguments to function

Re: unknown io writes in 2.6.16

[Erik Mouw]

On Wed, May 10, 2006 at 10:54:43PM -0700, Mark Hedges wrote:
> Just a wishlist that process IO could be monitored.  I hate to 
> say it but ctl-alt-esc in W2K can monitor io by process, and 
> that's really useful.  (I will never go back though.)

IIRC this can be done with Jens Axboe's blktrace. It got merged in
2.6.17, enable CONFIG_BLK_DEV_IO_TRACE and get the blktrace userspace
tools from git://brick.kernel.dk/data/git/blktrace.git .


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: unknown io writes in 2.6.16

[dean gaudet]

On Wed, 10 May 2006, Mark Hedges wrote:

> Just a wishlist that process IO could be monitored.  I hate to 
> say it but ctl-alt-esc in W2K can monitor io by process, and 
> that's really useful.  (I will never go back though.)

/etc/init.d/klogd stop
echo 1 >/proc/sys/vm/block_dump
/sbin/klogd -n -f -

watch lots of spew

^C
echo 0 >/proc/sys/vm/block_dump
/etc/init.d/klogd start

that will tell you a lot of stuff.  it will probably lead you to a process 
or two as culprits... use strace(8) to attach to those and get more 
specific information.

now... share with me how to do this on windoze?  :)  i'm trying to get rid 
of the last write or two on my laptop...

-dean

Re: unknown io writes in 2.6.16

[Mark Hedges]

On Fri, 12 May 2006, dean gaudet wrote:
> On Wed, 10 May 2006, Mark Hedges wrote:
> 
> > Just a wishlist that process IO could be monitored.  I hate to 
> > say it but ctl-alt-esc in W2K can monitor io by process, and 
> > that's really useful.  (I will never go back though.)
> 
> /etc/init.d/klogd stop
> echo 1 >/proc/sys/vm/block_dump
> /sbin/klogd -n -f -
> 
> watch lots of spew
> 
> ^C
> echo 0 >/proc/sys/vm/block_dump
> /etc/init.d/klogd start
> 
> that will tell you a lot of stuff.  it will probably lead you to a process 
> or two as culprits... use strace(8) to attach to those and get more 
> specific information.
> 
> now... share with me how to do this on windoze?  :)  i'm trying to get rid 
> of the last write or two on my laptop...

Cool, thanks, I'll check this out.  It's actually about 12k 
every 5 seconds, not 12 bytes. Seems excessive for atime 
updates.

In W2K at any rate, ctl-shift-esc to get the task manager.  
Processes tab. View Menu > Select Columns.  Sort by column.

Mark

Re: unknown io writes in 2.6.16

[dean gaudet]

On Mon, 15 May 2006, Mark Hedges wrote:

> In W2K at any rate, ctl-shift-esc to get the task manager.  
> Processes tab. View Menu > Select Columns.  Sort by column.

heh i already know about that... it's fairly useless though when the disk 
writes are due to logged events going through the same services dll... i 
was looking for something a little more detailed... like the block_dump + 
strace.

-dean

Re: unknown io writes in 2.6.16

[Con Kolivas]

On Tuesday 16 May 2006 10:12, Mark Hedges wrote:
> Cool, thanks, I'll check this out.  It's actually about 12k
> every 5 seconds, not 12 bytes. Seems excessive for atime
> updates.

That just sounds like the journal updating... the default journal time is 5 
seconds. Try transiently disabling the journal to see if that's it:

mount -o remount,noload /mountpoint

Don't forget to re-enable it afterwards.

You could set laptop mode if the writeouts are too frequent for your liking:

echo 1 > /proc/sys/vm/laptop_mode

-- 
-ck