Heavy disk activity without apperant reason (added more info)

Heavy disk activity without apperant reason (added more info)

[Stef van der Made]

On my AMD athlon system with 512MB memory I sometimes get a lot of disk 
activity the activity normaly lasts for about 10 seconds and after that 
the disk stays relativily quiet as expected with the load on the system. 
When I look into top I don't see any programs that could explain the 
disk activity. The system is in most cases not using any swap.

The system configuration is as following.

Software
Slackware 8.0
glibc 2.3.1
gcc 3.3.1
kernel 2.6.0-test9
all the needed software updates to run 2.5 and 2.6 kernels

AMD athlon 1400
512MB main mem
18GB scsi disk 10K
29160 adaptec scsi controller
using a via kt2666 chipset

Disk setup

bash-2.05$ cat /etc/fstab
/dev/sda1       /        ext2        defaults        1  1
/dev/sda6       /squid   reiserfs    defaults        1  1
/dev/sda5       /home    reiserfs    defaults        1  1
/dev/sda7       none     swap        defaults        0  0
/dev/sda8       /usr     reiserfs    defaults        1  1
none            /dev/pts devpts      gid=5,mode=620  0  0
none            /proc    proc        defaults        0  0
none            /proc/bus/usb  usbfs  defaults  0  0
/dev/hda5       /music   ext2        defaults        1  1

dmesg

bash-2.05# dmesg
Linux version 2.6.0-test9 (root@made0120) (gcc version 3.3.2) #18 Sat 
Oct 25 23:05:26 CEST 2003
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000001fff0000 (usable)
 BIOS-e820: 000000001fff0000 - 000000001fff3000 (ACPI NVS)
 BIOS-e820: 000000001fff3000 - 0000000020000000 (ACPI data)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
511MB LOWMEM available.
On node 0 totalpages: 131056
  DMA zone: 4096 pages, LIFO batch:1
  Normal zone: 126960 pages, LIFO batch:16
  HighMem zone: 0 pages, LIFO batch:1
DMI 2.2 present.
Building zonelist for node : 0
Kernel command line: BOOT_IMAGE=Linux ro root=801
Initializing CPU#0
PID hash table entries: 2048 (order 11: 16384 bytes)
Detected 1402.029 MHz processor.
Console: colour VGA+ 80x25
Memory: 514988k/524224k available (2153k kernel code, 8488k reserved, 
635k data, 344k init, 0k highmem)
Calibrating delay loop... 2760.70 BogoMIPS
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU:     After generic identify, caps: 0183fbff c1c7fbff 00000000 00000000
CPU:     After vendor identify, caps: 0183fbff c1c7fbff 00000000 00000000
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 256K (64 bytes/line)
CPU:     After all inits, caps: 0183fbff c1c7fbff 00000000 00000020
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: AMD Athlon(tm) processor stepping 04
Enabling fast FPU save and restore... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfb470, last bus=1
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
Linux Plug and Play Support v0.97 (c) Adam Belay
SCSI subsystem initialized
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Using IRQ router default [1106/3099] at 0000:00:00.0
Machine check exception polling timer started.
apm: BIOS version 1.2 Flags 0x07 (Driver version 1.16ac)
pty: 256 Unix98 ptys configured
Linux agpgart interface v0.100 (c) Dave Jones
agpgart: Detected VIA KT266/KY266x/KT333 chipset
agpgart: Maximum main memory to use for agp memory: 439M
agpgart: AGP aperture is 64M @ 0xe8000000
[drm] Initialized radeon 1.9.0 20020828 on minor 0
Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing disabled
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
8139too Fast Ethernet driver 0.9.26
eth0: RealTek RTL8139 at 0xe0816000, 00:e0:4c:3b:5d:84, IRQ 5
eth0:  Identified 8139 chip type 'RTL-8139C'
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
VP_IDE: IDE controller at PCI slot 0000:00:11.1
VP_IDE: chipset revision 6
VP_IDE: not 100% native mode: will probe irqs later
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
VP_IDE: VIA vt8235 (rev 00) IDE UDMA133 controller on pci0000:00:11.1
    ide0: BM-DMA at 0xec00-0xec07, BIOS settings: hda:DMA, hdb:DMA
    ide1: BM-DMA at 0xec08-0xec0f, BIOS settings: hdc:pio, hdd:DMA
hda: WDC WD205BA, ATA DISK drive
hdb: AOpen 12xDVD-ROM DRIVE 10172000, ATAPI CD/DVD-ROM drive
Using anticipatory io scheduler
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hdd: OnStream DI-30, ATAPI TAPE drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 40088160 sectors (20525 MB) w/2048KiB Cache, CHS=39770/16/63, UDMA(66)
 hda: hda1 hda4 < hda5 hda6 >
hdb: ATAPI DVD-ROM drive, 512kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.12
ide-tape: hdd <-> ht0: OnStream DI-30 rev 1.09
ide-tape: hdd <-> ht0: Tape length 14429MB (19239 frames/track, 24 
tracks = 461736 blocks, density: 64Kbpi)
ide-tape: hdd <-> ht0: 990KBps, 64*32kB buffer, 10208kB pipeline, 62ms 
tDSC, DMAscsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.35
        <Adaptec 29160 Ultra160 SCSI adapter>
        aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
 
(scsi0:A:0): 40.000MB/s transfers (20.000MHz, offset 63, 16bit)
  Vendor: QUANTUM   Model: ATLAS_V_18_WLS    Rev: 0230
  Type:   Direct-Access                      ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled.  Depth 32
SCSI device sda: 35861388 512-byte hdwr sectors (18361 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2 < sda5 sda6 sda7 sda8 >
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0,  type 0
mice: PS/2 mouse device common for all mice
input: PS2++ Logitech Mouse on isa0060/serio1
serio: i8042 AUX port at 0x60,0x64 irq 12
input: AT Translated Set 2 keyboard on isa0060/serio0
serio: i8042 KBD port at 0x60,0x64 irq 1
Advanced Linux Sound Architecture Driver Version 0.9.7 (Thu Sep 25 
19:16:36 2003 UTC).
request_module: failed /sbin/modprobe -- snd-card-0. error = -16
ALSA device list:
  #0: Sound Blaster Live! (rev.8) at 0xd800, irq 11
NET: Registered protocol family 2
IP: routing cache hash table of 4096 buckets, 32Kbytes
TCP: Hash tables configured (established 32768 bind 65536)
NET: Registered protocol family 1
NET: Registered protocol family 17
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 344k freed
Adding 136512k swap on /dev/sda7.  Priority:-1 extents:1
warning: process `update' used the obsolete bdflush system call
Fix your initscripts?
found reiserfs format "3.6" with standard journal
Reiserfs journal params: device sda6, size 8192, journal first block 18, 
max trans len 1024, max batch 900, max commit age 30, max trans age 30
reiserfs: checking transaction log (sda6) for (sda6)
reiserfs: replayed 8 transactions in 0 seconds
Using r5 hash to sort names
found reiserfs format "3.6" with standard journal
Reiserfs journal params: device sda5, size 8192, journal first block 18, 
max trans len 1024, max batch 900, max commit age 30, max trans age 30
reiserfs: checking transaction log (sda5) for (sda5)
reiserfs: replayed 12 transactions in 1 seconds
Using r5 hash to sort names
found reiserfs format "3.6" with standard journal
Reiserfs journal params: device sda8, size 8192, journal first block 18, 
max trans len 1024, max batch 900, max commit age 30, max trans age 30
reiserfs: checking transaction log (sda8) for (sda8)
reiserfs: replayed 7 transactions in 1 seconds
Using r5 hash to sort names
eth0: link up, 100Mbps, half-duplex, lpa 0x40A1


I'm not sure if I should log a bug and what the problem area could be.

Thanks in advance for helping out,

Stef

Re: Heavy disk activity without apperant reason (added more info)

[Jose Luis Domingo Lopez]

On Sunday, 26 October 2003, at 13:55:05 +0100,
Stef van der Made wrote:

> The system configuration is as following.
> 
Try first to isolate the kind of disk activity happening: is it writes,
reads, a mix of them ?. Start some monitoring program like "vmstat",
"sar" or even "xosview" or "gkrellm" to see the kind of disk access
pattern happens when your box does things of its own :-)

Also, to put swap out of the equation you could disable it and see if
the problem happens again, etc.

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.6.0-test8-mm1)

Re: Heavy disk activity without apperant reason (added more info)

[jw schultz]

On Sun, Oct 26, 2003 at 01:55:05PM +0100, Stef van der Made wrote:
> 
> On my AMD athlon system with 512MB memory I sometimes get a lot of disk 
> activity the activity normaly lasts for about 10 seconds and after that 
> the disk stays relativily quiet as expected with the load on the system. 
> When I look into top I don't see any programs that could explain the 
> disk activity. The system is in most cases not using any swap.

In the vein of "make sure its plugged in", make sure it
isn't cron.

Re: Heavy disk activity without apperant reason (added more info)

[Helge Hafting]

Stef van der Made wrote:
> 
> On my AMD athlon system with 512MB memory I sometimes get a lot of disk 
> activity the activity normaly lasts for about 10 seconds and after that 
> the disk stays relativily quiet as expected with the load on the system. 
> When I look into top I don't see any programs that could explain the 
> disk activity. The system is in most cases not using any swap.
> 
Try finding out what is causing this.
Have a "vmstat 1" running.  Break it after this
disk activity starts.  You should be able to
see wether it is normal io or swap.

Also have a "top -d 1" running.  A normal
process issuing lots of io will probably
show up here too.  "ps aux" during
the activity might also be a good idea.

Note that such behaviour isn't necessarily unusual.
Perhaps cron started something that needed lots
of reads to start?  Perhaps you got a bunch of emails?
Email software often use synchronous writes, so they won't
loose any of your mail even in case of a crash.
This synchronous io makes for _lots_ of disk seeking.
Email filters (for spam and other purposes) may make this even worse, 
with email messages being written synchronously several times.
If you use "fetchmail" started by cron - see if these disk bursts
correspond with mail fetching.

Helge Hafting

Re: Heavy disk activity without apperant reason (added more info)

[Stef van der Made]

Dear Helge,

I had the very interesting disk activity again. No serious activity was 
happening. I've tried to make some sense of the 2 output commands you've 
asked, but they make not much sense. Sorry :-( that I need to ask so 
much help on this (bug) report.

This is the output of vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- 
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy 
id wa
 1  0      0 122124  37396 238892    0    0     0   100 1073  1246 98  
2  0  0
 1  0      0 121996  37396 239020    0    0   128     0 1079  1068 100  
0  0  0
 1  0      0 121988  37396 239020    0    0     0     0 1109  1266 99  
1  0  0
 1  0      0 121860  37396 239020    0    0     0     0 1124  1393 97  
3  0  0
 1  0      0 121924  37396 239020    0    0     0     0 1143  1220 98  
2  0  0
 2  0      0 121924  37416 239020    0    0     0    32 1081  1154 99  
1  0  0
 2  0      0 121924  37416 239020    0    0     0     0 1061  1096 99  
1  0  0
 1  0      0 121796  37448 239148    0    0   128    36 1063  1104 99  
1  0  0
 1  0      0 121732  37448 239152    0    0     0     0 1147  1471 98  
2  0  0
 2  0      0 121732  37448 239152    0    0     0     0 1118  1330 99  
1  0  0
 1  2      0 121212  37484 239160    0    0     0   180 1264  1631 96  
4  0  0
 1  1      0 121148  37484 239160    0    0     0   158 1305  1540 95  
5  0  0
 1  1      0 121148  37484 239164    0    0     0   166 1300  1678 95  
5  0  0
 1  1      0 121148  37484 239172    0    0     0   199 1315  6863 93  
7  0  0
 1  2      0 121020  37484 239316    0    0   128   217 1303  2197 94  
6  0  0
 1  2      0 121020  37500 239320    0    0     0   213 1300  7326 91  
9  0  0
 1  1      0 120956  37500 239328    0    0     0   225 1352  1842 93  
7  0  0
 3  1      0 120948  37500 239336    0    0     0   219 1346  1656 94  
6  0  0
 1  1      0 120772  37500 239348    0    0     0   228 1351  1678 94  
6  0  0
 2  1      0 120948  37500 239356    0    0     0   198 1252  1476 96  
4  0  0
 1  0      0 121204  37500 239484    0    0   128    40 1122  1162 98  
2  0  0

and this is top

top - 20:17:06 up  1:15,  1 user,  load average: 1.63, 1.26, 1.19
Tasks:  77 total,   3 running,  74 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.0% us,  2.0% sy, 93.1% ni,  0.0% id,  0.0% wa,  1.0% hi,  0.0% si
Mem:    515692k total,   394488k used,   121204k free,    37500k buffers
Swap:   136512k total,        0k used,   136512k free,   239484k cached
                                                                                

  PID USER      PR      NI  VIRT      RES      SHR S %CPU %MEM    TIME+  
COMMAND
  118     root      39      19      1120      584     1072 R 94.4  0.1  
68:55.07 dnetc
  225     root      16       0     73620      20m      54m R  1.0  4.2   
1:15.36 X
  263     stef      15       0     21116     6720     7412 S  1.0  1.3   
0:05.12 xmms
  303     stef      15       0     22948      14m      14m S  1.0  3.0   
0:14.47 gnome-terminal
  318     stef      15       0     66528      49m      26m S  1.0  9.9   
3:56.12 mozilla-bin
  667     stef      16       0      1912     1028     1756 R  1.0  0.2   
0:09.34 top
    1     root      16       0       420      216      388 S  0.0  0.0   
0:06.29 init
    2     root      34      19     0    0        0 S      0.0  0.0   
0:00.00 ksoftirqd/0
    3     root       5     -10      0    0        0 S      0.0  0.0   
0:00.02 events/0
    4     root       5     -10      0    0        0 S      0.0  0.0   
0:00.01 kblockd/0
    5     root      15       0      0    0        0 S      0.0  0.0   
0:00.03 kapmd
    6     root      25       0      0    0        0 S      0.0  0.0   
0:00.00 pdflush
    7     root      15       0      0    0        0 S      0.0  0.0   
0:00.28 pdflush
    8     root      25       0      0    0        0 S      0.0  0.0   
0:00.00 kswapd0
    9     root      10     -10     0    0        0 S      0.0  0.0   
0:00.00 aio/0
   10 root      21           0     0    0        0 S      0.0  0.0   
0:00.00 scsi_eh_0
   11 root      15           0     0    0        0 S      0.0  0.0   
0:00.00 ahc_dv_0

Best regards,

Stef

Helge Hafting wrote:

> Stef van der Made wrote:
>
>>
>> On my AMD athlon system with 512MB memory I sometimes get a lot of 
>> disk activity the activity normaly lasts for about 10 seconds and 
>> after that the disk stays relativily quiet as expected with the load 
>> on the system. When I look into top I don't see any programs that 
>> could explain the disk activity. The system is in most cases not 
>> using any swap.
>>
> Try finding out what is causing this.
> Have a "vmstat 1" running.  Break it after this
> disk activity starts.  You should be able to
> see wether it is normal io or swap.
>
> Also have a "top -d 1" running.  A normal
> process issuing lots of io will probably
> show up here too.  "ps aux" during
> the activity might also be a good idea.
>
> Note that such behaviour isn't necessarily unusual.
> Perhaps cron started something that needed lots
> of reads to start?  Perhaps you got a bunch of emails?
> Email software often use synchronous writes, so they won't
> loose any of your mail even in case of a crash.
> This synchronous io makes for _lots_ of disk seeking.
> Email filters (for spam and other purposes) may make this even worse, 
> with email messages being written synchronously several times.
> If you use "fetchmail" started by cron - see if these disk bursts
> correspond with mail fetching.
>
> Helge Hafting
>
>

Re: Heavy disk activity without apperant reason (added more info)

[Helge Hafting]

Stef van der Made wrote:
> 
> Dear Helge,
> 
> I had the very interesting disk activity again. No serious activity was 
> happening. I've tried to make some sense of the 2 output commands you've 
> asked, but they make not much sense. Sorry :-( that I need to ask so 
> much help on this (bug) report.
> 
> This is the output of vmstat

[...]
>From this we see that it isn't swapping (0 in the si and so
columns {"swap in" and "swap out"}). 
The machine occationally reads, and it writes a lot.

> and this is top
> 
dnetc is the most active process here.  I don't know
what it _is_ - could it be doing lots of disk
writes?  

mozilla is running, it may do a lot of disk
access under some circumstances, for example
when refreshing some webpage with lots
of content.

Helge Hafting

Re: Heavy disk activity without apperant reason (added more info)

[Mike Dresser]

On Tue, 28 Oct 2003, Helge Hafting wrote:

> dnetc is the most active process here.  I don't know
> what it _is_ - could it be doing lots of disk
> writes?

likely the distributed.net client, shouldnt' normally be doing any disk
writing at all except between blocks.

Re: Heavy disk activity without apperant reason (added more info)

[Stef van der Made]

Dear All,

Thanks for all the help. I reread all the emails sent and looked into 
cron. The dug into the logs and found some interesting stuff in the 
maillog and messages. Sendmail apperantly needed procmail to work 
properly with the latest version. This solved my issue.

Thanks everybody for your help. This again proves that I'm just a newbee 
in the debuging of problems on Linux. OS/400 seems to be more my area of 
expertise ;-)

Stef

Helge Hafting wrote:

> Stef van der Made wrote:
>
>>
>> On my AMD athlon system with 512MB memory I sometimes get a lot of 
>> disk activity the activity normaly lasts for about 10 seconds and 
>> after that the disk stays relativily quiet as expected with the load 
>> on the system. When I look into top I don't see any programs that 
>> could explain the disk activity. The system is in most cases not 
>> using any swap.
>>
> Try finding out what is causing this.
> Have a "vmstat 1" running.  Break it after this
> disk activity starts.  You should be able to
> see wether it is normal io or swap.
>
> Also have a "top -d 1" running.  A normal
> process issuing lots of io will probably
> show up here too.  "ps aux" during
> the activity might also be a good idea.
>
> Note that such behaviour isn't necessarily unusual.
> Perhaps cron started something that needed lots
> of reads to start?  Perhaps you got a bunch of emails?
> Email software often use synchronous writes, so they won't
> loose any of your mail even in case of a crash.
> This synchronous io makes for _lots_ of disk seeking.
> Email filters (for spam and other purposes) may make this even worse, 
> with email messages being written synchronously several times.
> If you use "fetchmail" started by cron - see if these disk bursts
> correspond with mail fetching.
>
> Helge Hafting
>
>