Re: high system cpu load during intense disk i/o

Dimitrios Apostolou
2007-08-17T22:19:39Z

[-- Attachment #1: Type: text/plain, Size: 2841 bytes --]

Hello list,

before trying to reproduce the problem with older kernels I did the
necessary step of compiling and using a *vanilla* simple monolithic
kernel for my measurements. The kernel config (attached config.gz) has
many standard things disabled (like ACPI for example) so the oprofile
output now seems very different. Please keep in mind that I switched
back from libata to the old IDE driver, to be able to use the same
config on old kernels.

The situations I attach are:

idle: The PC doing nothing. Note that now idle time is spent in
irq_handler and not in poll_idle. Strange...
one_disk: Destructive badblocks (badblocks -v -w) on one disk.
Everything is responsive and the CPU is 99% iowait as it should.
two_disks: Destructive badblocks on two disks before the problem
appears. Things are starting to get sluggy.
two_disks_bad2: *PROBLEM* The previous situation after several
minutes, and after several cron jobs kicked in (and never finished).
System in a bad state, highly unresponsive.

The situation seems now completely different (but practically the
problem is exactly the same), probably because of kernel options. Here
are the first lines from opreport with debugging info, for the
two_disks_bad2 scenario:

CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a
unit mask of 0x00 (No unit mask) count 100000
samples % linenr info symbol name
282 9.3377 ide-iops.c:1081 pre_reset
231 7.6490 stats.c:187 rpc_print_iostats
222 7.3510 ptrace.c:654 do_syscall_trace
146 4.8344 ide-io.c:1185 ide_do_request
144 4.7682 process.c:529 dump_task_regs
131 4.3377 stats.c:64 rpc_proc_open
122 4.0397 backing-dev.c:46 congestion_wait
98 3.2450 vsprintf.c:622 vsscanf
52 1.7219 process.c:643 __switch_to
33 1.0927 sched.c:4065 interruptible_sleep_on
32 1.0596 slub.c:597 check_object
32 1.0596 signal.c:244 setup_sigcontext
32 1.0596 signal.c:56 sys_sigaction
31 1.0265 buffer.c:2452 block_truncate_page
31 1.0265 fadvise.c:28 sys_fadvise64_64
31 1.0265 page-writeback.c:987 test_set_page_writeback

If you think I should enable/disable other options in the kernel please
tell me. Moreover it would be nice to know how to use the various
debugging options that I enabled, to help figuring out the problem. So
what do you think? Does this help or should I start trying older kernels
(which is *hard* to do with latest libc and udev that I have)?

Thanks again,
Dimitris

[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 5793 bytes --]

[-- Attachment #3: dmesg.txt --]
[-- Type: text/plain, Size: 9092 bytes --]

[ 0.000000] Linux version 2.6.22.3mango-monolithic-dbg (jimis@mango) (gcc version 4.2.1) #2 Fri Aug 17 19:18:43 EEST 2007
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
[ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000000fff0000 (usable)
[ 0.000000] BIOS-e820: 000000000fff0000 - 000000000fff3000 (ACPI NVS)
[ 0.000000] BIOS-e820: 000000000fff3000 - 0000000010000000 (ACPI data)
[ 0.000000] BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
[ 0.000000] using polling idle threads.
[ 0.000000] 255MB LOWMEM available.
[ 0.000000] Entering add_active_range(0, 0, 65520) 0 entries of 256 used
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0 -> 4096
[ 0.000000] Normal 4096 -> 65520
[ 0.000000] early_node_map[1] active PFN ranges
[ 0.000000] 0: 0 -> 65520
[ 0.000000] On node 0 totalpages: 65520
[ 0.000000] DMA zone: 32 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 4064 pages, LIFO batch:0
[ 0.000000] Normal zone: 479 pages used for memmap
[ 0.000000] Normal zone: 60945 pages, LIFO batch:15
[ 0.000000] DMI 2.2 present.
[ 0.000000] Allocating PCI resources starting at 20000000 (gap: 10000000:efff0000)
[ 0.000000] Built 1 zonelists. Total pages: 65009
[ 0.000000] Kernel command line: auto BOOT_IMAGE=2.6.22.3-dbg ro root=341 lapic nmi_watchdog=0 idle=poll
[ 0.000000] Local APIC disabled by BIOS -- reenabling.
[ 0.000000] Found and enabled local APIC!
[ 0.000000] mapped APIC to ffffd000 (fee00000)
[ 0.000000] Enabling fast FPU save and restore... done.
[ 0.000000] Enabling unmasked SIMD FPU exception support... done.
[ 0.000000] Initializing CPU#0
[ 0.000000] PID hash table entries: 1024 (order: 10, 4096 bytes)
[ 0.000000] Detected 798.020 MHz processor.
[ 36.983538] Console: colour VGA+ 80x25
[ 36.986206] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 36.986676] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
[ 37.012595] Memory: 256036k/262080k available (1963k kernel code, 5544k reserved, 611k data, 140k init, 0k highmem)
[ 37.012781] virtual kernel memory layout:
[ 37.012785] fixmap : 0xffffc000 - 0xfffff000 ( 12 kB)
[ 37.012789] vmalloc : 0xd0800000 - 0xffffa000 ( 759 MB)
[ 37.012792] lowmem : 0xc0000000 - 0xcfff0000 ( 255 MB)
[ 37.012796] .init : 0xc0386000 - 0xc03a9000 ( 140 kB)
[ 37.012800] .data : 0xc02eacd7 - 0xc0383b64 ( 611 kB)
[ 37.012804] .text : 0xc0100000 - 0xc02eacd7 (1963 kB)
[ 37.013432] Checking if this processor honours the WP bit even in supervisor mode... Ok.
[ 37.013699] SLUB: Genslabs=22, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
[ 37.163848] Calibrating delay using timer specific routine.. 1597.05 BogoMIPS (lpj=7985266)
[ 37.164075] Mount-cache hash table entries: 512
[ 37.164358] CPU: After generic identify, caps: 0387fbff 00000000 00000000 00000000 00000000 00000000 00000000
[ 37.164382] CPU: L1 I cache: 16K, L1 D cache: 16K
[ 37.164501] CPU: L2 cache: 256K
[ 37.164587] CPU serial number disabled.
[ 37.164675] CPU: After all inits, caps: 0383fbff 00000000 00000000 00000040 00000000 00000000 00000000
[ 37.164688] Intel machine check architecture supported.
[ 37.164782] Intel machine check reporting enabled on CPU#0.
[ 37.164880] Compat vDSO mapped to ffffe000.
[ 37.164986] CPU: Intel Pentium III (Coppermine) stepping 06
[ 37.165168] Checking 'hlt' instruction... OK.
[ 37.444042] NET: Registered protocol family 16
[ 37.475344] PCI: PCI BIOS revision 2.10 entry at 0xfb370, last bus=1
[ 37.475446] PCI: Using configuration type 1
[ 37.475541] Setting up standard PCI resources
[ 37.478177] PCI: Probing PCI hardware
[ 37.478315] PCI: Probing PCI hardware (bus 00)
[ 37.480044] PCI: Using IRQ router VIA [1106/0596] at 0000:00:07.0
[ 37.489977] PCI: Bridge: 0000:00:01.0
[ 37.490080] IO window: disabled.
[ 37.490180] MEM window: d8000000-dfffffff
[ 37.490276] PREFETCH window: 20000000-200fffff
[ 37.490390] PCI: Setting latency timer of device 0000:00:01.0 to 64
[ 37.490422] NET: Registered protocol family 2
[ 37.493810] Time: tsc clocksource has been installed.
[ 37.573854] IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 37.574034] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[ 37.574363] TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
[ 37.574578] TCP: Hash tables configured (established 8192 bind 8192)
[ 37.574679] TCP reno registered
[ 37.604371] IA-32 Microcode Update Driver: v1.14a <tigran@aivazian.fsnet.co.uk>
[ 37.611242] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 37.611678] fuse init (API version 7.8)
[ 37.612390] io scheduler noop registered
[ 37.612492] io scheduler anticipatory registered (default)
[ 37.612592] io scheduler deadline registered
[ 37.612862] io scheduler cfq registered
[ 37.612980] PCI: VIA PCI bridge detected. Disabling DAC.
[ 37.613082] Activating ISA DMA hang workarounds.
[ 37.613213] Boot video device is 0000:01:00.0
[ 37.678453] Real Time Clock Driver v1.12ac
[ 37.678571] Hangcheck: starting hangcheck timer 0.9.0 (tick is 180 seconds, margin is 60 seconds).
[ 37.678740] Hangcheck: Using get_cycles().
[ 37.678847] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
[ 37.680289] serial8250.0: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 37.680872] serial8250.0: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[ 37.682616] loop: module loaded
[ 37.682825] PCI: setting IRQ 12 as level-triggered
[ 37.682834] PCI: Found IRQ 12 for device 0000:00:0a.0
[ 37.683001] skge 1.11 addr 0xe4000000 irq 12 chip Yukon rev 1
[ 37.683579] skge eth0: addr 00:0f:38:6a:9c:fe
[ 37.684108] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
[ 37.684219] ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
[ 37.684498] VP_IDE: IDE controller at PCI slot 0000:00:07.1
[ 37.684617] VP_IDE: chipset revision 6
[ 37.684714] VP_IDE: not 100% native mode: will probe irqs later
[ 37.684828] VP_IDE: VIA vt82c596b (rev 12) IDE UDMA66 controller on pci0000:00:07.1
[ 37.685004] ide0: BM-DMA at 0xe000-0xe007, BIOS settings: hda:DMA, hdb:DMA
[ 37.685233] ide1: BM-DMA at 0xe008-0xe00f, BIOS settings: hdc:DMA, hdd:pio
[ 37.685452] Probing IDE interface ide0...
[ 38.133759] hda: WDC WD2500JB-55REA0, ATA DISK drive
[ 38.433703] hdb: MAXTOR 6L020J1, ATA DISK drive
[ 38.494132] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
[ 38.494492] Probing IDE interface ide1...
[ 38.933603] hdc: WDC WD2500JB-55REA0, ATA DISK drive
[ 39.653537] ide1 at 0x170-0x177,0x376 on irq 15
[ 39.653980] hda: max request size: 512KiB
[ 39.662415] hda: 488397168 sectors (250059 MB) w/8192KiB Cache, CHS=30401/255/63, UDMA(66)
[ 39.662778] hda: cache flushes supported
[ 39.662941] hda: unknown partition table
[ 39.671391] hdb: max request size: 128KiB
[ 39.672508] hdb: 40132503 sectors (20547 MB) w/1819KiB Cache, CHS=39813/16/63, UDMA(66)
[ 39.672897] hdb: cache flushes supported
[ 39.673023] hdb: hdb1 hdb2
[ 39.679890] hdc: max request size: 512KiB
[ 39.688535] hdc: 488397168 sectors (250059 MB) w/8192KiB Cache, CHS=30401/255/63, UDMA(66)
[ 39.688882] hdc: cache flushes supported
[ 39.689028] hdc: unknown partition table
[ 39.943833] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 39.944151] mice: PS/2 mouse device common for all mice
[ 39.964472] input: AT Translated Set 2 keyboard as /class/input/input0
[ 39.964662] md: raid0 personality registered for level 0
[ 39.964776] oprofile: using NMI interrupt.
[ 39.964882] TCP cubic registered
[ 39.965194] NET: Registered protocol family 1
[ 39.965294] NET: Registered protocol family 17
[ 39.965605] Using IPI Shortcut mode
[ 39.966053] md: Autodetecting RAID arrays.
[ 39.966151] md: autorun ...
[ 39.966246] md: ... autorun DONE.
[ 39.981714] kjournald starting. Commit interval 5 seconds
[ 39.981839] EXT3-fs: mounted filesystem with ordered data mode.
[ 39.981955] VFS: Mounted root (ext3 filesystem) readonly.
[ 39.982331] Freeing unused kernel memory: 140k freed
[ 43.937797] md: md0 stopped.
[ 47.134790] EXT3 FS on hdb1, internal journal
[ 47.250784] ReiserFS: hdb2: found reiserfs format "3.6" with standard journal
[ 47.250820] ReiserFS: hdb2: using ordered data mode
[ 47.251544] ReiserFS: hdb2: journal params: device hdb2, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
[ 47.256771] ReiserFS: hdb2: checking transaction log (hdb2)
[ 47.312336] ReiserFS: hdb2: Using r5 hash to sort names
[ 48.732000] skge eth0: enabling interface
[ 51.219364] skge eth0: Link is up at 1000 Mbps, full duplex, flow control both

[-- Attachment #4: oprof_idle.txt --]
[-- Type: text/plain, Size: 9502 bytes --]

+ date
Fri Aug 17 20:07:32 EEST 2007
+ rm -rf /var/lib/oprofile/
+ opcontrol --vmlinux=/home/jimis/dist/src/linux-2.6.22.3/vmlinux
+ opcontrol --start
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
+ sleep 5
+ opcontrol --shutdown
Stopping profiling.
Killing daemon.
+ echo

+ echo

+ echo

+ opreport
CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
27648 96.8746 vmlinux
360 1.2614 libc-2.6.1.so
178 0.6237 bash
139 0.4870 oprofiled
CPU_CLK_UNHALT...|
samples| %|
------------------
138 99.2806 oprofiled
1 0.7194 [vdso] (tgid:11997 range:0xb7f96000-0xb7f97000)
131 0.4590 ld-2.6.1.so
59 0.2067 ISO8859-1.so
6 0.0210 locale-archive
5 0.0175 gawk
3 0.0105 libcrypto.so.0.9.8
2 0.0070 libpthread-2.6.1.so
2 0.0070 imap-login
1 0.0035 ls
1 0.0035 libhistory.so.5.2
1 0.0035 screen-4.0.3
1 0.0035 libnetsnmp.so.15.0.0
1 0.0035 dovecot-auth
1 0.0035 dovecot
1 0.0035 sshd
+ echo

+ echo

+ echo

+ opreport -l /home/jimis/dist/src/linux-2.6.22.3/vmlinux
CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
26423 95.5693 irq_handler
288 1.0417 kmem_cache_shrink
190 0.6872 create_kmalloc_cache
105 0.3798 congestion_wait
38 0.1374 vsscanf
30 0.1085 assign_all_busses
30 0.1085 sys_fadvise64_64
28 0.1013 interruptible_sleep_on
25 0.0904 sys_madvise
24 0.0868 do_wp_page
14 0.0506 print_hex_dump
12 0.0434 check_object
12 0.0434 ext3_reserve_inode_write
11 0.0398 generic_file_buffered_write
11 0.0398 handle_vm86_fault
11 0.0398 vsnprintf
10 0.0362 kobject_rename
9 0.0326 __relay_reset
9 0.0326 do_syscall_trace
9 0.0326 rtc_cmos_read
8 0.0289 __handle_mm_fault
8 0.0289 setup_sigcontext
7 0.0253 access_process_vm
7 0.0253 locks_mandatory_area
7 0.0253 pcibios_setup
7 0.0253 print_bad_pte
6 0.0217 load_elf_binary
6 0.0217 pirq_piix_set
6 0.0217 vfs_mknod
5 0.0181 ext3_orphan_get
5 0.0181 fcntl_setlk
5 0.0181 loop_alloc
5 0.0181 process_slab
5 0.0181 rt_mutex_setprio
4 0.0145 __switch_to_xtra
4 0.0145 shrink_zone
3 0.0109 __dequeue_signal
3 0.0109 __switch_to
3 0.0109 __vmalloc_area_node
3 0.0109 block_invalidatepage
3 0.0109 do_mremap
3 0.0109 do_sync
3 0.0109 ext3_free_inode
3 0.0109 ext3_new_inode
3 0.0109 follow_page
3 0.0109 out_of_memory
3 0.0109 pcibios_enable_device
3 0.0109 pipe_write
3 0.0109 pirq_enable_irq
3 0.0109 posix_cpu_nsleep
3 0.0109 sys_remap_file_pages
3 0.0109 try_to_wake_up
3 0.0109 vfs_mkdir
3 0.0109 vfs_rename
3 0.0109 zoneinfo_show
2 0.0072 __free_slab
2 0.0072 __remove_shared_vm_struct
2 0.0072 background_writeout
2 0.0072 bitmap_find_free_region
2 0.0072 calibrate_delay
2 0.0072 dentry_open
2 0.0072 do_getitimer
2 0.0072 do_mmap_pgoff
2 0.0072 do_munmap
2 0.0072 do_page_fault
2 0.0072 elf_core_dump
2 0.0072 ext3_xattr_set
2 0.0072 install_page
2 0.0072 log_do_checkpoint
2 0.0072 lookup_one_len
2 0.0072 madvise_need_mmap_write
2 0.0072 pdflush
2 0.0072 prepare_timeout
2 0.0072 register_chrdev
2 0.0072 remap_pfn_range
2 0.0072 send_group_sigqueue
2 0.0072 swap_readpage
2 0.0072 sys_mincore
2 0.0072 sys_mprotect
2 0.0072 sys_timerfd
2 0.0072 t_start
2 0.0072 user_shm_lock
2 0.0072 vm_normal_page
2 0.0072 wake_up_new_task
1 0.0036 __cond_resched
1 0.0036 __filemap_copy_from_user_iovec_inatomic
1 0.0036 __follow_mount
1 0.0036 __free_pages_ok
1 0.0036 __get_user_2
1 0.0036 __inode_dir_notify
1 0.0036 __ip_route_output_key
1 0.0036 __journal_refile_buffer
1 0.0036 __journal_temp_unlink_buffer
1 0.0036 __kill_fasync
1 0.0036 __oom_kill_task
1 0.0036 __posix_lock_file
1 0.0036 __pte_alloc
1 0.0036 __set_special_pids
1 0.0036 __setscheduler
1 0.0036 __sigqueue_alloc
1 0.0036 __vma_link
1 0.0036 __wait_on_freeing_inode
1 0.0036 aio_complete
1 0.0036 badness
1 0.0036 clear_page_dirty_for_io
1 0.0036 clockevents_set_mode
1 0.0036 congestion_wait_interruptible
1 0.0036 copy_page_range
1 0.0036 copy_process
1 0.0036 cpu_idle
1 0.0036 d_lookup
1 0.0036 daemonize
1 0.0036 dcache_dir_lseek
1 0.0036 do_cpu_nanosleep
1 0.0036 do_lookup
1 0.0036 do_msgrcv
1 0.0036 do_notify_parent
1 0.0036 do_sendfile
1 0.0036 do_sync_read
1 0.0036 do_sysctl
1 0.0036 do_utimes
1 0.0036 dup_fd
1 0.0036 early_serial_putc
1 0.0036 early_serial_write
1 0.0036 ext3_count_dirs
1 0.0036 ext3_count_free_inodes
1 0.0036 ext3_read_inode
1 0.0036 ext3_readdir
1 0.0036 ext3_rename
1 0.0036 ext3_xattr_block_set
1 0.0036 ext3_xattr_get
1 0.0036 ext3_xattr_set_handle
1 0.0036 ext3_xattr_trusted_list
1 0.0036 file_send_actor
1 0.0036 find_inode
1 0.0036 find_mergeable_anon_vma
1 0.0036 force_sigsegv
1 0.0036 generic_permission
1 0.0036 generic_segment_checks
1 0.0036 generic_shutdown_super
1 0.0036 generic_write_checks
1 0.0036 get_next_timer_interrupt
1 0.0036 hrtimer_nanosleep
1 0.0036 i8259A_shutdown
1 0.0036 ide_do_request
1 0.0036 inode_sub_bytes
1 0.0036 insert_vm_struct
1 0.0036 internal_add_timer
1 0.0036 interruptible_sleep_on_timeout
1 0.0036 ioremap_nocache
1 0.0036 itimer_get_remtime
1 0.0036 journal_commit_transaction
1 0.0036 journal_get_create_access
1 0.0036 journal_recover
1 0.0036 journal_refile_buffer
1 0.0036 kill_block_super
1 0.0036 kill_litter_super
1 0.0036 kmem_cache_create
1 0.0036 kmem_cache_destroy
1 0.0036 kmem_ptr_validate
1 0.0036 kobject_get_path
1 0.0036 kobject_shadow_add
1 0.0036 kobject_uevent_env
1 0.0036 lock_timer
1 0.0036 locks_insert_block
1 0.0036 may_open
1 0.0036 memcmp
1 0.0036 memcpy
1 0.0036 mempool_resize
1 0.0036 microcode_write
1 0.0036 modify_acceptable_latency
1 0.0036 number
1 0.0036 pci_write
1 0.0036 percpu_pagelist_fraction_sysctl_handler
1 0.0036 pirq_serverworks_set
1 0.0036 posix_cpu_timer_set
1 0.0036 posix_timer_event
1 0.0036 prepare_to_wait
1 0.0036 prio_tree_remove
1 0.0036 proc_dodebug
1 0.0036 put_files_struct
1 0.0036 put_io_context
1 0.0036 rb_erase
1 0.0036 reiserfs_delete_xattrs
1 0.0036 reiserfs_listxattr
1 0.0036 reiserfs_xattr_init
1 0.0036 relay_create_buf
1 0.0036 relay_switch_subbuf
1 0.0036 rpc_print_iostats
1 0.0036 rpc_proc_exit
1 0.0036 rpc_proc_init
1 0.0036 sched_setscheduler
1 0.0036 send_signal
1 0.0036 set_bdi_congested
1 0.0036 set_page_dirty
1 0.0036 show_regs
1 0.0036 show_state_filter
1 0.0036 skb_checksum
1 0.0036 skb_seq_read
1 0.0036 skge_poll
1 0.0036 sprint_symbol
1 0.0036 submit_bh
1 0.0036 sync_cmos_clock
1 0.0036 sys_chroot
1 0.0036 sys_close
1 0.0036 sys_fchdir
1 0.0036 sys_get_thread_area
1 0.0036 sys_mq_timedreceive
1 0.0036 sys_sched_getparam
1 0.0036 sys_setreuid
1 0.0036 sys_setuid
1 0.0036 sys_sync_file_range
1 0.0036 sys_vm86old
1 0.0036 task_running_tick
1 0.0036 tcp_rcv_established
1 0.0036 test_clear_page_writeback
1 0.0036 test_set_page_writeback
1 0.0036 throttle_vm_writeout
1 0.0036 tick_notify
1 0.0036 timer_list_show
1 0.0036 trace
1 0.0036 unregister_timer_hook
1 0.0036 update_wall_time
1 0.0036 vfs_create
1 0.0036 vfs_ioctl
1 0.0036 vfs_unlink
1 0.0036 vm_insert_pfn
1 0.0036 vma_adjust
1 0.0036 vma_link
1 0.0036 vma_merge
1 0.0036 vprintk
1 0.0036 wakeup_pdflush
1 0.0036 walk_page_buffers
+ date
Fri Aug 17 20:07:38 EEST 2007

[-- Attachment #5: oprof_one_disk.txt --]
[-- Type: text/plain, Size: 17721 bytes --]

+ date
Fri Aug 17 20:17:46 EEST 2007
+ rm -rf /var/lib/oprofile/
+ opcontrol --vmlinux=/home/jimis/dist/src/linux-2.6.22.3/vmlinux
+ opcontrol --start
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
+ sleep 5
+ opcontrol --shutdown
Stopping profiling.
Killing daemon.
+ echo

+ echo

+ echo

+ opreport
CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
10455 90.6922 vmlinux
410 3.5566 libc-2.6.1.so
239 2.0732 bash
CPU_CLK_UNHALT...|
samples| %|
------------------
237 99.1632 bash
2 0.8368 [vdso] (tgid:12553 range:0xb7f81000-0xb7f82000)
190 1.6482 ld-2.6.1.so
121 1.0496 oprofiled
57 0.4944 ISO8859-1.so
10 0.0867 gawk
10 0.0867 badblocks
CPU_CLK_UNHALT...|
samples| %|
------------------
6 60.0000 badblocks
4 40.0000 [vdso] (tgid:10904 range:0xb7eed000-0xb7eee000)
5 0.0434 grep
5 0.0434 locale-archive
4 0.0347 libext2fs.so.2.4
3 0.0260 libcrypto.so.0.9.8
3 0.0260 imap-login
CPU_CLK_UNHALT...|
samples| %|
------------------
2 66.6667 imap-login
1 33.3333 [vdso] (tgid:1946 range:0xb7f17000-0xb7f18000)
2 0.0173 libncurses.so.5.6
2 0.0173 screen-4.0.3
2 0.0173 dovecot
2 0.0173 sshd
1 0.0087 cat
1 0.0087 mkdir
1 0.0087 rm
1 0.0087 libreadline.so.5.2
1 0.0087 libnetsnmp.so.15.0.0
1 0.0087 libnetsnmphelpers.so.15.0.0
1 0.0087 libnetsnmpmibs.so.15.0.0
1 0.0087 syslog-ng
+ echo

+ echo

+ echo

+ opreport -l /home/jimis/dist/src/linux-2.6.22.3/vmlinux
CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
6879 65.7963 irq_handler
881 8.4266 kmem_cache_shrink
672 6.4275 create_kmalloc_cache
188 1.7982 congestion_wait
140 1.3391 pre_reset
104 0.9947 vsscanf
84 0.8034 do_syscall_trace
78 0.7461 ide_do_request
51 0.4878 assign_all_busses
46 0.4400 interruptible_sleep_on
35 0.3348 sys_fadvise64_64
33 0.3156 sys_madvise
30 0.2869 rtc_cmos_read
29 0.2774 do_wp_page
28 0.2678 kobject_rename
25 0.2391 check_object
25 0.2391 print_hex_dump
19 0.1817 __relay_reset
17 0.1626 setup_sigcontext
16 0.1530 vsnprintf
15 0.1435 ide_setup_pci_devices
15 0.1435 zoneinfo_show
14 0.1339 block_truncate_page
14 0.1339 test_set_page_writeback
13 0.1243 __blkdev_put
13 0.1243 cont_prepare_write
13 0.1243 load_elf_binary
12 0.1148 access_process_vm
12 0.1148 pcibios_setup
11 0.1052 __handle_mm_fault
11 0.1052 __switch_to
11 0.1052 do_ide_setup_pci_device
11 0.1052 dump_task_extended_fpu
11 0.1052 rpc_print_iostats
11 0.1052 zap_pte
10 0.0956 bdget
10 0.0956 blk_release_queue
10 0.0956 do_page_fault
10 0.0956 elv_next_request
10 0.0956 sys_mq_open
9 0.0861 bdev_clear_inode
9 0.0861 generic_file_buffered_write
9 0.0861 locks_mandatory_area
9 0.0861 scsi_cmd_ioctl
8 0.0765 bd_claim_by_disk
8 0.0765 bd_release_from_disk
8 0.0765 dump_task_regs
8 0.0765 fcntl_setlk
7 0.0670 blk_cleanup_queue
7 0.0670 do_mpage_readpage
7 0.0670 loop_alloc
7 0.0670 unmap_vmas
7 0.0670 vfs_rename
6 0.0574 __end_that_request_first
6 0.0574 bio_split
6 0.0574 blkdev_close
6 0.0574 do_generic_mapping_read
6 0.0574 do_mremap
6 0.0574 generic_ide_ioctl
6 0.0574 pirq_piix_set
6 0.0574 prio_tree_insert
6 0.0574 test_clear_page_writeback
5 0.0478 __break_lease
5 0.0478 __switch_to_xtra
5 0.0478 daemonize
5 0.0478 dio_complete
5 0.0478 flush_old_exec
5 0.0478 handle_vm86_fault
5 0.0478 kmem_cache_create
5 0.0478 pipe_write
5 0.0478 pirq_enable_irq
5 0.0478 process_slab
5 0.0478 read_cache_page_async
5 0.0478 vfs_mkdir
4 0.0383 __blk_put_request
4 0.0383 add_to_page_cache
4 0.0383 cap_inode_removexattr
4 0.0383 cap_task_post_setuid
4 0.0383 dio_cleanup
4 0.0383 dio_get_page
4 0.0383 do_kern_mount
4 0.0383 do_mmap_pgoff
4 0.0383 do_notify_parent
4 0.0383 do_sendfile
4 0.0383 do_sysctl
4 0.0383 do_sysctl_strategy
4 0.0383 ext3_xattr_set_handle
4 0.0383 generic_write_checks
4 0.0383 idedisk_check_hpa
4 0.0383 interruptible_sleep_on_timeout
4 0.0383 log_do_checkpoint
4 0.0383 pcibios_scan_root
4 0.0383 print_bad_pte
4 0.0383 ptrace_request
4 0.0383 send_group_sigqueue
4 0.0383 set_using_dma
4 0.0383 sys_remap_file_pages
4 0.0383 vmtruncate
3 0.0287 __pte_alloc
3 0.0287 alloc_node_mem_map
3 0.0287 as_read_expire_store
3 0.0287 bitmap_find_free_region
3 0.0287 check_disk_change
3 0.0287 do_open
3 0.0287 do_sync
3 0.0287 do_sync_readv_writev
3 0.0287 early_serial_putc
3 0.0287 generic_ide_resume
3 0.0287 generic_shutdown_super
3 0.0287 ide_intr
3 0.0287 ide_register_hw_with_fixup
3 0.0287 install_file_pte
3 0.0287 install_page
3 0.0287 kobject_get_path
3 0.0287 ll_back_merge_fn
3 0.0287 lock_timer
3 0.0287 mpage_readpages
3 0.0287 on_freelist
3 0.0287 posix_cpu_nsleep
3 0.0287 rb_erase
3 0.0287 release_task
3 0.0287 rt_mutex_setprio
3 0.0287 sched_exit
3 0.0287 sg_scsi_ioctl
3 0.0287 show_partition
3 0.0287 show_schedstat
3 0.0287 sprint_symbol
3 0.0287 svc_seq_show
3 0.0287 sync_page_range
3 0.0287 sys_chown
3 0.0287 sys_mincore
3 0.0287 sys_mprotect
3 0.0287 sys_statfs
3 0.0287 sys_tee
3 0.0287 wake_up_new_task
2 0.0191 __elv_add_request
2 0.0191 __is_prefetch
2 0.0191 __register_chrdev_region
2 0.0191 account_steal_time
2 0.0191 apply_microcode
2 0.0191 as_antic_expire_store
2 0.0191 bd_forget
2 0.0191 bdput
2 0.0191 blk_init_queue_node
2 0.0191 blk_recount_segments
2 0.0191 blkdev_open
2 0.0191 calibrate_delay
2 0.0191 clear_page_dirty_for_io
2 0.0191 copy_page_range
2 0.0191 copy_process
2 0.0191 d_invalidate
2 0.0191 dentry_open
2 0.0191 dio_new_bio
2 0.0191 dio_send_cur_page
2 0.0191 do_brk
2 0.0191 do_msgrcv
2 0.0191 do_sync_read
2 0.0191 do_utimes
2 0.0191 dx_probe
2 0.0191 elv_insert
2 0.0191 elv_iosched_allow_merge
2 0.0191 elv_rb_del
2 0.0191 est_time_show
2 0.0191 exit_itimers
2 0.0191 ext3_free_inode
2 0.0191 ext3_mknod
2 0.0191 ext3_new_inode
2 0.0191 ext3_orphan_get
2 0.0191 filemap_nopage
2 0.0191 find_mergeable_anon_vma
2 0.0191 free_as_io_context
2 0.0191 freed_request
2 0.0191 get_signal_to_deliver
2 0.0191 get_timestamp
2 0.0191 grab_cache_page_nowait
2 0.0191 handle_stop_signal
2 0.0191 hrtimer_run_queues
2 0.0191 init_once
2 0.0191 inode_wait
2 0.0191 kobject_move
2 0.0191 kobject_shadow_rename
2 0.0191 load_elf_library
2 0.0191 lookup_address
2 0.0191 number
2 0.0191 open_namei
2 0.0191 pcibios_enable_device
2 0.0191 pcibios_fixup_bus
2 0.0191 poison_store
2 0.0191 prio_tree_next
2 0.0191 prio_tree_remove
2 0.0191 put_io_context
2 0.0191 read_port
2 0.0191 register_blkdev
2 0.0191 register_chrdev
2 0.0191 reiserfs_delete_xattrs
2 0.0191 reiserfs_xattr_init
2 0.0191 relay_destroy_buf
2 0.0191 request_irq
2 0.0191 rpc_proc_exit
2 0.0191 rpc_proc_init
2 0.0191 rpc_proc_open
2 0.0191 rw_copy_check_uvector
2 0.0191 sb_set_blocksize
2 0.0191 sched_setscheduler
2 0.0191 simple_strtoul
2 0.0191 skge_probe
2 0.0191 slab_pad_check
2 0.0191 strcspn
2 0.0191 sys_chdir
2 0.0191 sys_faccessat
2 0.0191 sys_fchdir
2 0.0191 sys_mq_timedreceive
2 0.0191 sys_mq_timedsend
2 0.0191 sys_openat
2 0.0191 sys_sync_file_range
2 0.0191 t_start
2 0.0191 tick_notify
2 0.0191 timer_list_show
2 0.0191 try_to_wake_up
2 0.0191 vfs_mknod
2 0.0191 vgacon_startup
2 0.0191 vm_normal_page
2 0.0191 vmtruncate_range
2 0.0191 zone_watermark_ok
1 0.0096 __activate_task
1 0.0096 __blockdev_direct_IO
1 0.0096 __call_usermodehelper
1 0.0096 __cleanup_sighand
1 0.0096 __d_lookup
1 0.0096 __ext3_get_inode_loc
1 0.0096 __free_pages_ok
1 0.0096 __free_slab
1 0.0096 __generic_file_aio_write_nolock
1 0.0096 __journal_drop_transaction
1 0.0096 __link_path_walk
1 0.0096 __posix_lock_file
1 0.0096 __ptrace_link
1 0.0096 __remove_shared_vm_struct
1 0.0096 __round_jiffies_relative
1 0.0096 __sigqueue_alloc
1 0.0096 __vma_link_rb
1 0.0096 __vmalloc_area_node
1 0.0096 __wait_on_freeing_inode
1 0.0096 __wake_up_common
1 0.0096 arch_ptrace
1 0.0096 arp_ioctl
1 0.0096 as_can_break_anticipation
1 0.0096 as_choose_req
1 0.0096 athlon_setup_ctrs
1 0.0096 background_writeout
1 0.0096 badness
1 0.0096 bd_get_sb
1 0.0096 bio_alloc_bioset
1 0.0096 blk_execute_rq_nowait
1 0.0096 blk_hw_contig_segment
1 0.0096 blk_ordered_complete_seq
1 0.0096 blk_ordered_cur_seq
1 0.0096 blk_ordered_req_seq
1 0.0096 blk_queue_find_tag
1 0.0096 blk_queue_make_request
1 0.0096 blk_queue_resize_tags
1 0.0096 blk_sync_queue
1 0.0096 blkdev_get_block
1 0.0096 blkdev_get_blocks
1 0.0096 block_fsync
1 0.0096 block_invalidatepage
1 0.0096 block_uevent_filter
1 0.0096 call_filldir
1 0.0096 call_usermodehelper_pipe
1 0.0096 cap_bprm_apply_creds
1 0.0096 cap_bprm_secureexec
1 0.0096 cap_capset_check
1 0.0096 cap_inode_setxattr
1 0.0096 cap_settime
1 0.0096 cap_vm_enough_memory
1 0.0096 cdev_get
1 0.0096 chrdev_show
1 0.0096 clockevents_set_mode
1 0.0096 common_timer_get
1 0.0096 complete_all
1 0.0096 congestion_wait_interruptible
1 0.0096 cp_old_stat
1 0.0096 cpu_idle
1 0.0096 current_io_context
1 0.0096 d_rehash
1 0.0096 deactivate_slab
1 0.0096 dev_kfree_skb_any
1 0.0096 discard_slab
1 0.0096 dma_declare_coherent_memory
1 0.0096 do_fcntl
1 0.0096 do_getitimer
1 0.0096 do_lookup
1 0.0096 do_munmap
1 0.0096 do_sched_setscheduler
1 0.0096 do_tkill
1 0.0096 do_wait
1 0.0096 drive_stat_acct
1 0.0096 dupfd
1 0.0096 elevator_alloc
1 0.0096 elevator_init
1 0.0096 elf_core_dump
1 0.0096 elv_attr_store
1 0.0096 elv_completed_request
1 0.0096 elv_unregister
1 0.0096 end_buffer_async_write
1 0.0096 expand_files
1 0.0096 expand_stack
1 0.0096 ext3_count_dirs
1 0.0096 ext3_find_entry
1 0.0096 ext3_free_blocks_sb
1 0.0096 ext3_getblk
1 0.0096 ext3_group_add
1 0.0096 ext3_new_blocks
1 0.0096 ext3_read_inode
1 0.0096 ext3_reserve_inode_write
1 0.0096 ext3_setattr
1 0.0096 ext3_symlink
1 0.0096 ext3_try_to_allocate
1 0.0096 ext3_xattr_block_set
1 0.0096 ext3_xattr_get
1 0.0096 ext3_xattr_trusted_list
1 0.0096 ext3_xattr_user_set
1 0.0096 file_send_actor
1 0.0096 filemap_fdatawait
1 0.0096 find_extend_vma
1 0.0096 follow_up
1 0.0096 force_sig_info_fault
1 0.0096 frag_show
1 0.0096 free_compound_page
1 0.0096 free_pgd_range
1 0.0096 generic_drop_inode
1 0.0096 generic_permission
1 0.0096 generic_unplug_device
1 0.0096 get_request_wait
1 0.0096 get_symbol_pos
1 0.0096 getname
1 0.0096 ide_hwif_request_regions
1 0.0096 ide_mm_inb
1 0.0096 ide_pci_setup_ports
1 0.0096 ide_setup_ports
1 0.0096 init_idedisk_capacity
1 0.0096 init_object
1 0.0096 init_request_from_bio
1 0.0096 init_tag_map
1 0.0096 inode_sub_bytes
1 0.0096 invalidate_bh_lrus
1 0.0096 ioctl_by_bdev
1 0.0096 ip_fragment
1 0.0096 iput
1 0.0096 it_real_fn
1 0.0096 itimer_get_remtime
1 0.0096 journal_commit_transaction
1 0.0096 journal_recover
1 0.0096 journal_refile_buffer
1 0.0096 kill_anon_super
1 0.0096 kill_litter_super
1 0.0096 kmem_cache_destroy
1 0.0096 kobject_shadow_add
1 0.0096 ll_rw_block
1 0.0096 lo_ioctl
1 0.0096 lookup_bdev
1 0.0096 may_attach
1 0.0096 may_open
1 0.0096 mb_cache_create
1 0.0096 mb_cache_entry_insert
1 0.0096 microcode_write
1 0.0096 modify_acceptable_latency
1 0.0096 mounts_release
1 0.0096 mpage_end_io_write
1 0.0096 neigh_resolve_output
1 0.0096 netif_rx
1 0.0096 nobh_prepare_write
1 0.0096 notify_arch_cmos_timer
1 0.0096 open_by_devnum
1 0.0096 p4_fill_in_addresses
1 0.0096 page_address_in_vma
1 0.0096 pci_write
1 0.0096 pcibios_assign_all_busses
1 0.0096 pcibios_disable_device
1 0.0096 pdflush
1 0.0096 pipe_read_open
1 0.0096 pirq_serverworks_set
1 0.0096 poll_idle
1 0.0096 posix_cpu_timer_del
1 0.0096 posix_timer_event
1 0.0096 prepare_to_wait_exclusive
1 0.0096 proc_dodebug
1 0.0096 proc_pid_auxv
1 0.0096 proc_task_lookup
1 0.0096 ptrace_attach
1 0.0096 ptrace_writedata
1 0.0096 raw_bind
1 0.0096 read_chan
1 0.0096 register_chrdev_region
1 0.0096 reiserfs_listxattr
1 0.0096 relay_file_mmap
1 0.0096 relay_file_open
1 0.0096 rpc_mkdir
1 0.0096 rpc_proc_show
1 0.0096 rt_check_expire
1 0.0096 run_local_timers
1 0.0096 run_posix_cpu_timers
1 0.0096 sb_min_blocksize
1 0.0096 schedule_tail
1 0.0096 set_blocksize
1 0.0096 set_ksettings
1 0.0096 set_load_weight
1 0.0096 setup_arg_pages
1 0.0096 setup_irq
1 0.0096 sg_io
1 0.0096 show_regs
1 0.0096 show_stat
1 0.0096 show_state_filter
1 0.0096 shrink_zone
1 0.0096 skb_checksum
1 0.0096 skge_get_coalesce
1 0.0096 skge_poll
1 0.0096 skge_set_coalesce
1 0.0096 sock_aio_write
1 0.0096 strnicmp
1 0.0096 svc_proc_register
1 0.0096 sys_chroot
1 0.0096 sys_fchmodat
1 0.0096 sys_get_thread_area
1 0.0096 sys_getcwd
1 0.0096 sys_mknodat
1 0.0096 sys_mlockall
1 0.0096 sys_nice
1 0.0096 sys_renameat
1 0.0096 sys_sched_get_priority_min
1 0.0096 sys_sendfile
1 0.0096 sys_sendfile64
1 0.0096 sys_timer_settime
1 0.0096 sys_uselib
1 0.0096 sysctl_head_next
1 0.0096 sysfs_follow_link
1 0.0096 sysfs_slab_add
1 0.0096 tcp_enter_quickack_mode
1 0.0096 tcp_v4_connect
1 0.0096 tcp_v4_hash
1 0.0096 throttle_vm_writeout
1 0.0096 trace
1 0.0096 udp_lib_unhash
1 0.0096 udp_proc_unregister
1 0.0096 uevent_helper_store
1 0.0096 unmap_mapping_range_vma
1 0.0096 unuse_table
1 0.0096 user_shm_unlock
1 0.0096 validate_store
1 0.0096 vfs_ioctl
1 0.0096 vgacon_scrolldelta
1 0.0096 vm_stat_account
1 0.0096 vma_adjust
1 0.0096 vprintk
1 0.0096 wait_for_helper
1 0.0096 wait_for_partner
1 0.0096 wait_on_retry_sync_kiocb
1 0.0096 wq_sleep
1 0.0096 write_boundary_block
1 0.0096 write_cache_pages
+ date
Fri Aug 17 20:17:53 EEST 2007

[-- Attachment #6: oprof_two_disks.txt --]
[-- Type: text/plain, Size: 14877 bytes --]

+ date
Fri Aug 17 20:22:29 EEST 2007
+ rm -rf /var/lib/oprofile/
+ opcontrol --vmlinux=/home/jimis/dist/src/linux-2.6.22.3/vmlinux
+ opcontrol --start
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
+ sleep 5
+ opcontrol --shutdown
Stopping profiling.
Killing daemon.
+ echo

+ echo

+ echo

+ opreport
CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
2528 52.9535 vmlinux
874 18.3075 libc-2.6.1.so
759 15.8986 ld-2.6.1.so
358 7.4990 bash
CPU_CLK_UNHALT...|
samples| %|
------------------
357 99.7207 bash
1 0.2793 [vdso] (tgid:13041 range:0xb7f12000-0xb7f13000)
109 2.2832 gawk
68 1.4244 ISO8859-1.so
13 0.2723 grep
10 0.2095 locale-archive
7 0.1466 badblocks
CPU_CLK_UNHALT...|
samples| %|
------------------
5 71.4286 badblocks
2 28.5714 [vdso] (tgid:10910 range:0xb7fc6000-0xb7fc7000)
7 0.1466 libnetsnmp.so.15.0.0
7 0.1466 imap-login
5 0.1047 libdl-2.6.1.so
4 0.0838 libncurses.so.5.6
3 0.0628 ophelp
3 0.0628 libcrypto.so.0.9.8
3 0.0628 libpopt.so.0.0.0
3 0.0628 dovecot-auth
2 0.0419 libhistory.so.5.2
2 0.0419 libnetsnmpmibs.so.15.0.0
1 0.0209 ls
1 0.0209 libext2fs.so.2.4
1 0.0209 libreadline.so.5.2
1 0.0209 init
1 0.0209 id
1 0.0209 oprofiled
1 0.0209 screen-4.0.3
1 0.0209 libnetsnmpagent.so.15.0.0
1 0.0209 sshd
CPU_CLK_UNHALT...|
samples| %|
------------------
1 100.000 [vdso] (tgid:6746 range:0xb7f31000-0xb7f32000)
+ echo

+ echo

+ echo

+ opreport -l /home/jimis/dist/src/linux-2.6.22.3/vmlinux
CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
227 8.9794 rpc_print_iostats
206 8.1487 pre_reset
175 6.9225 kmem_cache_shrink
160 6.3291 irq_handler
112 4.4304 congestion_wait
100 3.9557 rpc_proc_open
79 3.1250 create_kmalloc_cache
76 3.0063 ide_do_request
72 2.8481 dump_task_extended_fpu
64 2.5316 vsscanf
56 2.2152 dump_task_regs
52 2.0570 do_syscall_trace
41 1.6218 interruptible_sleep_on
39 1.5427 __relay_reset
32 1.2658 do_wp_page
31 1.2263 kobject_rename
29 1.1472 sys_madvise
27 1.0680 __switch_to
24 0.9494 print_hex_dump
22 0.8703 test_set_page_writeback
20 0.7911 check_object
19 0.7516 do_ide_setup_pci_device
18 0.7120 sys_fadvise64_64
17 0.6725 rpc_proc_show
15 0.5934 ide_setup_pci_devices
14 0.5538 __handle_mm_fault
14 0.5538 handle_vm86_fault
14 0.5538 iput
13 0.5142 svc_seq_show
12 0.4747 do_page_fault
12 0.4747 vsnprintf
11 0.4351 calibrate_delay
11 0.4351 generic_file_buffered_write
11 0.4351 zap_pte
11 0.4351 zoneinfo_show
10 0.3956 access_process_vm
10 0.3956 vm_normal_page
9 0.3560 blk_release_queue
9 0.3560 block_truncate_page
9 0.3560 kmem_cache_create
9 0.3560 setup_sigcontext
8 0.3165 dio_get_page
7 0.2769 bdev_clear_inode
7 0.2769 elv_next_request
7 0.2769 loop_alloc
7 0.2769 print_bad_pte
7 0.2769 rpc_proc_exit
7 0.2769 unmap_vmas
7 0.2769 write_cache_pages
6 0.2373 __switch_to_xtra
6 0.2373 bio_split
6 0.2373 load_elf_binary
6 0.2373 on_freelist
6 0.2373 sys_mprotect
5 0.1978 __blk_put_request
5 0.1978 __blkdev_put
5 0.1978 bdget
5 0.1978 blk_cleanup_queue
5 0.1978 cont_prepare_write
5 0.1978 do_open
5 0.1978 do_sysctl_strategy
5 0.1978 idedisk_check_hpa
5 0.1978 lock_timer
5 0.1978 posix_cpu_nsleep
5 0.1978 sys_mq_timedsend
5 0.1978 test_clear_page_writeback
4 0.1582 __pte_alloc
4 0.1582 __setlease
4 0.1582 assign_all_busses
4 0.1582 bd_release_from_disk
4 0.1582 blk_get_request
4 0.1582 cdev_get
4 0.1582 dio_cleanup
4 0.1582 generic_ide_ioctl
4 0.1582 ioctl_by_bdev
4 0.1582 lookup_bdev
4 0.1582 prio_tree_insert
4 0.1582 relay_switch_subbuf
4 0.1582 rpc_count_iostats
4 0.1582 rpc_proc_init
4 0.1582 sched_setscheduler
4 0.1582 scsi_cmd_ioctl
4 0.1582 svc_proc_unregister
4 0.1582 sys_chroot
4 0.1582 sys_mincore
4 0.1582 sys_mq_open
4 0.1582 wake_up_new_task
3 0.1187 as_can_break_anticipation
3 0.1187 blk_init_queue_node
3 0.1187 blk_phys_contig_segment
3 0.1187 blkdev_close
3 0.1187 cap_bprm_apply_creds
3 0.1187 dio_new_bio
3 0.1187 do_sendfile
3 0.1187 do_sync_readv_writev
3 0.1187 do_sysctl
3 0.1187 elf_core_dump
3 0.1187 fcntl_setlk
3 0.1187 filemap_nopage
3 0.1187 follow_page
3 0.1187 generic_unplug_device
3 0.1187 locks_mandatory_area
3 0.1187 pcibios_setup
3 0.1187 pipe_write
3 0.1187 prio_tree_next
3 0.1187 proc_dodebug
3 0.1187 register_chrdev
3 0.1187 remap_pfn_range
3 0.1187 rw_copy_check_uvector
3 0.1187 send_group_sigqueue
3 0.1187 sys_fchmodat
3 0.1187 sys_remap_file_pages
3 0.1187 vfs_mkdir
3 0.1187 vfs_mknod
3 0.1187 vma_adjust
3 0.1187 vma_merge
2 0.0791 __end_that_request_first
2 0.0791 __free_slab
2 0.0791 add_to_page_cache_lru
2 0.0791 as_fifo_expired
2 0.0791 bd_claim_by_disk
2 0.0791 bio_pair_end_2
2 0.0791 blk_alloc_queue_node
2 0.0791 blk_done_softirq
2 0.0791 blk_ordered_req_seq
2 0.0791 blk_sync_queue
2 0.0791 calculate_totalreserve_pages
2 0.0791 copy_page_range
2 0.0791 dentry_open
2 0.0791 do_exit
2 0.0791 do_mmap_pgoff
2 0.0791 do_mremap
2 0.0791 do_msgrcv
2 0.0791 do_munmap
2 0.0791 do_notify_parent
2 0.0791 dump_thread
2 0.0791 elv_rb_del
2 0.0791 ext3_count_dirs
2 0.0791 ext3_free_inode
2 0.0791 ext3_new_inode
2 0.0791 flush_old_exec
2 0.0791 force_sig_info_fault
2 0.0791 generic_shutdown_super
2 0.0791 grab_cache_page_nowait
2 0.0791 handle_stop_signal
2 0.0791 ide_hwif_request_regions
2 0.0791 init_cpu_workqueue
2 0.0791 init_idedisk_capacity
2 0.0791 insert_vm_struct
2 0.0791 install_file_pte
2 0.0791 interruptible_sleep_on_timeout
2 0.0791 ll_back_merge_fn
2 0.0791 page_address_in_vma
2 0.0791 pirq_enable_irq
2 0.0791 proc_task_lookup
2 0.0791 read_port
2 0.0791 relay_file_read_consume
2 0.0791 rt_mutex_setprio
2 0.0791 rtc_cmos_read
2 0.0791 sg_scsi_ioctl
2 0.0791 subbuf_send_actor
2 0.0791 sync_page_range
2 0.0791 sys_chdir
2 0.0791 sys_chown
2 0.0791 sys_mlockall
2 0.0791 sys_sendfile
2 0.0791 sys_setfsuid
2 0.0791 sys_sysinfo
2 0.0791 t_start
2 0.0791 tick_notify
2 0.0791 vfs_rename
2 0.0791 vfs_unlink
2 0.0791 vgacon_startup
2 0.0791 vmtruncate
1 0.0396 __blk_free_tags
1 0.0396 __d_find_alias
1 0.0396 __elv_add_request
1 0.0396 __f_setown
1 0.0396 __filemap_copy_from_user_iovec_inatomic
1 0.0396 __group_complete_signal
1 0.0396 __is_prefetch
1 0.0396 __link_path_walk
1 0.0396 __netif_rx_schedule
1 0.0396 __oom_kill_task
1 0.0396 __pte_alloc_kernel
1 0.0396 __put_user_1
1 0.0396 __remove_suid
1 0.0396 __set_page_dirty_buffers
1 0.0396 __set_special_pids
1 0.0396 __strncpy_from_user
1 0.0396 __vmalloc_area_node
1 0.0396 __wait_on_freeing_inode
1 0.0396 __wake_up_common
1 0.0396 alloc_chrdev_region
1 0.0396 apply_microcode
1 0.0396 arch_align_stack
1 0.0396 as_antic_stop
1 0.0396 as_choose_req
1 0.0396 as_read_batch_expire_show
1 0.0396 as_read_expire_store
1 0.0396 as_update_iohist
1 0.0396 badness
1 0.0396 blk_execute_rq
1 0.0396 blk_execute_rq_nowait
1 0.0396 blk_free_tags
1 0.0396 blk_hw_contig_segment
1 0.0396 blk_plug_device
1 0.0396 blk_queue_find_tag
1 0.0396 blk_queue_hardsect_size
1 0.0396 blk_register_queue
1 0.0396 blk_remove_plug
1 0.0396 blkdev_direct_IO
1 0.0396 blkdev_get
1 0.0396 blkdev_open
1 0.0396 block_fsync
1 0.0396 call_usermodehelper_pipe
1 0.0396 can_do_mlock
1 0.0396 cap_ptrace
1 0.0396 cap_settime
1 0.0396 cap_task_post_setuid
1 0.0396 cascade
1 0.0396 cdev_del
1 0.0396 check_disk_change
1 0.0396 check_slab
1 0.0396 congestion_wait_interruptible
1 0.0396 cpu_idle_wait
1 0.0396 create_new_namespaces
1 0.0396 current_io_context
1 0.0396 d_alloc
1 0.0396 d_hash_and_lookup
1 0.0396 del_timer
1 0.0396 dev_hard_start_xmit
1 0.0396 dio_complete
1 0.0396 do_brk
1 0.0396 do_fcntl
1 0.0396 do_generic_mapping_read
1 0.0396 do_iret_error
1 0.0396 do_mpage_readpage
1 0.0396 do_sched_setscheduler
1 0.0396 do_sync
1 0.0396 do_writepages
1 0.0396 dup_fd
1 0.0396 elevator_alloc
1 0.0396 elv_completed_request
1 0.0396 elv_insert
1 0.0396 elv_merged_request
1 0.0396 elv_requeue_request
1 0.0396 elv_unregister
1 0.0396 end_buffer_read_sync
1 0.0396 exit_itimers
1 0.0396 ext3_bread
1 0.0396 ext3_rename
1 0.0396 ext3_xattr_trusted_list
1 0.0396 free_pgtables
1 0.0396 freed_request
1 0.0396 generic_drop_inode
1 0.0396 generic_file_sendfile
1 0.0396 generic_fillattr
1 0.0396 generic_ide_resume
1 0.0396 generic_ide_suspend
1 0.0396 generic_permission
1 0.0396 get_next_timer_interrupt
1 0.0396 get_symbol_offset
1 0.0396 get_timestamp
1 0.0396 getname
1 0.0396 ide_intr
1 0.0396 ide_pci_setup_ports
1 0.0396 ide_setup_pci_device
1 0.0396 ide_taskfile_ioctl
1 0.0396 init_object
1 0.0396 init_once
1 0.0396 internal_add_timer
1 0.0396 ip_options_get_from_user
1 0.0396 journal_add_journal_head
1 0.0396 kill_anon_super
1 0.0396 kmem_ptr_validate
1 0.0396 kobject_get_path
1 0.0396 kobject_move
1 0.0396 laptop_io_completion
1 0.0396 link_path_walk
1 0.0396 lock_task_sighand
1 0.0396 log_do_checkpoint
1 0.0396 lookup_one_len
1 0.0396 madvise_need_mmap_write
1 0.0396 may_open
1 0.0396 mb_cache_create
1 0.0396 microcode_write
1 0.0396 mm_release
1 0.0396 mounts_release
1 0.0396 mpage_readpages
1 0.0396 mqueue_read_file
1 0.0396 netstat_show
1 0.0396 nobh_prepare_write
1 0.0396 normalize_rt_tasks
1 0.0396 notify_arch_cmos_timer
1 0.0396 nr_processes
1 0.0396 number
1 0.0396 open_bdev_excl
1 0.0396 open_namei
1 0.0396 out_of_memory
1 0.0396 p4_start
1 0.0396 page_cache_read
1 0.0396 page_mkclean
1 0.0396 pcibios_disable_device
1 0.0396 pdflush
1 0.0396 pipe_poll
1 0.0396 poison_store
1 0.0396 posix_cpu_nsleep_restart
1 0.0396 prepare_timeout
1 0.0396 prio_tree_remove
1 0.0396 proc_pid_auxv
1 0.0396 profile_event_unregister
1 0.0396 profile_hits
1 0.0396 ptmx_open
1 0.0396 ptrace_attach
1 0.0396 read_cache_page_async
1 0.0396 read_profile
1 0.0396 recalc_task_prio
1 0.0396 register_console
1 0.0396 relay_file_mmap
1 0.0396 relay_file_open
1 0.0396 relay_file_poll
1 0.0396 release_task
1 0.0396 reparent_thread
1 0.0396 request_irq
1 0.0396 rpc_alloc_iostats
1 0.0396 rt_fill_info
1 0.0396 run_posix_cpu_timers
1 0.0396 rw_verify_area
1 0.0396 sb_set_blocksize
1 0.0396 sched_exit
1 0.0396 sched_fork
1 0.0396 send_signal
1 0.0396 set_bdi_congested
1 0.0396 set_blocksize
1 0.0396 set_load_weight
1 0.0396 set_page_dirty_lock
1 0.0396 setup_irq
1 0.0396 sg_io
1 0.0396 show_partition
1 0.0396 show_regs
1 0.0396 show_schedstat
1 0.0396 shrink_zone
1 0.0396 strpbrk
1 0.0396 subbuf_read_actor
1 0.0396 svc_proc_register
1 0.0396 sys_faccessat
1 0.0396 sys_io_getevents
1 0.0396 sys_mlock
1 0.0396 sys_munlock
1 0.0396 sys_sched_get_priority_min
1 0.0396 sys_setitimer
1 0.0396 sys_setregid
1 0.0396 sys_sigaction
1 0.0396 sys_splice
1 0.0396 sys_stat
1 0.0396 sys_statfs
1 0.0396 sys_sync_file_range
1 0.0396 sys_sysctl
1 0.0396 sys_vm86old
1 0.0396 sysctl_head_finish
1 0.0396 sysctl_head_next
1 0.0396 try_to_free_buffers
1 0.0396 unuse_table
1 0.0396 user_shm_lock
1 0.0396 validate_store
1 0.0396 vfs_ioctl
1 0.0396 vm_stat_account
1 0.0396 vma_link
1 0.0396 vprintk
1 0.0396 wait_on_retry_sync_kiocb
1 0.0396 wait_on_work
1 0.0396 wake_up_inode
1 0.0396 will_become_orphaned_pgrp
1 0.0396 write_boundary_block
1 0.0396 write_full
+ date
Fri Aug 17 20:23:34 EEST 2007

[-- Attachment #7: oprof_two_disks_bad2.txt --]
[-- Type: text/plain, Size: 17366 bytes --]

+ date
Sat Aug 18 00:13:48 EEST 2007
+ rm -rf /var/lib/oprofile/
+ opcontrol --vmlinux=/home/jimis/dist/src/linux-2.6.22.3/vmlinux
+ opcontrol --start
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
+ sleep 5
+ opcontrol --shutdown
Stopping profiling.
Killing daemon.
+ echo

+ echo

+ echo

+ opreport
CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
3020 34.9902 vmlinux
1920 22.2454 libc-2.6.1.so
1274 14.7607 libpython2.5.so.1.0
1140 13.2082 perl
432 5.0052 mpop
CPU_CLK_UNHALT...|
samples| %|
------------------
430 99.5370 mpop
2 0.4630 [vdso] (tgid:16432 range:0xb7f24000-0xb7f25000)
253 2.9313 bash
CPU_CLK_UNHALT...|
samples| %|
------------------
252 99.6047 bash
1 0.3953 [vdso] (tgid:16796 range:0xb7fbe000-0xb7fbf000)
229 2.6532 ld-2.6.1.so
105 1.2165 libgnutls.so.13.3.0
49 0.5677 ISO8859-1.so
46 0.5330 libgcrypt.so.11.2.3
38 0.4403 libpthread-2.6.1.so
27 0.3128 badblocks
CPU_CLK_UNHALT...|
samples| %|
------------------
17 62.9630 badblocks
8 29.6296 [vdso] (tgid:16297 range:0xb7fb6000-0xb7fb7000)
2 7.4074 [vdso] (tgid:16298 range:0xb7f62000-0xb7f63000)
20 0.2317 screen-4.0.3
CPU_CLK_UNHALT...|
samples| %|
------------------
19 95.0000 screen-4.0.3
1 5.0000 [vdso] (tgid:16282 range:0xb7f25000-0xb7f26000)
11 0.1274 slocate
CPU_CLK_UNHALT...|
samples| %|
------------------
7 63.6364 slocate
4 36.3636 [vdso] (tgid:16652 range:0xb7efe000-0xb7eff000)
10 0.1159 imap-login
CPU_CLK_UNHALT...|
samples| %|
------------------
8 80.0000 imap-login
2 20.0000 [vdso] (tgid:15896 range:0xb7ef9000-0xb7efa000)
9 0.1043 libncurses.so.5.6
8 0.0927 gawk
7 0.0811 grep
6 0.0695 python2.5
CPU_CLK_UNHALT...|
samples| %|
------------------
5 83.3333 [vdso] (tgid:16627 range:0xb7efd000-0xb7efe000)
1 16.6667 [vdso] (tgid:16686 range:0xb7fa7000-0xb7fa8000)
6 0.0695 libnetsnmp.so.15.0.0
5 0.0579 libext2fs.so.2.4
3 0.0348 dovecot
CPU_CLK_UNHALT...|
samples| %|
------------------
2 66.6667 dovecot
1 33.3333 [vdso] (tgid:1923 range:0xb7ef7000-0xb7ef8000)
3 0.0348 sshd
CPU_CLK_UNHALT...|
samples| %|
------------------
2 66.6667 sshd
1 33.3333 [vdso] (tgid:15858 range:0xb7f64000-0xb7f65000)
2 0.0232 libcrypto.so.0.9.8
1 0.0116 ls
1 0.0116 tr
1 0.0116 libpcre.so.0.0.1
1 0.0116 which
1 0.0116 libnetsnmpagent.so.15.0.0
1 0.0116 libnetsnmpmibs.so.15.0.0
1 0.0116 locale-archive
1 0.0116 dovecot-auth
+ echo

+ echo

+ echo

+ opreport -l /home/jimis/dist/src/linux-2.6.22.3/vmlinux
CPU: PIII, speed 798.02 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
282 9.3377 pre_reset
231 7.6490 rpc_print_iostats
222 7.3510 do_syscall_trace
146 4.8344 ide_do_request
144 4.7682 dump_task_regs
131 4.3377 rpc_proc_open
122 4.0397 congestion_wait
98 3.2450 vsscanf
52 1.7219 __switch_to
33 1.0927 interruptible_sleep_on
32 1.0596 check_object
32 1.0596 setup_sigcontext
32 1.0596 sys_sigaction
31 1.0265 block_truncate_page
31 1.0265 sys_fadvise64_64
31 1.0265 test_set_page_writeback
28 0.9272 __blkdev_put
28 0.9272 sys_madvise
26 0.8609 svc_seq_show
22 0.7285 do_wp_page
22 0.7285 zap_pte
21 0.6954 dump_task_extended_fpu
21 0.6954 elv_next_request
21 0.6954 generic_file_buffered_write
21 0.6954 unmap_vmas
20 0.6623 zoneinfo_show
19 0.6291 __relay_reset
19 0.6291 cont_prepare_write
18 0.5960 bdget
18 0.5960 calibrate_delay
18 0.5960 do_page_fault
17 0.5629 ide_setup_pci_devices
17 0.5629 scsi_cmd_ioctl
17 0.5629 vsnprintf
16 0.5298 bd_release_from_disk
16 0.5298 bdev_clear_inode
16 0.5298 do_ide_setup_pci_device
15 0.4967 kobject_rename
14 0.4636 blkdev_close
14 0.4636 print_hex_dump
13 0.4305 access_process_vm
13 0.4305 do_open
12 0.3974 bio_split
12 0.3974 blk_release_queue
12 0.3974 do_sendfile
12 0.3974 handle_vm86_fault
11 0.3642 __handle_mm_fault
11 0.3642 bd_claim_by_disk
11 0.3642 rpc_proc_show
10 0.3311 __switch_to_xtra
10 0.3311 posix_cpu_nsleep
9 0.2980 do_generic_mapping_read
9 0.2980 install_file_pte
9 0.2980 sg_scsi_ioctl
9 0.2980 vgacon_startup
8 0.2649 do_mpage_readpage
8 0.2649 prio_tree_insert
8 0.2649 test_clear_page_writeback
7 0.2318 init_once
7 0.2318 locks_mandatory_area
7 0.2318 loop_alloc
7 0.2318 register_blkdev
7 0.2318 rt_mutex_setprio
6 0.1987 daemonize
6 0.1987 idedisk_check_hpa
6 0.1987 pcibios_setup
6 0.1987 ptrace_request
6 0.1987 shrink_zone
6 0.1987 sys_chdir
6 0.1987 sys_mq_open
5 0.1656 as_update_iohist
5 0.1656 blk_cleanup_queue
5 0.1656 blkdev_direct_IO
5 0.1656 current_io_context
5 0.1656 do_sysctl_strategy
5 0.1656 ext3_xattr_list
5 0.1656 ide_setup_ports
5 0.1656 igrab
5 0.1656 on_freelist
5 0.1656 register_chrdev
5 0.1656 show_partition
5 0.1656 vfs_rename
5 0.1656 vgacon_scrolldelta
5 0.1656 vm_normal_page
5 0.1656 vmtruncate
5 0.1656 write_cache_pages
4 0.1325 __free_slab
4 0.1325 __generic_file_aio_write_nolock
4 0.1325 __make_request
4 0.1325 bd_forget
4 0.1325 blk_queue_make_request
4 0.1325 dio_get_page
4 0.1325 dio_new_bio
4 0.1325 do_exit
4 0.1325 do_fcntl
4 0.1325 elv_insert
4 0.1325 generic_file_sendfile
4 0.1325 generic_ide_suspend
4 0.1325 generic_segment_checks
4 0.1325 init_idedisk_capacity
4 0.1325 kill_litter_super
4 0.1325 load_elf_binary
4 0.1325 lock_timer
4 0.1325 number
4 0.1325 page_cache_read
4 0.1325 prio_tree_next
4 0.1325 process_slab
4 0.1325 read_cache_page_async
4 0.1325 setup_irq
4 0.1325 sg_io
4 0.1325 sync_page_range
3 0.0993 __break_lease
3 0.0993 __end_that_request_first
3 0.0993 add_to_page_cache
3 0.0993 athlon_setup_ctrs
3 0.0993 blk_execute_rq_nowait
3 0.0993 blk_hw_contig_segment
3 0.0993 blk_ordered_req_seq
3 0.0993 blk_queue_find_tag
3 0.0993 blk_rq_map_kern
3 0.0993 cap_bprm_apply_creds
3 0.0993 cap_settime
3 0.0993 cap_task_post_setuid
3 0.0993 check_disk_change
3 0.0993 check_slab
3 0.0993 d_invalidate
3 0.0993 dio_cleanup
3 0.0993 dio_complete
3 0.0993 do_mremap
3 0.0993 do_sync_write
3 0.0993 do_wait
3 0.0993 elv_completed_request
3 0.0993 est_time_show
3 0.0993 ext3_xattr_set_handle
3 0.0993 fcntl_setlk
3 0.0993 generic_ide_ioctl
3 0.0993 ioctl_by_bdev
3 0.0993 iput
3 0.0993 ll_back_merge_fn
3 0.0993 lookup_bdev
3 0.0993 open_by_devnum
3 0.0993 page_address_in_vma
3 0.0993 pipe_write
3 0.0993 posix_cpu_nsleep_restart
3 0.0993 relay_file_read
3 0.0993 rpc_proc_init
3 0.0993 sb_min_blocksize
3 0.0993 sb_set_blocksize
3 0.0993 svc_proc_unregister
3 0.0993 sys_mprotect
3 0.0993 sys_mq_timedsend
3 0.0993 tcp_rcv_state_process
3 0.0993 vma_adjust
2 0.0662 __blk_free_tags
2 0.0662 __journal_abort_hard
2 0.0662 __journal_drop_transaction
2 0.0662 __pte_alloc
2 0.0662 __vmalloc_area_node
2 0.0662 add_to_page_cache_lru
2 0.0662 aio_complete
2 0.0662 alloc_node_mem_map
2 0.0662 as_can_break_anticipation
2 0.0662 as_choose_req
2 0.0662 as_read_batch_expire_store
2 0.0662 as_read_expire_store
2 0.0662 assign_all_busses
2 0.0662 background_writeout
2 0.0662 bdput
2 0.0662 bio_alloc_bioset
2 0.0662 bio_endio
2 0.0662 blk_alloc_queue_node
2 0.0662 blk_end_sync_rq
2 0.0662 blkdev_open
2 0.0662 block_uevent_filter
2 0.0662 cdev_del
2 0.0662 cdev_get
2 0.0662 copy_page_range
2 0.0662 dio_send_cur_page
2 0.0662 do_mmap_pgoff
2 0.0662 do_notify_parent
2 0.0662 do_sync_readv_writev
2 0.0662 early_serial_putc
2 0.0662 early_serial_write
2 0.0662 elevator_init
2 0.0662 elv_iosched_allow_merge
2 0.0662 elv_rq_merge_ok
2 0.0662 free_as_io_context
2 0.0662 generic_shutdown_super
2 0.0662 grab_cache_page_nowait
2 0.0662 hwif_request_region
2 0.0662 ide_pci_setup_ports
2 0.0662 ide_setup_pci_device
2 0.0662 ide_taskfile_ioctl
2 0.0662 idle_cpu
2 0.0662 journal_flush
2 0.0662 kmem_cache_create
2 0.0662 lookup_one_len
2 0.0662 mpage_readpages
2 0.0662 notify_arch_cmos_timer
2 0.0662 open_bdev_excl
2 0.0662 pirq_piix_set
2 0.0662 print_bad_pte
2 0.0662 prio_tree_remove
2 0.0662 proc_dodebug
2 0.0662 release_task
2 0.0662 rpc_proc_exit
2 0.0662 rtc_cmos_read
2 0.0662 send_sigio
2 0.0662 set_blocksize
2 0.0662 set_ksettings
2 0.0662 set_using_dma
2 0.0662 sha_transform
2 0.0662 show_schedstat
2 0.0662 sprint_symbol
2 0.0662 sys_faccessat
2 0.0662 sys_fchmodat
2 0.0662 sys_remap_file_pages
2 0.0662 sys_vm86old
2 0.0662 sysctl_head_next
2 0.0662 t_start
2 0.0662 throttle_vm_writeout
2 0.0662 vfs_ioctl
2 0.0662 vfs_mknod
2 0.0662 vfs_unlink
2 0.0662 vgacon_deinit
2 0.0662 vmalloc_sync_all
2 0.0662 wake_up_new_task
2 0.0662 write_boundary_block
2 0.0662 zone_watermark_ok
1 0.0331 __blk_put_request
1 0.0331 __blkdev_get
1 0.0331 __filemap_copy_from_user_iovec_inatomic
1 0.0331 __find_get_block
1 0.0331 __follow_mount
1 0.0331 __free_pages_ok
1 0.0331 __is_prefetch
1 0.0331 __netif_schedule
1 0.0331 __register_chrdev_region
1 0.0331 __remove_hrtimer
1 0.0331 __set_page_dirty_buffers
1 0.0331 __set_page_dirty_nobuffers
1 0.0331 add_timer_randomness
1 0.0331 arch_ptrace
1 0.0331 as_fifo_expired
1 0.0331 as_put_io_context
1 0.0331 as_trim
1 0.0331 badness
1 0.0331 bio_pair_end_2
1 0.0331 bitmap_find_free_region
1 0.0331 blk_free_tags
1 0.0331 blk_init_queue_node
1 0.0331 blk_ordered_cur_seq
1 0.0331 blk_queue_resize_tags
1 0.0331 blk_remove_plug
1 0.0331 blk_sync_queue
1 0.0331 blkdev_get_block
1 0.0331 calculate_totalreserve_pages
1 0.0331 cap_vm_enough_memory
1 0.0331 clocksource_watchdog
1 0.0331 complete
1 0.0331 complete_all
1 0.0331 congestion_wait_interruptible
1 0.0331 copy_process
1 0.0331 cpu_idle
1 0.0331 create_new_namespaces
1 0.0331 current_is_keventd
1 0.0331 dentry_open
1 0.0331 dma_declare_coherent_memory
1 0.0331 do_alignment_check
1 0.0331 do_coredump
1 0.0331 do_getitimer
1 0.0331 do_kern_mount
1 0.0331 do_munmap
1 0.0331 do_sched_setscheduler
1 0.0331 do_sync
1 0.0331 do_sync_read
1 0.0331 do_sys_poll
1 0.0331 do_sysctl
1 0.0331 do_syslog
1 0.0331 do_timer
1 0.0331 do_utimes
1 0.0331 drive_stat_acct
1 0.0331 dump_thread
1 0.0331 dup_fd
1 0.0331 elevator_alloc
1 0.0331 eligible_child
1 0.0331 elv_attr_store
1 0.0331 elv_rb_add
1 0.0331 elv_rb_del
1 0.0331 elv_unregister
1 0.0331 end_buffer_async_write
1 0.0331 expand_stack
1 0.0331 ext3_count_dirs
1 0.0331 ext3_new_blocks
1 0.0331 ext3_orphan_get
1 0.0331 ext3_rename
1 0.0331 ext3_xattr_block_set
1 0.0331 ext3_xattr_get
1 0.0331 ext3_xattr_set
1 0.0331 f_delown
1 0.0331 filemap_fdatawait
1 0.0331 filemap_nopage
1 0.0331 flush_old_exec
1 0.0331 flush_thread
1 0.0331 fn_hash_insert
1 0.0331 follow_mount
1 0.0331 force_sig_info_fault
1 0.0331 force_sigsegv
1 0.0331 frag_start
1 0.0331 free_fdtable_work
1 0.0331 freed_request
1 0.0331 generic_fillattr
1 0.0331 generic_ide_resume
1 0.0331 generic_permission
1 0.0331 generic_unplug_device
1 0.0331 get_request_wait
1 0.0331 get_signal_to_deliver
1 0.0331 get_symbol_offset
1 0.0331 ide_abort
1 0.0331 init_object
1 0.0331 init_tag_map
1 0.0331 inode_add_bytes
1 0.0331 insert_wq_barrier
1 0.0331 install_page
1 0.0331 interruptible_sleep_on_timeout
1 0.0331 ip_fragment
1 0.0331 itimer_get_remtime
1 0.0331 journal_start
1 0.0331 kill_anon_super
1 0.0331 kill_fasync
1 0.0331 kmem_ptr_validate
1 0.0331 kobject_register
1 0.0331 kobject_shadow_add
1 0.0331 kobject_uevent_env
1 0.0331 ktime_get_real
1 0.0331 link_path_walk
1 0.0331 lo_ioctl
1 0.0331 locks_insert_block
1 0.0331 log_do_checkpoint
1 0.0331 madvise_need_mmap_write
1 0.0331 memory_open
1 0.0331 mincore_page
1 0.0331 nobh_prepare_write
1 0.0331 normalize_rt_tasks
1 0.0331 out_of_memory
1 0.0331 page_mkclean
1 0.0331 pcibios_fixup_bus
1 0.0331 posix_cpu_timer_set
1 0.0331 posix_timer_event
1 0.0331 prepare_to_wait_exclusive
1 0.0331 prio_tree_left
1 0.0331 proc_pid_auxv
1 0.0331 proc_task_lookup
1 0.0331 profile_hits
1 0.0331 profile_task_exit
1 0.0331 ptrace_attach
1 0.0331 ptrace_detach
1 0.0331 ptrace_writedata
1 0.0331 put_io_context
1 0.0331 rb_first
1 0.0331 read_port
1 0.0331 read_profile
1 0.0331 red_zone_store
1 0.0331 register_posix_clock
1 0.0331 relay_file_mmap
1 0.0331 relay_file_open
1 0.0331 relay_file_read_consume
1 0.0331 relay_file_release
1 0.0331 release_console_sem
1 0.0331 reparent_thread
1 0.0331 request_irq
1 0.0331 rq_init
1 0.0331 run_local_timers
1 0.0331 run_posix_cpu_timers
1 0.0331 rw_copy_check_uvector
1 0.0331 rw_verify_area
1 0.0331 sched_exit
1 0.0331 sched_setscheduler
1 0.0331 send_group_sigqueue
1 0.0331 set_close_on_exec
1 0.0331 set_load_weight
1 0.0331 skge_set_coalesce
1 0.0331 sock_aio_write
1 0.0331 sock_sendmsg
1 0.0331 strcasecmp
1 0.0331 strncasecmp
1 0.0331 strstr
1 0.0331 subbuf_send_actor
1 0.0331 svc_proc_register
1 0.0331 sys_chroot
1 0.0331 sys_mincore
1 0.0331 sys_munlockall
1 0.0331 sys_openat
1 0.0331 sys_sendfile64
1 0.0331 sys_setfsuid
1 0.0331 sys_sysctl
1 0.0331 sys_tee
1 0.0331 sysfs_follow_link
1 0.0331 task_prio
1 0.0331 tcp_add_reno_sack
1 0.0331 timekeeping_resume
1 0.0331 try_acquire_console_sem
1 0.0331 uevent_helper_store
1 0.0331 unix_release_sock
1 0.0331 unregister_timer_hook
1 0.0331 unuse_table
1 0.0331 update_iter
1 0.0331 user_shm_lock
1 0.0331 vfs_mkdir
1 0.0331 vma_merge
1 0.0331 wait_on_page_writeback_range
1 0.0331 wait_on_retry_sync_kiocb
1 0.0331 wb_kupdate
+ date
Sat Aug 18 00:16:13 EEST 2007

[-- Attachment #8: vmstat_idle.txt --]
[-- Type: text/plain, Size: 944 bytes --]

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 138400 28520 68632 0 0 35 1854 143 96 20 3 72 5
0 0 0 138400 28520 68632 0 0 0 0 101 11 0 0 100 0
0 0 0 138400 28520 68632 0 0 0 0 103 13 0 0 100 0
0 0 0 138400 28520 68632 0 0 0 756 154 15 0 0 100 0
0 0 0 138400 28520 68632 0 0 0 0 106 13 0 0 100 0
0 0 0 138400 28520 68632 0 0 0 0 106 13 0 0 100 0
0 0 0 138400 28528 68632 0 0 0 52 111 21 0 0 100 0
0 0 0 138400 28528 68632 0 0 0 0 104 18 0 1 99 0
0 0 0 138400 28528 68632 0 0 0 0 102 18 0 0 100 0
0 0 0 138400 28528 68632 0 0 0 0 103 9 0 0 100 0

[-- Attachment #9: vmstat_one_disk.txt --]
[-- Type: text/plain, Size: 936 bytes --]

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 0 137068 29776 68644 0 0 28 2948 160 126 23 4 67 6
0 1 0 137068 29776 68644 0 0 0 21696 442 690 0 2 0 98
0 1 0 137068 29776 68644 0 0 0 21632 440 690 0 1 0 99
0 1 0 137068 29776 68644 0 0 0 21696 440 688 0 3 0 97
1 1 0 137068 29776 68644 0 0 0 21632 440 690 0 3 0 97
1 1 0 137068 29784 68644 0 0 0 21688 441 697 0 2 0 98
1 1 0 137068 29784 68644 0 0 0 21696 444 695 0 1 0 99
0 1 0 137068 29784 68644 0 0 0 21632 439 689 0 1 0 99
0 1 0 137068 29784 68644 0 0 0 21632 439 689 0 3 0 97
0 1 0 137068 29784 68644 0 0 0 21696 440 687 0 2 0 98

[-- Attachment #10: vmstat_two_disks.txt --]
[-- Type: text/plain, Size: 936 bytes --]

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 2 0 133908 31200 70236 0 0 26 4585 185 177 22 4 61 14
2 2 0 133908 31200 70236 0 0 0 15936 354 488 0 2 0 98
1 2 0 133908 31200 70236 0 0 0 16064 356 514 0 1 0 99
1 2 0 133908 31200 70236 0 0 0 16000 354 496 0 1 0 99
2 2 0 133908 31200 70236 0 0 0 15872 350 481 0 3 0 97
2 2 0 133908 31200 70236 0 0 0 15872 350 477 0 3 0 97
2 2 0 133908 31208 70236 0 0 0 15984 357 507 0 2 0 98
2 2 0 133908 31208 70236 0 0 0 15872 352 481 0 4 0 96
3 2 0 133908 31208 70236 0 0 0 15872 351 469 0 2 0 98
3 2 0 133908 31208 70236 0 0 0 16000 355 500 0 1 0 99

[-- Attachment #11: vmstat_two_disks_bad2.txt --]
[-- Type: text/plain, Size: 936 bytes --]

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
8 1 0 8260 64260 117276 0 0 18 2481 239 279 44 7 44 5
8 2 0 8200 64260 117276 0 0 0 15920 361 510 89 11 0 0
9 2 0 8200 64268 117276 0 0 0 16100 362 511 89 11 0 0
9 2 0 8200 64268 117276 0 0 0 16000 353 495 85 15 0 0
8 2 0 8200 64268 117276 0 0 0 16000 352 502 90 10 0 0
9 2 0 8200 64268 117276 0 0 0 15872 350 494 88 12 0 0
8 2 0 8200 64268 117276 0 0 0 15872 350 497 91 9 0 0
9 2 0 8200 64268 117276 0 0 0 15872 353 471 75 25 0 0
8 2 0 8200 64276 117276 0 0 0 15956 356 521 88 12 0 0
9 2 0 8200 64276 117276 0 0 0 16000 353 516 92 8 0 0