Re: Mainline kernel OLTP performance update

[Andrew Morton <akpm@linux-foundation.org>]

On Tue, 28 Apr 2009 10:08:22 -0700 "Styner, Douglas W" <douglas.w.styner@intel.com> wrote:

> Summary: Measured the mainline kernel from kernel.org (2.6.30-rc3). 
> 
> The regression for 2.6.30-rc3 against the baseline, 2.6.24.2 is 1.91%.  Oprofile reports 71.1626% user, 28.8295% system.  
> 
> Linux OLTP Performance summary
> Kernel#            Speedup(x)   Intr/s  CtxSw/s us%     sys%    idle%   iowait%
> 2.6.24.2                1.000   22106   43709   75      24      0       0
> 2.6.30-rc3              0.981   30645   43027   75      25      0       0

The main difference there is the interrupt frequency.  Do we know which
interrupt source(s) caused this?

> Server configurations:
> Intel Xeon Quad-core 2.0GHz  2 cpus/8 cores/8 threads
> 64GB memory, 3 qle2462 FC HBA, 450 spindles (30 logical units)
> 
> 
> ======oprofile CPU_CLK_UNHALTED for top 30 functions
> Cycles% 2.6.24.2                   Cycles% 2.6.30-rc3
> 74.8578 <database>                 69.1925 <database>

ouch, that's a large drop in userspace CPU occupancy.  It seems
inconsistent with the 1.91% above.

> 1.0500 qla24xx_start_scsi          1.1314 qla24xx_intr_handler
> 0.8089 schedule                    1.0031 qla24xx_start_scsi
> 0.5864 kmem_cache_alloc            0.8476 __schedule
> 0.4989 __blockdev_direct_IO        0.6532 kmem_cache_alloc
> 0.4357 __sigsetjmp                 0.4490 __blockdev_direct_IO
> 0.4152 copy_user_generic_string    0.4199 __sigsetjmp
> 0.3953 qla24xx_intr_handler        0.3946 __switch_to
> 0.3850 memcpy                      0.3538 __list_add
> 0.3596 scsi_request_fn             0.3499 task_rq_lock
> 0.3188 __switch_to                 0.3402 scsi_request_fn
> 0.2889 lock_timer_base             0.3382 rb_get_reader_page
> 0.2750 memmove                     0.3363 copy_user_generic_string
> 0.2519 task_rq_lock                0.3324 aio_complete
> 0.2474 aio_complete                0.3110 try_to_wake_up
> 0.2460 scsi_alloc_sgtable          0.2877 ring_buffer_consume
> 0.2445 generic_make_request        0.2683 mod_timer
> 0.2263 qla2x00_process_completed_re0.2605 qla2x00_process_completed_re
> 0.2118 blk_queue_end_tag           0.2566 blk_queue_end_tag
> 0.2085 dio_bio_complete            0.2566 generic_make_request
> 0.2021 e1000_xmit_frame            0.2547 tcp_sendmsg
> 0.2006 __end_that_request_first    0.2372 lock_timer_base
> 0.1954 generic_file_aio_read       0.2333 memmove
> 0.1949 kfree                       0.2294 memset_c
> 0.1915 tcp_sendmsg                 0.2080 mempool_free
> 0.1901 try_to_wake_up              0.2022 generic_file_aio_read
> 0.1895 kref_get                    0.1963 scsi_device_unbusy
> 0.1864 __mod_timer                 0.1963 plist_del
> 0.1863 thread_return               0.1944 dequeue_rt_stack
> 0.1854 math_state_restore          0.1924 e1000_xmit_frame