From mboxrd@z Thu Jan 1 00:00:00 1970 From: Shaohua Li Subject: Re: more dput lock contentions in 2.6.38-rc? Date: Tue, 25 Jan 2011 10:46:29 +0800 Message-ID: <20110125024629.GA8931@sli10-conroe.sh.intel.com> References: <1295915725.1949.967.camel@sli10-conroe> <1295917864.1949.970.camel@sli10-conroe> <1295919258.1949.973.camel@sli10-conroe> <20110125020123.GA30965@sli10-conroe.sh.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "linux-fsdevel@vger.kernel.org" , lkml , Andrew Morton , Nick Piggin , "Chen, Tim C" To: Nick Piggin Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org On Tue, Jan 25, 2011 at 10:09:48AM +0800, Nick Piggin wrote: > On Tue, Jan 25, 2011 at 1:01 PM, Shaohua Li wr= ote: > > On Tue, Jan 25, 2011 at 09:44:45AM +0800, Nick Piggin wrote: > >> On Tue, Jan 25, 2011 at 12:34 PM, Shaohua Li wrote: > >> > On Tue, 2011-01-25 at 09:26 +0800, Nick Piggin wrote: > >> >> On Tue, Jan 25, 2011 at 12:11 PM, Shaohua Li wrote: > >> >> > On Tue, 2011-01-25 at 09:04 +0800, Nick Piggin wrote: > >> >> >> On Tue, Jan 25, 2011 at 11:35 AM, Shaohua Li wrote: > >> >> >> > Hi, > >> >> >> > we are testing dbench benchmark and see big drop of 2.6.38= -rc compared > >> >> >> > to 2.6.37 in several machines with 2 sockets or 4 sockets.= We have 12 > >> >> >> > disks mount to /mnt/stp/dbenchdata/sd*/ and dbench runs ag= ainst data of > >> >> >> > the disks. According to perf, we saw more lock contentions= : > >> >> >> > In 2.6.37: 13.00% =A0 =A0 =A0 =A0dbench =A0[kernel.kallsym= s] =A0 [k] _raw_spin_lock > >> >> >> > In 2.6.38-rc: 69.45% =A0 =A0 =A0 =A0dbench =A0[kernel.kall= syms] =A0 [k]_raw_spin_lock > >> >> >> > - =A0 =A0 69.45% =A0 =A0 =A0 =A0dbench =A0[kernel.kallsyms= ] =A0 [k] _raw_spin_lock > >> >> >> > =A0 - _raw_spin_lock > >> >> >> > =A0 =A0 =A0- 48.41% dput > >> >> >> > =A0 =A0 =A0 =A0 - 61.17% path_put > >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0- 60.47% do_path_lookup > >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 53.18% user_path_at > >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 42.13% do_filp_open > >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 + 4.69% user_path_parent > >> >> >> > >> >> >> What filesystems are mounted on the path? > >> >> > ext3 or ext4 > >> >> > >> >> ext3 or 4 along every step of the path? Are there > >> >> any acls loaded, or security policy running? > >> > all disks are formated with the same fs, just some machines use = ext3 and > >> > others ext4. no we don't have acl or security policy. > >> >> It may be possible that they're all coming from > >> >> /proc/ access. > >> > I added trace in dput just after the lock taken. and most files = are '/', > >> > 'mnt', 'stp'. the percentage of 'proc' is small actually. > >> > >> Hm, OK well I could send you a patch to gather some statistics for > >> why rcu-walk gets dropped. It'll have to wait until I get home, th= ough. > > Sure, I'm still at office tomorrow. I can test before that day, oth= erwise > > maybe Tim can help. >=20 > There _are_ a lot of dput contentions coming from d_path. Some of the > dentries you're seeing in dput could be coming from path_put in > d_path (mountpoints, eg. 'mnt'). >=20 > What does an actual snippet from perf with callgraphs look like? here is another perf report # Events: 198K cycles # # Overhead Command Shared Object = Symbol # ........ ............ .................. .........................= =2E................ # 70.20% dbench [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock | =20 |---1.79%-- dput | | =20 | |---45.22%-- path_put | | | =20 | | |--58.16%-- do_path_lookup | | | | =20 | | | |--55.25%-- user_= path_at | | | | | = =20 | | | | |--96.= 85%-- vfs_fstatat | | | | | = vfs_stat | | | | | = sys_newstat | | | | | = system_call_fastpath | | | | | = __GI___xstat64 | | | | | = | =20 | | | | | = --100.00%-- 0x1000 | | | | | = =20 | | | | --3.1= 5%-- sys_statfs | | | | = system_call_fastpath | | | | = __GI___statfs | | | | = | =20 | | | | = |--80.01%-- _int_malloc | | | | = | =20 | | | | = --19.99%-- 0x6a | | | | =20 | | | |--40.19%-- do_fi= lp_open | | | | do_sys= _open | | | | sys_op= en | | | | system= _call_fastpath | | | | __GI_o= pen64 | | | | | = =20 | | | | --100= =2E00%-- __fopen_internal | | | | = | =20 | | | | = |--80.02%-- 0x200000000 | | | | = | =20 | | | | = --19.98%-- 0x34c333c0ef | | | | =20 | | | --4.56%-- user_p= ath_parent | | | | = =20 | | | |--66.= 67%-- do_unlinkat | | | | = sys_unlink | | | | = system_call_fastpath | | | | = __unlink | | | | = =20 | | | --33.= 33%-- sys_renameat | | | = sys_rename | | | = system_call_fastpath | | | = __GI_rename | | | =20 | | |--37.67%-- d_path | | | seq_path | | | show_vfsmnt | | | seq_read | | | vfs_read | | | sys_read | | | system_call_fastp= ath | | | __GI___libc_read | | | =20 | | |--2.15%-- do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | | =20 | | |--1.89%-- mounts_release | | | fput | | | filp_close | | | sys_close | | | system_call_fastp= ath | | | __GI_close | | --0.13%-- [...] | | =20 | |--41.39%-- link_path_walk | | | =20 | | |--96.97%-- do_path_lookup | | | | =20 | | | |--51.11%-- user_= path_at | | | | | = =20 | | | | |--94.= 58%-- vfs_fstatat | | | | | = vfs_stat | | | | | = sys_newstat | | | | | = system_call_fastpath | | | | | = __GI___xstat64 | | | | | = | =20 | | | | | = --100.00%-- 0x1000 | | | | | = =20 | | | | |--3.6= 2%-- sys_statfs | | | | | = system_call_fastpath | | | | | = __GI___statfs | | | | | = | =20 | | | | | = |--50.32%-- _int_malloc | | | | | = | =20 | | | | | = |--24.79%-- 0x1000 | | | | | = | =20 | | | | | = |--12.50%-- 0x4 | | | | | = | 0x544345004b4f5f53 | | | | | = | =20 | | | | | = --12.39%-- 0x34c3571740 | | | | | = =20 | | | | --1.8= 0%-- do_utimes | | | | = sys_utime | | | | = system_call_fastpath | | | | = __GI_utime | | | | =20 | | | |--44.11%-- do_fi= lp_open | | | | do_sys= _open | | | | sys_op= en | | | | system= _call_fastpath | | | | __GI_o= pen64 | | | | =20 | | | --4.78%-- user_p= ath_parent | | | | = =20 | | | |--61.= 54%-- do_unlinkat | | | | = sys_unlink | | | | = system_call_fastpath | | | | = __unlink | | | | = =20 | | | --38.= 46%-- sys_renameat | | | = sys_rename | | | = system_call_fastpath | | | = __GI_rename | | | =20 | | |--2.85%-- do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | --0.18%-- [...] | --0.22%-- [...] | =20 |---22.62%-- path_get | | =20 | |--61.51%-- nameidata_drop_rcu | | link_path_walk | | | =20 | | |--96.48%-- do_path_lookup | | | | =20 | | | |--54.89%-- user_= path_at | | | | | = =20 | | | | |--93.= 35%-- vfs_fstatat | | | | | = vfs_stat | | | | | = sys_newstat | | | | | = system_call_fastpath | | | | | = __GI___xstat64 | | | | | = | =20 | | | | | = --100.00%-- 0x1000 | | | | | = =20 | | | | |--5.0= 9%-- sys_statfs | | | | | = system_call_fastpath | | | | | = __GI___statfs | | | | | = | =20 | | | | | = |--50.01%-- 0x1000 | | | | | = | =20 | | | | | = --49.99%-- 0x9 | | | | | = =20 | | | | --1.5= 7%-- do_utimes | | | | = sys_utime | | | | = system_call_fastpath | | | | = __GI_utime | | | | =20 | | | |--40.17%-- do_fi= lp_open | | | | do_sys= _open | | | | sys_op= en | | | | system= _call_fastpath | | | | __GI_o= pen64 | | | | | = =20 | | | | --100= =2E00%-- __fopen_internal | | | | = | =20 | | | | = |--66.67%-- 0x34c333c0ef | | | | = | =20 | | | | = --33.33%-- 0x200000000 | | | | =20 | | | --4.94%-- user_p= ath_parent | | | | = =20 | | | |--73.= 91%-- do_unlinkat | | | | = sys_unlink | | | | = system_call_fastpath | | | | = __unlink | | | | = =20 | | | --26.= 09%-- sys_renameat | | | = sys_rename | | | = system_call_fastpath | | | = __GI_rename | | | =20 | | --3.52%-- do_filp_open | | do_sys_open | | sys_open | | system_call_fastp= ath | | __GI_open64 | | =20 | |--37.35%-- d_path | | seq_path | | show_vfsmnt | | seq_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | =20 | --1.15%-- get_task_root | mounts_open_common | mounts_open | __dentry_open | nameidata_to_filp | finish_open | do_filp_open | do_sys_open | sys_open | system_call_fastpath | __GI_open64 | | =20 | --100.00%-- __fopen_interna= l | | =20 | |--50.01%-- 0x34c= 333c0ef | | =20 | --49.99%-- 0x200= 000000 | =20 |--19.30%-- nameidata_drop_rcu | link_path_walk | | =20 | |--96.22%-- do_path_lookup | | | =20 | | |--54.40%-- user_path_at | | | | =20 | | | |--90.61%-- vfs_f= statat | | | | vfs_st= at | | | | sys_ne= wstat | | | | system= _call_fastpath | | | | __GI__= _xstat64 | | | | | = =20 | | | | --100= =2E00%-- 0x1000 | | | | =20 | | | |--7.22%-- sys_st= atfs | | | | system= _call_fastpath | | | | __GI__= _statfs | | | | | = =20 | | | | |--60.= 00%-- _int_malloc | | | | | = =20 | | | | |--20.= 02%-- 0x1000 | | | | | = =20 | | | | --19.= 98%-- 0x34c3571740 | | | | =20 | | | --2.17%-- do_uti= mes | | | sys_ut= ime | | | system= _call_fastpath | | | __GI_u= time | | | =20 | | |--40.88%-- do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | | | =20 | | | --100.00%-- __fo= pen_internal | | | | = =20 | | | |--64.= 00%-- 0x200000000 | | | | = =20 | | | --36.= 00%-- 0x34c333c0ef | | | =20 | | --4.71%-- user_path_parent | | | =20 | | |--79.16%-- do_un= linkat | | | sys_un= link | | | system= _call_fastpath | | | __unli= nk | | | =20 | | --20.84%-- sys_r= enameat | | sys_re= name | | system= _call_fastpath | | __GI_r= ename | | =20 | --3.78%-- do_filp_open | do_sys_open | sys_open | system_call_fastpath | __GI_open64 | =20 |--0.95%-- __d_lookup | do_lookup | link_path_walk | | =20 | |--88.46%-- do_path_lookup | | | =20 | | |--52.16%-- user_path_at | | | | =20 | | | |--91.67%-- vfs_f= statat | | | | vfs_st= at | | | | sys_ne= wstat | | | | system= _call_fastpath | | | | __GI__= _xstat64 | | | | =20 | | | --8.33%-- sys_st= atfs | | | system= _call_fastpath | | | __GI__= _statfs | | | =20 | | --47.84%-- do_filp_open | | do_sys_open | | sys_open | | system_call_fastp= ath | | __GI_open64 | | =20 | --11.54%-- do_filp_open | do_sys_open | sys_open | system_call_fastpath | __GI_open64 | =20 |--0.51%-- d_path | seq_path | show_vfsmnt | seq_read | vfs_read | sys_read | system_call_fastpath | __GI___libc_read --1.17%-- [...] 2.26% dbench [kernel.kallsyms] [k] copy_user_generic_str= ing | --- copy_user_generic_string | =20 |--67.84%-- generic_file_aio_read | do_sync_read | vfs_read | | =20 | |--98.11%-- sys_pread64 | | system_call_fastpath | | __libc_pread | | =20 | --1.89%-- sys_read | system_call_fastpath | __GI___libc_read | =20 |--28.32%-- generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | do_sync_write | vfs_write | sys_pwrite64 | system_call_fastpath | __GI___pwrite64 | =20 |--2.56%-- call_filldir | ext3_readdir | vfs_readdir | sys_getdents | system_call_fastpath | __getdents64 | =20 --1.28%-- vfs_read sys_read system_call_fastpath __GI___libc_read 1.54% dbench [kernel.kallsyms] [k] read_hpet | --- read_hpet ktime_get | =20 |--47.29%-- tick_sched_timer | __run_hrtimer | hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | | =20 | |--34.29%-- dput | | | =20 | | |--50.00%-- path_put | | | | =20 | | | |--50.00%-- do_pa= th_lookup | | | | do_fil= p_open | | | | do_sys= _open | | | | sys_op= en | | | | system= _call_fastpath | | | | __GI_o= pen64 | | | | =20 | | | |--33.33%-- d_pat= h | | | | seq_pa= th | | | | show_v= fsmnt | | | | seq_re= ad | | | | vfs_re= ad | | | | sys_re= ad | | | | system= _call_fastpath | | | | __GI__= _libc_read | | | | =20 | | | --16.67%-- do_fi= lp_open | | | do_sys= _open | | | sys_op= en | | | system= _call_fastpath | | | __GI_o= pen64 | | | =20 | | --50.00%-- link_path_walk | | do_path_lookup | | | =20 | | |--50.00%-- user_= path_at | | | vfs_fs= tatat | | | vfs_st= at | | | sys_ne= wstat | | | system= _call_fastpath | | | __GI__= _xstat64 | | | =20 | | |--33.34%-- do_fi= lp_open | | | do_sys= _open | | | sys_op= en | | | system= _call_fastpath | | | __GI_o= pen64 | | | =20 | | --16.66%-- user_= path_parent | | do_unl= inkat | | sys_un= link | | system= _call_fastpath | | __unli= nk | | =20 | |--22.86%-- path_get | | | =20 | | |--62.49%-- d_path | | | seq_path | | | show_vfsmnt | | | seq_read | | | vfs_read | | | sys_read | | | system_call_fastp= ath | | | __GI___libc_read | | | =20 | | --37.51%-- nameidata_drop_r= cu | | link_path_walk | | do_path_lookup | | | =20 | | |--66.66%-- user_= path_at | | | vfs_fs= tatat | | | vfs_st= at | | | sys_ne= wstat | | | system= _call_fastpath | | | __GI__= _xstat64 | | | | = =20 | | | --100= =2E00%-- 0x1000 | | | =20 | | --33.34%-- do_fi= lp_open | | do_sys= _open | | sys_op= en | | system= _call_fastpath | | __GI_o= pen64 | | =20 | |--11.42%-- nameidata_drop_rcu | | link_path_walk | | do_path_lookup | | | =20 | | |--50.00%-- user_path_at | | | | =20 | | | |--50.01%-- sys_s= tatfs | | | | system= _call_fastpath | | | | __GI__= _statfs | | | | _int_m= alloc | | | | =20 | | | --49.99%-- vfs_f= statat | | | vfs_st= at | | | sys_ne= wstat | | | system= _call_fastpath | | | __GI__= _xstat64 | | | =20 | | |--25.00%-- do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | | =20 | | --24.99%-- user_path_parent | | sys_renameat | | sys_rename | | system_call_fastp= ath | | __GI_rename | | =20 | |--5.72%-- link_path_walk | | do_path_lookup | | | =20 | | |--50.04%-- user_path_at | | | vfs_fstatat | | | vfs_stat | | | sys_newstat | | | system_call_fastp= ath | | | __GI___xstat64 | | | =20 | | --49.96%-- do_filp_open | | do_sys_open | | sys_open | | system_call_fastp= ath | | __GI_open64 | | =20 | |--2.86%-- show_vfsmnt | | seq_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | =20 | |--2.86%-- journal_get_write_access | | __ext3_journal_get_write_acc= ess | | ext3_reserve_inode_write | | ext3_orphan_del | | ext3_evict_inode | | evict | | iput | | do_unlinkat | | sys_unlink | | system_call_fastpath | | __unlink | | =20 | |--2.86%-- follow_managed | | do_lookup | | link_path_walk | | do_path_lookup | | do_filp_open | | do_sys_open | | sys_open | | system_call_fastpath | | __GI_open64 | | =20 | |--2.86%-- m_start | | seq_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | =20 | |--2.86%-- malloc_consolidate | | =20 | |--2.86%-- ext3fs_dirhash | | htree_dirblock_to_tree | | ext3_htree_fill_tree | | ext3_readdir | | vfs_readdir | | sys_getdents | | system_call_fastpath | | __getdents64 | | =20 | |--2.86%-- ext3_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | do_sync_write | | vfs_write | | sys_pwrite64 | | system_call_fastpath | | __GI___pwrite64 | | =20 | |--2.86%-- handle_pte_fault | | handle_mm_fault | | do_page_fault | | page_fault | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | =20 | --2.85%-- __strlen_sse2 | =20 |--32.43%-- sched_clock_tick | scheduler_tick | update_process_times | tick_sched_timer | __run_hrtimer | hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | | =20 | |--37.51%-- path_get | | | =20 | | |--88.89%-- nameidata_drop_r= cu | | | link_path_walk | | | do_path_lookup | | | | =20 | | | |--62.49%-- user_= path_at | | | | vfs_fs= tatat | | | | vfs_st= at | | | | sys_ne= wstat | | | | system= _call_fastpath | | | | __GI__= _xstat64 | | | | =20 | | | --37.51%-- do_fi= lp_open | | | do_sys= _open | | | sys_op= en | | | system= _call_fastpath | | | __GI_o= pen64 | | | =20 | | --11.11%-- d_path | | seq_path | | show_vfsmnt | | seq_read | | vfs_read | | sys_read | | system_call_fastp= ath | | __GI___libc_read | | =20 | |--29.16%-- dput | | | =20 | | |--57.16%-- path_put | | | | =20 | | | |--50.01%-- do_pa= th_lookup | | | | user_p= ath_at | | | | vfs_fs= tatat | | | | vfs_st= at | | | | sys_ne= wstat | | | | system= _call_fastpath | | | | __GI__= _xstat64 | | | | =20 | | | --49.99%-- d_pat= h | | | seq_pa= th | | | show_v= fsmnt | | | seq_re= ad | | | vfs_re= ad | | | sys_re= ad | | | system= _call_fastpath | | | __GI__= _libc_read | | | =20 | | --42.84%-- link_path_walk | | do_path_lookup | | | =20 | | |--66.65%-- user_= path_at | | | vfs_fs= tatat | | | vfs_st= at | | | sys_ne= wstat | | | system= _call_fastpath | | | __GI__= _xstat64 | | | =20 | | --33.35%-- do_fi= lp_open | | do_sys= _open | | sys_op= en | | system= _call_fastpath | | __GI_o= pen64 | | =20 | |--12.49%-- nameidata_drop_rcu | | link_path_walk | | | =20 | | |--66.65%-- do_path_lookup | | | do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | | =20 | | --33.35%-- do_filp_open | | do_sys_open | | sys_open | | system_call_fastp= ath | | __GI_open64 | | =20 | |--4.17%-- journal_get_write_access | | __ext3_journal_get_write_acc= ess | | ext3_get_blocks_handle | | ext3_get_block | | __block_write_begin | | ext3_write_begin | | generic_file_buffered_write | | __generic_file_aio_write | | generic_file_aio_write | | do_sync_write | | vfs_write | | sys_pwrite64 | | system_call_fastpath | | __GI___pwrite64 | | =20 | |--4.17%-- path_put | | do_path_lookup | | do_filp_open | | do_sys_open | | sys_open | | system_call_fastpath | | __GI_open64 | | =20 | |--4.17%-- htree_dirblock_to_tree | | ext3_htree_fill_tree | | ext3_readdir | | vfs_readdir | | sys_getdents | | system_call_fastpath | | __getdents64 | | =20 | |--4.17%-- link_path_walk | | do_path_lookup | | do_filp_open | | do_sys_open | | sys_open | | system_call_fastpath | | __GI_open64 | | =20 | --4.16%-- _int_free | =20 |--10.81%-- hrtimer_interrupt | smp_apic_timer_interrupt | apic_timer_interrupt | | =20 | |--50.00%-- dput | | path_put | | do_path_lookup | | | =20 | | |--75.00%-- user_path_at | | | | =20 | | | |--66.67%-- vfs_f= statat | | | | vfs_st= at | | | | sys_ne= wstat | | | | system= _call_fastpath | | | | __GI__= _xstat64 | | | | =20 | | | --33.33%-- sys_s= tatfs | | | system= _call_fastpath | | | __GI__= _statfs | | | =20 | | --25.00%-- do_filp_open | | do_sys_open | | sys_open | | system_call_fastp= ath | | __GI_open64 | | =20 | |--12.51%-- nameidata_drop_rcu | | link_path_walk | | do_path_lookup | | do_filp_open | | do_sys_open | | sys_open | | system_call_fastpath | | __GI_open64 | | =20 | |--12.50%-- path_get | | d_path | | seq_path | | show_vfsmnt | | seq_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | =20 | |--12.50%-- d_path | | seq_path | | show_vfsmnt | | seq_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | =20 | --12.50%-- getmntent_r | =20 --9.46%-- tick_dev_program_event tick_program_event hrtimer_interrupt smp_apic_timer_interrupt apic_timer_interrupt | =20 |--71.41%-- dput | | =20 | |--59.99%-- link_path_walk | | do_path_lookup | | | =20 | | |--66.65%-- do_fi= lp_open | | | do_sys= _open | | | sys_op= en | | | system= _call_fastpath | | | __GI_o= pen64 | | | =20 | | --33.35%-- user_= path_at | | vfs_fs= tatat | | vfs_st= at | | sys_ne= wstat | | system= _call_fastpath | | __GI__= _xstat64 | | =20 | --40.01%-- path_put | | =20 | |--50.02%-- do_pa= th_lookup | | do_fil= p_open | | do_sys= _open | | sys_op= en | | system= _call_fastpath | | __GI_o= pen64 | | =20 | --49.98%-- d_pat= h | seq_pa= th | show_v= fsmnt | seq_re= ad | vfs_re= ad | sys_re= ad | system= _call_fastpath | __GI__= _libc_read | =20 |--14.30%-- find_lock_page | grab_cache_page_write_begin | ext3_write_begin | generic_file_buffered_write | __generic_file_aio_write | generic_file_aio_write | do_sync_write | vfs_write | sys_pwrite64 | system_call_fastpath | __GI___pwrite64 | =20 --14.30%-- path_get d_path seq_path show_vfsmnt seq_read vfs_read sys_read system_call_fastpath __GI___libc_read 1.46% dbench [kernel.kallsyms] [k] dput | --- dput | =20 |--52.83%-- path_put | | =20 | |--57.98%-- do_path_lookup | | | =20 | | |--50.17%-- do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | | =20 | | |--40.77%-- user_path_at | | | | =20 | | | |--77.78%-- vfs_f= statat | | | | vfs_st= at | | | | sys_ne= wstat | | | | system= _call_fastpath | | | | __GI__= _xstat64 | | | | | = =20 | | | | --100= =2E00%-- 0x1000 | | | | =20 | | | --22.22%-- sys_s= tatfs | | | system= _call_fastpath | | | __GI__= _statfs | | | | = =20 | | | --100= =2E00%-- 0x340000006b | | | =20 | | --9.06%-- user_path_parent | | do_unlinkat | | sys_unlink | | system_call_fastp= ath | | __unlink | | =20 | |--39.39%-- d_path | | seq_path | | show_vfsmnt | | seq_read | | vfs_read | | sys_read | | system_call_fastpath | | __GI___libc_read | | =20 | --2.63%-- mounts_release | fput | filp_close | sys_close | system_call_fastpath | __GI_close | =20 |--44.40%-- link_path_walk | | =20 | |--93.75%-- do_path_lookup | | | =20 | | |--56.66%-- user_path_at | | | | =20 | | | |--88.24%-- vfs_f= statat | | | | vfs_st= at | | | | sys_ne= wstat | | | | system= _call_fastpath | | | | __GI__= _xstat64 | | | | =20 | | | --11.76%-- sys_s= tatfs | | | system= _call_fastpath | | | __GI__= _statfs | | | | = =20 | | | --100= =2E00%-- _int_malloc | | | =20 | | |--40.00%-- do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | | =20 | | --3.33%-- user_path_parent | | do_unlinkat | | sys_unlink | | system_call_fastp= ath | | __unlink | | =20 | --6.25%-- do_filp_open | do_sys_open | sys_open | system_call_fastpath | __GI_open64 | =20 --2.78%-- follow_managed do_lookup link_path_walk do_path_lookup | =20 |--50.00%-- do_filp_open | do_sys_open | sys_open | system_call_fastpath | __GI_open64 | =20 --50.00%-- user_path_at vfs_fstatat vfs_stat sys_newstat system_call_fastpath __GI___xstat64 0.77% swapper [kernel.kallsyms] [k] poll_idle | --- poll_idle cpu_idle | =20 |--4.82%-- start_secondary | =20 --3.90%-- rest_init start_kernel x86_64_start_reservations x86_64_start_kernel 0.74% dbench [kernel.kallsyms] [k] link_path_walk | --- link_path_walk | =20 |--87.50%-- do_path_lookup | | =20 | |--52.37%-- user_path_at | | vfs_fstatat | | vfs_stat | | sys_newstat | | system_call_fastpath | | __GI___xstat64 | | =20 | |--38.10%-- do_filp_open | | do_sys_open | | sys_open | | system_call_fastpath | | __GI_open64 | | =20 | --9.52%-- user_path_parent | do_unlinkat | sys_unlink | system_call_fastpath | __unlink | =20 |--8.34%-- do_filp_open | do_sys_open | sys_open | system_call_fastpath | __GI_open64 | =20 --4.17%-- user_path_at vfs_fstatat vfs_stat sys_newstat system_call_fastpath __GI___xstat64 0.61% dbench libc-2.11.so [.] __strchr_sse42 | --- __strchr_sse42 0x608500 | =20 |--11.99%-- 0x524f464e495f4854 | =20 |--8.00%-- 0x353300004b4f5f53 | =20 |--8.00%-- 0x746e65696c632f73 | =20 |--8.00%-- 0x5443454a424f5f53 | =20 |--8.00%-- 0x65696c632f222058 | =20 |--4.01%-- 0x647e00004b4f5f53 | =20 |--4.00%-- 0x4355535f4f4e5f53 | =20 |--4.00%-- 0x4b4f00004b4f5f53 | =20 |--4.00%-- 0x4f4e00004b4f5f53 | =20 |--4.00%-- 0x3120302035303130 | =20 |--4.00%-- 0x3335353620363335 | =20 |--4.00%-- 0x535f544e20343132 | =20 |--4.00%-- 0x3535362032373031 | =20 |--4.00%-- 0x4d5500004b4f5f53 | =20 |--4.00%-- 0x4154535f544e2036 | =20 |--4.00%-- 0x4f5f535554415453 | =20 |--4.00%-- 0x5f5355544154535f | =20 |--4.00%-- 0x2036393034203032 | =20 --4.00%-- 0x535f544e20363930 0.59% dbench [kernel.kallsyms] [k] mntput_no_expire | --- mntput_no_expire | =20 |--97.30%-- mntput | | =20 | |--88.89%-- path_put | | | =20 | | |--49.99%-- do_path_lookup | | | | =20 | | | |--62.50%-- user_= path_at | | | | vfs_fs= tatat | | | | vfs_st= at | | | | sys_ne= wstat | | | | system= _call_fastpath | | | | __GI__= _xstat64 | | | | =20 | | | |--31.25%-- do_fi= lp_open | | | | do_sys= _open | | | | sys_op= en | | | | system= _call_fastpath | | | | __GI_o= pen64 | | | | =20 | | | --6.25%-- user_p= ath_parent | | | do_unl= inkat | | | sys_un= link | | | system= _call_fastpath | | | __unli= nk | | | =20 | | |--40.63%-- d_path | | | seq_path | | | show_vfsmnt | | | seq_read | | | vfs_read | | | sys_read | | | system_call_fastp= ath | | | __GI___libc_read | | | =20 | | |--3.13%-- do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | | =20 | | |--3.13%-- mounts_release | | | fput | | | filp_close | | | sys_close | | | system_call_fastp= ath | | | __GI_close | | | =20 | | --3.13%-- vfs_fstatat | | vfs_stat | | sys_newstat | | system_call_fastp= ath | | __GI___xstat64 | | =20 | |--5.56%-- link_path_walk | | do_path_lookup | | | =20 | | |--50.02%-- user_path_parent | | | do_unlinkat | | | sys_unlink | | | system_call_fastp= ath | | | __unlink | | | =20 | | --49.98%-- user_path_at | | vfs_fstatat | | vfs_stat | | sys_newstat | | system_call_fastp= ath | | __GI___xstat64 | | =20 | --5.55%-- fput | filp_close | sys_close | system_call_fastpath | __GI_close | =20 --2.70%-- path_put vfs_fstatat vfs_stat sys_newstat system_call_fastpath __GI___xstat64 0.53% dbench [kernel.kallsyms] [k] __d_lookup | --- __d_lookup | =20 |--93.55%-- do_lookup | link_path_walk | | =20 | |--96.55%-- do_path_lookup | | | =20 | | |--46.43%-- user_path_at | | | vfs_fstatat | | | vfs_stat | | | sys_newstat | | | system_call_fastp= ath | | | __GI___xstat64 | | | =20 | | |--39.28%-- do_filp_open | | | do_sys_open | | | sys_open | | | system_call_fastp= ath | | | __GI_open64 | | | =20 | | --14.29%-- user_path_parent | | | =20 | | |--74.99%-- do_un= linkat | | | sys_un= link | | | system= _call_fastpath | | | __unli= nk | | | =20 | | --25.01%-- sys_r= enameat | | sys_re= name | | system= _call_fastpath | | __GI_r= ename | | =20 | --3.45%-- do_filp_open | do_sys_open | sys_open | system_call_fastpath | __GI_open64 | =20 --6.45%-- link_path_walk do_path_lookup do_filp_open do_sys_open sys_open system_call_fastpath __GI_open64