Re: Quadrant write performance degradation - kernel3.10 vs kernel3.4

[Zheng Liu <gnehzuil.liu@gmail.com>]

On Tue, Jun 17, 2014 at 09:52:46AM +0200, Lukáš Czerner wrote:
> On Mon, 16 Jun 2014, Darrick J. Wong wrote:
> 
> > Date: Mon, 16 Jun 2014 12:20:09 -0700
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > To: Tanya Brokhman <tlinder@codeaurora.org>
> > Cc: linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org,
> >     kdorfman@codeaurora.org, merez@codeaurora.org,
> >     Dolev Raviv <draviv@codeaurora.org>
> > Subject: Re: Quadrant write performance degradation - kernel3.10 vs kernel3.4
> > 
> > On Mon, Jun 16, 2014 at 09:02:08AM +0300, Tanya Brokhman wrote:
> > > Hello,
> > > Recently we encountered a performance degradation on 3.10kernel
> > > based build, compared to 3.4 based one, when running the fs_write
> > > Quadrant benchmark.
> > > We profiled the test and came to the conclusion that the root cause
> > > of the degradation is in the vfs_write call stack (overhead of
> > > 2611.2us is observed in 3.10 kernel compared to 3.4):
> > > 
> > > ret_fast_syscall
> > > SyS_write
> > > vfs_write (total time spent: 3.10kernel-21295us, 3.4kernel-18683.79us)
> > > do_sync_write
> > > ext4_file_write
> > > generic_file_aio_write (total time spent: 3.10kernel-19124.4us,
> > > 3.4kernel-16815us)
> > > __generic_file_aio_write
> > > generic_file_buffered_write
> > > ext4_da_write_begin (total time spent: 3.10kernel-10935.2us,
> > > 3.4kernel-8444.6us)
> > > __block_write_begin
> > > ext4_da_get_block_prep (total time spent: 3.10kernel-5402.6us,
> > > 3.4kernel-3576.8us)
> > > ext4_es_lookup_extent  (total time spent: 3.10kernel-2219.7us,
> > > 3.4kernel-0us)
[snip]
> > > 2) Extents status tracking: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ext4/extents_status.c?id=refs/tags/v3.10.42#n20
> > > “There is a cache extent for write access, so if writes are not very
> > > random, adding space operations are in O(1) time.”
> > 
> > I'm no expert on the extent status cache, but this seems like a possible cause.
> 
> Exactly, there has been some fixes since the introduction of extent
> status tree, however I've noticed some performance going down as
> well and I believe that extent status tree is to blame.
> 
> AFAIK you can not turn it off in any way, but there might be some
> way to test it's overhead. Zheng, do you have any suggestions ?

Sigh, sorry for the delay reply.

Lukas, Could you please share your test with me?  From the calltrace it
seems that the latency is in ext4_da_get_block_prep.  It is not easy to
disable ext4_es_lookup_extent() because we need to lookup delayed extent
from extent status tree and determine whether or not we need to reserve
some disk spaces.

Tanya, I really appreciate if you can disable delalloc and re-run your
test.  You can use the following command to turn off the delalloc
feature.

 $ sudo mount -t ext4 -o remount,nodelalloc ${DEV} ${MNT}

Thanks,
                                                - Zheng

> 
> Thanks!
> -Lukas
> 
> > 
> > --D
> > > 
> > > We tried pick up several performance-enhancement patches from the
> > > community, released between 3.10 and 3.14 kernel versions. The
> > > performance was almost the same.
> > > 
> > > I was wondering what performance tests were performed on these
> > > features? Has anyone encountered same issue?
> > > 
> > > Best Regards
> > > Tanya Brokhman
> > > -- 
> > > QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member
> > > of Code Aurora Forum, hosted by The Linux Foundation
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html