Re: Quadrant write performance degradation - kernel3.10 vs kernel3.4

["Darrick J. Wong" <darrick.wong@oracle.com>]

On Mon, Jun 16, 2014 at 09:02:08AM +0300, Tanya Brokhman wrote:
> Hello,
> Recently we encountered a performance degradation on 3.10kernel
> based build, compared to 3.4 based one, when running the fs_write
> Quadrant benchmark.
> We profiled the test and came to the conclusion that the root cause
> of the degradation is in the vfs_write call stack (overhead of
> 2611.2us is observed in 3.10 kernel compared to 3.4):
> 
> ret_fast_syscall
> SyS_write
> vfs_write (total time spent: 3.10kernel-21295us, 3.4kernel-18683.79us)
> do_sync_write
> ext4_file_write
> generic_file_aio_write (total time spent: 3.10kernel-19124.4us,
> 3.4kernel-16815us)
> __generic_file_aio_write
> generic_file_buffered_write
> ext4_da_write_begin (total time spent: 3.10kernel-10935.2us,
> 3.4kernel-8444.6us)
> __block_write_begin
> ext4_da_get_block_prep (total time spent: 3.10kernel-5402.6us,
> 3.4kernel-3576.8us)
> ext4_es_lookup_extent  (total time spent: 3.10kernel-2219.7us,
> 3.4kernel-0us)
> 
> 
> We tried to revert just the ext4 code back to 3.4 (on a 3.10 kernel)
> build and got an improvement of 50% in the test result.
> When looking deeper into the changes made to the ext4 FS between 3.4
> and 3.10 versions we stumbled across two major features making an
> explicit tradeoff in favor of robustness and good design over
> performance in some use cases:
> 
> 1) Metadata Checksums http://kernelnewbies.org/Linux_3.5#head-e8ea0d70436ea63590eac3dc25a7b417333147f8
> “As far as performance impact goes, it shouldn't be noticeable for
> common desktop and server workloads. A mail server ffsb simulation
> show nearly no change. On a test doing only file creation and
> deletion and extent tree modifications, a performance drop of about
> 20 percent was measured. However, it's a workload very heavily
> oriented towards metadata, in most real-world workloads metadata is
> usually a small fraction of total IO, so unless your workload is
> metadata-oriented, the cost of enabling this feature should be
> negligible.”

Dumb question, but do you have metadata_csum enabled?  That would be a little
surprising, since (afaik) the only way you can turn it on is via unreleased
e2fsprogs-1.43.

(Otoh if you /do/ have it enabled and it's slowing you down, I'd like to hear
about it. ;))

> 2) Extents status tracking: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ext4/extents_status.c?id=refs/tags/v3.10.42#n20
> “There is a cache extent for write access, so if writes are not very
> random, adding space operations are in O(1) time.”

I'm no expert on the extent status cache, but this seems like a possible cause.

--D
> 
> We tried pick up several performance-enhancement patches from the
> community, released between 3.10 and 3.14 kernel versions. The
> performance was almost the same.
> 
> I was wondering what performance tests were performed on these
> features? Has anyone encountered same issue?
> 
> Best Regards
> Tanya Brokhman
> -- 
> QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member
> of Code Aurora Forum, hosted by The Linux Foundation
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html