From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752832AbbCSQh3 (ORCPT ); Thu, 19 Mar 2015 12:37:29 -0400 Received: from mx2.parallels.com ([199.115.105.18]:39967 "EHLO mx2.parallels.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751202AbbCSQh1 (ORCPT ); Thu, 19 Mar 2015 12:37:27 -0400 Message-ID: <550AFB31.80608@parallels.com> Date: Thu, 19 Mar 2015 09:37:05 -0700 From: Maxim Patlasov User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Ming Lei CC: Linux Kernel Mailing List , Dave Kleikamp , Jens Axboe , Zach Brown , Christoph Hellwig , Andrew Morton , Alexander Viro , Benjamin LaHaise Subject: Re: [PATCH v2 4/4] block: loop: support to submit I/O via kernel aio based References: <1421163888-21452-1-git-send-email-ming.lei@canonical.com> <1421163888-21452-5-git-send-email-ming.lei@canonical.com> <5509C3E7.1030307@parallels.com> In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.10.39.160] X-ClientProxiedBy: US-EXCH2.sw.swsoft.com (10.255.249.46) To US-EXCH.sw.swsoft.com (10.255.249.47) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 03/18/2015 07:57 PM, Ming Lei wrote: > On Thu, Mar 19, 2015 at 2:28 AM, Maxim Patlasov wrote: >> On 01/13/2015 07:44 AM, Ming Lei wrote: >>> Part of the patch is based on Dave's previous post. >>> >>> This patch submits I/O to fs via kernel aio, and we >>> can obtain following benefits: >>> >>> - double cache in both loop file system and backend file >>> gets avoided >>> - context switch decreased a lot, and finally CPU utilization >>> is decreased >>> - cached memory got decreased a lot >>> >>> One main side effect is that throughput is decreased when >>> accessing raw loop block(not by filesystem) with kernel aio. >>> >>> This patch has passed xfstests test(./check -g auto), and >>> both test and scratch devices are loop block, file system is ext4. >>> >>> Follows two fio tests' result: >>> >>> 1. fio test inside ext4 file system over loop block >>> 1) How to run >>> - linux kernel base: 3.19.0-rc3-next-20150108(loop-mq merged) >>> - loop over SSD image 1 in ext4 >>> - linux psync, 16 jobs, size 200M, ext4 over loop block >>> - test result: IOPS from fio output >>> >>> 2) Throughput result: >>> ------------------------------------------------------------- >>> test cases |randread |read |randwrite |write | >>> ------------------------------------------------------------- >>> base |16799 |59508 |31059 |58829 >>> ------------------------------------------------------------- >>> base+kernel aio |15480 |64453 |30187 |57222 >>> ------------------------------------------------------------- >> >> Ming, it's important to understand the overhead of aio_kernel_() >> implementation. So could you please add test results for raw SSD device to >> the table above next time (in v3 of your patches). > what aio_kernel_() does is to just call ->read_iter()/->write_iter(), > so it should not have introduced extra overload. > > From performance view, the effect is only from switching to > O_DIRECT. With O_DIRECT, double cache can be avoided, > meantime both page caches and CPU utilization can be decreased. The way how you reused loop_queue_rq() --> queue_work() functionality (added early, by commit b5dd2f604) may affect performance of O_DIRECT operations. It can be easily demonstrated on ram-drive, but measurements on real storage h/w would be more convincing. Btw, when you wrote "linux psync, 16 jobs, size 200M, ext4 over loop block" -- does it mean that there were 16 threads in userspace submitting I/O concurrently? If yes, throughput comparison for a single job test would be also useful to look at. Thanks, Maxim