From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.fusionio.com ([64.244.102.30]:43179 "EHLO mx1.fusionio.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753876Ab1B1NHr (ORCPT ); Mon, 28 Feb 2011 08:07:47 -0500 Message-ID: <4D6B9E21.70705@fusionio.com> Date: Mon, 28 Feb 2011 08:07:45 -0500 From: Jens Axboe MIME-Version: 1.0 Subject: Re: uneven IOPS among HD References: <932143.86482.qm@web161913.mail.bf1.yahoo.com> <4D616775.8050908@fusionio.com> In-Reply-To: Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit Sender: fio-owner@vger.kernel.org List-Id: fio@vger.kernel.org To: Josh Aune Cc: , john smith , "fio@vger.kernel.org" On 2011-02-26 15:31, Josh Aune wrote: > On Sunday, February 20, 2011, Jens Axboe wrote: >> On 2011-02-17 03:14, john smith wrote: >>> I have a couple of questions: >>> >>> 1) if it is normal (and if so, why) that while running a number of identical jobs (e.g 4), each bound to a different CPU (cpu_allowed=i) and each performing sequential (bs=512) reads on (e.g. 4) different HD, fio reports IOPS numbers that are highly unbalanced among HD with one HD reporting more than 7x of the IOPS of the minimum IOPS HD. >>> Different runs may show most of the IOPS on a different HD, so it's not that one HD is faster. >>> >>> Here are some possibly interesting parameters: >>> >>> ioengine=libaio >>> iodepth=32 >>> direct=1 >> >> There should be very little variance between runs, and between jobs. >> Each are separate threads/processes, so most of the variance you will >> see will be due to system "artifacts". So I'd suggest you look there. If >> you don't pin to specific CPUs, do you see smaller or larger >> differences? >> >> Are you doing random or sequential IO? Could also be timing differences >> in submission and dispatch from the IO scheduler. > > Is hyper threading enabled? Could be running some of the jobs on the > same CPU core. > > Does it track to different sockets? No, the replay doesn't factor in CPU placement of the apps submitting IO. -- Jens Axboe