From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8137437334612924727==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth Date: Fri, 29 Dec 2017 20:42:55 +0000 Message-ID: <1514580173.2658.126.camel@intel.com> In-Reply-To: CAKkMQdxpYbj5fuALOLLuWUkvTc6HgfD1C65JzsfZX_VNPKRSsg@mail.gmail.com List-ID: To: spdk@lists.01.org --===============8137437334612924727== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote: > Dear all, > = > I am running nvme/perf with master branch on a EC2 instance (Redhat, 8 CPU > cores). I noticed that the IOPS is around 1K when the queue depth is 1. By > simply increase the queue length to 2, the IOPS grows to over 60K. May I = ask > if this is normal? Is this an EC2 dedicated server (i.e. physical hardware) or a virtual machi= ne? Is the NVMe device you're connected to backed by a local NVMe device, a rem= ote NVMe device, or an EBS volume? Amazon presents basically all storage as NVMe devices these days, but the actual backing storage can perform differently depending on the circumstances. In all cases I'm aware of, the NVMe device presented is not the actual NVMe= SSD. It's either a virtual NVMe device emulated in software, or it's implemented= on a special card that Amazon puts in between the platform and the real SSD. SPDK hasn't been tested thoroughly with Amazon's emulated NVMe devices, so there could be some issues there. Most NVMe devices historically were not validat= ed against polled mode, ultra high performance drivers so when SPDK was first released we found a number of firmware bugs in existing SSDs that were only exposed because SPDK was so fast. We're seeing less and less of that now, b= ut you could be hitting something like that here. --===============8137437334612924727==--