From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2346EC25B0C for ; Sun, 7 Aug 2022 18:46:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date: Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fRF6tDwiED09Qm8+jjF0m2XwEBk/N1T42MZbGGhUU2s=; b=kT0g6LdbHMt/MkrZJ2wVR98715 vLSTzRVAI8C4J+IfEOnBrEizX032sgc+E6NOURufHE9GelKAbMEAtPjbO6dvmW3+uKrrgZzz6hdhG gzl0cjnn6Cq2VRNGuZHZj9deuwJNvD6D6FdwTidB14ieLnAq6iy1vkQomq7v4T36vCNDF50w1Aew+ 0wNzmz6r7qR0h/SrQ8RPOeJL5tTH20/H5+2yplHgNs0uzOedsM0pIFKYQPgVtEC4GcKK8kgZngmhu 6bymSVh6SPEEOkn2eC8TEukt7VkhUYU2VaNnlLZ4JppaH3PeVOqKJOxELxmR3bHESuyXSIbQQqm8U 1u2SuZFA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oKlIc-001RoK-GD; Sun, 07 Aug 2022 18:46:54 +0000 Received: from mail-pj1-x102c.google.com ([2607:f8b0:4864:20::102c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oKlIY-001Rk6-BK for linux-nvme@lists.infradead.org; Sun, 07 Aug 2022 18:46:52 +0000 Received: by mail-pj1-x102c.google.com with SMTP id t2-20020a17090a4e4200b001f21572f3a4so7290108pjl.0 for ; Sun, 07 Aug 2022 11:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc; bh=fRF6tDwiED09Qm8+jjF0m2XwEBk/N1T42MZbGGhUU2s=; b=sUfIjnfFg6C7BTgJCAUiobCnbcymxCqjth0VgNCg5PRpdOLoFXN06GP/722jVG/hut 4X/7qgNXMYcShr8q27lbxVxfQ21iOwZdtwyZmgX8B72vmSMeYt2fLKSFupJubJr7uXAz bIz7blhJEPqDbCHiBgiaBkm9/9AiWj1jDbf2w0cgyUeGKRlvs72m0HeNSYhJBhb06cru GAJV2EIdUH2dYA6Vj1qmWmrRyWb2wpxuW9W2/7qjWeavx9CM5E59Ip9GnyxEE0GLELSd QQuhncoy6aalVGBpadAattGstyajxkl5ceqlxvc/h1nN4eA8unicErsc1OGzuRb8ytB2 Yjbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc; bh=fRF6tDwiED09Qm8+jjF0m2XwEBk/N1T42MZbGGhUU2s=; b=pFc/23cRl40odCk17ncmZ4MDhL58G2h1Sku8J7CKVopSkhHtf2sNiLgOkkh556pI0t f5/0f+evWaSdOCtfHhxLTEnbZnbt0Egg9LkjBZmn/iBFdl088K8ABSvHpy+HpKV/c23K /pgw/Co/dMVlA7oMOf7hEBISaJB+oRtgNaQs3KsOyTWT4jIbNyLySUW6IT7m3dyE2J7A 2lCqu+R5QXK5G9U0tWMUgCytY8kA2WwOOEfthPALN6EbNlNa3fifg0TjY/WAncB68PzB bTJPDaAHD2LbVGwDuPrNd4PYaiEpPwr/XOgePSicAipRVwFBejKQrPCBd+kDsiJ1dDSm cBFA== X-Gm-Message-State: ACgBeo2ZaJit83lbgpGrcDRwR9ajaEJqSHts4baQHqZBacSne6Bh2Cjx RDPUF58St+GOnVbIAHg8hvWztg== X-Google-Smtp-Source: AA6agR52R7nPXz6hA/NEJIpwExQy73zN+UQ2YPf1Pb4UGZRqFX/ApCGAZKP7G39++6dymp7prQKsLg== X-Received: by 2002:a17:902:c40a:b0:16e:cc02:b9ab with SMTP id k10-20020a170902c40a00b0016ecc02b9abmr15621295plk.81.1659898006947; Sun, 07 Aug 2022 11:46:46 -0700 (PDT) Received: from [192.168.1.100] ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id i25-20020a635419000000b0041c89bba5a8sm4787876pgb.25.2022.08.07.11.46.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 07 Aug 2022 11:46:46 -0700 (PDT) Message-ID: <068737bb-7729-decd-bd3b-60380c6443fc@kernel.dk> Date: Sun, 7 Aug 2022 12:46:45 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Subject: Re: [PATCH 0/4] iopoll support for io_uring/nvme passthrough Content-Language: en-US To: Kanchan Joshi Cc: Keith Busch , hch@lst.de, io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, ming.lei@redhat.com, joshiiitr@gmail.com, gost.dev@samsung.com References: <20220805154226.155008-1-joshi.k@samsung.com> <78f0ac8e-cd45-d71d-4e10-e6d2f910ae45@kernel.dk> <6bd091d6-e0e6-3095-fc6b-d32ec89db054@kernel.dk> <20220807175803.GA13140@test-zns> From: Jens Axboe In-Reply-To: <20220807175803.GA13140@test-zns> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220807_114650_444396_892869ED X-CRM114-Status: GOOD ( 14.97 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org On 8/7/22 11:58 AM, Kanchan Joshi wrote: > On Fri, Aug 05, 2022 at 12:15:24PM -0600, Jens Axboe wrote: >> On 8/5/22 12:11 PM, Keith Busch wrote: >>> On Fri, Aug 05, 2022 at 11:18:38AM -0600, Jens Axboe wrote: >>>> On 8/5/22 11:04 AM, Jens Axboe wrote: >>>>> On 8/5/22 9:42 AM, Kanchan Joshi wrote: >>>>>> Hi, >>>>>> >>>>>> Series enables async polling on io_uring command, and nvme passthrough >>>>>> (for io-commands) is wired up to leverage that. >>>>>> >>>>>> 512b randread performance (KIOP) below: >>>>>> >>>>>> QD_batch block passthru passthru-poll block-poll >>>>>> 1_1 80 81 158 157 >>>>>> 8_2 406 470 680 700 >>>>>> 16_4 620 656 931 920 >>>>>> 128_32 879 1056 1120 1132 >>>>> >>>>> Curious on why passthru is slower than block-poll? Are we missing >>>>> something here? >>>> >>>> I took a quick peek, running it here. List of items making it slower: >>>> >>>> - No fixedbufs support for passthru, each each request will go through >>>> get_user_pages() and put_pages() on completion. This is about a 10% >>>> change for me, by itself. >>> >>> Enabling fixed buffer support through here looks like it will take a >>> little bit of work. The driver needs an opcode or flag to tell it the >>> user address is a fixed buffer, and io_uring needs to export its >>> registered buffer for a driver like nvme to get to. >> >> Yeah, it's not a straight forward thing. But if this will be used with >> recycled buffers, then it'll definitely be worthwhile to look into. > > Had posted bio-cache and fixedbufs in the initial round but retracted > to get the foundation settled first. > https://lore.kernel.org/linux-nvme/20220308152105.309618-1-joshi.k@samsung.com/ > > I see that you brought back bio-cache already. I can refresh fixedbufs. Excellent, yes please bring back the fixedbufs. It's a 5-10% win, nothing to sneeze at. > Completion-batching seems too tightly coupled to block-path. It's really not, in fact it'd be even simpler to do for passthru. The rq->end_io handler just needs to know about it. -- Jens Axboe