From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EACE2C2D0C2 for ; Fri, 3 Jan 2020 16:29:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C5EE321734 for ; Fri, 3 Jan 2020 16:29:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727905AbgACQ3C (ORCPT ); Fri, 3 Jan 2020 11:29:02 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:43337 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727793AbgACQ3C (ORCPT ); Fri, 3 Jan 2020 11:29:02 -0500 Received: by mail-pg1-f194.google.com with SMTP id k197so23634087pga.10; Fri, 03 Jan 2020 08:29:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=JqvZcEu8W8qbxYaL1d0CeIMIcPYQQjtkLM1EHupMOTw=; b=pkSMVe+mOkmpGhX/5kjiHQCYoIk/6az+FjYDZtOYadvUvtpHzVGtOnNxgbPYz/rca2 4r4yF8MioIZq+Vn6wbhEtaMaQHREnsB4fl44dzHYm5jSqOK5WuqkD4JpIeZ/VP7g/coJ 5ME5FZdoVZRbx3FjMHS/itg61FhgVmtlqP6yRiqU/KvAN1ERO1F+D0vMaVS3m7RkAQOm Xkxgx86t4l13EIUNPx3UotxtrT7BaxkwJQxC47FvqH/Ym86Ros+Q4e8TtOwcmeZHQeeB Hip9TuB8IoF13OWorloMA5+OjKyJuRPc/FOlKRbsLcb4dDD+biaxVjMSOxU7PiYljQMx I30g== X-Gm-Message-State: APjAAAWrkOjTkKLEVdWzIkmC9cSdmBcYBAnOAGTjiYItUnT4tPjnWoUH 0juIk5HyXlvijlWP+FYYNgw= X-Google-Smtp-Source: APXvYqznV4MvK/lkK8CWTfG4BOxMM8tNc6vMJpxLLLOwaxirSgdxE3YG2JCSDNMjUGy+c2DE88WWEg== X-Received: by 2002:a63:1a08:: with SMTP id a8mr73067236pga.425.1578068941359; Fri, 03 Jan 2020 08:29:01 -0800 (PST) Received: from desktop-bart.svl.corp.google.com ([2620:15c:2cd:202:4308:52a3:24b6:2c60]) by smtp.gmail.com with ESMTPSA id k10sm15027522pjs.13.2020.01.03.08.29.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 03 Jan 2020 08:29:00 -0800 (PST) Subject: Re: [PATCH v5 00/25] RTRS (former IBTRS) rdma transport library and the corresponding RNBD (former IBNBD) rdma network block device To: Jinpu Wang , Jason Gunthorpe Cc: Jack Wang , linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, Jens Axboe , Christoph Hellwig , Sagi Grimberg , Leon Romanovsky , Doug Ledford , Danil Kipnis , rpenyaev@suse.de References: <20191220155109.8959-1-jinpuwang@gmail.com> <20200102181859.GC9282@ziepe.ca> From: Bart Van Assche Message-ID: Date: Fri, 3 Jan 2020 08:28:59 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On 1/3/20 4:39 AM, Jinpu Wang wrote: > Performance results for the v5.5-rc1 kernel are here: > link: https://github.com/ionos-enterprise/ibnbd/tree/develop/performance/v5-v5.5-rc1 > > Some workloads RNBD are faster, some workloads NVMeoF are faster. Thank you for having shared these graphs. Do the graphs in RNBD-SinglePath.pdf show that NVMeOF achieves similar or higher IOPS, higher bandwidth and lower latency than RNBD for workloads with a block size of 4 KB and also for mixed workloads with less than 20 disks, whether or not invalidation is enabled for RNBD? Is it already clear why NVMeOF performance drops if the number of disks is above 25? Is that perhaps caused by contention on the block layer tag allocator because multiple NVMe namespaces share a tag set? Can that contention be avoided by increasing the NVMeOF queue depth further? Thanks, Bart.