From mboxrd@z Thu Jan  1 00:00:00 1970
From: Jakub Kicinski <jakub.kicinski@netronome.com>
Subject: Re: [net-next 1/6] net/dcb: Add dcbnl buffer attribute
Date: Mon, 21 May 2018 22:20:26 -0700
Message-ID: <20180521222026.4f54f479@cakuba>
References: <20180521210502.11082-1-saeedm@mellanox.com>
        <20180521210502.11082-2-saeedm@mellanox.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Cc: "David S. Miller" <davem@davemloft.net>, netdev@vger.kernel.org,
        Huy Nguyen <huyn@mellanox.com>, Jiri Pirko <jiri@resnulli.us>,
        Or Gerlitz <gerlitz.or@gmail.com>
To: Saeed Mahameed <saeedm@mellanox.com>
Return-path: <netdev-owner@vger.kernel.org>
Received: from mail-qk0-f169.google.com ([209.85.220.169]:37316 "EHLO
        mail-qk0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1751060AbeEVFUa (ORCPT
        <rfc822;netdev@vger.kernel.org>); Tue, 22 May 2018 01:20:30 -0400
Received: by mail-qk0-f169.google.com with SMTP id r28-v6so11344861qkk.4
        for <netdev@vger.kernel.org>; Mon, 21 May 2018 22:20:30 -0700 (PDT)
In-Reply-To: <20180521210502.11082-2-saeedm@mellanox.com>
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

On Mon, 21 May 2018 14:04:57 -0700, Saeed Mahameed wrote:
> From: Huy Nguyen <huyn@mellanox.com>
> 
> In this patch, we add dcbnl buffer attribute to allow user
> change the NIC's buffer configuration such as priority
> to buffer mapping and buffer size of individual buffer.
> 
> This attribute combined with pfc attribute allows advance user to
> fine tune the qos setting for specific priority queue. For example,
> user can give dedicated buffer for one or more prirorities or user
> can give large buffer to certain priorities.
> 
> We present an use case scenario where dcbnl buffer attribute configured
> by advance user helps reduce the latency of messages of different sizes.
> 
> Scenarios description:
> On ConnectX-5, we run latency sensitive traffic with
> small/medium message sizes ranging from 64B to 256KB and bandwidth sensitive
> traffic with large messages sizes 512KB and 1MB. We group small, medium,
> and large message sizes to their own pfc enables priorities as follow.
>   Priorities 1 & 2 (64B, 256B and 1KB)
>   Priorities 3 & 4 (4KB, 8KB, 16KB, 64KB, 128KB and 256KB)
>   Priorities 5 & 6 (512KB and 1MB)
> 
> By default, ConnectX-5 maps all pfc enabled priorities to a single
> lossless fixed buffer size of 50% of total available buffer space. The
> other 50% is assigned to lossy buffer. Using dcbnl buffer attribute,
> we create three equal size lossless buffers. Each buffer has 25% of total
> available buffer space. Thus, the lossy buffer size reduces to 25%. Priority
> to lossless  buffer mappings are set as follow.
>   Priorities 1 & 2 on lossless buffer #1
>   Priorities 3 & 4 on lossless buffer #2
>   Priorities 5 & 6 on lossless buffer #3
> 
> We observe improvements in latency for small and medium message sizes
> as follows. Please note that the large message sizes bandwidth performance is
> reduced but the total bandwidth remains the same.
>   256B message size (42 % latency reduction)
>   4K message size (21% latency reduction)
>   64K message size (16% latency reduction)
> 
> Signed-off-by: Huy Nguyen <huyn@mellanox.com>
> Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>

On a cursory look this bares a lot of resemblance to devlink shared
buffer configuration ABI.  Did you look into using that?  

Just to be clear devlink shared buffer ABIs don't require representors
and "switchdev mode".