From mboxrd@z Thu Jan  1 00:00:00 1970
From: Sagi Grimberg <sagi@grimberg.me>
Subject: Question on mlx5 PMD txq memory registration
Date: Mon, 17 Jul 2017 16:29:34 +0300
Message-ID: <75d08202-1882-7660-924c-b6dbb4455b88@grimberg.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
To: dev@dpdk.org, Shahaf Shuler <shahafs@mellanox.com>,
 =?UTF-8?Q?N=c3=a9lio_Laranjeiro?= <nelio.laranjeiro@6wind.com>,
 Yongseok Koh <yskoh@mellanox.com>, Roy Shterman <roys@lightbitslabs.com>,
 Alexander Solganik <sashas@lightbitslabs.com>
Return-path: <dev-bounces@dpdk.org>
Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com
 [209.85.128.193]) by dpdk.org (Postfix) with ESMTP id A5624377E
 for <dev@dpdk.org>; Mon, 17 Jul 2017 15:29:46 +0200 (CEST)
Received: by mail-wr0-f193.google.com with SMTP id 40so2679230wrw.0
 for <dev@dpdk.org>; Mon, 17 Jul 2017 06:29:46 -0700 (PDT)
Content-Language: en-US
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

Hi,

Looking at the code, it looks like mlx5 keeps a MR cache per TX queue
(each MR registers a rte_mempool).

Once a TX queue is created, mlx5 scans existing mempools and
pre-registers a MR for each mempool it meets (using rte_mempool_walk).
For each MR registration that exceeds the TX queue cache, it removes the
first entry from the cache and deregisters that MR (in txq_mp2mr_reg).

Now on TX burst, if the driver sees a mbuf from an unknown mempool, it
registers the mempool on the fly and again *deregister* the first MR in
the cache.

My question is, what guarantees that no inflight send operations posted
on the TX queue when we deregister and remove a MR from the cache?

AFAICT, it is the driver responsibility to guarantee to never deregister
a memory region that has inflight send operations posted, otherwise
the send operation *will* complete with a local protection error. Is
that taken care of?

Another question, why is the MR cache maintained per TX queue and not
per device? If the application starts N TX queues then a single mempool
will be registered N times instead of just once. Having lots of MR
instances will pollute the device ICMC pretty badly. Am I missing
something?

Thanks,
Sagi.