From mboxrd@z Thu Jan  1 00:00:00 1970
From: Ilya Maximets <i.maximets@samsung.com>
Subject: Re: Memory leak when adding/removing vhost_user ports
Date: Thu, 21 Apr 2016 14:01:26 +0300
Message-ID: <5718B306.5070801@samsung.com>
References: <CAATJJ0JLd9iqNBvD=kyfpzk5DjGF_pGz+hBNeionZpuctNBymQ@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Cc: Daniele Di Proietto <diproiettod@vmware.com>,
 "dev@openvswitch.org" <dev@openvswitch.org>,
 "yuanhan.liu@linux.intel.com" <yuanhan.liu@linux.intel.com>,
 Dyasly Sergey <s.dyasly@samsung.com>
To: dev@dpdk.org, christian.ehrhardt@canonical.com
Return-path: <dev-bounces@dpdk.org>
Received: from mailout3.w1.samsung.com (mailout3.w1.samsung.com
 [210.118.77.13]) by dpdk.org (Postfix) with ESMTP id 0F99CA2F
 for <dev@dpdk.org>; Thu, 21 Apr 2016 13:01:29 +0200 (CEST)
Received: from eucpsbgm2.samsung.com (unknown [203.254.199.245])
 by mailout3.w1.samsung.com
 (Oracle Communications Messaging Server 7.0.5.31.0 64bit (built May 5 2014))
 with ESMTP id <0O5Z0049WDAFEJ30@mailout3.w1.samsung.com> for dev@dpdk.org;
 Thu, 21 Apr 2016 12:01:27 +0100 (BST)
In-reply-to: <CAATJJ0JLd9iqNBvD=kyfpzk5DjGF_pGz+hBNeionZpuctNBymQ@mail.gmail.com>
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

Hi, Christian.
You're, likely, using tar archive with openvswitch from openvswitch.org.
It doesn't contain many bug fixes from git/branch-2.5 unfortunately.

The problem that you are facing has been solved in branch-2.5 by

commit d9df7b9206831631ddbd90f9cbeef1b4fc5a8e89
Author: Ilya Maximets <i.maximets@samsung.com>
Date:   Thu Mar 3 11:30:06 2016 +0300

    netdev-dpdk: Fix memory leak in netdev_dpdk_vhost_destruct().
    
    Fixes: 4573fbd38fa1 ("netdev-dpdk: Add vhost-user multiqueue support")
    Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
    Acked-by: Flavio Leitner <fbl@sysclose.org>
    Acked-by: Daniele Di Proietto <diproiettod@vmware.com>

Best regards, Ilya Maximets.

> I assume there is a leak somewhere on adding/removing vhost_user ports.
> Although it could also be "only" a fragmentation issue.
> 
> Reproduction is easy:
> I set up a pair of nicely working OVS-DPDK connected KVM Guests.
> Then in a loop I
>    - add up to more 512 ports
>    - test connectivity between the two guests
>    - remove up to 512 ports
> 
> Depending on memory and the amount of multiqueue/rxq I use it seems to
> slightly change when exactly it breaks. But for my default setup of 4
> queues and 5G Hugepages initialized by DPDK it always breaks at the sixth
> iteration.
> Here a link to the stack trace indicating a memory shortage (TBC):
> https://launchpadlibrarian.net/253916410/apport-retrace.log
> 
> Known Todos:
> - I want to track it down more, and will try to come up with a non
> openvswitch based looping testcase that might show it as well to simplify
> debugging.
> - in use were Openvswitch-dpdk 2.5 and DPDK 2.2; Retest with DPDK 16.04 and
> Openvswitch master is planned.
> 
> I will go on debugging this and let you know, but I wanted to give a heads
> up to everyone.
> In case this is a known issue for some of you please let me know.
> 
> Kind Regards,
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
> 
> P.S. I think it is a dpdk issue, but adding Daniele on CC to represent
> ovs-dpdk as well.