From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DE62C43381 for ; Thu, 21 Mar 2019 15:44:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 626D1218E2 for ; Thu, 21 Mar 2019 15:44:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=networkplumber-org.20150623.gappssmtp.com header.i=@networkplumber-org.20150623.gappssmtp.com header.b="gJufJROk" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728172AbfCUPog (ORCPT ); Thu, 21 Mar 2019 11:44:36 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:39928 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727823AbfCUPof (ORCPT ); Thu, 21 Mar 2019 11:44:35 -0400 Received: by mail-pg1-f194.google.com with SMTP id h8so4463000pgp.6 for ; Thu, 21 Mar 2019 08:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=m5JoclAZER6Be9HmkWVl8wn5UkTyaCiHMnZMhv1b0bQ=; b=gJufJROk088fOAc37jDMDGdlRZVXXZd+FNFEIQBzwc6AzzVu7Ng8inuWUcZ6q1QoQc +IOEXNReclEgCJvFGYkccXT0bpUVOuQmEUAPxj4DI/me/1YO6FAGEzkbyDz51MjLGHBT iTjbLq+Excpaz7yxIZ2Bu7ri7w4Dx/E8870w9+/7n/S5VSQ69xlG8jHzFVjzYz/jrZaJ IjprGATYlI6zO5ClF7QaovRc7cKSi9Fv+iWLNH/aEjsuFC1qf+cNBYSf/o5IftHiObBy FXboA4MVggRCk5+cR//fjw589B8lKB/fZmJtbNHNh7HWMhG+9+ldRS0Y75zlUgrTWld7 68xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=m5JoclAZER6Be9HmkWVl8wn5UkTyaCiHMnZMhv1b0bQ=; b=amC7fn4fOua6XFlAqood89qUxSlaR1aMkRg0R+kPjS88yHnxIg4jGbX3fajTmfWv+L 8+f7vR8xPKcoqHLf56E+mOoqXXdL8w63XXea5r7qJhu8mKsLyfe775tkvS41D9UPOZCO J++8+jaT1dpjiFwahGPrSQmQBT/aE1+KsFRYgDzF5wPdQLn08zk/bG/I3UMmrXVdgnlM S2kwoFu15TK2/Fe2rTRmR9FlHWuLlu8yaTPm4Xt/wKbf9X9AyCSktx1sm9KKr8rTWiQ9 HG2N69BDyTAYAZJs5Z4OYTMvasFluegs91LYUvyERo7I7lApXb3CG5ckCzntn4nziXrv GrSg== X-Gm-Message-State: APjAAAW2dHyl+X/R08cfKsD5A+NHzFoXQ8SwbPecFfIJl8js0wzF9PyX 4Cn4QNRZ/aLynsmi8B87/QXghg== X-Google-Smtp-Source: APXvYqzH3CAuo584Dcv4rosI5Flb1U89sGYR1LqTDCSJpQ8d5rgz5xHV2UBfPnk4BsxR5xucnGwFLQ== X-Received: by 2002:a63:5652:: with SMTP id g18mr4035848pgm.290.1553183074315; Thu, 21 Mar 2019 08:44:34 -0700 (PDT) Received: from shemminger-XPS-13-9360 (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id l64sm7456269pfj.179.2019.03.21.08.44.33 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 21 Mar 2019 08:44:34 -0700 (PDT) Date: Thu, 21 Mar 2019 08:44:30 -0700 From: Stephen Hemminger To: "Michael S. Tsirkin" Cc: Liran Alon , Si-Wei Liu , Sridhar Samudrala , Alexander Duyck , Jakub Kicinski , Jiri Pirko , David Miller , Netdev , virtualization@lists.linux-foundation.org, boris.ostrovsky@oracle.com, vijay.balakrishna@oracle.com, jfreimann@redhat.com, ogerlitz@mellanox.com, vuhuong@mellanox.com Subject: Re: [summary] virtio network device failover writeup Message-ID: <20190321084430.4750bd9b@shemminger-XPS-13-9360> In-Reply-To: <20190321085159-mutt-send-email-mst@kernel.org> References: <20190320061632-mutt-send-email-mst@kernel.org> <20190320100747-mutt-send-email-mst@kernel.org> <36772E22-7A8F-4C42-A731-398E3204B418@oracle.com> <20190320180641-mutt-send-email-mst@kernel.org> <20190321044920-mutt-send-email-mst@kernel.org> <20190321082532-mutt-send-email-mst@kernel.org> <20190321085159-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Thu, 21 Mar 2019 08:57:03 -0400 "Michael S. Tsirkin" wrote: > On Thu, Mar 21, 2019 at 02:47:50PM +0200, Liran Alon wrote: > >=20 > > =20 > > > On 21 Mar 2019, at 14:37, Michael S. Tsirkin wrote: > > >=20 > > > On Thu, Mar 21, 2019 at 12:07:57PM +0200, Liran Alon wrote: =20 > > >>>>>> 2) It brings non-intuitive customer experience. For example, a c= ustomer may attempt to analyse connectivity issue by checking the connectiv= ity > > >>>>>> on a net-failover slave (e.g. the VF) but will see no connectivi= ty when in-fact checking the connectivity on the net-failover master netdev= shows correct connectivity. > > >>>>>>=20 > > >>>>>> The set of changes I vision to fix our issues are: > > >>>>>> 1) Hide net-failover slaves in a different netns created and man= aged by the kernel. But that user can enter to it and manage the netdevs th= ere if wishes to do so explicitly. > > >>>>>> (E.g. Configure the net-failover VF slave in some special way). > > >>>>>> 2) Match the virtio-net and the VF based on a PV attribute inste= ad of MAC. (Similar to as done in NetVSC). E.g. Provide a virtio-net interf= ace to get PCI slot where the matching VF will be hot-plugged by hypervisor. > > >>>>>> 3) Have an explicit virtio-net control message to command hyperv= isor to switch data-path from virtio-net to VF and vice-versa. Instead of r= elying on intercepting the PCI master enable-bit > > >>>>>> as an indicator on when VF is about to be set up. (Similar to as= done in NetVSC). > > >>>>>>=20 > > >>>>>> Is there any clear issue we see regarding the above suggestion? > > >>>>>>=20 > > >>>>>> -Liran =20 > > >>>>>=20 > > >>>>> The issue would be this: how do we avoid conflicting with namespa= ces > > >>>>> created by users? =20 > > >>>>=20 > > >>>> This is kinda controversial, but maybe separate netns names into 2= groups: hidden and normal. > > >>>> To reference a hidden netns, you need to do it explicitly.=20 > > >>>> Hidden and normal netns names can collide as they will be maintain= ed in different namespaces (Yes I=E2=80=99m overloading the term namespace = here=E2=80=A6). =20 > > >>>=20 > > >>> Maybe it's an unnamed namespace. Hidden until userspace gives it a = name? =20 > > >>=20 > > >> This is also a good idea that will solve the issue. Yes. > > >> =20 > > >>> =20 > > >>>> Does this seems reasonable? > > >>>>=20 > > >>>> -Liran =20 > > >>>=20 > > >>> Reasonable I'd say yes, easy to implement probably no. But maybe I > > >>> missed a trick or two. =20 > > >>=20 > > >> BTW, from a practical point of view, I think that even until we figu= re out a solution on how to implement this, > > >> it was better to create an kernel auto-generated name (e.g. =E2=80= =9Ckernel_net_failover_slaves") > > >> that will break only userspace workloads that by a very rare-chance = have a netns that collides with this then > > >> the breakage we have today for the various userspace components. > > >>=20 > > >> -Liran =20 > > >=20 > > > It seems quite easy to supply that as a module parameter. Do we need = two > > > namespaces though? Won't some userspace still be confused by the two > > > slaves sharing the MAC address? =20 > >=20 > > That=E2=80=99s one reasonable option. > > Another one is that we will indeed change the mechanism by which we det= ermine a VF should be bonded with a virtio-net device. > > i.e. Expose a new virtio-net property that specify the PCI slot of the = VF to be bonded with. > >=20 > > The second seems cleaner but I don=E2=80=99t have a strong opinion on t= his. Both seem reasonable to me and your suggestion is faster to implement = from current state of things. > >=20 > > -Liran =20 >=20 > OK. Now what happens if master is moved to another namespace? Do we need > to move the slaves too? >=20 > Also siwei's patch is then kind of extraneous right? > Attempts to rename a slave will now fail as it's in a namespace... I did try moving slave device into a namespace at one point. The problem is that introduces all sorts of locking problems in the code because you can't do it directly in the context of when the callback happens that a new slave device is discovered. Since you can't safely change device namespace in the notifier, it requires a work queue. Then you add more complexity and error cases because the slave is exposed for a short period, and handling all the state race unwinds... Good idea but hard to implement