From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 864B3C43381 for ; Tue, 26 Feb 2019 01:39:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 249C5217F4 for ; Tue, 26 Feb 2019 01:39:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=networkplumber-org.20150623.gappssmtp.com header.i=@networkplumber-org.20150623.gappssmtp.com header.b="BUTvEVg/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725963AbfBZBjS (ORCPT ); Mon, 25 Feb 2019 20:39:18 -0500 Received: from mail-pl1-f193.google.com ([209.85.214.193]:42448 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725842AbfBZBjS (ORCPT ); Mon, 25 Feb 2019 20:39:18 -0500 Received: by mail-pl1-f193.google.com with SMTP id s1so5364800plp.9 for ; Mon, 25 Feb 2019 17:39:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eo+rd415hpAAFNaAJb2FJoQUR4vymlvxgxEllwwA5s4=; b=BUTvEVg/DucPHWGACypz7H/XJlURW9SOvf+P/M6ME03UvCQzqpl/y9RoYopgPN+l2B 2YgwKmtUINT+WZ0lrusjpDxCtqXmXYVkwomML3oPZUA3vJyBWpUBSRYC+4mo4biPwDO6 LeXKU9fFqswGdkj1gzVDGTkR3tTmt+YR+Y9OCu5EGw2IYGTVmXSx9hLJDbUklGZMRshb jH/htWC1R7caXqM2MULlgh/537ob7rIG3N29RQdHnh22KOr2yyZDvl5PQKDZfjBIXuZ7 qWktcVGsLSgTcIunEDaOgEczg4OP0fr7e4rckI8DWsdQ9IOir2VOg0GFjQ7RKbC2/6VG jX/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eo+rd415hpAAFNaAJb2FJoQUR4vymlvxgxEllwwA5s4=; b=ePLkK9N9rDdEIa45Jj4b9DFg0zee8sSSgGSTwgLUqA9Gw1BR0p7lOTj9HT/pZNKkOK 8ygbvnOirukwCRZP9iRrihMMssstB0NrQTBxOWd+fLYeL+5BBfdXOekvngCTPmsmjneg 0HLGAfvkeJDTMRGMxbg5Xk3DETXca2NJiZhPqbMMWwRMrRFEpyDgRL3HTS1g6Tm4VBFS t37t9qwzO9kz33zS2IeW8qeusiF2VLuIkWSL1mirsS+jqGqleY+rxA19ZW/LUqSo6tgs lBw+BRCXMDqMbvlQchPCAYdOkrwPXKvaEtwAFB4W2lOJQi8MdmF0Xge9Y4QCd2+KokAh JvTw== X-Gm-Message-State: AHQUAuax3C2fjkNF8LXAJq31PskfznvB8/ualyBqL8rWiBo3hTVqYQ2B rJzzG9CeZeColHk9S1MyOxXQaQ== X-Google-Smtp-Source: AHgI3IYFY7w2+Vm0JjZp4QR6yr13wwkNGpNWRnH/Ash5nd54EjQwjuvXJmWbaDXTopzg28TcZXy3ZQ== X-Received: by 2002:a17:902:2e03:: with SMTP id q3mr24102672plb.330.1551145156721; Mon, 25 Feb 2019 17:39:16 -0800 (PST) Received: from shemminger-XPS-13-9360 (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id u67sm16235446pfu.51.2019.02.25.17.39.16 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 25 Feb 2019 17:39:16 -0800 (PST) Date: Mon, 25 Feb 2019 17:39:12 -0800 From: Stephen Hemminger To: si-wei liu Cc: "Michael S. Tsirkin" , "Samudrala, Sridhar" , Siwei Liu , Jiri Pirko , David Miller , Netdev , virtualization@lists.linux-foundation.org, virtio-dev , "Brandeburg, Jesse" , Alexander Duyck , Jakub Kicinski , Jason Wang , liran.alon@oracle.com Subject: Re: [virtio-dev] Re: net_failover slave udev renaming (was Re: [RFC PATCH net-next v6 4/4] netvsc: refactor notifier/event handling code to use the bypass framework) Message-ID: <20190225173912.26b93422@shemminger-XPS-13-9360> In-Reply-To: References: <1523386790-12396-1-git-send-email-sridhar.samudrala@intel.com> <1523386790-12396-5-git-send-email-sridhar.samudrala@intel.com> <20180410142608.50f15b45@xeon-e3> <20180411075334.GK2028@nanopsycho> <20190221203808-mutt-send-email-mst@kernel.org> <581e4399-3969-aecd-e923-03bbc0880733@oracle.com> <91d4cbb1-be7a-b53c-6b2a-99bef07e7c53@intel.com> <20190222100753-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Mon, 25 Feb 2019 16:58:07 -0800 si-wei liu wrote: > On 2/22/2019 7:14 AM, Michael S. Tsirkin wrote: > > On Thu, Feb 21, 2019 at 11:55:11PM -0800, si-wei liu wrote: > >> > >> On 2/21/2019 11:00 PM, Samudrala, Sridhar wrote: > >>> > >>> On 2/21/2019 7:33 PM, si-wei liu wrote: > >>>> > >>>> On 2/21/2019 5:39 PM, Michael S. Tsirkin wrote: > >>>>> On Thu, Feb 21, 2019 at 05:14:44PM -0800, Siwei Liu wrote: > >>>>>> Sorry for replying to this ancient thread. There was some remaining > >>>>>> issue that I don't think the initial net_failover patch got addressed > >>>>>> cleanly, see: > >>>>>> > >>>>>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1815268 > >>>>>> > >>>>>> The renaming of 'eth0' to 'ens4' fails because the udev userspace was > >>>>>> not specifically writtten for such kernel automatic enslavement. > >>>>>> Specifically, if it is a bond or team, the slave would typically get > >>>>>> renamed *before* virtual device gets created, that's what udev can > >>>>>> control (without getting netdev opened early by the other part of > >>>>>> kernel) and other userspace components for e.g. initramfs, > >>>>>> init-scripts can coordinate well in between. The in-kernel > >>>>>> auto-enslavement of net_failover breaks this userspace convention, > >>>>>> which don't provides a solution if user care about consistent naming > >>>>>> on the slave netdevs specifically. > >>>>>> > >>>>>> Previously this issue had been specifically called out when IFF_HIDDEN > >>>>>> and the 1-netdev was proposed, but no one gives out a solution to this > >>>>>> problem ever since. Please share your mind how to proceed and solve > >>>>>> this userspace issue if netdev does not welcome a 1-netdev model. > >>>>> Above says: > >>>>> > >>>>> there's no motivation in the systemd/udevd community at > >>>>> this point to refactor the rename logic and make it work well with > >>>>> 3-netdev. > >>>>> > >>>>> What would the fix be? Skip slave devices? > >>>>> > >>>> There's nothing user can get if just skipping slave devices - the > >>>> name is still unchanged and unpredictable e.g. eth0, or eth1 the > >>>> next reboot, while the rest may conform to the naming scheme (ens3 > >>>> and such). There's no way one can fix this in userspace alone - when > >>>> the failover is created the enslaved netdev was opened by the kernel > >>>> earlier than the userspace is made aware of, and there's no > >>>> negotiation protocol for kernel to know when userspace has done > >>>> initial renaming of the interface. I would expect netdev list should > >>>> at least provide the direction in general for how this can be > >>>> solved... > > > > I was just wondering what did you mean when you said > > "refactor the rename logic and make it work well with 3-netdev" - > > was there a proposal udev rejected? > No. I never believed this particular issue can be fixed in userspace > alone. Previously someone had said it could be, but I never see any work > or relevant discussion ever happened in various userspace communities > (for e.g. dracut, initramfs-tools, systemd, udev, and NetworkManager). > IMHO the root of the issue derives from the kernel, it makes more sense > to start from netdev, work out and decide on a solution: see what can be > done in the kernel in order to fix it, then after that engage userspace > community for the feasibility... > > > Anyway, can we write a time diagram for what happens in which order that > > leads to failure? That would help look for triggers that we can tie > > into, or add new ones. > > > > See attached diagram. > > > > > > > > > > >>> Is there an issue if slave device names are not predictable? The user/admin scripts are expected > >>> to only work with the master failover device. > >> Where does this expectation come from? > >> > >> Admin users may have ethtool or tc configurations that need to deal with > >> predictable interface name. Third-party app which was built upon specifying > >> certain interface name can't be modified to chase dynamic names. > >> > >> Specifically, we have pre-canned image that uses ethtool to fine tune VF > >> offload settings post boot for specific workload. Those images won't work > >> well if the name is constantly changing just after couple rounds of live > >> migration. > > It should be possible to specify the ethtool configuration on the > > master and have it automatically propagated to the slave. > > > > BTW this is something we should look at IMHO. > I was elaborating a few examples that the expectation and assumption > that user/admin scripts only deal with master failover device is > incorrect. It had never been taken good care of, although I did try to > emphasize it from the very beginning. > > Basically what you said about propagating the ethtool configuration down > to the slave is the key pursuance of 1-netdev model. However, what I am > seeking now is any alternative that can also fix the specific udev > rename problem, before concluding that 1-netdev is the only solution. > Generally a 1-netdev scheme would take time to implement, while I'm > trying to find a way out to fix this particular naming problem under > 3-netdev. > > > > >>> Moreover, you were suggesting hiding the lower slave devices anyway. There was some discussion > >>> about moving them to a hidden network namespace so that they are not visible from the default namespace. > >>> I looked into this sometime back, but did not find the right kernel api to create a network namespace within > >>> kernel. If so, we could use this mechanism to simulate a 1-netdev model. > >> Yes, that's one possible implementation (IMHO the key is to make 1-netdev > >> model as much transparent to a real NIC as possible, while a hidden netns is > >> just the vehicle). However, I recall there was resistance around this > >> discussion that even the concept of hiding itself is a taboo for Linux > >> netdev. I would like to summon potential alternatives before concluding > >> 1-netdev is the only solution too soon. > >> > >> Thanks, > >> -Siwei > > Your scripts would not work at all then, right? > At this point we don't claim images with such usage as SR-IOV live > migrate-able. We would flag it as live migrate-able until this ethtool > config issue is fully addressed and a transparent live migration > solution emerges in upstream eventually. The hyper-v netvsc with 1-dev model uses a timeout to allow udev to do its rename. I proposed a patch to key state change off of the udev rename, but that patch was rejected.