From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3ACD6C5CFC2 for ; Thu, 15 Jul 2021 17:34:24 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F1C6261357 for ; Thu, 15 Jul 2021 17:34:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F1C6261357 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1Em1RpGJzKuL3YxBEmnjwNw4odSdcopO+AlAPlYTDKY=; b=xpFA7FSyoB83zU 5Pg29Icgzw3cm+cmcyy972TjvPKAC81ywGCGefFRrePZzEw1X7io+0YzpU0DRnqy7sDHdTnVunsxu hnUDOVKOpov7ah8RPCIsm/enie+qSPVMvkQ9F1FHKcxgg1OP58DLKd3ZWuXxQdrxZlbWuCZOd+ami RC92xdu7RUZmjZI17gSSyI5p5L324haWQGafh3VVvtjmGlpS/BrtV3WTjiP1GBrpFKKCOObGtg18L 1N0ricjLWzsCzMS/6DBEvNrYEoTI99PQSnLbV9SJVdoAPS5s7JVyQwY0aq1EPYXdi2YADpUrHEskB DARRc7lgh8FvZSxhZ4nA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1m45E5-001kfh-NK; Thu, 15 Jul 2021 17:32:46 +0000 Received: from mail-qv1-xf29.google.com ([2607:f8b0:4864:20::f29]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1m45Dz-001kZd-G1 for linux-arm-kernel@lists.infradead.org; Thu, 15 Jul 2021 17:32:41 +0000 Received: by mail-qv1-xf29.google.com with SMTP id h18so3247157qve.1 for ; Thu, 15 Jul 2021 10:32:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ZMhm9Q1ft2ZuJHnsgYob9UiHLcsqHs1PTnUjZmty0Ss=; b=de2SvrF1oRHJ1RaUTUgDxztdq/aLJitMAt74d4K3e+TPceDirxz+g7MUVorInP8RCe n4Y/Wh8ocWciGZp/lgpKTFdM8CQTP8iquH75D7ETaN4eGuQ8CG2fQga5yawfHzjH46is geb98vOpP6lOVVVCq+WvVQfxyNb7mJwi7Fe17kojvJXI+mSKSDoRcinUg1Ywz+Mnh6HL ujVKAhBjfkiBuUHKAQx1P3ko4Fyf9zR70NH8PRPybe/xXdgkfHG4LDx8GabXdHjliuGV ubFDcwe/v5npezWOmPbfvrGu5rXaoigwILGnDu0HFBtUtFhmzxt/QQ/RmqZkd3redq+p vYag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ZMhm9Q1ft2ZuJHnsgYob9UiHLcsqHs1PTnUjZmty0Ss=; b=VEHfyySnr/BN7z7bznowDbJ/ShU40Dr0tgaIUS4l5MuXJFtfAnqdM63kHxUHGNnw+i DPhBdcMEkVrHUT0roQAFkK94d7lLrTwyN9AkMx3JfiTP0pb0viVxRudGkL5WODTBxOV3 0K63++VO4VbyRDXqXYXOLCO9P5a4pNBzGJi62S5zCPaPL4iluDAR/l5KkSa7mcCexDZq Hc9wK8+H/VY/RRVHyaj9M2DJioEh+DjKXiBrRhjrkutBfZx5OCpoVZBpTZLPJ2L3ME/8 a7Ea33Lr3px5rJIJpTqVEA5vFBvquohXGPU4lR2me4aaBkNLh1NeNQT1ijp1a7G2qCN5 lyBA== X-Gm-Message-State: AOAM530tzMSZEn8RdrQ7exrt79Fn2oXJ71m0vigIMqtGUbEWcdgCk5GU +0qfbCTI3ZlTQ7dr4BspZZ4= X-Google-Smtp-Source: ABdhPJzKHeH+osOcaM58zeKsiokt6uEE3J17OoNO8AREwf9A+BMGPyp6zlg0BpheemYCI32KDnDO2A== X-Received: by 2002:ad4:5343:: with SMTP id v3mr5422776qvs.45.1626370357369; Thu, 15 Jul 2021 10:32:37 -0700 (PDT) Received: from auth2-smtp.messagingengine.com (auth2-smtp.messagingengine.com. [66.111.4.228]) by smtp.gmail.com with ESMTPSA id p21sm2801615qki.36.2021.07.15.10.32.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Jul 2021 10:32:36 -0700 (PDT) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailauth.nyi.internal (Postfix) with ESMTP id E4C6E27C005A; Thu, 15 Jul 2021 13:32:35 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Thu, 15 Jul 2021 13:32:35 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrvddtgdejjecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkfhggtggujgesthdtredttddtvdenucfhrhhomhepuehoqhhunhcu hfgvnhhguceosghoqhhunhdrfhgvnhhgsehgmhgrihhlrdgtohhmqeenucggtffrrghtth gvrhhnpedvleeigedugfegveejhfejveeuveeiteejieekvdfgjeefudehfefhgfegvdeg jeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegsoh hquhhnodhmvghsmhhtphgruhhthhhpvghrshhonhgrlhhithihqdeiledvgeehtdeigedq udejjeekheehhedvqdgsohhquhhnrdhfvghngheppehgmhgrihhlrdgtohhmsehfihigmh gvrdhnrghmvg X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 15 Jul 2021 13:32:34 -0400 (EDT) Date: Fri, 16 Jul 2021 01:30:52 +0800 From: Boqun Feng To: Bjorn Helgaas Cc: Bjorn Helgaas , Arnd Bergmann , Marc Zyngier , Catalin Marinas , Will Deacon , "K. Y. Srinivasan" , Haiyang Zhang , Stephen Hemminger , Wei Liu , Dexuan Cui , Lorenzo Pieralisi , Rob Herring , Krzysztof =?utf-8?Q?Wilczy=C5=84ski?= , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pci@vger.kernel.org, Sunil Muthuswamy , Mike Rapoport Subject: Re: [RFC v4 1/7] PCI: Introduce domain_nr in pci_host_bridge Message-ID: References: <20210714102737.198432-2-boqun.feng@gmail.com> <20210714193319.GA1867593@bjorn-Precision-5520> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210714193319.GA1867593@bjorn-Precision-5520> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210715_103239_611454_C0B35D5E X-CRM114-Status: GOOD ( 48.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jul 14, 2021 at 02:33:19PM -0500, Bjorn Helgaas wrote: > On Wed, Jul 14, 2021 at 06:27:31PM +0800, Boqun Feng wrote: > > Currently we retrieve the PCI domain number of the host bridge from the > > bus sysdata (or pci_config_window if PCI_DOMAINS_GENERIC=y). Actually > > we have the information at PCI host bridge probing time, and it makes > > sense that we store it into pci_host_bridge. One benefit of doing so is > > the requirement for supporting PCI on Hyper-V for ARM64, because the > > host bridge of Hyper-V doesn't have pci_config_window, whereas ARM64 is > > a PCI_DOMAINS_GENERIC=y arch, so we cannot retrieve the PCI domain > > number from pci_config_window on ARM64 Hyper-V guest. > > > > As the preparation for ARM64 Hyper-V PCI support, we introduce the > > domain_nr in pci_host_bridge and a sentinel value to allow drivers to > > set domain numbers properly at probing time. Currently > > CONFIG_PCI_DOMAINS_GENERIC=y archs are only users of this > > newly-introduced field. > > Thanks for pushing on this. PCI_DOMAINS_GENERIC is really not very > generic today and it will be good to make it more so. > > > Signed-off-by: Boqun Feng > > --- > > drivers/pci/probe.c | 6 +++++- > > include/linux/pci.h | 10 ++++++++++ > > 2 files changed, 15 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > > index 79177ac37880..60c50d4f156f 100644 > > --- a/drivers/pci/probe.c > > +++ b/drivers/pci/probe.c > > @@ -594,6 +594,7 @@ static void pci_init_host_bridge(struct pci_host_bridge *bridge) > > bridge->native_pme = 1; > > bridge->native_ltr = 1; > > bridge->native_dpc = 1; > > + bridge->domain_nr = PCI_DOMAIN_NR_NOT_SET; > > > > device_initialize(&bridge->dev); > > } > > @@ -898,7 +899,10 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge) > > bus->ops = bridge->ops; > > bus->number = bus->busn_res.start = bridge->busnr; > > #ifdef CONFIG_PCI_DOMAINS_GENERIC > > - bus->domain_nr = pci_bus_find_domain_nr(bus, parent); > > + if (bridge->domain_nr == PCI_DOMAIN_NR_NOT_SET) > > + bus->domain_nr = pci_bus_find_domain_nr(bus, parent); > > + else > > + bus->domain_nr = bridge->domain_nr; > > The domain_nr in struct pci_bus is really only used by > pci_domain_nr(). It seems like it really belongs in the struct > pci_host_bridge and probably doesn't need to be duplicated in the > struct pci_bus. But that's probably a project for the future. > Agreed. Maybe we can define pci_bus_domain_nr() as: static inline int pci_domain_nr(struct pci_bus *bus) { struct device *bridge = bus->bridge; struct pci_host_bridge *b = container_of(bridge, struct pci_host_bridge, dev); return b->domain_nr; } but apart from corretness (e.g. should we use get_device() for bus->bridge?), it makes more sense if ->domain_nr of pci_host_bridge is used (as a way to set domain number at probing time) for most of drivers and archs. ;-) > > #endif > > > > b = pci_find_bus(pci_domain_nr(bus), bridge->busnr); > > diff --git a/include/linux/pci.h b/include/linux/pci.h > > index 540b377ca8f6..952bb7d46576 100644 > > --- a/include/linux/pci.h > > +++ b/include/linux/pci.h > > @@ -526,6 +526,15 @@ static inline int pci_channel_offline(struct pci_dev *pdev) > > return (pdev->error_state != pci_channel_io_normal); > > } > > > > +/* > > + * PCI Conventional has at most 256 PCI bus segments and PCI Express has at > > + * most 65536 "PCI Segments Groups", therefore -1 is not a valid PCI domain > > s/Segments/Segment/ > > Do you have a reference for these limits? I don't think either > Conventional PCI or PCIe actually specifies a hardware limit on the > number of domains (I think PCI uses "segment group" to mean the same > thing). > > "Segment" in the Conventional PCI spec, r3.0, means a bus segment, > which connects all the devices on a single bus. Obviously there's a > limit of 256 buses under a single host bridge, but that's different > concept than a domain/segment group. > > The PCI Firmware spec, r3.3, defines "Segment Group Number" as being > in the range 0..65535, but as far as I know, that's just a firmware > issue, and it applies equally to Conventional PCI and PCIe. > > I think you're right that -1 is a reasonable sentinel; I just don't > want to claim a difference here unless there really is one. > I think you're right, I got confused on the concepts of "Segment" and "Segment Group". After digging in specs, I haven't find any difference on the limitation between Conventional PCI and PCIe. The PCI Firmware spec, r3.2, refers ACPI (3.0 and later) spec for the details of "Segment Group", and in ACPI spec v6.3, the description _SEG object says: """ The lower 16 bits of _SEG returned integer is the PCI Segment Group number. Other bits are reserved. """ So I'm thinking replacing the comments with: Currently in ACPI spec, for each PCI host bridge, PCI Segment Group number is limited to a 16-bit value, therefore (int)-1 is not a valid PCI domain number, and can be used as a sentinel value indicating ->domain_nr is not set by the driver (and CONFIG_PCI_DOMAINS_GENERIC=y archs will set it with pci_bus_find_domain_nr()). Thoughts? Regards, BOqun > > + * number, and can be used as a sentinel value indicating ->domain_nr is not > > + * set by the driver (and CONFIG_PCI_DOMAINS_GENERIC=y can set it in generic > > + * code). > > + */ > > +#define PCI_DOMAIN_NR_NOT_SET (-1) > > + > > struct pci_host_bridge { > > struct device dev; > > struct pci_bus *bus; /* Root bus */ > > @@ -533,6 +542,7 @@ struct pci_host_bridge { > > struct pci_ops *child_ops; > > void *sysdata; > > int busnr; > > + int domain_nr; > > struct list_head windows; /* resource_entry */ > > struct list_head dma_ranges; /* dma ranges resource list */ > > u8 (*swizzle_irq)(struct pci_dev *, u8 *); /* Platform IRQ swizzler */ > > -- > > 2.30.2 > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel