From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 2002:a17:505:564d:b0:1be9:327d:8ee3 with SMTP id jl13csp3293866njb; Tue, 9 Jul 2024 06:33:37 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCW4xcLLqYYHaBPO0+8iLvOkC0M9i7Y4hAADSXtfRQrDefvafNo2tWmVn8BcOxvcZ9JQeOV9mxmX0T5jf2f/bHcBWwAkY8gf X-Google-Smtp-Source: AGHT+IEMYgTdrzVDeRcga7XqK6IvkmcpL3JAG1SLDx/hkAj7JuIJUcEFo3CwefkMwxQi9l0dAl5C X-Received: by 2002:a05:622a:3:b0:447:c051:cb40 with SMTP id d75a77b69052e-447faa5be08mr28419761cf.58.1720532017277; Tue, 09 Jul 2024 06:33:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1720532017; cv=none; d=google.com; s=arc-20160816; b=z6fmPKV35CKohr9dkT5ou0YFQ5z9ncbev45JVgtJZRjHafJUsQFm5kuVb4WU6al3JZ cD8dWqS6liImVFnLALzaTwB43b5aNKJyQbzi1gRYwdPxIOHRczN8rJI7b9PbY5aA5mHp 1Iv6Qnbh96AgxTdI4WNwgHECRbBY5Cg+hCbu+OYF1m/A45TtTlAJhMvHZy2//0fqtEda 4nandcIjVRoTx9jccDCNH7Gdp48LoODFNoHLKd0WzhM1bU1zwxpTLoDns4z55lUa1Nsa 0D4u4RuAEJxeLkDgazVxmoxvagOIX7hp1fClz8xQEL+Y53NMOqMFFyMFoXwrr1w1iTsE W8Mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:reply-to:list-subscribe:list-help:list-post :list-archive:list-unsubscribe:list-id:precedence :content-transfer-encoding:content-language:in-reply-to:from :references:cc:to:subject:user-agent:mime-version:date:message-id :dkim-signature; bh=hcYNPiAxxZiFlbDUPTi8jag3xNX3lLeMyqZc5BZbBs4=; fh=+YM4KGCiyJ4tFP3cTVMGnasPgrFMaPT8/hVYokrMVk8=; b=E6PEmh61OMMMd5OSRuVwJvO3TIf/keQxPOt6Kp0SFqVqgXsvQqLqrfowra0v4Rj5NU +jnVg6ekS0TZwS1ADJ+CPTsxYzncftpLEzRXuDymJ7N2X1gWCJHSvE5Xv2NrFtoSbFNh iJK2Lr+TSftyzQf/cZ8x1lOvuqlugZ7FeXX/92NeCrsEy/V5xQ4KswFRsO3L7hTZ2tEX p5MdKw8JOgfglplelq4n39nbDXEAidHDc+JBrCQa3sDuXb/b1B2t9sFAFOVguBOKr7U5 8KrQM9GIFt7BcuJJGUVtSVvDI/r13phg17hN7mrhKAaDISTUkfximNqevG5pkil0a7Hf Gk9A==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZKzdDxIb; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id d75a77b69052e-447f9bdaccbsi21544961cf.532.2024.07.09.06.33.37 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 09 Jul 2024 06:33:37 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZKzdDxIb; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sRAy3-0002nk-61; Tue, 09 Jul 2024 09:33:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sRAxu-0002an-2L for qemu-arm@nongnu.org; Tue, 09 Jul 2024 09:33:06 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sRAxi-0008NK-HI for qemu-arm@nongnu.org; Tue, 09 Jul 2024 09:33:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720531972; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hcYNPiAxxZiFlbDUPTi8jag3xNX3lLeMyqZc5BZbBs4=; b=ZKzdDxIb0Tp6SOj+SK8la1MX4YqmXxYNRFyyd5LTB9n25196DbCgoPFNBHWpqFqBydxBSu Cf+m3ZaL4wC8tflKhloOcI7RhtjDCD7IHw4Li9y0dnNkC1oyMvUuZQYtrioHj5zxwHUm24 MIhyXEHFhdSLGxtXgGbW0PCevuJcQpk= Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-76-0rlry41XPYCCQ9bVnLlRWA-1; Tue, 09 Jul 2024 09:32:50 -0400 X-MC-Unique: 0rlry41XPYCCQ9bVnLlRWA-1 Received: by mail-oi1-f197.google.com with SMTP id 5614622812f47-3d92b366160so2511226b6e.3 for ; Tue, 09 Jul 2024 06:32:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720531970; x=1721136770; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:reply-to:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hcYNPiAxxZiFlbDUPTi8jag3xNX3lLeMyqZc5BZbBs4=; b=n14yWiGjHcgilicKk6F/8+vxJGi89aIxWsVVF+EocCzJ/BPCWFVJ3LtDFnrs2j+1L0 1F21veWOLSCmI+z+r/ecYwFQKvHr3yR5qbn0BxAUfWQzmtvOr6w4lXJMeEnEnfvVoocr zuTHGjrADtMgN2HhdDL5pdaSklohKCE+rpb2hDfSwOYdGNTjP/nxSglOBLxQN1jJM2kY IqU1k+GPiuK7WiRTLZVUXrIxXIm5YRocyoPhJ9kpVm3HkvPT3vIikqpREWkzQHoBFmlB U1NEkW4heIiLEzweEN90wpQDstDSLWCie9oRFQf8KDeV0NFgshdKM3740GIkw5Vr87E3 7RTA== X-Gm-Message-State: AOJu0Yzts+xFZrTXE+XFRBzAPd4ZvHc4t19w6aN3AUwSVaJUBGFsXmwn dbQZiejYUAYcGd9xA2xdh9+HZdMaNBC4/LFrrTnIxrdyXL/TQuRkLwuOmajeDRQ5TwNX0GJ8twh /4LQTTR99Al/Y1xHu7X7WEoxgQJEIqdbjjf1DEJ69OdcuTzciXQ== X-Received: by 2002:a05:6808:17a5:b0:3d2:23e0:d7aa with SMTP id 5614622812f47-3d93bef8b38mr3153318b6e.13.1720531969847; Tue, 09 Jul 2024 06:32:49 -0700 (PDT) X-Received: by 2002:a05:6808:17a5:b0:3d2:23e0:d7aa with SMTP id 5614622812f47-3d93bef8b38mr3153287b6e.13.1720531969386; Tue, 09 Jul 2024 06:32:49 -0700 (PDT) Received: from ?IPV6:2a01:e0a:59e:9d80:527b:9dff:feef:3874? ([2a01:e0a:59e:9d80:527b:9dff:feef:3874]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-447fa56d88csm9767081cf.27.2024.07.09.06.32.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 09 Jul 2024 06:32:48 -0700 (PDT) Message-ID: Date: Tue, 9 Jul 2024 15:32:41 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFCv1 06/10] hw/arm/virt: Assign vfio-pci devices to nested SMMUs To: Nicolin Chen , peter.maydell@linaro.org, shannon.zhaosl@gmail.com, mst@redhat.com, imammedo@redhat.com, anisinha@redhat.com, peterx@redhat.com Cc: qemu-arm@nongnu.org, qemu-devel@nongnu.org, jgg@nvidia.com, shameerali.kolothum.thodi@huawei.com, jasowang@redhat.com, Andrea Bolognani References: <67c6311756de2a6e827e3dd0563f939dcf334418.1719361174.git.nicolinc@nvidia.com> From: Eric Auger In-Reply-To: <67c6311756de2a6e827e3dd0563f939dcf334418.1719361174.git.nicolinc@nvidia.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=170.10.133.124; envelope-from=eric.auger@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.144, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-arm@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: eric.auger@redhat.com Errors-To: qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org Sender: qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org X-TUID: ShhUhdLXI0GB On 6/26/24 02:28, Nicolin Chen wrote: > With iommu=nested-smmuv3, there could be multiple nested SMMU instances in > the vms. A passthrough device must to look up for its iommu handler in its > sysfs node, and then link to the nested SMMU instance created for the same > iommu handler. This isn't easy to do. > > Add an auto-assign piece after all vSMMU backed pxb buses are created. It > loops the existing input devices, and sets/replaces their pci bus numbers > with a newly created pcie-root-port to the pxb bus. Here again I don't think it is acceptable to create such topology under the hood. Libvirt shall master the whole PCIe topology. Eric > > Note that this is not an ideal solution to handle hot plug device. > > Signed-off-by: Nicolin Chen > --- > hw/arm/virt.c | 110 ++++++++++++++++++++++++++++++++++++++++++ > include/hw/arm/virt.h | 13 +++++ > 2 files changed, 123 insertions(+) > > diff --git a/hw/arm/virt.c b/hw/arm/virt.c > index a54332fca8..3610f53304 100644 > --- a/hw/arm/virt.c > +++ b/hw/arm/virt.c > @@ -38,6 +38,7 @@ > #include "hw/arm/primecell.h" > #include "hw/arm/virt.h" > #include "hw/block/flash.h" > +#include "hw/vfio/pci.h" > #include "hw/vfio/vfio-calxeda-xgmac.h" > #include "hw/vfio/vfio-amd-xgbe.h" > #include "hw/display/ramfb.h" > @@ -1491,6 +1492,112 @@ static void create_virtio_iommu_dt_bindings(VirtMachineState *vms) > bdf + 1, vms->iommu_phandle, bdf + 1, 0xffff - bdf); > } > > +static char *create_new_pcie_port(VirtNestedSmmu *nested_smmu, Error **errp) > +{ > + uint32_t port_nr = nested_smmu->pci_bus->qbus.num_children; > + uint32_t chassis_nr = UINT8_MAX - nested_smmu->index; > + uint32_t bus_nr = pci_bus_num(nested_smmu->pci_bus); > + DeviceState *dev; > + char *name_port; > + > + /* Create a root port */ > + dev = qdev_new("pcie-root-port"); > + name_port = g_strdup_printf("smmu_bus0x%x_port%d", bus_nr, port_nr); > + > + if (!qdev_set_id(dev, name_port, &error_fatal)) { > + /* FIXME retry with a different port num? */ > + error_setg(errp, "Could not set pcie-root-port ID %s", name_port); > + g_free(name_port); > + g_free(dev); > + return NULL; > + } > + qdev_prop_set_uint32(dev, "chassis", chassis_nr); > + qdev_prop_set_uint32(dev, "slot", port_nr); > + qdev_prop_set_uint64(dev, "io-reserve", 0); > + qdev_realize_and_unref(dev, BUS(nested_smmu->pci_bus), &error_fatal); > + return name_port; > +} > + > +static int assign_nested_smmu(void *opaque, QemuOpts *opts, Error **errp) > +{ > + VirtMachineState *vms = (VirtMachineState *)opaque; > + const char *sysfsdev = qemu_opt_get(opts, "sysfsdev"); > + const char *iommufd = qemu_opt_get(opts, "iommufd"); > + const char *driver = qemu_opt_get(opts, "driver"); > + const char *host = qemu_opt_get(opts, "host"); > + const char *bus = qemu_opt_get(opts, "bus"); > + VirtNestedSmmu *nested_smmu; > + char *link_iommu; > + char *dir_iommu; > + char *smmu_node; > + char *name_port; > + int ret = 0; > + > + if (!iommufd || !driver) { > + return 0; > + } > + if (!sysfsdev && !host) { > + return 0; > + } > + if (strncmp(driver, TYPE_VFIO_PCI, strlen(TYPE_VFIO_PCI))) { > + return 0; > + } > + /* If the device wants to attach to the default bus, do not reassign it */ > + if (bus && !strncmp(bus, "pcie.0", strlen(bus))) { > + return 0; > + } > + > + if (sysfsdev) { > + link_iommu = g_strdup_printf("%s/iommu", sysfsdev); > + } else { > + link_iommu = g_strdup_printf("/sys/bus/pci/devices/%s/iommu", host); > + } > + > + dir_iommu = realpath(link_iommu, NULL); > + if (!dir_iommu) { > + error_setg(errp, "Could not get the real path for iommu link: %s", > + link_iommu); > + ret = -EINVAL; > + goto free_link; > + } > + > + smmu_node = g_path_get_basename(dir_iommu); > + if (!smmu_node) { > + error_setg(errp, "Could not get SMMU node name for iommu at: %s", > + dir_iommu); > + ret = -EINVAL; > + goto free_dir; > + } > + > + nested_smmu = find_nested_smmu_by_sysfs(vms, smmu_node); > + if (!nested_smmu) { > + error_setg(errp, "Could not find any detected SMMU matching node: %s", > + smmu_node); > + ret = -EINVAL; > + goto free_node; > + } > + > + name_port = create_new_pcie_port(nested_smmu, errp); > + if (!name_port) { > + ret = -EBUSY; > + goto free_node; > + } > + > + qemu_opt_set(opts, "bus", name_port, &error_fatal); > + if (bus) { > + error_report("overriding PCI bus %s to %s for device %s [%s]", > + bus, name_port, host, sysfsdev); > + } > + > +free_node: > + free(smmu_node); > +free_dir: > + free(dir_iommu); > +free_link: > + free(link_iommu); > + return ret; > +} > + > /* > * FIXME this is used to reverse for hotplug devices, yet it could result in a > * big waste of PCI bus numbners. > @@ -1669,6 +1776,9 @@ static void create_pcie(VirtMachineState *vms) > qemu_fdt_setprop_cells(ms->fdt, nodename, "iommu-map", 0x0, > vms->nested_smmu_phandle[i], 0x0, 0x10000); > } > + > + qemu_opts_foreach(qemu_find_opts("device"), > + assign_nested_smmu, vms, &error_fatal); > } else if (vms->iommu) { > vms->iommu_phandle = qemu_fdt_alloc_phandle(ms->fdt); > > diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h > index 0a3f1ab8b5..dfbc4bba3c 100644 > --- a/include/hw/arm/virt.h > +++ b/include/hw/arm/virt.h > @@ -246,4 +246,17 @@ find_nested_smmu_by_index(VirtMachineState *vms, int index) > return NULL; > } > > +static inline VirtNestedSmmu * > +find_nested_smmu_by_sysfs(VirtMachineState *vms, char *node) > +{ > + VirtNestedSmmu *nested_smmu; > + > + QLIST_FOREACH(nested_smmu, &vms->nested_smmu_list, next) { > + if (!strncmp(nested_smmu->smmu_node, node, strlen(node))) { > + return nested_smmu; > + } > + } > + return NULL; > +} > + > #endif /* QEMU_ARM_VIRT_H */