From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 253512206E for ; Fri, 8 Mar 2024 02:53:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.137.202.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709866422; cv=none; b=oIFqz5pXOteqqnozdwA1H9lApsQ4T3nGKY8yY/ZcDw3HTxNdL01CaxwOGfXrPW+Ph8pURcXJ9FRWnz8peTPV86r5Va90u06Ga6ezc28olY4AtyNv/a9szf605RQmGpu1qxSqj6czrRG/pajFPYHSsP9hZIrkw22gxnZkuJRqTnw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709866422; c=relaxed/simple; bh=4gErI6XB5XcKl7INpCRiauEQFJc4HCnSi6IakJcGHzY=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=VhNGCfb4ns6W+EDdg/KqKmiuVXmWbpGXVJn4REoqZc4yZ1jwOThJFOGM6J+/ksz5NE5zVfpmyZOmK9JLjg865jjZHBV3K5qS6edRS3R6zgKaDCBipZVkIWsOCs7B43sEFBAfW2CFA6eTJCHRiNmDpqGH/TA7PazKMRNgKPKnxoA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=2VacbEwn; arc=none smtp.client-ip=198.137.202.133 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="2VacbEwn" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-Type: Content-ID:Content-Description:In-Reply-To:References; bh=g7HPNFt9vw7oLuJ9BObDVUOTEKdXSxwswiJlSLCWN7I=; b=2VacbEwnU7bK9kqknp+u0dcMHL E87Yzo2P7wX7Xhi+5QGIJsChKRHyedl4+ufglShQ22opm8kTd/g+AkPCO5CSGzd61hGgBdCD1JnSa ZDGcx/GnuTt7Rj2ojyReWLV6YfZbv30YBYkDyDuF+yHeRVCxyKoJUleu6fBcb3epfXkS+W66UWg2N 7YPl0xc0B93cM58RT9ruCQ/so94c/EcQxFzDVcsyUb8A2IvNCUe5LtmlTlJyB4AzODxR55neTwqvc 6UKCZQdYZ9F/r1vfO2AsaYVh+NlfG/RPb9CMx0UygR1+zWSWNyFyvfcLminaM36iMeZ6vUhyXRcBP XPib/mnQ==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1riQMd-00000007Q5B-01mW; Fri, 08 Mar 2024 02:53:39 +0000 From: Luis Chamberlain To: kdevops@lists.linux.dev Cc: dave@stgolabs.net, a.manzanares@samsung.com, fan.ni@samsung.com, jim.harris@samsung.com, Luis Chamberlain Subject: [PATCH] guestfs: add pcie passthrough XML generation / attachsupport Date: Thu, 7 Mar 2024 18:53:38 -0800 Message-ID: <20240308025338.1768534-1-mcgrof@kernel.org> X-Mailer: git-send-email 2.44.0 Precedence: bulk X-Mailing-List: kdevops@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: Luis Chamberlain Most of the hard work to support PCIE passthrough support was already done before for vagrant and through 'make dynconfig' for the dynamic Kconfig stuff. This let's associate a PCIE device to a guest or lets us pick which device onto which guest to peg a device to. The only thing left to do for guestfs was to generate the XML files and then attach them. When and how we do this was quite tricky though. We just generate an XML file with the input source information, we do this at 'make' time. CXL folks may want to pay attention to this, it means you can use XML output files for each target guest, or you dynamically generate guests and their respective PCI topology at bring up time later. To attach we just wait after we define the guest. As soon as we define it, we can virsh attach devices to it. To simplify things further, instead of assuming we know where exactly on the guest we want to put a device we let libvirt figure that out. This proves to be super clean and removes all the PCI passthrough stuff from the XML input source file. This should also mean the CXL dynamic topology we may want to later add support for could all be kept out of the file source file and instead consist of XML input files post device definition. All we'd really need for the XML device to support CXL is to define the PCI-E reqs before we virsh define each guest, and that is pretty generic. Screenshot: kdevops@gdemo ~ $ sudo nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 /dev/ng0n1 S332NXAH300376 SAMSUNG MZQLW1T9HMJP-000ZU 1 119.93 GB / 1.92 TB 4 KiB + 0 B CXV8301Q Signed-off-by: Luis Chamberlain --- .../gen_pcie_passthrough_guestfs_xml.py | 78 +++++++++++++++++++ playbooks/roles/gen_nodes/tasks/main.yml | 13 ++++ scripts/bringup_guestfs.sh | 4 + 3 files changed, 95 insertions(+) create mode 100755 playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py diff --git a/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py b/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py new file mode 100755 index 000000000000..fcf96002e608 --- /dev/null +++ b/playbooks/roles/gen_nodes/python/gen_pcie_passthrough_guestfs_xml.py @@ -0,0 +1,78 @@ +#!/usr/bin/python3 +# +# add_pcie_passthrough_guestfs +# +# For each kdevops guest, gets its PCIe passthrough devices on the yaml file +# and augment them on the XML file. + +import yaml +import json +import sys +import pprint +import subprocess +import time +import os +from pathlib import Path + +pcie_hotplug_template = """ + + + +
+ + + +""" + +def main(): + topdir = os.environ.get('TOPDIR', '.') + + # load extra_vars + with open(f'{topdir}/extra_vars.yaml') as stream: + extra_vars = yaml.safe_load(stream) + + yaml_nodes_file = f'{topdir}/{extra_vars["kdevops_nodes"]}' + + # slurp in the guestfs_nodes list + with open(yaml_nodes_file) as stream: + nodes = yaml.safe_load(stream) + + # add pcie devices + for node in nodes['guestfs_nodes']: + name = node['name'] + pcipassthrough = node.get('pcipassthrough') + if not pcipassthrough: + continue + for dev_key_name in pcipassthrough: + dev = pcipassthrough.get(dev_key_name) + dev_keys = list(dev.keys()) + if 'domain' not in dev_keys or 'bus' not in dev_keys or 'slot' not in dev_keys or 'function' not in dev_keys: + raise Exception(f"Missing pcie attributes for device %s in %s" % + (dev_key_name, yaml_nodes_file)) + domain = dev.get('domain') + bus = dev.get('bus') + slot = dev.get('slot') + function = dev.get('function') + + pcie_xml = f"{extra_vars['guestfs_path']}/{name}/pcie_passthrough_" + dev_key_name + ".xml" + + if os.path.exists(pcie_xml): + os.remove(pcie_xml) + + device_xml = open(pcie_xml, 'w') + context = { + "domain" : domain, + "bus" : bus, + "slot" : slot, + "function" : function, + } + device_xml.write(pcie_hotplug_template.format(**context)) + device_xml.close() + +# result = subprocess.run(['virsh', 'attach-device', name, '--file', device_xml, '--config'], capture_output=True) +# if result.returncode != 0: +# raise Exception(f"Unable to vish attach-device to guest %s pci passthrough device %s using %s" % +# (name, dev_key_name, pcie_xml)) + +if __name__ == "__main__": + main() diff --git a/playbooks/roles/gen_nodes/tasks/main.yml b/playbooks/roles/gen_nodes/tasks/main.yml index 2f5c48b6cdcf..92cf5b3065a2 100644 --- a/playbooks/roles/gen_nodes/tasks/main.yml +++ b/playbooks/roles/gen_nodes/tasks/main.yml @@ -385,6 +385,19 @@ with_items: "{{ guestfs_nodes }}" when: kdevops_enable_guestfs +- name: Generate XML files for the libvirt guests for PCI passthrough for guestfs + delegate_to: localhost + run_once: true + shell: | + set -o pipefail + {{ role_path }}/python/gen_pcie_passthrough_guestfs_xml.py + args: + executable: /bin/bash + chdir: "{{ topdir_path }}" + when: + - kdevops_enable_guestfs + - pcie_passthrough_enable|bool + - name: Generate XML files for the libvirt guests vars: hostname: "{{ item.name }}" diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh index 2b5b38571265..b02c3ef3ba4c 100755 --- a/scripts/bringup_guestfs.sh +++ b/scripts/bringup_guestfs.sh @@ -135,6 +135,10 @@ do fi virsh define $GUESTFSDIR/$name/$name.xml + for xml in $GUESTFSDIR/$name/pcie_passthrough_*.xml; do + echo "Doing PCI-E passthrough for device $xml" + virsh attach-device $name $xml --config + done virsh start $name if [[ $? -ne 0 ]]; then echo "Failed to start $name" -- 2.43.0