* [PATCH 1/5] bringup_guestfs.sh: use bash as the default shell
2024-03-07 0:14 [PATCH 0/5] guestfs: start moving guest data to macros Luis Chamberlain
@ 2024-03-07 0:14 ` Luis Chamberlain
2024-03-07 0:14 ` [PATCH 2/5] guestfs_q35: use hex for pci addr Luis Chamberlain
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-03-07 0:14 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
We don't use bash but we rely on bash for everthing so just make the default.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
scripts/bringup_guestfs.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index 58e5e3d2..7dca84fe 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -53,8 +53,8 @@ _EOT
# basic pre-install customization
cat <<_EOT >>$cmdfile
-install sudo,qemu-guest-agent,python3
-run-command useradd -m kdevops
+install sudo,qemu-guest-agent,python3,bash
+run-command useradd -m kdevops -s /bin/bash
append-line /etc/sudoers.d/kdevops:kdevops ALL=(ALL) NOPASSWD: ALL
_EOT
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 2/5] guestfs_q35: use hex for pci addr
2024-03-07 0:14 [PATCH 0/5] guestfs: start moving guest data to macros Luis Chamberlain
2024-03-07 0:14 ` [PATCH 1/5] bringup_guestfs.sh: use bash as the default shell Luis Chamberlain
@ 2024-03-07 0:14 ` Luis Chamberlain
2024-03-07 0:14 ` [PATCH 3/5] guestfs_q35: use libvirt_extra_storage_nvme_logical_block_size Luis Chamberlain
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-03-07 0:14 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
Since we are prepending as a hex value ensure the input is converted
as such. This creates no functional changes but if we later want to
increase the drives beyond 9 then this will ensure we use the address
space appropriately. This will become more important later when we
add support for large-io experimentation to guestfs which does add
tons of drives onto a guest, enough to be able to experiment with
LBS support on different types of drives all in one system.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
index 86b26346..ee340aca 100644
--- a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
+++ b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
@@ -187,7 +187,7 @@
{% elif libvirt_extra_storage_drive_virtio %}
{% for n in range(0,4) %}
<qemu:arg value='-device'/>
- <qemu:arg value='pcie-root-port,id=pcie-port-for-virtio-{{ n }},multifunction=on,bus=pcie.1,addr=0x{{ n }},chassis=5{{ n }}'/>
+ <qemu:arg value='pcie-root-port,id=pcie-port-for-virtio-{{ n }},multifunction=on,bus=pcie.1,addr=0x{{ "%0x" | format( n | int) }},chassis=5{{ n }}'/>
<qemu:arg value="-object"/>
<qemu:arg value="iothread,id=kdevops-virtio-iothread-{{ n }}"/>
<qemu:arg value="-drive"/>
@@ -198,7 +198,7 @@
{% elif libvirt_extra_storage_drive_nvme %}
{% for n in range(0,4) %}
<qemu:arg value='-device'/>
- <qemu:arg value='pcie-root-port,id=pcie-port-for-nvme-{{ n }},multifunction=on,bus=pcie.1,addr=0x{{ n }},chassis=5{{ n }}'/>
+ <qemu:arg value='pcie-root-port,id=pcie-port-for-nvme-{{ n }},multifunction=on,bus=pcie.1,addr=0x{{ "%0x" | format( n | int) }},chassis=5{{ n }}'/>
<qemu:arg value='-drive'/>
<qemu:arg value='file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ n }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},if=none,id=drv{{ n }}'/>
<qemu:arg value='-device'/>
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 3/5] guestfs_q35: use libvirt_extra_storage_nvme_logical_block_size
2024-03-07 0:14 [PATCH 0/5] guestfs: start moving guest data to macros Luis Chamberlain
2024-03-07 0:14 ` [PATCH 1/5] bringup_guestfs.sh: use bash as the default shell Luis Chamberlain
2024-03-07 0:14 ` [PATCH 2/5] guestfs_q35: use hex for pci addr Luis Chamberlain
@ 2024-03-07 0:14 ` Luis Chamberlain
2024-03-07 0:14 ` [PATCH 4/5] gen_nodes: move drive generation for guestfs to macros Luis Chamberlain
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-03-07 0:14 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
The libvirt_extra_storage_nvme_logical_block_size was not being used, so
fix that. Also clarify on the defaults why only one variable is set.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
playbooks/roles/gen_nodes/defaults/main.yml | 2 ++
playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml | 2 +-
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/playbooks/roles/gen_nodes/defaults/main.yml b/playbooks/roles/gen_nodes/defaults/main.yml
index 0e41e0ad..5f46a7da 100644
--- a/playbooks/roles/gen_nodes/defaults/main.yml
+++ b/playbooks/roles/gen_nodes/defaults/main.yml
@@ -63,6 +63,8 @@ libvirt_extra_storage_drive_virtio: False
libvirt_extra_storage_drive_ide: False
libvirt_extra_storage_aio_mode: "native"
libvirt_extra_storage_aio_cache_mode: "none"
+# Note that NVMe on qemu does not allow the physical block size
+# to differ from the logical block size so we just have one variable.
libvirt_extra_storage_nvme_logical_block_size: 512
libvirt_extra_storage_virtio_logical_block_size: 512
libvirt_extra_storage_virtio_physical_block_size: 512
diff --git a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
index ee340aca..266ced32 100644
--- a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
+++ b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
@@ -204,7 +204,7 @@
<qemu:arg value='-device'/>
<qemu:arg value='nvme,id=nvme{{ n }},serial=kdevops{{ n }},bus=pcie-port-for-nvme-{{ n }},addr=0x0'/>
<qemu:arg value='-device'/>
- <qemu:arg value='nvme-ns,drive=drv{{ n }},bus=nvme{{ n }},nsid=1,logical_block_size=512,physical_block_size=512'/>
+ <qemu:arg value='nvme-ns,drive=drv{{ n }},bus=nvme{{ n }},nsid=1,logical_block_size={{ libvirt_extra_storage_nvme_logical_block_size }},physical_block_size={{ libvirt_extra_storage_nvme_logical_block_size }}'/>
{% endfor %}
{% endif %}
{% if bootlinux_9p %}
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 4/5] gen_nodes: move drive generation for guestfs to macros
2024-03-07 0:14 [PATCH 0/5] guestfs: start moving guest data to macros Luis Chamberlain
` (2 preceding siblings ...)
2024-03-07 0:14 ` [PATCH 3/5] guestfs_q35: use libvirt_extra_storage_nvme_logical_block_size Luis Chamberlain
@ 2024-03-07 0:14 ` Luis Chamberlain
2024-03-07 0:14 ` [PATCH 5/5] guestfs: add large IO support Luis Chamberlain
2024-03-07 18:43 ` [PATCH 0/5] guestfs: start moving guest data to macros Luis Chamberlain
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-03-07 0:14 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
Make the guestfs template easier to read by using macros for
the different types of drive generation methods. This will also
let us scale this to more easily support other architectures with
as much shared code as possible later.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
playbooks/roles/gen_nodes/templates/drives.j2 | 76 +++++++++++++++++++
.../gen_nodes/templates/guestfs_q35.j2.xml | 59 +++++++-------
2 files changed, 104 insertions(+), 31 deletions(-)
create mode 100644 playbooks/roles/gen_nodes/templates/drives.j2
diff --git a/playbooks/roles/gen_nodes/templates/drives.j2 b/playbooks/roles/gen_nodes/templates/drives.j2
new file mode 100644
index 00000000..4878cff9
--- /dev/null
+++ b/playbooks/roles/gen_nodes/templates/drives.j2
@@ -0,0 +1,76 @@
+{#
+This is a collection of macros to generate extra drives used by guestfs,
+the drives can vary by type, so we have one macro by type of drive.
+#}
+{%- macro gen_drive_ide(num_drives,
+ kdevops_storage_pool_path,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode) -%}
+<!-- We generated {{ num_drives }} ide dives -->
+{% for n in range(0,num_drives) %}
+ <!-- This is IDE drive # {{ n + 1 }} -->
+ <qemu:arg value='-drive'/>
+ <qemu:arg value='file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ n }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},aio={{ libvirt_extra_storage_aio_mode }},cache={{ libvirt_extra_storage_aio_cache_mode }},if=none,id=drv{{ n }}'/>
+ <qemu:arg value='-device'/>
+ <qemu:arg value="ide-hd,drive=drv{{ n }},bus=ide.{{ n }},serial=kdevops{{ n }}"/>
+{% endfor %}
+<!-- End of ide drives-->
+{%- endmacro -%}
+{%- macro gen_drive_virtio(num_drives,
+ kdevops_storage_pool_path,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode,
+ libvirt_extra_storage_virtio_logical_block_size,
+ libvirt_extra_storage_virtio_physical_block_size) -%}
+<!-- We generated {{ num_drives }} virtio dives -->
+{% for n in range(0,num_drives) %}
+ <!-- This is virtio drive # {{ n + 1 }} -->
+ <qemu:arg value='-device'/>
+ <qemu:arg value='pcie-root-port,id=pcie-port-for-virtio-{{ n }},multifunction=on,bus=pcie.1,addr=0x{{ "%0x" | format( n | int) }},chassis=5{{ n }}'/>
+ <qemu:arg value="-object"/>
+ <qemu:arg value="iothread,id=kdevops-virtio-iothread-{{ n }}"/>
+ <qemu:arg value="-drive"/>
+ <qemu:arg value="file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ n }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},if=none,aio={{ libvirt_extra_storage_aio_mode }},cache={{ libvirt_extra_storage_aio_cache_mode }},id=drv{{ n }}"/>
+ <qemu:arg value="-device"/>
+ <qemu:arg value="virtio-blk-pci,scsi=off,drive=drv{{ n }},id=virtio-drv{{ n }},serial=kdevops{{ n }},bus=pcie-port-for-virtio-{{ n }},addr=0x0,iothread=kdevops-virtio-iothread-{{ n }},logical_block_size={{ libvirt_extra_storage_virtio_logical_block_size }},physical_block_size={{ libvirt_extra_storage_virtio_physical_block_size }}"/>
+{% endfor %}
+<!-- End of virtio drives-->
+{%- endmacro -%}
+{%- macro gen_drive_nvme(num_drives,
+ kdevops_storage_pool_path,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode,
+ libvirt_extra_storage_nvme_logical_block_size) -%}
+<!-- We generated {{ num_drives }} NVMe dives -->
+{% for n in range(0,num_drives) %}
+ <!-- This is NVMe drive # {{ n + 1 }} -->
+ <qemu:arg value='-device'/>
+ <qemu:arg value='pcie-root-port,id=pcie-port-for-nvme-{{ n }},multifunction=on,bus=pcie.1,addr=0x{{ "%0x" | format( n | int) }},chassis=5{{ n }}'/>
+ <qemu:arg value='-drive'/>
+ <qemu:arg value='file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ n }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},if=none,id=drv{{ n }}'/>
+ <qemu:arg value='-device'/>
+ <qemu:arg value='nvme,id=nvme{{ n }},serial=kdevops{{ n }},bus=pcie-port-for-nvme-{{ n }},addr=0x0'/>
+ <qemu:arg value='-device'/>
+ <qemu:arg value='nvme-ns,drive=drv{{ n }},bus=nvme{{ n }},nsid=1,logical_block_size={{ libvirt_extra_storage_nvme_logical_block_size }},physical_block_size={{ libvirt_extra_storage_nvme_logical_block_size }}'/>
+{% endfor %}
+<!-- End of NVMe drives-->
+{%- endmacro -%}
+{% macro gen_9p_mount(bootlinux_9p_driver,
+ bootlinux_9p_fsdev,
+ bootlinux_9p_host_path,
+ bootlinux_9p_mount_tag,
+ bootlinux_9p_security_model,
+ pcie_addr) %}
+<!-- This is a 9p mounted drive -->
+ <qemu:arg value='-device'/>
+ <qemu:arg value='{{ bootlinux_9p_driver }},fsdev={{ bootlinux_9p_fsdev }},mount_tag={{ bootlinux_9p_mount_tag }},bus=pcie.0,addr=0x{{ pcie_addr }}'/>
+ <qemu:arg value='-fsdev'/>
+ <qemu:arg value='local,id={{ bootlinux_9p_fsdev }},path={{ bootlinux_9p_host_path }},security_model={{ bootlinux_9p_security_model }}'/>
+<!-- End of 9p mount -->
+{% endmacro %}
diff --git a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
index 266ced32..16364ea2 100644
--- a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
+++ b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
@@ -1,3 +1,4 @@
+{% import './templates/drives.j2' as drives %}
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>{{ hostname }}</name>
<memory unit='MiB'>{{ libvirt_mem_mb }}</memory>
@@ -178,41 +179,37 @@
<qemu:arg value='-device'/>
<qemu:arg value='pxb-pcie,id=pcie.1,bus_nr=32,bus=pcie.0,addr=0x8'/>
{% if libvirt_extra_storage_drive_ide %}
-{% for n in range(0,4) %}
- <qemu:arg value='-drive'/>
- <qemu:arg value='file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ n }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},aio={{ libvirt_extra_storage_aio_mode }},cache={{ libvirt_extra_storage_aio_cache_mode }},if=none,id=drv{{ n }}'/>
- <qemu:arg value='-device'/>
- <qemu:arg value="ide-hd,drive=drv{{ n }},bus=ide.{{ n }},serial=kdevops{{ n }}"/>
-{% endfor %}
+{{ drives.gen_drive_ide(4,
+ kdevops_storage_pool_path,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode) }}
{% elif libvirt_extra_storage_drive_virtio %}
-{% for n in range(0,4) %}
- <qemu:arg value='-device'/>
- <qemu:arg value='pcie-root-port,id=pcie-port-for-virtio-{{ n }},multifunction=on,bus=pcie.1,addr=0x{{ "%0x" | format( n | int) }},chassis=5{{ n }}'/>
- <qemu:arg value="-object"/>
- <qemu:arg value="iothread,id=kdevops-virtio-iothread-{{ n }}"/>
- <qemu:arg value="-drive"/>
- <qemu:arg value="file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ n }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},if=none,aio={{ libvirt_extra_storage_aio_mode }},cache={{ libvirt_extra_storage_aio_cache_mode }},id=drv{{ n }}"/>
- <qemu:arg value="-device"/>
- <qemu:arg value="virtio-blk-pci,scsi=off,drive=drv{{ n }},id=virtio-drv{{ n }},serial=kdevops{{ n }},bus=pcie-port-for-virtio-{{ n }},addr=0x0,iothread=kdevops-virtio-iothread-{{ n }},logical_block_size={{ libvirt_extra_storage_virtio_logical_block_size }},physical_block_size={{ libvirt_extra_storage_virtio_physical_block_size }}"/>
-{% endfor %}
+{{ drives.gen_drive_virtio(4,
+ kdevops_storage_pool_path,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode,
+ libvirt_extra_storage_virtio_logical_block_size,
+ libvirt_extra_storage_virtio_physical_block_size) }}
{% elif libvirt_extra_storage_drive_nvme %}
-{% for n in range(0,4) %}
- <qemu:arg value='-device'/>
- <qemu:arg value='pcie-root-port,id=pcie-port-for-nvme-{{ n }},multifunction=on,bus=pcie.1,addr=0x{{ "%0x" | format( n | int) }},chassis=5{{ n }}'/>
- <qemu:arg value='-drive'/>
- <qemu:arg value='file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ n }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},if=none,id=drv{{ n }}'/>
- <qemu:arg value='-device'/>
- <qemu:arg value='nvme,id=nvme{{ n }},serial=kdevops{{ n }},bus=pcie-port-for-nvme-{{ n }},addr=0x0'/>
- <qemu:arg value='-device'/>
- <qemu:arg value='nvme-ns,drive=drv{{ n }},bus=nvme{{ n }},nsid=1,logical_block_size={{ libvirt_extra_storage_nvme_logical_block_size }},physical_block_size={{ libvirt_extra_storage_nvme_logical_block_size }}'/>
-{% endfor %}
+{{ drives.gen_drive_nvme(4,
+ kdevops_storage_pool_path,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode,
+ libvirt_extra_storage_nvme_logical_block_size) }}
{% endif %}
{% if bootlinux_9p %}
- <qemu:arg value='-device'/>
- <qemu:arg value='{{ bootlinux_9p_driver }},fsdev={{ bootlinux_9p_fsdev }},mount_tag={{ bootlinux_9p_mount_tag }},bus=pcie.0,addr=0x10'/>
- <qemu:arg value='-fsdev'/>
- <qemu:arg value='local,id={{ bootlinux_9p_fsdev }},path={{ bootlinux_9p_host_path }},security_model={{ bootlinux_9p_security_model }}'/>
+ {{ drives.gen_9p_mount(bootlinux_9p_driver,
+ bootlinux_9p_fsdev,
+ bootlinux_9p_host_path,
+ bootlinux_9p_mount_tag,
+ bootlinux_9p_security_model,
+ 10) }}
{% endif %}
</qemu:commandline>
</domain>
-
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 5/5] guestfs: add large IO support
2024-03-07 0:14 [PATCH 0/5] guestfs: start moving guest data to macros Luis Chamberlain
` (3 preceding siblings ...)
2024-03-07 0:14 ` [PATCH 4/5] gen_nodes: move drive generation for guestfs to macros Luis Chamberlain
@ 2024-03-07 0:14 ` Luis Chamberlain
2024-03-07 18:43 ` [PATCH 0/5] guestfs: start moving guest data to macros Luis Chamberlain
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-03-07 0:14 UTC (permalink / raw)
To: kdevops; +Cc: Luis Chamberlain
Large IO experimentation support was added to vagrant a while ago so to
enable experimentation with LBS support on the kernel [0]. Add this support
to guestfs now that guestfs is all rave, and we plan to deprecate
vagrant support as soon as we can.
Screenshot with NVMe:
$ sudo nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme9n1 /dev/ng9n1 kdevops22 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 16 KiB + 0 B 8.2.1
/dev/nvme8n1 /dev/ng8n1 kdevops21 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 16 KiB + 0 B 8.2.1
/dev/nvme7n1 /dev/ng7n1 kdevops16 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 4 KiB + 0 B 8.2.1
/dev/nvme6n1 /dev/ng6n1 kdevops15 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 4 KiB + 0 B 8.2.1
/dev/nvme5n1 /dev/ng5n1 kdevops14 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 4 KiB + 0 B 8.2.1
/dev/nvme4n1 /dev/ng4n1 kdevops13 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 4 KiB + 0 B 8.2.1
/dev/nvme3n1 /dev/ng3n1 kdevops4 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 512 B + 0 B 8.2.1
/dev/nvme2n1 /dev/ng2n1 kdevops3 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 512 B + 0 B 8.2.1
/dev/nvme16n1 /dev/ng16n1 kdevops30 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 64 KiB + 0 B 8.2.1
/dev/nvme15n1 /dev/ng15n1 kdevops29 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 64 KiB + 0 B 8.2.1
/dev/nvme14n1 /dev/ng14n1 kdevops27 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 32 KiB + 0 B 8.2.1
/dev/nvme13n1 /dev/ng13n1 kdevops26 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 32 KiB + 0 B 8.2.1
/dev/nvme12n1 /dev/ng12n1 kdevops25 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 32 KiB + 0 B 8.2.1
/dev/nvme11n1 /dev/ng11n1 kdevops24 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 16 KiB + 0 B 8.2.1
/dev/nvme10n1 /dev/ng10n1 kdevops23 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 16 KiB + 0 B 8.2.1
/dev/nvme1n1 /dev/ng1n1 kdevops2 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 512 B + 0 B 8.2.1
/dev/nvme0n1 /dev/ng0n1 kdevops1 QEMU NVMe Ctrl 1 107.37 GB / 107.37 GB 512 B + 0 B 8.2.1
$ uname -r
6.1.0-9-amd64
To use drives with larger LBA formats we need further changes than the
LBS patches, but that's later. We have patches to get this functional up
to 1 MiB LBA format.
For virtio, we end up failing at virtio device 14:
Mar 06 19:03:55 d5 kernel: virtio_blk virtio14: virtio_blk: invalid block size: 0x4000
Mar 06 19:03:55 d5 kernel: virtio_blk: probe of virtio14 failed with error -22
Mar 06 19:03:55 d5 kernel: virtio_blk virtio15: 8/0/0 default/read/poll queues
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
playbooks/roles/gen_nodes/templates/drives.j2 | 72 +++++++++++++++++++
.../gen_nodes/templates/guestfs_q35.j2.xml | 24 +++++++
scripts/bringup_guestfs.sh | 38 +++++++---
3 files changed, 125 insertions(+), 9 deletions(-)
diff --git a/playbooks/roles/gen_nodes/templates/drives.j2 b/playbooks/roles/gen_nodes/templates/drives.j2
index 4878cff9..676722ae 100644
--- a/playbooks/roles/gen_nodes/templates/drives.j2
+++ b/playbooks/roles/gen_nodes/templates/drives.j2
@@ -40,6 +40,42 @@ the drives can vary by type, so we have one macro by type of drive.
{% endfor %}
<!-- End of virtio drives-->
{%- endmacro -%}
+{%- macro gen_drive_large_io_virtio(libvirt_largeio_logical_compat,
+ libvirt_largeio_logical_compat_size,
+ libvirt_largeio_pow_limit,
+ libvirt_largeio_drives_per_space,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode,
+ kdevops_storage_pool_path) -%}
+<!-- These are virtio drives used for large IO experimentaiton, with LBS support -->
+{% set ns = namespace(lbs_idx=1) %}
+{% set max_pbs = libvirt_largeio_logical_compat_size * (2 ** libvirt_largeio_pow_limit) %}
+{% for n in range(0,libvirt_largeio_pow_limit+1) %}
+{% for x in range(0,libvirt_largeio_drives_per_space) %}
+{% set ns2 = namespace(pbs=libvirt_largeio_logical_compat_size * (2 ** n)) %}
+{% set ns3 = namespace(pbs_next_two=ns2.pbs * (2*(x-1))) %}
+{% if libvirt_largeio_logical_compat %}
+{% set ns4 = namespace(lbs=libvirt_largeio_logical_compat_size) %}
+{% else %}
+{% set ns4 = namespace(lbs=ns2.pbs) %}
+{% endif %}
+{% if (ns2.pbs == 512 or ns2.pbs == 4096 or ns2.pbs >= 16384) and (ns3.pbs_next_two <= max_pbs) %}
+ <qemu:arg value='-device'/>
+ <qemu:arg value='pcie-root-port,id=pcie-port-for-virtio-{{ ns.lbs_idx }},multifunction=on,bus=pcie.1,addr=0x{{ "%0x" | format( ns.lbs_idx | int) }},chassis={{ 50 + ns.lbs_idx }}'/>
+ <qemu:arg value="-object"/>
+ <qemu:arg value="iothread,id=kdevops-virtio-iothread-{{ ns.lbs_idx }}"/>
+ <qemu:arg value='-drive'/>
+ <qemu:arg value='file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ ns.lbs_idx }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},if=none,aio={{ libvirt_extra_storage_aio_mode }},cache={{ libvirt_extra_storage_aio_cache_mode }},id=drv{{ ns.lbs_idx }}'/>
+ <qemu:arg value='-device'/>
+ <qemu:arg value="virtio-blk-pci,scsi=off,drive=drv{{ ns.lbs_idx }},id=virtio-drv{{ ns.lbs_idx }},serial=kdevops{{ ns.lbs_idx }},bus=pcie-port-for-virtio-{{ ns.lbs_idx }},addr=0x0,iothread=kdevops-virtio-iothread-{{ ns.lbs_idx }},logical_block_size={{ ns4.lbs }},physical_block_size={{ ns2.pbs }}"/>
+{% endif %}
+{% set ns.lbs_idx = ns.lbs_idx + 1 %}
+{% endfor %}
+{% endfor %}
+<!-- End of virtio drives for large IO experimentation -->
+{%- endmacro -%}
{%- macro gen_drive_nvme(num_drives,
kdevops_storage_pool_path,
hostname,
@@ -61,6 +97,42 @@ the drives can vary by type, so we have one macro by type of drive.
{% endfor %}
<!-- End of NVMe drives-->
{%- endmacro -%}
+{%- macro gen_drive_large_io_nvme(libvirt_largeio_logical_compat,
+ libvirt_largeio_logical_compat_size,
+ libvirt_largeio_pow_limit,
+ libvirt_largeio_drives_per_space,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode,
+ kdevops_storage_pool_path) -%}
+<!-- These are NVMe drives used for large IO experimentaiton, with LBS support -->
+{% set ns = namespace(lbs_idx=1) %}
+{% set max_pbs = libvirt_largeio_logical_compat_size * (2 ** libvirt_largeio_pow_limit) %}
+{% for n in range(0,libvirt_largeio_pow_limit+1) %}
+{% for x in range(0,libvirt_largeio_drives_per_space) %}
+{% set ns2 = namespace(pbs=libvirt_largeio_logical_compat_size * (2 ** n)) %}
+{% set ns3 = namespace(pbs_next_two=ns2.pbs * (2*(x-1))) %}
+{% if libvirt_largeio_logical_compat %}
+{% set ns4 = namespace(lbs=libvirt_largeio_logical_compat_size) %}
+{% else %}
+{% set ns4 = namespace(lbs=ns2.pbs) %}
+{% endif %}
+{% if (ns2.pbs == 512 or ns2.pbs == 4096 or ns2.pbs >= 16384) and (ns3.pbs_next_two <= max_pbs) %}
+ <qemu:arg value='-device'/>
+ <qemu:arg value='pcie-root-port,id=pcie-port-for-nvme-{{ ns.lbs_idx }},multifunction=on,bus=pcie.1,addr=0x{{ "%0x" | format( ns.lbs_idx | int) }},chassis={{ 50 + ns.lbs_idx }}'/>
+ <qemu:arg value='-drive'/>
+ <qemu:arg value='file={{ kdevops_storage_pool_path }}/guestfs/{{ hostname }}/extra{{ ns.lbs_idx }}.{{ libvirt_extra_drive_format }},format={{ libvirt_extra_drive_format }},if=none,id=drv{{ ns.lbs_idx }}'/>
+ <qemu:arg value='-device'/>
+ <qemu:arg value='nvme,id=nvme{{ ns.lbs_idx }},serial=kdevops{{ ns.lbs_idx }},bus=pcie-port-for-nvme-{{ ns.lbs_idx }},addr=0x0'/>
+ <qemu:arg value='-device'/>
+ <qemu:arg value='nvme-ns,drive=drv{{ ns.lbs_idx }},bus=nvme{{ ns.lbs_idx }},nsid=1,logical_block_size={{ ns4.lbs }},physical_block_size={{ ns2.pbs }}'/>
+{% endif %}
+{% set ns.lbs_idx = ns.lbs_idx + 1 %}
+{% endfor %}
+{% endfor %}
+<!-- End of NVMe drives for large IO experimentation -->
+{%- endmacro -%}
{% macro gen_9p_mount(bootlinux_9p_driver,
bootlinux_9p_fsdev,
bootlinux_9p_host_path,
diff --git a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
index 16364ea2..fe8be827 100644
--- a/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
+++ b/playbooks/roles/gen_nodes/templates/guestfs_q35.j2.xml
@@ -186,6 +186,17 @@
libvirt_extra_storage_aio_mode,
libvirt_extra_storage_aio_cache_mode) }}
{% elif libvirt_extra_storage_drive_virtio %}
+{% if libvirt_largeio_enable %}
+{{ drives.gen_drive_large_io_virtio(libvirt_largeio_logical_compat,
+ libvirt_largeio_logical_compat_size,
+ libvirt_largeio_pow_limit,
+ libvirt_largeio_drives_per_space,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode,
+ kdevops_storage_pool_path) }}
+{% else %}
{{ drives.gen_drive_virtio(4,
kdevops_storage_pool_path,
hostname,
@@ -194,7 +205,19 @@
libvirt_extra_storage_aio_cache_mode,
libvirt_extra_storage_virtio_logical_block_size,
libvirt_extra_storage_virtio_physical_block_size) }}
+{% endif %}
{% elif libvirt_extra_storage_drive_nvme %}
+{% if libvirt_largeio_enable %}
+{{ drives.gen_drive_large_io_nvme(libvirt_largeio_logical_compat,
+ libvirt_largeio_logical_compat_size,
+ libvirt_largeio_pow_limit,
+ libvirt_largeio_drives_per_space,
+ hostname,
+ libvirt_extra_drive_format,
+ libvirt_extra_storage_aio_mode,
+ libvirt_extra_storage_aio_cache_mode,
+ kdevops_storage_pool_path) }}
+{% else %}
{{ drives.gen_drive_nvme(4,
kdevops_storage_pool_path,
hostname,
@@ -203,6 +226,7 @@
libvirt_extra_storage_aio_cache_mode,
libvirt_extra_storage_nvme_logical_block_size) }}
{% endif %}
+{% endif %}
{% if bootlinux_9p %}
{{ drives.gen_9p_mount(bootlinux_9p_driver,
bootlinux_9p_fsdev,
diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
index 7dca84fe..6f621785 100755
--- a/scripts/bringup_guestfs.sh
+++ b/scripts/bringup_guestfs.sh
@@ -109,16 +109,36 @@ do
cp --reflink=auto $BASE_IMAGE $ROOTIMG
virt-sysprep -a $ROOTIMG --hostname $name --ssh-inject "kdevops:file:$SSH_KEY.pub"
- # build some extra disks
- for i in $(seq 0 3); do
- diskimg="$STORAGEDIR/$name/extra${i}.${IMG_FMT}"
- rm -f $diskimg
- qemu-img create -f $IMG_FMT "$STORAGEDIR/$name/extra${i}.$IMG_FMT" 100G
- if [[ "$CONFIG_LIBVIRT_URI_SYSTEM" == "y" ]]; then
- chmod g+rw $STORAGEDIR/$name/extra${i}.$IMG_FMT
- fi
- done
+
+ if [[ "$CONFIG_LIBVIRT_ENABLE_LARGEIO" == "y" ]]; then
+ lbs_idx=1
+ for i in $(seq 1 $(($CONFIG_QEMU_LARGEIO_MAX_POW_LIMIT+1))); do
+ for x in $(seq 0 $CONFIG_QEMU_EXTRA_DRIVE_LARGEIO_NUM_DRIVES_PER_SPACE); do
+ diskimg="$STORAGEDIR/$name/extra${lbs_idx}.${IMG_FMT}"
+ rm -f $diskimg
+ qemu-img create -f $IMG_FMT "$diskimg" 100G
+ if [[ "$CONFIG_LIBVIRT_URI_SYSTEM" == "y" ]]; then
+ chmod g+rw $diskimg
+ fi
+ let lbs_idx=$lbs_idx+1
+ done
+ done
+ else
+ # build some extra disks
+ for i in $(seq 0 3); do
+ diskimg="$STORAGEDIR/$name/extra${i}.${IMG_FMT}"
+ rm -f $diskimg
+ qemu-img create -f $IMG_FMT "$STORAGEDIR/$name/extra${i}.$IMG_FMT" 100G
+ if [[ "$CONFIG_LIBVIRT_URI_SYSTEM" == "y" ]]; then
+ chmod g+rw $STORAGEDIR/$name/extra${i}.$IMG_FMT
+ fi
+ done
+ fi
virsh define $GUESTFSDIR/$name/$name.xml
virsh start $name
+ if [[ $? -ne 0 ]]; then
+ echo "Failed to start $name"
+ exit 1
+ fi
done
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH 0/5] guestfs: start moving guest data to macros
2024-03-07 0:14 [PATCH 0/5] guestfs: start moving guest data to macros Luis Chamberlain
` (4 preceding siblings ...)
2024-03-07 0:14 ` [PATCH 5/5] guestfs: add large IO support Luis Chamberlain
@ 2024-03-07 18:43 ` Luis Chamberlain
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-03-07 18:43 UTC (permalink / raw)
To: kdevops
On Wed, Mar 06, 2024 at 04:14:21PM -0800, Luis Chamberlain wrote:
> This implements initial macro support usage for using guestfs for
> dynamic libvirt XML data for the guests, it starts with moving the
> existing extra drive support to its own file as macros, and later
> builds on this to demonstrate more complex functionality like support
> for large IO experimentation.
>
> Luis Chamberlain (5):
> bringup_guestfs.sh: use bash as the default shell
> guestfs_q35: use hex for pci addr
> guestfs_q35: use libvirt_extra_storage_nvme_logical_block_size
> gen_nodes: move drive generation for guestfs to macros
> guestfs: add large IO support
Merged.
Luis
^ permalink raw reply [flat|nested] 7+ messages in thread