* [PULL 1/7] s390x/ap: fix missing subsystem reset registration
2023-09-12 11:41 [PULL 0/7] s390x (and one qtest) patches Thomas Huth
@ 2023-09-12 11:41 ` Thomas Huth
2023-09-12 11:41 ` [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot Thomas Huth
` (5 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Thomas Huth @ 2023-09-12 11:41 UTC (permalink / raw)
To: qemu-devel; +Cc: Stefan Hajnoczi, Janosch Frank, Jason J . Herne, Tony Krowiak
From: Janosch Frank <frankja@linux.ibm.com>
A subsystem reset contains a reset of AP resources which has been
missing. Adding the AP bridge to the list of device types that need
reset fixes this issue.
Reviewed-by: Jason J. Herne <jjherne@linux.ibm.com>
Reviewed-by: Tony Krowiak <akrowiak@linux.ibm.com>
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Fixes: a51b3153 ("s390x/ap: base Adjunct Processor (AP) object model")
Message-ID: <20230823142219.1046522-2-seiden@linux.ibm.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
hw/s390x/s390-virtio-ccw.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index bfcf64d007..3dd0b2372d 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -109,6 +109,7 @@ static const char *const reset_dev_types[] = {
"s390-flic",
"diag288",
TYPE_S390_PCI_HOST_BRIDGE,
+ TYPE_AP_BRIDGE,
};
static void subsystem_reset(void)
--
2.41.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot
2023-09-12 11:41 [PULL 0/7] s390x (and one qtest) patches Thomas Huth
2023-09-12 11:41 ` [PULL 1/7] s390x/ap: fix missing subsystem reset registration Thomas Huth
@ 2023-09-12 11:41 ` Thomas Huth
2024-01-10 18:30 ` Cédric Le Goater
2023-09-12 11:41 ` [PULL 3/7] linux-headers: Update to Linux v6.6-rc1 Thomas Huth
` (4 subsequent siblings)
6 siblings, 1 reply; 15+ messages in thread
From: Thomas Huth @ 2023-09-12 11:41 UTC (permalink / raw)
To: qemu-devel
Cc: Stefan Hajnoczi, Janosch Frank, Viktor Mihajlovski,
Christian Borntraeger
From: Janosch Frank <frankja@linux.ibm.com>
Bound APQNs have to be reset before tearing down the secure config via
s390_machine_unprotect(). Otherwise the Ultravisor will return a error
code.
So let's do a subsystem_reset() which includes a AP reset before the
unprotect call. We'll do a full device_reset() afterwards which will
reset some devices twice. That's ok since we can't move the
device_reset() before the unprotect as it includes a CPU clear reset
which the Ultravisor does not expect at that point in time.
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
Message-ID: <20230901114851.154357-1-frankja@linux.ibm.com>
Tested-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
hw/s390x/s390-virtio-ccw.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
index 3dd0b2372d..2d75f2131f 100644
--- a/hw/s390x/s390-virtio-ccw.c
+++ b/hw/s390x/s390-virtio-ccw.c
@@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
switch (reset_type) {
case S390_RESET_EXTERNAL:
case S390_RESET_REIPL:
+ /*
+ * Reset the subsystem which includes a AP reset. If a PV
+ * guest had APQNs attached the AP reset is a prerequisite to
+ * unprotecting since the UV checks if all APQNs are reset.
+ */
+ subsystem_reset();
if (s390_is_pv()) {
s390_machine_unprotect(ms);
}
+ /*
+ * Device reset includes CPU clear resets so this has to be
+ * done AFTER the unprotect call above.
+ */
qemu_devices_reset(reason);
s390_crypto_reset();
--
2.41.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot
2023-09-12 11:41 ` [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot Thomas Huth
@ 2024-01-10 18:30 ` Cédric Le Goater
2024-01-10 20:28 ` Matthew Rosato
0 siblings, 1 reply; 15+ messages in thread
From: Cédric Le Goater @ 2024-01-10 18:30 UTC (permalink / raw)
To: Thomas Huth, qemu-devel
Cc: Stefan Hajnoczi, Janosch Frank, Viktor Mihajlovski,
Christian Borntraeger, Matthew Rosato
On 9/12/23 13:41, Thomas Huth wrote:
> From: Janosch Frank <frankja@linux.ibm.com>
>
> Bound APQNs have to be reset before tearing down the secure config via
> s390_machine_unprotect(). Otherwise the Ultravisor will return a error
> code.
>
> So let's do a subsystem_reset() which includes a AP reset before the
> unprotect call. We'll do a full device_reset() afterwards which will
> reset some devices twice. That's ok since we can't move the
> device_reset() before the unprotect as it includes a CPU clear reset
> which the Ultravisor does not expect at that point in time.
>
> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
> Message-ID: <20230901114851.154357-1-frankja@linux.ibm.com>
> Tested-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
> Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> hw/s390x/s390-virtio-ccw.c | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
> index 3dd0b2372d..2d75f2131f 100644
> --- a/hw/s390x/s390-virtio-ccw.c
> +++ b/hw/s390x/s390-virtio-ccw.c
> @@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
> switch (reset_type) {
> case S390_RESET_EXTERNAL:
> case S390_RESET_REIPL:
> + /*
> + * Reset the subsystem which includes a AP reset. If a PV
> + * guest had APQNs attached the AP reset is a prerequisite to
> + * unprotecting since the UV checks if all APQNs are reset.
> + */
> + subsystem_reset();
This commit introduced a regression with pass-though ISM devices.
After startup, a reboot will generate extra device resets (vfio-pci in
this case) which break the pass-though ISM device in a subtle way,
probably related to IOMMU mapping according to 03451953c79e
("s390x/pci: reset ISM passthrough devices on shutdown and system
reset"). After poweroff, the device is left in a sort-of-a-use state
on the host and the LPAR has to be rebooted to clear the invalid state
of the device. To be noted, that standard PCI devices are immune to
this change.
The extra resets should avoided in some ways, (a shutdown notifier and
a reset callback are already registered for ISM devices by 03451953c79e)
and, most important, once the VM terminates, the device resources
should be cleared in the host kernel. So there seem to be two issues
to address in mainline QEMU and in Linux AFAICT.
Thanks,
C.
> if (s390_is_pv()) {
> s390_machine_unprotect(ms);
> }
>
> + /*
> + * Device reset includes CPU clear resets so this has to be
> + * done AFTER the unprotect call above.
> + */
> qemu_devices_reset(reason);
> s390_crypto_reset();
>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot
2024-01-10 18:30 ` Cédric Le Goater
@ 2024-01-10 20:28 ` Matthew Rosato
2024-01-11 9:43 ` Cédric Le Goater
0 siblings, 1 reply; 15+ messages in thread
From: Matthew Rosato @ 2024-01-10 20:28 UTC (permalink / raw)
To: Cédric Le Goater, Thomas Huth, qemu-devel
Cc: Stefan Hajnoczi, Janosch Frank, Viktor Mihajlovski,
Christian Borntraeger, Eric Farman
On 1/10/24 1:30 PM, Cédric Le Goater wrote:
> On 9/12/23 13:41, Thomas Huth wrote:
>> From: Janosch Frank <frankja@linux.ibm.com>
>>
>> Bound APQNs have to be reset before tearing down the secure config via
>> s390_machine_unprotect(). Otherwise the Ultravisor will return a error
>> code.
>>
>> So let's do a subsystem_reset() which includes a AP reset before the
>> unprotect call. We'll do a full device_reset() afterwards which will
>> reset some devices twice. That's ok since we can't move the
>> device_reset() before the unprotect as it includes a CPU clear reset
>> which the Ultravisor does not expect at that point in time.
>>
>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>> Message-ID: <20230901114851.154357-1-frankja@linux.ibm.com>
>> Tested-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
>> Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>> hw/s390x/s390-virtio-ccw.c | 10 ++++++++++
>> 1 file changed, 10 insertions(+)
>>
>> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
>> index 3dd0b2372d..2d75f2131f 100644
>> --- a/hw/s390x/s390-virtio-ccw.c
>> +++ b/hw/s390x/s390-virtio-ccw.c
>> @@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
>> switch (reset_type) {
>> case S390_RESET_EXTERNAL:
>> case S390_RESET_REIPL:
>> + /*
>> + * Reset the subsystem which includes a AP reset. If a PV
>> + * guest had APQNs attached the AP reset is a prerequisite to
>> + * unprotecting since the UV checks if all APQNs are reset.
>> + */
>> + subsystem_reset();
>
>
> This commit introduced a regression with pass-though ISM devices.
>
> After startup, a reboot will generate extra device resets (vfio-pci in
> this case) which break the pass-though ISM device in a subtle way,
Hi Cedric, thanks for reporting this... I was able to reproduce just now, and it looks like ISM firmware is unhappy specifically with this susbystem_reset call added by ef1535901a0, not necessarily the multiple attempts at reset -- I verified that reverting ef1535901a0 resolves the ISM issue, but if I instead try reverting the older 03451953c79e while leaving ef1535901a0 in place then ISM devices still break on guest reboot.
> probably related to IOMMU mapping according to 03451953c79e
> ("s390x/pci: reset ISM passthrough devices on shutdown and system
> reset"). After poweroff, the device is left in a sort-of-a-use state
> on the host and the LPAR has to be rebooted to clear the invalid state
> of the device. To be noted, that standard PCI devices are immune to
> this change.
As a bit of background, ISM firmware is very sensitive re: the contents of the (host) IOMMU and attempts at manipulation that it deems to be out-of-order; the point of 03451953c79e was to ensure that the device gets a reset before we attempt at unmapping anything that wasn't cleaned up in an orderly fashion by the (guest) ism driver at the time of shutdown/reset (e.g. underlying firmware may view guest SBAs in the IOMMU as still registered for use and will throw an error condition at attempts to remove their entries in the IOMMU without first going through an unregistration process).
The unmap that would make ISM upset would generally be coming out of vfio_listener_region_del where we just do one big vfio_dma_unmap -- a quick trace shows that the subsystem_reset call added by ef1535901a0 is causing the vfio_listener_region_del to once again trigger before the pci reset of the ISM device, effectively re-introducing the condition that 03451953c79e was trying to resolve.
>
> The extra resets should avoided in some ways, (a shutdown notifier and
> a reset callback are already registered for ISM devices by 03451953c79e)
So as mentioned above, it's not the extra resets that are the issue, it's the order of operations. Basically, we need to drive pci_device_reset for any ISM device associated with the guest before we destroy the vfio memory listener (now triggered in this case via subsystem_reset). So if we must drive this subsystem_reset before we trigger the device reset callbacks then it might require a s390 pci bus routine that is called before or during subystem_reset just to reset the ISM devices associated with this guest first; I'm not sure yet.
As an aside: I wonder why we are always doing the subsystem_reset here unconditionally rather than only when s390_is_pv() since that seems to be the only case that requires it.
> and, most important, once the VM terminates, the device resources
> should be cleared in the host kernel. So there seem to be two issues
> to address in mainline QEMU and in Linux AFAICT.
Because of the condition detected by ISM firmware as described above, the host device was placed in an error state and remains in that state. After shutting down the guest, you should be able to use zpcictl --reset on the affected host device(s) to clear the error condition and re-enable it for use.
Thanks,
Matt
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot
2024-01-10 20:28 ` Matthew Rosato
@ 2024-01-11 9:43 ` Cédric Le Goater
2024-01-11 10:18 ` Christian Borntraeger
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Cédric Le Goater @ 2024-01-11 9:43 UTC (permalink / raw)
To: Matthew Rosato, Thomas Huth, qemu-devel
Cc: Stefan Hajnoczi, Janosch Frank, Viktor Mihajlovski,
Christian Borntraeger, Eric Farman
On 1/10/24 21:28, Matthew Rosato wrote:
> On 1/10/24 1:30 PM, Cédric Le Goater wrote:
>> On 9/12/23 13:41, Thomas Huth wrote:
>>> From: Janosch Frank <frankja@linux.ibm.com>
>>>
>>> Bound APQNs have to be reset before tearing down the secure config via
>>> s390_machine_unprotect(). Otherwise the Ultravisor will return a error
>>> code.
>>>
>>> So let's do a subsystem_reset() which includes a AP reset before the
>>> unprotect call. We'll do a full device_reset() afterwards which will
>>> reset some devices twice. That's ok since we can't move the
>>> device_reset() before the unprotect as it includes a CPU clear reset
>>> which the Ultravisor does not expect at that point in time.
>>>
>>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>>> Message-ID: <20230901114851.154357-1-frankja@linux.ibm.com>
>>> Tested-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
>>> Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>> hw/s390x/s390-virtio-ccw.c | 10 ++++++++++
>>> 1 file changed, 10 insertions(+)
>>>
>>> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
>>> index 3dd0b2372d..2d75f2131f 100644
>>> --- a/hw/s390x/s390-virtio-ccw.c
>>> +++ b/hw/s390x/s390-virtio-ccw.c
>>> @@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
>>> switch (reset_type) {
>>> case S390_RESET_EXTERNAL:
>>> case S390_RESET_REIPL:
>>> + /*
>>> + * Reset the subsystem which includes a AP reset. If a PV
>>> + * guest had APQNs attached the AP reset is a prerequisite to
>>> + * unprotecting since the UV checks if all APQNs are reset.
>>> + */
>>> + subsystem_reset();
>>
>>
>> This commit introduced a regression with pass-though ISM devices.
>>
>> After startup, a reboot will generate extra device resets (vfio-pci in
>> this case) which break the pass-though ISM device in a subtle way,
>
> Hi Cedric, thanks for reporting this... I was able to reproduce just now, and it looks like ISM firmware is unhappy specifically with this susbystem_reset call added by ef1535901a0, not necessarily the multiple attempts at reset -- I verified that reverting ef1535901a0 resolves the ISM issue, but if I instead try reverting the older 03451953c79e while leaving ef1535901a0 in place then ISM devices still break on guest reboot.
>
>
>> probably related to IOMMU mapping according to 03451953c79e
>> ("s390x/pci: reset ISM passthrough devices on shutdown and system
>> reset"). After poweroff, the device is left in a sort-of-a-use state
>> on the host and the LPAR has to be rebooted to clear the invalid state
>> of the device. To be noted, that standard PCI devices are immune to
>> this change.
>
> As a bit of background, ISM firmware is very sensitive re: the contents of the (host) IOMMU and attempts at manipulation that it deems to be out-of-order; the point of 03451953c79e was to ensure that the device gets a reset before we attempt at unmapping anything that wasn't cleaned up in an orderly fashion by the (guest) ism driver at the time of shutdown/reset (e.g. underlying firmware may view guest SBAs in the IOMMU as still registered for use and will throw an error condition at attempts to remove their entries in the IOMMU without first going through an unregistration process).
>
> The unmap that would make ISM upset would generally be coming out of vfio_listener_region_del where we just do one big vfio_dma_unmap -- a quick trace shows that the subsystem_reset call added by ef1535901a0 is causing the vfio_listener_region_del to once again trigger before the pci reset of the ISM device, effectively re-introducing the condition that 03451953c79e was trying to resolve.
Yes. I saw the vfio_listener_region_del trace coming first and came to
the conclusion it was related to IOMMU mappings.
>> The extra resets should avoided in some ways, (a shutdown notifier and
>> a reset callback are already registered for ISM devices by 03451953c79e)
>
> So as mentioned above, it's not the extra resets that are the issue, it's the order of operations. Basically, we need to drive pci_device_reset for any ISM device associated with the guest before we destroy the vfio memory listener (now triggered in this case via subsystem_reset). So if we must drive this subsystem_reset before we trigger the device reset callbacks then it might require a s390 pci bus routine that is called before or during subystem_reset just to reset the ISM devices associated with this guest first; I'm not sure yet.
>
> As an aside: I wonder why we are always doing the subsystem_reset here unconditionally rather than only when s390_is_pv() since that seems to be the only case that requires it.
That would be a start to workaround the issue.
>> and, most important, once the VM terminates, the device resources
>> should be cleared in the host kernel. So there seem to be two issues
>> to address in mainline QEMU and in Linux AFAICT.
>
> Because of the condition detected by ISM firmware as described above, the host device was placed in an error state and remains in that state.
OK. this condition is considered serious enough to be reported to a
management level. This seems a bit excessive since the recovery can be
handled by software, but manually. Are there any plans to address this
problem ?
> After shutting down the guest, you should be able to use zpcictl --reset on the affected host device(s) to clear the error condition and re-enable it for use.
ok. That's better than reboot.
On a side note, I am also seeing :
[ 73.989688] ------------[ cut here ]------------
[ 73.989696] unexpected non zero alert.mask 0x20
[ 73.989748] WARNING: CPU: 9 PID: 4503 at arch/s390/kvm/interrupt.c:3214 kvm_s390_gisa_destroy+0xd4/0xe8 [kvm]
[ 73.989791] Modules linked in: vfio_pci vfio_pci_core irqbypass vhost_net vhost vhost_iotlb tap tun xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink 8021q garp mrp rfkill sunrpc ext4 mbcache jbd2 vfio_ap zcrypt_cex4 vfio_ccw mdev vfio_iommu_type1 vfio drm fuse i2c_core drm_panel_orientation_quirks xfs libcrc32c dm_service_time mlx5_core sd_mod t10_pi ghash_s390 sg prng des_s390 libdes sha3_512_s390 sha3_256_s390 mlxfw tls scm_block psample eadm_sch qeth_l2 bridge stp llc dasd_eckd_mod zfcp qeth dasd_mod scsi_transport_fc ccwgroup qdio dm_multipath dm_mirror dm_region_hash dm_log dm_mod pkey zcrypt kvm aes_s390
[ 73.989825] CPU: 9 PID: 4503 Comm: worker Kdump: loaded Not tainted 6.7.0-clg-dirty #52
[ 73.989827] Hardware name: IBM 3931 LA1 400 (LPAR)
[ 73.989829] Krnl PSW : 0704c00180000000 000003ff7fcd2198 (kvm_s390_gisa_destroy+0xd8/0xe8 [kvm])
[ 73.989845] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[ 73.989847] Krnl GPRS: c0000000fffeffff 0000000700000027 0000000000000023 00000007df4249c8
[ 73.989849] 000003800649b858 000003800649b850 00000007fcb9db00 0000000000000000
[ 73.989851] 000000008ebae8c8 0000000083a8c4f0 0000000000b69900 000000008ebac000
[ 73.989853] 000003ff903aef68 000003800649bd98 000003ff7fcd2194 000003800649b9f8
[ 73.989859] Krnl Code: 000003ff7fcd2188: c02000024f88 larl %r2,000003ff7fd1c098
000003ff7fcd218e: c0e5fffea360 brasl %r14,000003ff7fca684e
#000003ff7fcd2194: af000000 mc 0,0
>000003ff7fcd2198: e310b7680204 lg %r1,10088(%r11)
000003ff7fcd219e: a7f4ffae brc 15,000003ff7fcd20fa
000003ff7fcd21a2: 0707 bcr 0,%r7
000003ff7fcd21a4: 0707 bcr 0,%r7
000003ff7fcd21a6: 0707 bcr 0,%r7
[ 73.989929] Call Trace:
[ 73.989931] [<000003ff7fcd2198>] kvm_s390_gisa_destroy+0xd8/0xe8 [kvm]
[ 73.989946] ([<000003ff7fcd2194>] kvm_s390_gisa_destroy+0xd4/0xe8 [kvm])
[ 73.989960] [<000003ff7fcc1578>] kvm_arch_destroy_vm+0x50/0x118 [kvm]
[ 73.989974] [<000003ff7fcb00a2>] kvm_destroy_vm+0x15a/0x260 [kvm]
[ 73.989985] [<000003ff7fcb021e>] kvm_vm_release+0x36/0x48 [kvm]
[ 73.989996] [<00000007de4f830c>] __fput+0x94/0x2d0
[ 73.990009] [<00000007de20d838>] task_work_run+0x88/0xe8
[ 73.990013] [<00000007de1e75e0>] do_exit+0x2e0/0x4e0
[ 73.990016] [<00000007de1e79c0>] do_group_exit+0x40/0xb8
[ 73.990017] [<00000007de1f96e8>] send_sig_info+0x0/0xa8
[ 73.990021] [<00000007de194b26>] arch_do_signal_or_restart+0x56/0x318
[ 73.990025] [<00000007de28bf12>] exit_to_user_mode_prepare+0x10a/0x1a0
[ 73.990028] [<00000007deb607d2>] __do_syscall+0x152/0x1f8
[ 73.990032] [<00000007deb70ac8>] system_call+0x70/0x98
[ 73.990036] Last Breaking-Event-Address:
[ 73.990037] [<00000007de1e0c58>] __warn_printk+0x78/0xe8
Thanks,
C.
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot
2024-01-11 9:43 ` Cédric Le Goater
@ 2024-01-11 10:18 ` Christian Borntraeger
2024-01-11 15:26 ` Matthew Rosato
2024-01-11 15:00 ` Janosch Frank
2024-01-11 15:27 ` Matthew Rosato
2 siblings, 1 reply; 15+ messages in thread
From: Christian Borntraeger @ 2024-01-11 10:18 UTC (permalink / raw)
To: Cédric Le Goater, Matthew Rosato, Thomas Huth, qemu-devel,
Michael Mueller
Cc: Stefan Hajnoczi, Janosch Frank, Eric Farman
Am 11.01.24 um 10:43 schrieb Cédric Le Goater:
[...]
>
>
> On a side note, I am also seeing :
Michael?
>
> [ 73.989688] ------------[ cut here ]------------
> [ 73.989696] unexpected non zero alert.mask 0x20
> [ 73.989748] WARNING: CPU: 9 PID: 4503 at arch/s390/kvm/interrupt.c:3214 kvm_s390_gisa_destroy+0xd4/0xe8 [kvm]
> [ 73.989791] Modules linked in: vfio_pci vfio_pci_core irqbypass vhost_net vhost vhost_iotlb tap tun xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink 8021q garp mrp rfkill sunrpc ext4 mbcache jbd2 vfio_ap zcrypt_cex4 vfio_ccw mdev vfio_iommu_type1 vfio drm fuse i2c_core drm_panel_orientation_quirks xfs libcrc32c dm_service_time mlx5_core sd_mod t10_pi ghash_s390 sg prng des_s390 libdes sha3_512_s390 sha3_256_s390 mlxfw tls scm_block psample eadm_sch qeth_l2 bridge stp llc dasd_eckd_mod zfcp qeth dasd_mod scsi_transport_fc ccwgroup qdio dm_multipath dm_mirror dm_region_hash dm_log dm_mod pkey zcrypt kvm aes_s390
> [ 73.989825] CPU: 9 PID: 4503 Comm: worker Kdump: loaded Not tainted 6.7.0-clg-dirty #52
> [ 73.989827] Hardware name: IBM 3931 LA1 400 (LPAR)
> [ 73.989829] Krnl PSW : 0704c00180000000 000003ff7fcd2198 (kvm_s390_gisa_destroy+0xd8/0xe8 [kvm])
> [ 73.989845] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
> [ 73.989847] Krnl GPRS: c0000000fffeffff 0000000700000027 0000000000000023 00000007df4249c8
> [ 73.989849] 000003800649b858 000003800649b850 00000007fcb9db00 0000000000000000
> [ 73.989851] 000000008ebae8c8 0000000083a8c4f0 0000000000b69900 000000008ebac000
> [ 73.989853] 000003ff903aef68 000003800649bd98 000003ff7fcd2194 000003800649b9f8
> [ 73.989859] Krnl Code: 000003ff7fcd2188: c02000024f88 larl %r2,000003ff7fd1c098
> 000003ff7fcd218e: c0e5fffea360 brasl %r14,000003ff7fca684e
> #000003ff7fcd2194: af000000 mc 0,0
> >000003ff7fcd2198: e310b7680204 lg %r1,10088(%r11)
> 000003ff7fcd219e: a7f4ffae brc 15,000003ff7fcd20fa
> 000003ff7fcd21a2: 0707 bcr 0,%r7
> 000003ff7fcd21a4: 0707 bcr 0,%r7
> 000003ff7fcd21a6: 0707 bcr 0,%r7
> [ 73.989929] Call Trace:
> [ 73.989931] [<000003ff7fcd2198>] kvm_s390_gisa_destroy+0xd8/0xe8 [kvm]
> [ 73.989946] ([<000003ff7fcd2194>] kvm_s390_gisa_destroy+0xd4/0xe8 [kvm])
> [ 73.989960] [<000003ff7fcc1578>] kvm_arch_destroy_vm+0x50/0x118 [kvm]
> [ 73.989974] [<000003ff7fcb00a2>] kvm_destroy_vm+0x15a/0x260 [kvm]
> [ 73.989985] [<000003ff7fcb021e>] kvm_vm_release+0x36/0x48 [kvm]
> [ 73.989996] [<00000007de4f830c>] __fput+0x94/0x2d0
> [ 73.990009] [<00000007de20d838>] task_work_run+0x88/0xe8
> [ 73.990013] [<00000007de1e75e0>] do_exit+0x2e0/0x4e0
> [ 73.990016] [<00000007de1e79c0>] do_group_exit+0x40/0xb8
> [ 73.990017] [<00000007de1f96e8>] send_sig_info+0x0/0xa8
> [ 73.990021] [<00000007de194b26>] arch_do_signal_or_restart+0x56/0x318
> [ 73.990025] [<00000007de28bf12>] exit_to_user_mode_prepare+0x10a/0x1a0
> [ 73.990028] [<00000007deb607d2>] __do_syscall+0x152/0x1f8
> [ 73.990032] [<00000007deb70ac8>] system_call+0x70/0x98
> [ 73.990036] Last Breaking-Event-Address:
> [ 73.990037] [<00000007de1e0c58>] __warn_printk+0x78/0xe8
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot
2024-01-11 10:18 ` Christian Borntraeger
@ 2024-01-11 15:26 ` Matthew Rosato
0 siblings, 0 replies; 15+ messages in thread
From: Matthew Rosato @ 2024-01-11 15:26 UTC (permalink / raw)
To: Christian Borntraeger, Cédric Le Goater, Thomas Huth,
qemu-devel, Michael Mueller
Cc: Stefan Hajnoczi, Janosch Frank, Eric Farman
On 1/11/24 5:18 AM, Christian Borntraeger wrote:
>
>
> Am 11.01.24 um 10:43 schrieb Cédric Le Goater:
> [...]
>>
>>
>> On a side note, I am also seeing :
>
> Michael?
>
Hmm, it looks like this warning is tripping because we have a path in PCI passthrough where we don't unregister the gisc. This warning notices that at the time we destroy the VM (it believes that are still consumers of the guest ISC so the bit in the alert mask is still on) -- after cutting the warning the code forces the alerts off at least.
Not sure yet if that is directly related to the device going into error condition or if it is an independent issue, will have a look.
>>
>> [ 73.989688] ------------[ cut here ]------------
>> [ 73.989696] unexpected non zero alert.mask 0x20
>> [ 73.989748] WARNING: CPU: 9 PID: 4503 at arch/s390/kvm/interrupt.c:3214 kvm_s390_gisa_destroy+0xd4/0xe8 [kvm]
>> [ 73.989791] Modules linked in: vfio_pci vfio_pci_core irqbypass vhost_net vhost vhost_iotlb tap tun xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink 8021q garp mrp rfkill sunrpc ext4 mbcache jbd2 vfio_ap zcrypt_cex4 vfio_ccw mdev vfio_iommu_type1 vfio drm fuse i2c_core drm_panel_orientation_quirks xfs libcrc32c dm_service_time mlx5_core sd_mod t10_pi ghash_s390 sg prng des_s390 libdes sha3_512_s390 sha3_256_s390 mlxfw tls scm_block psample eadm_sch qeth_l2 bridge stp llc dasd_eckd_mod zfcp qeth dasd_mod scsi_transport_fc ccwgroup qdio dm_multipath dm_mirror dm_region_hash dm_log dm_mod pkey zcrypt kvm aes_s390
>> [ 73.989825] CPU: 9 PID: 4503 Comm: worker Kdump: loaded Not tainted 6.7.0-clg-dirty #52
>> [ 73.989827] Hardware name: IBM 3931 LA1 400 (LPAR)
>> [ 73.989829] Krnl PSW : 0704c00180000000 000003ff7fcd2198 (kvm_s390_gisa_destroy+0xd8/0xe8 [kvm])
>> [ 73.989845] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
>> [ 73.989847] Krnl GPRS: c0000000fffeffff 0000000700000027 0000000000000023 00000007df4249c8
>> [ 73.989849] 000003800649b858 000003800649b850 00000007fcb9db00 0000000000000000
>> [ 73.989851] 000000008ebae8c8 0000000083a8c4f0 0000000000b69900 000000008ebac000
>> [ 73.989853] 000003ff903aef68 000003800649bd98 000003ff7fcd2194 000003800649b9f8
>> [ 73.989859] Krnl Code: 000003ff7fcd2188: c02000024f88 larl %r2,000003ff7fd1c098
>> 000003ff7fcd218e: c0e5fffea360 brasl %r14,000003ff7fca684e
>> #000003ff7fcd2194: af000000 mc 0,0
>> >000003ff7fcd2198: e310b7680204 lg %r1,10088(%r11)
>> 000003ff7fcd219e: a7f4ffae brc 15,000003ff7fcd20fa
>> 000003ff7fcd21a2: 0707 bcr 0,%r7
>> 000003ff7fcd21a4: 0707 bcr 0,%r7
>> 000003ff7fcd21a6: 0707 bcr 0,%r7
>> [ 73.989929] Call Trace:
>> [ 73.989931] [<000003ff7fcd2198>] kvm_s390_gisa_destroy+0xd8/0xe8 [kvm]
>> [ 73.989946] ([<000003ff7fcd2194>] kvm_s390_gisa_destroy+0xd4/0xe8 [kvm])
>> [ 73.989960] [<000003ff7fcc1578>] kvm_arch_destroy_vm+0x50/0x118 [kvm]
>> [ 73.989974] [<000003ff7fcb00a2>] kvm_destroy_vm+0x15a/0x260 [kvm]
>> [ 73.989985] [<000003ff7fcb021e>] kvm_vm_release+0x36/0x48 [kvm]
>> [ 73.989996] [<00000007de4f830c>] __fput+0x94/0x2d0
>> [ 73.990009] [<00000007de20d838>] task_work_run+0x88/0xe8
>> [ 73.990013] [<00000007de1e75e0>] do_exit+0x2e0/0x4e0
>> [ 73.990016] [<00000007de1e79c0>] do_group_exit+0x40/0xb8
>> [ 73.990017] [<00000007de1f96e8>] send_sig_info+0x0/0xa8
>> [ 73.990021] [<00000007de194b26>] arch_do_signal_or_restart+0x56/0x318
>> [ 73.990025] [<00000007de28bf12>] exit_to_user_mode_prepare+0x10a/0x1a0
>> [ 73.990028] [<00000007deb607d2>] __do_syscall+0x152/0x1f8
>> [ 73.990032] [<00000007deb70ac8>] system_call+0x70/0x98
>> [ 73.990036] Last Breaking-Event-Address:
>> [ 73.990037] [<00000007de1e0c58>] __warn_printk+0x78/0xe8
>>
>>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot
2024-01-11 9:43 ` Cédric Le Goater
2024-01-11 10:18 ` Christian Borntraeger
@ 2024-01-11 15:00 ` Janosch Frank
2024-01-11 15:27 ` Matthew Rosato
2 siblings, 0 replies; 15+ messages in thread
From: Janosch Frank @ 2024-01-11 15:00 UTC (permalink / raw)
To: Cédric Le Goater, Matthew Rosato, Thomas Huth, qemu-devel
Cc: Stefan Hajnoczi, Viktor Mihajlovski, Christian Borntraeger,
Eric Farman
On 1/11/24 10:43, Cédric Le Goater wrote:
> On 1/10/24 21:28, Matthew Rosato wrote:
>> On 1/10/24 1:30 PM, Cédric Le Goater wrote:
>>> On 9/12/23 13:41, Thomas Huth wrote:
>>>> From: Janosch Frank <frankja@linux.ibm.com>
>>>>
>>>> Bound APQNs have to be reset before tearing down the secure config via
>>>> s390_machine_unprotect(). Otherwise the Ultravisor will return a error
>>>> code.
>>>>
>>>> So let's do a subsystem_reset() which includes a AP reset before the
>>>> unprotect call. We'll do a full device_reset() afterwards which will
>>>> reset some devices twice. That's ok since we can't move the
>>>> device_reset() before the unprotect as it includes a CPU clear reset
>>>> which the Ultravisor does not expect at that point in time.
>>>>
>>>> Signed-off-by: Janosch Frank <frankja@linux.ibm.com>
>>>> Message-ID: <20230901114851.154357-1-frankja@linux.ibm.com>
>>>> Tested-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
>>>> Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com>
>>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>>> ---
>>>> hw/s390x/s390-virtio-ccw.c | 10 ++++++++++
>>>> 1 file changed, 10 insertions(+)
>>>>
>>>> diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c
>>>> index 3dd0b2372d..2d75f2131f 100644
>>>> --- a/hw/s390x/s390-virtio-ccw.c
>>>> +++ b/hw/s390x/s390-virtio-ccw.c
>>>> @@ -438,10 +438,20 @@ static void s390_machine_reset(MachineState *machine, ShutdownCause reason)
>>>> switch (reset_type) {
>>>> case S390_RESET_EXTERNAL:
>>>> case S390_RESET_REIPL:
>>>> + /*
>>>> + * Reset the subsystem which includes a AP reset. If a PV
>>>> + * guest had APQNs attached the AP reset is a prerequisite to
>>>> + * unprotecting since the UV checks if all APQNs are reset.
>>>> + */
>>>> + subsystem_reset();
>>>
>>>
>>> This commit introduced a regression with pass-though ISM devices.
>>>
>>> After startup, a reboot will generate extra device resets (vfio-pci in
>>> this case) which break the pass-though ISM device in a subtle way,
>>
>> Hi Cedric, thanks for reporting this... I was able to reproduce just now, and it looks like ISM firmware is unhappy specifically with this susbystem_reset call added by ef1535901a0, not necessarily the multiple attempts at reset -- I verified that reverting ef1535901a0 resolves the ISM issue, but if I instead try reverting the older 03451953c79e while leaving ef1535901a0 in place then ISM devices still break on guest reboot.
>>
>>
>>> probably related to IOMMU mapping according to 03451953c79e
>>> ("s390x/pci: reset ISM passthrough devices on shutdown and system
>>> reset"). After poweroff, the device is left in a sort-of-a-use state
>>> on the host and the LPAR has to be rebooted to clear the invalid state
>>> of the device. To be noted, that standard PCI devices are immune to
>>> this change.
>>
>> As a bit of background, ISM firmware is very sensitive re: the contents of the (host) IOMMU and attempts at manipulation that it deems to be out-of-order; the point of 03451953c79e was to ensure that the device gets a reset before we attempt at unmapping anything that wasn't cleaned up in an orderly fashion by the (guest) ism driver at the time of shutdown/reset (e.g. underlying firmware may view guest SBAs in the IOMMU as still registered for use and will throw an error condition at attempts to remove their entries in the IOMMU without first going through an unregistration process).
>>
>> The unmap that would make ISM upset would generally be coming out of vfio_listener_region_del where we just do one big vfio_dma_unmap -- a quick trace shows that the subsystem_reset call added by ef1535901a0 is causing the vfio_listener_region_del to once again trigger before the pci reset of the ISM device, effectively re-introducing the condition that 03451953c79e was trying to resolve.
>
> Yes. I saw the vfio_listener_region_del trace coming first and came to
> the conclusion it was related to IOMMU mappings.
>
>>> The extra resets should avoided in some ways, (a shutdown notifier and
>>> a reset callback are already registered for ISM devices by 03451953c79e)
>>
>> So as mentioned above, it's not the extra resets that are the issue, it's the order of operations. Basically, we need to drive pci_device_reset for any ISM device associated with the guest before we destroy the vfio memory listener (now triggered in this case via subsystem_reset). So if we must drive this subsystem_reset before we trigger the device reset callbacks then it might require a s390 pci bus routine that is called before or during subystem_reset just to reset the ISM devices associated with this guest first; I'm not sure yet.
>>
>> As an aside: I wonder why we are always doing the subsystem_reset here unconditionally rather than only when s390_is_pv() since that seems to be the only case that requires it.
>
> That would be a start to workaround the issue.
>
I can have a look into that, I'm just a bit hesitant since there's an
ominous feeling in my brain that tells me that there might have been a
reason that I forgot after this patch was pulled.
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot
2024-01-11 9:43 ` Cédric Le Goater
2024-01-11 10:18 ` Christian Borntraeger
2024-01-11 15:00 ` Janosch Frank
@ 2024-01-11 15:27 ` Matthew Rosato
2 siblings, 0 replies; 15+ messages in thread
From: Matthew Rosato @ 2024-01-11 15:27 UTC (permalink / raw)
To: Cédric Le Goater, Thomas Huth, qemu-devel
Cc: Stefan Hajnoczi, Janosch Frank, Viktor Mihajlovski,
Christian Borntraeger, Eric Farman
On 1/11/24 4:43 AM, Cédric Le Goater wrote:
> OK. this condition is considered serious enough to be reported to a
> management level. This seems a bit excessive since the recovery can be
> handled by software, but manually. Are there any plans to address this
> problem ?
Yes, eventually. Currently for drivers other than QEMU+vfio-pci we rely on automated PCI recovery to handle this event, but today we do not forward the event to QEMU when using vfio-pci because the code in vfio_err_notifier_handler will halt the guest. This leaves the device in an error state to be recovered manually. The intent is to eventually address this by triggering the guest to initiate the recovery action rather than halting the guest for s390.
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PULL 3/7] linux-headers: Update to Linux v6.6-rc1
2023-09-12 11:41 [PULL 0/7] s390x (and one qtest) patches Thomas Huth
2023-09-12 11:41 ` [PULL 1/7] s390x/ap: fix missing subsystem reset registration Thomas Huth
2023-09-12 11:41 ` [PULL 2/7] s390x: do a subsystem reset before the unprotect on reboot Thomas Huth
@ 2023-09-12 11:41 ` Thomas Huth
2023-09-12 11:41 ` [PULL 4/7] target/s390x/kvm: Refactor AP functionalities Thomas Huth
` (3 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Thomas Huth @ 2023-09-12 11:41 UTC (permalink / raw)
To: qemu-devel; +Cc: Stefan Hajnoczi
This update contains the required header changes for the
"target/s390x: AP-passthrough for PV guests" patch from
Steffen Eiden.
Message-ID: <20230912093432.180041-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
include/standard-headers/linux/fuse.h | 63 +++++++-
include/standard-headers/linux/vhost_types.h | 4 +
include/standard-headers/linux/virtio_net.h | 14 ++
| 23 +++
| 5 +-
| 1 +
| 1 +
| 1 +
| 1 +
| 1 +
| 13 ++
| 16 ++
| 16 ++
| 1 +
| 1 +
| 10 +-
| 1 +
| 2 +
| 1 +
| 13 +-
| 4 +
| 25 +++-
| 150 ++++++++++++++++++-
23 files changed, 351 insertions(+), 16 deletions(-)
diff --git a/include/standard-headers/linux/fuse.h b/include/standard-headers/linux/fuse.h
index 35c131a107..6b9793842c 100644
--- a/include/standard-headers/linux/fuse.h
+++ b/include/standard-headers/linux/fuse.h
@@ -206,6 +206,11 @@
* - add extension header
* - add FUSE_EXT_GROUPS
* - add FUSE_CREATE_SUPP_GROUP
+ * - add FUSE_HAS_EXPIRE_ONLY
+ *
+ * 7.39
+ * - add FUSE_DIRECT_IO_RELAX
+ * - add FUSE_STATX and related structures
*/
#ifndef _LINUX_FUSE_H
@@ -237,7 +242,7 @@
#define FUSE_KERNEL_VERSION 7
/** Minor version number of this interface */
-#define FUSE_KERNEL_MINOR_VERSION 38
+#define FUSE_KERNEL_MINOR_VERSION 39
/** The node ID of the root inode */
#define FUSE_ROOT_ID 1
@@ -264,6 +269,40 @@ struct fuse_attr {
uint32_t flags;
};
+/*
+ * The following structures are bit-for-bit compatible with the statx(2) ABI in
+ * Linux.
+ */
+struct fuse_sx_time {
+ int64_t tv_sec;
+ uint32_t tv_nsec;
+ int32_t __reserved;
+};
+
+struct fuse_statx {
+ uint32_t mask;
+ uint32_t blksize;
+ uint64_t attributes;
+ uint32_t nlink;
+ uint32_t uid;
+ uint32_t gid;
+ uint16_t mode;
+ uint16_t __spare0[1];
+ uint64_t ino;
+ uint64_t size;
+ uint64_t blocks;
+ uint64_t attributes_mask;
+ struct fuse_sx_time atime;
+ struct fuse_sx_time btime;
+ struct fuse_sx_time ctime;
+ struct fuse_sx_time mtime;
+ uint32_t rdev_major;
+ uint32_t rdev_minor;
+ uint32_t dev_major;
+ uint32_t dev_minor;
+ uint64_t __spare2[14];
+};
+
struct fuse_kstatfs {
uint64_t blocks;
uint64_t bfree;
@@ -365,6 +404,9 @@ struct fuse_file_lock {
* FUSE_HAS_INODE_DAX: use per inode DAX
* FUSE_CREATE_SUPP_GROUP: add supplementary group info to create, mkdir,
* symlink and mknod (single group that matches parent)
+ * FUSE_HAS_EXPIRE_ONLY: kernel supports expiry-only entry invalidation
+ * FUSE_DIRECT_IO_RELAX: relax restrictions in FOPEN_DIRECT_IO mode, for now
+ * allow shared mmap
*/
#define FUSE_ASYNC_READ (1 << 0)
#define FUSE_POSIX_LOCKS (1 << 1)
@@ -402,6 +444,8 @@ struct fuse_file_lock {
#define FUSE_SECURITY_CTX (1ULL << 32)
#define FUSE_HAS_INODE_DAX (1ULL << 33)
#define FUSE_CREATE_SUPP_GROUP (1ULL << 34)
+#define FUSE_HAS_EXPIRE_ONLY (1ULL << 35)
+#define FUSE_DIRECT_IO_RELAX (1ULL << 36)
/**
* CUSE INIT request/reply flags
@@ -568,6 +612,7 @@ enum fuse_opcode {
FUSE_REMOVEMAPPING = 49,
FUSE_SYNCFS = 50,
FUSE_TMPFILE = 51,
+ FUSE_STATX = 52,
/* CUSE specific operations */
CUSE_INIT = 4096,
@@ -632,6 +677,22 @@ struct fuse_attr_out {
struct fuse_attr attr;
};
+struct fuse_statx_in {
+ uint32_t getattr_flags;
+ uint32_t reserved;
+ uint64_t fh;
+ uint32_t sx_flags;
+ uint32_t sx_mask;
+};
+
+struct fuse_statx_out {
+ uint64_t attr_valid; /* Cache timeout for the attributes */
+ uint32_t attr_valid_nsec;
+ uint32_t flags;
+ uint64_t spare[2];
+ struct fuse_statx stat;
+};
+
#define FUSE_COMPAT_MKNOD_IN_SIZE 8
struct fuse_mknod_in {
diff --git a/include/standard-headers/linux/vhost_types.h b/include/standard-headers/linux/vhost_types.h
index 6691a3ce24..5ad07e134a 100644
--- a/include/standard-headers/linux/vhost_types.h
+++ b/include/standard-headers/linux/vhost_types.h
@@ -181,5 +181,9 @@ struct vhost_vdpa_iova_range {
#define VHOST_BACKEND_F_SUSPEND 0x4
/* Device can be resumed */
#define VHOST_BACKEND_F_RESUME 0x5
+/* Device supports the driver enabling virtqueues both before and after
+ * DRIVER_OK
+ */
+#define VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK 0x6
#endif
diff --git a/include/standard-headers/linux/virtio_net.h b/include/standard-headers/linux/virtio_net.h
index 2325485f2c..0f88417742 100644
--- a/include/standard-headers/linux/virtio_net.h
+++ b/include/standard-headers/linux/virtio_net.h
@@ -56,6 +56,7 @@
#define VIRTIO_NET_F_MQ 22 /* Device supports Receive Flow
* Steering */
#define VIRTIO_NET_F_CTRL_MAC_ADDR 23 /* Set MAC address */
+#define VIRTIO_NET_F_VQ_NOTF_COAL 52 /* Device supports virtqueue notification coalescing */
#define VIRTIO_NET_F_NOTF_COAL 53 /* Device supports notifications coalescing */
#define VIRTIO_NET_F_GUEST_USO4 54 /* Guest can handle USOv4 in. */
#define VIRTIO_NET_F_GUEST_USO6 55 /* Guest can handle USOv6 in. */
@@ -391,5 +392,18 @@ struct virtio_net_ctrl_coal_rx {
};
#define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+#define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+#define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+
+struct virtio_net_ctrl_coal {
+ uint32_t max_packets;
+ uint32_t max_usecs;
+};
+
+struct virtio_net_ctrl_coal_vq {
+ uint16_t vqn;
+ uint16_t reserved;
+ struct virtio_net_ctrl_coal coal;
+};
#endif /* _LINUX_VIRTIO_NET_H */
--git a/linux-headers/asm-arm64/bitsperlong.h b/linux-headers/asm-arm64/bitsperlong.h
index 6dc0bb0c13..485d60bee2 100644
--- a/linux-headers/asm-arm64/bitsperlong.h
+++ b/linux-headers/asm-arm64/bitsperlong.h
@@ -1 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ASM_BITSPERLONG_H
+#define __ASM_BITSPERLONG_H
+
+#define __BITS_PER_LONG 64
+
#include <asm-generic/bitsperlong.h>
+
+#endif /* __ASM_BITSPERLONG_H */
--git a/linux-headers/asm-generic/unistd.h b/linux-headers/asm-generic/unistd.h
index fd6c1cb585..abe087c53b 100644
--- a/linux-headers/asm-generic/unistd.h
+++ b/linux-headers/asm-generic/unistd.h
@@ -820,8 +820,11 @@ __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node)
#define __NR_cachestat 451
__SYSCALL(__NR_cachestat, sys_cachestat)
+#define __NR_fchmodat2 452
+__SYSCALL(__NR_fchmodat2, sys_fchmodat2)
+
#undef __NR_syscalls
-#define __NR_syscalls 452
+#define __NR_syscalls 453
/*
* 32 bit systems traditionally used different
--git a/linux-headers/asm-mips/unistd_n32.h b/linux-headers/asm-mips/unistd_n32.h
index 8233f061c4..46d8500654 100644
--- a/linux-headers/asm-mips/unistd_n32.h
+++ b/linux-headers/asm-mips/unistd_n32.h
@@ -380,5 +380,6 @@
#define __NR_futex_waitv (__NR_Linux + 449)
#define __NR_set_mempolicy_home_node (__NR_Linux + 450)
#define __NR_cachestat (__NR_Linux + 451)
+#define __NR_fchmodat2 (__NR_Linux + 452)
#endif /* _ASM_UNISTD_N32_H */
--git a/linux-headers/asm-mips/unistd_n64.h b/linux-headers/asm-mips/unistd_n64.h
index a174edc768..c2f7ac673b 100644
--- a/linux-headers/asm-mips/unistd_n64.h
+++ b/linux-headers/asm-mips/unistd_n64.h
@@ -356,5 +356,6 @@
#define __NR_futex_waitv (__NR_Linux + 449)
#define __NR_set_mempolicy_home_node (__NR_Linux + 450)
#define __NR_cachestat (__NR_Linux + 451)
+#define __NR_fchmodat2 (__NR_Linux + 452)
#endif /* _ASM_UNISTD_N64_H */
--git a/linux-headers/asm-mips/unistd_o32.h b/linux-headers/asm-mips/unistd_o32.h
index c1a5351d9b..757c68f2ad 100644
--- a/linux-headers/asm-mips/unistd_o32.h
+++ b/linux-headers/asm-mips/unistd_o32.h
@@ -426,5 +426,6 @@
#define __NR_futex_waitv (__NR_Linux + 449)
#define __NR_set_mempolicy_home_node (__NR_Linux + 450)
#define __NR_cachestat (__NR_Linux + 451)
+#define __NR_fchmodat2 (__NR_Linux + 452)
#endif /* _ASM_UNISTD_O32_H */
--git a/linux-headers/asm-powerpc/unistd_32.h b/linux-headers/asm-powerpc/unistd_32.h
index 8206758691..8ef94bbac1 100644
--- a/linux-headers/asm-powerpc/unistd_32.h
+++ b/linux-headers/asm-powerpc/unistd_32.h
@@ -433,6 +433,7 @@
#define __NR_futex_waitv 449
#define __NR_set_mempolicy_home_node 450
#define __NR_cachestat 451
+#define __NR_fchmodat2 452
#endif /* _ASM_UNISTD_32_H */
--git a/linux-headers/asm-powerpc/unistd_64.h b/linux-headers/asm-powerpc/unistd_64.h
index 7be98c15f0..0e7ee43e88 100644
--- a/linux-headers/asm-powerpc/unistd_64.h
+++ b/linux-headers/asm-powerpc/unistd_64.h
@@ -405,6 +405,7 @@
#define __NR_futex_waitv 449
#define __NR_set_mempolicy_home_node 450
#define __NR_cachestat 451
+#define __NR_fchmodat2 452
#endif /* _ASM_UNISTD_64_H */
--git a/linux-headers/asm-riscv/bitsperlong.h b/linux-headers/asm-riscv/bitsperlong.h
index 6dc0bb0c13..cc5c45a9ce 100644
--- a/linux-headers/asm-riscv/bitsperlong.h
+++ b/linux-headers/asm-riscv/bitsperlong.h
@@ -1 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ * Copyright (C) 2015 Regents of the University of California
+ */
+
+#ifndef _ASM_RISCV_BITSPERLONG_H
+#define _ASM_RISCV_BITSPERLONG_H
+
+#define __BITS_PER_LONG (__SIZEOF_POINTER__ * 8)
+
#include <asm-generic/bitsperlong.h>
+
+#endif /* _ASM_RISCV_BITSPERLONG_H */
--git a/linux-headers/asm-riscv/kvm.h b/linux-headers/asm-riscv/kvm.h
index 930fdc4101..992c5e4071 100644
--- a/linux-headers/asm-riscv/kvm.h
+++ b/linux-headers/asm-riscv/kvm.h
@@ -55,6 +55,7 @@ struct kvm_riscv_config {
unsigned long marchid;
unsigned long mimpid;
unsigned long zicboz_block_size;
+ unsigned long satp_mode;
};
/* CORE registers for KVM_GET_ONE_REG and KVM_SET_ONE_REG */
@@ -124,6 +125,12 @@ enum KVM_RISCV_ISA_EXT_ID {
KVM_RISCV_ISA_EXT_SSAIA,
KVM_RISCV_ISA_EXT_V,
KVM_RISCV_ISA_EXT_SVNAPOT,
+ KVM_RISCV_ISA_EXT_ZBA,
+ KVM_RISCV_ISA_EXT_ZBS,
+ KVM_RISCV_ISA_EXT_ZICNTR,
+ KVM_RISCV_ISA_EXT_ZICSR,
+ KVM_RISCV_ISA_EXT_ZIFENCEI,
+ KVM_RISCV_ISA_EXT_ZIHPM,
KVM_RISCV_ISA_EXT_MAX,
};
@@ -193,6 +200,15 @@ enum KVM_RISCV_SBI_EXT_ID {
/* ISA Extension registers are mapped as type 7 */
#define KVM_REG_RISCV_ISA_EXT (0x07 << KVM_REG_RISCV_TYPE_SHIFT)
+#define KVM_REG_RISCV_ISA_SINGLE (0x0 << KVM_REG_RISCV_SUBTYPE_SHIFT)
+#define KVM_REG_RISCV_ISA_MULTI_EN (0x1 << KVM_REG_RISCV_SUBTYPE_SHIFT)
+#define KVM_REG_RISCV_ISA_MULTI_DIS (0x2 << KVM_REG_RISCV_SUBTYPE_SHIFT)
+#define KVM_REG_RISCV_ISA_MULTI_REG(__ext_id) \
+ ((__ext_id) / __BITS_PER_LONG)
+#define KVM_REG_RISCV_ISA_MULTI_MASK(__ext_id) \
+ (1UL << ((__ext_id) % __BITS_PER_LONG))
+#define KVM_REG_RISCV_ISA_MULTI_REG_LAST \
+ KVM_REG_RISCV_ISA_MULTI_REG(KVM_RISCV_ISA_EXT_MAX - 1)
/* SBI extension registers are mapped as type 8 */
#define KVM_REG_RISCV_SBI_EXT (0x08 << KVM_REG_RISCV_TYPE_SHIFT)
--git a/linux-headers/asm-s390/kvm.h b/linux-headers/asm-s390/kvm.h
index e2afd95420..023a2763a9 100644
--- a/linux-headers/asm-s390/kvm.h
+++ b/linux-headers/asm-s390/kvm.h
@@ -159,6 +159,22 @@ struct kvm_s390_vm_cpu_subfunc {
__u8 reserved[1728];
};
+#define KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST 6
+#define KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST 7
+
+#define KVM_S390_VM_CPU_UV_FEAT_NR_BITS 64
+struct kvm_s390_vm_cpu_uv_feat {
+ union {
+ struct {
+ __u64 : 4;
+ __u64 ap : 1; /* bit 4 */
+ __u64 ap_intr : 1; /* bit 5 */
+ __u64 : 58;
+ };
+ __u64 feat;
+ };
+};
+
/* kvm attributes for crypto */
#define KVM_S390_VM_CRYPTO_ENABLE_AES_KW 0
#define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW 1
--git a/linux-headers/asm-s390/unistd_32.h b/linux-headers/asm-s390/unistd_32.h
index ef772cc5f8..716fa368ca 100644
--- a/linux-headers/asm-s390/unistd_32.h
+++ b/linux-headers/asm-s390/unistd_32.h
@@ -424,5 +424,6 @@
#define __NR_futex_waitv 449
#define __NR_set_mempolicy_home_node 450
#define __NR_cachestat 451
+#define __NR_fchmodat2 452
#endif /* _ASM_S390_UNISTD_32_H */
--git a/linux-headers/asm-s390/unistd_64.h b/linux-headers/asm-s390/unistd_64.h
index 32354a0459..b2a11b1d13 100644
--- a/linux-headers/asm-s390/unistd_64.h
+++ b/linux-headers/asm-s390/unistd_64.h
@@ -372,5 +372,6 @@
#define __NR_futex_waitv 449
#define __NR_set_mempolicy_home_node 450
#define __NR_cachestat 451
+#define __NR_fchmodat2 452
#endif /* _ASM_S390_UNISTD_64_H */
--git a/linux-headers/asm-x86/mman.h b/linux-headers/asm-x86/mman.h
index 775dbd3aff..46cdc941f9 100644
--- a/linux-headers/asm-x86/mman.h
+++ b/linux-headers/asm-x86/mman.h
@@ -3,14 +3,10 @@
#define _ASM_X86_MMAN_H
#define MAP_32BIT 0x40 /* only give out 32bit addresses */
+#define MAP_ABOVE4G 0x80 /* only map above 4GB */
-#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
-#define arch_calc_vm_prot_bits(prot, key) ( \
- ((key) & 0x1 ? VM_PKEY_BIT0 : 0) | \
- ((key) & 0x2 ? VM_PKEY_BIT1 : 0) | \
- ((key) & 0x4 ? VM_PKEY_BIT2 : 0) | \
- ((key) & 0x8 ? VM_PKEY_BIT3 : 0))
-#endif
+/* Flags for map_shadow_stack(2) */
+#define SHADOW_STACK_SET_TOKEN (1ULL << 0) /* Set up a restore token in the shadow stack */
#include <asm-generic/mman.h>
--git a/linux-headers/asm-x86/unistd_32.h b/linux-headers/asm-x86/unistd_32.h
index 37b32d8139..d749ad1c24 100644
--- a/linux-headers/asm-x86/unistd_32.h
+++ b/linux-headers/asm-x86/unistd_32.h
@@ -442,6 +442,7 @@
#define __NR_futex_waitv 449
#define __NR_set_mempolicy_home_node 450
#define __NR_cachestat 451
+#define __NR_fchmodat2 452
#endif /* _ASM_UNISTD_32_H */
--git a/linux-headers/asm-x86/unistd_64.h b/linux-headers/asm-x86/unistd_64.h
index 5b55d6729a..cea67282eb 100644
--- a/linux-headers/asm-x86/unistd_64.h
+++ b/linux-headers/asm-x86/unistd_64.h
@@ -364,6 +364,8 @@
#define __NR_futex_waitv 449
#define __NR_set_mempolicy_home_node 450
#define __NR_cachestat 451
+#define __NR_fchmodat2 452
+#define __NR_map_shadow_stack 453
#endif /* _ASM_UNISTD_64_H */
--git a/linux-headers/asm-x86/unistd_x32.h b/linux-headers/asm-x86/unistd_x32.h
index e8a007543d..5b2e79bf4c 100644
--- a/linux-headers/asm-x86/unistd_x32.h
+++ b/linux-headers/asm-x86/unistd_x32.h
@@ -317,6 +317,7 @@
#define __NR_futex_waitv (__X32_SYSCALL_BIT + 449)
#define __NR_set_mempolicy_home_node (__X32_SYSCALL_BIT + 450)
#define __NR_cachestat (__X32_SYSCALL_BIT + 451)
+#define __NR_fchmodat2 (__X32_SYSCALL_BIT + 452)
#define __NR_rt_sigaction (__X32_SYSCALL_BIT + 512)
#define __NR_rt_sigreturn (__X32_SYSCALL_BIT + 513)
#define __NR_ioctl (__X32_SYSCALL_BIT + 514)
--git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h
index 1f3f3333a4..0d74ee999a 100644
--- a/linux-headers/linux/kvm.h
+++ b/linux-headers/linux/kvm.h
@@ -1414,9 +1414,16 @@ struct kvm_device_attr {
__u64 addr; /* userspace address of attr data */
};
-#define KVM_DEV_VFIO_GROUP 1
-#define KVM_DEV_VFIO_GROUP_ADD 1
-#define KVM_DEV_VFIO_GROUP_DEL 2
+#define KVM_DEV_VFIO_FILE 1
+
+#define KVM_DEV_VFIO_FILE_ADD 1
+#define KVM_DEV_VFIO_FILE_DEL 2
+
+/* KVM_DEV_VFIO_GROUP aliases are for compile time uapi compatibility */
+#define KVM_DEV_VFIO_GROUP KVM_DEV_VFIO_FILE
+
+#define KVM_DEV_VFIO_GROUP_ADD KVM_DEV_VFIO_FILE_ADD
+#define KVM_DEV_VFIO_GROUP_DEL KVM_DEV_VFIO_FILE_DEL
#define KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE 3
enum kvm_device_type {
--git a/linux-headers/linux/stddef.h b/linux-headers/linux/stddef.h
index bb6ea517ef..9bb07083ac 100644
--- a/linux-headers/linux/stddef.h
+++ b/linux-headers/linux/stddef.h
@@ -45,3 +45,7 @@
TYPE NAME[]; \
}
#endif
+
+#ifndef __counted_by
+#define __counted_by(m)
+#endif
--git a/linux-headers/linux/userfaultfd.h b/linux-headers/linux/userfaultfd.h
index 14e402263a..59978fbaae 100644
--- a/linux-headers/linux/userfaultfd.h
+++ b/linux-headers/linux/userfaultfd.h
@@ -39,7 +39,8 @@
UFFD_FEATURE_MINOR_SHMEM | \
UFFD_FEATURE_EXACT_ADDRESS | \
UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \
- UFFD_FEATURE_WP_UNPOPULATED)
+ UFFD_FEATURE_WP_UNPOPULATED | \
+ UFFD_FEATURE_POISON)
#define UFFD_API_IOCTLS \
((__u64)1 << _UFFDIO_REGISTER | \
(__u64)1 << _UFFDIO_UNREGISTER | \
@@ -49,12 +50,14 @@
(__u64)1 << _UFFDIO_COPY | \
(__u64)1 << _UFFDIO_ZEROPAGE | \
(__u64)1 << _UFFDIO_WRITEPROTECT | \
- (__u64)1 << _UFFDIO_CONTINUE)
+ (__u64)1 << _UFFDIO_CONTINUE | \
+ (__u64)1 << _UFFDIO_POISON)
#define UFFD_API_RANGE_IOCTLS_BASIC \
((__u64)1 << _UFFDIO_WAKE | \
(__u64)1 << _UFFDIO_COPY | \
+ (__u64)1 << _UFFDIO_WRITEPROTECT | \
(__u64)1 << _UFFDIO_CONTINUE | \
- (__u64)1 << _UFFDIO_WRITEPROTECT)
+ (__u64)1 << _UFFDIO_POISON)
/*
* Valid ioctl command number range with this API is from 0x00 to
@@ -71,6 +74,7 @@
#define _UFFDIO_ZEROPAGE (0x04)
#define _UFFDIO_WRITEPROTECT (0x06)
#define _UFFDIO_CONTINUE (0x07)
+#define _UFFDIO_POISON (0x08)
#define _UFFDIO_API (0x3F)
/* userfaultfd ioctl ids */
@@ -91,6 +95,8 @@
struct uffdio_writeprotect)
#define UFFDIO_CONTINUE _IOWR(UFFDIO, _UFFDIO_CONTINUE, \
struct uffdio_continue)
+#define UFFDIO_POISON _IOWR(UFFDIO, _UFFDIO_POISON, \
+ struct uffdio_poison)
/* read() structure */
struct uffd_msg {
@@ -225,6 +231,7 @@ struct uffdio_api {
#define UFFD_FEATURE_EXACT_ADDRESS (1<<11)
#define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12)
#define UFFD_FEATURE_WP_UNPOPULATED (1<<13)
+#define UFFD_FEATURE_POISON (1<<14)
__u64 features;
__u64 ioctls;
@@ -321,6 +328,18 @@ struct uffdio_continue {
__s64 mapped;
};
+struct uffdio_poison {
+ struct uffdio_range range;
+#define UFFDIO_POISON_MODE_DONTWAKE ((__u64)1<<0)
+ __u64 mode;
+
+ /*
+ * Fields below here are written by the ioctl and must be at the end:
+ * the copy_from_user will not read past here.
+ */
+ __s64 updated;
+};
+
/*
* Flags for the userfaultfd(2) system call itself.
*/
--git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h
index 16db89071e..acf72b4999 100644
--- a/linux-headers/linux/vfio.h
+++ b/linux-headers/linux/vfio.h
@@ -217,6 +217,7 @@ struct vfio_device_info {
__u32 num_regions; /* Max region index + 1 */
__u32 num_irqs; /* Max IRQ index + 1 */
__u32 cap_offset; /* Offset within info struct of first cap */
+ __u32 pad;
};
#define VFIO_DEVICE_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 7)
@@ -677,11 +678,60 @@ enum {
* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12,
* struct vfio_pci_hot_reset_info)
*
+ * This command is used to query the affected devices in the hot reset for
+ * a given device.
+ *
+ * This command always reports the segment, bus, and devfn information for
+ * each affected device, and selectively reports the group_id or devid per
+ * the way how the calling device is opened.
+ *
+ * - If the calling device is opened via the traditional group/container
+ * API, group_id is reported. User should check if it has owned all
+ * the affected devices and provides a set of group fds to prove the
+ * ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl.
+ *
+ * - If the calling device is opened as a cdev, devid is reported.
+ * Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this
+ * data type. All the affected devices should be represented in
+ * the dev_set, ex. bound to a vfio driver, and also be owned by
+ * this interface which is determined by the following conditions:
+ * 1) Has a valid devid within the iommufd_ctx of the calling device.
+ * Ownership cannot be determined across separate iommufd_ctx and
+ * the cdev calling conventions do not support a proof-of-ownership
+ * model as provided in the legacy group interface. In this case
+ * valid devid with value greater than zero is provided in the return
+ * structure.
+ * 2) Does not have a valid devid within the iommufd_ctx of the calling
+ * device, but belongs to the same IOMMU group as the calling device
+ * or another opened device that has a valid devid within the
+ * iommufd_ctx of the calling device. This provides implicit ownership
+ * for devices within the same DMA isolation context. In this case
+ * the devid value of VFIO_PCI_DEVID_OWNED is provided in the return
+ * structure.
+ *
+ * A devid value of VFIO_PCI_DEVID_NOT_OWNED is provided in the return
+ * structure for affected devices where device is NOT represented in the
+ * dev_set or ownership is not available. Such devices prevent the use
+ * of VFIO_DEVICE_PCI_HOT_RESET ioctl outside of the proof-of-ownership
+ * calling conventions (ie. via legacy group accessed devices). Flag
+ * VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the
+ * affected devices are represented in the dev_set and also owned by
+ * the user. This flag is available only when
+ * flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved.
+ * When set, user could invoke VFIO_DEVICE_PCI_HOT_RESET with a zero
+ * length fd array on the calling device as the ownership is validated
+ * by iommufd_ctx.
+ *
* Return: 0 on success, -errno on failure:
* -enospc = insufficient buffer, -enodev = unsupported for device.
*/
struct vfio_pci_dependent_device {
- __u32 group_id;
+ union {
+ __u32 group_id;
+ __u32 devid;
+#define VFIO_PCI_DEVID_OWNED 0
+#define VFIO_PCI_DEVID_NOT_OWNED -1
+ };
__u16 segment;
__u8 bus;
__u8 devfn; /* Use PCI_SLOT/PCI_FUNC */
@@ -690,6 +740,8 @@ struct vfio_pci_dependent_device {
struct vfio_pci_hot_reset_info {
__u32 argsz;
__u32 flags;
+#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID (1 << 0)
+#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED (1 << 1)
__u32 count;
struct vfio_pci_dependent_device devices[];
};
@@ -700,6 +752,24 @@ struct vfio_pci_hot_reset_info {
* VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
* struct vfio_pci_hot_reset)
*
+ * A PCI hot reset results in either a bus or slot reset which may affect
+ * other devices sharing the bus/slot. The calling user must have
+ * ownership of the full set of affected devices as determined by the
+ * VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl.
+ *
+ * When called on a device file descriptor acquired through the vfio
+ * group interface, the user is required to provide proof of ownership
+ * of those affected devices via the group_fds array in struct
+ * vfio_pci_hot_reset.
+ *
+ * When called on a direct cdev opened vfio device, the flags field of
+ * struct vfio_pci_hot_reset_info reports the ownership status of the
+ * affected devices and this ioctl must be called with an empty group_fds
+ * array. See above INFO ioctl definition for ownership requirements.
+ *
+ * Mixed usage of legacy groups and cdevs across the set of affected
+ * devices is not supported.
+ *
* Return: 0 on success, -errno on failure.
*/
struct vfio_pci_hot_reset {
@@ -828,6 +898,83 @@ struct vfio_device_feature {
#define VFIO_DEVICE_FEATURE _IO(VFIO_TYPE, VFIO_BASE + 17)
+/*
+ * VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
+ * struct vfio_device_bind_iommufd)
+ * @argsz: User filled size of this data.
+ * @flags: Must be 0.
+ * @iommufd: iommufd to bind.
+ * @out_devid: The device id generated by this bind. devid is a handle for
+ * this device/iommufd bond and can be used in IOMMUFD commands.
+ *
+ * Bind a vfio_device to the specified iommufd.
+ *
+ * User is restricted from accessing the device before the binding operation
+ * is completed. Only allowed on cdev fds.
+ *
+ * Unbind is automatically conducted when device fd is closed.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+struct vfio_device_bind_iommufd {
+ __u32 argsz;
+ __u32 flags;
+ __s32 iommufd;
+ __u32 out_devid;
+};
+
+#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
+
+/*
+ * VFIO_DEVICE_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 19,
+ * struct vfio_device_attach_iommufd_pt)
+ * @argsz: User filled size of this data.
+ * @flags: Must be 0.
+ * @pt_id: Input the target id which can represent an ioas or a hwpt
+ * allocated via iommufd subsystem.
+ * Output the input ioas id or the attached hwpt id which could
+ * be the specified hwpt itself or a hwpt automatically created
+ * for the specified ioas by kernel during the attachment.
+ *
+ * Associate the device with an address space within the bound iommufd.
+ * Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. This is only
+ * allowed on cdev fds.
+ *
+ * If a vfio device is currently attached to a valid hw_pagetable, without doing
+ * a VFIO_DEVICE_DETACH_IOMMUFD_PT, a second VFIO_DEVICE_ATTACH_IOMMUFD_PT ioctl
+ * passing in another hw_pagetable (hwpt) id is allowed. This action, also known
+ * as a hw_pagetable replacement, will replace the device's currently attached
+ * hw_pagetable with a new hw_pagetable corresponding to the given pt_id.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+struct vfio_device_attach_iommufd_pt {
+ __u32 argsz;
+ __u32 flags;
+ __u32 pt_id;
+};
+
+#define VFIO_DEVICE_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 19)
+
+/*
+ * VFIO_DEVICE_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 20,
+ * struct vfio_device_detach_iommufd_pt)
+ * @argsz: User filled size of this data.
+ * @flags: Must be 0.
+ *
+ * Remove the association of the device and its current associated address
+ * space. After it, the device should be in a blocking DMA state. This is only
+ * allowed on cdev fds.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+struct vfio_device_detach_iommufd_pt {
+ __u32 argsz;
+ __u32 flags;
+};
+
+#define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20)
+
/*
* Provide support for setting a PCI VF Token, which is used as a shared
* secret between PF and VF drivers. This feature may only be set on a
@@ -1304,6 +1451,7 @@ struct vfio_iommu_type1_info {
#define VFIO_IOMMU_INFO_CAPS (1 << 1) /* Info supports caps */
__u64 iova_pgsizes; /* Bitmap of supported page sizes */
__u32 cap_offset; /* Offset within info struct of first cap */
+ __u32 pad;
};
/*
--
2.41.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PULL 4/7] target/s390x/kvm: Refactor AP functionalities
2023-09-12 11:41 [PULL 0/7] s390x (and one qtest) patches Thomas Huth
` (2 preceding siblings ...)
2023-09-12 11:41 ` [PULL 3/7] linux-headers: Update to Linux v6.6-rc1 Thomas Huth
@ 2023-09-12 11:41 ` Thomas Huth
2023-09-12 11:41 ` [PULL 5/7] target/s390x: AP-passthrough for PV guests Thomas Huth
` (2 subsequent siblings)
6 siblings, 0 replies; 15+ messages in thread
From: Thomas Huth @ 2023-09-12 11:41 UTC (permalink / raw)
To: qemu-devel; +Cc: Stefan Hajnoczi, Steffen Eiden, Michael Mueller
From: Steffen Eiden <seiden@linux.ibm.com>
kvm_s390_set_attr() is a misleading name as it only sets attributes for
the KVM_S390_VM_CRYPTO group. Therefore, rename it to
kvm_s390_set_crypto_attr().
Add new functions ap_available() and ap_enabled() to avoid code
duplication later.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Michael Mueller <mimu@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Message-ID: <20230823142219.1046522-5-seiden@linux.ibm.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
target/s390x/kvm/kvm.c | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index 852fbd0df7..f034b06abc 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -240,7 +240,7 @@ static void kvm_s390_enable_cmma(void)
trace_kvm_enable_cmma(rc);
}
-static void kvm_s390_set_attr(uint64_t attr)
+static void kvm_s390_set_crypto_attr(uint64_t attr)
{
struct kvm_device_attr attribute = {
.group = KVM_S390_VM_CRYPTO,
@@ -265,7 +265,7 @@ static void kvm_s390_init_aes_kw(void)
}
if (kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO, attr)) {
- kvm_s390_set_attr(attr);
+ kvm_s390_set_crypto_attr(attr);
}
}
@@ -279,7 +279,7 @@ static void kvm_s390_init_dea_kw(void)
}
if (kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO, attr)) {
- kvm_s390_set_attr(attr);
+ kvm_s390_set_crypto_attr(attr);
}
}
@@ -2288,6 +2288,17 @@ static int configure_cpu_subfunc(const S390FeatBitmap features)
return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attr);
}
+static bool ap_available(void)
+{
+ return kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
+ KVM_S390_VM_CRYPTO_ENABLE_APIE);
+}
+
+static bool ap_enabled(const S390FeatBitmap features)
+{
+ return test_bit(S390_FEAT_AP, features);
+}
+
static int kvm_to_feat[][2] = {
{ KVM_S390_VM_CPU_FEAT_ESOP, S390_FEAT_ESOP },
{ KVM_S390_VM_CPU_FEAT_SIEF2, S390_FEAT_SIE_F2 },
@@ -2467,8 +2478,7 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)
return;
}
/* for now, we can only provide the AP feature with HW support */
- if (kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO,
- KVM_S390_VM_CRYPTO_ENABLE_APIE)) {
+ if (ap_available()) {
set_bit(S390_FEAT_AP, model->features);
}
@@ -2494,7 +2504,7 @@ static void kvm_s390_configure_apie(bool interpret)
KVM_S390_VM_CRYPTO_DISABLE_APIE;
if (kvm_vm_check_attr(kvm_state, KVM_S390_VM_CRYPTO, attr)) {
- kvm_s390_set_attr(attr);
+ kvm_s390_set_crypto_attr(attr);
}
}
@@ -2548,7 +2558,7 @@ void kvm_s390_apply_cpu_model(const S390CPUModel *model, Error **errp)
kvm_s390_enable_cmma();
}
- if (test_bit(S390_FEAT_AP, model->features)) {
+ if (ap_enabled(model->features)) {
kvm_s390_configure_apie(true);
}
}
--
2.41.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PULL 5/7] target/s390x: AP-passthrough for PV guests
2023-09-12 11:41 [PULL 0/7] s390x (and one qtest) patches Thomas Huth
` (3 preceding siblings ...)
2023-09-12 11:41 ` [PULL 4/7] target/s390x/kvm: Refactor AP functionalities Thomas Huth
@ 2023-09-12 11:41 ` Thomas Huth
2023-09-12 11:41 ` [PULL 6/7] kconfig: Add NVME to s390x machines Thomas Huth
2023-09-12 11:41 ` [PULL 7/7] tests/qtest/pflash: Clean up local variable shadowing Thomas Huth
6 siblings, 0 replies; 15+ messages in thread
From: Thomas Huth @ 2023-09-12 11:41 UTC (permalink / raw)
To: qemu-devel; +Cc: Stefan Hajnoczi, Steffen Eiden, Michael Mueller
From: Steffen Eiden <seiden@linux.ibm.com>
Enabling AP-passthrough(AP-pt) for PV-guest by using the new CPU
features for PV-AP-pt of KVM.
As usual QEMU first checks which CPU features are available and then
sets them if available and selected by user. An additional check is done
to verify that PV-AP can only be enabled if "regular" AP-pt is enabled
as well. Note that KVM itself does not enforce this restriction.
Reviewed-by: Michael Mueller <mimu@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Message-ID: <20230823142219.1046522-6-seiden@linux.ibm.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
target/s390x/cpu_features.h | 1 +
target/s390x/cpu_features_def.h.inc | 4 ++
target/s390x/cpu_models.c | 2 +
target/s390x/gen-features.c | 2 +
target/s390x/kvm/kvm.c | 70 +++++++++++++++++++++++++++++
5 files changed, 79 insertions(+)
diff --git a/target/s390x/cpu_features.h b/target/s390x/cpu_features.h
index 87463f064d..a9bd68a2e1 100644
--- a/target/s390x/cpu_features.h
+++ b/target/s390x/cpu_features.h
@@ -43,6 +43,7 @@ typedef enum {
S390_FEAT_TYPE_KDSA,
S390_FEAT_TYPE_SORTL,
S390_FEAT_TYPE_DFLTCC,
+ S390_FEAT_TYPE_UV_FEAT_GUEST,
} S390FeatType;
/* Definition of a CPU feature */
diff --git a/target/s390x/cpu_features_def.h.inc b/target/s390x/cpu_features_def.h.inc
index e3cfe63735..e68da9b8ff 100644
--- a/target/s390x/cpu_features_def.h.inc
+++ b/target/s390x/cpu_features_def.h.inc
@@ -379,3 +379,7 @@ DEF_FEAT(DEFLATE_GHDT, "dfltcc-gdht", DFLTCC, 1, "DFLTCC GDHT")
DEF_FEAT(DEFLATE_CMPR, "dfltcc-cmpr", DFLTCC, 2, "DFLTCC CMPR")
DEF_FEAT(DEFLATE_XPND, "dfltcc-xpnd", DFLTCC, 4, "DFLTCC XPND")
DEF_FEAT(DEFLATE_F0, "dfltcc-f0", DFLTCC, 192, "DFLTCC format 0 parameter-block")
+
+/* Features exposed via the UV-CALL instruction */
+DEF_FEAT(UV_FEAT_AP, "appv", UV_FEAT_GUEST, 4, "AP instructions installed for secure guests")
+DEF_FEAT(UV_FEAT_AP_INTR, "appvi", UV_FEAT_GUEST, 5, "AP instructions interruption support for secure guests")
diff --git a/target/s390x/cpu_models.c b/target/s390x/cpu_models.c
index 91ce896491..98f14c09c2 100644
--- a/target/s390x/cpu_models.c
+++ b/target/s390x/cpu_models.c
@@ -483,6 +483,8 @@ static void check_consistency(const S390CPUModel *model)
{ S390_FEAT_DIAG_318, S390_FEAT_EXTENDED_LENGTH_SCCB },
{ S390_FEAT_NNPA, S390_FEAT_VECTOR },
{ S390_FEAT_RDP, S390_FEAT_LOCAL_TLB_CLEARING },
+ { S390_FEAT_UV_FEAT_AP, S390_FEAT_AP },
+ { S390_FEAT_UV_FEAT_AP_INTR, S390_FEAT_UV_FEAT_AP },
};
int i;
diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c
index 1e3b7c0dc9..2b2bfc3736 100644
--- a/target/s390x/gen-features.c
+++ b/target/s390x/gen-features.c
@@ -576,6 +576,8 @@ static uint16_t full_GEN16_GA1[] = {
S390_FEAT_RDP,
S390_FEAT_PAI,
S390_FEAT_PAIE,
+ S390_FEAT_UV_FEAT_AP,
+ S390_FEAT_UV_FEAT_AP_INTR,
};
diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c
index f034b06abc..bc5c56a305 100644
--- a/target/s390x/kvm/kvm.c
+++ b/target/s390x/kvm/kvm.c
@@ -2299,6 +2299,42 @@ static bool ap_enabled(const S390FeatBitmap features)
return test_bit(S390_FEAT_AP, features);
}
+static bool uv_feat_supported(void)
+{
+ return kvm_vm_check_attr(kvm_state, KVM_S390_VM_CPU_MODEL,
+ KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST);
+}
+
+static int query_uv_feat_guest(S390FeatBitmap features)
+{
+ struct kvm_s390_vm_cpu_uv_feat prop = {};
+ struct kvm_device_attr attr = {
+ .group = KVM_S390_VM_CPU_MODEL,
+ .attr = KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST,
+ .addr = (uint64_t) &prop,
+ };
+ int rc;
+
+ /* AP support check is currently the only user of the UV feature test */
+ if (!(uv_feat_supported() && ap_available())) {
+ return 0;
+ }
+
+ rc = kvm_vm_ioctl(kvm_state, KVM_GET_DEVICE_ATTR, &attr);
+ if (rc) {
+ return rc;
+ }
+
+ if (prop.ap) {
+ set_bit(S390_FEAT_UV_FEAT_AP, features);
+ }
+ if (prop.ap_intr) {
+ set_bit(S390_FEAT_UV_FEAT_AP_INTR, features);
+ }
+
+ return 0;
+}
+
static int kvm_to_feat[][2] = {
{ KVM_S390_VM_CPU_FEAT_ESOP, S390_FEAT_ESOP },
{ KVM_S390_VM_CPU_FEAT_SIEF2, S390_FEAT_SIE_F2 },
@@ -2493,11 +2529,38 @@ void kvm_s390_get_host_cpu_model(S390CPUModel *model, Error **errp)
set_bit(S390_FEAT_DIAG_318, model->features);
}
+ /* Test for Ultravisor features that influence secure guest behavior */
+ query_uv_feat_guest(model->features);
+
/* strip of features that are not part of the maximum model */
bitmap_and(model->features, model->features, model->def->full_feat,
S390_FEAT_MAX);
}
+static int configure_uv_feat_guest(const S390FeatBitmap features)
+{
+ struct kvm_s390_vm_cpu_uv_feat uv_feat = {};
+ struct kvm_device_attr attribute = {
+ .group = KVM_S390_VM_CPU_MODEL,
+ .attr = KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST,
+ .addr = (__u64) &uv_feat,
+ };
+
+ /* AP support check is currently the only user of the UV feature test */
+ if (!(uv_feat_supported() && ap_enabled(features))) {
+ return 0;
+ }
+
+ if (test_bit(S390_FEAT_UV_FEAT_AP, features)) {
+ uv_feat.ap = 1;
+ }
+ if (test_bit(S390_FEAT_UV_FEAT_AP_INTR, features)) {
+ uv_feat.ap_intr = 1;
+ }
+
+ return kvm_vm_ioctl(kvm_state, KVM_SET_DEVICE_ATTR, &attribute);
+}
+
static void kvm_s390_configure_apie(bool interpret)
{
uint64_t attr = interpret ? KVM_S390_VM_CRYPTO_ENABLE_APIE :
@@ -2561,6 +2624,13 @@ void kvm_s390_apply_cpu_model(const S390CPUModel *model, Error **errp)
if (ap_enabled(model->features)) {
kvm_s390_configure_apie(true);
}
+
+ /* configure UV-features for the guest indicated via query / test_bit */
+ rc = configure_uv_feat_guest(model->features);
+ if (rc) {
+ error_setg(errp, "KVM: Error configuring CPU UV features %d", rc);
+ return;
+ }
}
void kvm_s390_restart_interrupt(S390CPU *cpu)
--
2.41.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PULL 6/7] kconfig: Add NVME to s390x machines
2023-09-12 11:41 [PULL 0/7] s390x (and one qtest) patches Thomas Huth
` (4 preceding siblings ...)
2023-09-12 11:41 ` [PULL 5/7] target/s390x: AP-passthrough for PV guests Thomas Huth
@ 2023-09-12 11:41 ` Thomas Huth
2023-09-12 11:41 ` [PULL 7/7] tests/qtest/pflash: Clean up local variable shadowing Thomas Huth
6 siblings, 0 replies; 15+ messages in thread
From: Thomas Huth @ 2023-09-12 11:41 UTC (permalink / raw)
To: qemu-devel; +Cc: Stefan Hajnoczi, Cédric Le Goater, Klaus Jensen
From: Cédric Le Goater <clg@redhat.com>
We recently had issues with nvme devices on big endian platforms.
Include their compilation on s390x to ease tests.
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Message-ID: <20230828150148.120031-1-clg@kaod.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Klaus Jensen <k.jensen@samsung.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
hw/nvme/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hw/nvme/Kconfig b/hw/nvme/Kconfig
index 8ac90942e5..cfa2ab0f9d 100644
--- a/hw/nvme/Kconfig
+++ b/hw/nvme/Kconfig
@@ -1,4 +1,4 @@
config NVME_PCI
bool
- default y if PCI_DEVICES
+ default y if PCI_DEVICES || PCIE_DEVICES
depends on PCI
--
2.41.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PULL 7/7] tests/qtest/pflash: Clean up local variable shadowing
2023-09-12 11:41 [PULL 0/7] s390x (and one qtest) patches Thomas Huth
` (5 preceding siblings ...)
2023-09-12 11:41 ` [PULL 6/7] kconfig: Add NVME to s390x machines Thomas Huth
@ 2023-09-12 11:41 ` Thomas Huth
6 siblings, 0 replies; 15+ messages in thread
From: Thomas Huth @ 2023-09-12 11:41 UTC (permalink / raw)
To: qemu-devel; +Cc: Stefan Hajnoczi, Philippe Mathieu-Daudé, Peter Maydell
From: Philippe Mathieu-Daudé <philmd@linaro.org>
Fix:
tests/qtest/pflash-cfi02-test.c: In function ‘test_geometry’:
tests/qtest/pflash-cfi02-test.c:409:22: warning: declaration of ‘byte_addr’ shadows a previous local [-Wshadow=compatible-local]
409 | uint64_t byte_addr = (uint64_t)i * c->sector_len[region];
| ^~~~~~~~~
tests/qtest/pflash-cfi02-test.c:342:14: note: shadowed declaration is here
342 | uint64_t byte_addr = 0;
| ^~~~~~~~~
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20230904162824.85385-4-philmd@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
tests/qtest/pflash-cfi02-test.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qtest/pflash-cfi02-test.c b/tests/qtest/pflash-cfi02-test.c
index 0b52c2ca5c..8c073efcb4 100644
--- a/tests/qtest/pflash-cfi02-test.c
+++ b/tests/qtest/pflash-cfi02-test.c
@@ -406,7 +406,7 @@ static void test_geometry(const void *opaque)
for (int region = 0; region < nb_erase_regions; ++region) {
for (uint32_t i = 0; i < c->nb_blocs[region]; ++i) {
- uint64_t byte_addr = (uint64_t)i * c->sector_len[region];
+ byte_addr = (uint64_t)i * c->sector_len[region];
g_assert_cmphex(flash_read(c, byte_addr), ==, bank_mask(c));
}
}
--
2.41.0
^ permalink raw reply related [flat|nested] 15+ messages in thread