[PATCH v4 0/4] KVM: s390: Enable AP instructions for PV-guests

[PATCH v4 0/4] KVM: s390: Enable AP instructions for PV-guests

[Steffen Eiden]

This series enables general KVM support for AP-passthrough for Secure
Execution guests (PV-guests).

To enable AP inside PV-guests two things have to be done/considered:
	1) set corresponding flags in the Create Secure Configuration UVC if
     firmware supports AP for PV-guests (patch 4).
	2) enable/disable AP in PV-guests if the VMM wants this (patch 3).

since v3:
  - add a patch from Viktor that handles a new rc that can occur with AP-pt.
  - remove KVM_S390_VM_CPU_UV_FEAT_GUEST_DEFAULT define (Janosch)
  - add a boundary check in uv_has_feature() (Janosch)
  - add r-b from Janosch

since v2:
  - applied styling recommendations from Heiko

since v1:
  - PATCH 1: r-b from Claudio
  - PATCH 2: fixed formatting issues (Claudio)
  - PATCH 3: removed unnecessary checks (Claudio)

Steffen

Steffen Eiden (3):
  s390: uv: UV feature check utility
  KVM: s390: Add UV feature negotiation
  KVM: s390: pv:  Allow AP-instructions for pv-guests

Viktor Mihajlovski (1):
  KVM: s390: pv: relax WARN_ONCE condition for destroy fast

 arch/s390/include/asm/kvm_host.h |  2 +
 arch/s390/include/asm/uv.h       | 19 +++++++-
 arch/s390/include/uapi/asm/kvm.h | 16 +++++++
 arch/s390/kernel/uv.c            |  2 +-
 arch/s390/kvm/kvm-s390.c         | 75 +++++++++++++++++++++++++++++++-
 arch/s390/kvm/pv.c               |  9 ++--
 arch/s390/mm/fault.c             |  2 +-
 7 files changed, 118 insertions(+), 7 deletions(-)

-- 
2.41.0

[PATCH v4 1/4] KVM: s390: pv: relax WARN_ONCE condition for destroy fast

[Steffen Eiden]

From: Viktor Mihajlovski <mihajlov@linux.ibm.com>

Destroy configuration fast may return with RC 0x104 if there
are still bound APQNs in the configuration. The final cleanup
will occur with the standard destroy configuration UVC as
at this point in time all APQNs have been reset and thus
unbound. Therefore, don't warn if RC 0x104 is reported.

Signed-off-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
Reviewed-by: Steffen Eiden <seiden@linux.ibm.com>
---
 arch/s390/kvm/pv.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
index 8d3f39a8a11e..8570ee324607 100644
--- a/arch/s390/kvm/pv.c
+++ b/arch/s390/kvm/pv.c
@@ -285,7 +285,8 @@ static int kvm_s390_pv_deinit_vm_fast(struct kvm *kvm, u16 *rc, u16 *rrc)
 	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
 	KVM_UV_EVENT(kvm, 3, "PROTVIRT DESTROY VM FAST: rc %x rrc %x",
 		     uvcb.header.rc, uvcb.header.rrc);
-	WARN_ONCE(cc, "protvirt destroy vm fast failed handle %llx rc %x rrc %x",
+	WARN_ONCE(cc && uvcb.header.rc != 0x104,
+		  "protvirt destroy vm fast failed handle %llx rc %x rrc %x",
 		  kvm_s390_pv_get_handle(kvm), uvcb.header.rc, uvcb.header.rrc);
 	/* Intended memory leak on "impossible" error */
 	if (!cc)
-- 
2.41.0

[PATCH v4 2/4] s390: uv: UV feature check utility

[Steffen Eiden]

Introduces a function to check the existence of an UV feature.
Refactor feature bit checks to use the new function.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/uv.h | 7 +++++++
 arch/s390/kernel/uv.c      | 2 +-
 arch/s390/kvm/kvm-s390.c   | 2 +-
 arch/s390/mm/fault.c       | 2 +-
 4 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index d2cd42bb2c26..823adfff7315 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -397,6 +397,13 @@ struct uv_info {
 
 extern struct uv_info uv_info;
 
+static inline bool uv_has_feature(u8 feature_bit)
+{
+	if (feature_bit >= sizeof(uv_info.uv_feature_indications) * 8)
+		return false;
+	return test_bit_inv(feature_bit, &uv_info.uv_feature_indications);
+}
+
 #ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
 extern int prot_virt_guest;
 
diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
index b771f1b4cdd1..fc07bc39e698 100644
--- a/arch/s390/kernel/uv.c
+++ b/arch/s390/kernel/uv.c
@@ -258,7 +258,7 @@ static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_str
 	 * shared page from a different protected VM will automatically also
 	 * transfer its ownership.
 	 */
-	if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications))
+	if (uv_has_feature(BIT_UV_FEAT_MISC))
 		return false;
 	if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
 		return false;
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index e6511608280c..813cc3d59c90 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -2406,7 +2406,7 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)
 	struct kvm_vcpu *vcpu;
 
 	/* Disable the GISA if the ultravisor does not support AIV. */
-	if (!test_bit_inv(BIT_UV_FEAT_AIV, &uv_info.uv_feature_indications))
+	if (!uv_has_feature(BIT_UV_FEAT_AIV))
 		kvm_s390_gisa_disable(kvm);
 
 	kvm_for_each_vcpu(i, vcpu, kvm) {
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index b5e1bea9194c..8a86dd725870 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -599,7 +599,7 @@ void do_secure_storage_access(struct pt_regs *regs)
 	 * reliable without the misc UV feature so we need to check
 	 * for that as well.
 	 */
-	if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications) &&
+	if (uv_has_feature(BIT_UV_FEAT_MISC) &&
 	    !test_bit_inv(61, &regs->int_parm_long)) {
 		/*
 		 * When this happens, userspace did something that it
-- 
2.41.0

[PATCH v4 3/4] KVM: s390: Add UV feature negotiation

[Steffen Eiden]

Add a uv_feature list for pv-guests to the KVM cpu-model.
The feature bits 'AP-interpretation for secure guests' and
'AP-interrupt for secure guests' are available.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/kvm_host.h |  2 +
 arch/s390/include/uapi/asm/kvm.h | 16 +++++++
 arch/s390/kvm/kvm-s390.c         | 73 ++++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 91bfecb91321..427f9528a7b6 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -817,6 +817,8 @@ struct kvm_s390_cpu_model {
 	__u64 *fac_list;
 	u64 cpuid;
 	unsigned short ibc;
+	/* subset of available UV-features for pv-guests enabled by user space */
+	struct kvm_s390_vm_cpu_uv_feat uv_feat_guest;
 };
 
 typedef int (*crypto_hook)(struct kvm_vcpu *vcpu);
diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
index a73cf01a1606..abe926d43cbe 100644
--- a/arch/s390/include/uapi/asm/kvm.h
+++ b/arch/s390/include/uapi/asm/kvm.h
@@ -159,6 +159,22 @@ struct kvm_s390_vm_cpu_subfunc {
 	__u8 reserved[1728];
 };
 
+#define KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST	6
+#define KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST	7
+
+#define KVM_S390_VM_CPU_UV_FEAT_NR_BITS	64
+struct kvm_s390_vm_cpu_uv_feat {
+	union {
+		struct {
+			__u64 : 4;
+			__u64 ap : 1;		/* bit 4 */
+			__u64 ap_intr : 1;	/* bit 5 */
+			__u64 : 58;
+		};
+		__u64 feat;
+	};
+};
+
 /* kvm attributes for crypto */
 #define KVM_S390_VM_CRYPTO_ENABLE_AES_KW	0
 #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW	1
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 813cc3d59c90..b3f17e014cab 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -1531,6 +1531,39 @@ static int kvm_s390_set_processor_subfunc(struct kvm *kvm,
 	return 0;
 }
 
+#define KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK	\
+(						\
+	((struct kvm_s390_vm_cpu_uv_feat){	\
+		.ap = 1,			\
+		.ap_intr = 1,			\
+	})					\
+	.feat					\
+)
+
+static int kvm_s390_set_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr)
+{
+	struct kvm_s390_vm_cpu_uv_feat __user *ptr = (void __user *)attr->addr;
+	unsigned long data, filter;
+
+	filter = uv_info.uv_feature_indications & KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK;
+	if (get_user(data, &ptr->feat))
+		return -EFAULT;
+	if (!bitmap_subset(&data, &filter, KVM_S390_VM_CPU_UV_FEAT_NR_BITS))
+		return -EINVAL;
+
+	mutex_lock(&kvm->lock);
+	if (kvm->created_vcpus) {
+		mutex_unlock(&kvm->lock);
+		return -EBUSY;
+	}
+	kvm->arch.model.uv_feat_guest.feat = data;
+	mutex_unlock(&kvm->lock);
+
+	VM_EVENT(kvm, 3, "SET: guest UV-feat: 0x%16.16lx", data);
+
+	return 0;
+}
+
 static int kvm_s390_set_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr)
 {
 	int ret = -ENXIO;
@@ -1545,6 +1578,9 @@ static int kvm_s390_set_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr)
 	case KVM_S390_VM_CPU_PROCESSOR_SUBFUNC:
 		ret = kvm_s390_set_processor_subfunc(kvm, attr);
 		break;
+	case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST:
+		ret = kvm_s390_set_uv_feat(kvm, attr);
+		break;
 	}
 	return ret;
 }
@@ -1777,6 +1813,33 @@ static int kvm_s390_get_machine_subfunc(struct kvm *kvm,
 	return 0;
 }
 
+static int kvm_s390_get_processor_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr)
+{
+	struct kvm_s390_vm_cpu_uv_feat __user *dst = (void __user *)attr->addr;
+	unsigned long feat = kvm->arch.model.uv_feat_guest.feat;
+
+	if (put_user(feat, &dst->feat))
+		return -EFAULT;
+	VM_EVENT(kvm, 3, "GET: guest UV-feat: 0x%16.16lx", feat);
+
+	return 0;
+}
+
+static int kvm_s390_get_machine_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr)
+{
+	struct kvm_s390_vm_cpu_uv_feat __user *dst = (void __user *)attr->addr;
+	unsigned long feat;
+
+	BUILD_BUG_ON(sizeof(*dst) != sizeof(uv_info.uv_feature_indications));
+
+	feat = uv_info.uv_feature_indications & KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK;
+	if (put_user(feat, &dst->feat))
+		return -EFAULT;
+	VM_EVENT(kvm, 3, "GET: guest UV-feat: 0x%16.16lx", feat);
+
+	return 0;
+}
+
 static int kvm_s390_get_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr)
 {
 	int ret = -ENXIO;
@@ -1800,6 +1863,12 @@ static int kvm_s390_get_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr)
 	case KVM_S390_VM_CPU_MACHINE_SUBFUNC:
 		ret = kvm_s390_get_machine_subfunc(kvm, attr);
 		break;
+	case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST:
+		ret = kvm_s390_get_processor_uv_feat(kvm, attr);
+		break;
+	case KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST:
+		ret = kvm_s390_get_machine_uv_feat(kvm, attr);
+		break;
 	}
 	return ret;
 }
@@ -1952,6 +2021,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
 		case KVM_S390_VM_CPU_MACHINE_FEAT:
 		case KVM_S390_VM_CPU_MACHINE_SUBFUNC:
 		case KVM_S390_VM_CPU_PROCESSOR_SUBFUNC:
+		case KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST:
+		case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST:
 			ret = 0;
 			break;
 		default:
@@ -3296,6 +3367,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.model.cpuid = kvm_s390_get_initial_cpuid();
 	kvm->arch.model.ibc = sclp.ibc & 0x0fff;
 
+	kvm->arch.model.uv_feat_guest.feat = 0;
+
 	kvm_s390_crypto_init(kvm);
 
 	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) {
-- 
2.41.0

[PATCH v4 4/4] KVM: s390: pv: Allow AP-instructions for pv-guests

[Steffen Eiden]

Introduces new feature bits and enablement flags for AP and AP IRQ
support.

Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
---
 arch/s390/include/asm/uv.h | 12 +++++++++++-
 arch/s390/kvm/pv.c         |  6 ++++--
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
index 823adfff7315..0e7bd3873907 100644
--- a/arch/s390/include/asm/uv.h
+++ b/arch/s390/include/asm/uv.h
@@ -99,6 +99,8 @@ enum uv_cmds_inst {
 enum uv_feat_ind {
 	BIT_UV_FEAT_MISC = 0,
 	BIT_UV_FEAT_AIV = 1,
+	BIT_UV_FEAT_AP = 4,
+	BIT_UV_FEAT_AP_INTR = 5,
 };
 
 struct uv_cb_header {
@@ -159,7 +161,15 @@ struct uv_cb_cgc {
 	u64 guest_handle;
 	u64 conf_base_stor_origin;
 	u64 conf_virt_stor_origin;
-	u64 reserved30;
+	u8  reserved30[6];
+	union {
+		struct {
+			u16 : 14;
+			u16 ap_instr_intr : 1;
+			u16 ap_allow_instr : 1;
+		};
+		u16 raw;
+	} flags;
 	u64 guest_stor_origin;
 	u64 guest_stor_len;
 	u64 guest_sca;
diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
index 8570ee324607..75e81ba26d04 100644
--- a/arch/s390/kvm/pv.c
+++ b/arch/s390/kvm/pv.c
@@ -576,12 +576,14 @@ int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 	uvcb.conf_base_stor_origin =
 		virt_to_phys((void *)kvm->arch.pv.stor_base);
 	uvcb.conf_virt_stor_origin = (u64)kvm->arch.pv.stor_var;
+	uvcb.flags.ap_allow_instr = kvm->arch.model.uv_feat_guest.ap;
+	uvcb.flags.ap_instr_intr = kvm->arch.model.uv_feat_guest.ap_intr;
 
 	cc = uv_call_sched(0, (u64)&uvcb);
 	*rc = uvcb.header.rc;
 	*rrc = uvcb.header.rrc;
-	KVM_UV_EVENT(kvm, 3, "PROTVIRT CREATE VM: handle %llx len %llx rc %x rrc %x",
-		     uvcb.guest_handle, uvcb.guest_stor_len, *rc, *rrc);
+	KVM_UV_EVENT(kvm, 3, "PROTVIRT CREATE VM: handle %llx len %llx rc %x rrc %x flags %04x",
+		     uvcb.guest_handle, uvcb.guest_stor_len, *rc, *rrc, uvcb.flags.raw);
 
 	/* Outputs */
 	kvm->arch.pv.handle = uvcb.guest_handle;
-- 
2.41.0

Re: [PATCH v4 1/4] KVM: s390: pv: relax WARN_ONCE condition for destroy fast

[Michael Mueller]

On 15.08.23 17:14, Steffen Eiden wrote:
> From: Viktor Mihajlovski <mihajlov@linux.ibm.com>
> 
> Destroy configuration fast may return with RC 0x104 if there
> are still bound APQNs in the configuration. The final cleanup
> will occur with the standard destroy configuration UVC as
> at this point in time all APQNs have been reset and thus
> unbound. Therefore, don't warn if RC 0x104 is reported.
> 
> Signed-off-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
> Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> Reviewed-by: Janosch Frank <frankja@linux.ibm.com>
> Reviewed-by: Steffen Eiden <seiden@linux.ibm.com>


Reviewed-by: Michael Mueller <mimu@linux.ibm.com>

> ---
>   arch/s390/kvm/pv.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
> index 8d3f39a8a11e..8570ee324607 100644
> --- a/arch/s390/kvm/pv.c
> +++ b/arch/s390/kvm/pv.c
> @@ -285,7 +285,8 @@ static int kvm_s390_pv_deinit_vm_fast(struct kvm *kvm, u16 *rc, u16 *rrc)
>   	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
>   	KVM_UV_EVENT(kvm, 3, "PROTVIRT DESTROY VM FAST: rc %x rrc %x",
>   		     uvcb.header.rc, uvcb.header.rrc);
> -	WARN_ONCE(cc, "protvirt destroy vm fast failed handle %llx rc %x rrc %x",
> +	WARN_ONCE(cc && uvcb.header.rc != 0x104,
> +		  "protvirt destroy vm fast failed handle %llx rc %x rrc %x",
>   		  kvm_s390_pv_get_handle(kvm), uvcb.header.rc, uvcb.header.rrc);
>   	/* Intended memory leak on "impossible" error */
>   	if (!cc)

Re: [PATCH v4 2/4] s390: uv: UV feature check utility

[Michael Mueller]

On 15.08.23 17:14, Steffen Eiden wrote:
> Introduces a function to check the existence of an UV feature.
> Refactor feature bit checks to use the new function.
> 
> Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
> Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> Reviewed-by: Janosch Frank <frankja@linux.ibm.com>

Reviewed-by: Michael Mueller <mimu@linux.ibm.com>

> ---
>   arch/s390/include/asm/uv.h | 7 +++++++
>   arch/s390/kernel/uv.c      | 2 +-
>   arch/s390/kvm/kvm-s390.c   | 2 +-
>   arch/s390/mm/fault.c       | 2 +-
>   4 files changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
> index d2cd42bb2c26..823adfff7315 100644
> --- a/arch/s390/include/asm/uv.h
> +++ b/arch/s390/include/asm/uv.h
> @@ -397,6 +397,13 @@ struct uv_info {
> 
>   extern struct uv_info uv_info;
> 
> +static inline bool uv_has_feature(u8 feature_bit)
> +{
> +	if (feature_bit >= sizeof(uv_info.uv_feature_indications) * 8)
> +		return false;
> +	return test_bit_inv(feature_bit, &uv_info.uv_feature_indications);
> +}
> +
>   #ifdef CONFIG_PROTECTED_VIRTUALIZATION_GUEST
>   extern int prot_virt_guest;
> 
> diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c
> index b771f1b4cdd1..fc07bc39e698 100644
> --- a/arch/s390/kernel/uv.c
> +++ b/arch/s390/kernel/uv.c
> @@ -258,7 +258,7 @@ static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_str
>   	 * shared page from a different protected VM will automatically also
>   	 * transfer its ownership.
>   	 */
> -	if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications))
> +	if (uv_has_feature(BIT_UV_FEAT_MISC))
>   		return false;
>   	if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
>   		return false;
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index e6511608280c..813cc3d59c90 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -2406,7 +2406,7 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)
>   	struct kvm_vcpu *vcpu;
> 
>   	/* Disable the GISA if the ultravisor does not support AIV. */
> -	if (!test_bit_inv(BIT_UV_FEAT_AIV, &uv_info.uv_feature_indications))
> +	if (!uv_has_feature(BIT_UV_FEAT_AIV))
>   		kvm_s390_gisa_disable(kvm);
> 
>   	kvm_for_each_vcpu(i, vcpu, kvm) {
> diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
> index b5e1bea9194c..8a86dd725870 100644
> --- a/arch/s390/mm/fault.c
> +++ b/arch/s390/mm/fault.c
> @@ -599,7 +599,7 @@ void do_secure_storage_access(struct pt_regs *regs)
>   	 * reliable without the misc UV feature so we need to check
>   	 * for that as well.
>   	 */
> -	if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications) &&
> +	if (uv_has_feature(BIT_UV_FEAT_MISC) &&
>   	    !test_bit_inv(61, &regs->int_parm_long)) {
>   		/*
>   		 * When this happens, userspace did something that it

Re: [PATCH v4 3/4] KVM: s390: Add UV feature negotiation

[Michael Mueller]

On 15.08.23 17:14, Steffen Eiden wrote:
> Add a uv_feature list for pv-guests to the KVM cpu-model.
> The feature bits 'AP-interpretation for secure guests' and
> 'AP-interrupt for secure guests' are available.
> 
> Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
> Reviewed-by: Janosch Frank <frankja@linux.ibm.com>

Reviewed-by: Michael Mueller <mimu@linux.ibm.com>

but a minor comment, see below.

> ---
>   arch/s390/include/asm/kvm_host.h |  2 +
>   arch/s390/include/uapi/asm/kvm.h | 16 +++++++
>   arch/s390/kvm/kvm-s390.c         | 73 ++++++++++++++++++++++++++++++++
>   3 files changed, 91 insertions(+)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 91bfecb91321..427f9528a7b6 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -817,6 +817,8 @@ struct kvm_s390_cpu_model {
>   	__u64 *fac_list;
>   	u64 cpuid;
>   	unsigned short ibc;
> +	/* subset of available UV-features for pv-guests enabled by user space */
> +	struct kvm_s390_vm_cpu_uv_feat uv_feat_guest;
>   };
> 
>   typedef int (*crypto_hook)(struct kvm_vcpu *vcpu);
> diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
> index a73cf01a1606..abe926d43cbe 100644
> --- a/arch/s390/include/uapi/asm/kvm.h
> +++ b/arch/s390/include/uapi/asm/kvm.h
> @@ -159,6 +159,22 @@ struct kvm_s390_vm_cpu_subfunc {
>   	__u8 reserved[1728];
>   };
> 
> +#define KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST	6
> +#define KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST	7
> +
> +#define KVM_S390_VM_CPU_UV_FEAT_NR_BITS	64
> +struct kvm_s390_vm_cpu_uv_feat {
> +	union {
> +		struct {
> +			__u64 : 4;
> +			__u64 ap : 1;		/* bit 4 */
> +			__u64 ap_intr : 1;	/* bit 5 */
> +			__u64 : 58;
> +		};
> +		__u64 feat;
> +	};
> +};
> +
>   /* kvm attributes for crypto */
>   #define KVM_S390_VM_CRYPTO_ENABLE_AES_KW	0
>   #define KVM_S390_VM_CRYPTO_ENABLE_DEA_KW	1
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 813cc3d59c90..b3f17e014cab 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -1531,6 +1531,39 @@ static int kvm_s390_set_processor_subfunc(struct kvm *kvm,
>   	return 0;
>   }
> 
> +#define KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK	\
> +(						\
> +	((struct kvm_s390_vm_cpu_uv_feat){	\
> +		.ap = 1,			\
> +		.ap_intr = 1,			\
> +	})					\
> +	.feat					\
> +)
> +
> +static int kvm_s390_set_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr)
> +{
> +	struct kvm_s390_vm_cpu_uv_feat __user *ptr = (void __user *)attr->addr;
> +	unsigned long data, filter;
> +
> +	filter = uv_info.uv_feature_indications & KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK;

if get_user() fails the uv feature bits in "filter" are
calculated in vain

> +	if (get_user(data, &ptr->feat))
> +		return -EFAULT;

could be done here, but that's really minor to me

> +	if (!bitmap_subset(&data, &filter, KVM_S390_VM_CPU_UV_FEAT_NR_BITS))
> +		return -EINVAL;
> +
> +	mutex_lock(&kvm->lock);
> +	if (kvm->created_vcpus) {
> +		mutex_unlock(&kvm->lock);
> +		return -EBUSY;
> +	}
> +	kvm->arch.model.uv_feat_guest.feat = data;
> +	mutex_unlock(&kvm->lock);
> +
> +	VM_EVENT(kvm, 3, "SET: guest UV-feat: 0x%16.16lx", data);
> +
> +	return 0;
> +}
> +
>   static int kvm_s390_set_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr)
>   {
>   	int ret = -ENXIO;
> @@ -1545,6 +1578,9 @@ static int kvm_s390_set_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr)
>   	case KVM_S390_VM_CPU_PROCESSOR_SUBFUNC:
>   		ret = kvm_s390_set_processor_subfunc(kvm, attr);
>   		break;
> +	case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST:
> +		ret = kvm_s390_set_uv_feat(kvm, attr);
> +		break;
>   	}
>   	return ret;
>   }
> @@ -1777,6 +1813,33 @@ static int kvm_s390_get_machine_subfunc(struct kvm *kvm,
>   	return 0;
>   }
> 
> +static int kvm_s390_get_processor_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr)
> +{
> +	struct kvm_s390_vm_cpu_uv_feat __user *dst = (void __user *)attr->addr;
> +	unsigned long feat = kvm->arch.model.uv_feat_guest.feat;
> +
> +	if (put_user(feat, &dst->feat))
> +		return -EFAULT;
> +	VM_EVENT(kvm, 3, "GET: guest UV-feat: 0x%16.16lx", feat);
> +
> +	return 0;
> +}
> +
> +static int kvm_s390_get_machine_uv_feat(struct kvm *kvm, struct kvm_device_attr *attr)
> +{
> +	struct kvm_s390_vm_cpu_uv_feat __user *dst = (void __user *)attr->addr;
> +	unsigned long feat;
> +
> +	BUILD_BUG_ON(sizeof(*dst) != sizeof(uv_info.uv_feature_indications));
> +
> +	feat = uv_info.uv_feature_indications & KVM_S390_VM_CPU_UV_FEAT_GUEST_MASK;
> +	if (put_user(feat, &dst->feat))
> +		return -EFAULT;
> +	VM_EVENT(kvm, 3, "GET: guest UV-feat: 0x%16.16lx", feat);
> +
> +	return 0;
> +}
> +
>   static int kvm_s390_get_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr)
>   {
>   	int ret = -ENXIO;
> @@ -1800,6 +1863,12 @@ static int kvm_s390_get_cpu_model(struct kvm *kvm, struct kvm_device_attr *attr)
>   	case KVM_S390_VM_CPU_MACHINE_SUBFUNC:
>   		ret = kvm_s390_get_machine_subfunc(kvm, attr);
>   		break;
> +	case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST:
> +		ret = kvm_s390_get_processor_uv_feat(kvm, attr);
> +		break;
> +	case KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST:
> +		ret = kvm_s390_get_machine_uv_feat(kvm, attr);
> +		break;
>   	}
>   	return ret;
>   }
> @@ -1952,6 +2021,8 @@ static int kvm_s390_vm_has_attr(struct kvm *kvm, struct kvm_device_attr *attr)
>   		case KVM_S390_VM_CPU_MACHINE_FEAT:
>   		case KVM_S390_VM_CPU_MACHINE_SUBFUNC:
>   		case KVM_S390_VM_CPU_PROCESSOR_SUBFUNC:
> +		case KVM_S390_VM_CPU_MACHINE_UV_FEAT_GUEST:
> +		case KVM_S390_VM_CPU_PROCESSOR_UV_FEAT_GUEST:
>   			ret = 0;
>   			break;
>   		default:
> @@ -3296,6 +3367,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
>   	kvm->arch.model.cpuid = kvm_s390_get_initial_cpuid();
>   	kvm->arch.model.ibc = sclp.ibc & 0x0fff;
> 
> +	kvm->arch.model.uv_feat_guest.feat = 0;
> +
>   	kvm_s390_crypto_init(kvm);
> 
>   	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) {

Re: [PATCH v4 4/4] KVM: s390: pv: Allow AP-instructions for pv-guests

[Michael Mueller]

On 15.08.23 17:14, Steffen Eiden wrote:
> Introduces new feature bits and enablement flags for AP and AP IRQ
> support.
> 
> Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
> Reviewed-by: Janosch Frank <frankja@linux.ibm.com>

Reviewed-by: Michael Mueller <mimu@linux.ibm.com>

> ---
>   arch/s390/include/asm/uv.h | 12 +++++++++++-
>   arch/s390/kvm/pv.c         |  6 ++++--
>   2 files changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/s390/include/asm/uv.h b/arch/s390/include/asm/uv.h
> index 823adfff7315..0e7bd3873907 100644
> --- a/arch/s390/include/asm/uv.h
> +++ b/arch/s390/include/asm/uv.h
> @@ -99,6 +99,8 @@ enum uv_cmds_inst {
>   enum uv_feat_ind {
>   	BIT_UV_FEAT_MISC = 0,
>   	BIT_UV_FEAT_AIV = 1,
> +	BIT_UV_FEAT_AP = 4,
> +	BIT_UV_FEAT_AP_INTR = 5,
>   };
>   
>   struct uv_cb_header {
> @@ -159,7 +161,15 @@ struct uv_cb_cgc {
>   	u64 guest_handle;
>   	u64 conf_base_stor_origin;
>   	u64 conf_virt_stor_origin;
> -	u64 reserved30;
> +	u8  reserved30[6];
> +	union {
> +		struct {
> +			u16 : 14;
> +			u16 ap_instr_intr : 1;
> +			u16 ap_allow_instr : 1;
> +		};
> +		u16 raw;
> +	} flags;
>   	u64 guest_stor_origin;
>   	u64 guest_stor_len;
>   	u64 guest_sca;
> diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/pv.c
> index 8570ee324607..75e81ba26d04 100644
> --- a/arch/s390/kvm/pv.c
> +++ b/arch/s390/kvm/pv.c
> @@ -576,12 +576,14 @@ int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
>   	uvcb.conf_base_stor_origin =
>   		virt_to_phys((void *)kvm->arch.pv.stor_base);
>   	uvcb.conf_virt_stor_origin = (u64)kvm->arch.pv.stor_var;
> +	uvcb.flags.ap_allow_instr = kvm->arch.model.uv_feat_guest.ap;
> +	uvcb.flags.ap_instr_intr = kvm->arch.model.uv_feat_guest.ap_intr;
>   
>   	cc = uv_call_sched(0, (u64)&uvcb);
>   	*rc = uvcb.header.rc;
>   	*rrc = uvcb.header.rrc;
> -	KVM_UV_EVENT(kvm, 3, "PROTVIRT CREATE VM: handle %llx len %llx rc %x rrc %x",
> -		     uvcb.guest_handle, uvcb.guest_stor_len, *rc, *rrc);
> +	KVM_UV_EVENT(kvm, 3, "PROTVIRT CREATE VM: handle %llx len %llx rc %x rrc %x flags %04x",
> +		     uvcb.guest_handle, uvcb.guest_stor_len, *rc, *rrc, uvcb.flags.raw);
>   
>   	/* Outputs */
>   	kvm->arch.pv.handle = uvcb.guest_handle;

Re: [PATCH v4 0/4] KVM: s390: Enable AP instructions for PV-guests

[Janosch Frank]

On 8/15/23 17:14, Steffen Eiden wrote:
> This series enables general KVM support for AP-passthrough for Secure
> Execution guests (PV-guests).

Thanks, picked