[PATCH 0/3] firmware: arm_ffa: Add support to run inside a guest/VM under a hypervisor

[PATCH 0/3] firmware: arm_ffa: Add support to run inside a guest/VM under a hypervisor

[Sudeep Holla]

Add support for running the driver inside a guest/VM under a hypervisor.

The main difference include:
1. supporting introducing notification pending interrupt and
2. Avoiding creation of all the notification bitmaps as they must be
   created by the hypervisor before the VM is initialised

The guest may need to use a notification pending interrupt instead of
the schedule receiver interrupt.

Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
---
Jens Wiklander (1):
      firmware: arm_ffa: Skip creation of the notification bitmaps

Sudeep Holla (2):
      firmware: arm_ffa: Rename the references to SRI to include NPI
      firmware: arm_ffa: Add support for handling notification pending interrupt(NPI)

 drivers/firmware/arm_ffa/driver.c | 94 ++++++++++++++++++++++++---------------
 1 file changed, 59 insertions(+), 35 deletions(-)
---
base-commit: 2c71fdf02a95b3dd425b42f28fd47fb2b1d22702
change-id: 20240410-ffa_npi_support-98edfdcc4882

Best regards,
-- 
Regards,
Sudeep


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 1/3] firmware: arm_ffa: Skip creation of the notification bitmaps

[Sudeep Holla]

From: Jens Wiklander <jens.wiklander@linaro.org>

When the FF-A driver is running inside a guest VM under an hypervisor,
the driver/guest VM doesn't have the permission/capability to request
the creation of notification bitmaps. For a VM, the hypervisor reserves
memory for its VM and hypervisor framework notification bitmaps and the
SPMC reserves memory for its SP and SPMC framework notification bitmaps
before the hypervisor initializes it.

The hypervisor does not initialize a VM if memory cannot be reserved
for all its notification bitmaps. So the creation of all the necessary
bitmaps are already done when the driver initialises and hence it can be
skipped. We rely on FFA_FEATURES(FFA_NOTIFICATION_BITMAP_CREATE) to fail
when running in the guest to handle this in the driver.

Signed-off-by:Jens Wiklander <jens.wiklander@linaro.org>
[sudeep.holla: Updated the commit message]
Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
---
 drivers/firmware/arm_ffa/driver.c | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c
index f2556a8e9401..4a576af7b8fd 100644
--- a/drivers/firmware/arm_ffa/driver.c
+++ b/drivers/firmware/arm_ffa/driver.c
@@ -1442,17 +1442,15 @@ static void ffa_notifications_setup(void)
 	int ret, irq;
 
 	ret = ffa_features(FFA_NOTIFICATION_BITMAP_CREATE, 0, NULL, NULL);
-	if (ret) {
-		pr_info("Notifications not supported, continuing with it ..\n");
-		return;
-	}
+	if (!ret) {
+		ret = ffa_notification_bitmap_create();
+		if (ret) {
+			pr_info("Notification bitmap create error %d\n", ret);
+			return;
+		}
 
-	ret = ffa_notification_bitmap_create();
-	if (ret) {
-		pr_info("Notification bitmap create error %d\n", ret);
-		return;
+		drv_info->bitmap_created = true;
 	}
-	drv_info->bitmap_created = true;
 
 	irq = ffa_sched_recv_irq_map();
 	if (irq <= 0) {

-- 
2.43.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 2/3] firmware: arm_ffa: Rename the references to SRI to include NPI

[Sudeep Holla]

In preparation to support handling of Notification Pending Interrupt(NPI)
in addition to the existing support for Schedule Reciever Interrupt(SRI),
rename all the references to SRI to reflect that it can be either SRI or
NRI. This change is just cosmetic and doesn't impact any features yet.
It also doesn't add the support for NPIs yet.

Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
---
 drivers/firmware/arm_ffa/driver.c | 41 ++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 20 deletions(-)

diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c
index 4a576af7b8fd..c299cb9e6889 100644
--- a/drivers/firmware/arm_ffa/driver.c
+++ b/drivers/firmware/arm_ffa/driver.c
@@ -100,7 +100,8 @@ struct ffa_drv_info {
 	bool mem_ops_native;
 	bool bitmap_created;
 	bool notif_enabled;
-	unsigned int sched_recv_irq;
+	/* Schedule Receiver(SRI) or Notification pending interrupt(NPI) */
+	unsigned int sr_or_np_irq;
 	unsigned int cpuhp_state;
 	struct ffa_pcpu_irq __percpu *irq_pcpu;
 	struct workqueue_struct *notif_pcpu_wq;
@@ -1291,7 +1292,7 @@ static void ffa_partitions_cleanup(void)
 #define FFA_FEAT_SCHEDULE_RECEIVER_INT		(2)
 #define FFA_FEAT_MANAGED_EXIT_INT		(3)
 
-static irqreturn_t irq_handler(int irq, void *irq_data)
+static irqreturn_t sri_handler(int irq, void *irq_data)
 {
 	struct ffa_pcpu_irq *pcpu = irq_data;
 	struct ffa_drv_info *info = pcpu->info;
@@ -1306,12 +1307,12 @@ static void ffa_sched_recv_irq_work_fn(struct work_struct *work)
 	ffa_notification_info_get();
 }
 
-static int ffa_sched_recv_irq_map(void)
+static int ffa_sr_or_np_irq_map(void)
 {
-	int ret, irq, sr_intid;
+	int ret, irq, intr_id;
 
-	/* The returned sr_intid is assumed to be SGI donated to NS world */
-	ret = ffa_features(FFA_FEAT_SCHEDULE_RECEIVER_INT, 0, &sr_intid, NULL);
+	/* The returned intr_id is assumed to be SGI donated to NS world */
+	ret = ffa_features(FFA_FEAT_SCHEDULE_RECEIVER_INT, 0, &intr_id, NULL);
 	if (ret < 0) {
 		if (ret != -EOPNOTSUPP)
 			pr_err("Failed to retrieve scheduler Rx interrupt\n");
@@ -1329,12 +1330,12 @@ static int ffa_sched_recv_irq_map(void)
 
 		oirq.np = gic;
 		oirq.args_count = 1;
-		oirq.args[0] = sr_intid;
+		oirq.args[0] = intr_id;
 		irq = irq_create_of_mapping(&oirq);
 		of_node_put(gic);
 #ifdef CONFIG_ACPI
 	} else {
-		irq = acpi_register_gsi(NULL, sr_intid, ACPI_EDGE_SENSITIVE,
+		irq = acpi_register_gsi(NULL, intr_id, ACPI_EDGE_SENSITIVE,
 					ACPI_ACTIVE_HIGH);
 #endif
 	}
@@ -1347,23 +1348,23 @@ static int ffa_sched_recv_irq_map(void)
 	return irq;
 }
 
-static void ffa_sched_recv_irq_unmap(void)
+static void ffa_sr_or_np_irq_unmap(void)
 {
-	if (drv_info->sched_recv_irq) {
-		irq_dispose_mapping(drv_info->sched_recv_irq);
-		drv_info->sched_recv_irq = 0;
+	if (drv_info->sr_or_np_irq) {
+		irq_dispose_mapping(drv_info->sr_or_np_irq);
+		drv_info->sr_or_np_irq = 0;
 	}
 }
 
 static int ffa_cpuhp_pcpu_irq_enable(unsigned int cpu)
 {
-	enable_percpu_irq(drv_info->sched_recv_irq, IRQ_TYPE_NONE);
+	enable_percpu_irq(drv_info->sr_or_np_irq, IRQ_TYPE_NONE);
 	return 0;
 }
 
 static int ffa_cpuhp_pcpu_irq_disable(unsigned int cpu)
 {
-	disable_percpu_irq(drv_info->sched_recv_irq);
+	disable_percpu_irq(drv_info->sr_or_np_irq);
 	return 0;
 }
 
@@ -1379,8 +1380,8 @@ static void ffa_uninit_pcpu_irq(void)
 		drv_info->notif_pcpu_wq = NULL;
 	}
 
-	if (drv_info->sched_recv_irq)
-		free_percpu_irq(drv_info->sched_recv_irq, drv_info->irq_pcpu);
+	if (drv_info->sr_or_np_irq)
+		free_percpu_irq(drv_info->sr_or_np_irq, drv_info->irq_pcpu);
 
 	if (drv_info->irq_pcpu) {
 		free_percpu(drv_info->irq_pcpu);
@@ -1402,7 +1403,7 @@ static int ffa_init_pcpu_irq(unsigned int irq)
 
 	drv_info->irq_pcpu = irq_pcpu;
 
-	ret = request_percpu_irq(irq, irq_handler, "ARM-FFA", irq_pcpu);
+	ret = request_percpu_irq(irq, sri_handler, "ARM-FFA-SRI", irq_pcpu);
 	if (ret) {
 		pr_err("Error registering notification IRQ %d: %d\n", irq, ret);
 		return ret;
@@ -1428,7 +1429,7 @@ static int ffa_init_pcpu_irq(unsigned int irq)
 static void ffa_notifications_cleanup(void)
 {
 	ffa_uninit_pcpu_irq();
-	ffa_sched_recv_irq_unmap();
+	ffa_sr_or_np_irq_unmap();
 
 	if (drv_info->bitmap_created) {
 		ffa_notification_bitmap_destroy();
@@ -1452,13 +1453,13 @@ static void ffa_notifications_setup(void)
 		drv_info->bitmap_created = true;
 	}
 
-	irq = ffa_sched_recv_irq_map();
+	irq = ffa_sr_or_np_irq_map();
 	if (irq <= 0) {
 		ret = irq;
 		goto cleanup;
 	}
 
-	drv_info->sched_recv_irq = irq;
+	drv_info->sr_or_np_irq = irq;
 
 	ret = ffa_init_pcpu_irq(irq);
 	if (ret)

-- 
2.43.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 3/3] firmware: arm_ffa: Add support for handling notification pending interrupt(NPI)

[Sudeep Holla]

The FF-A uses the notification pending interrupt to inform the receiver
that it has a pending notification. This is a virtual interrupt and is
used by the following type of receivers:
 1. A guest/VM running under a hypervisor.
 2. An S-EL1 SP running under a S-EL2 SPMC.

The rules that govern the properties of the NPI are the same as the
rules for the SRI with couple of exceptions. If the SRI is not supported,
then an attempt to discover NPI will be made in the driver.

The handling of NPI is also same as the handling of notification for the
self/primary VM with ID 0 except the absence of global notification.

Co-Developed-by: Jens Wiklander <jens.wiklander@linaro.org>
Signed-off-by: Sudeep Holla <sudeep.holla@arm.com>
---
 drivers/firmware/arm_ffa/driver.c | 43 +++++++++++++++++++++++++++++++--------
 1 file changed, 34 insertions(+), 9 deletions(-)

diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c
index c299cb9e6889..d50b41a815b9 100644
--- a/drivers/firmware/arm_ffa/driver.c
+++ b/drivers/firmware/arm_ffa/driver.c
@@ -100,6 +100,8 @@ struct ffa_drv_info {
 	bool mem_ops_native;
 	bool bitmap_created;
 	bool notif_enabled;
+	/* Flag to indicate it is Notification pending interrupt */
+	bool irq_is_npi;
 	/* Schedule Receiver(SRI) or Notification pending interrupt(NPI) */
 	unsigned int sr_or_np_irq;
 	unsigned int cpuhp_state;
@@ -1138,6 +1140,15 @@ ffa_self_notif_handle(u16 vcpu, bool is_per_vcpu, void *cb_data)
 			      &info->notif_pcpu_work);
 }
 
+static irqreturn_t npi_handler(int irq, void *irq_data)
+{
+	struct ffa_pcpu_irq *pcpu = irq_data;
+
+	ffa_self_notif_handle(smp_processor_id(), true, pcpu->info);
+
+	return IRQ_HANDLED;
+}
+
 static const struct ffa_info_ops ffa_drv_info_ops = {
 	.api_version_get = ffa_api_version_get,
 	.partition_info_get = ffa_partition_info_get,
@@ -1307,16 +1318,24 @@ static void ffa_sched_recv_irq_work_fn(struct work_struct *work)
 	ffa_notification_info_get();
 }
 
-static int ffa_sr_or_np_irq_map(void)
+static int ffa_sr_or_np_irq_map(bool *irq_is_npi)
 {
 	int ret, irq, intr_id;
 
 	/* The returned intr_id is assumed to be SGI donated to NS world */
 	ret = ffa_features(FFA_FEAT_SCHEDULE_RECEIVER_INT, 0, &intr_id, NULL);
-	if (ret < 0) {
-		if (ret != -EOPNOTSUPP)
-			pr_err("Failed to retrieve scheduler Rx interrupt\n");
+	if (ret && ret != -EOPNOTSUPP) {
+		pr_err("Failed to retrieve scheduler Rx interrupt\n");
 		return ret;
+	} else if (ret) {
+		ret = ffa_features(FFA_FEAT_NOTIFICATION_PENDING_INT, 0,
+				   &intr_id, NULL);
+		if (ret) {
+			if (ret != -EOPNOTSUPP)
+				pr_err("Failed to retrieve notification pending interrupt\n");
+			return ret;
+		}
+		*irq_is_npi = true;
 	}
 
 	if (acpi_disabled) {
@@ -1389,7 +1408,7 @@ static void ffa_uninit_pcpu_irq(void)
 	}
 }
 
-static int ffa_init_pcpu_irq(unsigned int irq)
+static int ffa_init_pcpu_irq(void)
 {
 	struct ffa_pcpu_irq __percpu *irq_pcpu;
 	int ret, cpu;
@@ -1403,9 +1422,15 @@ static int ffa_init_pcpu_irq(unsigned int irq)
 
 	drv_info->irq_pcpu = irq_pcpu;
 
-	ret = request_percpu_irq(irq, sri_handler, "ARM-FFA-SRI", irq_pcpu);
+	if (drv_info->irq_is_npi)
+		ret = request_percpu_irq(drv_info->sr_or_np_irq, npi_handler,
+					 "ARM-FFA-NPI", irq_pcpu);
+	else
+		ret = request_percpu_irq(drv_info->sr_or_np_irq, sri_handler,
+					 "ARM-FFA-SRI", irq_pcpu);
 	if (ret) {
-		pr_err("Error registering notification IRQ %d: %d\n", irq, ret);
+		pr_err("Error registering NPI/SRI IRQ %d: %d\n",
+		       drv_info->sr_or_np_irq, ret);
 		return ret;
 	}
 
@@ -1453,7 +1478,7 @@ static void ffa_notifications_setup(void)
 		drv_info->bitmap_created = true;
 	}
 
-	irq = ffa_sr_or_np_irq_map();
+	irq = ffa_sr_or_np_irq_map(&drv_info->irq_is_npi);
 	if (irq <= 0) {
 		ret = irq;
 		goto cleanup;
@@ -1461,7 +1486,7 @@ static void ffa_notifications_setup(void)
 
 	drv_info->sr_or_np_irq = irq;
 
-	ret = ffa_init_pcpu_irq(irq);
+	ret = ffa_init_pcpu_irq();
 	if (ret)
 		goto cleanup;
 

-- 
2.43.2


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 3/3] firmware: arm_ffa: Add support for handling notification pending interrupt(NPI)

[Sudeep Holla]

On Wed, Apr 10, 2024 at 12:39:48PM +0100, Sudeep Holla wrote:
> The FF-A uses the notification pending interrupt to inform the receiver
> that it has a pending notification. This is a virtual interrupt and is
> used by the following type of receivers:
>  1. A guest/VM running under a hypervisor.
>  2. An S-EL1 SP running under a S-EL2 SPMC.
> 
> The rules that govern the properties of the NPI are the same as the
> rules for the SRI with couple of exceptions. If the SRI is not supported,
> then an attempt to discover NPI will be made in the driver.
> 
> The handling of NPI is also same as the handling of notification for the
> self/primary VM with ID 0 except the absence of global notification.
>

OK I think I rushed to send this. Bertrand had now convinced me with a
use case that needs the support for both SRI and NPI in the driver.
XEN hypervisor can inject both SRI and NPI to the Dom0. The choice of
NPI over SRI is to avoid the context switch latency with the
FFA_NOTIFICATION_INFO_GET as NPI will just need FFA_NOTIFICATION_GET
which may completely avoid switching to secure side as XEN hyp handles
the same.

--
Regards,
Sudeep

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel