[PATCH 0/2] powerpc/pseries: Improve serialization of PRRN events

[PATCH 0/2] powerpc/pseries: Improve serialization of PRRN events

[John Allen]

Stress testing has uncovered issues with handling continuously queued PRRN
events. Running PRRN events in this way can seriously load the system given
the sheer volume of dlpar being handled. This patchset ensures that PRRN
events are handled more synchronously, only allowing the PRRN handler to
queue a single dlpar event at any given time.  Additionally, it ensures
that rtas polling continues normally when multiple PRRN events are queued
simultaneously.

John Allen (2):
  pseries/prrn: Avoid blocking rtas polling handling multiple PRRN
    events
  pseries/prrn: Wait for completion of hotplug events during PRRN
    handling

 arch/powerpc/kernel/rtasd.c               | 11 ++++++++---
 arch/powerpc/platforms/pseries/mobility.c |  5 ++++-
 2 files changed, 12 insertions(+), 4 deletions(-)

-- 
2.17.1

[PATCH 1/2] powerpc/pseries: Avoid blocking rtas polling handling multiple PRRN events

[John Allen]

When a PRRN event is being handled and another PRRN event comes in, the
second event will block rtas polling waiting on the first to complete,
preventing any further rtas events from being handled. This can be
especially problematic in case that PRRN events are continuously being
queued in which case rtas polling gets indefinitely blocked completely.

This patch introduces a mutex that prevents any subsequent PRRN events from
running while there is a prrn event being handled, allowing rtas polling to
continue normally.

Signed-off-by: John Allen <jallen@linux.ibm.com>
---
 arch/powerpc/kernel/rtasd.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/kernel/rtasd.c b/arch/powerpc/kernel/rtasd.c
index 44d66c33d59d..8a72a53d62c0 100644
--- a/arch/powerpc/kernel/rtasd.c
+++ b/arch/powerpc/kernel/rtasd.c
@@ -35,6 +35,8 @@
 
 static DEFINE_SPINLOCK(rtasd_log_lock);
 
+static DEFINE_MUTEX(prrn_lock);
+
 static DECLARE_WAIT_QUEUE_HEAD(rtas_log_wait);
 
 static char *rtas_log_buf;
@@ -290,9 +292,12 @@ static DECLARE_WORK(prrn_work, prrn_work_fn);
 
 static void prrn_schedule_update(u32 scope)
 {
-	flush_work(&prrn_work);
-	prrn_update_scope = scope;
-	schedule_work(&prrn_work);
+	if (mutex_trylock(&prrn_lock)) {
+		flush_work(&prrn_work);
+		prrn_update_scope = scope;
+		schedule_work(&prrn_work);
+		mutex_unlock(&prrn_lock);
+	}
 }
 
 static void handle_rtas_event(const struct rtas_error_log *log)
-- 
2.17.1

Re: [PATCH 1/2] powerpc/pseries: Avoid blocking rtas polling handling multiple PRRN events

[John Allen]

On Fri, Jul 13, 2018 at 09:22:23AM -0500, John Allen wrote:
>When a PRRN event is being handled and another PRRN event comes in, the
>second event will block rtas polling waiting on the first to complete,
>preventing any further rtas events from being handled. This can be
>especially problematic in case that PRRN events are continuously being
>queued in which case rtas polling gets indefinitely blocked completely.
>
>This patch introduces a mutex that prevents any subsequent PRRN events from
>running while there is a prrn event being handled, allowing rtas polling to
>continue normally.
>
>Signed-off-by: John Allen <jallen@linux.ibm.com>
>---
> arch/powerpc/kernel/rtasd.c | 11 ++++++++---
> 1 file changed, 8 insertions(+), 3 deletions(-)
>
>diff --git a/arch/powerpc/kernel/rtasd.c b/arch/powerpc/kernel/rtasd.c
>index 44d66c33d59d..8a72a53d62c0 100644
>--- a/arch/powerpc/kernel/rtasd.c
>+++ b/arch/powerpc/kernel/rtasd.c
>@@ -35,6 +35,8 @@
>
> static DEFINE_SPINLOCK(rtasd_log_lock);
>
>+static DEFINE_MUTEX(prrn_lock);
>+
> static DECLARE_WAIT_QUEUE_HEAD(rtas_log_wait);
>
> static char *rtas_log_buf;
>@@ -290,9 +292,12 @@ static DECLARE_WORK(prrn_work, prrn_work_fn);
>
> static void prrn_schedule_update(u32 scope)
> {
>-	flush_work(&prrn_work);
>-	prrn_update_scope = scope;
>-	schedule_work(&prrn_work);
>+	if (mutex_trylock(&prrn_lock)) {
>+		flush_work(&prrn_work);
>+		prrn_update_scope = scope;
>+		schedule_work(&prrn_work);
>+		mutex_unlock(&prrn_lock);

This appears to be bugged. The mutex_unlock should be done elsewhere.  
Will send an updated version.

-John

>+	}
> }
>
> static void handle_rtas_event(const struct rtas_error_log *log)
>-- 
>2.17.1
>

[PATCH 2/2] powerpc/pseries: Wait for completion of hotplug events during PRRN handling

[John Allen]

While handling PRRN events, the time to handle the actual hotplug events
dwarfs the time it takes to perform the device tree updates and queue the
hotplug events. In the case that PRRN events are being queued continuously,
hotplug events have been observed to be queued faster than the kernel can
actually handle them. This patch avoids the problem by waiting for a
hotplug request to complete before queueing more hotplug events.

Signed-off-by: John Allen <jallen@linux.ibm.com>
---
 arch/powerpc/platforms/pseries/mobility.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/platforms/pseries/mobility.c b/arch/powerpc/platforms/pseries/mobility.c
index 8a8033a249c7..49930848fa78 100644
--- a/arch/powerpc/platforms/pseries/mobility.c
+++ b/arch/powerpc/platforms/pseries/mobility.c
@@ -242,6 +242,7 @@ static int add_dt_node(__be32 parent_phandle, __be32 drc_index)
 static void prrn_update_node(__be32 phandle)
 {
 	struct pseries_hp_errorlog *hp_elog;
+	struct completion hotplug_done;
 	struct device_node *dn;
 
 	/*
@@ -263,7 +264,9 @@ static void prrn_update_node(__be32 phandle)
 	hp_elog->id_type = PSERIES_HP_ELOG_ID_DRC_INDEX;
 	hp_elog->_drc_u.drc_index = phandle;
 
-	queue_hotplug_event(hp_elog, NULL, NULL);
+	init_completion(&hotplug_done);
+	queue_hotplug_event(hp_elog, &hotplug_done, NULL);
+	wait_for_completion(&hotplug_done);
 
 	kfree(hp_elog);
 }
-- 
2.17.1