[RFC PATCH v6 0/3] arm/arm64: fix a migrating irq bug when hotplug cpu

[RFC PATCH v6 0/3] arm/arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

----
Changes in v6:
 - Don't touch the existing file kernel/irq/migration.c and add
   migrating interrupts code to a new file kernel/irq/cpuhotplug.c
   suggested by Thomas.
 - Change some comments in patch #2 and #3.

Changes in v5:
 - remove the macro that guard move_irqs()
 - use name irq_migrate_all_off_this_cpu instead of move_irqs

Changes in v4:
 - select GENERIC_IRQ_MIGRATION when config GENERIC_IRQ_MIGRATION is enabled
 - add move_irqs() into kerne/irq/migration.c for cpu hotplug
 - use move_irqs() to migrate interrupts on arm/arm64 when cpu is going to down
 - split bugfix patch into seperate patches for arm and arm64

Changes in v3:
 - introduce config GENERIC_IRQ_MIGRATION for compiling migration.c
 - rename migrate_irqs in arch/ia64/kernel/irq.c to avoid compiling error

Changes in v2:
 - use the exiting helper to set IRQD_MOVE_PCNTXT flag
 - use for_each_active_irq() instead of for_each_irq_desc()
 - add some warn messages when affinity is null or do set affinity failed
----

Hi All,

There is a bug:

When cpu is disabled, all irqs will be migratged to another cpu.
In some cases, a new affinity is different, it needed to be coppied
to irq's affinity. But if the type of irq is LPI, it's affinity will
not be coppied because of irq_set_affinity's return value.



As Marc and Will suggested, I refactor the arm/arm64 migrating interrupts
code and fix the migrating irq bug before cpu is offline.

I'm trying let the core code do the migrating interrupts matter. Add the migrating code
to a new file kernel/irq/cpuhotplug.c and make it depend on GENERIC_IRQ_MIGRATION. Fix
the bug by using irq_do_set_affinity. And use the new code to migrate interrupts before
cpu is offline on arm and arm64.


Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>

Yang Yingliang (3):
  genirq: introduce CONFIG_GENERIC_IRQ_MIGRATION and
    kernel/irq/cpuhotplug.c
  arm64: fix a migrating irq bug when hotplug cpu
  arm: fix a migrating irq bug when hotplug cpu

 arch/arm/Kconfig             |  1 +
 arch/arm/include/asm/irq.h   |  1 -
 arch/arm/kernel/irq.c        | 62 ----------------------------------------
 arch/arm/kernel/smp.c        |  2 +-
 arch/arm64/Kconfig           |  1 +
 arch/arm64/include/asm/irq.h |  1 -
 arch/arm64/kernel/irq.c      | 62 ----------------------------------------
 arch/arm64/kernel/smp.c      |  2 +-
 include/linux/irq.h          |  2 ++
 kernel/irq/Kconfig           |  4 +++
 kernel/irq/Makefile          |  1 +
 kernel/irq/cpuhotplug.c      | 68 ++++++++++++++++++++++++++++++++++++++++++++
 12 files changed, 79 insertions(+), 128 deletions(-)
 create mode 100644 kernel/irq/cpuhotplug.c

-- 
2.5.0

[RFC PATCH v6 1/3] genirq: introduce CONFIG_GENERIC_IRQ_MIGRATION and kernel/irq/cpuhotplug.c

[Yang Yingliang]

Add migrating interrupts code to a new file kernel/irq/cpuhotplug.c and
make it depends on CONFIG_GENERIC_IRQ_MIGRATION. So we can use it to migrate
interrupts, before cpu is offline.

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 include/linux/irq.h     |  2 ++
 kernel/irq/Kconfig      |  4 +++
 kernel/irq/Makefile     |  1 +
 kernel/irq/cpuhotplug.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 75 insertions(+)
 create mode 100644 kernel/irq/cpuhotplug.c

diff --git a/include/linux/irq.h b/include/linux/irq.h
index 11bf092..45cc729 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -452,6 +452,8 @@ extern int irq_set_affinity_locked(struct irq_data *data,
 				   const struct cpumask *cpumask, bool force);
 extern int irq_set_vcpu_affinity(unsigned int irq, void *vcpu_info);
 
+extern void irq_migrate_all_off_this_cpu(void);
+
 #if defined(CONFIG_SMP) && defined(CONFIG_GENERIC_PENDING_IRQ)
 void irq_move_irq(struct irq_data *data);
 void irq_move_masked_irq(struct irq_data *data);
diff --git a/kernel/irq/Kconfig b/kernel/irq/Kconfig
index 9a76e3b..3b48dab 100644
--- a/kernel/irq/Kconfig
+++ b/kernel/irq/Kconfig
@@ -30,6 +30,10 @@ config GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
 config GENERIC_PENDING_IRQ
 	bool
 
+# Support for generic irq migrating off cpu before the cpu is offline.
+config GENERIC_IRQ_MIGRATION
+	bool
+
 # Alpha specific irq affinity mechanism
 config AUTO_IRQ_AFFINITY
        bool
diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile
index d121235..2fc9cbd 100644
--- a/kernel/irq/Makefile
+++ b/kernel/irq/Makefile
@@ -5,5 +5,6 @@ obj-$(CONFIG_GENERIC_IRQ_PROBE) += autoprobe.o
 obj-$(CONFIG_IRQ_DOMAIN) += irqdomain.o
 obj-$(CONFIG_PROC_FS) += proc.o
 obj-$(CONFIG_GENERIC_PENDING_IRQ) += migration.o
+obj-$(CONFIG_GENERIC_IRQ_MIGRATION) += cpuhotplug.o
 obj-$(CONFIG_PM_SLEEP) += pm.o
 obj-$(CONFIG_GENERIC_MSI_IRQ) += msi.o
diff --git a/kernel/irq/cpuhotplug.c b/kernel/irq/cpuhotplug.c
new file mode 100644
index 0000000..8f2e9c0
--- /dev/null
+++ b/kernel/irq/cpuhotplug.c
@@ -0,0 +1,68 @@
+#include <linux/irq.h>
+#include <linux/interrupt.h>
+#include <linux/ratelimit.h>
+
+#include "internals.h"
+
+static bool migrate_one_irq(struct irq_desc *desc)
+{
+	struct irq_data *d = irq_desc_get_irq_data(desc);
+	const struct cpumask *affinity = d->common->affinity;
+	struct irq_chip *c;
+	bool ret = false;
+
+	/*
+	 * If this is a per-CPU interrupt, or the affinity does not
+	 * include this CPU, then we have nothing to do.
+	 */
+	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
+		return false;
+
+	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
+		affinity = cpu_online_mask;
+		ret = true;
+	}
+
+	c = irq_data_get_irq_chip(d);
+	if (!c->irq_set_affinity) {
+		pr_warn_ratelimited("IRQ%u: unable to set affinity\n", d->irq);
+	} else {
+		int r = irq_do_set_affinity(d, affinity, false);
+		if (r)
+			pr_warn_ratelimited("IRQ%u: set affinity failed(%d).\n", d->irq, r);
+	}
+
+	return ret;
+}
+
+/*
+ * The current CPU has been marked offline.  Migrate IRQs off this CPU.
+ * If the affinity settings do not allow other CPUs, force them onto any
+ * available CPU.
+ *
+ * Note: we must iterate over all IRQs, whether they have an attached
+ * action structure or not, as we need to get chained interrupts too.
+ */
+void irq_migrate_all_off_this_cpu(void)
+{
+	unsigned int irq;
+	struct irq_desc *desc;
+	unsigned long flags;
+
+	local_irq_save(flags);
+
+	for_each_active_irq(irq) {
+		bool affinity_broken;
+
+		desc = irq_to_desc(irq);
+		raw_spin_lock(&desc->lock);
+		affinity_broken = migrate_one_irq(desc);
+		raw_spin_unlock(&desc->lock);
+
+		if (affinity_broken)
+			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
+					    irq, smp_processor_id());
+	}
+
+	local_irq_restore(flags);
+}
-- 
2.5.0

[tip:irq/core] genirq: Introduce generic irq migration for cpu hotunplug

[Thomas Gleixner]

On Thu, 1 Oct 2015, tip-bot for Yang Yingliang wrote:

> Commit-ID:  f1e0bb0ad473a32d1b7e6d285ae9f7e47710bb5e
> Gitweb:     http://git.kernel.org/tip/f1e0bb0ad473a32d1b7e6d285ae9f7e47710bb5e
> Author:     Yang Yingliang <yangyingliang@huawei.com>
> AuthorDate: Thu, 24 Sep 2015 17:32:13 +0800
> Committer:  Thomas Gleixner <tglx@linutronix.de>
> CommitDate: Thu, 1 Oct 2015 14:51:15 +0200
> 
> genirq: Introduce generic irq migration for cpu hotunplug
> 
> ARM and ARM64 have almost identical code for migrating interrupts on
> cpu hotunplug. Provide a generic version which can be used by both.
> 
> The new code addresses a shortcoming in the ARM[64] variants which
> fails to update the affinity change in some cases. The solution for
> this is to use the core function irq_do_set_affinity() instead of open
> coding it.
> 
> [ tglx: Added copyright notice and license boilerplate. Rewrote
>   	subject and changelog. ]
> 
> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> Acked-by: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> Cc: <linux-arm-kernel@lists.infradead.org>
> Link: http://lkml.kernel.org/r/1443087135-17044-2-git-send-email-yangyingliang at huawei.com
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

Russell, Will,

I applied that to a seperate branch

  git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git irq/for-arm

You can pull that into arm[64] so you can apply the architecture
specific part.

Thanks,

	tglx

[tip:irq/core] genirq: Introduce generic irq migration for cpu hotunplug

[Catalin Marinas]

On Thu, Oct 01, 2015 at 04:24:24PM +0200, Thomas Gleixner wrote:
> On Thu, 1 Oct 2015, tip-bot for Yang Yingliang wrote:
> 
> > Commit-ID:  f1e0bb0ad473a32d1b7e6d285ae9f7e47710bb5e
> > Gitweb:     http://git.kernel.org/tip/f1e0bb0ad473a32d1b7e6d285ae9f7e47710bb5e
> > Author:     Yang Yingliang <yangyingliang@huawei.com>
> > AuthorDate: Thu, 24 Sep 2015 17:32:13 +0800
> > Committer:  Thomas Gleixner <tglx@linutronix.de>
> > CommitDate: Thu, 1 Oct 2015 14:51:15 +0200
> > 
> > genirq: Introduce generic irq migration for cpu hotunplug
> > 
> > ARM and ARM64 have almost identical code for migrating interrupts on
> > cpu hotunplug. Provide a generic version which can be used by both.
> > 
> > The new code addresses a shortcoming in the ARM[64] variants which
> > fails to update the affinity change in some cases. The solution for
> > this is to use the core function irq_do_set_affinity() instead of open
> > coding it.
> > 
> > [ tglx: Added copyright notice and license boilerplate. Rewrote
> >   	subject and changelog. ]
> > 
> > Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> > Acked-by: Russell King - ARM Linux <linux@arm.linux.org.uk>
> > Cc: Jiang Liu <jiang.liu@linux.intel.com>
> > Cc: Marc Zyngier <marc.zyngier@arm.com>
> > Cc: Mark Rutland <mark.rutland@arm.com>
> > Cc: Will Deacon <will.deacon@arm.com>
> > Cc: Hanjun Guo <hanjun.guo@linaro.org>
> > Cc: <linux-arm-kernel@lists.infradead.org>
> > Link: http://lkml.kernel.org/r/1443087135-17044-2-git-send-email-yangyingliang at huawei.com
> > Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> 
> Russell, Will,
> 
> I applied that to a seperate branch
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git irq/for-arm
> 
> You can pull that into arm[64] so you can apply the architecture
> specific part.

Thanks. I'll queue the corresponding arm64 patch for 4.4.

-- 
Catalin

[RFC PATCH v6 2/3] arm64: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

When cpu is disabled, all irqs will be migratged to another cpu.
In some cases, a new affinity is different, the old affinity need
to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
the old affinity can not be updated. Fix it by using irq_do_set_affinity.

And migrating interrupts is a core code matter, so use the generic
function irq_migrate_all_off_this_cpu() to migrate interrupts in
kernel/irq/migration.c.

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 arch/arm64/Kconfig           |  1 +
 arch/arm64/include/asm/irq.h |  1 -
 arch/arm64/kernel/irq.c      | 62 --------------------------------------------
 arch/arm64/kernel/smp.c      |  2 +-
 4 files changed, 2 insertions(+), 64 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 07d1811..279d71c 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -427,6 +427,7 @@ config NR_CPUS
 
 config HOTPLUG_CPU
 	bool "Support for hot-pluggable CPUs"
+	select GENERIC_IRQ_MIGRATION
 	help
 	  Say Y here to experiment with turning CPUs off and on.  CPUs
 	  can be controlled through /sys/devices/system/cpu.
diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h
index bbb251b..0916929 100644
--- a/arch/arm64/include/asm/irq.h
+++ b/arch/arm64/include/asm/irq.h
@@ -7,7 +7,6 @@
 
 struct pt_regs;
 
-extern void migrate_irqs(void);
 extern void set_handle_irq(void (*handle_irq)(struct pt_regs *));
 
 static inline void acpi_irq_init(void)
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 11dc3fd..9f17ec0 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -27,7 +27,6 @@
 #include <linux/init.h>
 #include <linux/irqchip.h>
 #include <linux/seq_file.h>
-#include <linux/ratelimit.h>
 
 unsigned long irq_err_count;
 
@@ -54,64 +53,3 @@ void __init init_IRQ(void)
 	if (!handle_arch_irq)
 		panic("No interrupt controller found.");
 }
-
-#ifdef CONFIG_HOTPLUG_CPU
-static bool migrate_one_irq(struct irq_desc *desc)
-{
-	struct irq_data *d = irq_desc_get_irq_data(desc);
-	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
-	struct irq_chip *c;
-	bool ret = false;
-
-	/*
-	 * If this is a per-CPU interrupt, or the affinity does not
-	 * include this CPU, then we have nothing to do.
-	 */
-	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
-		return false;
-
-	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
-		affinity = cpu_online_mask;
-		ret = true;
-	}
-
-	c = irq_data_get_irq_chip(d);
-	if (!c->irq_set_affinity)
-		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
-		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
-
-	return ret;
-}
-
-/*
- * The current CPU has been marked offline.  Migrate IRQs off this CPU.
- * If the affinity settings do not allow other CPUs, force them onto any
- * available CPU.
- *
- * Note: we must iterate over all IRQs, whether they have an attached
- * action structure or not, as we need to get chained interrupts too.
- */
-void migrate_irqs(void)
-{
-	unsigned int i;
-	struct irq_desc *desc;
-	unsigned long flags;
-
-	local_irq_save(flags);
-
-	for_each_irq_desc(i, desc) {
-		bool affinity_broken;
-
-		raw_spin_lock(&desc->lock);
-		affinity_broken = migrate_one_irq(desc);
-		raw_spin_unlock(&desc->lock);
-
-		if (affinity_broken)
-			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
-					    i, smp_processor_id());
-	}
-
-	local_irq_restore(flags);
-}
-#endif /* CONFIG_HOTPLUG_CPU */
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index dbdaacd..ee5cafa 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -232,7 +232,7 @@ int __cpu_disable(void)
 	/*
 	 * OK - migrate IRQs away from this CPU
 	 */
-	migrate_irqs();
+	irq_migrate_all_off_this_cpu();
 
 	/*
 	 * Remove this CPU from the vm mask set of all processes.
-- 
2.5.0

[RFC PATCH v6 2/3] arm64: fix a migrating irq bug when hotplug cpu

[Catalin Marinas]

On Thu, Sep 24, 2015 at 05:32:14PM +0800, Yang Yingliang wrote:
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, the old affinity need
> to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
> the old affinity can not be updated. Fix it by using irq_do_set_affinity.
> 
> And migrating interrupts is a core code matter, so use the generic
> function irq_migrate_all_off_this_cpu() to migrate interrupts in
> kernel/irq/migration.c.
> 
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>

I queued this patch for 4.4, on top of this branch from Thomas:

  git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git irq/for-arm

The arm (32-bit) patch would need to go via Russell's tree (patch
system, pull request or waiting until 4.4-rc1, it's up to Russell).

Thanks.

-- 
Catalin

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

When cpu is disabled, all irqs will be migratged to another cpu.
In some cases, a new affinity is different, the old affinity need
to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
the old affinity can not be updated. Fix it by using irq_do_set_affinity.

And migrating interrupts is a core code matter, so use the generic
function irq_migrate_all_off_this_cpu() to migrate interrupts in
kernel/irq/migration.c.

Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
Cc: Hanjun Guo <hanjun.guo@linaro.org>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
---
 arch/arm/Kconfig           |  1 +
 arch/arm/include/asm/irq.h |  1 -
 arch/arm/kernel/irq.c      | 62 ----------------------------------------------
 arch/arm/kernel/smp.c      |  2 +-
 4 files changed, 2 insertions(+), 64 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 72ad724..bffba78 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1492,6 +1492,7 @@ config NR_CPUS
 config HOTPLUG_CPU
 	bool "Support for hot-pluggable CPUs"
 	depends on SMP
+	select GENERIC_IRQ_MIGRATION
 	help
 	  Say Y here to experiment with turning CPUs off and on.  CPUs
 	  can be controlled through /sys/devices/system/cpu.
diff --git a/arch/arm/include/asm/irq.h b/arch/arm/include/asm/irq.h
index be1d07d..882bf98 100644
--- a/arch/arm/include/asm/irq.h
+++ b/arch/arm/include/asm/irq.h
@@ -24,7 +24,6 @@
 #ifndef __ASSEMBLY__
 struct irqaction;
 struct pt_regs;
-extern void migrate_irqs(void);
 
 extern void asm_do_IRQ(unsigned int, struct pt_regs *);
 void handle_IRQ(unsigned int, struct pt_regs *);
diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c
index 2766183..85299b7 100644
--- a/arch/arm/kernel/irq.c
+++ b/arch/arm/kernel/irq.c
@@ -31,7 +31,6 @@
 #include <linux/smp.h>
 #include <linux/init.h>
 #include <linux/seq_file.h>
-#include <linux/ratelimit.h>
 #include <linux/errno.h>
 #include <linux/list.h>
 #include <linux/kallsyms.h>
@@ -116,64 +115,3 @@ int __init arch_probe_nr_irqs(void)
 	return nr_irqs;
 }
 #endif
-
-#ifdef CONFIG_HOTPLUG_CPU
-static bool migrate_one_irq(struct irq_desc *desc)
-{
-	struct irq_data *d = irq_desc_get_irq_data(desc);
-	const struct cpumask *affinity = irq_data_get_affinity_mask(d);
-	struct irq_chip *c;
-	bool ret = false;
-
-	/*
-	 * If this is a per-CPU interrupt, or the affinity does not
-	 * include this CPU, then we have nothing to do.
-	 */
-	if (irqd_is_per_cpu(d) || !cpumask_test_cpu(smp_processor_id(), affinity))
-		return false;
-
-	if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
-		affinity = cpu_online_mask;
-		ret = true;
-	}
-
-	c = irq_data_get_irq_chip(d);
-	if (!c->irq_set_affinity)
-		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
-	else if (c->irq_set_affinity(d, affinity, false) == IRQ_SET_MASK_OK && ret)
-		cpumask_copy(irq_data_get_affinity_mask(d), affinity);
-
-	return ret;
-}
-
-/*
- * The current CPU has been marked offline.  Migrate IRQs off this CPU.
- * If the affinity settings do not allow other CPUs, force them onto any
- * available CPU.
- *
- * Note: we must iterate over all IRQs, whether they have an attached
- * action structure or not, as we need to get chained interrupts too.
- */
-void migrate_irqs(void)
-{
-	unsigned int i;
-	struct irq_desc *desc;
-	unsigned long flags;
-
-	local_irq_save(flags);
-
-	for_each_irq_desc(i, desc) {
-		bool affinity_broken;
-
-		raw_spin_lock(&desc->lock);
-		affinity_broken = migrate_one_irq(desc);
-		raw_spin_unlock(&desc->lock);
-
-		if (affinity_broken)
-			pr_warn_ratelimited("IRQ%u no longer affine to CPU%u\n",
-				i, smp_processor_id());
-	}
-
-	local_irq_restore(flags);
-}
-#endif /* CONFIG_HOTPLUG_CPU */
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index 48185a7..fc26384 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -218,7 +218,7 @@ int __cpu_disable(void)
 	/*
 	 * OK - migrate IRQs away from this CPU
 	 */
-	migrate_irqs();
+	irq_migrate_all_off_this_cpu();
 
 	/*
 	 * Flush user cache and TLB mappings, and then remove this CPU
-- 
2.5.0

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Russell King - ARM Linux]

On Thu, Sep 24, 2015 at 05:32:15PM +0800, Yang Yingliang wrote:
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, the old affinity need
> to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
> the old affinity can not be updated. Fix it by using irq_do_set_affinity.
> 
> And migrating interrupts is a core code matter, so use the generic
> function irq_migrate_all_off_this_cpu() to migrate interrupts in
> kernel/irq/migration.c.

Please can you put this patch in the patch system so I don't forget it
(it's taken tglx prompting and quite a long time to find this in my
mailbox already.)

See http:/www.arm.linux.org.uk/developer/patches/

Thanks.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Geert Uytterhoeven]

On Thu, Sep 24, 2015 at 11:32 AM, Yang Yingliang
<yangyingliang@huawei.com> wrote:
> When cpu is disabled, all irqs will be migratged to another cpu.
> In some cases, a new affinity is different, the old affinity need
> to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
> the old affinity can not be updated. Fix it by using irq_do_set_affinity.
>
> And migrating interrupts is a core code matter, so use the generic
> function irq_migrate_all_off_this_cpu() to migrate interrupts in
> kernel/irq/migration.c.
>
> Cc: Jiang Liu <jiang.liu@linux.intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marc Zyngier <marc.zyngier@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> Cc: Hanjun Guo <hanjun.guo@linaro.org>
> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> ---
>  arch/arm/Kconfig           |  1 +
>  arch/arm/include/asm/irq.h |  1 -
>  arch/arm/kernel/irq.c      | 62 ----------------------------------------------
>  arch/arm/kernel/smp.c      |  2 +-
>  4 files changed, 2 insertions(+), 64 deletions(-)
>
> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> index 72ad724..bffba78 100644
> --- a/arch/arm/Kconfig
> +++ b/arch/arm/Kconfig
> @@ -1492,6 +1492,7 @@ config NR_CPUS
>  config HOTPLUG_CPU
>         bool "Support for hot-pluggable CPUs"
>         depends on SMP
> +       select GENERIC_IRQ_MIGRATION

This causes the following warnings during s2ram on r8a7791/koelsch
(dual-core CA15):

 Freezing remaining freezable tasks ... (elapsed 0.003 seconds) done.
 PM: Suspending system (mem)
 Suspending console(s) (use no_console_suspend to debug)
 PM: suspend of devices complete after 56.533 msecs
 PM: late suspend of devices complete after 21.229 msecs
 PM: noirq suspend of devices complete after 21.093 msecs
 Disabling non-boot CPUs ...
+IRQ0: unable to set affinity
+IRQ1: unable to set affinity
+IRQ2: unable to set affinity
+IRQ3: unable to set affinity
+IRQ4: unable to set affinity
+IRQ5: unable to set affinity
+IRQ6: unable to set affinity
+IRQ7: unable to set affinity
+IRQ8: unable to set affinity
+IRQ9: unable to set affinity
 CPU1: shutdown
 Enabling non-boot CPUs ...
 CPU1 is up
 PM: noirq resume of devices complete after 13.605 msecs
 PM: early resume of devices complete after 15.177 msecs

/proc/interrupts doesn't have IRQ[0-9].

Printing the irqchip's name and removing the rate limiting reveals:

chip none IRQ0: unable to set affinity
chip none IRQ1: unable to set affinity
chip none IRQ2: unable to set affinity
chip none IRQ3: unable to set affinity
chip none IRQ4: unable to set affinity
chip none IRQ5: unable to set affinity
chip none IRQ6: unable to set affinity
chip none IRQ7: unable to set affinity
chip none IRQ8: unable to set affinity
chip none IRQ9: unable to set affinity
chip none IRQ10: unable to set affinity
chip none IRQ11: unable to set affinity
chip none IRQ12: unable to set affinity
chip none IRQ13: unable to set affinity
chip none IRQ14: unable to set affinity
chip none IRQ15: unable to set affinity
chip e6050000.gpio IRQ128: unable to set affinity
chip e6050000.gpio IRQ129: unable to set affinity
chip e6050000.gpio IRQ130: unable to set affinity
chip e6050000.gpio IRQ131: unable to set affinity
chip e6050000.gpio IRQ132: unable to set affinity
chip e6050000.gpio IRQ133: unable to set affinity
chip e6050000.gpio IRQ134: unable to set affinity
chip e6050000.gpio IRQ135: unable to set affinity
chip e6050000.gpio IRQ136: unable to set affinity
chip e6050000.gpio IRQ137: unable to set affinity
chip e6050000.gpio IRQ138: unable to set affinity
chip e6050000.gpio IRQ139: unable to set affinity
chip e6050000.gpio IRQ140: unable to set affinity
chip e6050000.gpio IRQ141: unable to set affinity
chip e6050000.gpio IRQ142: unable to set affinity
chip e6050000.gpio IRQ143: unable to set affinity
chip e6050000.gpio IRQ144: unable to set affinity
chip e6050000.gpio IRQ145: unable to set affinity
chip e6050000.gpio IRQ146: unable to set affinity
chip e6050000.gpio IRQ147: unable to set affinity
chip e6050000.gpio IRQ148: unable to set affinity
chip e6050000.gpio IRQ149: unable to set affinity
chip e6050000.gpio IRQ150: unable to set affinity
chip e6050000.gpio IRQ151: unable to set affinity
chip e6050000.gpio IRQ152: unable to set affinity
chip e6050000.gpio IRQ153: unable to set affinity
chip e6050000.gpio IRQ154: unable to set affinity
chip e6050000.gpio IRQ155: unable to set affinity
chip e6050000.gpio IRQ156: unable to set affinity
chip e6050000.gpio IRQ157: unable to set affinity
chip e6050000.gpio IRQ158: unable to set affinity
chip e6050000.gpio IRQ159: unable to set affinity
chip e6051000.gpio IRQ160: unable to set affinity
chip e6051000.gpio IRQ161: unable to set affinity
chip e6051000.gpio IRQ162: unable to set affinity
chip e6051000.gpio IRQ163: unable to set affinity
chip e6051000.gpio IRQ164: unable to set affinity
chip e6051000.gpio IRQ165: unable to set affinity
chip e6051000.gpio IRQ166: unable to set affinity
chip e6051000.gpio IRQ167: unable to set affinity
chip e6051000.gpio IRQ168: unable to set affinity
chip e6051000.gpio IRQ169: unable to set affinity
chip e6051000.gpio IRQ170: unable to set affinity
chip e6051000.gpio IRQ171: unable to set affinity
chip e6051000.gpio IRQ172: unable to set affinity
chip e6051000.gpio IRQ173: unable to set affinity
chip e6051000.gpio IRQ174: unable to set affinity
chip e6051000.gpio IRQ175: unable to set affinity
chip e6051000.gpio IRQ176: unable to set affinity
chip e6051000.gpio IRQ177: unable to set affinity
chip e6051000.gpio IRQ178: unable to set affinity
chip e6051000.gpio IRQ179: unable to set affinity
chip e6051000.gpio IRQ180: unable to set affinity
chip e6051000.gpio IRQ181: unable to set affinity
chip e6051000.gpio IRQ182: unable to set affinity
chip e6051000.gpio IRQ183: unable to set affinity
chip e6051000.gpio IRQ184: unable to set affinity
chip e6051000.gpio IRQ185: unable to set affinity
chip e6051000.gpio IRQ186: unable to set affinity
chip e6051000.gpio IRQ187: unable to set affinity
chip e6051000.gpio IRQ188: unable to set affinity
chip e6051000.gpio IRQ189: unable to set affinity
chip e6051000.gpio IRQ190: unable to set affinity
chip e6051000.gpio IRQ191: unable to set affinity
chip e6052000.gpio IRQ192: unable to set affinity
chip e6052000.gpio IRQ193: unable to set affinity
chip e6052000.gpio IRQ194: unable to set affinity
chip e6052000.gpio IRQ195: unable to set affinity
chip e6052000.gpio IRQ196: unable to set affinity
chip e6052000.gpio IRQ197: unable to set affinity
chip e6052000.gpio IRQ198: unable to set affinity
chip e6052000.gpio IRQ199: unable to set affinity
chip e6052000.gpio IRQ200: unable to set affinity
chip e6052000.gpio IRQ201: unable to set affinity
chip e6052000.gpio IRQ202: unable to set affinity
chip e6052000.gpio IRQ203: unable to set affinity
chip e6052000.gpio IRQ204: unable to set affinity
chip e6052000.gpio IRQ205: unable to set affinity
chip e6052000.gpio IRQ206: unable to set affinity
chip e6052000.gpio IRQ207: unable to set affinity
chip e6052000.gpio IRQ208: unable to set affinity
chip e6052000.gpio IRQ209: unable to set affinity
chip e6052000.gpio IRQ210: unable to set affinity
chip e6052000.gpio IRQ211: unable to set affinity
chip e6052000.gpio IRQ212: unable to set affinity
chip e6052000.gpio IRQ213: unable to set affinity
chip e6052000.gpio IRQ214: unable to set affinity
chip e6052000.gpio IRQ215: unable to set affinity
chip e6052000.gpio IRQ216: unable to set affinity
chip e6052000.gpio IRQ217: unable to set affinity
chip e6052000.gpio IRQ218: unable to set affinity
chip e6052000.gpio IRQ219: unable to set affinity
chip e6052000.gpio IRQ220: unable to set affinity
chip e6052000.gpio IRQ221: unable to set affinity
chip e6052000.gpio IRQ222: unable to set affinity
chip e6052000.gpio IRQ223: unable to set affinity
chip e6053000.gpio IRQ224: unable to set affinity
chip e6053000.gpio IRQ225: unable to set affinity
chip e6053000.gpio IRQ226: unable to set affinity
chip e6053000.gpio IRQ227: unable to set affinity
chip e6053000.gpio IRQ228: unable to set affinity
chip e6053000.gpio IRQ229: unable to set affinity
chip e6053000.gpio IRQ230: unable to set affinity
chip e6053000.gpio IRQ231: unable to set affinity
chip e6053000.gpio IRQ232: unable to set affinity
chip e6053000.gpio IRQ233: unable to set affinity
chip e6053000.gpio IRQ234: unable to set affinity
chip e6053000.gpio IRQ235: unable to set affinity
chip e6053000.gpio IRQ236: unable to set affinity
chip e6053000.gpio IRQ237: unable to set affinity
chip e6053000.gpio IRQ238: unable to set affinity
chip e6053000.gpio IRQ239: unable to set affinity
chip e6053000.gpio IRQ240: unable to set affinity
chip e6053000.gpio IRQ241: unable to set affinity
chip e6053000.gpio IRQ242: unable to set affinity
chip e6053000.gpio IRQ243: unable to set affinity
chip e6053000.gpio IRQ244: unable to set affinity
chip e6053000.gpio IRQ245: unable to set affinity
chip e6053000.gpio IRQ246: unable to set affinity
chip e6053000.gpio IRQ247: unable to set affinity
chip e6053000.gpio IRQ248: unable to set affinity
chip e6053000.gpio IRQ249: unable to set affinity
chip e6053000.gpio IRQ250: unable to set affinity
chip e6053000.gpio IRQ251: unable to set affinity
chip e6053000.gpio IRQ252: unable to set affinity
chip e6053000.gpio IRQ253: unable to set affinity
chip e6053000.gpio IRQ254: unable to set affinity
chip e6053000.gpio IRQ255: unable to set affinity
chip e6054000.gpio IRQ256: unable to set affinity
chip e6054000.gpio IRQ257: unable to set affinity
chip e6054000.gpio IRQ258: unable to set affinity
chip e6054000.gpio IRQ259: unable to set affinity
chip e6054000.gpio IRQ260: unable to set affinity
chip e6054000.gpio IRQ261: unable to set affinity
chip e6054000.gpio IRQ262: unable to set affinity
chip e6054000.gpio IRQ263: unable to set affinity
chip e6054000.gpio IRQ264: unable to set affinity
chip e6054000.gpio IRQ265: unable to set affinity
chip e6054000.gpio IRQ266: unable to set affinity
chip e6054000.gpio IRQ267: unable to set affinity
chip e6054000.gpio IRQ268: unable to set affinity
chip e6054000.gpio IRQ269: unable to set affinity
chip e6054000.gpio IRQ270: unable to set affinity
chip e6054000.gpio IRQ271: unable to set affinity
chip e6054000.gpio IRQ272: unable to set affinity
chip e6054000.gpio IRQ273: unable to set affinity
chip e6054000.gpio IRQ274: unable to set affinity
chip e6054000.gpio IRQ275: unable to set affinity
chip e6054000.gpio IRQ276: unable to set affinity
chip e6054000.gpio IRQ277: unable to set affinity
chip e6054000.gpio IRQ278: unable to set affinity
chip e6054000.gpio IRQ279: unable to set affinity
chip e6054000.gpio IRQ280: unable to set affinity
chip e6054000.gpio IRQ281: unable to set affinity
chip e6054000.gpio IRQ282: unable to set affinity
chip e6054000.gpio IRQ283: unable to set affinity
chip e6054000.gpio IRQ284: unable to set affinity
chip e6054000.gpio IRQ285: unable to set affinity
chip e6054000.gpio IRQ286: unable to set affinity
chip e6054000.gpio IRQ287: unable to set affinity
chip e6055000.gpio IRQ288: unable to set affinity
chip e6055000.gpio IRQ289: unable to set affinity
chip e6055000.gpio IRQ290: unable to set affinity
chip e6055000.gpio IRQ291: unable to set affinity
chip e6055000.gpio IRQ292: unable to set affinity
chip e6055000.gpio IRQ293: unable to set affinity
chip e6055000.gpio IRQ294: unable to set affinity
chip e6055000.gpio IRQ295: unable to set affinity
chip e6055000.gpio IRQ296: unable to set affinity
chip e6055000.gpio IRQ297: unable to set affinity
chip e6055000.gpio IRQ298: unable to set affinity
chip e6055000.gpio IRQ299: unable to set affinity
chip e6055000.gpio IRQ300: unable to set affinity
chip e6055000.gpio IRQ301: unable to set affinity
chip e6055000.gpio IRQ302: unable to set affinity
chip e6055000.gpio IRQ303: unable to set affinity
chip e6055000.gpio IRQ304: unable to set affinity
chip e6055000.gpio IRQ305: unable to set affinity
chip e6055000.gpio IRQ306: unable to set affinity
chip e6055000.gpio IRQ307: unable to set affinity
chip e6055000.gpio IRQ308: unable to set affinity
chip e6055000.gpio IRQ309: unable to set affinity
chip e6055000.gpio IRQ310: unable to set affinity
chip e6055000.gpio IRQ311: unable to set affinity
chip e6055000.gpio IRQ312: unable to set affinity
chip e6055000.gpio IRQ313: unable to set affinity
chip e6055000.gpio IRQ314: unable to set affinity
chip e6055000.gpio IRQ315: unable to set affinity
chip e6055000.gpio IRQ316: unable to set affinity
chip e6055000.gpio IRQ317: unable to set affinity
chip e6055000.gpio IRQ318: unable to set affinity
chip e6055000.gpio IRQ319: unable to set affinity
chip e6055400.gpio IRQ320: unable to set affinity
chip e6055400.gpio IRQ321: unable to set affinity
chip e6055400.gpio IRQ322: unable to set affinity
chip e6055400.gpio IRQ323: unable to set affinity
chip e6055400.gpio IRQ324: unable to set affinity
chip e6055400.gpio IRQ325: unable to set affinity
chip e6055400.gpio IRQ326: unable to set affinity
chip e6055400.gpio IRQ327: unable to set affinity
chip e6055400.gpio IRQ328: unable to set affinity
chip e6055400.gpio IRQ329: unable to set affinity
chip e6055400.gpio IRQ330: unable to set affinity
chip e6055400.gpio IRQ331: unable to set affinity
chip e6055400.gpio IRQ332: unable to set affinity
chip e6055400.gpio IRQ333: unable to set affinity
chip e6055400.gpio IRQ334: unable to set affinity
chip e6055400.gpio IRQ335: unable to set affinity
chip e6055400.gpio IRQ336: unable to set affinity
chip e6055400.gpio IRQ337: unable to set affinity
chip e6055400.gpio IRQ338: unable to set affinity
chip e6055400.gpio IRQ339: unable to set affinity
chip e6055400.gpio IRQ340: unable to set affinity
chip e6055400.gpio IRQ341: unable to set affinity
chip e6055400.gpio IRQ342: unable to set affinity
chip e6055400.gpio IRQ343: unable to set affinity
chip e6055400.gpio IRQ344: unable to set affinity
chip e6055400.gpio IRQ345: unable to set affinity
chip e6055400.gpio IRQ346: unable to set affinity
chip e6055400.gpio IRQ347: unable to set affinity
chip e6055400.gpio IRQ348: unable to set affinity
chip e6055400.gpio IRQ349: unable to set affinity
chip e6055400.gpio IRQ350: unable to set affinity
chip e6055400.gpio IRQ351: unable to set affinity
chip e6055800.gpio IRQ352: unable to set affinity
chip e6055800.gpio IRQ353: unable to set affinity
chip e6055800.gpio IRQ354: unable to set affinity
chip e6055800.gpio IRQ355: unable to set affinity
chip e6055800.gpio IRQ356: unable to set affinity
chip e6055800.gpio IRQ357: unable to set affinity
chip e6055800.gpio IRQ358: unable to set affinity
chip e6055800.gpio IRQ359: unable to set affinity
chip e6055800.gpio IRQ360: unable to set affinity
chip e6055800.gpio IRQ361: unable to set affinity
chip e6055800.gpio IRQ362: unable to set affinity
chip e6055800.gpio IRQ363: unable to set affinity
chip e6055800.gpio IRQ364: unable to set affinity
chip e6055800.gpio IRQ365: unable to set affinity
chip e6055800.gpio IRQ366: unable to set affinity
chip e6055800.gpio IRQ367: unable to set affinity
chip e6055800.gpio IRQ368: unable to set affinity
chip e6055800.gpio IRQ369: unable to set affinity
chip e6055800.gpio IRQ370: unable to set affinity
chip e6055800.gpio IRQ371: unable to set affinity
chip e6055800.gpio IRQ372: unable to set affinity
chip e6055800.gpio IRQ373: unable to set affinity
chip e6055800.gpio IRQ374: unable to set affinity
chip e6055800.gpio IRQ375: unable to set affinity
chip e6055800.gpio IRQ376: unable to set affinity
chip e6055800.gpio IRQ377: unable to set affinity
chip e61c0000.interrupt-controller IRQ378: unable to set affinity
chip da9063-irq IRQ379: unable to set affinity
chip da9063-irq IRQ380: unable to set affinity
chip da9063-irq IRQ381: unable to set affinity
chip da9063-irq IRQ382: unable to set affinity
chip da9063-irq IRQ383: unable to set affinity
chip da9063-irq IRQ384: unable to set affinity
chip da9063-irq IRQ385: unable to set affinity
chip da9063-irq IRQ386: unable to set affinity
chip da9063-irq IRQ387: unable to set affinity
chip da9063-irq IRQ388: unable to set affinity
chip da9063-irq IRQ389: unable to set affinity
chip da9063-irq IRQ390: unable to set affinity
chip da9063-irq IRQ391: unable to set affinity
chip da9063-irq IRQ392: unable to set affinity
chip da9063-irq IRQ393: unable to set affinity
chip da9063-irq IRQ394: unable to set affinity
chip da9063-irq IRQ395: unable to set affinity
chip da9063-irq IRQ396: unable to set affinity
chip da9063-irq IRQ397: unable to set affinity
chip da9063-irq IRQ398: unable to set affinity
chip da9063-irq IRQ399: unable to set affinity
chip da9063-irq IRQ400: unable to set affinity
chip da9063-irq IRQ401: unable to set affinity
chip da9063-irq IRQ402: unable to set affinity
chip da9063-irq IRQ403: unable to set affinity
chip da9063-irq IRQ404: unable to set affinity
chip da9063-irq IRQ405: unable to set affinity
chip da9063-irq IRQ406: unable to set affinity
chip da9063-irq IRQ407: unable to set affinity
chip e61c0000.interrupt-controller IRQ408: unable to set affinity

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Russell King - ARM Linux]

On Wed, Oct 21, 2015 at 01:47:49PM +0200, Geert Uytterhoeven wrote:
> On Thu, Sep 24, 2015 at 11:32 AM, Yang Yingliang
> <yangyingliang@huawei.com> wrote:
> > When cpu is disabled, all irqs will be migratged to another cpu.
> > In some cases, a new affinity is different, the old affinity need
> > to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
> > the old affinity can not be updated. Fix it by using irq_do_set_affinity.
> >
> > And migrating interrupts is a core code matter, so use the generic
> > function irq_migrate_all_off_this_cpu() to migrate interrupts in
> > kernel/irq/migration.c.
> >
> > Cc: Jiang Liu <jiang.liu@linux.intel.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Marc Zyngier <marc.zyngier@arm.com>
> > Cc: Mark Rutland <mark.rutland@arm.com>
> > Cc: Will Deacon <will.deacon@arm.com>
> > Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> > Cc: Hanjun Guo <hanjun.guo@linaro.org>
> > Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> > ---
> >  arch/arm/Kconfig           |  1 +
> >  arch/arm/include/asm/irq.h |  1 -
> >  arch/arm/kernel/irq.c      | 62 ----------------------------------------------
> >  arch/arm/kernel/smp.c      |  2 +-
> >  4 files changed, 2 insertions(+), 64 deletions(-)
> >
> > diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> > index 72ad724..bffba78 100644
> > --- a/arch/arm/Kconfig
> > +++ b/arch/arm/Kconfig
> > @@ -1492,6 +1492,7 @@ config NR_CPUS
> >  config HOTPLUG_CPU
> >         bool "Support for hot-pluggable CPUs"
> >         depends on SMP
> > +       select GENERIC_IRQ_MIGRATION
> 
> This causes the following warnings during s2ram on r8a7791/koelsch
> (dual-core CA15):

Thanks for the report.  I'll see what tonight's boot run says for my
platforms.  Hopefully, the author of these changes can help debug
this.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Russell King - ARM Linux]

On Wed, Oct 21, 2015 at 09:29:08PM +0100, Russell King - ARM Linux wrote:
> On Wed, Oct 21, 2015 at 01:47:49PM +0200, Geert Uytterhoeven wrote:
> > On Thu, Sep 24, 2015 at 11:32 AM, Yang Yingliang
> > <yangyingliang@huawei.com> wrote:
> > > When cpu is disabled, all irqs will be migratged to another cpu.
> > > In some cases, a new affinity is different, the old affinity need
> > > to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
> > > the old affinity can not be updated. Fix it by using irq_do_set_affinity.
> > >
> > > And migrating interrupts is a core code matter, so use the generic
> > > function irq_migrate_all_off_this_cpu() to migrate interrupts in
> > > kernel/irq/migration.c.
> > >
> > > Cc: Jiang Liu <jiang.liu@linux.intel.com>
> > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > Cc: Marc Zyngier <marc.zyngier@arm.com>
> > > Cc: Mark Rutland <mark.rutland@arm.com>
> > > Cc: Will Deacon <will.deacon@arm.com>
> > > Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
> > > Cc: Hanjun Guo <hanjun.guo@linaro.org>
> > > Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
> > > ---
> > >  arch/arm/Kconfig           |  1 +
> > >  arch/arm/include/asm/irq.h |  1 -
> > >  arch/arm/kernel/irq.c      | 62 ----------------------------------------------
> > >  arch/arm/kernel/smp.c      |  2 +-
> > >  4 files changed, 2 insertions(+), 64 deletions(-)
> > >
> > > diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
> > > index 72ad724..bffba78 100644
> > > --- a/arch/arm/Kconfig
> > > +++ b/arch/arm/Kconfig
> > > @@ -1492,6 +1492,7 @@ config NR_CPUS
> > >  config HOTPLUG_CPU
> > >         bool "Support for hot-pluggable CPUs"
> > >         depends on SMP
> > > +       select GENERIC_IRQ_MIGRATION
> > 
> > This causes the following warnings during s2ram on r8a7791/koelsch
> > (dual-core CA15):
> 
> Thanks for the report.  I'll see what tonight's boot run says for my
> platforms.  Hopefully, the author of these changes can help debug
> this.

What's happened is that:

-	c = irq_data_get_irq_chip(d);
-	if (!c->irq_set_affinity)
-		pr_debug("IRQ%u: unable to set affinity\n", d->irq);

has become:

+	c = irq_data_get_irq_chip(d);
+	if (!c->irq_set_affinity) {
+		pr_warn_ratelimited("IRQ%u: unable to set affinity\n", d->irq);

which makes things more noisy.

This is a change that was not described in the commit message for the
patch Thomas merged.

So, I think the right thing to do is to drop the patch set and revert
back to what we knew was sane, and get the submitter to do the job
properly: cleanly move the code from one location to another with _no_
changes what so ever, convert ARM and ARM64 to use it, and _then_ to
modify the resulting code.

>From what I can see, both ARM and ARM64 implementations here are
identical.

I'm certainly dropping this from the ARM tree.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Geert Uytterhoeven]

Hi Russell,

On Thu, Oct 22, 2015 at 11:26 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Wed, Oct 21, 2015 at 09:29:08PM +0100, Russell King - ARM Linux wrote:
>> On Wed, Oct 21, 2015 at 01:47:49PM +0200, Geert Uytterhoeven wrote:
>> > On Thu, Sep 24, 2015 at 11:32 AM, Yang Yingliang
>> > <yangyingliang@huawei.com> wrote:
>> > > When cpu is disabled, all irqs will be migratged to another cpu.
>> > > In some cases, a new affinity is different, the old affinity need
>> > > to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
>> > > the old affinity can not be updated. Fix it by using irq_do_set_affinity.
>> > >
>> > > And migrating interrupts is a core code matter, so use the generic
>> > > function irq_migrate_all_off_this_cpu() to migrate interrupts in
>> > > kernel/irq/migration.c.
>> > >
>> > > Cc: Jiang Liu <jiang.liu@linux.intel.com>
>> > > Cc: Thomas Gleixner <tglx@linutronix.de>
>> > > Cc: Marc Zyngier <marc.zyngier@arm.com>
>> > > Cc: Mark Rutland <mark.rutland@arm.com>
>> > > Cc: Will Deacon <will.deacon@arm.com>
>> > > Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
>> > > Cc: Hanjun Guo <hanjun.guo@linaro.org>
>> > > Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
>> > > ---
>> > >  arch/arm/Kconfig           |  1 +
>> > >  arch/arm/include/asm/irq.h |  1 -
>> > >  arch/arm/kernel/irq.c      | 62 ----------------------------------------------
>> > >  arch/arm/kernel/smp.c      |  2 +-
>> > >  4 files changed, 2 insertions(+), 64 deletions(-)
>> > >
>> > > diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
>> > > index 72ad724..bffba78 100644
>> > > --- a/arch/arm/Kconfig
>> > > +++ b/arch/arm/Kconfig
>> > > @@ -1492,6 +1492,7 @@ config NR_CPUS
>> > >  config HOTPLUG_CPU
>> > >         bool "Support for hot-pluggable CPUs"
>> > >         depends on SMP
>> > > +       select GENERIC_IRQ_MIGRATION
>> >
>> > This causes the following warnings during s2ram on r8a7791/koelsch
>> > (dual-core CA15):
>>
>> Thanks for the report.  I'll see what tonight's boot run says for my
>> platforms.  Hopefully, the author of these changes can help debug
>> this.
>
> What's happened is that:
>
> -       c = irq_data_get_irq_chip(d);
> -       if (!c->irq_set_affinity)
> -               pr_debug("IRQ%u: unable to set affinity\n", d->irq);
>
> has become:
>
> +       c = irq_data_get_irq_chip(d);
> +       if (!c->irq_set_affinity) {
> +               pr_warn_ratelimited("IRQ%u: unable to set affinity\n", d->irq);
>
> which makes things more noisy.
>
> This is a change that was not described in the commit message for the
> patch Thomas merged.
>
> So, I think the right thing to do is to drop the patch set and revert
> back to what we knew was sane, and get the submitter to do the job
> properly: cleanly move the code from one location to another with _no_
> changes what so ever, convert ARM and ARM64 to use it, and _then_ to
> modify the resulting code.
>
> From what I can see, both ARM and ARM64 implementations here are
> identical.
>
> I'm certainly dropping this from the ARM tree.

Thanks for your analysis!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

On 2015/10/22 17:26, Russell King - ARM Linux wrote:
> On Wed, Oct 21, 2015 at 09:29:08PM +0100, Russell King - ARM Linux wrote:
>> On Wed, Oct 21, 2015 at 01:47:49PM +0200, Geert Uytterhoeven wrote:
>>> On Thu, Sep 24, 2015 at 11:32 AM, Yang Yingliang
>>> <yangyingliang@huawei.com> wrote:
>>>> When cpu is disabled, all irqs will be migratged to another cpu.
>>>> In some cases, a new affinity is different, the old affinity need
>>>> to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE,
>>>> the old affinity can not be updated. Fix it by using irq_do_set_affinity.
>>>>
>>>> And migrating interrupts is a core code matter, so use the generic
>>>> function irq_migrate_all_off_this_cpu() to migrate interrupts in
>>>> kernel/irq/migration.c.
>>>>
>>>> Cc: Jiang Liu <jiang.liu@linux.intel.com>
>>>> Cc: Thomas Gleixner <tglx@linutronix.de>
>>>> Cc: Marc Zyngier <marc.zyngier@arm.com>
>>>> Cc: Mark Rutland <mark.rutland@arm.com>
>>>> Cc: Will Deacon <will.deacon@arm.com>
>>>> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>
>>>> Cc: Hanjun Guo <hanjun.guo@linaro.org>
>>>> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
>>>> ---
>>>>   arch/arm/Kconfig           |  1 +
>>>>   arch/arm/include/asm/irq.h |  1 -
>>>>   arch/arm/kernel/irq.c      | 62 ----------------------------------------------
>>>>   arch/arm/kernel/smp.c      |  2 +-
>>>>   4 files changed, 2 insertions(+), 64 deletions(-)
>>>>
>>>> diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
>>>> index 72ad724..bffba78 100644
>>>> --- a/arch/arm/Kconfig
>>>> +++ b/arch/arm/Kconfig
>>>> @@ -1492,6 +1492,7 @@ config NR_CPUS
>>>>   config HOTPLUG_CPU
>>>>          bool "Support for hot-pluggable CPUs"
>>>>          depends on SMP
>>>> +       select GENERIC_IRQ_MIGRATION
>>>
>>> This causes the following warnings during s2ram on r8a7791/koelsch
>>> (dual-core CA15):
>>
>> Thanks for the report.  I'll see what tonight's boot run says for my
>> platforms.  Hopefully, the author of these changes can help debug
>> this.
>
> What's happened is that:
>
> -	c = irq_data_get_irq_chip(d);
> -	if (!c->irq_set_affinity)
> -		pr_debug("IRQ%u: unable to set affinity\n", d->irq);
>
> has become:
>
> +	c = irq_data_get_irq_chip(d);
> +	if (!c->irq_set_affinity) {
> +		pr_warn_ratelimited("IRQ%u: unable to set affinity\n", d->irq);
>
> which makes things more noisy.
>
> This is a change that was not described in the commit message for the
> patch Thomas merged.

I described it in v2 cover letter and kept the change history in v6
cover letter. There is no comment on the change when patch the was
reviewing in v2, so I thought it's ok and I kept the change in the
next versions.

Need I send a patch to the Thomas branch to revert the change ?

Thanks,
Yang

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Russell King - ARM Linux]

On Thu, Oct 22, 2015 at 06:56:29PM +0800, Yang Yingliang wrote:
> I described it in v2 cover letter and kept the change history in v6
> cover letter. There is no comment on the change when patch the was
> reviewing in v2, so I thought it's ok and I kept the change in the
> next versions.

Cover letters don't always get read, neither do changelogs.

However, there's a principle here: never mix moving code around with
changes to that code.  Always move code with as few changes as possible
in one patch, and then make changes in a subsequent patch.

The "few changes as possible" means that if you need to make changes
for it to end up building in its new location, such as removing a
'static' or adding an 'EXPORT_SYMBOL' then those are fine, but the
main body of the code should remain identical, even down to style.

Any changes (such as, in this case, replacing pr_debug with pr_warn)
should be done as a distinctly separate patch so that such changes
are immediately obvious to reviewers.

> Need I send a patch to the Thomas branch to revert the change ?

I think wait for Thomas and Catalin to reply.  Your patch series is
currently merged into two different trees (Thomas' and Catalin's
trees) and what action is needed depends on how they want to handle
it.

The solutions are:
* A patch to restore the pr_debug() which Thomas applies, and Catalin
  and myself then pull Thomas' tree again, which potentially creates
  a messier history.

* Catalin drops the ARM64 change and Thomas' tree from the ARM64 tree,
  Thomas drops the original commit, and we start again doing it
  correctly.

Which is up to Catalin and Thomas.

I've dropped it from my tree as an easy way to fix the regression
on ARM for the time being, pending the outcome of deciding how to
fix this.

-- 
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Thomas Gleixner]

On Thu, 22 Oct 2015, Russell King - ARM Linux wrote:
> On Thu, Oct 22, 2015 at 06:56:29PM +0800, Yang Yingliang wrote:
> > I described it in v2 cover letter and kept the change history in v6
> > cover letter. There is no comment on the change when patch the was
> > reviewing in v2, so I thought it's ok and I kept the change in the
> > next versions.
> 
> Cover letters don't always get read, neither do changelogs.
> 
> However, there's a principle here: never mix moving code around with
> changes to that code.  Always move code with as few changes as possible
> in one patch, and then make changes in a subsequent patch.
> 
> The "few changes as possible" means that if you need to make changes
> for it to end up building in its new location, such as removing a
> 'static' or adding an 'EXPORT_SYMBOL' then those are fine, but the
> main body of the code should remain identical, even down to style.
> 
> Any changes (such as, in this case, replacing pr_debug with pr_warn)
> should be done as a distinctly separate patch so that such changes
> are immediately obvious to reviewers.
> 
> > Need I send a patch to the Thomas branch to revert the change ?
> 
> I think wait for Thomas and Catalin to reply.  Your patch series is
> currently merged into two different trees (Thomas' and Catalin's
> trees) and what action is needed depends on how they want to handle
> it.
> 
> The solutions are:
> * A patch to restore the pr_debug() which Thomas applies, and Catalin
>   and myself then pull Thomas' tree again, which potentially creates
>   a messier history.
> 
> * Catalin drops the ARM64 change and Thomas' tree from the ARM64 tree,
>   Thomas drops the original commit, and we start again doing it
>   correctly.
> 
> Which is up to Catalin and Thomas.

I'd have to do a revert as it's in the middle of other changes. So I
prefer to do an incremental fix.

I committed the change into the irq/for-arm branch and pushed it out.

Thanks,

	tglx

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Catalin Marinas]

On Thu, Oct 22, 2015 at 02:43:34PM +0200, Thomas Gleixner wrote:
> On Thu, 22 Oct 2015, Russell King - ARM Linux wrote:
> > The solutions are:
> > * A patch to restore the pr_debug() which Thomas applies, and Catalin
> >   and myself then pull Thomas' tree again, which potentially creates
> >   a messier history.
> > 
> > * Catalin drops the ARM64 change and Thomas' tree from the ARM64 tree,
> >   Thomas drops the original commit, and we start again doing it
> >   correctly.
> > 
> > Which is up to Catalin and Thomas.
> 
> I'd have to do a revert as it's in the middle of other changes. So I
> prefer to do an incremental fix.
> 
> I committed the change into the irq/for-arm branch and pushed it out.

I pulled the branch again into arm64 for-next/core. Thanks.

-- 
Catalin

[RFC PATCH v6 3/3] arm: fix a migrating irq bug when hotplug cpu

[Yang Yingliang]

Hi, Russell

On 2015/10/22 19:13, Russell King - ARM Linux wrote:
> On Thu, Oct 22, 2015 at 06:56:29PM +0800, Yang Yingliang wrote:
>> I described it in v2 cover letter and kept the change history in v6
>> cover letter. There is no comment on the change when patch the was
>> reviewing in v2, so I thought it's ok and I kept the change in the
>> next versions.
>
> Cover letters don't always get read, neither do changelogs.
>
> However, there's a principle here: never mix moving code around with
> changes to that code.  Always move code with as few changes as possible
> in one patch, and then make changes in a subsequent patch.
>
> The "few changes as possible" means that if you need to make changes
> for it to end up building in its new location, such as removing a
> 'static' or adding an 'EXPORT_SYMBOL' then those are fine, but the
> main body of the code should remain identical, even down to style.
>
> Any changes (such as, in this case, replacing pr_debug with pr_warn)
> should be done as a distinctly separate patch so that such changes
> are immediately obvious to reviewers.
>
>> Need I send a patch to the Thomas branch to revert the change ?
>
> I think wait for Thomas and Catalin to reply.  Your patch series is
> currently merged into two different trees (Thomas' and Catalin's
> trees) and what action is needed depends on how they want to handle
> it.
>
> The solutions are:
> * A patch to restore the pr_debug() which Thomas applies, and Catalin
>    and myself then pull Thomas' tree again, which potentially creates
>    a messier history.
>
> * Catalin drops the ARM64 change and Thomas' tree from the ARM64 tree,
>    Thomas drops the original commit, and we start again doing it
>    correctly.
>
> Which is up to Catalin and Thomas.
>
> I've dropped it from my tree as an easy way to fix the regression
> on ARM for the time being, pending the outcome of deciding how to
> fix this.
>

The regression had been fixed.
Do I need to put the patch in the patch system again ?

Regards,
Yang