[PATCH 00/10] MIPS CPS cpuidle

[PATCH 00/10] MIPS CPS cpuidle

[Paul Burton]

This series introduces a cpuidle driver for systems based around the
MIPS Coherent Processing System, ie. those with a Coherence Manager and
optionally a Cluster Power Controller. The resulting driver is very
much foundational work to which deeper power states will later be added.

Tested on Malta using interAptiv bitstreams with both 1 & 2 VPEs in each
of 2 cores, and a multi-core proAptiv bitstream.

This series depends upon the CM changes of my "MIPS CPS SMP" series.

Paul Burton (10):
  MIPS: inst.h: define COP0 wait op
  MIPS: inst.h: define missing microMIPS POOL32AXf ops
  MIPS: uasm: add sync instruction
  MIPS: uasm: add wait instruction
  MIPS: uasm: add jr.hb instruction
  MIPS: uasm: add mftc0 & mttc0 instructions
  MIPS: uasm: add a label variant of beq
  MIPS: support use of cpuidle
  cpuidle: declare cpuidle_dev in cpuidle.h
  cpuidle: cpuidle driver for MIPS CPS

 arch/mips/Kconfig                  |   2 +
 arch/mips/include/asm/idle.h       |  14 +
 arch/mips/include/asm/uasm.h       |   7 +
 arch/mips/include/uapi/asm/inst.h  |  15 +-
 arch/mips/kernel/idle.c            |  20 +-
 arch/mips/mm/uasm-micromips.c      |   3 +
 arch/mips/mm/uasm-mips.c           |   5 +
 arch/mips/mm/uasm.c                |  25 +-
 drivers/cpuidle/Kconfig            |   5 +
 drivers/cpuidle/Kconfig.mips       |  14 +
 drivers/cpuidle/Makefile           |   3 +
 drivers/cpuidle/cpuidle-mips-cps.c | 545 +++++++++++++++++++++++++++++++++++++
 include/linux/cpuidle.h            |   1 +
 13 files changed, 650 insertions(+), 9 deletions(-)
 create mode 100644 drivers/cpuidle/Kconfig.mips
 create mode 100644 drivers/cpuidle/cpuidle-mips-cps.c

-- 
1.8.4.2

[PATCH 00/10] MIPS CPS cpuidle

[Paul Burton]

This series introduces a cpuidle driver for systems based around the
MIPS Coherent Processing System, ie. those with a Coherence Manager and
optionally a Cluster Power Controller. The resulting driver is very
much foundational work to which deeper power states will later be added.

Tested on Malta using interAptiv bitstreams with both 1 & 2 VPEs in each
of 2 cores, and a multi-core proAptiv bitstream.

This series depends upon the CM changes of my "MIPS CPS SMP" series.

Paul Burton (10):
  MIPS: inst.h: define COP0 wait op
  MIPS: inst.h: define missing microMIPS POOL32AXf ops
  MIPS: uasm: add sync instruction
  MIPS: uasm: add wait instruction
  MIPS: uasm: add jr.hb instruction
  MIPS: uasm: add mftc0 & mttc0 instructions
  MIPS: uasm: add a label variant of beq
  MIPS: support use of cpuidle
  cpuidle: declare cpuidle_dev in cpuidle.h
  cpuidle: cpuidle driver for MIPS CPS

 arch/mips/Kconfig                  |   2 +
 arch/mips/include/asm/idle.h       |  14 +
 arch/mips/include/asm/uasm.h       |   7 +
 arch/mips/include/uapi/asm/inst.h  |  15 +-
 arch/mips/kernel/idle.c            |  20 +-
 arch/mips/mm/uasm-micromips.c      |   3 +
 arch/mips/mm/uasm-mips.c           |   5 +
 arch/mips/mm/uasm.c                |  25 +-
 drivers/cpuidle/Kconfig            |   5 +
 drivers/cpuidle/Kconfig.mips       |  14 +
 drivers/cpuidle/Makefile           |   3 +
 drivers/cpuidle/cpuidle-mips-cps.c | 545 +++++++++++++++++++++++++++++++++++++
 include/linux/cpuidle.h            |   1 +
 13 files changed, 650 insertions(+), 9 deletions(-)
 create mode 100644 drivers/cpuidle/Kconfig.mips
 create mode 100644 drivers/cpuidle/cpuidle-mips-cps.c

-- 
1.8.4.2

[PATCH 01/10] MIPS: inst.h: define COP0 wait op

[Paul Burton]

The func field for the wait instruction was missing from inst.h - this
patch adds it.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/uapi/asm/inst.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index b39ba25..b4a24bd 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -118,7 +118,8 @@ enum bcop_op {
 enum cop0_coi_func {
 	tlbr_op	      = 0x01, tlbwi_op	    = 0x02,
 	tlbwr_op      = 0x06, tlbp_op	    = 0x08,
-	rfe_op	      = 0x10, eret_op	    = 0x18
+	rfe_op	      = 0x10, eret_op	    = 0x18,
+	wait_op       = 0x20,
 };
 
 /*
-- 
1.8.4.2

[PATCH 01/10] MIPS: inst.h: define COP0 wait op

[Paul Burton]

The func field for the wait instruction was missing from inst.h - this
patch adds it.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/uapi/asm/inst.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index b39ba25..b4a24bd 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -118,7 +118,8 @@ enum bcop_op {
 enum cop0_coi_func {
 	tlbr_op	      = 0x01, tlbwi_op	    = 0x02,
 	tlbwr_op      = 0x06, tlbp_op	    = 0x08,
-	rfe_op	      = 0x10, eret_op	    = 0x18
+	rfe_op	      = 0x10, eret_op	    = 0x18,
+	wait_op       = 0x20,
 };
 
 /*
-- 
1.8.4.2

[PATCH 02/10] MIPS: inst.h: define missing microMIPS POOL32AXf ops

[Paul Burton]

The opcodes for the sync & wait instructions within POOL32AXf were
missing. This patch adds them for use by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/uapi/asm/inst.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index b4a24bd..203f28b 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -295,7 +295,9 @@ enum mm_32axf_minor_op {
 	mm_tlbwr_op = 0x0cd,
 	mm_jalrs_op = 0x13c,
 	mm_jalrshb_op = 0x17c,
+	mm_sync_op = 0x1ad,
 	mm_syscall_op = 0x22d,
+	mm_wait_op = 0x24d,
 	mm_eret_op = 0x3cd,
 };
 
-- 
1.8.4.2

[PATCH 02/10] MIPS: inst.h: define missing microMIPS POOL32AXf ops

[Paul Burton]

The opcodes for the sync & wait instructions within POOL32AXf were
missing. This patch adds them for use by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/uapi/asm/inst.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index b4a24bd..203f28b 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -295,7 +295,9 @@ enum mm_32axf_minor_op {
 	mm_tlbwr_op = 0x0cd,
 	mm_jalrs_op = 0x13c,
 	mm_jalrshb_op = 0x17c,
+	mm_sync_op = 0x1ad,
 	mm_syscall_op = 0x22d,
+	mm_wait_op = 0x24d,
 	mm_eret_op = 0x3cd,
 };
 
-- 
1.8.4.2

[PATCH 03/10] MIPS: uasm: add sync instruction

[Paul Burton]

This patch allows use of the sync instruction from uasm. It will be used
by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h  | 1 +
 arch/mips/mm/uasm-micromips.c | 1 +
 arch/mips/mm/uasm-mips.c      | 1 +
 arch/mips/mm/uasm.c           | 3 ++-
 4 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index c33a956..4f4ed49 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -137,6 +137,7 @@ Ip_u2u1u3(_sra);
 Ip_u2u1u3(_srl);
 Ip_u3u1u2(_subu);
 Ip_u2s3u1(_sw);
+Ip_u1(_sync);
 Ip_u1(_syscall);
 Ip_0(_tlbp);
 Ip_0(_tlbr);
diff --git a/arch/mips/mm/uasm-micromips.c b/arch/mips/mm/uasm-micromips.c
index 060000f..15b46da 100644
--- a/arch/mips/mm/uasm-micromips.c
+++ b/arch/mips/mm/uasm-micromips.c
@@ -100,6 +100,7 @@ static struct insn insn_table_MM[] = {
 	{ insn_rotr, M(mm_pool32a_op, 0, 0, 0, 0, mm_rotr_op), RT | RS | RD },
 	{ insn_subu, M(mm_pool32a_op, 0, 0, 0, 0, mm_subu32_op), RT | RS | RD },
 	{ insn_sw, M(mm_sw32_op, 0, 0, 0, 0, 0), RT | RS | SIMM },
+	{ insn_sync, M(mm_pool32a_op, 0, 0, 0, mm_sync_op, mm_pool32axf_op), RS },
 	{ insn_tlbp, M(mm_pool32a_op, 0, 0, 0, mm_tlbp_op, mm_pool32axf_op), 0 },
 	{ insn_tlbr, M(mm_pool32a_op, 0, 0, 0, mm_tlbr_op, mm_pool32axf_op), 0 },
 	{ insn_tlbwi, M(mm_pool32a_op, 0, 0, 0, mm_tlbwi_op, mm_pool32axf_op), 0 },
diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c
index 0c72458..07ec837 100644
--- a/arch/mips/mm/uasm-mips.c
+++ b/arch/mips/mm/uasm-mips.c
@@ -107,6 +107,7 @@ static struct insn insn_table[] = {
 	{ insn_srl,  M(spec_op, 0, 0, 0, 0, srl_op),  RT | RD | RE },
 	{ insn_subu,  M(spec_op, 0, 0, 0, 0, subu_op),	RS | RT | RD },
 	{ insn_sw,  M(sw_op, 0, 0, 0, 0, 0),  RS | RT | SIMM },
+	{ insn_sync, M(spec_op, 0, 0, 0, 0, sync_op), RE },
 	{ insn_syscall, M(spec_op, 0, 0, 0, 0, syscall_op), SCIMM},
 	{ insn_tlbp,  M(cop0_op, cop_op, 0, 0, 0, tlbp_op),  0 },
 	{ insn_tlbr,  M(cop0_op, cop_op, 0, 0, 0, tlbr_op),  0 },
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index b9d14b6..e688800 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -52,7 +52,7 @@ enum opcode {
 	insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_ld, insn_ldx,
 	insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0, insn_mtc0,
 	insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc, insn_scd,
-	insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw,
+	insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw, insn_sync,
 	insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_xor,
 	insn_xori,
 };
@@ -270,6 +270,7 @@ I_u2u1u3(_srl)
 I_u2u1u3(_rotr)
 I_u3u1u2(_subu)
 I_u2s3u1(_sw)
+I_u1(_sync)
 I_0(_tlbp)
 I_0(_tlbr)
 I_0(_tlbwi)
-- 
1.8.4.2

[PATCH 03/10] MIPS: uasm: add sync instruction

[Paul Burton]

This patch allows use of the sync instruction from uasm. It will be used
by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h  | 1 +
 arch/mips/mm/uasm-micromips.c | 1 +
 arch/mips/mm/uasm-mips.c      | 1 +
 arch/mips/mm/uasm.c           | 3 ++-
 4 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index c33a956..4f4ed49 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -137,6 +137,7 @@ Ip_u2u1u3(_sra);
 Ip_u2u1u3(_srl);
 Ip_u3u1u2(_subu);
 Ip_u2s3u1(_sw);
+Ip_u1(_sync);
 Ip_u1(_syscall);
 Ip_0(_tlbp);
 Ip_0(_tlbr);
diff --git a/arch/mips/mm/uasm-micromips.c b/arch/mips/mm/uasm-micromips.c
index 060000f..15b46da 100644
--- a/arch/mips/mm/uasm-micromips.c
+++ b/arch/mips/mm/uasm-micromips.c
@@ -100,6 +100,7 @@ static struct insn insn_table_MM[] = {
 	{ insn_rotr, M(mm_pool32a_op, 0, 0, 0, 0, mm_rotr_op), RT | RS | RD },
 	{ insn_subu, M(mm_pool32a_op, 0, 0, 0, 0, mm_subu32_op), RT | RS | RD },
 	{ insn_sw, M(mm_sw32_op, 0, 0, 0, 0, 0), RT | RS | SIMM },
+	{ insn_sync, M(mm_pool32a_op, 0, 0, 0, mm_sync_op, mm_pool32axf_op), RS },
 	{ insn_tlbp, M(mm_pool32a_op, 0, 0, 0, mm_tlbp_op, mm_pool32axf_op), 0 },
 	{ insn_tlbr, M(mm_pool32a_op, 0, 0, 0, mm_tlbr_op, mm_pool32axf_op), 0 },
 	{ insn_tlbwi, M(mm_pool32a_op, 0, 0, 0, mm_tlbwi_op, mm_pool32axf_op), 0 },
diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c
index 0c72458..07ec837 100644
--- a/arch/mips/mm/uasm-mips.c
+++ b/arch/mips/mm/uasm-mips.c
@@ -107,6 +107,7 @@ static struct insn insn_table[] = {
 	{ insn_srl,  M(spec_op, 0, 0, 0, 0, srl_op),  RT | RD | RE },
 	{ insn_subu,  M(spec_op, 0, 0, 0, 0, subu_op),	RS | RT | RD },
 	{ insn_sw,  M(sw_op, 0, 0, 0, 0, 0),  RS | RT | SIMM },
+	{ insn_sync, M(spec_op, 0, 0, 0, 0, sync_op), RE },
 	{ insn_syscall, M(spec_op, 0, 0, 0, 0, syscall_op), SCIMM},
 	{ insn_tlbp,  M(cop0_op, cop_op, 0, 0, 0, tlbp_op),  0 },
 	{ insn_tlbr,  M(cop0_op, cop_op, 0, 0, 0, tlbr_op),  0 },
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index b9d14b6..e688800 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -52,7 +52,7 @@ enum opcode {
 	insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_ld, insn_ldx,
 	insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0, insn_mtc0,
 	insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc, insn_scd,
-	insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw,
+	insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw, insn_sync,
 	insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_xor,
 	insn_xori,
 };
@@ -270,6 +270,7 @@ I_u2u1u3(_srl)
 I_u2u1u3(_rotr)
 I_u3u1u2(_subu)
 I_u2s3u1(_sw)
+I_u1(_sync)
 I_0(_tlbp)
 I_0(_tlbr)
 I_0(_tlbwi)
-- 
1.8.4.2

[PATCH 04/10] MIPS: uasm: add wait instruction

[Paul Burton]

This patch allows use of the wait instruction from uasm. It will be used
by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h  | 1 +
 arch/mips/mm/uasm-micromips.c | 1 +
 arch/mips/mm/uasm-mips.c      | 1 +
 arch/mips/mm/uasm.c           | 5 +++--
 4 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 4f4ed49..7458a94 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -143,6 +143,7 @@ Ip_0(_tlbp);
 Ip_0(_tlbr);
 Ip_0(_tlbwi);
 Ip_0(_tlbwr);
+Ip_u1(_wait);
 Ip_u3u1u2(_xor);
 Ip_u2u1u3(_xori);
 
diff --git a/arch/mips/mm/uasm-micromips.c b/arch/mips/mm/uasm-micromips.c
index 15b46da..1150eda 100644
--- a/arch/mips/mm/uasm-micromips.c
+++ b/arch/mips/mm/uasm-micromips.c
@@ -105,6 +105,7 @@ static struct insn insn_table_MM[] = {
 	{ insn_tlbr, M(mm_pool32a_op, 0, 0, 0, mm_tlbr_op, mm_pool32axf_op), 0 },
 	{ insn_tlbwi, M(mm_pool32a_op, 0, 0, 0, mm_tlbwi_op, mm_pool32axf_op), 0 },
 	{ insn_tlbwr, M(mm_pool32a_op, 0, 0, 0, mm_tlbwr_op, mm_pool32axf_op), 0 },
+	{ insn_wait, M(mm_pool32a_op, 0, 0, 0, mm_wait_op, mm_pool32axf_op), SCIMM },
 	{ insn_xor, M(mm_pool32a_op, 0, 0, 0, 0, mm_xor32_op), RT | RS | RD },
 	{ insn_xori, M(mm_xori32_op, 0, 0, 0, 0, 0), RT | RS | UIMM },
 	{ insn_dins, 0, 0 },
diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c
index 07ec837..343840a 100644
--- a/arch/mips/mm/uasm-mips.c
+++ b/arch/mips/mm/uasm-mips.c
@@ -113,6 +113,7 @@ static struct insn insn_table[] = {
 	{ insn_tlbr,  M(cop0_op, cop_op, 0, 0, 0, tlbr_op),  0 },
 	{ insn_tlbwi,  M(cop0_op, cop_op, 0, 0, 0, tlbwi_op),  0 },
 	{ insn_tlbwr,  M(cop0_op, cop_op, 0, 0, 0, tlbwr_op),  0 },
+	{ insn_wait, M(cop0_op, cop_op, 0, 0, 0, wait_op), SCIMM },
 	{ insn_xori,  M(xori_op, 0, 0, 0, 0, 0),  RS | RT | UIMM },
 	{ insn_xor,  M(spec_op, 0, 0, 0, 0, xor_op),  RS | RT | RD },
 	{ insn_invalid, 0, 0 }
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index e688800..1b42934 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -53,8 +53,8 @@ enum opcode {
 	insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0, insn_mtc0,
 	insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc, insn_scd,
 	insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw, insn_sync,
-	insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_xor,
-	insn_xori,
+	insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait,
+	insn_xor, insn_xori,
 };
 
 struct insn {
@@ -275,6 +275,7 @@ I_0(_tlbp)
 I_0(_tlbr)
 I_0(_tlbwi)
 I_0(_tlbwr)
+I_u1(_wait);
 I_u3u1u2(_xor)
 I_u2u1u3(_xori)
 I_u2u1msbu3(_dins);
-- 
1.8.4.2

[PATCH 04/10] MIPS: uasm: add wait instruction

[Paul Burton]

This patch allows use of the wait instruction from uasm. It will be used
by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h  | 1 +
 arch/mips/mm/uasm-micromips.c | 1 +
 arch/mips/mm/uasm-mips.c      | 1 +
 arch/mips/mm/uasm.c           | 5 +++--
 4 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 4f4ed49..7458a94 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -143,6 +143,7 @@ Ip_0(_tlbp);
 Ip_0(_tlbr);
 Ip_0(_tlbwi);
 Ip_0(_tlbwr);
+Ip_u1(_wait);
 Ip_u3u1u2(_xor);
 Ip_u2u1u3(_xori);
 
diff --git a/arch/mips/mm/uasm-micromips.c b/arch/mips/mm/uasm-micromips.c
index 15b46da..1150eda 100644
--- a/arch/mips/mm/uasm-micromips.c
+++ b/arch/mips/mm/uasm-micromips.c
@@ -105,6 +105,7 @@ static struct insn insn_table_MM[] = {
 	{ insn_tlbr, M(mm_pool32a_op, 0, 0, 0, mm_tlbr_op, mm_pool32axf_op), 0 },
 	{ insn_tlbwi, M(mm_pool32a_op, 0, 0, 0, mm_tlbwi_op, mm_pool32axf_op), 0 },
 	{ insn_tlbwr, M(mm_pool32a_op, 0, 0, 0, mm_tlbwr_op, mm_pool32axf_op), 0 },
+	{ insn_wait, M(mm_pool32a_op, 0, 0, 0, mm_wait_op, mm_pool32axf_op), SCIMM },
 	{ insn_xor, M(mm_pool32a_op, 0, 0, 0, 0, mm_xor32_op), RT | RS | RD },
 	{ insn_xori, M(mm_xori32_op, 0, 0, 0, 0, 0), RT | RS | UIMM },
 	{ insn_dins, 0, 0 },
diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c
index 07ec837..343840a 100644
--- a/arch/mips/mm/uasm-mips.c
+++ b/arch/mips/mm/uasm-mips.c
@@ -113,6 +113,7 @@ static struct insn insn_table[] = {
 	{ insn_tlbr,  M(cop0_op, cop_op, 0, 0, 0, tlbr_op),  0 },
 	{ insn_tlbwi,  M(cop0_op, cop_op, 0, 0, 0, tlbwi_op),  0 },
 	{ insn_tlbwr,  M(cop0_op, cop_op, 0, 0, 0, tlbwr_op),  0 },
+	{ insn_wait, M(cop0_op, cop_op, 0, 0, 0, wait_op), SCIMM },
 	{ insn_xori,  M(xori_op, 0, 0, 0, 0, 0),  RS | RT | UIMM },
 	{ insn_xor,  M(spec_op, 0, 0, 0, 0, xor_op),  RS | RT | RD },
 	{ insn_invalid, 0, 0 }
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index e688800..1b42934 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -53,8 +53,8 @@ enum opcode {
 	insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0, insn_mtc0,
 	insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc, insn_scd,
 	insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw, insn_sync,
-	insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_xor,
-	insn_xori,
+	insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait,
+	insn_xor, insn_xori,
 };
 
 struct insn {
@@ -275,6 +275,7 @@ I_0(_tlbp)
 I_0(_tlbr)
 I_0(_tlbwi)
 I_0(_tlbwr)
+I_u1(_wait);
 I_u3u1u2(_xor)
 I_u2u1u3(_xori)
 I_u2u1msbu3(_dins);
-- 
1.8.4.2

[PATCH 05/10] MIPS: uasm: add jr.hb instruction

[Paul Burton]

This patch allows use of the jr.hb instruction from uasm, in order to
clear instruction hazards. It will be used by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h      |  1 +
 arch/mips/include/uapi/asm/inst.h |  7 +++++++
 arch/mips/mm/uasm-micromips.c     |  1 +
 arch/mips/mm/uasm-mips.c          |  1 +
 arch/mips/mm/uasm.c               | 13 +++++++------
 5 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 7458a94..2d9cedd 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -115,6 +115,7 @@ Ip_u2u1msbu3(_ins);
 Ip_u1(_j);
 Ip_u1(_jal);
 Ip_u1(_jr);
+Ip_u1(_jr_hb);
 Ip_u2s3u1(_ld);
 Ip_u3u1u2(_ldx);
 Ip_u2s3u1(_ll);
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index 203f28b..e19d63e 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -196,6 +196,13 @@ enum lx_func {
 };
 
 /*
+ * hint/re field for jr instructions.
+ */
+enum jr_hint {
+	jr_hb_hint = 0x10,
+};
+
+/*
  * (microMIPS) Major opcodes.
  */
 enum mm_major_op {
diff --git a/arch/mips/mm/uasm-micromips.c b/arch/mips/mm/uasm-micromips.c
index 1150eda..19ebbcd 100644
--- a/arch/mips/mm/uasm-micromips.c
+++ b/arch/mips/mm/uasm-micromips.c
@@ -80,6 +80,7 @@ static struct insn insn_table_MM[] = {
 	{ insn_j, M(mm_j32_op, 0, 0, 0, 0, 0), JIMM },
 	{ insn_jal, M(mm_jal32_op, 0, 0, 0, 0, 0), JIMM },
 	{ insn_jr, M(mm_pool32a_op, 0, 0, 0, mm_jalr_op, mm_pool32axf_op), RS },
+	{ insn_jr_hb, M(mm_pool32a_op, 0, 0, 0, mm_jalrhb_op, mm_pool32axf_op), RS },
 	{ insn_ld, 0, 0 },
 	{ insn_ll, M(mm_pool32c_op, 0, 0, (mm_ll_func << 1), 0, 0), RS | RT | SIMM },
 	{ insn_lld, 0, 0 },
diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c
index 343840a..46ca072 100644
--- a/arch/mips/mm/uasm-mips.c
+++ b/arch/mips/mm/uasm-mips.c
@@ -85,6 +85,7 @@ static struct insn insn_table[] = {
 	{ insn_jal,  M(jal_op, 0, 0, 0, 0, 0),	JIMM },
 	{ insn_j,  M(j_op, 0, 0, 0, 0, 0),  JIMM },
 	{ insn_jr,  M(spec_op, 0, 0, 0, 0, jr_op),  RS },
+	{ insn_jr_hb,  M(spec_op, 0, 0, 0, jr_hb_hint, jr_op),  RS },
 	{ insn_ld,  M(ld_op, 0, 0, 0, 0, 0),  RS | RT | SIMM },
 	{ insn_ldx, M(spec3_op, 0, 0, 0, ldx_op, lx_op), RS | RT | RD },
 	{ insn_lld,  M(lld_op, 0, 0, 0, 0, 0),	RS | RT | SIMM },
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index 1b42934..f725309 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -49,12 +49,12 @@ enum opcode {
 	insn_bne, insn_cache, insn_daddiu, insn_daddu, insn_dins, insn_dinsm,
 	insn_dmfc0, insn_dmtc0, insn_drotr, insn_drotr32, insn_dsll,
 	insn_dsll32, insn_dsra, insn_dsrl, insn_dsrl32, insn_dsubu, insn_eret,
-	insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_ld, insn_ldx,
-	insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0, insn_mtc0,
-	insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc, insn_scd,
-	insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw, insn_sync,
-	insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait,
-	insn_xor, insn_xori,
+	insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_jr_hb, insn_ld,
+	insn_ldx, insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0,
+	insn_mtc0, insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc,
+	insn_scd, insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw,
+	insn_sync, insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr,
+	insn_wait, insn_xor, insn_xori,
 };
 
 struct insn {
@@ -251,6 +251,7 @@ I_u2u1msbu3(_ins)
 I_u1(_j)
 I_u1(_jal)
 I_u1(_jr)
+I_u1(_jr_hb)
 I_u2s3u1(_ld)
 I_u2s3u1(_ll)
 I_u2s3u1(_lld)
-- 
1.8.4.2

[PATCH 05/10] MIPS: uasm: add jr.hb instruction

[Paul Burton]

This patch allows use of the jr.hb instruction from uasm, in order to
clear instruction hazards. It will be used by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h      |  1 +
 arch/mips/include/uapi/asm/inst.h |  7 +++++++
 arch/mips/mm/uasm-micromips.c     |  1 +
 arch/mips/mm/uasm-mips.c          |  1 +
 arch/mips/mm/uasm.c               | 13 +++++++------
 5 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 7458a94..2d9cedd 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -115,6 +115,7 @@ Ip_u2u1msbu3(_ins);
 Ip_u1(_j);
 Ip_u1(_jal);
 Ip_u1(_jr);
+Ip_u1(_jr_hb);
 Ip_u2s3u1(_ld);
 Ip_u3u1u2(_ldx);
 Ip_u2s3u1(_ll);
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index 203f28b..e19d63e 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -196,6 +196,13 @@ enum lx_func {
 };
 
 /*
+ * hint/re field for jr instructions.
+ */
+enum jr_hint {
+	jr_hb_hint = 0x10,
+};
+
+/*
  * (microMIPS) Major opcodes.
  */
 enum mm_major_op {
diff --git a/arch/mips/mm/uasm-micromips.c b/arch/mips/mm/uasm-micromips.c
index 1150eda..19ebbcd 100644
--- a/arch/mips/mm/uasm-micromips.c
+++ b/arch/mips/mm/uasm-micromips.c
@@ -80,6 +80,7 @@ static struct insn insn_table_MM[] = {
 	{ insn_j, M(mm_j32_op, 0, 0, 0, 0, 0), JIMM },
 	{ insn_jal, M(mm_jal32_op, 0, 0, 0, 0, 0), JIMM },
 	{ insn_jr, M(mm_pool32a_op, 0, 0, 0, mm_jalr_op, mm_pool32axf_op), RS },
+	{ insn_jr_hb, M(mm_pool32a_op, 0, 0, 0, mm_jalrhb_op, mm_pool32axf_op), RS },
 	{ insn_ld, 0, 0 },
 	{ insn_ll, M(mm_pool32c_op, 0, 0, (mm_ll_func << 1), 0, 0), RS | RT | SIMM },
 	{ insn_lld, 0, 0 },
diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c
index 343840a..46ca072 100644
--- a/arch/mips/mm/uasm-mips.c
+++ b/arch/mips/mm/uasm-mips.c
@@ -85,6 +85,7 @@ static struct insn insn_table[] = {
 	{ insn_jal,  M(jal_op, 0, 0, 0, 0, 0),	JIMM },
 	{ insn_j,  M(j_op, 0, 0, 0, 0, 0),  JIMM },
 	{ insn_jr,  M(spec_op, 0, 0, 0, 0, jr_op),  RS },
+	{ insn_jr_hb,  M(spec_op, 0, 0, 0, jr_hb_hint, jr_op),  RS },
 	{ insn_ld,  M(ld_op, 0, 0, 0, 0, 0),  RS | RT | SIMM },
 	{ insn_ldx, M(spec3_op, 0, 0, 0, ldx_op, lx_op), RS | RT | RD },
 	{ insn_lld,  M(lld_op, 0, 0, 0, 0, 0),	RS | RT | SIMM },
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index 1b42934..f725309 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -49,12 +49,12 @@ enum opcode {
 	insn_bne, insn_cache, insn_daddiu, insn_daddu, insn_dins, insn_dinsm,
 	insn_dmfc0, insn_dmtc0, insn_drotr, insn_drotr32, insn_dsll,
 	insn_dsll32, insn_dsra, insn_dsrl, insn_dsrl32, insn_dsubu, insn_eret,
-	insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_ld, insn_ldx,
-	insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0, insn_mtc0,
-	insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc, insn_scd,
-	insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw, insn_sync,
-	insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait,
-	insn_xor, insn_xori,
+	insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_jr_hb, insn_ld,
+	insn_ldx, insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0,
+	insn_mtc0, insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc,
+	insn_scd, insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw,
+	insn_sync, insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr,
+	insn_wait, insn_xor, insn_xori,
 };
 
 struct insn {
@@ -251,6 +251,7 @@ I_u2u1msbu3(_ins)
 I_u1(_j)
 I_u1(_jal)
 I_u1(_jr)
+I_u1(_jr_hb)
 I_u2s3u1(_ld)
 I_u2s3u1(_ll)
 I_u2s3u1(_lld)
-- 
1.8.4.2

[PATCH 06/10] MIPS: uasm: add mftc0 & mttc0 instructions

[Paul Burton]

This patch allows use of mftc0 & mttc0 instructions from uasm, in order
to access the cop0 registers of another TC within the core. It will be
used by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h      |  2 ++
 arch/mips/include/uapi/asm/inst.h |  3 ++-
 arch/mips/mm/uasm-mips.c          |  2 ++
 arch/mips/mm/uasm.c               | 10 ++++++----
 4 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 2d9cedd..86c3535 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -124,7 +124,9 @@ Ip_u1s2(_lui);
 Ip_u2s3u1(_lw);
 Ip_u3u1u2(_lwx);
 Ip_u1u2u3(_mfc0);
+Ip_u2u1u3(_mftc0);
 Ip_u1u2u3(_mtc0);
+Ip_u1u2u3(_mttc0);
 Ip_u3u1u2(_or);
 Ip_u2u1u3(_ori);
 Ip_u2s3u1(_pref);
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index e19d63e..757f472 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -101,7 +101,8 @@ enum cop_op {
 	cfc_op	      = 0x02, mfhc_op	    = 0x03,
 	mtc_op        = 0x04, dmtc_op	    = 0x05,
 	ctc_op	      = 0x06, mthc_op	    = 0x07,
-	bc_op	      = 0x08, cop_op	    = 0x10,
+	bc_op	      = 0x08, mftr_op	    = 0x08,
+	mttr_op	      = 0x0c, cop_op	    = 0x10,
 	copm_op	      = 0x18
 };
 
diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c
index 46ca072..814265f 100644
--- a/arch/mips/mm/uasm-mips.c
+++ b/arch/mips/mm/uasm-mips.c
@@ -94,7 +94,9 @@ static struct insn insn_table[] = {
 	{ insn_lw,  M(lw_op, 0, 0, 0, 0, 0),  RS | RT | SIMM },
 	{ insn_lwx, M(spec3_op, 0, 0, 0, lwx_op, lx_op), RS | RT | RD },
 	{ insn_mfc0,  M(cop0_op, mfc_op, 0, 0, 0, 0),  RT | RD | SET},
+	{ insn_mftc0,  M(cop0_op, mftr_op, 0, 0, 0, 0),  RT | RD | SET},
 	{ insn_mtc0,  M(cop0_op, mtc_op, 0, 0, 0, 0),  RT | RD | SET},
+	{ insn_mttc0,  M(cop0_op, mttr_op, 0, 0, 0, 0),  RT | RD | SET},
 	{ insn_ori,  M(ori_op, 0, 0, 0, 0, 0),	RS | RT | UIMM },
 	{ insn_or,  M(spec_op, 0, 0, 0, 0, or_op),  RS | RT | RD },
 	{ insn_pref,  M(pref_op, 0, 0, 0, 0, 0),  RS | RT | SIMM },
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index f725309..99cb34b 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -51,10 +51,10 @@ enum opcode {
 	insn_dsll32, insn_dsra, insn_dsrl, insn_dsrl32, insn_dsubu, insn_eret,
 	insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_jr_hb, insn_ld,
 	insn_ldx, insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0,
-	insn_mtc0, insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc,
-	insn_scd, insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw,
-	insn_sync, insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr,
-	insn_wait, insn_xor, insn_xori,
+	insn_mftc0, insn_mtc0, insn_mttc0, insn_or, insn_ori, insn_pref,
+	insn_rfe, insn_rotr, insn_sc, insn_scd, insn_sd, insn_sll, insn_sra,
+	insn_srl, insn_subu, insn_sw, insn_sync, insn_syscall, insn_tlbp,
+	insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait, insn_xor, insn_xori,
 };
 
 struct insn {
@@ -258,7 +258,9 @@ I_u2s3u1(_lld)
 I_u1s2(_lui)
 I_u2s3u1(_lw)
 I_u1u2u3(_mfc0)
+I_u2u1u3(_mftc0)
 I_u1u2u3(_mtc0)
+I_u1u2u3(_mttc0)
 I_u2u1u3(_ori)
 I_u3u1u2(_or)
 I_0(_rfe)
-- 
1.8.4.2

[PATCH 06/10] MIPS: uasm: add mftc0 & mttc0 instructions

[Paul Burton]

This patch allows use of mftc0 & mttc0 instructions from uasm, in order
to access the cop0 registers of another TC within the core. It will be
used by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h      |  2 ++
 arch/mips/include/uapi/asm/inst.h |  3 ++-
 arch/mips/mm/uasm-mips.c          |  2 ++
 arch/mips/mm/uasm.c               | 10 ++++++----
 4 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 2d9cedd..86c3535 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -124,7 +124,9 @@ Ip_u1s2(_lui);
 Ip_u2s3u1(_lw);
 Ip_u3u1u2(_lwx);
 Ip_u1u2u3(_mfc0);
+Ip_u2u1u3(_mftc0);
 Ip_u1u2u3(_mtc0);
+Ip_u1u2u3(_mttc0);
 Ip_u3u1u2(_or);
 Ip_u2u1u3(_ori);
 Ip_u2s3u1(_pref);
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index e19d63e..757f472 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -101,7 +101,8 @@ enum cop_op {
 	cfc_op	      = 0x02, mfhc_op	    = 0x03,
 	mtc_op        = 0x04, dmtc_op	    = 0x05,
 	ctc_op	      = 0x06, mthc_op	    = 0x07,
-	bc_op	      = 0x08, cop_op	    = 0x10,
+	bc_op	      = 0x08, mftr_op	    = 0x08,
+	mttr_op	      = 0x0c, cop_op	    = 0x10,
 	copm_op	      = 0x18
 };
 
diff --git a/arch/mips/mm/uasm-mips.c b/arch/mips/mm/uasm-mips.c
index 46ca072..814265f 100644
--- a/arch/mips/mm/uasm-mips.c
+++ b/arch/mips/mm/uasm-mips.c
@@ -94,7 +94,9 @@ static struct insn insn_table[] = {
 	{ insn_lw,  M(lw_op, 0, 0, 0, 0, 0),  RS | RT | SIMM },
 	{ insn_lwx, M(spec3_op, 0, 0, 0, lwx_op, lx_op), RS | RT | RD },
 	{ insn_mfc0,  M(cop0_op, mfc_op, 0, 0, 0, 0),  RT | RD | SET},
+	{ insn_mftc0,  M(cop0_op, mftr_op, 0, 0, 0, 0),  RT | RD | SET},
 	{ insn_mtc0,  M(cop0_op, mtc_op, 0, 0, 0, 0),  RT | RD | SET},
+	{ insn_mttc0,  M(cop0_op, mttr_op, 0, 0, 0, 0),  RT | RD | SET},
 	{ insn_ori,  M(ori_op, 0, 0, 0, 0, 0),	RS | RT | UIMM },
 	{ insn_or,  M(spec_op, 0, 0, 0, 0, or_op),  RS | RT | RD },
 	{ insn_pref,  M(pref_op, 0, 0, 0, 0, 0),  RS | RT | SIMM },
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index f725309..99cb34b 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -51,10 +51,10 @@ enum opcode {
 	insn_dsll32, insn_dsra, insn_dsrl, insn_dsrl32, insn_dsubu, insn_eret,
 	insn_ext, insn_ins, insn_j, insn_jal, insn_jr, insn_jr_hb, insn_ld,
 	insn_ldx, insn_ll, insn_lld, insn_lui, insn_lw, insn_lwx, insn_mfc0,
-	insn_mtc0, insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sc,
-	insn_scd, insn_sd, insn_sll, insn_sra, insn_srl, insn_subu, insn_sw,
-	insn_sync, insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr,
-	insn_wait, insn_xor, insn_xori,
+	insn_mftc0, insn_mtc0, insn_mttc0, insn_or, insn_ori, insn_pref,
+	insn_rfe, insn_rotr, insn_sc, insn_scd, insn_sd, insn_sll, insn_sra,
+	insn_srl, insn_subu, insn_sw, insn_sync, insn_syscall, insn_tlbp,
+	insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait, insn_xor, insn_xori,
 };
 
 struct insn {
@@ -258,7 +258,9 @@ I_u2s3u1(_lld)
 I_u1s2(_lui)
 I_u2s3u1(_lw)
 I_u1u2u3(_mfc0)
+I_u2u1u3(_mftc0)
 I_u1u2u3(_mtc0)
+I_u1u2u3(_mttc0)
 I_u2u1u3(_ori)
 I_u3u1u2(_or)
 I_0(_rfe)
-- 
1.8.4.2

[PATCH 07/10] MIPS: uasm: add a label variant of beq

[Paul Burton]

This patch allows for use of the beq instruction with labels from uasm,
much as bne & others already do. It will be used by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h | 2 ++
 arch/mips/mm/uasm.c          | 8 ++++++++
 2 files changed, 10 insertions(+)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 86c3535..53bb110 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -269,6 +269,8 @@ void uasm_il_bbit0(u32 **p, struct uasm_reloc **r, unsigned int reg,
 		   unsigned int bit, int lid);
 void uasm_il_bbit1(u32 **p, struct uasm_reloc **r, unsigned int reg,
 		   unsigned int bit, int lid);
+void uasm_il_beq(u32 **p, struct uasm_reloc **r, unsigned int reg1,
+		 unsigned int reg2, int lid);
 void uasm_il_beqz(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid);
 void uasm_il_beqzl(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid);
 void uasm_il_bgezl(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid);
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index 99cb34b..15fc233 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -474,6 +474,14 @@ void ISAFUNC(uasm_il_b)(u32 **p, struct uasm_reloc **r, int lid)
 }
 UASM_EXPORT_SYMBOL(ISAFUNC(uasm_il_b));
 
+void ISAFUNC(uasm_il_beq)(u32 **p, struct uasm_reloc **r, unsigned int reg1,
+			  unsigned int reg2, int lid)
+{
+	uasm_r_mips_pc16(r, *p, lid);
+	ISAFUNC(uasm_i_beq)(p, reg1, reg2, 0);
+}
+UASM_EXPORT_SYMBOL(ISAFUNC(uasm_il_beq));
+
 void ISAFUNC(uasm_il_beqz)(u32 **p, struct uasm_reloc **r, unsigned int reg,
 			   int lid)
 {
-- 
1.8.4.2

[PATCH 07/10] MIPS: uasm: add a label variant of beq

[Paul Burton]

This patch allows for use of the beq instruction with labels from uasm,
much as bne & others already do. It will be used by a subsequent patch.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---
 arch/mips/include/asm/uasm.h | 2 ++
 arch/mips/mm/uasm.c          | 8 ++++++++
 2 files changed, 10 insertions(+)

diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 86c3535..53bb110 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -269,6 +269,8 @@ void uasm_il_bbit0(u32 **p, struct uasm_reloc **r, unsigned int reg,
 		   unsigned int bit, int lid);
 void uasm_il_bbit1(u32 **p, struct uasm_reloc **r, unsigned int reg,
 		   unsigned int bit, int lid);
+void uasm_il_beq(u32 **p, struct uasm_reloc **r, unsigned int reg1,
+		 unsigned int reg2, int lid);
 void uasm_il_beqz(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid);
 void uasm_il_beqzl(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid);
 void uasm_il_bgezl(u32 **p, struct uasm_reloc **r, unsigned int reg, int lid);
diff --git a/arch/mips/mm/uasm.c b/arch/mips/mm/uasm.c
index 99cb34b..15fc233 100644
--- a/arch/mips/mm/uasm.c
+++ b/arch/mips/mm/uasm.c
@@ -474,6 +474,14 @@ void ISAFUNC(uasm_il_b)(u32 **p, struct uasm_reloc **r, int lid)
 }
 UASM_EXPORT_SYMBOL(ISAFUNC(uasm_il_b));
 
+void ISAFUNC(uasm_il_beq)(u32 **p, struct uasm_reloc **r, unsigned int reg1,
+			  unsigned int reg2, int lid)
+{
+	uasm_r_mips_pc16(r, *p, lid);
+	ISAFUNC(uasm_i_beq)(p, reg1, reg2, 0);
+}
+UASM_EXPORT_SYMBOL(ISAFUNC(uasm_il_beq));
+
 void ISAFUNC(uasm_il_beqz)(u32 **p, struct uasm_reloc **r, unsigned int reg,
 			   int lid)
 {
-- 
1.8.4.2

[PATCH 08/10] MIPS: support use of cpuidle

[Paul Burton]

This patch lays the groundwork for MIPS platforms to make use of the
cpuidle framework. The arch_cpu_idle function simply calls cpuidle &
falls back to the regular cpu_wait path if cpuidle should fail (eg. if
it's not selected or no driver is registered). A generic cpuidle state
for the wait instruction is introduced, intended to ease use of the wait
instruction from cpuidle drivers and reduce code duplication.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: linux-pm@vger.kernel.org
---
 arch/mips/Kconfig            |  2 ++
 arch/mips/include/asm/idle.h | 14 ++++++++++++++
 arch/mips/kernel/idle.c      | 20 +++++++++++++++++++-
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 5bc27c0..95f2f11 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1838,6 +1838,8 @@ config CPU_R4K_CACHE_TLB
 	bool
 	default y if !(CPU_R3000 || CPU_R8000 || CPU_SB1 || CPU_TX39XX || CPU_CAVIUM_OCTEON)
 
+source "drivers/cpuidle/Kconfig"
+
 choice
 	prompt "MIPS MT options"
 
diff --git a/arch/mips/include/asm/idle.h b/arch/mips/include/asm/idle.h
index d192158..d9f932d 100644
--- a/arch/mips/include/asm/idle.h
+++ b/arch/mips/include/asm/idle.h
@@ -1,6 +1,7 @@
 #ifndef __ASM_IDLE_H
 #define __ASM_IDLE_H
 
+#include <linux/cpuidle.h>
 #include <linux/linkage.h>
 
 extern void (*cpu_wait)(void);
@@ -20,4 +21,17 @@ static inline int address_is_in_r4k_wait_irqoff(unsigned long addr)
 	       addr < (unsigned long)__pastwait;
 }
 
+extern int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
+				   struct cpuidle_driver *drv, int index);
+
+#define MIPS_CPUIDLE_WAIT_STATE {\
+	.enter			= mips_cpuidle_wait_enter,\
+	.exit_latency		= 1,\
+	.target_residency	= 1,\
+	.power_usage		= UINT_MAX,\
+	.flags			= CPUIDLE_FLAG_TIME_VALID,\
+	.name			= "wait",\
+	.desc			= "MIPS wait",\
+}
+
 #endif /* __ASM_IDLE_H  */
diff --git a/arch/mips/kernel/idle.c b/arch/mips/kernel/idle.c
index 3553243..64e91e4 100644
--- a/arch/mips/kernel/idle.c
+++ b/arch/mips/kernel/idle.c
@@ -11,6 +11,7 @@
  * as published by the Free Software Foundation; either version
  * 2 of the License, or (at your option) any later version.
  */
+#include <linux/cpuidle.h>
 #include <linux/export.h>
 #include <linux/init.h>
 #include <linux/irqflags.h>
@@ -239,7 +240,7 @@ static void smtc_idle_hook(void)
 #endif
 }
 
-void arch_cpu_idle(void)
+static void mips_cpu_idle(void)
 {
 	smtc_idle_hook();
 	if (cpu_wait)
@@ -247,3 +248,20 @@ void arch_cpu_idle(void)
 	else
 		local_irq_enable();
 }
+
+void arch_cpu_idle(void)
+{
+	if (cpuidle_idle_call())
+		mips_cpu_idle();
+}
+
+#ifdef CONFIG_CPU_IDLE
+
+int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
+			    struct cpuidle_driver *drv, int index)
+{
+	mips_cpu_idle();
+	return index;
+}
+
+#endif
-- 
1.8.4.2

[PATCH 08/10] MIPS: support use of cpuidle

[Paul Burton]

This patch lays the groundwork for MIPS platforms to make use of the
cpuidle framework. The arch_cpu_idle function simply calls cpuidle &
falls back to the regular cpu_wait path if cpuidle should fail (eg. if
it's not selected or no driver is registered). A generic cpuidle state
for the wait instruction is introduced, intended to ease use of the wait
instruction from cpuidle drivers and reduce code duplication.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: linux-pm@vger.kernel.org
---
 arch/mips/Kconfig            |  2 ++
 arch/mips/include/asm/idle.h | 14 ++++++++++++++
 arch/mips/kernel/idle.c      | 20 +++++++++++++++++++-
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 5bc27c0..95f2f11 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1838,6 +1838,8 @@ config CPU_R4K_CACHE_TLB
 	bool
 	default y if !(CPU_R3000 || CPU_R8000 || CPU_SB1 || CPU_TX39XX || CPU_CAVIUM_OCTEON)
 
+source "drivers/cpuidle/Kconfig"
+
 choice
 	prompt "MIPS MT options"
 
diff --git a/arch/mips/include/asm/idle.h b/arch/mips/include/asm/idle.h
index d192158..d9f932d 100644
--- a/arch/mips/include/asm/idle.h
+++ b/arch/mips/include/asm/idle.h
@@ -1,6 +1,7 @@
 #ifndef __ASM_IDLE_H
 #define __ASM_IDLE_H
 
+#include <linux/cpuidle.h>
 #include <linux/linkage.h>
 
 extern void (*cpu_wait)(void);
@@ -20,4 +21,17 @@ static inline int address_is_in_r4k_wait_irqoff(unsigned long addr)
 	       addr < (unsigned long)__pastwait;
 }
 
+extern int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
+				   struct cpuidle_driver *drv, int index);
+
+#define MIPS_CPUIDLE_WAIT_STATE {\
+	.enter			= mips_cpuidle_wait_enter,\
+	.exit_latency		= 1,\
+	.target_residency	= 1,\
+	.power_usage		= UINT_MAX,\
+	.flags			= CPUIDLE_FLAG_TIME_VALID,\
+	.name			= "wait",\
+	.desc			= "MIPS wait",\
+}
+
 #endif /* __ASM_IDLE_H  */
diff --git a/arch/mips/kernel/idle.c b/arch/mips/kernel/idle.c
index 3553243..64e91e4 100644
--- a/arch/mips/kernel/idle.c
+++ b/arch/mips/kernel/idle.c
@@ -11,6 +11,7 @@
  * as published by the Free Software Foundation; either version
  * 2 of the License, or (at your option) any later version.
  */
+#include <linux/cpuidle.h>
 #include <linux/export.h>
 #include <linux/init.h>
 #include <linux/irqflags.h>
@@ -239,7 +240,7 @@ static void smtc_idle_hook(void)
 #endif
 }
 
-void arch_cpu_idle(void)
+static void mips_cpu_idle(void)
 {
 	smtc_idle_hook();
 	if (cpu_wait)
@@ -247,3 +248,20 @@ void arch_cpu_idle(void)
 	else
 		local_irq_enable();
 }
+
+void arch_cpu_idle(void)
+{
+	if (cpuidle_idle_call())
+		mips_cpu_idle();
+}
+
+#ifdef CONFIG_CPU_IDLE
+
+int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
+			    struct cpuidle_driver *drv, int index)
+{
+	mips_cpu_idle();
+	return index;
+}
+
+#endif
-- 
1.8.4.2

Re: [PATCH 08/10] MIPS: support use of cpuidle

[Daniel Lezcano]

On 01/15/2014 02:55 PM, Paul Burton wrote:
> This patch lays the groundwork for MIPS platforms to make use of the
> cpuidle framework. The arch_cpu_idle function simply calls cpuidle &
> falls back to the regular cpu_wait path if cpuidle should fail (eg. if
> it's not selected or no driver is registered). A generic cpuidle state
> for the wait instruction is introduced, intended to ease use of the wait
> instruction from cpuidle drivers and reduce code duplication.

Hi,

What is the status of this patchset ? Still need comments ?

Thanks
   -- Daniel

>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> Cc: linux-pm@vger.kernel.org
> ---
>   arch/mips/Kconfig            |  2 ++
>   arch/mips/include/asm/idle.h | 14 ++++++++++++++
>   arch/mips/kernel/idle.c      | 20 +++++++++++++++++++-
>   3 files changed, 35 insertions(+), 1 deletion(-)
>
> diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
> index 5bc27c0..95f2f11 100644
> --- a/arch/mips/Kconfig
> +++ b/arch/mips/Kconfig
> @@ -1838,6 +1838,8 @@ config CPU_R4K_CACHE_TLB
>   	bool
>   	default y if !(CPU_R3000 || CPU_R8000 || CPU_SB1 || CPU_TX39XX || CPU_CAVIUM_OCTEON)
>
> +source "drivers/cpuidle/Kconfig"
> +
>   choice
>   	prompt "MIPS MT options"
>
> diff --git a/arch/mips/include/asm/idle.h b/arch/mips/include/asm/idle.h
> index d192158..d9f932d 100644
> --- a/arch/mips/include/asm/idle.h
> +++ b/arch/mips/include/asm/idle.h
> @@ -1,6 +1,7 @@
>   #ifndef __ASM_IDLE_H
>   #define __ASM_IDLE_H
>
> +#include <linux/cpuidle.h>
>   #include <linux/linkage.h>
>
>   extern void (*cpu_wait)(void);
> @@ -20,4 +21,17 @@ static inline int address_is_in_r4k_wait_irqoff(unsigned long addr)
>   	       addr < (unsigned long)__pastwait;
>   }
>
> +extern int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
> +				   struct cpuidle_driver *drv, int index);
> +
> +#define MIPS_CPUIDLE_WAIT_STATE {\
> +	.enter			= mips_cpuidle_wait_enter,\
> +	.exit_latency		= 1,\
> +	.target_residency	= 1,\
> +	.power_usage		= UINT_MAX,\
> +	.flags			= CPUIDLE_FLAG_TIME_VALID,\
> +	.name			= "wait",\
> +	.desc			= "MIPS wait",\
> +}
> +
>   #endif /* __ASM_IDLE_H  */
> diff --git a/arch/mips/kernel/idle.c b/arch/mips/kernel/idle.c
> index 3553243..64e91e4 100644
> --- a/arch/mips/kernel/idle.c
> +++ b/arch/mips/kernel/idle.c
> @@ -11,6 +11,7 @@
>    * as published by the Free Software Foundation; either version
>    * 2 of the License, or (at your option) any later version.
>    */
> +#include <linux/cpuidle.h>
>   #include <linux/export.h>
>   #include <linux/init.h>
>   #include <linux/irqflags.h>
> @@ -239,7 +240,7 @@ static void smtc_idle_hook(void)
>   #endif
>   }
>
> -void arch_cpu_idle(void)
> +static void mips_cpu_idle(void)
>   {
>   	smtc_idle_hook();
>   	if (cpu_wait)
> @@ -247,3 +248,20 @@ void arch_cpu_idle(void)
>   	else
>   		local_irq_enable();
>   }
> +
> +void arch_cpu_idle(void)
> +{
> +	if (cpuidle_idle_call())
> +		mips_cpu_idle();
> +}
> +
> +#ifdef CONFIG_CPU_IDLE
> +
> +int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
> +			    struct cpuidle_driver *drv, int index)
> +{
> +	mips_cpu_idle();
> +	return index;
> +}
> +
> +#endif
>


-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: [PATCH 08/10] MIPS: support use of cpuidle

[Paul Burton]

On Thu, Feb 20, 2014 at 09:52:26AM +0100, Daniel Lezcano wrote:
> On 01/15/2014 02:55 PM, Paul Burton wrote:
> >This patch lays the groundwork for MIPS platforms to make use of the
> >cpuidle framework. The arch_cpu_idle function simply calls cpuidle &
> >falls back to the regular cpu_wait path if cpuidle should fail (eg. if
> >it's not selected or no driver is registered). A generic cpuidle state
> >for the wait instruction is introduced, intended to ease use of the wait
> >instruction from cpuidle drivers and reduce code duplication.
> 
> Hi,
> 
> What is the status of this patchset ? Still need comments ?
> 
> Thanks
>   -- Daniel
> 

It's present in Ralf's current mips-for-linux-next branch, and in
linux-next from next-20140210 up to todays next-20140220. So I presume
barring any problems it should make it into 3.15.

Thanks,
    Paul

> >
> >Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> >Cc: linux-pm@vger.kernel.org
> >---
> >  arch/mips/Kconfig            |  2 ++
> >  arch/mips/include/asm/idle.h | 14 ++++++++++++++
> >  arch/mips/kernel/idle.c      | 20 +++++++++++++++++++-
> >  3 files changed, 35 insertions(+), 1 deletion(-)
> >
> >diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
> >index 5bc27c0..95f2f11 100644
> >--- a/arch/mips/Kconfig
> >+++ b/arch/mips/Kconfig
> >@@ -1838,6 +1838,8 @@ config CPU_R4K_CACHE_TLB
> >  	bool
> >  	default y if !(CPU_R3000 || CPU_R8000 || CPU_SB1 || CPU_TX39XX || CPU_CAVIUM_OCTEON)
> >
> >+source "drivers/cpuidle/Kconfig"
> >+
> >  choice
> >  	prompt "MIPS MT options"
> >
> >diff --git a/arch/mips/include/asm/idle.h b/arch/mips/include/asm/idle.h
> >index d192158..d9f932d 100644
> >--- a/arch/mips/include/asm/idle.h
> >+++ b/arch/mips/include/asm/idle.h
> >@@ -1,6 +1,7 @@
> >  #ifndef __ASM_IDLE_H
> >  #define __ASM_IDLE_H
> >
> >+#include <linux/cpuidle.h>
> >  #include <linux/linkage.h>
> >
> >  extern void (*cpu_wait)(void);
> >@@ -20,4 +21,17 @@ static inline int address_is_in_r4k_wait_irqoff(unsigned long addr)
> >  	       addr < (unsigned long)__pastwait;
> >  }
> >
> >+extern int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
> >+				   struct cpuidle_driver *drv, int index);
> >+
> >+#define MIPS_CPUIDLE_WAIT_STATE {\
> >+	.enter			= mips_cpuidle_wait_enter,\
> >+	.exit_latency		= 1,\
> >+	.target_residency	= 1,\
> >+	.power_usage		= UINT_MAX,\
> >+	.flags			= CPUIDLE_FLAG_TIME_VALID,\
> >+	.name			= "wait",\
> >+	.desc			= "MIPS wait",\
> >+}
> >+
> >  #endif /* __ASM_IDLE_H  */
> >diff --git a/arch/mips/kernel/idle.c b/arch/mips/kernel/idle.c
> >index 3553243..64e91e4 100644
> >--- a/arch/mips/kernel/idle.c
> >+++ b/arch/mips/kernel/idle.c
> >@@ -11,6 +11,7 @@
> >   * as published by the Free Software Foundation; either version
> >   * 2 of the License, or (at your option) any later version.
> >   */
> >+#include <linux/cpuidle.h>
> >  #include <linux/export.h>
> >  #include <linux/init.h>
> >  #include <linux/irqflags.h>
> >@@ -239,7 +240,7 @@ static void smtc_idle_hook(void)
> >  #endif
> >  }
> >
> >-void arch_cpu_idle(void)
> >+static void mips_cpu_idle(void)
> >  {
> >  	smtc_idle_hook();
> >  	if (cpu_wait)
> >@@ -247,3 +248,20 @@ void arch_cpu_idle(void)
> >  	else
> >  		local_irq_enable();
> >  }
> >+
> >+void arch_cpu_idle(void)
> >+{
> >+	if (cpuidle_idle_call())
> >+		mips_cpu_idle();
> >+}
> >+
> >+#ifdef CONFIG_CPU_IDLE
> >+
> >+int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
> >+			    struct cpuidle_driver *drv, int index)
> >+{
> >+	mips_cpu_idle();
> >+	return index;
> >+}
> >+
> >+#endif
> >
> 
> 
> -- 
>  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
> 
> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
> <http://twitter.com/#!/linaroorg> Twitter |
> <http://www.linaro.org/linaro-blog/> Blog
> 

Re: [PATCH 08/10] MIPS: support use of cpuidle

[Paul Burton]

On Thu, Feb 20, 2014 at 09:52:26AM +0100, Daniel Lezcano wrote:
> On 01/15/2014 02:55 PM, Paul Burton wrote:
> >This patch lays the groundwork for MIPS platforms to make use of the
> >cpuidle framework. The arch_cpu_idle function simply calls cpuidle &
> >falls back to the regular cpu_wait path if cpuidle should fail (eg. if
> >it's not selected or no driver is registered). A generic cpuidle state
> >for the wait instruction is introduced, intended to ease use of the wait
> >instruction from cpuidle drivers and reduce code duplication.
> 
> Hi,
> 
> What is the status of this patchset ? Still need comments ?
> 
> Thanks
>   -- Daniel
> 

It's present in Ralf's current mips-for-linux-next branch, and in
linux-next from next-20140210 up to todays next-20140220. So I presume
barring any problems it should make it into 3.15.

Thanks,
    Paul

> >
> >Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> >Cc: linux-pm@vger.kernel.org
> >---
> >  arch/mips/Kconfig            |  2 ++
> >  arch/mips/include/asm/idle.h | 14 ++++++++++++++
> >  arch/mips/kernel/idle.c      | 20 +++++++++++++++++++-
> >  3 files changed, 35 insertions(+), 1 deletion(-)
> >
> >diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
> >index 5bc27c0..95f2f11 100644
> >--- a/arch/mips/Kconfig
> >+++ b/arch/mips/Kconfig
> >@@ -1838,6 +1838,8 @@ config CPU_R4K_CACHE_TLB
> >  	bool
> >  	default y if !(CPU_R3000 || CPU_R8000 || CPU_SB1 || CPU_TX39XX || CPU_CAVIUM_OCTEON)
> >
> >+source "drivers/cpuidle/Kconfig"
> >+
> >  choice
> >  	prompt "MIPS MT options"
> >
> >diff --git a/arch/mips/include/asm/idle.h b/arch/mips/include/asm/idle.h
> >index d192158..d9f932d 100644
> >--- a/arch/mips/include/asm/idle.h
> >+++ b/arch/mips/include/asm/idle.h
> >@@ -1,6 +1,7 @@
> >  #ifndef __ASM_IDLE_H
> >  #define __ASM_IDLE_H
> >
> >+#include <linux/cpuidle.h>
> >  #include <linux/linkage.h>
> >
> >  extern void (*cpu_wait)(void);
> >@@ -20,4 +21,17 @@ static inline int address_is_in_r4k_wait_irqoff(unsigned long addr)
> >  	       addr < (unsigned long)__pastwait;
> >  }
> >
> >+extern int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
> >+				   struct cpuidle_driver *drv, int index);
> >+
> >+#define MIPS_CPUIDLE_WAIT_STATE {\
> >+	.enter			= mips_cpuidle_wait_enter,\
> >+	.exit_latency		= 1,\
> >+	.target_residency	= 1,\
> >+	.power_usage		= UINT_MAX,\
> >+	.flags			= CPUIDLE_FLAG_TIME_VALID,\
> >+	.name			= "wait",\
> >+	.desc			= "MIPS wait",\
> >+}
> >+
> >  #endif /* __ASM_IDLE_H  */
> >diff --git a/arch/mips/kernel/idle.c b/arch/mips/kernel/idle.c
> >index 3553243..64e91e4 100644
> >--- a/arch/mips/kernel/idle.c
> >+++ b/arch/mips/kernel/idle.c
> >@@ -11,6 +11,7 @@
> >   * as published by the Free Software Foundation; either version
> >   * 2 of the License, or (at your option) any later version.
> >   */
> >+#include <linux/cpuidle.h>
> >  #include <linux/export.h>
> >  #include <linux/init.h>
> >  #include <linux/irqflags.h>
> >@@ -239,7 +240,7 @@ static void smtc_idle_hook(void)
> >  #endif
> >  }
> >
> >-void arch_cpu_idle(void)
> >+static void mips_cpu_idle(void)
> >  {
> >  	smtc_idle_hook();
> >  	if (cpu_wait)
> >@@ -247,3 +248,20 @@ void arch_cpu_idle(void)
> >  	else
> >  		local_irq_enable();
> >  }
> >+
> >+void arch_cpu_idle(void)
> >+{
> >+	if (cpuidle_idle_call())
> >+		mips_cpu_idle();
> >+}
> >+
> >+#ifdef CONFIG_CPU_IDLE
> >+
> >+int mips_cpuidle_wait_enter(struct cpuidle_device *dev,
> >+			    struct cpuidle_driver *drv, int index)
> >+{
> >+	mips_cpu_idle();
> >+	return index;
> >+}
> >+
> >+#endif
> >
> 
> 
> -- 
>  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
> 
> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
> <http://twitter.com/#!/linaroorg> Twitter |
> <http://www.linaro.org/linaro-blog/> Blog
> 

[PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Paul Burton]

Declaring this allows drivers which need to initialise each struct
cpuidle_device at initialisation time to make use of the structures
already defined in cpuidle.c, rather than having to wastefully define
their own.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: linux-pm@vger.kernel.org
---
 include/linux/cpuidle.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
index 50fcbb0..bab4f33 100644
--- a/include/linux/cpuidle.h
+++ b/include/linux/cpuidle.h
@@ -84,6 +84,7 @@ struct cpuidle_device {
 };
 
 DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
+DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
 
 /**
  * cpuidle_get_last_residency - retrieves the last state's residency time
-- 
1.8.4.2

[PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Paul Burton]

Declaring this allows drivers which need to initialise each struct
cpuidle_device at initialisation time to make use of the structures
already defined in cpuidle.c, rather than having to wastefully define
their own.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: linux-pm@vger.kernel.org
---
 include/linux/cpuidle.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
index 50fcbb0..bab4f33 100644
--- a/include/linux/cpuidle.h
+++ b/include/linux/cpuidle.h
@@ -84,6 +84,7 @@ struct cpuidle_device {
 };
 
 DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
+DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
 
 /**
  * cpuidle_get_last_residency - retrieves the last state's residency time
-- 
1.8.4.2

Re: [PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Daniel Lezcano]

On 01/15/2014 02:55 PM, Paul Burton wrote:
> Declaring this allows drivers which need to initialise each struct
> cpuidle_device at initialisation time to make use of the structures
> already defined in cpuidle.c, rather than having to wastefully define
> their own.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> Cc: linux-pm@vger.kernel.org
> ---
>   include/linux/cpuidle.h | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
> index 50fcbb0..bab4f33 100644
> --- a/include/linux/cpuidle.h
> +++ b/include/linux/cpuidle.h
> @@ -84,6 +84,7 @@ struct cpuidle_device {
>   };
>
>   DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
> +DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);


Nak. When a device is registered, it is assigned to the cpuidle_devices 
pointer and the backend driver should use it.



-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: [PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Paul Burton]

On Thu, Feb 20, 2014 at 02:35:18PM +0100, Daniel Lezcano wrote:
> On 01/15/2014 02:55 PM, Paul Burton wrote:
> >Declaring this allows drivers which need to initialise each struct
> >cpuidle_device at initialisation time to make use of the structures
> >already defined in cpuidle.c, rather than having to wastefully define
> >their own.
> >
> >Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> >Cc: linux-pm@vger.kernel.org
> >---
> >  include/linux/cpuidle.h | 1 +
> >  1 file changed, 1 insertion(+)
> >
> >diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
> >index 50fcbb0..bab4f33 100644
> >--- a/include/linux/cpuidle.h
> >+++ b/include/linux/cpuidle.h
> >@@ -84,6 +84,7 @@ struct cpuidle_device {
> >  };
> >
> >  DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
> >+DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
> 
> 
> Nak. When a device is registered, it is assigned to the cpuidle_devices
> pointer and the backend driver should use it.
> 

Yes, but then if the driver needs to initialise the coupled_cpus mask
then it cannot do so until after the device has been registered. During
registration the cpuidle_coupled_register_device will then see the empty
coupled_cpus mask & do nothing. The only other ways around this would be
for the driver to define its own per-cpu struct cpuidle_device (which as
I state in the commit message seems wasteful when cpuidle already
defined them), or for cpuidle_coupled_register_device to be called later
after the driver had a chance to modify devices via the cpuidle_devices
pointers.

Paul

> 
> 
> -- 
>  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
> 
> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
> <http://twitter.com/#!/linaroorg> Twitter |
> <http://www.linaro.org/linaro-blog/> Blog
> 

Re: [PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Paul Burton]

On Thu, Feb 20, 2014 at 02:35:18PM +0100, Daniel Lezcano wrote:
> On 01/15/2014 02:55 PM, Paul Burton wrote:
> >Declaring this allows drivers which need to initialise each struct
> >cpuidle_device at initialisation time to make use of the structures
> >already defined in cpuidle.c, rather than having to wastefully define
> >their own.
> >
> >Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> >Cc: linux-pm@vger.kernel.org
> >---
> >  include/linux/cpuidle.h | 1 +
> >  1 file changed, 1 insertion(+)
> >
> >diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
> >index 50fcbb0..bab4f33 100644
> >--- a/include/linux/cpuidle.h
> >+++ b/include/linux/cpuidle.h
> >@@ -84,6 +84,7 @@ struct cpuidle_device {
> >  };
> >
> >  DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
> >+DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
> 
> 
> Nak. When a device is registered, it is assigned to the cpuidle_devices
> pointer and the backend driver should use it.
> 

Yes, but then if the driver needs to initialise the coupled_cpus mask
then it cannot do so until after the device has been registered. During
registration the cpuidle_coupled_register_device will then see the empty
coupled_cpus mask & do nothing. The only other ways around this would be
for the driver to define its own per-cpu struct cpuidle_device (which as
I state in the commit message seems wasteful when cpuidle already
defined them), or for cpuidle_coupled_register_device to be called later
after the driver had a chance to modify devices via the cpuidle_devices
pointers.

Paul

> 
> 
> -- 
>  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
> 
> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
> <http://twitter.com/#!/linaroorg> Twitter |
> <http://www.linaro.org/linaro-blog/> Blog
> 

Re: [PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Daniel Lezcano]

On 02/20/2014 02:41 PM, Paul Burton wrote:
> On Thu, Feb 20, 2014 at 02:35:18PM +0100, Daniel Lezcano wrote:
>> On 01/15/2014 02:55 PM, Paul Burton wrote:
>>> Declaring this allows drivers which need to initialise each struct
>>> cpuidle_device at initialisation time to make use of the structures
>>> already defined in cpuidle.c, rather than having to wastefully define
>>> their own.
>>>
>>> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>> Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
>>> Cc: linux-pm@vger.kernel.org
>>> ---
>>>   include/linux/cpuidle.h | 1 +
>>>   1 file changed, 1 insertion(+)
>>>
>>> diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
>>> index 50fcbb0..bab4f33 100644
>>> --- a/include/linux/cpuidle.h
>>> +++ b/include/linux/cpuidle.h
>>> @@ -84,6 +84,7 @@ struct cpuidle_device {
>>>   };
>>>
>>>   DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
>>> +DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
>>
>>
>> Nak. When a device is registered, it is assigned to the cpuidle_devices
>> pointer and the backend driver should use it.
>>
>
> Yes, but then if the driver needs to initialise the coupled_cpus mask
> then it cannot do so until after the device has been registered. During
> registration the cpuidle_coupled_register_device will then see the empty
> coupled_cpus mask & do nothing. The only other ways around this would be
> for the driver to define its own per-cpu struct cpuidle_device (which as
> I state in the commit message seems wasteful when cpuidle already
> defined them), or for cpuidle_coupled_register_device to be called later
> after the driver had a chance to modify devices via the cpuidle_devices
> pointers.

Yeah. I understand why you wanted to declare these cpu variables.

The mips cpuidle driver sounds like a bit particular. I believe I need 
some clarification on the behavior of the hardware to understand 
correctly the driver. Could you explain how the couples act vs the cpu ? 
And why cpu_sibling is used instead of cpu_possible_mask ?



>> --
>>   <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
>>
>> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
>> <http://twitter.com/#!/linaroorg> Twitter |
>> <http://www.linaro.org/linaro-blog/> Blog
>>
>


-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: [PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Paul Burton]

On Thu, Feb 20, 2014 at 02:53:06PM +0100, Daniel Lezcano wrote:
> On 02/20/2014 02:41 PM, Paul Burton wrote:
> >On Thu, Feb 20, 2014 at 02:35:18PM +0100, Daniel Lezcano wrote:
> >>On 01/15/2014 02:55 PM, Paul Burton wrote:
> >>>Declaring this allows drivers which need to initialise each struct
> >>>cpuidle_device at initialisation time to make use of the structures
> >>>already defined in cpuidle.c, rather than having to wastefully define
> >>>their own.
> >>>
> >>>Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >>>Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >>>Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> >>>Cc: linux-pm@vger.kernel.org
> >>>---
> >>>  include/linux/cpuidle.h | 1 +
> >>>  1 file changed, 1 insertion(+)
> >>>
> >>>diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
> >>>index 50fcbb0..bab4f33 100644
> >>>--- a/include/linux/cpuidle.h
> >>>+++ b/include/linux/cpuidle.h
> >>>@@ -84,6 +84,7 @@ struct cpuidle_device {
> >>>  };
> >>>
> >>>  DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
> >>>+DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
> >>
> >>
> >>Nak. When a device is registered, it is assigned to the cpuidle_devices
> >>pointer and the backend driver should use it.
> >>
> >
> >Yes, but then if the driver needs to initialise the coupled_cpus mask
> >then it cannot do so until after the device has been registered. During
> >registration the cpuidle_coupled_register_device will then see the empty
> >coupled_cpus mask & do nothing. The only other ways around this would be
> >for the driver to define its own per-cpu struct cpuidle_device (which as
> >I state in the commit message seems wasteful when cpuidle already
> >defined them), or for cpuidle_coupled_register_device to be called later
> >after the driver had a chance to modify devices via the cpuidle_devices
> >pointers.
> 
> Yeah. I understand why you wanted to declare these cpu variables.
> 
> The mips cpuidle driver sounds like a bit particular. I believe I need some
> clarification on the behavior of the hardware to understand correctly the
> driver. Could you explain how the couples act vs the cpu ? And why
> cpu_sibling is used instead of cpu_possible_mask ?
> 
> 

Sure. The CPUs that are coupled are actually VPEs (Virtual Processor
Elements) within a single core. They share the compute resource (ALUs
etc) of the core but have their own register state, ie. they're a form
of simultaneous multithreading.

Coherence within a MIPS Coherent Processing System is a property of the
core rather than of an individual VPE (since the VPEs within a core
share the same L1 caches). So for idle states which are non-coherent the
VPEs within the core are coupled. That covers all idle states beyond a
simple "wait" instruction - clock gating or powering down a core
requires it to become non-coherent first.

cpu_sibling_mask is already setup to indicate which CPUs (VPEs) are
within the same core as each other, which is why it is simply copied for
coupled_cpus.

Paul

Re: [PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Paul Burton]

On Thu, Feb 20, 2014 at 02:53:06PM +0100, Daniel Lezcano wrote:
> On 02/20/2014 02:41 PM, Paul Burton wrote:
> >On Thu, Feb 20, 2014 at 02:35:18PM +0100, Daniel Lezcano wrote:
> >>On 01/15/2014 02:55 PM, Paul Burton wrote:
> >>>Declaring this allows drivers which need to initialise each struct
> >>>cpuidle_device at initialisation time to make use of the structures
> >>>already defined in cpuidle.c, rather than having to wastefully define
> >>>their own.
> >>>
> >>>Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >>>Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >>>Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> >>>Cc: linux-pm@vger.kernel.org
> >>>---
> >>>  include/linux/cpuidle.h | 1 +
> >>>  1 file changed, 1 insertion(+)
> >>>
> >>>diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
> >>>index 50fcbb0..bab4f33 100644
> >>>--- a/include/linux/cpuidle.h
> >>>+++ b/include/linux/cpuidle.h
> >>>@@ -84,6 +84,7 @@ struct cpuidle_device {
> >>>  };
> >>>
> >>>  DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
> >>>+DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
> >>
> >>
> >>Nak. When a device is registered, it is assigned to the cpuidle_devices
> >>pointer and the backend driver should use it.
> >>
> >
> >Yes, but then if the driver needs to initialise the coupled_cpus mask
> >then it cannot do so until after the device has been registered. During
> >registration the cpuidle_coupled_register_device will then see the empty
> >coupled_cpus mask & do nothing. The only other ways around this would be
> >for the driver to define its own per-cpu struct cpuidle_device (which as
> >I state in the commit message seems wasteful when cpuidle already
> >defined them), or for cpuidle_coupled_register_device to be called later
> >after the driver had a chance to modify devices via the cpuidle_devices
> >pointers.
> 
> Yeah. I understand why you wanted to declare these cpu variables.
> 
> The mips cpuidle driver sounds like a bit particular. I believe I need some
> clarification on the behavior of the hardware to understand correctly the
> driver. Could you explain how the couples act vs the cpu ? And why
> cpu_sibling is used instead of cpu_possible_mask ?
> 
> 

Sure. The CPUs that are coupled are actually VPEs (Virtual Processor
Elements) within a single core. They share the compute resource (ALUs
etc) of the core but have their own register state, ie. they're a form
of simultaneous multithreading.

Coherence within a MIPS Coherent Processing System is a property of the
core rather than of an individual VPE (since the VPEs within a core
share the same L1 caches). So for idle states which are non-coherent the
VPEs within the core are coupled. That covers all idle states beyond a
simple "wait" instruction - clock gating or powering down a core
requires it to become non-coherent first.

cpu_sibling_mask is already setup to indicate which CPUs (VPEs) are
within the same core as each other, which is why it is simply copied for
coupled_cpus.

Paul

Re: [PATCH 09/10] cpuidle: declare cpuidle_dev in cpuidle.h

[Daniel Lezcano]

On 02/20/2014 03:00 PM, Paul Burton wrote:
> On Thu, Feb 20, 2014 at 02:53:06PM +0100, Daniel Lezcano wrote:
>> On 02/20/2014 02:41 PM, Paul Burton wrote:
>>> On Thu, Feb 20, 2014 at 02:35:18PM +0100, Daniel Lezcano wrote:
>>>> On 01/15/2014 02:55 PM, Paul Burton wrote:
>>>>> Declaring this allows drivers which need to initialise each struct
>>>>> cpuidle_device at initialisation time to make use of the structures
>>>>> already defined in cpuidle.c, rather than having to wastefully define
>>>>> their own.
>>>>>
>>>>> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>>>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>>>> Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
>>>>> Cc: linux-pm@vger.kernel.org
>>>>> ---
>>>>>   include/linux/cpuidle.h | 1 +
>>>>>   1 file changed, 1 insertion(+)
>>>>>
>>>>> diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h
>>>>> index 50fcbb0..bab4f33 100644
>>>>> --- a/include/linux/cpuidle.h
>>>>> +++ b/include/linux/cpuidle.h
>>>>> @@ -84,6 +84,7 @@ struct cpuidle_device {
>>>>>   };
>>>>>
>>>>>   DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices);
>>>>> +DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev);
>>>>
>>>>
>>>> Nak. When a device is registered, it is assigned to the cpuidle_devices
>>>> pointer and the backend driver should use it.
>>>>
>>>
>>> Yes, but then if the driver needs to initialise the coupled_cpus mask
>>> then it cannot do so until after the device has been registered. During
>>> registration the cpuidle_coupled_register_device will then see the empty
>>> coupled_cpus mask & do nothing. The only other ways around this would be
>>> for the driver to define its own per-cpu struct cpuidle_device (which as
>>> I state in the commit message seems wasteful when cpuidle already
>>> defined them), or for cpuidle_coupled_register_device to be called later
>>> after the driver had a chance to modify devices via the cpuidle_devices
>>> pointers.
>>
>> Yeah. I understand why you wanted to declare these cpu variables.
>>
>> The mips cpuidle driver sounds like a bit particular. I believe I need some
>> clarification on the behavior of the hardware to understand correctly the
>> driver. Could you explain how the couples act vs the cpu ? And why
>> cpu_sibling is used instead of cpu_possible_mask ?
>>
>>
>
> Sure. The CPUs that are coupled are actually VPEs (Virtual Processor
> Elements) within a single core. They share the compute resource (ALUs
> etc) of the core but have their own register state, ie. they're a form
> of simultaneous multithreading.
>
> Coherence within a MIPS Coherent Processing System is a property of the
> core rather than of an individual VPE (since the VPEs within a core
> share the same L1 caches). So for idle states which are non-coherent the
> VPEs within the core are coupled. That covers all idle states beyond a
> simple "wait" instruction - clock gating or powering down a core
> requires it to become non-coherent first.
>
> cpu_sibling_mask is already setup to indicate which CPUs (VPEs) are
> within the same core as each other, which is why it is simply copied for
> coupled_cpus.

Hi Paul,

thanks for the explanation.

   -- Daniel




-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

[PATCH 10/10] cpuidle: cpuidle driver for MIPS CPS

[Paul Burton]

This patch introduces a cpuidle driver implementation for the MIPS
Coherent Processing System (ie. Coherence Manager, Cluster Power
Controller). It allows for use of the following idle states:

 - Coherent wait. This is the usual MIPS wait instruction.

 - Non-coherent wait. In this state a core will disable coherency with
   the rest of the system before running the wait instruction. This
   eliminates coherence interventions which would typically be used to
   keep cores coherent.

These two states lay the groundwork for deeper states to be implemented
later, since all deeper states require the core to become non-coherent.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: linux-pm@vger.kernel.org
---
 drivers/cpuidle/Kconfig            |   5 +
 drivers/cpuidle/Kconfig.mips       |  14 +
 drivers/cpuidle/Makefile           |   3 +
 drivers/cpuidle/cpuidle-mips-cps.c | 545 +++++++++++++++++++++++++++++++++++++
 4 files changed, 567 insertions(+)
 create mode 100644 drivers/cpuidle/Kconfig.mips
 create mode 100644 drivers/cpuidle/cpuidle-mips-cps.c

diff --git a/drivers/cpuidle/Kconfig b/drivers/cpuidle/Kconfig
index b3fb81d..11ff281 100644
--- a/drivers/cpuidle/Kconfig
+++ b/drivers/cpuidle/Kconfig
@@ -35,6 +35,11 @@ depends on ARM
 source "drivers/cpuidle/Kconfig.arm"
 endmenu
 
+menu "MIPS CPU Idle Drivers"
+depends on MIPS
+source "drivers/cpuidle/Kconfig.mips"
+endmenu
+
 endif
 
 config ARCH_NEEDS_CPU_IDLE_COUPLED
diff --git a/drivers/cpuidle/Kconfig.mips b/drivers/cpuidle/Kconfig.mips
new file mode 100644
index 0000000..dc96691
--- /dev/null
+++ b/drivers/cpuidle/Kconfig.mips
@@ -0,0 +1,14 @@
+#
+# MIPS CPU Idle drivers
+#
+
+config MIPS_CPS_CPUIDLE
+	bool "Support for MIPS Coherent Processing Systems"
+	depends on SYS_SUPPORTS_MIPS_CPS && CPU_MIPSR2 && !MIPS_MT_SMTC
+	select ARCH_NEEDS_CPU_IDLE_COUPLED if MIPS_MT
+	select MIPS_CM
+	help
+	  Select this option to enable CPU idle driver for systems based
+	  around the MIPS Coherent Processing System architecture - that
+	  is, those with a Coherence Manager & optionally a Cluster
+	  Power Controller.
diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile
index 527be28..693cd95 100644
--- a/drivers/cpuidle/Makefile
+++ b/drivers/cpuidle/Makefile
@@ -13,3 +13,6 @@ obj-$(CONFIG_ARM_KIRKWOOD_CPUIDLE)	+= cpuidle-kirkwood.o
 obj-$(CONFIG_ARM_ZYNQ_CPUIDLE)		+= cpuidle-zynq.o
 obj-$(CONFIG_ARM_U8500_CPUIDLE)         += cpuidle-ux500.o
 obj-$(CONFIG_ARM_AT91_CPUIDLE)          += cpuidle-at91.o
+
+# MIPS SoC drivers
+obj-$(CONFIG_MIPS_CPS_CPUIDLE)		+= cpuidle-mips-cps.o
diff --git a/drivers/cpuidle/cpuidle-mips-cps.c b/drivers/cpuidle/cpuidle-mips-cps.c
new file mode 100644
index 0000000..a78bfb4
--- /dev/null
+++ b/drivers/cpuidle/cpuidle-mips-cps.c
@@ -0,0 +1,545 @@
+/*
+ * Copyright (C) 2013 Imagination Technologies
+ * Author: Paul Burton <paul.burton@imgtec.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation;  either version 2 of the  License, or (at your
+ * option) any later version.
+ */
+
+#include <linux/cpuidle.h>
+#include <linux/init.h>
+#include <linux/kconfig.h>
+#include <linux/slab.h>
+
+#include <asm/cacheflush.h>
+#include <asm/cacheops.h>
+#include <asm/idle.h>
+#include <asm/mips-cm.h>
+#include <asm/mipsmtregs.h>
+#include <asm/uasm.h>
+
+/*
+ * The CM & CPC can only handle coherence & power control on a per-core basis,
+ * thus in an MT system the VPEs within each core are coupled and can only
+ * enter or exit states requiring CM or CPC assistance in unison.
+ */
+#ifdef CONFIG_MIPS_MT
+# define coupled_coherence cpu_has_mipsmt
+#else
+# define coupled_coherence 0
+#endif
+
+/*
+ * cps_nc_entry_fn - type of a generated non-coherent state entry function
+ * @vpe_mask: a bitmap of online coupled VPEs, excluding this one
+ * @online: the count of online coupled VPEs (weight of vpe_mask + 1)
+ *
+ * The code entering & exiting non-coherent states is generated at runtime
+ * using uasm, in order to ensure that the compiler cannot insert a stray
+ * memory access at an unfortunate time and to allow the generation of optimal
+ * core-specific code particularly for cache routines. If coupled_coherence
+ * is non-zero, returns the number of VPEs that were in the wait state at the
+ * point this VPE left it. Returns garbage if coupled_coherence is zero.
+ */
+typedef unsigned (*cps_nc_entry_fn)(unsigned vpe_mask, unsigned online);
+
+/*
+ * The entry point of the generated non-coherent wait entry/exit function.
+ * Actually per-core rather than per-CPU.
+ */
+static DEFINE_PER_CPU_READ_MOSTLY(cps_nc_entry_fn, ncwait_asm_enter);
+
+/*
+ * Indicates the number of coupled VPEs ready to operate in a non-coherent
+ * state. Actually per-core rather than per-CPU.
+ */
+static DEFINE_PER_CPU_ALIGNED(u32, nc_ready_count);
+
+/* A somewhat arbitrary number of labels & relocs for uasm */
+static struct uasm_label labels[32] __initdata;
+static struct uasm_reloc relocs[32] __initdata;
+
+/* CPU dependant sync types */
+static unsigned stype_intervention;
+static unsigned stype_memory;
+static unsigned stype_ordering;
+
+enum mips_reg {
+	zero, at, v0, v1, a0, a1, a2, a3,
+	t0, t1, t2, t3, t4, t5, t6, t7,
+	s0, s1, s2, s3, s4, s5, s6, s7,
+	t8, t9, k0, k1, gp, sp, fp, ra,
+};
+
+static int cps_ncwait_enter(struct cpuidle_device *dev,
+			    struct cpuidle_driver *drv, int index)
+{
+	unsigned core = cpu_data[dev->cpu].core;
+	unsigned online, first_cpu, num_left;
+	cpumask_var_t coupled_mask, vpe_mask;
+
+	if (!alloc_cpumask_var(&coupled_mask, GFP_KERNEL))
+		return -ENOMEM;
+
+	if (!alloc_cpumask_var(&vpe_mask, GFP_KERNEL)) {
+		free_cpumask_var(coupled_mask);
+		return -ENOMEM;
+	}
+
+	/* Calculate which coupled CPUs (VPEs) are online */
+#ifdef CONFIG_MIPS_MT
+	cpumask_and(coupled_mask, cpu_online_mask, &dev->coupled_cpus);
+	first_cpu = cpumask_first(coupled_mask);
+	online = cpumask_weight(coupled_mask);
+	cpumask_clear_cpu(dev->cpu, coupled_mask);
+	cpumask_shift_right(vpe_mask, coupled_mask,
+			    cpumask_first(&dev->coupled_cpus));
+#else
+	cpumask_clear(coupled_mask);
+	cpumask_clear(vpe_mask);
+	first_cpu = dev->cpu;
+	online = 1;
+#endif
+
+	/*
+	 * Run the generated entry code. Note that we assume the number of VPEs
+	 * within this core does not exceed the width in bits of a long. Since
+	 * MVPConf0.PVPE is 4 bits wide this seems like a safe assumption.
+	 */
+	num_left = per_cpu(ncwait_asm_enter, core)(vpe_mask->bits[0], online);
+
+	/*
+	 * If this VPE is the first to leave the non-coherent wait state then
+	 * it needs to wake up any coupled VPEs still running their wait
+	 * instruction so that they return to cpuidle, which can then complete
+	 * coordination between the coupled VPEs & provide the governor with
+	 * a chance to reflect on the length of time the VPEs were in the
+	 * idle state.
+	 */
+	if (coupled_coherence && (num_left == online))
+		arch_send_call_function_ipi_mask(coupled_mask);
+
+	free_cpumask_var(vpe_mask);
+	free_cpumask_var(coupled_mask);
+	return index;
+}
+
+static struct cpuidle_driver cps_driver = {
+	.name			= "cpc_cpuidle",
+	.owner			= THIS_MODULE,
+	.states = {
+		MIPS_CPUIDLE_WAIT_STATE,
+		{
+			.enter	= cps_ncwait_enter,
+			.exit_latency		= 200,
+			.target_residency	= 450,
+			.flags	= CPUIDLE_FLAG_TIME_VALID,
+			.name	= "nc-wait",
+			.desc	= "non-coherent MIPS wait",
+		},
+	},
+	.state_count		= 2,
+	.safe_state_index	= 0,
+};
+
+static void __init cps_gen_cache_routine(u32 **pp, struct uasm_label **pl,
+					 struct uasm_reloc **pr,
+					 const struct cache_desc *cache,
+					 unsigned op, int lbl)
+{
+	unsigned cache_size = cache->ways << cache->waybit;
+	unsigned i;
+	const unsigned unroll_lines = 32;
+
+	/* If the cache isn't present this function has it easy */
+	if (cache->flags & MIPS_CACHE_NOT_PRESENT)
+		return;
+
+	/* Load base address */
+	UASM_i_LA(pp, t0, (long)CKSEG0);
+
+	/* Calculate end address */
+	if (cache_size < 0x8000)
+		uasm_i_addiu(pp, t1, t0, cache_size);
+	else
+		UASM_i_LA(pp, t1, (long)(CKSEG0 + cache_size));
+
+	/* Start of cache op loop */
+	uasm_build_label(pl, *pp, lbl);
+
+	/* Generate the cache ops */
+	for (i = 0; i < unroll_lines; i++)
+		uasm_i_cache(pp, op, i * cache->linesz, t0);
+
+	/* Update the base address */
+	uasm_i_addiu(pp, t0, t0, unroll_lines * cache->linesz);
+
+	/* Loop if we haven't reached the end address yet */
+	uasm_il_bne(pp, pr, t0, t1, lbl);
+	uasm_i_nop(pp);
+}
+
+static void * __init cps_gen_entry_code(struct cpuidle_device *dev)
+{
+	unsigned core = cpu_data[dev->cpu].core;
+	struct uasm_label *l = labels;
+	struct uasm_reloc *r = relocs;
+	u32 *buf, *p;
+	const unsigned r_vpemask = a0;
+	const unsigned r_online = a1;
+	const unsigned r_pcount = t6;
+	const unsigned r_pcohctl = t7;
+	const unsigned max_instrs = 256;
+	enum {
+		lbl_incready = 1,
+		lbl_lastvpe,
+		lbl_vpehalt_loop,
+		lbl_vpehalt_poll,
+		lbl_vpehalt_next,
+		lbl_disable_coherence,
+		lbl_invicache,
+		lbl_flushdcache,
+		lbl_vpeactivate_loop,
+		lbl_vpeactivate_next,
+		lbl_wait,
+		lbl_decready,
+	};
+
+	/* Allocate a buffer to hold the generated code */
+	p = buf = kcalloc(max_instrs, sizeof(u32), GFP_KERNEL);
+	if (!buf)
+		return NULL;
+
+	/* Clear labels & relocs ready for (re)use */
+	memset(labels, 0, sizeof(labels));
+	memset(relocs, 0, sizeof(relocs));
+
+	/*
+	 * Load address of the CM GCR_CL_COHERENCE register. This is done early
+	 * because it's needed in both the enable & disable coherence steps but
+	 * in the coupled case the enable step will only run on one VPE.
+	 */
+	UASM_i_LA(&p, r_pcohctl, (long)addr_gcr_cl_coherence());
+
+	if (coupled_coherence) {
+		/* Load address of nc_ready_count */
+		UASM_i_LA(&p, r_pcount, (long)&per_cpu(nc_ready_count, core));
+
+		/* Increment nc_ready_count */
+		uasm_build_label(&l, p, lbl_incready);
+		uasm_i_sync(&p, stype_ordering);
+		uasm_i_ll(&p, t1, 0, r_pcount);
+		uasm_i_addiu(&p, t2, t1, 1);
+		uasm_i_sc(&p, t2, 0, r_pcount);
+		uasm_il_beqz(&p, &r, t2, lbl_incready);
+		uasm_i_addiu(&p, t1, t1, 1);
+
+		/*
+		 * If this is the last VPE to become ready for non-coherence
+		 * then it should branch below.
+		 */
+		uasm_il_beq(&p, &r, t1, r_online, lbl_lastvpe);
+		uasm_i_nop(&p);
+
+		/*
+		 * Otherwise this is not the last VPE to become ready for
+		 * non-coherence. It needs to wait until coherence has been
+		 * disabled before executing a wait instruction, otherwise it
+		 * may return from wait quickly & re-enable coherence causing
+		 * a race with the VPE disabling coherence. It can't simply
+		 * poll the CPC sequencer for a non-coherent state as that
+		 * would race with any other VPE which may spot the
+		 * non-coherent state, run wait, return quickly & re-enable
+		 * coherence before this VPE ever saw the non-coherent state.
+		 * Instead this VPE will halt its TC such that it ceases to
+		 * execute for the moment.
+		 */
+		uasm_i_addiu(&p, t0, zero, TCHALT_H);
+		uasm_i_mtc0(&p, t0, 2, 4); /* TCHalt */
+
+		/* instruction_hazard(), to ensure the TC halts */
+		UASM_i_LA(&p, t0, (long)p + 12);
+		uasm_i_jr_hb(&p, t0);
+		uasm_i_nop(&p);
+
+		/*
+		 * The VPE which disables coherence will then clear the halt
+		 * bit for this VPE's TC once coherence has been disabled and
+		 * it can safely proceed to execute the wait instruction.
+		 */
+		uasm_il_b(&p, &r, lbl_wait);
+		uasm_i_nop(&p);
+
+		/*
+		 * The last VPE to increment nc_ready_count will continue from
+		 * here and must spin until all other VPEs within the core have
+		 * been halted, at which point it can be sure that it is safe
+		 * to disable coherence.
+		 *
+		 *   t0: number of VPEs left to handle
+		 *   t1: (shifted) mask of online VPEs
+		 *   t2: current VPE index
+		 */
+		uasm_build_label(&l, p, lbl_lastvpe);
+		uasm_i_addiu(&p, t0, r_online, -1);
+		uasm_il_beqz(&p, &r, t0, lbl_disable_coherence);
+		uasm_i_move(&p, t1, r_vpemask);
+		uasm_i_move(&p, t2, zero);
+
+		/*
+		 * Now loop through all VPEs within the core checking whether
+		 * they are online & not this VPE, which can be determined by
+		 * checking the vpe_mask argument. If a VPE is offline or is
+		 * this VPE, skip it.
+		 */
+		uasm_build_label(&l, p, lbl_vpehalt_loop);
+		uasm_i_andi(&p, t3, t1, 1);
+		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_next);
+
+		/* settc(vpe) */
+		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
+		uasm_i_ins(&p, t3, t2, 0, 8);
+		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
+		uasm_i_ehb(&p);
+
+		/*
+		 * It's very likely that the VPE has already halted itself
+		 * by now, but there's theoretically a chance that it may not
+		 * have. Wait until the VPE's TC is halted.
+		 */
+		uasm_build_label(&l, p, lbl_vpehalt_poll);
+		uasm_i_mftc0(&p, t3, 2, 4); /* TCHalt */
+		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_poll);
+		uasm_i_nop(&p);
+
+		/* Decrement the count of VPEs to be handled */
+		uasm_i_addiu(&p, t0, t0, -1);
+
+		/* Proceed to the next VPE, if there is one */
+		uasm_build_label(&l, p, lbl_vpehalt_next);
+		uasm_i_srl(&p, t1, t1, 1);
+		uasm_il_bnez(&p, &r, t0, lbl_vpehalt_loop);
+		uasm_i_addiu(&p, t2, t2, 1);
+	}
+
+	/*
+	 * This is the point of no return - this VPE will now proceed to
+	 * disable coherence. At this point we *must* be sure that no other
+	 * VPE within the core will interfere with the L1 dcache.
+	 */
+	uasm_build_label(&l, p, lbl_disable_coherence);
+
+	/* Completion barrier */
+	uasm_i_sync(&p, stype_memory);
+
+	/* Invalidate the L1 icache */
+	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].icache,
+			      Index_Invalidate_I, lbl_invicache);
+
+	/* Writeback & invalidate the L1 dcache */
+	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].dcache,
+			      Index_Writeback_Inv_D, lbl_flushdcache);
+
+	/*
+	 * Disable all but self interventions. The load from COHCTL is defined
+	 * by the interAptiv & proAptiv SUMs as ensuring that the operation
+	 * resulting from the preceeding store is complete.
+	 */
+	uasm_i_addiu(&p, t0, zero, 1 << cpu_data[dev->cpu].core);
+	uasm_i_sw(&p, t0, 0, r_pcohctl);
+	uasm_i_lw(&p, t0, 0, r_pcohctl);
+
+	/* Sync to ensure previous interventions are complete */
+	uasm_i_sync(&p, stype_intervention);
+
+	/* Disable coherence */
+	uasm_i_sw(&p, zero, 0, r_pcohctl);
+	uasm_i_lw(&p, t0, 0, r_pcohctl);
+
+	if (coupled_coherence) {
+		/*
+		 * Now that coherence is disabled it is safe for all VPEs to
+		 * proceed with executing their wait instruction, so this VPE
+		 * will go ahead and clear the halt bit of the TCs associated
+		 * with all other online VPEs within the core. Start by
+		 * initialising variables used throughout the loop, and
+		 * skipping the loop entirely if there are no VPEs to handle.
+		 *
+		 *   t0: number of VPEs left to handle
+		 *   t1: (shifted) mask of online VPEs
+		 *   t2: current VPE index
+		 */
+		uasm_i_addiu(&p, t0, r_online, -1);
+		uasm_il_beqz(&p, &r, t0, lbl_wait);
+		uasm_i_move(&p, t1, r_vpemask);
+		uasm_i_move(&p, t2, zero);
+
+		/*
+		 * Now loop through all VPEs within the core checking whether
+		 * they are online & not this VPE, which can be determined by
+		 * checking the vpe_mask argument. If a VPE is offline or is
+		 * this VPE, skip it.
+		 */
+		uasm_build_label(&l, p, lbl_vpeactivate_loop);
+		uasm_i_andi(&p, t3, t1, 1);
+		uasm_il_beqz(&p, &r, t3, lbl_vpeactivate_next);
+
+		/* settc(vpe) */
+		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
+		uasm_i_ins(&p, t3, t2, 0, 8);
+		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
+		uasm_i_ehb(&p);
+
+		/* Clear TCHalt */
+		uasm_i_mttc0(&p, zero, 2, 4); /* TCHalt */
+
+		/* Decrement the count of VPEs to be handled */
+		uasm_i_addiu(&p, t0, t0, -1);
+
+		/* Proceed to the next VPE, if there is one */
+		uasm_build_label(&l, p, lbl_vpeactivate_next);
+		uasm_i_srl(&p, t1, t1, 1);
+		uasm_il_bnez(&p, &r, t0, lbl_vpeactivate_loop);
+		uasm_i_addiu(&p, t2, t2, 1);
+	}
+
+	/* Now perform our wait */
+	uasm_build_label(&l, p, lbl_wait);
+	uasm_i_wait(&p, 0);
+
+	/*
+	 * Re-enable coherence. Note that all coupled VPEs will run this, the
+	 * first will actually re-enable coherence & the rest will just be
+	 * performing a rather unusual nop.
+	 */
+	uasm_i_addiu(&p, t0, zero, CM_GCR_Cx_COHERENCE_COHDOMAINEN_MSK);
+	uasm_i_sw(&p, t0, 0, r_pcohctl);
+	uasm_i_lw(&p, t0, 0, r_pcohctl);
+
+	/* Ordering barrier */
+	uasm_i_sync(&p, stype_ordering);
+
+	if (coupled_coherence) {
+		/* Decrement nc_ready_count */
+		uasm_build_label(&l, p, lbl_decready);
+		uasm_i_sync(&p, stype_ordering);
+		uasm_i_ll(&p, t1, 0, r_pcount);
+		uasm_i_addiu(&p, t2, t1, -1);
+		uasm_i_sc(&p, t2, 0, r_pcount);
+		uasm_il_beqz(&p, &r, t2, lbl_decready);
+		uasm_i_move(&p, v0, t1);
+	}
+
+	/* The core is coherent, time to return to C code */
+	uasm_i_jr(&p, ra);
+	uasm_i_nop(&p);
+
+	/* Ensure the code didn't exceed the resources allocated for it */
+	BUG_ON((p - buf) > max_instrs);
+	BUG_ON((l - labels) > ARRAY_SIZE(labels));
+	BUG_ON((r - relocs) > ARRAY_SIZE(relocs));
+
+	/* Patch branch offsets */
+	uasm_resolve_relocs(relocs, labels);
+
+	/* Flush the icache */
+	local_flush_icache_range((unsigned long)buf, (unsigned long)p);
+
+	return buf;
+}
+
+static void __init cps_cpuidle_unregister(void)
+{
+	int cpu;
+	struct cpuidle_device *device;
+	cps_nc_entry_fn *fn;
+
+	for_each_possible_cpu(cpu) {
+		device = &per_cpu(cpuidle_dev, cpu);
+		cpuidle_unregister_device(device);
+
+		/* Free entry code */
+		fn = &per_cpu(ncwait_asm_enter, cpu_data[cpu].core);
+		kfree(*fn);
+		*fn = NULL;
+	}
+
+	cpuidle_unregister_driver(&cps_driver);
+}
+
+static int __init cps_cpuidle_init(void)
+{
+	int err, cpu, core, i;
+	struct cpuidle_device *device;
+	void *core_entry;
+
+	/*
+	 * If interrupts were enabled whilst running the wait instruction then
+	 * the VPE may end up processing interrupts whilst non-coherent.
+	 */
+	if (cpu_wait != r4k_wait_irqoff) {
+		pr_warn("cpuidle-cps requires that masked interrupts restart the CPU pipeline following a wait\n");
+		return -ENODEV;
+	}
+
+	/* Detect appropriate sync types for the system */
+	switch (current_cpu_data.cputype) {
+	case CPU_INTERAPTIV:
+	case CPU_PROAPTIV:
+		stype_intervention = 0x2;
+		stype_memory = 0x3;
+		stype_ordering = 0x10;
+		break;
+
+	default:
+		pr_warn("cpuidle-cps using heavyweight sync 0\n");
+	}
+
+	/*
+	 * Set the coupled flag on the appropriate states if this system
+	 * requires it.
+	 */
+	if (coupled_coherence)
+		for (i = 1; i < cps_driver.state_count; i++)
+			cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED;
+
+	err = cpuidle_register_driver(&cps_driver);
+	if (err) {
+		pr_err("Failed to register CPS cpuidle driver\n");
+		return err;
+	}
+
+	for_each_possible_cpu(cpu) {
+		core = cpu_data[cpu].core;
+		device = &per_cpu(cpuidle_dev, cpu);
+		device->cpu = cpu;
+#ifdef CONFIG_MIPS_MT
+		cpumask_copy(&device->coupled_cpus, &cpu_sibling_map[cpu]);
+#endif
+		if (!per_cpu(ncwait_asm_enter, core)) {
+			core_entry = cps_gen_entry_code(device);
+			if (!core_entry) {
+				pr_err("Failed to generate core %u entry\n",
+				       core);
+				err = -ENOMEM;
+				goto err_out;
+			}
+			per_cpu(ncwait_asm_enter, core) = core_entry;
+		}
+
+		err = cpuidle_register_device(device);
+		if (err) {
+			pr_err("Failed to register CPU%d cpuidle device\n",
+			       cpu);
+			goto err_out;
+		}
+	}
+
+	return 0;
+err_out:
+	cps_cpuidle_unregister();
+	return err;
+}
+device_initcall(cps_cpuidle_init);
-- 
1.8.4.2

[PATCH 10/10] cpuidle: cpuidle driver for MIPS CPS

[Paul Burton]

This patch introduces a cpuidle driver implementation for the MIPS
Coherent Processing System (ie. Coherence Manager, Cluster Power
Controller). It allows for use of the following idle states:

 - Coherent wait. This is the usual MIPS wait instruction.

 - Non-coherent wait. In this state a core will disable coherency with
   the rest of the system before running the wait instruction. This
   eliminates coherence interventions which would typically be used to
   keep cores coherent.

These two states lay the groundwork for deeper states to be implemented
later, since all deeper states require the core to become non-coherent.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: linux-pm@vger.kernel.org
---
 drivers/cpuidle/Kconfig            |   5 +
 drivers/cpuidle/Kconfig.mips       |  14 +
 drivers/cpuidle/Makefile           |   3 +
 drivers/cpuidle/cpuidle-mips-cps.c | 545 +++++++++++++++++++++++++++++++++++++
 4 files changed, 567 insertions(+)
 create mode 100644 drivers/cpuidle/Kconfig.mips
 create mode 100644 drivers/cpuidle/cpuidle-mips-cps.c

diff --git a/drivers/cpuidle/Kconfig b/drivers/cpuidle/Kconfig
index b3fb81d..11ff281 100644
--- a/drivers/cpuidle/Kconfig
+++ b/drivers/cpuidle/Kconfig
@@ -35,6 +35,11 @@ depends on ARM
 source "drivers/cpuidle/Kconfig.arm"
 endmenu
 
+menu "MIPS CPU Idle Drivers"
+depends on MIPS
+source "drivers/cpuidle/Kconfig.mips"
+endmenu
+
 endif
 
 config ARCH_NEEDS_CPU_IDLE_COUPLED
diff --git a/drivers/cpuidle/Kconfig.mips b/drivers/cpuidle/Kconfig.mips
new file mode 100644
index 0000000..dc96691
--- /dev/null
+++ b/drivers/cpuidle/Kconfig.mips
@@ -0,0 +1,14 @@
+#
+# MIPS CPU Idle drivers
+#
+
+config MIPS_CPS_CPUIDLE
+	bool "Support for MIPS Coherent Processing Systems"
+	depends on SYS_SUPPORTS_MIPS_CPS && CPU_MIPSR2 && !MIPS_MT_SMTC
+	select ARCH_NEEDS_CPU_IDLE_COUPLED if MIPS_MT
+	select MIPS_CM
+	help
+	  Select this option to enable CPU idle driver for systems based
+	  around the MIPS Coherent Processing System architecture - that
+	  is, those with a Coherence Manager & optionally a Cluster
+	  Power Controller.
diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile
index 527be28..693cd95 100644
--- a/drivers/cpuidle/Makefile
+++ b/drivers/cpuidle/Makefile
@@ -13,3 +13,6 @@ obj-$(CONFIG_ARM_KIRKWOOD_CPUIDLE)	+= cpuidle-kirkwood.o
 obj-$(CONFIG_ARM_ZYNQ_CPUIDLE)		+= cpuidle-zynq.o
 obj-$(CONFIG_ARM_U8500_CPUIDLE)         += cpuidle-ux500.o
 obj-$(CONFIG_ARM_AT91_CPUIDLE)          += cpuidle-at91.o
+
+# MIPS SoC drivers
+obj-$(CONFIG_MIPS_CPS_CPUIDLE)		+= cpuidle-mips-cps.o
diff --git a/drivers/cpuidle/cpuidle-mips-cps.c b/drivers/cpuidle/cpuidle-mips-cps.c
new file mode 100644
index 0000000..a78bfb4
--- /dev/null
+++ b/drivers/cpuidle/cpuidle-mips-cps.c
@@ -0,0 +1,545 @@
+/*
+ * Copyright (C) 2013 Imagination Technologies
+ * Author: Paul Burton <paul.burton@imgtec.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation;  either version 2 of the  License, or (at your
+ * option) any later version.
+ */
+
+#include <linux/cpuidle.h>
+#include <linux/init.h>
+#include <linux/kconfig.h>
+#include <linux/slab.h>
+
+#include <asm/cacheflush.h>
+#include <asm/cacheops.h>
+#include <asm/idle.h>
+#include <asm/mips-cm.h>
+#include <asm/mipsmtregs.h>
+#include <asm/uasm.h>
+
+/*
+ * The CM & CPC can only handle coherence & power control on a per-core basis,
+ * thus in an MT system the VPEs within each core are coupled and can only
+ * enter or exit states requiring CM or CPC assistance in unison.
+ */
+#ifdef CONFIG_MIPS_MT
+# define coupled_coherence cpu_has_mipsmt
+#else
+# define coupled_coherence 0
+#endif
+
+/*
+ * cps_nc_entry_fn - type of a generated non-coherent state entry function
+ * @vpe_mask: a bitmap of online coupled VPEs, excluding this one
+ * @online: the count of online coupled VPEs (weight of vpe_mask + 1)
+ *
+ * The code entering & exiting non-coherent states is generated at runtime
+ * using uasm, in order to ensure that the compiler cannot insert a stray
+ * memory access at an unfortunate time and to allow the generation of optimal
+ * core-specific code particularly for cache routines. If coupled_coherence
+ * is non-zero, returns the number of VPEs that were in the wait state at the
+ * point this VPE left it. Returns garbage if coupled_coherence is zero.
+ */
+typedef unsigned (*cps_nc_entry_fn)(unsigned vpe_mask, unsigned online);
+
+/*
+ * The entry point of the generated non-coherent wait entry/exit function.
+ * Actually per-core rather than per-CPU.
+ */
+static DEFINE_PER_CPU_READ_MOSTLY(cps_nc_entry_fn, ncwait_asm_enter);
+
+/*
+ * Indicates the number of coupled VPEs ready to operate in a non-coherent
+ * state. Actually per-core rather than per-CPU.
+ */
+static DEFINE_PER_CPU_ALIGNED(u32, nc_ready_count);
+
+/* A somewhat arbitrary number of labels & relocs for uasm */
+static struct uasm_label labels[32] __initdata;
+static struct uasm_reloc relocs[32] __initdata;
+
+/* CPU dependant sync types */
+static unsigned stype_intervention;
+static unsigned stype_memory;
+static unsigned stype_ordering;
+
+enum mips_reg {
+	zero, at, v0, v1, a0, a1, a2, a3,
+	t0, t1, t2, t3, t4, t5, t6, t7,
+	s0, s1, s2, s3, s4, s5, s6, s7,
+	t8, t9, k0, k1, gp, sp, fp, ra,
+};
+
+static int cps_ncwait_enter(struct cpuidle_device *dev,
+			    struct cpuidle_driver *drv, int index)
+{
+	unsigned core = cpu_data[dev->cpu].core;
+	unsigned online, first_cpu, num_left;
+	cpumask_var_t coupled_mask, vpe_mask;
+
+	if (!alloc_cpumask_var(&coupled_mask, GFP_KERNEL))
+		return -ENOMEM;
+
+	if (!alloc_cpumask_var(&vpe_mask, GFP_KERNEL)) {
+		free_cpumask_var(coupled_mask);
+		return -ENOMEM;
+	}
+
+	/* Calculate which coupled CPUs (VPEs) are online */
+#ifdef CONFIG_MIPS_MT
+	cpumask_and(coupled_mask, cpu_online_mask, &dev->coupled_cpus);
+	first_cpu = cpumask_first(coupled_mask);
+	online = cpumask_weight(coupled_mask);
+	cpumask_clear_cpu(dev->cpu, coupled_mask);
+	cpumask_shift_right(vpe_mask, coupled_mask,
+			    cpumask_first(&dev->coupled_cpus));
+#else
+	cpumask_clear(coupled_mask);
+	cpumask_clear(vpe_mask);
+	first_cpu = dev->cpu;
+	online = 1;
+#endif
+
+	/*
+	 * Run the generated entry code. Note that we assume the number of VPEs
+	 * within this core does not exceed the width in bits of a long. Since
+	 * MVPConf0.PVPE is 4 bits wide this seems like a safe assumption.
+	 */
+	num_left = per_cpu(ncwait_asm_enter, core)(vpe_mask->bits[0], online);
+
+	/*
+	 * If this VPE is the first to leave the non-coherent wait state then
+	 * it needs to wake up any coupled VPEs still running their wait
+	 * instruction so that they return to cpuidle, which can then complete
+	 * coordination between the coupled VPEs & provide the governor with
+	 * a chance to reflect on the length of time the VPEs were in the
+	 * idle state.
+	 */
+	if (coupled_coherence && (num_left == online))
+		arch_send_call_function_ipi_mask(coupled_mask);
+
+	free_cpumask_var(vpe_mask);
+	free_cpumask_var(coupled_mask);
+	return index;
+}
+
+static struct cpuidle_driver cps_driver = {
+	.name			= "cpc_cpuidle",
+	.owner			= THIS_MODULE,
+	.states = {
+		MIPS_CPUIDLE_WAIT_STATE,
+		{
+			.enter	= cps_ncwait_enter,
+			.exit_latency		= 200,
+			.target_residency	= 450,
+			.flags	= CPUIDLE_FLAG_TIME_VALID,
+			.name	= "nc-wait",
+			.desc	= "non-coherent MIPS wait",
+		},
+	},
+	.state_count		= 2,
+	.safe_state_index	= 0,
+};
+
+static void __init cps_gen_cache_routine(u32 **pp, struct uasm_label **pl,
+					 struct uasm_reloc **pr,
+					 const struct cache_desc *cache,
+					 unsigned op, int lbl)
+{
+	unsigned cache_size = cache->ways << cache->waybit;
+	unsigned i;
+	const unsigned unroll_lines = 32;
+
+	/* If the cache isn't present this function has it easy */
+	if (cache->flags & MIPS_CACHE_NOT_PRESENT)
+		return;
+
+	/* Load base address */
+	UASM_i_LA(pp, t0, (long)CKSEG0);
+
+	/* Calculate end address */
+	if (cache_size < 0x8000)
+		uasm_i_addiu(pp, t1, t0, cache_size);
+	else
+		UASM_i_LA(pp, t1, (long)(CKSEG0 + cache_size));
+
+	/* Start of cache op loop */
+	uasm_build_label(pl, *pp, lbl);
+
+	/* Generate the cache ops */
+	for (i = 0; i < unroll_lines; i++)
+		uasm_i_cache(pp, op, i * cache->linesz, t0);
+
+	/* Update the base address */
+	uasm_i_addiu(pp, t0, t0, unroll_lines * cache->linesz);
+
+	/* Loop if we haven't reached the end address yet */
+	uasm_il_bne(pp, pr, t0, t1, lbl);
+	uasm_i_nop(pp);
+}
+
+static void * __init cps_gen_entry_code(struct cpuidle_device *dev)
+{
+	unsigned core = cpu_data[dev->cpu].core;
+	struct uasm_label *l = labels;
+	struct uasm_reloc *r = relocs;
+	u32 *buf, *p;
+	const unsigned r_vpemask = a0;
+	const unsigned r_online = a1;
+	const unsigned r_pcount = t6;
+	const unsigned r_pcohctl = t7;
+	const unsigned max_instrs = 256;
+	enum {
+		lbl_incready = 1,
+		lbl_lastvpe,
+		lbl_vpehalt_loop,
+		lbl_vpehalt_poll,
+		lbl_vpehalt_next,
+		lbl_disable_coherence,
+		lbl_invicache,
+		lbl_flushdcache,
+		lbl_vpeactivate_loop,
+		lbl_vpeactivate_next,
+		lbl_wait,
+		lbl_decready,
+	};
+
+	/* Allocate a buffer to hold the generated code */
+	p = buf = kcalloc(max_instrs, sizeof(u32), GFP_KERNEL);
+	if (!buf)
+		return NULL;
+
+	/* Clear labels & relocs ready for (re)use */
+	memset(labels, 0, sizeof(labels));
+	memset(relocs, 0, sizeof(relocs));
+
+	/*
+	 * Load address of the CM GCR_CL_COHERENCE register. This is done early
+	 * because it's needed in both the enable & disable coherence steps but
+	 * in the coupled case the enable step will only run on one VPE.
+	 */
+	UASM_i_LA(&p, r_pcohctl, (long)addr_gcr_cl_coherence());
+
+	if (coupled_coherence) {
+		/* Load address of nc_ready_count */
+		UASM_i_LA(&p, r_pcount, (long)&per_cpu(nc_ready_count, core));
+
+		/* Increment nc_ready_count */
+		uasm_build_label(&l, p, lbl_incready);
+		uasm_i_sync(&p, stype_ordering);
+		uasm_i_ll(&p, t1, 0, r_pcount);
+		uasm_i_addiu(&p, t2, t1, 1);
+		uasm_i_sc(&p, t2, 0, r_pcount);
+		uasm_il_beqz(&p, &r, t2, lbl_incready);
+		uasm_i_addiu(&p, t1, t1, 1);
+
+		/*
+		 * If this is the last VPE to become ready for non-coherence
+		 * then it should branch below.
+		 */
+		uasm_il_beq(&p, &r, t1, r_online, lbl_lastvpe);
+		uasm_i_nop(&p);
+
+		/*
+		 * Otherwise this is not the last VPE to become ready for
+		 * non-coherence. It needs to wait until coherence has been
+		 * disabled before executing a wait instruction, otherwise it
+		 * may return from wait quickly & re-enable coherence causing
+		 * a race with the VPE disabling coherence. It can't simply
+		 * poll the CPC sequencer for a non-coherent state as that
+		 * would race with any other VPE which may spot the
+		 * non-coherent state, run wait, return quickly & re-enable
+		 * coherence before this VPE ever saw the non-coherent state.
+		 * Instead this VPE will halt its TC such that it ceases to
+		 * execute for the moment.
+		 */
+		uasm_i_addiu(&p, t0, zero, TCHALT_H);
+		uasm_i_mtc0(&p, t0, 2, 4); /* TCHalt */
+
+		/* instruction_hazard(), to ensure the TC halts */
+		UASM_i_LA(&p, t0, (long)p + 12);
+		uasm_i_jr_hb(&p, t0);
+		uasm_i_nop(&p);
+
+		/*
+		 * The VPE which disables coherence will then clear the halt
+		 * bit for this VPE's TC once coherence has been disabled and
+		 * it can safely proceed to execute the wait instruction.
+		 */
+		uasm_il_b(&p, &r, lbl_wait);
+		uasm_i_nop(&p);
+
+		/*
+		 * The last VPE to increment nc_ready_count will continue from
+		 * here and must spin until all other VPEs within the core have
+		 * been halted, at which point it can be sure that it is safe
+		 * to disable coherence.
+		 *
+		 *   t0: number of VPEs left to handle
+		 *   t1: (shifted) mask of online VPEs
+		 *   t2: current VPE index
+		 */
+		uasm_build_label(&l, p, lbl_lastvpe);
+		uasm_i_addiu(&p, t0, r_online, -1);
+		uasm_il_beqz(&p, &r, t0, lbl_disable_coherence);
+		uasm_i_move(&p, t1, r_vpemask);
+		uasm_i_move(&p, t2, zero);
+
+		/*
+		 * Now loop through all VPEs within the core checking whether
+		 * they are online & not this VPE, which can be determined by
+		 * checking the vpe_mask argument. If a VPE is offline or is
+		 * this VPE, skip it.
+		 */
+		uasm_build_label(&l, p, lbl_vpehalt_loop);
+		uasm_i_andi(&p, t3, t1, 1);
+		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_next);
+
+		/* settc(vpe) */
+		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
+		uasm_i_ins(&p, t3, t2, 0, 8);
+		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
+		uasm_i_ehb(&p);
+
+		/*
+		 * It's very likely that the VPE has already halted itself
+		 * by now, but there's theoretically a chance that it may not
+		 * have. Wait until the VPE's TC is halted.
+		 */
+		uasm_build_label(&l, p, lbl_vpehalt_poll);
+		uasm_i_mftc0(&p, t3, 2, 4); /* TCHalt */
+		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_poll);
+		uasm_i_nop(&p);
+
+		/* Decrement the count of VPEs to be handled */
+		uasm_i_addiu(&p, t0, t0, -1);
+
+		/* Proceed to the next VPE, if there is one */
+		uasm_build_label(&l, p, lbl_vpehalt_next);
+		uasm_i_srl(&p, t1, t1, 1);
+		uasm_il_bnez(&p, &r, t0, lbl_vpehalt_loop);
+		uasm_i_addiu(&p, t2, t2, 1);
+	}
+
+	/*
+	 * This is the point of no return - this VPE will now proceed to
+	 * disable coherence. At this point we *must* be sure that no other
+	 * VPE within the core will interfere with the L1 dcache.
+	 */
+	uasm_build_label(&l, p, lbl_disable_coherence);
+
+	/* Completion barrier */
+	uasm_i_sync(&p, stype_memory);
+
+	/* Invalidate the L1 icache */
+	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].icache,
+			      Index_Invalidate_I, lbl_invicache);
+
+	/* Writeback & invalidate the L1 dcache */
+	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].dcache,
+			      Index_Writeback_Inv_D, lbl_flushdcache);
+
+	/*
+	 * Disable all but self interventions. The load from COHCTL is defined
+	 * by the interAptiv & proAptiv SUMs as ensuring that the operation
+	 * resulting from the preceeding store is complete.
+	 */
+	uasm_i_addiu(&p, t0, zero, 1 << cpu_data[dev->cpu].core);
+	uasm_i_sw(&p, t0, 0, r_pcohctl);
+	uasm_i_lw(&p, t0, 0, r_pcohctl);
+
+	/* Sync to ensure previous interventions are complete */
+	uasm_i_sync(&p, stype_intervention);
+
+	/* Disable coherence */
+	uasm_i_sw(&p, zero, 0, r_pcohctl);
+	uasm_i_lw(&p, t0, 0, r_pcohctl);
+
+	if (coupled_coherence) {
+		/*
+		 * Now that coherence is disabled it is safe for all VPEs to
+		 * proceed with executing their wait instruction, so this VPE
+		 * will go ahead and clear the halt bit of the TCs associated
+		 * with all other online VPEs within the core. Start by
+		 * initialising variables used throughout the loop, and
+		 * skipping the loop entirely if there are no VPEs to handle.
+		 *
+		 *   t0: number of VPEs left to handle
+		 *   t1: (shifted) mask of online VPEs
+		 *   t2: current VPE index
+		 */
+		uasm_i_addiu(&p, t0, r_online, -1);
+		uasm_il_beqz(&p, &r, t0, lbl_wait);
+		uasm_i_move(&p, t1, r_vpemask);
+		uasm_i_move(&p, t2, zero);
+
+		/*
+		 * Now loop through all VPEs within the core checking whether
+		 * they are online & not this VPE, which can be determined by
+		 * checking the vpe_mask argument. If a VPE is offline or is
+		 * this VPE, skip it.
+		 */
+		uasm_build_label(&l, p, lbl_vpeactivate_loop);
+		uasm_i_andi(&p, t3, t1, 1);
+		uasm_il_beqz(&p, &r, t3, lbl_vpeactivate_next);
+
+		/* settc(vpe) */
+		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
+		uasm_i_ins(&p, t3, t2, 0, 8);
+		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
+		uasm_i_ehb(&p);
+
+		/* Clear TCHalt */
+		uasm_i_mttc0(&p, zero, 2, 4); /* TCHalt */
+
+		/* Decrement the count of VPEs to be handled */
+		uasm_i_addiu(&p, t0, t0, -1);
+
+		/* Proceed to the next VPE, if there is one */
+		uasm_build_label(&l, p, lbl_vpeactivate_next);
+		uasm_i_srl(&p, t1, t1, 1);
+		uasm_il_bnez(&p, &r, t0, lbl_vpeactivate_loop);
+		uasm_i_addiu(&p, t2, t2, 1);
+	}
+
+	/* Now perform our wait */
+	uasm_build_label(&l, p, lbl_wait);
+	uasm_i_wait(&p, 0);
+
+	/*
+	 * Re-enable coherence. Note that all coupled VPEs will run this, the
+	 * first will actually re-enable coherence & the rest will just be
+	 * performing a rather unusual nop.
+	 */
+	uasm_i_addiu(&p, t0, zero, CM_GCR_Cx_COHERENCE_COHDOMAINEN_MSK);
+	uasm_i_sw(&p, t0, 0, r_pcohctl);
+	uasm_i_lw(&p, t0, 0, r_pcohctl);
+
+	/* Ordering barrier */
+	uasm_i_sync(&p, stype_ordering);
+
+	if (coupled_coherence) {
+		/* Decrement nc_ready_count */
+		uasm_build_label(&l, p, lbl_decready);
+		uasm_i_sync(&p, stype_ordering);
+		uasm_i_ll(&p, t1, 0, r_pcount);
+		uasm_i_addiu(&p, t2, t1, -1);
+		uasm_i_sc(&p, t2, 0, r_pcount);
+		uasm_il_beqz(&p, &r, t2, lbl_decready);
+		uasm_i_move(&p, v0, t1);
+	}
+
+	/* The core is coherent, time to return to C code */
+	uasm_i_jr(&p, ra);
+	uasm_i_nop(&p);
+
+	/* Ensure the code didn't exceed the resources allocated for it */
+	BUG_ON((p - buf) > max_instrs);
+	BUG_ON((l - labels) > ARRAY_SIZE(labels));
+	BUG_ON((r - relocs) > ARRAY_SIZE(relocs));
+
+	/* Patch branch offsets */
+	uasm_resolve_relocs(relocs, labels);
+
+	/* Flush the icache */
+	local_flush_icache_range((unsigned long)buf, (unsigned long)p);
+
+	return buf;
+}
+
+static void __init cps_cpuidle_unregister(void)
+{
+	int cpu;
+	struct cpuidle_device *device;
+	cps_nc_entry_fn *fn;
+
+	for_each_possible_cpu(cpu) {
+		device = &per_cpu(cpuidle_dev, cpu);
+		cpuidle_unregister_device(device);
+
+		/* Free entry code */
+		fn = &per_cpu(ncwait_asm_enter, cpu_data[cpu].core);
+		kfree(*fn);
+		*fn = NULL;
+	}
+
+	cpuidle_unregister_driver(&cps_driver);
+}
+
+static int __init cps_cpuidle_init(void)
+{
+	int err, cpu, core, i;
+	struct cpuidle_device *device;
+	void *core_entry;
+
+	/*
+	 * If interrupts were enabled whilst running the wait instruction then
+	 * the VPE may end up processing interrupts whilst non-coherent.
+	 */
+	if (cpu_wait != r4k_wait_irqoff) {
+		pr_warn("cpuidle-cps requires that masked interrupts restart the CPU pipeline following a wait\n");
+		return -ENODEV;
+	}
+
+	/* Detect appropriate sync types for the system */
+	switch (current_cpu_data.cputype) {
+	case CPU_INTERAPTIV:
+	case CPU_PROAPTIV:
+		stype_intervention = 0x2;
+		stype_memory = 0x3;
+		stype_ordering = 0x10;
+		break;
+
+	default:
+		pr_warn("cpuidle-cps using heavyweight sync 0\n");
+	}
+
+	/*
+	 * Set the coupled flag on the appropriate states if this system
+	 * requires it.
+	 */
+	if (coupled_coherence)
+		for (i = 1; i < cps_driver.state_count; i++)
+			cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED;
+
+	err = cpuidle_register_driver(&cps_driver);
+	if (err) {
+		pr_err("Failed to register CPS cpuidle driver\n");
+		return err;
+	}
+
+	for_each_possible_cpu(cpu) {
+		core = cpu_data[cpu].core;
+		device = &per_cpu(cpuidle_dev, cpu);
+		device->cpu = cpu;
+#ifdef CONFIG_MIPS_MT
+		cpumask_copy(&device->coupled_cpus, &cpu_sibling_map[cpu]);
+#endif
+		if (!per_cpu(ncwait_asm_enter, core)) {
+			core_entry = cps_gen_entry_code(device);
+			if (!core_entry) {
+				pr_err("Failed to generate core %u entry\n",
+				       core);
+				err = -ENOMEM;
+				goto err_out;
+			}
+			per_cpu(ncwait_asm_enter, core) = core_entry;
+		}
+
+		err = cpuidle_register_device(device);
+		if (err) {
+			pr_err("Failed to register CPU%d cpuidle device\n",
+			       cpu);
+			goto err_out;
+		}
+	}
+
+	return 0;
+err_out:
+	cps_cpuidle_unregister();
+	return err;
+}
+device_initcall(cps_cpuidle_init);
-- 
1.8.4.2

Re: [PATCH 10/10] cpuidle: cpuidle driver for MIPS CPS

[Daniel Lezcano]

On 01/15/2014 02:55 PM, Paul Burton wrote:
> This patch introduces a cpuidle driver implementation for the MIPS
> Coherent Processing System (ie. Coherence Manager, Cluster Power
> Controller). It allows for use of the following idle states:
>
>   - Coherent wait. This is the usual MIPS wait instruction.
>
>   - Non-coherent wait. In this state a core will disable coherency with
>     the rest of the system before running the wait instruction. This
>     eliminates coherence interventions which would typically be used to
>     keep cores coherent.
>
> These two states lay the groundwork for deeper states to be implemented
> later, since all deeper states require the core to become non-coherent.
>
> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> Cc: linux-pm@vger.kernel.org
> ---
>   drivers/cpuidle/Kconfig            |   5 +
>   drivers/cpuidle/Kconfig.mips       |  14 +
>   drivers/cpuidle/Makefile           |   3 +
>   drivers/cpuidle/cpuidle-mips-cps.c | 545 +++++++++++++++++++++++++++++++++++++
>   4 files changed, 567 insertions(+)
>   create mode 100644 drivers/cpuidle/Kconfig.mips
>   create mode 100644 drivers/cpuidle/cpuidle-mips-cps.c
>
> diff --git a/drivers/cpuidle/Kconfig b/drivers/cpuidle/Kconfig
> index b3fb81d..11ff281 100644
> --- a/drivers/cpuidle/Kconfig
> +++ b/drivers/cpuidle/Kconfig
> @@ -35,6 +35,11 @@ depends on ARM
>   source "drivers/cpuidle/Kconfig.arm"
>   endmenu
>
> +menu "MIPS CPU Idle Drivers"
> +depends on MIPS
> +source "drivers/cpuidle/Kconfig.mips"
> +endmenu
> +
>   endif
>
>   config ARCH_NEEDS_CPU_IDLE_COUPLED
> diff --git a/drivers/cpuidle/Kconfig.mips b/drivers/cpuidle/Kconfig.mips
> new file mode 100644
> index 0000000..dc96691
> --- /dev/null
> +++ b/drivers/cpuidle/Kconfig.mips
> @@ -0,0 +1,14 @@
> +#
> +# MIPS CPU Idle drivers
> +#
> +
> +config MIPS_CPS_CPUIDLE
> +	bool "Support for MIPS Coherent Processing Systems"
> +	depends on SYS_SUPPORTS_MIPS_CPS && CPU_MIPSR2 && !MIPS_MT_SMTC
> +	select ARCH_NEEDS_CPU_IDLE_COUPLED if MIPS_MT
> +	select MIPS_CM
> +	help
> +	  Select this option to enable CPU idle driver for systems based
> +	  around the MIPS Coherent Processing System architecture - that
> +	  is, those with a Coherence Manager & optionally a Cluster
> +	  Power Controller.
> diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile
> index 527be28..693cd95 100644
> --- a/drivers/cpuidle/Makefile
> +++ b/drivers/cpuidle/Makefile
> @@ -13,3 +13,6 @@ obj-$(CONFIG_ARM_KIRKWOOD_CPUIDLE)	+= cpuidle-kirkwood.o
>   obj-$(CONFIG_ARM_ZYNQ_CPUIDLE)		+= cpuidle-zynq.o
>   obj-$(CONFIG_ARM_U8500_CPUIDLE)         += cpuidle-ux500.o
>   obj-$(CONFIG_ARM_AT91_CPUIDLE)          += cpuidle-at91.o
> +
> +# MIPS SoC drivers
> +obj-$(CONFIG_MIPS_CPS_CPUIDLE)		+= cpuidle-mips-cps.o
> diff --git a/drivers/cpuidle/cpuidle-mips-cps.c b/drivers/cpuidle/cpuidle-mips-cps.c
> new file mode 100644
> index 0000000..a78bfb4
> --- /dev/null
> +++ b/drivers/cpuidle/cpuidle-mips-cps.c
> @@ -0,0 +1,545 @@
> +/*
> + * Copyright (C) 2013 Imagination Technologies
> + * Author: Paul Burton <paul.burton@imgtec.com>
> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms of the GNU General Public License as published by the
> + * Free Software Foundation;  either version 2 of the  License, or (at your
> + * option) any later version.
> + */
> +
> +#include <linux/cpuidle.h>
> +#include <linux/init.h>
> +#include <linux/kconfig.h>
> +#include <linux/slab.h>
> +
> +#include <asm/cacheflush.h>
> +#include <asm/cacheops.h>
> +#include <asm/idle.h>
> +#include <asm/mips-cm.h>
> +#include <asm/mipsmtregs.h>
> +#include <asm/uasm.h>

The convention is to not include headers from arch. These headers 
shouldn't appear in this driver.

> +/*
> + * The CM & CPC can only handle coherence & power control on a per-core basis,
> + * thus in an MT system the VPEs within each core are coupled and can only
> + * enter or exit states requiring CM or CPC assistance in unison.
> + */
> +#ifdef CONFIG_MIPS_MT
> +# define coupled_coherence cpu_has_mipsmt
> +#else
> +# define coupled_coherence 0
> +#endif
> +
> +/*
> + * cps_nc_entry_fn - type of a generated non-coherent state entry function
> + * @vpe_mask: a bitmap of online coupled VPEs, excluding this one
> + * @online: the count of online coupled VPEs (weight of vpe_mask + 1)
> + *
> + * The code entering & exiting non-coherent states is generated at runtime
> + * using uasm, in order to ensure that the compiler cannot insert a stray
> + * memory access at an unfortunate time and to allow the generation of optimal
> + * core-specific code particularly for cache routines. If coupled_coherence
> + * is non-zero, returns the number of VPEs that were in the wait state at the
> + * point this VPE left it. Returns garbage if coupled_coherence is zero.
> + */
> +typedef unsigned (*cps_nc_entry_fn)(unsigned vpe_mask, unsigned online);
> +
> +/*
> + * The entry point of the generated non-coherent wait entry/exit function.
> + * Actually per-core rather than per-CPU.
> + */
> +static DEFINE_PER_CPU_READ_MOSTLY(cps_nc_entry_fn, ncwait_asm_enter);
> +
> +/*
> + * Indicates the number of coupled VPEs ready to operate in a non-coherent
> + * state. Actually per-core rather than per-CPU.
> + */
> +static DEFINE_PER_CPU_ALIGNED(u32, nc_ready_count);
> +
> +/* A somewhat arbitrary number of labels & relocs for uasm */
> +static struct uasm_label labels[32] __initdata;
> +static struct uasm_reloc relocs[32] __initdata;
> +
> +/* CPU dependant sync types */
> +static unsigned stype_intervention;
> +static unsigned stype_memory;
> +static unsigned stype_ordering;
> +
> +enum mips_reg {
> +	zero, at, v0, v1, a0, a1, a2, a3,
> +	t0, t1, t2, t3, t4, t5, t6, t7,
> +	s0, s1, s2, s3, s4, s5, s6, s7,
> +	t8, t9, k0, k1, gp, sp, fp, ra,
> +};
> +
> +static int cps_ncwait_enter(struct cpuidle_device *dev,
> +			    struct cpuidle_driver *drv, int index)
> +{
> +	unsigned core = cpu_data[dev->cpu].core;
> +	unsigned online, first_cpu, num_left;
> +	cpumask_var_t coupled_mask, vpe_mask;
> +
> +	if (!alloc_cpumask_var(&coupled_mask, GFP_KERNEL))
> +		return -ENOMEM;
> +
> +	if (!alloc_cpumask_var(&vpe_mask, GFP_KERNEL)) {
> +		free_cpumask_var(coupled_mask);
> +		return -ENOMEM;
> +	}

You can't do that in this function where the local irqs are disabled. 
IMO, if you set CONFIG_CPUMASK_OFFSTACK and CONFIG_DEBUG_ATOMIC_SLEEP, 
you should see a kernel warning.

> +	/* Calculate which coupled CPUs (VPEs) are online */
> +#ifdef CONFIG_MIPS_MT
> +	cpumask_and(coupled_mask, cpu_online_mask, &dev->coupled_cpus);
> +	first_cpu = cpumask_first(coupled_mask);
> +	online = cpumask_weight(coupled_mask);
> +	cpumask_clear_cpu(dev->cpu, coupled_mask);
> +	cpumask_shift_right(vpe_mask, coupled_mask,
> +			    cpumask_first(&dev->coupled_cpus));

What is the purpose of this computation ?

> +#else
> +	cpumask_clear(coupled_mask);
> +	cpumask_clear(vpe_mask);
> +	first_cpu = dev->cpu;
> +	online = 1;
> +#endif

first_cpu is not used.

> +	/*
> +	 * Run the generated entry code. Note that we assume the number of VPEs
> +	 * within this core does not exceed the width in bits of a long. Since
> +	 * MVPConf0.PVPE is 4 bits wide this seems like a safe assumption.
> +	 */
> +	num_left = per_cpu(ncwait_asm_enter, core)(vpe_mask->bits[0], online);
> +
> +	/*
> +	 * If this VPE is the first to leave the non-coherent wait state then
> +	 * it needs to wake up any coupled VPEs still running their wait
> +	 * instruction so that they return to cpuidle,

Why is it needed ? Can't the other cpus stay idle ?

>        * which can then complete
> +	 * coordination between the coupled VPEs & provide the governor with
> +	 * a chance to reflect on the length of time the VPEs were in the
> +	 * idle state.
> +	 */
> +	if (coupled_coherence && (num_left == online))
> +		arch_send_call_function_ipi_mask(coupled_mask);

Except there is no choice due to hardware limitations, I don't think 
this is valid.

> +	free_cpumask_var(vpe_mask);
> +	free_cpumask_var(coupled_mask);
> +	return index;
> +}
> +
> +static struct cpuidle_driver cps_driver = {
> +	.name			= "cpc_cpuidle",
> +	.owner			= THIS_MODULE,
> +	.states = {
> +		MIPS_CPUIDLE_WAIT_STATE,
> +		{
> +			.enter	= cps_ncwait_enter,
> +			.exit_latency		= 200,
> +			.target_residency	= 450,
> +			.flags	= CPUIDLE_FLAG_TIME_VALID,
> +			.name	= "nc-wait",
> +			.desc	= "non-coherent MIPS wait",
> +		},
> +	},
> +	.state_count		= 2,
> +	.safe_state_index	= 0,
> +};



> +static void __init cps_gen_cache_routine(u32 **pp, struct uasm_label **pl,
> +					 struct uasm_reloc **pr,
> +					 const struct cache_desc *cache,
> +					 unsigned op, int lbl)
> +{
> +	unsigned cache_size = cache->ways << cache->waybit;
> +	unsigned i;
> +	const unsigned unroll_lines = 32;
> +
> +	/* If the cache isn't present this function has it easy */
> +	if (cache->flags & MIPS_CACHE_NOT_PRESENT)
> +		return;
> +
> +	/* Load base address */
> +	UASM_i_LA(pp, t0, (long)CKSEG0);
> +
> +	/* Calculate end address */
> +	if (cache_size < 0x8000)
> +		uasm_i_addiu(pp, t1, t0, cache_size);
> +	else
> +		UASM_i_LA(pp, t1, (long)(CKSEG0 + cache_size));
> +
> +	/* Start of cache op loop */
> +	uasm_build_label(pl, *pp, lbl);
> +
> +	/* Generate the cache ops */
> +	for (i = 0; i < unroll_lines; i++)
> +		uasm_i_cache(pp, op, i * cache->linesz, t0);
> +
> +	/* Update the base address */
> +	uasm_i_addiu(pp, t0, t0, unroll_lines * cache->linesz);
> +
> +	/* Loop if we haven't reached the end address yet */
> +	uasm_il_bne(pp, pr, t0, t1, lbl);
> +	uasm_i_nop(pp);
> +}
> +
> +static void * __init cps_gen_entry_code(struct cpuidle_device *dev)
> +{
> +	unsigned core = cpu_data[dev->cpu].core;
> +	struct uasm_label *l = labels;
> +	struct uasm_reloc *r = relocs;
> +	u32 *buf, *p;
> +	const unsigned r_vpemask = a0;
> +	const unsigned r_online = a1;
> +	const unsigned r_pcount = t6;
> +	const unsigned r_pcohctl = t7;
> +	const unsigned max_instrs = 256;
> +	enum {
> +		lbl_incready = 1,
> +		lbl_lastvpe,
> +		lbl_vpehalt_loop,
> +		lbl_vpehalt_poll,
> +		lbl_vpehalt_next,
> +		lbl_disable_coherence,
> +		lbl_invicache,
> +		lbl_flushdcache,
> +		lbl_vpeactivate_loop,
> +		lbl_vpeactivate_next,
> +		lbl_wait,
> +		lbl_decready,
> +	};
> +
> +	/* Allocate a buffer to hold the generated code */
> +	p = buf = kcalloc(max_instrs, sizeof(u32), GFP_KERNEL);
> +	if (!buf)
> +		return NULL;
> +
> +	/* Clear labels & relocs ready for (re)use */
> +	memset(labels, 0, sizeof(labels));
> +	memset(relocs, 0, sizeof(relocs));
> +
> +	/*
> +	 * Load address of the CM GCR_CL_COHERENCE register. This is done early
> +	 * because it's needed in both the enable & disable coherence steps but
> +	 * in the coupled case the enable step will only run on one VPE.
> +	 */
> +	UASM_i_LA(&p, r_pcohctl, (long)addr_gcr_cl_coherence());
> +
> +	if (coupled_coherence) {
> +		/* Load address of nc_ready_count */
> +		UASM_i_LA(&p, r_pcount, (long)&per_cpu(nc_ready_count, core));
> +
> +		/* Increment nc_ready_count */
> +		uasm_build_label(&l, p, lbl_incready);
> +		uasm_i_sync(&p, stype_ordering);
> +		uasm_i_ll(&p, t1, 0, r_pcount);
> +		uasm_i_addiu(&p, t2, t1, 1);
> +		uasm_i_sc(&p, t2, 0, r_pcount);
> +		uasm_il_beqz(&p, &r, t2, lbl_incready);
> +		uasm_i_addiu(&p, t1, t1, 1);
> +
> +		/*
> +		 * If this is the last VPE to become ready for non-coherence
> +		 * then it should branch below.
> +		 */
> +		uasm_il_beq(&p, &r, t1, r_online, lbl_lastvpe);
> +		uasm_i_nop(&p);
> +
> +		/*
> +		 * Otherwise this is not the last VPE to become ready for
> +		 * non-coherence. It needs to wait until coherence has been
> +		 * disabled before executing a wait instruction, otherwise it
> +		 * may return from wait quickly & re-enable coherence causing
> +		 * a race with the VPE disabling coherence. It can't simply
> +		 * poll the CPC sequencer for a non-coherent state as that
> +		 * would race with any other VPE which may spot the
> +		 * non-coherent state, run wait, return quickly & re-enable
> +		 * coherence before this VPE ever saw the non-coherent state.
> +		 * Instead this VPE will halt its TC such that it ceases to
> +		 * execute for the moment.
> +		 */
> +		uasm_i_addiu(&p, t0, zero, TCHALT_H);
> +		uasm_i_mtc0(&p, t0, 2, 4); /* TCHalt */
> +
> +		/* instruction_hazard(), to ensure the TC halts */
> +		UASM_i_LA(&p, t0, (long)p + 12);
> +		uasm_i_jr_hb(&p, t0);
> +		uasm_i_nop(&p);
> +
> +		/*
> +		 * The VPE which disables coherence will then clear the halt
> +		 * bit for this VPE's TC once coherence has been disabled and
> +		 * it can safely proceed to execute the wait instruction.
> +		 */
> +		uasm_il_b(&p, &r, lbl_wait);
> +		uasm_i_nop(&p);
> +
> +		/*
> +		 * The last VPE to increment nc_ready_count will continue from
> +		 * here and must spin until all other VPEs within the core have
> +		 * been halted, at which point it can be sure that it is safe
> +		 * to disable coherence.
> +		 *
> +		 *   t0: number of VPEs left to handle
> +		 *   t1: (shifted) mask of online VPEs
> +		 *   t2: current VPE index
> +		 */
> +		uasm_build_label(&l, p, lbl_lastvpe);
> +		uasm_i_addiu(&p, t0, r_online, -1);
> +		uasm_il_beqz(&p, &r, t0, lbl_disable_coherence);
> +		uasm_i_move(&p, t1, r_vpemask);
> +		uasm_i_move(&p, t2, zero);
> +
> +		/*
> +		 * Now loop through all VPEs within the core checking whether
> +		 * they are online & not this VPE, which can be determined by
> +		 * checking the vpe_mask argument. If a VPE is offline or is
> +		 * this VPE, skip it.
> +		 */
> +		uasm_build_label(&l, p, lbl_vpehalt_loop);
> +		uasm_i_andi(&p, t3, t1, 1);
> +		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_next);
> +
> +		/* settc(vpe) */
> +		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
> +		uasm_i_ins(&p, t3, t2, 0, 8);
> +		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
> +		uasm_i_ehb(&p);
> +
> +		/*
> +		 * It's very likely that the VPE has already halted itself
> +		 * by now, but there's theoretically a chance that it may not
> +		 * have. Wait until the VPE's TC is halted.
> +		 */
> +		uasm_build_label(&l, p, lbl_vpehalt_poll);
> +		uasm_i_mftc0(&p, t3, 2, 4); /* TCHalt */
> +		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_poll);
> +		uasm_i_nop(&p);
> +
> +		/* Decrement the count of VPEs to be handled */
> +		uasm_i_addiu(&p, t0, t0, -1);
> +
> +		/* Proceed to the next VPE, if there is one */
> +		uasm_build_label(&l, p, lbl_vpehalt_next);
> +		uasm_i_srl(&p, t1, t1, 1);
> +		uasm_il_bnez(&p, &r, t0, lbl_vpehalt_loop);
> +		uasm_i_addiu(&p, t2, t2, 1);
> +	}
> +
> +	/*
> +	 * This is the point of no return - this VPE will now proceed to
> +	 * disable coherence. At this point we *must* be sure that no other
> +	 * VPE within the core will interfere with the L1 dcache.
> +	 */
> +	uasm_build_label(&l, p, lbl_disable_coherence);
> +
> +	/* Completion barrier */
> +	uasm_i_sync(&p, stype_memory);
> +
> +	/* Invalidate the L1 icache */
> +	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].icache,
> +			      Index_Invalidate_I, lbl_invicache);
> +
> +	/* Writeback & invalidate the L1 dcache */
> +	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].dcache,
> +			      Index_Writeback_Inv_D, lbl_flushdcache);
> +
> +	/*
> +	 * Disable all but self interventions. The load from COHCTL is defined
> +	 * by the interAptiv & proAptiv SUMs as ensuring that the operation
> +	 * resulting from the preceeding store is complete.
> +	 */
> +	uasm_i_addiu(&p, t0, zero, 1 << cpu_data[dev->cpu].core);
> +	uasm_i_sw(&p, t0, 0, r_pcohctl);
> +	uasm_i_lw(&p, t0, 0, r_pcohctl);
> +
> +	/* Sync to ensure previous interventions are complete */
> +	uasm_i_sync(&p, stype_intervention);
> +
> +	/* Disable coherence */
> +	uasm_i_sw(&p, zero, 0, r_pcohctl);
> +	uasm_i_lw(&p, t0, 0, r_pcohctl);
> +
> +	if (coupled_coherence) {
> +		/*
> +		 * Now that coherence is disabled it is safe for all VPEs to
> +		 * proceed with executing their wait instruction, so this VPE
> +		 * will go ahead and clear the halt bit of the TCs associated
> +		 * with all other online VPEs within the core. Start by
> +		 * initialising variables used throughout the loop, and
> +		 * skipping the loop entirely if there are no VPEs to handle.
> +		 *
> +		 *   t0: number of VPEs left to handle
> +		 *   t1: (shifted) mask of online VPEs
> +		 *   t2: current VPE index
> +		 */
> +		uasm_i_addiu(&p, t0, r_online, -1);
> +		uasm_il_beqz(&p, &r, t0, lbl_wait);
> +		uasm_i_move(&p, t1, r_vpemask);
> +		uasm_i_move(&p, t2, zero);
> +
> +		/*
> +		 * Now loop through all VPEs within the core checking whether
> +		 * they are online & not this VPE, which can be determined by
> +		 * checking the vpe_mask argument. If a VPE is offline or is
> +		 * this VPE, skip it.
> +		 */
> +		uasm_build_label(&l, p, lbl_vpeactivate_loop);
> +		uasm_i_andi(&p, t3, t1, 1);
> +		uasm_il_beqz(&p, &r, t3, lbl_vpeactivate_next);
> +
> +		/* settc(vpe) */
> +		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
> +		uasm_i_ins(&p, t3, t2, 0, 8);
> +		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
> +		uasm_i_ehb(&p);
> +
> +		/* Clear TCHalt */
> +		uasm_i_mttc0(&p, zero, 2, 4); /* TCHalt */
> +
> +		/* Decrement the count of VPEs to be handled */
> +		uasm_i_addiu(&p, t0, t0, -1);
> +
> +		/* Proceed to the next VPE, if there is one */
> +		uasm_build_label(&l, p, lbl_vpeactivate_next);
> +		uasm_i_srl(&p, t1, t1, 1);
> +		uasm_il_bnez(&p, &r, t0, lbl_vpeactivate_loop);
> +		uasm_i_addiu(&p, t2, t2, 1);
> +	}
> +
> +	/* Now perform our wait */
> +	uasm_build_label(&l, p, lbl_wait);
> +	uasm_i_wait(&p, 0);
> +
> +	/*
> +	 * Re-enable coherence. Note that all coupled VPEs will run this, the
> +	 * first will actually re-enable coherence & the rest will just be
> +	 * performing a rather unusual nop.
> +	 */
> +	uasm_i_addiu(&p, t0, zero, CM_GCR_Cx_COHERENCE_COHDOMAINEN_MSK);
> +	uasm_i_sw(&p, t0, 0, r_pcohctl);
> +	uasm_i_lw(&p, t0, 0, r_pcohctl);
> +
> +	/* Ordering barrier */
> +	uasm_i_sync(&p, stype_ordering);
> +
> +	if (coupled_coherence) {
> +		/* Decrement nc_ready_count */
> +		uasm_build_label(&l, p, lbl_decready);
> +		uasm_i_sync(&p, stype_ordering);
> +		uasm_i_ll(&p, t1, 0, r_pcount);
> +		uasm_i_addiu(&p, t2, t1, -1);
> +		uasm_i_sc(&p, t2, 0, r_pcount);
> +		uasm_il_beqz(&p, &r, t2, lbl_decready);
> +		uasm_i_move(&p, v0, t1);
> +	}
> +
> +	/* The core is coherent, time to return to C code */
> +	uasm_i_jr(&p, ra);
> +	uasm_i_nop(&p);
> +
> +	/* Ensure the code didn't exceed the resources allocated for it */
> +	BUG_ON((p - buf) > max_instrs);
> +	BUG_ON((l - labels) > ARRAY_SIZE(labels));
> +	BUG_ON((r - relocs) > ARRAY_SIZE(relocs));
> +
> +	/* Patch branch offsets */
> +	uasm_resolve_relocs(relocs, labels);
> +
> +	/* Flush the icache */
> +	local_flush_icache_range((unsigned long)buf, (unsigned long)p);
> +
> +	return buf;
> +}

The two functions above should go to somewhere in arch/mips.

> +
> +static void __init cps_cpuidle_unregister(void)
> +{
> +	int cpu;
> +	struct cpuidle_device *device;
> +	cps_nc_entry_fn *fn;
> +
> +	for_each_possible_cpu(cpu) {
> +		device = &per_cpu(cpuidle_dev, cpu);
> +		cpuidle_unregister_device(device);
> +
> +		/* Free entry code */
> +		fn = &per_cpu(ncwait_asm_enter, cpu_data[cpu].core);
> +		kfree(*fn);
> +		*fn = NULL;
> +	}
> +
> +	cpuidle_unregister_driver(&cps_driver);
> +}
> +
> +static int __init cps_cpuidle_init(void)
> +{
> +	int err, cpu, core, i;
> +	struct cpuidle_device *device;
> +	void *core_entry;
> +
> +	/*
> +	 * If interrupts were enabled whilst running the wait instruction then
> +	 * the VPE may end up processing interrupts whilst non-coherent.
> +	 */
> +	if (cpu_wait != r4k_wait_irqoff) {
> +		pr_warn("cpuidle-cps requires that masked interrupts restart the CPU pipeline following a wait\n");
> +		return -ENODEV;
> +	}
> +
> +	/* Detect appropriate sync types for the system */
> +	switch (current_cpu_data.cputype) {
> +	case CPU_INTERAPTIV:
> +	case CPU_PROAPTIV:
> +		stype_intervention = 0x2;
> +		stype_memory = 0x3;
> +		stype_ordering = 0x10;
> +		break;
> +
> +	default:
> +		pr_warn("cpuidle-cps using heavyweight sync 0\n");
> +	}
> +
> +	/*
> +	 * Set the coupled flag on the appropriate states if this system
> +	 * requires it.
> +	 */
> +	if (coupled_coherence)
> +		for (i = 1; i < cps_driver.state_count; i++)
> +			cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED;

I am not sure CPUIDLE_FLAG_COUPLED is the solution for this driver. 
IIUC, with the IPI above and the wakeup sync with the couple states, 
this driver is waking up everybody instead of sleeping as much as possible.

I would recommend to have a look at a different approach, like the one 
used for cpuidle-ux500.c.

> +
> +	err = cpuidle_register_driver(&cps_driver);
> +	if (err) {
> +		pr_err("Failed to register CPS cpuidle driver\n");
> +		return err;
> +	}
> +
> +	for_each_possible_cpu(cpu) {
> +		core = cpu_data[cpu].core;
> +		device = &per_cpu(cpuidle_dev, cpu);
> +		device->cpu = cpu;
> +#ifdef CONFIG_MIPS_MT
> +		cpumask_copy(&device->coupled_cpus, &cpu_sibling_map[cpu]);
> +#endif
> +		if (!per_cpu(ncwait_asm_enter, core)) {
> +			core_entry = cps_gen_entry_code(device);
> +			if (!core_entry) {
> +				pr_err("Failed to generate core %u entry\n",
> +				       core);
> +				err = -ENOMEM;
> +				goto err_out;
> +			}
> +			per_cpu(ncwait_asm_enter, core) = core_entry;
> +		}
> +
> +		err = cpuidle_register_device(device);
> +		if (err) {
> +			pr_err("Failed to register CPU%d cpuidle device\n",
> +			       cpu);
> +			goto err_out;
> +		}
> +	}
> +
> +	return 0;
> +err_out:
> +	cps_cpuidle_unregister();
> +	return err;
> +}
> +device_initcall(cps_cpuidle_init);
>


-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: [PATCH 10/10] cpuidle: cpuidle driver for MIPS CPS

[Paul Burton]

On Tue, Feb 25, 2014 at 04:33:46PM +0100, Daniel Lezcano wrote:
> On 01/15/2014 02:55 PM, Paul Burton wrote:
> >This patch introduces a cpuidle driver implementation for the MIPS
> >Coherent Processing System (ie. Coherence Manager, Cluster Power
> >Controller). It allows for use of the following idle states:
> >
> >  - Coherent wait. This is the usual MIPS wait instruction.
> >
> >  - Non-coherent wait. In this state a core will disable coherency with
> >    the rest of the system before running the wait instruction. This
> >    eliminates coherence interventions which would typically be used to
> >    keep cores coherent.
> >
> >These two states lay the groundwork for deeper states to be implemented
> >later, since all deeper states require the core to become non-coherent.
> >
> >Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> >Cc: linux-pm@vger.kernel.org
> >---
> >  drivers/cpuidle/Kconfig            |   5 +
> >  drivers/cpuidle/Kconfig.mips       |  14 +
> >  drivers/cpuidle/Makefile           |   3 +
> >  drivers/cpuidle/cpuidle-mips-cps.c | 545 +++++++++++++++++++++++++++++++++++++
> >  4 files changed, 567 insertions(+)
> >  create mode 100644 drivers/cpuidle/Kconfig.mips
> >  create mode 100644 drivers/cpuidle/cpuidle-mips-cps.c
> >
> >diff --git a/drivers/cpuidle/Kconfig b/drivers/cpuidle/Kconfig
> >index b3fb81d..11ff281 100644
> >--- a/drivers/cpuidle/Kconfig
> >+++ b/drivers/cpuidle/Kconfig
> >@@ -35,6 +35,11 @@ depends on ARM
> >  source "drivers/cpuidle/Kconfig.arm"
> >  endmenu
> >
> >+menu "MIPS CPU Idle Drivers"
> >+depends on MIPS
> >+source "drivers/cpuidle/Kconfig.mips"
> >+endmenu
> >+
> >  endif
> >
> >  config ARCH_NEEDS_CPU_IDLE_COUPLED
> >diff --git a/drivers/cpuidle/Kconfig.mips b/drivers/cpuidle/Kconfig.mips
> >new file mode 100644
> >index 0000000..dc96691
> >--- /dev/null
> >+++ b/drivers/cpuidle/Kconfig.mips
> >@@ -0,0 +1,14 @@
> >+#
> >+# MIPS CPU Idle drivers
> >+#
> >+
> >+config MIPS_CPS_CPUIDLE
> >+	bool "Support for MIPS Coherent Processing Systems"
> >+	depends on SYS_SUPPORTS_MIPS_CPS && CPU_MIPSR2 && !MIPS_MT_SMTC
> >+	select ARCH_NEEDS_CPU_IDLE_COUPLED if MIPS_MT
> >+	select MIPS_CM
> >+	help
> >+	  Select this option to enable CPU idle driver for systems based
> >+	  around the MIPS Coherent Processing System architecture - that
> >+	  is, those with a Coherence Manager & optionally a Cluster
> >+	  Power Controller.
> >diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile
> >index 527be28..693cd95 100644
> >--- a/drivers/cpuidle/Makefile
> >+++ b/drivers/cpuidle/Makefile
> >@@ -13,3 +13,6 @@ obj-$(CONFIG_ARM_KIRKWOOD_CPUIDLE)	+= cpuidle-kirkwood.o
> >  obj-$(CONFIG_ARM_ZYNQ_CPUIDLE)		+= cpuidle-zynq.o
> >  obj-$(CONFIG_ARM_U8500_CPUIDLE)         += cpuidle-ux500.o
> >  obj-$(CONFIG_ARM_AT91_CPUIDLE)          += cpuidle-at91.o
> >+
> >+# MIPS SoC drivers
> >+obj-$(CONFIG_MIPS_CPS_CPUIDLE)		+= cpuidle-mips-cps.o
> >diff --git a/drivers/cpuidle/cpuidle-mips-cps.c b/drivers/cpuidle/cpuidle-mips-cps.c
> >new file mode 100644
> >index 0000000..a78bfb4
> >--- /dev/null
> >+++ b/drivers/cpuidle/cpuidle-mips-cps.c
> >@@ -0,0 +1,545 @@
> >+/*
> >+ * Copyright (C) 2013 Imagination Technologies
> >+ * Author: Paul Burton <paul.burton@imgtec.com>
> >+ *
> >+ * This program is free software; you can redistribute it and/or modify it
> >+ * under the terms of the GNU General Public License as published by the
> >+ * Free Software Foundation;  either version 2 of the  License, or (at your
> >+ * option) any later version.
> >+ */
> >+
> >+#include <linux/cpuidle.h>
> >+#include <linux/init.h>
> >+#include <linux/kconfig.h>
> >+#include <linux/slab.h>
> >+
> >+#include <asm/cacheflush.h>
> >+#include <asm/cacheops.h>
> >+#include <asm/idle.h>
> >+#include <asm/mips-cm.h>
> >+#include <asm/mipsmtregs.h>
> >+#include <asm/uasm.h>
> 
> The convention is to not include headers from arch. These headers shouldn't
> appear in this driver.
> 

Without accessing those headers I can't really implement anything useful
- entering these idle states by their nature involves architecture
specifics. Would you rather the bulk of the driver is implemented under
arch/mips & the code in drivers/cpuidle simply calls elsewhere to do the
real work?

> >+/*
> >+ * The CM & CPC can only handle coherence & power control on a per-core basis,
> >+ * thus in an MT system the VPEs within each core are coupled and can only
> >+ * enter or exit states requiring CM or CPC assistance in unison.
> >+ */
> >+#ifdef CONFIG_MIPS_MT
> >+# define coupled_coherence cpu_has_mipsmt
> >+#else
> >+# define coupled_coherence 0
> >+#endif
> >+
> >+/*
> >+ * cps_nc_entry_fn - type of a generated non-coherent state entry function
> >+ * @vpe_mask: a bitmap of online coupled VPEs, excluding this one
> >+ * @online: the count of online coupled VPEs (weight of vpe_mask + 1)
> >+ *
> >+ * The code entering & exiting non-coherent states is generated at runtime
> >+ * using uasm, in order to ensure that the compiler cannot insert a stray
> >+ * memory access at an unfortunate time and to allow the generation of optimal
> >+ * core-specific code particularly for cache routines. If coupled_coherence
> >+ * is non-zero, returns the number of VPEs that were in the wait state at the
> >+ * point this VPE left it. Returns garbage if coupled_coherence is zero.
> >+ */
> >+typedef unsigned (*cps_nc_entry_fn)(unsigned vpe_mask, unsigned online);
> >+
> >+/*
> >+ * The entry point of the generated non-coherent wait entry/exit function.
> >+ * Actually per-core rather than per-CPU.
> >+ */
> >+static DEFINE_PER_CPU_READ_MOSTLY(cps_nc_entry_fn, ncwait_asm_enter);
> >+
> >+/*
> >+ * Indicates the number of coupled VPEs ready to operate in a non-coherent
> >+ * state. Actually per-core rather than per-CPU.
> >+ */
> >+static DEFINE_PER_CPU_ALIGNED(u32, nc_ready_count);
> >+
> >+/* A somewhat arbitrary number of labels & relocs for uasm */
> >+static struct uasm_label labels[32] __initdata;
> >+static struct uasm_reloc relocs[32] __initdata;
> >+
> >+/* CPU dependant sync types */
> >+static unsigned stype_intervention;
> >+static unsigned stype_memory;
> >+static unsigned stype_ordering;
> >+
> >+enum mips_reg {
> >+	zero, at, v0, v1, a0, a1, a2, a3,
> >+	t0, t1, t2, t3, t4, t5, t6, t7,
> >+	s0, s1, s2, s3, s4, s5, s6, s7,
> >+	t8, t9, k0, k1, gp, sp, fp, ra,
> >+};
> >+
> >+static int cps_ncwait_enter(struct cpuidle_device *dev,
> >+			    struct cpuidle_driver *drv, int index)
> >+{
> >+	unsigned core = cpu_data[dev->cpu].core;
> >+	unsigned online, first_cpu, num_left;
> >+	cpumask_var_t coupled_mask, vpe_mask;
> >+
> >+	if (!alloc_cpumask_var(&coupled_mask, GFP_KERNEL))
> >+		return -ENOMEM;
> >+
> >+	if (!alloc_cpumask_var(&vpe_mask, GFP_KERNEL)) {
> >+		free_cpumask_var(coupled_mask);
> >+		return -ENOMEM;
> >+	}
> 
> You can't do that in this function where the local irqs are disabled. IMO,
> if you set CONFIG_CPUMASK_OFFSTACK and CONFIG_DEBUG_ATOMIC_SLEEP, you should
> see a kernel warning.

Right you are, it should either use GFP_ATOMIC or just not handle the
off-stack case (which isn't currently used).

> 
> >+	/* Calculate which coupled CPUs (VPEs) are online */
> >+#ifdef CONFIG_MIPS_MT
> >+	cpumask_and(coupled_mask, cpu_online_mask, &dev->coupled_cpus);
> >+	first_cpu = cpumask_first(coupled_mask);
> >+	online = cpumask_weight(coupled_mask);
> >+	cpumask_clear_cpu(dev->cpu, coupled_mask);
> >+	cpumask_shift_right(vpe_mask, coupled_mask,
> >+			    cpumask_first(&dev->coupled_cpus));
> 
> What is the purpose of this computation ?

If you read through the code generated in cps_gen_entry_code, the
vpe_mask is used to indicate the VPEs within the current core which are
both online & not the VPE currently running the code.

> 
> >+#else
> >+	cpumask_clear(coupled_mask);
> >+	cpumask_clear(vpe_mask);
> >+	first_cpu = dev->cpu;
> >+	online = 1;
> >+#endif
> 
> first_cpu is not used.

Right you are. It's used in further work-in-progress patches but I'll
remove it from this one.

> 
> >+	/*
> >+	 * Run the generated entry code. Note that we assume the number of VPEs
> >+	 * within this core does not exceed the width in bits of a long. Since
> >+	 * MVPConf0.PVPE is 4 bits wide this seems like a safe assumption.
> >+	 */
> >+	num_left = per_cpu(ncwait_asm_enter, core)(vpe_mask->bits[0], online);
> >+
> >+	/*
> >+	 * If this VPE is the first to leave the non-coherent wait state then
> >+	 * it needs to wake up any coupled VPEs still running their wait
> >+	 * instruction so that they return to cpuidle,
> 
> Why is it needed ? Can't the other cpus stay idle ?

Not with the current cpuidle code. Please see the end of
cpuidle_enter_state_coupled in drivers/cpuidle/coupled.c. The code waits
for all coupled CPUs to exit the idle state before any of them proceed.
Whilst I suppose it would be possible to modify cpuidle to not require
that, it would leave you with inaccurate residence statistics mentioned
below.

> 
> >       * which can then complete
> >+	 * coordination between the coupled VPEs & provide the governor with
> >+	 * a chance to reflect on the length of time the VPEs were in the
> >+	 * idle state.
> >+	 */
> >+	if (coupled_coherence && (num_left == online))
> >+		arch_send_call_function_ipi_mask(coupled_mask);
> 
> Except there is no choice due to hardware limitations, I don't think this is
> valid.

By nature when one CPU (VPE) within a core leaves a non-coherent state
the rest do too, because as I mentioned coherence is a property of the
core not of individual VPEs. It would be possible to leave the other
VPEs idle if we didn't differentiate between the coherent & non-coherent
wait states, but again not with cpuidle as it is today (due to the
waiting for all CPUs at the end of cpuidle_enter_state_coupled).

> 
> >+	free_cpumask_var(vpe_mask);
> >+	free_cpumask_var(coupled_mask);
> >+	return index;
> >+}
> >+
> >+static struct cpuidle_driver cps_driver = {
> >+	.name			= "cpc_cpuidle",
> >+	.owner			= THIS_MODULE,
> >+	.states = {
> >+		MIPS_CPUIDLE_WAIT_STATE,
> >+		{
> >+			.enter	= cps_ncwait_enter,
> >+			.exit_latency		= 200,
> >+			.target_residency	= 450,
> >+			.flags	= CPUIDLE_FLAG_TIME_VALID,
> >+			.name	= "nc-wait",
> >+			.desc	= "non-coherent MIPS wait",
> >+		},
> >+	},
> >+	.state_count		= 2,
> >+	.safe_state_index	= 0,
> >+};
> 
> 
> 
> >+static void __init cps_gen_cache_routine(u32 **pp, struct uasm_label **pl,
> >+					 struct uasm_reloc **pr,
> >+					 const struct cache_desc *cache,
> >+					 unsigned op, int lbl)
> >+{
> >+	unsigned cache_size = cache->ways << cache->waybit;
> >+	unsigned i;
> >+	const unsigned unroll_lines = 32;
> >+
> >+	/* If the cache isn't present this function has it easy */
> >+	if (cache->flags & MIPS_CACHE_NOT_PRESENT)
> >+		return;
> >+
> >+	/* Load base address */
> >+	UASM_i_LA(pp, t0, (long)CKSEG0);
> >+
> >+	/* Calculate end address */
> >+	if (cache_size < 0x8000)
> >+		uasm_i_addiu(pp, t1, t0, cache_size);
> >+	else
> >+		UASM_i_LA(pp, t1, (long)(CKSEG0 + cache_size));
> >+
> >+	/* Start of cache op loop */
> >+	uasm_build_label(pl, *pp, lbl);
> >+
> >+	/* Generate the cache ops */
> >+	for (i = 0; i < unroll_lines; i++)
> >+		uasm_i_cache(pp, op, i * cache->linesz, t0);
> >+
> >+	/* Update the base address */
> >+	uasm_i_addiu(pp, t0, t0, unroll_lines * cache->linesz);
> >+
> >+	/* Loop if we haven't reached the end address yet */
> >+	uasm_il_bne(pp, pr, t0, t1, lbl);
> >+	uasm_i_nop(pp);
> >+}
> >+
> >+static void * __init cps_gen_entry_code(struct cpuidle_device *dev)
> >+{
> >+	unsigned core = cpu_data[dev->cpu].core;
> >+	struct uasm_label *l = labels;
> >+	struct uasm_reloc *r = relocs;
> >+	u32 *buf, *p;
> >+	const unsigned r_vpemask = a0;
> >+	const unsigned r_online = a1;
> >+	const unsigned r_pcount = t6;
> >+	const unsigned r_pcohctl = t7;
> >+	const unsigned max_instrs = 256;
> >+	enum {
> >+		lbl_incready = 1,
> >+		lbl_lastvpe,
> >+		lbl_vpehalt_loop,
> >+		lbl_vpehalt_poll,
> >+		lbl_vpehalt_next,
> >+		lbl_disable_coherence,
> >+		lbl_invicache,
> >+		lbl_flushdcache,
> >+		lbl_vpeactivate_loop,
> >+		lbl_vpeactivate_next,
> >+		lbl_wait,
> >+		lbl_decready,
> >+	};
> >+
> >+	/* Allocate a buffer to hold the generated code */
> >+	p = buf = kcalloc(max_instrs, sizeof(u32), GFP_KERNEL);
> >+	if (!buf)
> >+		return NULL;
> >+
> >+	/* Clear labels & relocs ready for (re)use */
> >+	memset(labels, 0, sizeof(labels));
> >+	memset(relocs, 0, sizeof(relocs));
> >+
> >+	/*
> >+	 * Load address of the CM GCR_CL_COHERENCE register. This is done early
> >+	 * because it's needed in both the enable & disable coherence steps but
> >+	 * in the coupled case the enable step will only run on one VPE.
> >+	 */
> >+	UASM_i_LA(&p, r_pcohctl, (long)addr_gcr_cl_coherence());
> >+
> >+	if (coupled_coherence) {
> >+		/* Load address of nc_ready_count */
> >+		UASM_i_LA(&p, r_pcount, (long)&per_cpu(nc_ready_count, core));
> >+
> >+		/* Increment nc_ready_count */
> >+		uasm_build_label(&l, p, lbl_incready);
> >+		uasm_i_sync(&p, stype_ordering);
> >+		uasm_i_ll(&p, t1, 0, r_pcount);
> >+		uasm_i_addiu(&p, t2, t1, 1);
> >+		uasm_i_sc(&p, t2, 0, r_pcount);
> >+		uasm_il_beqz(&p, &r, t2, lbl_incready);
> >+		uasm_i_addiu(&p, t1, t1, 1);
> >+
> >+		/*
> >+		 * If this is the last VPE to become ready for non-coherence
> >+		 * then it should branch below.
> >+		 */
> >+		uasm_il_beq(&p, &r, t1, r_online, lbl_lastvpe);
> >+		uasm_i_nop(&p);
> >+
> >+		/*
> >+		 * Otherwise this is not the last VPE to become ready for
> >+		 * non-coherence. It needs to wait until coherence has been
> >+		 * disabled before executing a wait instruction, otherwise it
> >+		 * may return from wait quickly & re-enable coherence causing
> >+		 * a race with the VPE disabling coherence. It can't simply
> >+		 * poll the CPC sequencer for a non-coherent state as that
> >+		 * would race with any other VPE which may spot the
> >+		 * non-coherent state, run wait, return quickly & re-enable
> >+		 * coherence before this VPE ever saw the non-coherent state.
> >+		 * Instead this VPE will halt its TC such that it ceases to
> >+		 * execute for the moment.
> >+		 */
> >+		uasm_i_addiu(&p, t0, zero, TCHALT_H);
> >+		uasm_i_mtc0(&p, t0, 2, 4); /* TCHalt */
> >+
> >+		/* instruction_hazard(), to ensure the TC halts */
> >+		UASM_i_LA(&p, t0, (long)p + 12);
> >+		uasm_i_jr_hb(&p, t0);
> >+		uasm_i_nop(&p);
> >+
> >+		/*
> >+		 * The VPE which disables coherence will then clear the halt
> >+		 * bit for this VPE's TC once coherence has been disabled and
> >+		 * it can safely proceed to execute the wait instruction.
> >+		 */
> >+		uasm_il_b(&p, &r, lbl_wait);
> >+		uasm_i_nop(&p);
> >+
> >+		/*
> >+		 * The last VPE to increment nc_ready_count will continue from
> >+		 * here and must spin until all other VPEs within the core have
> >+		 * been halted, at which point it can be sure that it is safe
> >+		 * to disable coherence.
> >+		 *
> >+		 *   t0: number of VPEs left to handle
> >+		 *   t1: (shifted) mask of online VPEs
> >+		 *   t2: current VPE index
> >+		 */
> >+		uasm_build_label(&l, p, lbl_lastvpe);
> >+		uasm_i_addiu(&p, t0, r_online, -1);
> >+		uasm_il_beqz(&p, &r, t0, lbl_disable_coherence);
> >+		uasm_i_move(&p, t1, r_vpemask);
> >+		uasm_i_move(&p, t2, zero);
> >+
> >+		/*
> >+		 * Now loop through all VPEs within the core checking whether
> >+		 * they are online & not this VPE, which can be determined by
> >+		 * checking the vpe_mask argument. If a VPE is offline or is
> >+		 * this VPE, skip it.
> >+		 */
> >+		uasm_build_label(&l, p, lbl_vpehalt_loop);
> >+		uasm_i_andi(&p, t3, t1, 1);
> >+		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_next);
> >+
> >+		/* settc(vpe) */
> >+		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
> >+		uasm_i_ins(&p, t3, t2, 0, 8);
> >+		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
> >+		uasm_i_ehb(&p);
> >+
> >+		/*
> >+		 * It's very likely that the VPE has already halted itself
> >+		 * by now, but there's theoretically a chance that it may not
> >+		 * have. Wait until the VPE's TC is halted.
> >+		 */
> >+		uasm_build_label(&l, p, lbl_vpehalt_poll);
> >+		uasm_i_mftc0(&p, t3, 2, 4); /* TCHalt */
> >+		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_poll);
> >+		uasm_i_nop(&p);
> >+
> >+		/* Decrement the count of VPEs to be handled */
> >+		uasm_i_addiu(&p, t0, t0, -1);
> >+
> >+		/* Proceed to the next VPE, if there is one */
> >+		uasm_build_label(&l, p, lbl_vpehalt_next);
> >+		uasm_i_srl(&p, t1, t1, 1);
> >+		uasm_il_bnez(&p, &r, t0, lbl_vpehalt_loop);
> >+		uasm_i_addiu(&p, t2, t2, 1);
> >+	}
> >+
> >+	/*
> >+	 * This is the point of no return - this VPE will now proceed to
> >+	 * disable coherence. At this point we *must* be sure that no other
> >+	 * VPE within the core will interfere with the L1 dcache.
> >+	 */
> >+	uasm_build_label(&l, p, lbl_disable_coherence);
> >+
> >+	/* Completion barrier */
> >+	uasm_i_sync(&p, stype_memory);
> >+
> >+	/* Invalidate the L1 icache */
> >+	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].icache,
> >+			      Index_Invalidate_I, lbl_invicache);
> >+
> >+	/* Writeback & invalidate the L1 dcache */
> >+	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].dcache,
> >+			      Index_Writeback_Inv_D, lbl_flushdcache);
> >+
> >+	/*
> >+	 * Disable all but self interventions. The load from COHCTL is defined
> >+	 * by the interAptiv & proAptiv SUMs as ensuring that the operation
> >+	 * resulting from the preceeding store is complete.
> >+	 */
> >+	uasm_i_addiu(&p, t0, zero, 1 << cpu_data[dev->cpu].core);
> >+	uasm_i_sw(&p, t0, 0, r_pcohctl);
> >+	uasm_i_lw(&p, t0, 0, r_pcohctl);
> >+
> >+	/* Sync to ensure previous interventions are complete */
> >+	uasm_i_sync(&p, stype_intervention);
> >+
> >+	/* Disable coherence */
> >+	uasm_i_sw(&p, zero, 0, r_pcohctl);
> >+	uasm_i_lw(&p, t0, 0, r_pcohctl);
> >+
> >+	if (coupled_coherence) {
> >+		/*
> >+		 * Now that coherence is disabled it is safe for all VPEs to
> >+		 * proceed with executing their wait instruction, so this VPE
> >+		 * will go ahead and clear the halt bit of the TCs associated
> >+		 * with all other online VPEs within the core. Start by
> >+		 * initialising variables used throughout the loop, and
> >+		 * skipping the loop entirely if there are no VPEs to handle.
> >+		 *
> >+		 *   t0: number of VPEs left to handle
> >+		 *   t1: (shifted) mask of online VPEs
> >+		 *   t2: current VPE index
> >+		 */
> >+		uasm_i_addiu(&p, t0, r_online, -1);
> >+		uasm_il_beqz(&p, &r, t0, lbl_wait);
> >+		uasm_i_move(&p, t1, r_vpemask);
> >+		uasm_i_move(&p, t2, zero);
> >+
> >+		/*
> >+		 * Now loop through all VPEs within the core checking whether
> >+		 * they are online & not this VPE, which can be determined by
> >+		 * checking the vpe_mask argument. If a VPE is offline or is
> >+		 * this VPE, skip it.
> >+		 */
> >+		uasm_build_label(&l, p, lbl_vpeactivate_loop);
> >+		uasm_i_andi(&p, t3, t1, 1);
> >+		uasm_il_beqz(&p, &r, t3, lbl_vpeactivate_next);
> >+
> >+		/* settc(vpe) */
> >+		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
> >+		uasm_i_ins(&p, t3, t2, 0, 8);
> >+		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
> >+		uasm_i_ehb(&p);
> >+
> >+		/* Clear TCHalt */
> >+		uasm_i_mttc0(&p, zero, 2, 4); /* TCHalt */
> >+
> >+		/* Decrement the count of VPEs to be handled */
> >+		uasm_i_addiu(&p, t0, t0, -1);
> >+
> >+		/* Proceed to the next VPE, if there is one */
> >+		uasm_build_label(&l, p, lbl_vpeactivate_next);
> >+		uasm_i_srl(&p, t1, t1, 1);
> >+		uasm_il_bnez(&p, &r, t0, lbl_vpeactivate_loop);
> >+		uasm_i_addiu(&p, t2, t2, 1);
> >+	}
> >+
> >+	/* Now perform our wait */
> >+	uasm_build_label(&l, p, lbl_wait);
> >+	uasm_i_wait(&p, 0);
> >+
> >+	/*
> >+	 * Re-enable coherence. Note that all coupled VPEs will run this, the
> >+	 * first will actually re-enable coherence & the rest will just be
> >+	 * performing a rather unusual nop.
> >+	 */
> >+	uasm_i_addiu(&p, t0, zero, CM_GCR_Cx_COHERENCE_COHDOMAINEN_MSK);
> >+	uasm_i_sw(&p, t0, 0, r_pcohctl);
> >+	uasm_i_lw(&p, t0, 0, r_pcohctl);
> >+
> >+	/* Ordering barrier */
> >+	uasm_i_sync(&p, stype_ordering);
> >+
> >+	if (coupled_coherence) {
> >+		/* Decrement nc_ready_count */
> >+		uasm_build_label(&l, p, lbl_decready);
> >+		uasm_i_sync(&p, stype_ordering);
> >+		uasm_i_ll(&p, t1, 0, r_pcount);
> >+		uasm_i_addiu(&p, t2, t1, -1);
> >+		uasm_i_sc(&p, t2, 0, r_pcount);
> >+		uasm_il_beqz(&p, &r, t2, lbl_decready);
> >+		uasm_i_move(&p, v0, t1);
> >+	}
> >+
> >+	/* The core is coherent, time to return to C code */
> >+	uasm_i_jr(&p, ra);
> >+	uasm_i_nop(&p);
> >+
> >+	/* Ensure the code didn't exceed the resources allocated for it */
> >+	BUG_ON((p - buf) > max_instrs);
> >+	BUG_ON((l - labels) > ARRAY_SIZE(labels));
> >+	BUG_ON((r - relocs) > ARRAY_SIZE(relocs));
> >+
> >+	/* Patch branch offsets */
> >+	uasm_resolve_relocs(relocs, labels);
> >+
> >+	/* Flush the icache */
> >+	local_flush_icache_range((unsigned long)buf, (unsigned long)p);
> >+
> >+	return buf;
> >+}
> 
> The two functions above should go to somewhere in arch/mips.

Well I guess they can, but they only make any sense in the context of
cpuidle which is why I implemented them here.

> 
> >+
> >+static void __init cps_cpuidle_unregister(void)
> >+{
> >+	int cpu;
> >+	struct cpuidle_device *device;
> >+	cps_nc_entry_fn *fn;
> >+
> >+	for_each_possible_cpu(cpu) {
> >+		device = &per_cpu(cpuidle_dev, cpu);
> >+		cpuidle_unregister_device(device);
> >+
> >+		/* Free entry code */
> >+		fn = &per_cpu(ncwait_asm_enter, cpu_data[cpu].core);
> >+		kfree(*fn);
> >+		*fn = NULL;
> >+	}
> >+
> >+	cpuidle_unregister_driver(&cps_driver);
> >+}
> >+
> >+static int __init cps_cpuidle_init(void)
> >+{
> >+	int err, cpu, core, i;
> >+	struct cpuidle_device *device;
> >+	void *core_entry;
> >+
> >+	/*
> >+	 * If interrupts were enabled whilst running the wait instruction then
> >+	 * the VPE may end up processing interrupts whilst non-coherent.
> >+	 */
> >+	if (cpu_wait != r4k_wait_irqoff) {
> >+		pr_warn("cpuidle-cps requires that masked interrupts restart the CPU pipeline following a wait\n");
> >+		return -ENODEV;
> >+	}
> >+
> >+	/* Detect appropriate sync types for the system */
> >+	switch (current_cpu_data.cputype) {
> >+	case CPU_INTERAPTIV:
> >+	case CPU_PROAPTIV:
> >+		stype_intervention = 0x2;
> >+		stype_memory = 0x3;
> >+		stype_ordering = 0x10;
> >+		break;
> >+
> >+	default:
> >+		pr_warn("cpuidle-cps using heavyweight sync 0\n");
> >+	}
> >+
> >+	/*
> >+	 * Set the coupled flag on the appropriate states if this system
> >+	 * requires it.
> >+	 */
> >+	if (coupled_coherence)
> >+		for (i = 1; i < cps_driver.state_count; i++)
> >+			cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED;
> 
> I am not sure CPUIDLE_FLAG_COUPLED is the solution for this driver. IIUC,
> with the IPI above and the wakeup sync with the couple states, this driver
> is waking up everybody instead of sleeping as much as possible.
> 
> I would recommend to have a look at a different approach, like the one used
> for cpuidle-ux500.c.

Ok it looks like that just counts & performs work only on the last
online CPU to run the code. As before that could work but only by
disregarding any differentiation between coherent & non-coherent wait
states at levels above the driver.

Paul

> 
> >+
> >+	err = cpuidle_register_driver(&cps_driver);
> >+	if (err) {
> >+		pr_err("Failed to register CPS cpuidle driver\n");
> >+		return err;
> >+	}
> >+
> >+	for_each_possible_cpu(cpu) {
> >+		core = cpu_data[cpu].core;
> >+		device = &per_cpu(cpuidle_dev, cpu);
> >+		device->cpu = cpu;
> >+#ifdef CONFIG_MIPS_MT
> >+		cpumask_copy(&device->coupled_cpus, &cpu_sibling_map[cpu]);
> >+#endif
> >+		if (!per_cpu(ncwait_asm_enter, core)) {
> >+			core_entry = cps_gen_entry_code(device);
> >+			if (!core_entry) {
> >+				pr_err("Failed to generate core %u entry\n",
> >+				       core);
> >+				err = -ENOMEM;
> >+				goto err_out;
> >+			}
> >+			per_cpu(ncwait_asm_enter, core) = core_entry;
> >+		}
> >+
> >+		err = cpuidle_register_device(device);
> >+		if (err) {
> >+			pr_err("Failed to register CPU%d cpuidle device\n",
> >+			       cpu);
> >+			goto err_out;
> >+		}
> >+	}
> >+
> >+	return 0;
> >+err_out:
> >+	cps_cpuidle_unregister();
> >+	return err;
> >+}
> >+device_initcall(cps_cpuidle_init);
> >
> 
> 
> -- 
>  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
> 
> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
> <http://twitter.com/#!/linaroorg> Twitter |
> <http://www.linaro.org/linaro-blog/> Blog
> 

Re: [PATCH 10/10] cpuidle: cpuidle driver for MIPS CPS

[Paul Burton]

On Tue, Feb 25, 2014 at 04:33:46PM +0100, Daniel Lezcano wrote:
> On 01/15/2014 02:55 PM, Paul Burton wrote:
> >This patch introduces a cpuidle driver implementation for the MIPS
> >Coherent Processing System (ie. Coherence Manager, Cluster Power
> >Controller). It allows for use of the following idle states:
> >
> >  - Coherent wait. This is the usual MIPS wait instruction.
> >
> >  - Non-coherent wait. In this state a core will disable coherency with
> >    the rest of the system before running the wait instruction. This
> >    eliminates coherence interventions which would typically be used to
> >    keep cores coherent.
> >
> >These two states lay the groundwork for deeper states to be implemented
> >later, since all deeper states require the core to become non-coherent.
> >
> >Signed-off-by: Paul Burton <paul.burton@imgtec.com>
> >Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
> >Cc: linux-pm@vger.kernel.org
> >---
> >  drivers/cpuidle/Kconfig            |   5 +
> >  drivers/cpuidle/Kconfig.mips       |  14 +
> >  drivers/cpuidle/Makefile           |   3 +
> >  drivers/cpuidle/cpuidle-mips-cps.c | 545 +++++++++++++++++++++++++++++++++++++
> >  4 files changed, 567 insertions(+)
> >  create mode 100644 drivers/cpuidle/Kconfig.mips
> >  create mode 100644 drivers/cpuidle/cpuidle-mips-cps.c
> >
> >diff --git a/drivers/cpuidle/Kconfig b/drivers/cpuidle/Kconfig
> >index b3fb81d..11ff281 100644
> >--- a/drivers/cpuidle/Kconfig
> >+++ b/drivers/cpuidle/Kconfig
> >@@ -35,6 +35,11 @@ depends on ARM
> >  source "drivers/cpuidle/Kconfig.arm"
> >  endmenu
> >
> >+menu "MIPS CPU Idle Drivers"
> >+depends on MIPS
> >+source "drivers/cpuidle/Kconfig.mips"
> >+endmenu
> >+
> >  endif
> >
> >  config ARCH_NEEDS_CPU_IDLE_COUPLED
> >diff --git a/drivers/cpuidle/Kconfig.mips b/drivers/cpuidle/Kconfig.mips
> >new file mode 100644
> >index 0000000..dc96691
> >--- /dev/null
> >+++ b/drivers/cpuidle/Kconfig.mips
> >@@ -0,0 +1,14 @@
> >+#
> >+# MIPS CPU Idle drivers
> >+#
> >+
> >+config MIPS_CPS_CPUIDLE
> >+	bool "Support for MIPS Coherent Processing Systems"
> >+	depends on SYS_SUPPORTS_MIPS_CPS && CPU_MIPSR2 && !MIPS_MT_SMTC
> >+	select ARCH_NEEDS_CPU_IDLE_COUPLED if MIPS_MT
> >+	select MIPS_CM
> >+	help
> >+	  Select this option to enable CPU idle driver for systems based
> >+	  around the MIPS Coherent Processing System architecture - that
> >+	  is, those with a Coherence Manager & optionally a Cluster
> >+	  Power Controller.
> >diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile
> >index 527be28..693cd95 100644
> >--- a/drivers/cpuidle/Makefile
> >+++ b/drivers/cpuidle/Makefile
> >@@ -13,3 +13,6 @@ obj-$(CONFIG_ARM_KIRKWOOD_CPUIDLE)	+= cpuidle-kirkwood.o
> >  obj-$(CONFIG_ARM_ZYNQ_CPUIDLE)		+= cpuidle-zynq.o
> >  obj-$(CONFIG_ARM_U8500_CPUIDLE)         += cpuidle-ux500.o
> >  obj-$(CONFIG_ARM_AT91_CPUIDLE)          += cpuidle-at91.o
> >+
> >+# MIPS SoC drivers
> >+obj-$(CONFIG_MIPS_CPS_CPUIDLE)		+= cpuidle-mips-cps.o
> >diff --git a/drivers/cpuidle/cpuidle-mips-cps.c b/drivers/cpuidle/cpuidle-mips-cps.c
> >new file mode 100644
> >index 0000000..a78bfb4
> >--- /dev/null
> >+++ b/drivers/cpuidle/cpuidle-mips-cps.c
> >@@ -0,0 +1,545 @@
> >+/*
> >+ * Copyright (C) 2013 Imagination Technologies
> >+ * Author: Paul Burton <paul.burton@imgtec.com>
> >+ *
> >+ * This program is free software; you can redistribute it and/or modify it
> >+ * under the terms of the GNU General Public License as published by the
> >+ * Free Software Foundation;  either version 2 of the  License, or (at your
> >+ * option) any later version.
> >+ */
> >+
> >+#include <linux/cpuidle.h>
> >+#include <linux/init.h>
> >+#include <linux/kconfig.h>
> >+#include <linux/slab.h>
> >+
> >+#include <asm/cacheflush.h>
> >+#include <asm/cacheops.h>
> >+#include <asm/idle.h>
> >+#include <asm/mips-cm.h>
> >+#include <asm/mipsmtregs.h>
> >+#include <asm/uasm.h>
> 
> The convention is to not include headers from arch. These headers shouldn't
> appear in this driver.
> 

Without accessing those headers I can't really implement anything useful
- entering these idle states by their nature involves architecture
specifics. Would you rather the bulk of the driver is implemented under
arch/mips & the code in drivers/cpuidle simply calls elsewhere to do the
real work?

> >+/*
> >+ * The CM & CPC can only handle coherence & power control on a per-core basis,
> >+ * thus in an MT system the VPEs within each core are coupled and can only
> >+ * enter or exit states requiring CM or CPC assistance in unison.
> >+ */
> >+#ifdef CONFIG_MIPS_MT
> >+# define coupled_coherence cpu_has_mipsmt
> >+#else
> >+# define coupled_coherence 0
> >+#endif
> >+
> >+/*
> >+ * cps_nc_entry_fn - type of a generated non-coherent state entry function
> >+ * @vpe_mask: a bitmap of online coupled VPEs, excluding this one
> >+ * @online: the count of online coupled VPEs (weight of vpe_mask + 1)
> >+ *
> >+ * The code entering & exiting non-coherent states is generated at runtime
> >+ * using uasm, in order to ensure that the compiler cannot insert a stray
> >+ * memory access at an unfortunate time and to allow the generation of optimal
> >+ * core-specific code particularly for cache routines. If coupled_coherence
> >+ * is non-zero, returns the number of VPEs that were in the wait state at the
> >+ * point this VPE left it. Returns garbage if coupled_coherence is zero.
> >+ */
> >+typedef unsigned (*cps_nc_entry_fn)(unsigned vpe_mask, unsigned online);
> >+
> >+/*
> >+ * The entry point of the generated non-coherent wait entry/exit function.
> >+ * Actually per-core rather than per-CPU.
> >+ */
> >+static DEFINE_PER_CPU_READ_MOSTLY(cps_nc_entry_fn, ncwait_asm_enter);
> >+
> >+/*
> >+ * Indicates the number of coupled VPEs ready to operate in a non-coherent
> >+ * state. Actually per-core rather than per-CPU.
> >+ */
> >+static DEFINE_PER_CPU_ALIGNED(u32, nc_ready_count);
> >+
> >+/* A somewhat arbitrary number of labels & relocs for uasm */
> >+static struct uasm_label labels[32] __initdata;
> >+static struct uasm_reloc relocs[32] __initdata;
> >+
> >+/* CPU dependant sync types */
> >+static unsigned stype_intervention;
> >+static unsigned stype_memory;
> >+static unsigned stype_ordering;
> >+
> >+enum mips_reg {
> >+	zero, at, v0, v1, a0, a1, a2, a3,
> >+	t0, t1, t2, t3, t4, t5, t6, t7,
> >+	s0, s1, s2, s3, s4, s5, s6, s7,
> >+	t8, t9, k0, k1, gp, sp, fp, ra,
> >+};
> >+
> >+static int cps_ncwait_enter(struct cpuidle_device *dev,
> >+			    struct cpuidle_driver *drv, int index)
> >+{
> >+	unsigned core = cpu_data[dev->cpu].core;
> >+	unsigned online, first_cpu, num_left;
> >+	cpumask_var_t coupled_mask, vpe_mask;
> >+
> >+	if (!alloc_cpumask_var(&coupled_mask, GFP_KERNEL))
> >+		return -ENOMEM;
> >+
> >+	if (!alloc_cpumask_var(&vpe_mask, GFP_KERNEL)) {
> >+		free_cpumask_var(coupled_mask);
> >+		return -ENOMEM;
> >+	}
> 
> You can't do that in this function where the local irqs are disabled. IMO,
> if you set CONFIG_CPUMASK_OFFSTACK and CONFIG_DEBUG_ATOMIC_SLEEP, you should
> see a kernel warning.

Right you are, it should either use GFP_ATOMIC or just not handle the
off-stack case (which isn't currently used).

> 
> >+	/* Calculate which coupled CPUs (VPEs) are online */
> >+#ifdef CONFIG_MIPS_MT
> >+	cpumask_and(coupled_mask, cpu_online_mask, &dev->coupled_cpus);
> >+	first_cpu = cpumask_first(coupled_mask);
> >+	online = cpumask_weight(coupled_mask);
> >+	cpumask_clear_cpu(dev->cpu, coupled_mask);
> >+	cpumask_shift_right(vpe_mask, coupled_mask,
> >+			    cpumask_first(&dev->coupled_cpus));
> 
> What is the purpose of this computation ?

If you read through the code generated in cps_gen_entry_code, the
vpe_mask is used to indicate the VPEs within the current core which are
both online & not the VPE currently running the code.

> 
> >+#else
> >+	cpumask_clear(coupled_mask);
> >+	cpumask_clear(vpe_mask);
> >+	first_cpu = dev->cpu;
> >+	online = 1;
> >+#endif
> 
> first_cpu is not used.

Right you are. It's used in further work-in-progress patches but I'll
remove it from this one.

> 
> >+	/*
> >+	 * Run the generated entry code. Note that we assume the number of VPEs
> >+	 * within this core does not exceed the width in bits of a long. Since
> >+	 * MVPConf0.PVPE is 4 bits wide this seems like a safe assumption.
> >+	 */
> >+	num_left = per_cpu(ncwait_asm_enter, core)(vpe_mask->bits[0], online);
> >+
> >+	/*
> >+	 * If this VPE is the first to leave the non-coherent wait state then
> >+	 * it needs to wake up any coupled VPEs still running their wait
> >+	 * instruction so that they return to cpuidle,
> 
> Why is it needed ? Can't the other cpus stay idle ?

Not with the current cpuidle code. Please see the end of
cpuidle_enter_state_coupled in drivers/cpuidle/coupled.c. The code waits
for all coupled CPUs to exit the idle state before any of them proceed.
Whilst I suppose it would be possible to modify cpuidle to not require
that, it would leave you with inaccurate residence statistics mentioned
below.

> 
> >       * which can then complete
> >+	 * coordination between the coupled VPEs & provide the governor with
> >+	 * a chance to reflect on the length of time the VPEs were in the
> >+	 * idle state.
> >+	 */
> >+	if (coupled_coherence && (num_left == online))
> >+		arch_send_call_function_ipi_mask(coupled_mask);
> 
> Except there is no choice due to hardware limitations, I don't think this is
> valid.

By nature when one CPU (VPE) within a core leaves a non-coherent state
the rest do too, because as I mentioned coherence is a property of the
core not of individual VPEs. It would be possible to leave the other
VPEs idle if we didn't differentiate between the coherent & non-coherent
wait states, but again not with cpuidle as it is today (due to the
waiting for all CPUs at the end of cpuidle_enter_state_coupled).

> 
> >+	free_cpumask_var(vpe_mask);
> >+	free_cpumask_var(coupled_mask);
> >+	return index;
> >+}
> >+
> >+static struct cpuidle_driver cps_driver = {
> >+	.name			= "cpc_cpuidle",
> >+	.owner			= THIS_MODULE,
> >+	.states = {
> >+		MIPS_CPUIDLE_WAIT_STATE,
> >+		{
> >+			.enter	= cps_ncwait_enter,
> >+			.exit_latency		= 200,
> >+			.target_residency	= 450,
> >+			.flags	= CPUIDLE_FLAG_TIME_VALID,
> >+			.name	= "nc-wait",
> >+			.desc	= "non-coherent MIPS wait",
> >+		},
> >+	},
> >+	.state_count		= 2,
> >+	.safe_state_index	= 0,
> >+};
> 
> 
> 
> >+static void __init cps_gen_cache_routine(u32 **pp, struct uasm_label **pl,
> >+					 struct uasm_reloc **pr,
> >+					 const struct cache_desc *cache,
> >+					 unsigned op, int lbl)
> >+{
> >+	unsigned cache_size = cache->ways << cache->waybit;
> >+	unsigned i;
> >+	const unsigned unroll_lines = 32;
> >+
> >+	/* If the cache isn't present this function has it easy */
> >+	if (cache->flags & MIPS_CACHE_NOT_PRESENT)
> >+		return;
> >+
> >+	/* Load base address */
> >+	UASM_i_LA(pp, t0, (long)CKSEG0);
> >+
> >+	/* Calculate end address */
> >+	if (cache_size < 0x8000)
> >+		uasm_i_addiu(pp, t1, t0, cache_size);
> >+	else
> >+		UASM_i_LA(pp, t1, (long)(CKSEG0 + cache_size));
> >+
> >+	/* Start of cache op loop */
> >+	uasm_build_label(pl, *pp, lbl);
> >+
> >+	/* Generate the cache ops */
> >+	for (i = 0; i < unroll_lines; i++)
> >+		uasm_i_cache(pp, op, i * cache->linesz, t0);
> >+
> >+	/* Update the base address */
> >+	uasm_i_addiu(pp, t0, t0, unroll_lines * cache->linesz);
> >+
> >+	/* Loop if we haven't reached the end address yet */
> >+	uasm_il_bne(pp, pr, t0, t1, lbl);
> >+	uasm_i_nop(pp);
> >+}
> >+
> >+static void * __init cps_gen_entry_code(struct cpuidle_device *dev)
> >+{
> >+	unsigned core = cpu_data[dev->cpu].core;
> >+	struct uasm_label *l = labels;
> >+	struct uasm_reloc *r = relocs;
> >+	u32 *buf, *p;
> >+	const unsigned r_vpemask = a0;
> >+	const unsigned r_online = a1;
> >+	const unsigned r_pcount = t6;
> >+	const unsigned r_pcohctl = t7;
> >+	const unsigned max_instrs = 256;
> >+	enum {
> >+		lbl_incready = 1,
> >+		lbl_lastvpe,
> >+		lbl_vpehalt_loop,
> >+		lbl_vpehalt_poll,
> >+		lbl_vpehalt_next,
> >+		lbl_disable_coherence,
> >+		lbl_invicache,
> >+		lbl_flushdcache,
> >+		lbl_vpeactivate_loop,
> >+		lbl_vpeactivate_next,
> >+		lbl_wait,
> >+		lbl_decready,
> >+	};
> >+
> >+	/* Allocate a buffer to hold the generated code */
> >+	p = buf = kcalloc(max_instrs, sizeof(u32), GFP_KERNEL);
> >+	if (!buf)
> >+		return NULL;
> >+
> >+	/* Clear labels & relocs ready for (re)use */
> >+	memset(labels, 0, sizeof(labels));
> >+	memset(relocs, 0, sizeof(relocs));
> >+
> >+	/*
> >+	 * Load address of the CM GCR_CL_COHERENCE register. This is done early
> >+	 * because it's needed in both the enable & disable coherence steps but
> >+	 * in the coupled case the enable step will only run on one VPE.
> >+	 */
> >+	UASM_i_LA(&p, r_pcohctl, (long)addr_gcr_cl_coherence());
> >+
> >+	if (coupled_coherence) {
> >+		/* Load address of nc_ready_count */
> >+		UASM_i_LA(&p, r_pcount, (long)&per_cpu(nc_ready_count, core));
> >+
> >+		/* Increment nc_ready_count */
> >+		uasm_build_label(&l, p, lbl_incready);
> >+		uasm_i_sync(&p, stype_ordering);
> >+		uasm_i_ll(&p, t1, 0, r_pcount);
> >+		uasm_i_addiu(&p, t2, t1, 1);
> >+		uasm_i_sc(&p, t2, 0, r_pcount);
> >+		uasm_il_beqz(&p, &r, t2, lbl_incready);
> >+		uasm_i_addiu(&p, t1, t1, 1);
> >+
> >+		/*
> >+		 * If this is the last VPE to become ready for non-coherence
> >+		 * then it should branch below.
> >+		 */
> >+		uasm_il_beq(&p, &r, t1, r_online, lbl_lastvpe);
> >+		uasm_i_nop(&p);
> >+
> >+		/*
> >+		 * Otherwise this is not the last VPE to become ready for
> >+		 * non-coherence. It needs to wait until coherence has been
> >+		 * disabled before executing a wait instruction, otherwise it
> >+		 * may return from wait quickly & re-enable coherence causing
> >+		 * a race with the VPE disabling coherence. It can't simply
> >+		 * poll the CPC sequencer for a non-coherent state as that
> >+		 * would race with any other VPE which may spot the
> >+		 * non-coherent state, run wait, return quickly & re-enable
> >+		 * coherence before this VPE ever saw the non-coherent state.
> >+		 * Instead this VPE will halt its TC such that it ceases to
> >+		 * execute for the moment.
> >+		 */
> >+		uasm_i_addiu(&p, t0, zero, TCHALT_H);
> >+		uasm_i_mtc0(&p, t0, 2, 4); /* TCHalt */
> >+
> >+		/* instruction_hazard(), to ensure the TC halts */
> >+		UASM_i_LA(&p, t0, (long)p + 12);
> >+		uasm_i_jr_hb(&p, t0);
> >+		uasm_i_nop(&p);
> >+
> >+		/*
> >+		 * The VPE which disables coherence will then clear the halt
> >+		 * bit for this VPE's TC once coherence has been disabled and
> >+		 * it can safely proceed to execute the wait instruction.
> >+		 */
> >+		uasm_il_b(&p, &r, lbl_wait);
> >+		uasm_i_nop(&p);
> >+
> >+		/*
> >+		 * The last VPE to increment nc_ready_count will continue from
> >+		 * here and must spin until all other VPEs within the core have
> >+		 * been halted, at which point it can be sure that it is safe
> >+		 * to disable coherence.
> >+		 *
> >+		 *   t0: number of VPEs left to handle
> >+		 *   t1: (shifted) mask of online VPEs
> >+		 *   t2: current VPE index
> >+		 */
> >+		uasm_build_label(&l, p, lbl_lastvpe);
> >+		uasm_i_addiu(&p, t0, r_online, -1);
> >+		uasm_il_beqz(&p, &r, t0, lbl_disable_coherence);
> >+		uasm_i_move(&p, t1, r_vpemask);
> >+		uasm_i_move(&p, t2, zero);
> >+
> >+		/*
> >+		 * Now loop through all VPEs within the core checking whether
> >+		 * they are online & not this VPE, which can be determined by
> >+		 * checking the vpe_mask argument. If a VPE is offline or is
> >+		 * this VPE, skip it.
> >+		 */
> >+		uasm_build_label(&l, p, lbl_vpehalt_loop);
> >+		uasm_i_andi(&p, t3, t1, 1);
> >+		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_next);
> >+
> >+		/* settc(vpe) */
> >+		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
> >+		uasm_i_ins(&p, t3, t2, 0, 8);
> >+		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
> >+		uasm_i_ehb(&p);
> >+
> >+		/*
> >+		 * It's very likely that the VPE has already halted itself
> >+		 * by now, but there's theoretically a chance that it may not
> >+		 * have. Wait until the VPE's TC is halted.
> >+		 */
> >+		uasm_build_label(&l, p, lbl_vpehalt_poll);
> >+		uasm_i_mftc0(&p, t3, 2, 4); /* TCHalt */
> >+		uasm_il_beqz(&p, &r, t3, lbl_vpehalt_poll);
> >+		uasm_i_nop(&p);
> >+
> >+		/* Decrement the count of VPEs to be handled */
> >+		uasm_i_addiu(&p, t0, t0, -1);
> >+
> >+		/* Proceed to the next VPE, if there is one */
> >+		uasm_build_label(&l, p, lbl_vpehalt_next);
> >+		uasm_i_srl(&p, t1, t1, 1);
> >+		uasm_il_bnez(&p, &r, t0, lbl_vpehalt_loop);
> >+		uasm_i_addiu(&p, t2, t2, 1);
> >+	}
> >+
> >+	/*
> >+	 * This is the point of no return - this VPE will now proceed to
> >+	 * disable coherence. At this point we *must* be sure that no other
> >+	 * VPE within the core will interfere with the L1 dcache.
> >+	 */
> >+	uasm_build_label(&l, p, lbl_disable_coherence);
> >+
> >+	/* Completion barrier */
> >+	uasm_i_sync(&p, stype_memory);
> >+
> >+	/* Invalidate the L1 icache */
> >+	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].icache,
> >+			      Index_Invalidate_I, lbl_invicache);
> >+
> >+	/* Writeback & invalidate the L1 dcache */
> >+	cps_gen_cache_routine(&p, &l, &r, &cpu_data[dev->cpu].dcache,
> >+			      Index_Writeback_Inv_D, lbl_flushdcache);
> >+
> >+	/*
> >+	 * Disable all but self interventions. The load from COHCTL is defined
> >+	 * by the interAptiv & proAptiv SUMs as ensuring that the operation
> >+	 * resulting from the preceeding store is complete.
> >+	 */
> >+	uasm_i_addiu(&p, t0, zero, 1 << cpu_data[dev->cpu].core);
> >+	uasm_i_sw(&p, t0, 0, r_pcohctl);
> >+	uasm_i_lw(&p, t0, 0, r_pcohctl);
> >+
> >+	/* Sync to ensure previous interventions are complete */
> >+	uasm_i_sync(&p, stype_intervention);
> >+
> >+	/* Disable coherence */
> >+	uasm_i_sw(&p, zero, 0, r_pcohctl);
> >+	uasm_i_lw(&p, t0, 0, r_pcohctl);
> >+
> >+	if (coupled_coherence) {
> >+		/*
> >+		 * Now that coherence is disabled it is safe for all VPEs to
> >+		 * proceed with executing their wait instruction, so this VPE
> >+		 * will go ahead and clear the halt bit of the TCs associated
> >+		 * with all other online VPEs within the core. Start by
> >+		 * initialising variables used throughout the loop, and
> >+		 * skipping the loop entirely if there are no VPEs to handle.
> >+		 *
> >+		 *   t0: number of VPEs left to handle
> >+		 *   t1: (shifted) mask of online VPEs
> >+		 *   t2: current VPE index
> >+		 */
> >+		uasm_i_addiu(&p, t0, r_online, -1);
> >+		uasm_il_beqz(&p, &r, t0, lbl_wait);
> >+		uasm_i_move(&p, t1, r_vpemask);
> >+		uasm_i_move(&p, t2, zero);
> >+
> >+		/*
> >+		 * Now loop through all VPEs within the core checking whether
> >+		 * they are online & not this VPE, which can be determined by
> >+		 * checking the vpe_mask argument. If a VPE is offline or is
> >+		 * this VPE, skip it.
> >+		 */
> >+		uasm_build_label(&l, p, lbl_vpeactivate_loop);
> >+		uasm_i_andi(&p, t3, t1, 1);
> >+		uasm_il_beqz(&p, &r, t3, lbl_vpeactivate_next);
> >+
> >+		/* settc(vpe) */
> >+		uasm_i_mfc0(&p, t3, 1, 1); /* VPEControl */
> >+		uasm_i_ins(&p, t3, t2, 0, 8);
> >+		uasm_i_mtc0(&p, t3, 1, 1); /* VPEControl */
> >+		uasm_i_ehb(&p);
> >+
> >+		/* Clear TCHalt */
> >+		uasm_i_mttc0(&p, zero, 2, 4); /* TCHalt */
> >+
> >+		/* Decrement the count of VPEs to be handled */
> >+		uasm_i_addiu(&p, t0, t0, -1);
> >+
> >+		/* Proceed to the next VPE, if there is one */
> >+		uasm_build_label(&l, p, lbl_vpeactivate_next);
> >+		uasm_i_srl(&p, t1, t1, 1);
> >+		uasm_il_bnez(&p, &r, t0, lbl_vpeactivate_loop);
> >+		uasm_i_addiu(&p, t2, t2, 1);
> >+	}
> >+
> >+	/* Now perform our wait */
> >+	uasm_build_label(&l, p, lbl_wait);
> >+	uasm_i_wait(&p, 0);
> >+
> >+	/*
> >+	 * Re-enable coherence. Note that all coupled VPEs will run this, the
> >+	 * first will actually re-enable coherence & the rest will just be
> >+	 * performing a rather unusual nop.
> >+	 */
> >+	uasm_i_addiu(&p, t0, zero, CM_GCR_Cx_COHERENCE_COHDOMAINEN_MSK);
> >+	uasm_i_sw(&p, t0, 0, r_pcohctl);
> >+	uasm_i_lw(&p, t0, 0, r_pcohctl);
> >+
> >+	/* Ordering barrier */
> >+	uasm_i_sync(&p, stype_ordering);
> >+
> >+	if (coupled_coherence) {
> >+		/* Decrement nc_ready_count */
> >+		uasm_build_label(&l, p, lbl_decready);
> >+		uasm_i_sync(&p, stype_ordering);
> >+		uasm_i_ll(&p, t1, 0, r_pcount);
> >+		uasm_i_addiu(&p, t2, t1, -1);
> >+		uasm_i_sc(&p, t2, 0, r_pcount);
> >+		uasm_il_beqz(&p, &r, t2, lbl_decready);
> >+		uasm_i_move(&p, v0, t1);
> >+	}
> >+
> >+	/* The core is coherent, time to return to C code */
> >+	uasm_i_jr(&p, ra);
> >+	uasm_i_nop(&p);
> >+
> >+	/* Ensure the code didn't exceed the resources allocated for it */
> >+	BUG_ON((p - buf) > max_instrs);
> >+	BUG_ON((l - labels) > ARRAY_SIZE(labels));
> >+	BUG_ON((r - relocs) > ARRAY_SIZE(relocs));
> >+
> >+	/* Patch branch offsets */
> >+	uasm_resolve_relocs(relocs, labels);
> >+
> >+	/* Flush the icache */
> >+	local_flush_icache_range((unsigned long)buf, (unsigned long)p);
> >+
> >+	return buf;
> >+}
> 
> The two functions above should go to somewhere in arch/mips.

Well I guess they can, but they only make any sense in the context of
cpuidle which is why I implemented them here.

> 
> >+
> >+static void __init cps_cpuidle_unregister(void)
> >+{
> >+	int cpu;
> >+	struct cpuidle_device *device;
> >+	cps_nc_entry_fn *fn;
> >+
> >+	for_each_possible_cpu(cpu) {
> >+		device = &per_cpu(cpuidle_dev, cpu);
> >+		cpuidle_unregister_device(device);
> >+
> >+		/* Free entry code */
> >+		fn = &per_cpu(ncwait_asm_enter, cpu_data[cpu].core);
> >+		kfree(*fn);
> >+		*fn = NULL;
> >+	}
> >+
> >+	cpuidle_unregister_driver(&cps_driver);
> >+}
> >+
> >+static int __init cps_cpuidle_init(void)
> >+{
> >+	int err, cpu, core, i;
> >+	struct cpuidle_device *device;
> >+	void *core_entry;
> >+
> >+	/*
> >+	 * If interrupts were enabled whilst running the wait instruction then
> >+	 * the VPE may end up processing interrupts whilst non-coherent.
> >+	 */
> >+	if (cpu_wait != r4k_wait_irqoff) {
> >+		pr_warn("cpuidle-cps requires that masked interrupts restart the CPU pipeline following a wait\n");
> >+		return -ENODEV;
> >+	}
> >+
> >+	/* Detect appropriate sync types for the system */
> >+	switch (current_cpu_data.cputype) {
> >+	case CPU_INTERAPTIV:
> >+	case CPU_PROAPTIV:
> >+		stype_intervention = 0x2;
> >+		stype_memory = 0x3;
> >+		stype_ordering = 0x10;
> >+		break;
> >+
> >+	default:
> >+		pr_warn("cpuidle-cps using heavyweight sync 0\n");
> >+	}
> >+
> >+	/*
> >+	 * Set the coupled flag on the appropriate states if this system
> >+	 * requires it.
> >+	 */
> >+	if (coupled_coherence)
> >+		for (i = 1; i < cps_driver.state_count; i++)
> >+			cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED;
> 
> I am not sure CPUIDLE_FLAG_COUPLED is the solution for this driver. IIUC,
> with the IPI above and the wakeup sync with the couple states, this driver
> is waking up everybody instead of sleeping as much as possible.
> 
> I would recommend to have a look at a different approach, like the one used
> for cpuidle-ux500.c.

Ok it looks like that just counts & performs work only on the last
online CPU to run the code. As before that could work but only by
disregarding any differentiation between coherent & non-coherent wait
states at levels above the driver.

Paul

> 
> >+
> >+	err = cpuidle_register_driver(&cps_driver);
> >+	if (err) {
> >+		pr_err("Failed to register CPS cpuidle driver\n");
> >+		return err;
> >+	}
> >+
> >+	for_each_possible_cpu(cpu) {
> >+		core = cpu_data[cpu].core;
> >+		device = &per_cpu(cpuidle_dev, cpu);
> >+		device->cpu = cpu;
> >+#ifdef CONFIG_MIPS_MT
> >+		cpumask_copy(&device->coupled_cpus, &cpu_sibling_map[cpu]);
> >+#endif
> >+		if (!per_cpu(ncwait_asm_enter, core)) {
> >+			core_entry = cps_gen_entry_code(device);
> >+			if (!core_entry) {
> >+				pr_err("Failed to generate core %u entry\n",
> >+				       core);
> >+				err = -ENOMEM;
> >+				goto err_out;
> >+			}
> >+			per_cpu(ncwait_asm_enter, core) = core_entry;
> >+		}
> >+
> >+		err = cpuidle_register_device(device);
> >+		if (err) {
> >+			pr_err("Failed to register CPU%d cpuidle device\n",
> >+			       cpu);
> >+			goto err_out;
> >+		}
> >+	}
> >+
> >+	return 0;
> >+err_out:
> >+	cps_cpuidle_unregister();
> >+	return err;
> >+}
> >+device_initcall(cps_cpuidle_init);
> >
> 
> 
> -- 
>  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
> 
> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
> <http://twitter.com/#!/linaroorg> Twitter |
> <http://www.linaro.org/linaro-blog/> Blog
> 

Re: [PATCH 10/10] cpuidle: cpuidle driver for MIPS CPS

[Daniel Lezcano]

On 02/25/2014 11:12 PM, Paul Burton wrote:
> On Tue, Feb 25, 2014 at 04:33:46PM +0100, Daniel Lezcano wrote:
>> On 01/15/2014 02:55 PM, Paul Burton wrote:
>>> This patch introduces a cpuidle driver implementation for the MIPS
>>> Coherent Processing System (ie. Coherence Manager, Cluster Power
>>> Controller). It allows for use of the following idle states:
>>>
>>>   - Coherent wait. This is the usual MIPS wait instruction.
>>>
>>>   - Non-coherent wait. In this state a core will disable coherency with
>>>     the rest of the system before running the wait instruction. This
>>>     eliminates coherence interventions which would typically be used to
>>>     keep cores coherent.
>>>
>>> These two states lay the groundwork for deeper states to be implemented
>>> later, since all deeper states require the core to become non-coherent.
>>>
>>> Signed-off-by: Paul Burton <paul.burton@imgtec.com>
>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>> Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
>>> Cc: linux-pm@vger.kernel.org
>>> ---
>>>   drivers/cpuidle/Kconfig            |   5 +
>>>   drivers/cpuidle/Kconfig.mips       |  14 +
>>>   drivers/cpuidle/Makefile           |   3 +
>>>   drivers/cpuidle/cpuidle-mips-cps.c | 545 +++++++++++++++++++++++++++++++++++++
>>>   4 files changed, 567 insertions(+)
>>>   create mode 100644 drivers/cpuidle/Kconfig.mips
>>>   create mode 100644 drivers/cpuidle/cpuidle-mips-cps.c
>>>
>>> diff --git a/drivers/cpuidle/Kconfig b/drivers/cpuidle/Kconfig
>>> index b3fb81d..11ff281 100644
>>> --- a/drivers/cpuidle/Kconfig
>>> +++ b/drivers/cpuidle/Kconfig
>>> @@ -35,6 +35,11 @@ depends on ARM
>>>   source "drivers/cpuidle/Kconfig.arm"
>>>   endmenu
>>>
>>> +menu "MIPS CPU Idle Drivers"
>>> +depends on MIPS
>>> +source "drivers/cpuidle/Kconfig.mips"
>>> +endmenu
>>> +
>>>   endif
>>>
>>>   config ARCH_NEEDS_CPU_IDLE_COUPLED
>>> diff --git a/drivers/cpuidle/Kconfig.mips b/drivers/cpuidle/Kconfig.mips
>>> new file mode 100644
>>> index 0000000..dc96691
>>> --- /dev/null
>>> +++ b/drivers/cpuidle/Kconfig.mips
>>> @@ -0,0 +1,14 @@
>>> +#
>>> +# MIPS CPU Idle drivers
>>> +#
>>> +
>>> +config MIPS_CPS_CPUIDLE
>>> +	bool "Support for MIPS Coherent Processing Systems"
>>> +	depends on SYS_SUPPORTS_MIPS_CPS && CPU_MIPSR2 && !MIPS_MT_SMTC
>>> +	select ARCH_NEEDS_CPU_IDLE_COUPLED if MIPS_MT
>>> +	select MIPS_CM
>>> +	help
>>> +	  Select this option to enable CPU idle driver for systems based
>>> +	  around the MIPS Coherent Processing System architecture - that
>>> +	  is, those with a Coherence Manager & optionally a Cluster
>>> +	  Power Controller.
>>> diff --git a/drivers/cpuidle/Makefile b/drivers/cpuidle/Makefile
>>> index 527be28..693cd95 100644
>>> --- a/drivers/cpuidle/Makefile
>>> +++ b/drivers/cpuidle/Makefile
>>> @@ -13,3 +13,6 @@ obj-$(CONFIG_ARM_KIRKWOOD_CPUIDLE)	+= cpuidle-kirkwood.o
>>>   obj-$(CONFIG_ARM_ZYNQ_CPUIDLE)		+= cpuidle-zynq.o
>>>   obj-$(CONFIG_ARM_U8500_CPUIDLE)         += cpuidle-ux500.o
>>>   obj-$(CONFIG_ARM_AT91_CPUIDLE)          += cpuidle-at91.o
>>> +
>>> +# MIPS SoC drivers
>>> +obj-$(CONFIG_MIPS_CPS_CPUIDLE)		+= cpuidle-mips-cps.o
>>> diff --git a/drivers/cpuidle/cpuidle-mips-cps.c b/drivers/cpuidle/cpuidle-mips-cps.c
>>> new file mode 100644
>>> index 0000000..a78bfb4
>>> --- /dev/null
>>> +++ b/drivers/cpuidle/cpuidle-mips-cps.c
>>> @@ -0,0 +1,545 @@
>>> +/*
>>> + * Copyright (C) 2013 Imagination Technologies
>>> + * Author: Paul Burton <paul.burton@imgtec.com>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify it
>>> + * under the terms of the GNU General Public License as published by the
>>> + * Free Software Foundation;  either version 2 of the  License, or (at your
>>> + * option) any later version.
>>> + */
>>> +
>>> +#include <linux/cpuidle.h>
>>> +#include <linux/init.h>
>>> +#include <linux/kconfig.h>
>>> +#include <linux/slab.h>
>>> +
>>> +#include <asm/cacheflush.h>
>>> +#include <asm/cacheops.h>
>>> +#include <asm/idle.h>
>>> +#include <asm/mips-cm.h>
>>> +#include <asm/mipsmtregs.h>
>>> +#include <asm/uasm.h>
>>
>> The convention is to not include headers from arch. These headers shouldn't
>> appear in this driver.
>>
>
> Without accessing those headers I can't really implement anything useful
> - entering these idle states by their nature involves architecture
> specifics. Would you rather the bulk of the driver is implemented under
> arch/mips & the code in drivers/cpuidle simply calls elsewhere to do the
> real work?

Usually, this is how it is done. If you can move the 'asm' code inside 
eg. pm.c somewhere in arch/mips and invoke from pm functions from 
cpuidle through a simple header that would be great.

>>> +/*
>>> + * The CM & CPC can only handle coherence & power control on a per-core basis,
>>> + * thus in an MT system the VPEs within each core are coupled and can only
>>> + * enter or exit states requiring CM or CPC assistance in unison.
>>> + */
>>> +#ifdef CONFIG_MIPS_MT
>>> +# define coupled_coherence cpu_has_mipsmt
>>> +#else
>>> +# define coupled_coherence 0
>>> +#endif
>>> +
>>> +/*
>>> + * cps_nc_entry_fn - type of a generated non-coherent state entry function
>>> + * @vpe_mask: a bitmap of online coupled VPEs, excluding this one
>>> + * @online: the count of online coupled VPEs (weight of vpe_mask + 1)
>>> + *
>>> + * The code entering & exiting non-coherent states is generated at runtime
>>> + * using uasm, in order to ensure that the compiler cannot insert a stray
>>> + * memory access at an unfortunate time and to allow the generation of optimal
>>> + * core-specific code particularly for cache routines. If coupled_coherence
>>> + * is non-zero, returns the number of VPEs that were in the wait state at the
>>> + * point this VPE left it. Returns garbage if coupled_coherence is zero.
>>> + */
>>> +typedef unsigned (*cps_nc_entry_fn)(unsigned vpe_mask, unsigned online);
>>> +
>>> +/*
>>> + * The entry point of the generated non-coherent wait entry/exit function.
>>> + * Actually per-core rather than per-CPU.
>>> + */
>>> +static DEFINE_PER_CPU_READ_MOSTLY(cps_nc_entry_fn, ncwait_asm_enter);
>>> +
>>> +/*
>>> + * Indicates the number of coupled VPEs ready to operate in a non-coherent
>>> + * state. Actually per-core rather than per-CPU.
>>> + */
>>> +static DEFINE_PER_CPU_ALIGNED(u32, nc_ready_count);
>>> +
>>> +/* A somewhat arbitrary number of labels & relocs for uasm */
>>> +static struct uasm_label labels[32] __initdata;
>>> +static struct uasm_reloc relocs[32] __initdata;
>>> +
>>> +/* CPU dependant sync types */
>>> +static unsigned stype_intervention;
>>> +static unsigned stype_memory;
>>> +static unsigned stype_ordering;
>>> +
>>> +enum mips_reg {
>>> +	zero, at, v0, v1, a0, a1, a2, a3,
>>> +	t0, t1, t2, t3, t4, t5, t6, t7,
>>> +	s0, s1, s2, s3, s4, s5, s6, s7,
>>> +	t8, t9, k0, k1, gp, sp, fp, ra,
>>> +};
>>> +
>>> +static int cps_ncwait_enter(struct cpuidle_device *dev,
>>> +			    struct cpuidle_driver *drv, int index)
>>> +{
>>> +	unsigned core = cpu_data[dev->cpu].core;
>>> +	unsigned online, first_cpu, num_left;
>>> +	cpumask_var_t coupled_mask, vpe_mask;
>>> +
>>> +	if (!alloc_cpumask_var(&coupled_mask, GFP_KERNEL))
>>> +		return -ENOMEM;
>>> +
>>> +	if (!alloc_cpumask_var(&vpe_mask, GFP_KERNEL)) {
>>> +		free_cpumask_var(coupled_mask);
>>> +		return -ENOMEM;
>>> +	}
>>
>> You can't do that in this function where the local irqs are disabled. IMO,
>> if you set CONFIG_CPUMASK_OFFSTACK and CONFIG_DEBUG_ATOMIC_SLEEP, you should
>> see a kernel warning.
>
> Right you are, it should either use GFP_ATOMIC or just not handle the
> off-stack case (which isn't currently used).

Why don't you just allocate these cpumasks at init time instead of 
allocating / freeing them in the fast path ?

>>> +	/* Calculate which coupled CPUs (VPEs) are online */
>>> +#ifdef CONFIG_MIPS_MT
>>> +	cpumask_and(coupled_mask, cpu_online_mask, &dev->coupled_cpus);
>>> +	first_cpu = cpumask_first(coupled_mask);
>>> +	online = cpumask_weight(coupled_mask);
>>> +	cpumask_clear_cpu(dev->cpu, coupled_mask);
>>> +	cpumask_shift_right(vpe_mask, coupled_mask,
>>> +			    cpumask_first(&dev->coupled_cpus));
>>
>> What is the purpose of this computation ?
>
> If you read through the code generated in cps_gen_entry_code, the
> vpe_mask is used to indicate the VPEs within the current core which are
> both online & not the VPE currently running the code.

Ok, thanks for the clarification.

Shouldn't 'online = cpumask_weight(coupled_mask)' be after clearing the 
current cpu ? Otherwise, online has also the current cpu which, IIUC, 
shouldn't be included, no ?

>>> +#else
>>> +	cpumask_clear(coupled_mask);
>>> +	cpumask_clear(vpe_mask);
>>> +	first_cpu = dev->cpu;
>>> +	online = 1;
>>> +#endif
>>
>> first_cpu is not used.
>
> Right you are. It's used in further work-in-progress patches but I'll
> remove it from this one.
>
>>
>>> +	/*
>>> +	 * Run the generated entry code. Note that we assume the number of VPEs
>>> +	 * within this core does not exceed the width in bits of a long. Since
>>> +	 * MVPConf0.PVPE is 4 bits wide this seems like a safe assumption.
>>> +	 */
>>> +	num_left = per_cpu(ncwait_asm_enter, core)(vpe_mask->bits[0], online);
>>> +
>>> +	/*
>>> +	 * If this VPE is the first to leave the non-coherent wait state then
>>> +	 * it needs to wake up any coupled VPEs still running their wait
>>> +	 * instruction so that they return to cpuidle,
>>
>> Why is it needed ? Can't the other cpus stay idle ?
>
> Not with the current cpuidle code. Please see the end of
> cpuidle_enter_state_coupled in drivers/cpuidle/coupled.c. The code waits
> for all coupled CPUs to exit the idle state before any of them proceed.
> Whilst I suppose it would be possible to modify cpuidle to not require
> that, it would leave you with inaccurate residence statistics mentioned
> below.
>
>>
>>>        * which can then complete
>>> +	 * coordination between the coupled VPEs & provide the governor with
>>> +	 * a chance to reflect on the length of time the VPEs were in the
>>> +	 * idle state.
>>> +	 */
>>> +	if (coupled_coherence && (num_left == online))
>>> +		arch_send_call_function_ipi_mask(coupled_mask);
>>
>> Except there is no choice due to hardware limitations, I don't think this is
>> valid.
>
> By nature when one CPU (VPE) within a core leaves a non-coherent state
> the rest do too, because as I mentioned coherence is a property of the
> core not of individual VPEs. It would be possible to leave the other
> VPEs idle if we didn't differentiate between the coherent & non-coherent
> wait states, but again not with cpuidle as it is today (due to the
> waiting for all CPUs at the end of cpuidle_enter_state_coupled).

[ ... ]

>>> +
>>> +	/*
>>> +	 * Set the coupled flag on the appropriate states if this system
>>> +	 * requires it.
>>> +	 */
>>> +	if (coupled_coherence)
>>> +		for (i = 1; i < cps_driver.state_count; i++)
>>> +			cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED;
>>
>> I am not sure CPUIDLE_FLAG_COUPLED is the solution for this driver. IIUC,
>> with the IPI above and the wakeup sync with the couple states, this driver
>> is waking up everybody instead of sleeping as much as possible.
>>
>> I would recommend to have a look at a different approach, like the one used
>> for cpuidle-ux500.c.
>
> Ok it looks like that just counts & performs work only on the last
> online CPU to run the code.

Yes, the last-man-standing approach. Usually, the couple idle states are 
used when the processors need to do some extra work or only the cpu0 can 
invoke PM routine for security reason.

In your case, IIUC, we don't care who will call the PM routine and if a 
cpu wakes up, the other ones (read the VPEs belonging to the same core), 
can stay idle until an interrupt occurs.

> As before that could work but only by
> disregarding any differentiation between coherent & non-coherent wait
> states at levels above the driver.

May be you can use the arch private flags for the idle states to 
differentiate coherent or non-coherent wait states ? Or alternatively 
create two drivers and register the right one.

>> --
>>   <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
>>
>> Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
>> <http://twitter.com/#!/linaroorg> Twitter |
>> <http://www.linaro.org/linaro-blog/> Blog
>>


-- 
  <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Re: [PATCH 10/10] cpuidle: cpuidle driver for MIPS CPS

[Paul Burton]

On Wed, Feb 26, 2014 at 03:37:39PM +0100, Daniel Lezcano wrote:
> >>The convention is to not include headers from arch. These headers shouldn't
> >>appear in this driver.
> >>
> >
> >Without accessing those headers I can't really implement anything useful
> >- entering these idle states by their nature involves architecture
> >specifics. Would you rather the bulk of the driver is implemented under
> >arch/mips & the code in drivers/cpuidle simply calls elsewhere to do the
> >real work?
> 
> Usually, this is how it is done. If you can move the 'asm' code inside eg.
> pm.c somewhere in arch/mips and invoke from pm functions from cpuidle
> through a simple header that would be great.

Ok.

> >Right you are, it should either use GFP_ATOMIC or just not handle the
> >off-stack case (which isn't currently used).
> 
> Why don't you just allocate these cpumasks at init time instead of
> allocating / freeing them in the fast path ?

Also a possibility. Right now nothing using this will have off-stack cpu
masks I was just attempting to be cautious by handling the case anyway.
I'll change it one way or another for v2.

> >>>+	/* Calculate which coupled CPUs (VPEs) are online */
> >>>+#ifdef CONFIG_MIPS_MT
> >>>+	cpumask_and(coupled_mask, cpu_online_mask, &dev->coupled_cpus);
> >>>+	first_cpu = cpumask_first(coupled_mask);
> >>>+	online = cpumask_weight(coupled_mask);
> >>>+	cpumask_clear_cpu(dev->cpu, coupled_mask);
> >>>+	cpumask_shift_right(vpe_mask, coupled_mask,
> >>>+			    cpumask_first(&dev->coupled_cpus));
> >>
> >>What is the purpose of this computation ?
> >
> >If you read through the code generated in cps_gen_entry_code, the
> >vpe_mask is used to indicate the VPEs within the current core which are
> >both online & not the VPE currently running the code.
> 
> Ok, thanks for the clarification.
> 
> Shouldn't 'online = cpumask_weight(coupled_mask)' be after clearing the
> current cpu ? Otherwise, online has also the current cpu which, IIUC,
> shouldn't be included, no ?

No, the generated code actually counts the number of VPEs which have
entered the generated code so that it knows it's safe to proceed to
disable coherence (which would of course be bad if any VPE was still
running C code). That count (nc_ready_count) has to equal online
(including the current CPU/VPE) & the final VPE goes on to disable
coherence whilst the others just halt.

> 
> >>>+#else
> >>>+	cpumask_clear(coupled_mask);
> >>>+	cpumask_clear(vpe_mask);
> >>>+	first_cpu = dev->cpu;
> >>>+	online = 1;
> >>>+#endif
> >>
> >>first_cpu is not used.
> >
> >Right you are. It's used in further work-in-progress patches but I'll
> >remove it from this one.
> >
> >>
> >>>+	/*
> >>>+	 * Run the generated entry code. Note that we assume the number of VPEs
> >>>+	 * within this core does not exceed the width in bits of a long. Since
> >>>+	 * MVPConf0.PVPE is 4 bits wide this seems like a safe assumption.
> >>>+	 */
> >>>+	num_left = per_cpu(ncwait_asm_enter, core)(vpe_mask->bits[0], online);
> >>>+
> >>>+	/*
> >>>+	 * If this VPE is the first to leave the non-coherent wait state then
> >>>+	 * it needs to wake up any coupled VPEs still running their wait
> >>>+	 * instruction so that they return to cpuidle,
> >>
> >>Why is it needed ? Can't the other cpus stay idle ?
> >
> >Not with the current cpuidle code. Please see the end of
> >cpuidle_enter_state_coupled in drivers/cpuidle/coupled.c. The code waits
> >for all coupled CPUs to exit the idle state before any of them proceed.
> >Whilst I suppose it would be possible to modify cpuidle to not require
> >that, it would leave you with inaccurate residence statistics mentioned
> >below.
> >
> >>
> >>>       * which can then complete
> >>>+	 * coordination between the coupled VPEs & provide the governor with
> >>>+	 * a chance to reflect on the length of time the VPEs were in the
> >>>+	 * idle state.
> >>>+	 */
> >>>+	if (coupled_coherence && (num_left == online))
> >>>+		arch_send_call_function_ipi_mask(coupled_mask);
> >>
> >>Except there is no choice due to hardware limitations, I don't think this is
> >>valid.
> >
> >By nature when one CPU (VPE) within a core leaves a non-coherent state
> >the rest do too, because as I mentioned coherence is a property of the
> >core not of individual VPEs. It would be possible to leave the other
> >VPEs idle if we didn't differentiate between the coherent & non-coherent
> >wait states, but again not with cpuidle as it is today (due to the
> >waiting for all CPUs at the end of cpuidle_enter_state_coupled).
> 
> [ ... ]
> 
> >>>+
> >>>+	/*
> >>>+	 * Set the coupled flag on the appropriate states if this system
> >>>+	 * requires it.
> >>>+	 */
> >>>+	if (coupled_coherence)
> >>>+		for (i = 1; i < cps_driver.state_count; i++)
> >>>+			cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED;
> >>
> >>I am not sure CPUIDLE_FLAG_COUPLED is the solution for this driver. IIUC,
> >>with the IPI above and the wakeup sync with the couple states, this driver
> >>is waking up everybody instead of sleeping as much as possible.
> >>
> >>I would recommend to have a look at a different approach, like the one used
> >>for cpuidle-ux500.c.
> >
> >Ok it looks like that just counts & performs work only on the last
> >online CPU to run the code.
> 
> Yes, the last-man-standing approach. Usually, the couple idle states are
> used when the processors need to do some extra work or only the cpu0 can
> invoke PM routine for security reason.
> 
> In your case, IIUC, we don't care who will call the PM routine and if a cpu
> wakes up, the other ones (read the VPEs belonging to the same core), can
> stay idle until an interrupt occurs.

Right, we don't care which CPU/VPE disables coherence or which one
re-enables coherence. However when one of them does so it does affect
the rest. So although they could stay idle it's a question of *which*
idle state they're now in. As soon as one VPE has left the idle state
the rest would now be in a coherent wait state. If they don't wake up
then cpuidle continues to believe they are in non-coherent wait.

> 
> >As before that could work but only by
> >disregarding any differentiation between coherent & non-coherent wait
> >states at levels above the driver.
> 
> May be you can use the arch private flags for the idle states to
> differentiate coherent or non-coherent wait states ?

I'm sure the driver could keep the coherent vs non-coherent
differentiation internally, but my point is that the governor would no
longer have access to it. That may or may not be a big issue, but one
other thing to bear in mind is that as I mentioned in the commit message
non-coherent wait is very much just a stepping stone towards entering
lower power states (clock gating or power gating the cores). Those lower
power states also require treating VPEs within a core as coupled, and
whilst the secondary ones could run a wait instruction on resume that
would be to further detriment of the residency statistics.

> Or alternatively create
> two drivers and register the right one.

I'm unsure how that would help here - could you explain what you mean?

Paul

Re: [PATCH 10/10] cpuidle: cpuidle driver for MIPS CPS

[Paul Burton]

On Wed, Feb 26, 2014 at 03:37:39PM +0100, Daniel Lezcano wrote:
> >>The convention is to not include headers from arch. These headers shouldn't
> >>appear in this driver.
> >>
> >
> >Without accessing those headers I can't really implement anything useful
> >- entering these idle states by their nature involves architecture
> >specifics. Would you rather the bulk of the driver is implemented under
> >arch/mips & the code in drivers/cpuidle simply calls elsewhere to do the
> >real work?
> 
> Usually, this is how it is done. If you can move the 'asm' code inside eg.
> pm.c somewhere in arch/mips and invoke from pm functions from cpuidle
> through a simple header that would be great.

Ok.

> >Right you are, it should either use GFP_ATOMIC or just not handle the
> >off-stack case (which isn't currently used).
> 
> Why don't you just allocate these cpumasks at init time instead of
> allocating / freeing them in the fast path ?

Also a possibility. Right now nothing using this will have off-stack cpu
masks I was just attempting to be cautious by handling the case anyway.
I'll change it one way or another for v2.

> >>>+	/* Calculate which coupled CPUs (VPEs) are online */
> >>>+#ifdef CONFIG_MIPS_MT
> >>>+	cpumask_and(coupled_mask, cpu_online_mask, &dev->coupled_cpus);
> >>>+	first_cpu = cpumask_first(coupled_mask);
> >>>+	online = cpumask_weight(coupled_mask);
> >>>+	cpumask_clear_cpu(dev->cpu, coupled_mask);
> >>>+	cpumask_shift_right(vpe_mask, coupled_mask,
> >>>+			    cpumask_first(&dev->coupled_cpus));
> >>
> >>What is the purpose of this computation ?
> >
> >If you read through the code generated in cps_gen_entry_code, the
> >vpe_mask is used to indicate the VPEs within the current core which are
> >both online & not the VPE currently running the code.
> 
> Ok, thanks for the clarification.
> 
> Shouldn't 'online = cpumask_weight(coupled_mask)' be after clearing the
> current cpu ? Otherwise, online has also the current cpu which, IIUC,
> shouldn't be included, no ?

No, the generated code actually counts the number of VPEs which have
entered the generated code so that it knows it's safe to proceed to
disable coherence (which would of course be bad if any VPE was still
running C code). That count (nc_ready_count) has to equal online
(including the current CPU/VPE) & the final VPE goes on to disable
coherence whilst the others just halt.

> 
> >>>+#else
> >>>+	cpumask_clear(coupled_mask);
> >>>+	cpumask_clear(vpe_mask);
> >>>+	first_cpu = dev->cpu;
> >>>+	online = 1;
> >>>+#endif
> >>
> >>first_cpu is not used.
> >
> >Right you are. It's used in further work-in-progress patches but I'll
> >remove it from this one.
> >
> >>
> >>>+	/*
> >>>+	 * Run the generated entry code. Note that we assume the number of VPEs
> >>>+	 * within this core does not exceed the width in bits of a long. Since
> >>>+	 * MVPConf0.PVPE is 4 bits wide this seems like a safe assumption.
> >>>+	 */
> >>>+	num_left = per_cpu(ncwait_asm_enter, core)(vpe_mask->bits[0], online);
> >>>+
> >>>+	/*
> >>>+	 * If this VPE is the first to leave the non-coherent wait state then
> >>>+	 * it needs to wake up any coupled VPEs still running their wait
> >>>+	 * instruction so that they return to cpuidle,
> >>
> >>Why is it needed ? Can't the other cpus stay idle ?
> >
> >Not with the current cpuidle code. Please see the end of
> >cpuidle_enter_state_coupled in drivers/cpuidle/coupled.c. The code waits
> >for all coupled CPUs to exit the idle state before any of them proceed.
> >Whilst I suppose it would be possible to modify cpuidle to not require
> >that, it would leave you with inaccurate residence statistics mentioned
> >below.
> >
> >>
> >>>       * which can then complete
> >>>+	 * coordination between the coupled VPEs & provide the governor with
> >>>+	 * a chance to reflect on the length of time the VPEs were in the
> >>>+	 * idle state.
> >>>+	 */
> >>>+	if (coupled_coherence && (num_left == online))
> >>>+		arch_send_call_function_ipi_mask(coupled_mask);
> >>
> >>Except there is no choice due to hardware limitations, I don't think this is
> >>valid.
> >
> >By nature when one CPU (VPE) within a core leaves a non-coherent state
> >the rest do too, because as I mentioned coherence is a property of the
> >core not of individual VPEs. It would be possible to leave the other
> >VPEs idle if we didn't differentiate between the coherent & non-coherent
> >wait states, but again not with cpuidle as it is today (due to the
> >waiting for all CPUs at the end of cpuidle_enter_state_coupled).
> 
> [ ... ]
> 
> >>>+
> >>>+	/*
> >>>+	 * Set the coupled flag on the appropriate states if this system
> >>>+	 * requires it.
> >>>+	 */
> >>>+	if (coupled_coherence)
> >>>+		for (i = 1; i < cps_driver.state_count; i++)
> >>>+			cps_driver.states[i].flags |= CPUIDLE_FLAG_COUPLED;
> >>
> >>I am not sure CPUIDLE_FLAG_COUPLED is the solution for this driver. IIUC,
> >>with the IPI above and the wakeup sync with the couple states, this driver
> >>is waking up everybody instead of sleeping as much as possible.
> >>
> >>I would recommend to have a look at a different approach, like the one used
> >>for cpuidle-ux500.c.
> >
> >Ok it looks like that just counts & performs work only on the last
> >online CPU to run the code.
> 
> Yes, the last-man-standing approach. Usually, the couple idle states are
> used when the processors need to do some extra work or only the cpu0 can
> invoke PM routine for security reason.
> 
> In your case, IIUC, we don't care who will call the PM routine and if a cpu
> wakes up, the other ones (read the VPEs belonging to the same core), can
> stay idle until an interrupt occurs.

Right, we don't care which CPU/VPE disables coherence or which one
re-enables coherence. However when one of them does so it does affect
the rest. So although they could stay idle it's a question of *which*
idle state they're now in. As soon as one VPE has left the idle state
the rest would now be in a coherent wait state. If they don't wake up
then cpuidle continues to believe they are in non-coherent wait.

> 
> >As before that could work but only by
> >disregarding any differentiation between coherent & non-coherent wait
> >states at levels above the driver.
> 
> May be you can use the arch private flags for the idle states to
> differentiate coherent or non-coherent wait states ?

I'm sure the driver could keep the coherent vs non-coherent
differentiation internally, but my point is that the governor would no
longer have access to it. That may or may not be a big issue, but one
other thing to bear in mind is that as I mentioned in the commit message
non-coherent wait is very much just a stepping stone towards entering
lower power states (clock gating or power gating the cores). Those lower
power states also require treating VPEs within a core as coupled, and
whilst the secondary ones could run a wait instruction on resume that
would be to further detriment of the residency statistics.

> Or alternatively create
> two drivers and register the right one.

I'm unsure how that would help here - could you explain what you mean?

Paul