[PATCH 1/3] Revert "qemumips: use 34Kf-64tlb CPU emulation"

[PATCH 1/3] Revert "qemumips: use 34Kf-64tlb CPU emulation"

[Victor Kamensky]

This reverts commit 6a9568401cbf8bed89d1f63dae61a07a83a20b50.

Will be replaced with upstreamed solution instead, that just
bumped number of TLBs to 64 in original 34Kf CPU model.
---
 meta/conf/machine/qemumips.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/machine/qemumips.conf b/meta/conf/machine/qemumips.conf
index b8c80f02ef..1373e4cba0 100644
--- a/meta/conf/machine/qemumips.conf
+++ b/meta/conf/machine/qemumips.conf
@@ -15,4 +15,4 @@ SERIAL_CONSOLES ?= "115200;ttyS0 115200;ttyS1"
 
 QB_SYSTEM_NAME = "qemu-system-mips"
 
-QB_CPU = "-cpu 34Kf-64tlb"
+QB_CPU = "-cpu 34Kf"
-- 
2.14.5

[PATCH 2/3] Revert "qemu: add 34Kf-64tlb fictitious cpu type"

[Victor Kamensky]

This reverts commit 4470a04943352224955f17e004962f0f9e1c9b0c.

Will be replaced with upstreamed solution instead, that just
bumped number of TLBs to 64 in original 34Kf CPU model.
---
 meta/recipes-devtools/qemu/qemu.inc                |   1 -
 ...Kf-64tlb-fictitious-cpu-type-like-34Kf-bu.patch | 118 ---------------------
 2 files changed, 119 deletions(-)
 delete mode 100644 meta/recipes-devtools/qemu/qemu/0001-mips-add-34Kf-64tlb-fictitious-cpu-type-like-34Kf-bu.patch

diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc
index 6c0edcb706..bbb9038961 100644
--- a/meta/recipes-devtools/qemu/qemu.inc
+++ b/meta/recipes-devtools/qemu/qemu.inc
@@ -31,7 +31,6 @@ SRC_URI = "https://download.qemu.org/${BPN}-${PV}.tar.xz \
            file://0001-qemu-Do-not-include-file-if-not-exists.patch \
            file://find_datadir.patch \
            file://usb-fix-setup_len-init.patch \
-           file://0001-mips-add-34Kf-64tlb-fictitious-cpu-type-like-34Kf-bu.patch \
            "
 UPSTREAM_CHECK_REGEX = "qemu-(?P<pver>\d+(\.\d+)+)\.tar"
 
diff --git a/meta/recipes-devtools/qemu/qemu/0001-mips-add-34Kf-64tlb-fictitious-cpu-type-like-34Kf-bu.patch b/meta/recipes-devtools/qemu/qemu/0001-mips-add-34Kf-64tlb-fictitious-cpu-type-like-34Kf-bu.patch
deleted file mode 100644
index b6312e1543..0000000000
--- a/meta/recipes-devtools/qemu/qemu/0001-mips-add-34Kf-64tlb-fictitious-cpu-type-like-34Kf-bu.patch
+++ /dev/null
@@ -1,118 +0,0 @@
-From b3fcc7d96523ad8e3ea28c09d495ef08529d01ce Mon Sep 17 00:00:00 2001
-From: Victor Kamensky <kamensky@cisco.com>
-Date: Wed, 7 Oct 2020 10:19:42 -0700
-Subject: [PATCH] mips: add 34Kf-64tlb fictitious cpu type like 34Kf but with
- 64 TLBs
-
-In Yocto Project CI runs it was observed that test run
-of 32 bit mips image takes almost twice longer than 64 bit
-mips image with the same logical load and CI execution
-hits timeout.
-
-See https://bugzilla.yoctoproject.org/show_bug.cgi?id=13992
-
-Yocto project uses 34Kf cpu type to run 32 bit mips image,
-and MIPS64R2-generic cpu type to run 64 bit mips64 image.
-
-Upon qemu behavior differences investigation between mips
-and mips64 two prominent observations came up: under
-logically similar load (same definition and configuration
-of user-land image) in case of mips get_physical_address
-function is called almost twice more often, meaning
-twice more memory accesses involved in this case. Also
-number of tlbwr instruction executed (r4k_helper_tlbwr
-qemu function) almost 16 time bigger in mips case than in
-mips64.
-
-It turns out that 34Kf cpu has 16 TLBs, but in case of
-MIPS64R2-generic it is 64 TLBs. So that explains why
-some many more tlbwr had to be execute by kernel TLB refill
-handler in case of 32 bit misp.
-
-The idea of the fix is to come up with new 34Kf-64tlb fictitious
-cpu type, that would behave exactly as 34Kf but it would
-contain 64 TLBs to reduce TLB trashing. After all, adding
-more TLBs to soft mmu is easy.
-
-Experiment with some significant non-trvial load in Yocto
-environment by running do_testimage load shows that 34Kf-64tlb
-cpu performs 40% or so better than original 34Kf cpu wrt test
-execution real time.
-
-It is not ideal to have cpu type that does not exist in the
-wild but given performance gains it seems to be justified.
-
-Signed-off-by: Victor Kamensky <kamensky@cisco.com>
----
- target/mips/translate_init.inc.c | 55 ++++++++++++++++++++++++++++++++++++++++
- 1 file changed, 55 insertions(+)
-
-diff --git a/target/mips/translate_init.inc.c b/target/mips/translate_init.inc.c
-index 637caccd89..b73ab48231 100644
---- a/target/mips/translate_init.inc.c
-+++ b/target/mips/translate_init.inc.c
-@@ -297,6 +297,61 @@ const mips_def_t mips_defs[] =
-         .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_MT,
-         .mmu_type = MMU_TYPE_R4000,
-     },
-+    /*
-+     * Verbatim copy of "34Kf" cpu, only bumped up number of TLB entries
-+     * from 16 to 64 (see CP0_Config0 value at CP0C1_MMU bits) to improve
-+     * performance by reducing number of TLB refill exceptions and
-+     * eliminating need to run all corresponding TLB refill handling
-+     * instructions.
-+     */
-+    {
-+        .name = "34Kf-64tlb",
-+        .CP0_PRid = 0x00019500,
-+        .CP0_Config0 = MIPS_CONFIG0 | (0x1 << CP0C0_AR) |
-+                       (MMU_TYPE_R4000 << CP0C0_MT),
-+        .CP0_Config1 = MIPS_CONFIG1 | (1 << CP0C1_FP) | (63 << CP0C1_MMU) |
-+                       (0 << CP0C1_IS) | (3 << CP0C1_IL) | (1 << CP0C1_IA) |
-+                       (0 << CP0C1_DS) | (3 << CP0C1_DL) | (1 << CP0C1_DA) |
-+                       (1 << CP0C1_CA),
-+        .CP0_Config2 = MIPS_CONFIG2,
-+        .CP0_Config3 = MIPS_CONFIG3 | (1 << CP0C3_VInt) | (1 << CP0C3_MT) |
-+                       (1 << CP0C3_DSPP),
-+        .CP0_LLAddr_rw_bitmask = 0,
-+        .CP0_LLAddr_shift = 0,
-+        .SYNCI_Step = 32,
-+        .CCRes = 2,
-+        .CP0_Status_rw_bitmask = 0x3778FF1F,
-+        .CP0_TCStatus_rw_bitmask = (0 << CP0TCSt_TCU3) | (0 << CP0TCSt_TCU2) |
-+                    (1 << CP0TCSt_TCU1) | (1 << CP0TCSt_TCU0) |
-+                    (0 << CP0TCSt_TMX) | (1 << CP0TCSt_DT) |
-+                    (1 << CP0TCSt_DA) | (1 << CP0TCSt_A) |
-+                    (0x3 << CP0TCSt_TKSU) | (1 << CP0TCSt_IXMT) |
-+                    (0xff << CP0TCSt_TASID),
-+        .CP1_fcr0 = (1 << FCR0_F64) | (1 << FCR0_L) | (1 << FCR0_W) |
-+                    (1 << FCR0_D) | (1 << FCR0_S) | (0x95 << FCR0_PRID),
-+        .CP1_fcr31 = 0,
-+        .CP1_fcr31_rw_bitmask = 0xFF83FFFF,
-+        .CP0_SRSCtl = (0xf << CP0SRSCtl_HSS),
-+        .CP0_SRSConf0_rw_bitmask = 0x3fffffff,
-+        .CP0_SRSConf0 = (1U << CP0SRSC0_M) | (0x3fe << CP0SRSC0_SRS3) |
-+                    (0x3fe << CP0SRSC0_SRS2) | (0x3fe << CP0SRSC0_SRS1),
-+        .CP0_SRSConf1_rw_bitmask = 0x3fffffff,
-+        .CP0_SRSConf1 = (1U << CP0SRSC1_M) | (0x3fe << CP0SRSC1_SRS6) |
-+                    (0x3fe << CP0SRSC1_SRS5) | (0x3fe << CP0SRSC1_SRS4),
-+        .CP0_SRSConf2_rw_bitmask = 0x3fffffff,
-+        .CP0_SRSConf2 = (1U << CP0SRSC2_M) | (0x3fe << CP0SRSC2_SRS9) |
-+                    (0x3fe << CP0SRSC2_SRS8) | (0x3fe << CP0SRSC2_SRS7),
-+        .CP0_SRSConf3_rw_bitmask = 0x3fffffff,
-+        .CP0_SRSConf3 = (1U << CP0SRSC3_M) | (0x3fe << CP0SRSC3_SRS12) |
-+                    (0x3fe << CP0SRSC3_SRS11) | (0x3fe << CP0SRSC3_SRS10),
-+        .CP0_SRSConf4_rw_bitmask = 0x3fffffff,
-+        .CP0_SRSConf4 = (0x3fe << CP0SRSC4_SRS15) |
-+                    (0x3fe << CP0SRSC4_SRS14) | (0x3fe << CP0SRSC4_SRS13),
-+        .SEGBITS = 32,
-+        .PABITS = 32,
-+        .insn_flags = CPU_MIPS32R2 | ASE_MIPS16 | ASE_DSP | ASE_MT,
-+        .mmu_type = MMU_TYPE_R4000,
-+    },
-     {
-         .name = "74Kf",
-         .CP0_PRid = 0x00019700,
--- 
-2.14.5
-
-- 
2.14.5

[PATCH 3/3] qemu: change TLBs number to 64 in 34Kf mips cpu model

[Victor Kamensky]

Replace OE private qemu patch with one that got upstreamed
and solves the same problem: increase qemumips CI performance
by increasing number of TLBs in CPU model and reduce need to
run software TLB refill code.

Signed-off-by: Victor Kamensky <kamensky@cisco.com>
---
 meta/recipes-devtools/qemu/qemu.inc                |  1 +
 ...-Increase-number-of-TLB-entries-on-the-34.patch | 59 ++++++++++++++++++++++
 2 files changed, 60 insertions(+)
 create mode 100644 meta/recipes-devtools/qemu/qemu/0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch

diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc
index bbb9038961..84f600cec0 100644
--- a/meta/recipes-devtools/qemu/qemu.inc
+++ b/meta/recipes-devtools/qemu/qemu.inc
@@ -31,6 +31,7 @@ SRC_URI = "https://download.qemu.org/${BPN}-${PV}.tar.xz \
            file://0001-qemu-Do-not-include-file-if-not-exists.patch \
            file://find_datadir.patch \
            file://usb-fix-setup_len-init.patch \
+           file://0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch \
            "
 UPSTREAM_CHECK_REGEX = "qemu-(?P<pver>\d+(\.\d+)+)\.tar"
 
diff --git a/meta/recipes-devtools/qemu/qemu/0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch b/meta/recipes-devtools/qemu/qemu/0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch
new file mode 100644
index 0000000000..5227b7cbd2
--- /dev/null
+++ b/meta/recipes-devtools/qemu/qemu/0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch
@@ -0,0 +1,59 @@
+From 68fa519a6cb455005317bd61f95214b58b2f1e69 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= <f4bug@amsat.org>
+Date: Fri, 16 Oct 2020 15:20:37 +0200
+Subject: [PATCH] target/mips: Increase number of TLB entries on the 34Kf core
+ (16 -> 64)
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Per "MIPS32 34K Processor Core Family Software User's Manual,
+Revision 01.13" page 8 in "Joint TLB (JTLB)" section:
+
+  "The JTLB is a fully associative TLB cache containing 16, 32,
+   or 64-dual-entries mapping up to 128 virtual pages to their
+   corresponding physical addresses."
+
+There is no particular reason to restrict the 34Kf core model to
+16 TLB entries, so raise its config to 64.
+
+This is helpful for other projects, in particular the Yocto Project:
+
+  Yocto Project uses qemu-system-mips 34Kf cpu model, to run 32bit
+  MIPS CI loop. It was observed that in this case CI test execution
+  time was almost twice longer than 64bit MIPS variant that runs
+  under MIPS64R2-generic model. It was investigated and concluded
+  that the difference in number of TLBs 16 in 34Kf case vs 64 in
+  MIPS64R2-generic is responsible for most of CI real time execution
+  difference. Because with 16 TLBs linux user-land trashes TLB more
+  and it needs to execute more instructions in TLB refill handler
+  calls, as result it runs much longer.
+
+(https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg03428.html)
+
+Buglink: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13992
+Reported-by: Victor Kamensky <kamensky@cisco.com>
+Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
+Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
+Message-Id: <20201016133317.553068-1-f4bug@amsat.org>
+
+Upstream-Status: Backport [https://github.com/qemu/qemu/commit/68fa519a6cb455005317bd61f95214b58b2f1e69]
+Signed-off-by: Victor Kamensky <kamensky@cisco.com>
+
+---
+ target/mips/translate_init.c.inc | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+Index: qemu-5.1.0/target/mips/translate_init.inc.c
+===================================================================
+--- qemu-5.1.0.orig/target/mips/translate_init.inc.c
++++ qemu-5.1.0/target/mips/translate_init.inc.c
+@@ -254,7 +254,7 @@ const mips_def_t mips_defs[] =
+         .CP0_PRid = 0x00019500,
+         .CP0_Config0 = MIPS_CONFIG0 | (0x1 << CP0C0_AR) |
+                        (MMU_TYPE_R4000 << CP0C0_MT),
+-        .CP0_Config1 = MIPS_CONFIG1 | (1 << CP0C1_FP) | (15 << CP0C1_MMU) |
++        .CP0_Config1 = MIPS_CONFIG1 | (1 << CP0C1_FP) | (63 << CP0C1_MMU) |
+                        (0 << CP0C1_IS) | (3 << CP0C1_IL) | (1 << CP0C1_IA) |
+                        (0 << CP0C1_DS) | (3 << CP0C1_DL) | (1 << CP0C1_DA) |
+                        (1 << CP0C1_CA),
-- 
2.14.5

Re: [PATCH 3/3] qemu: change TLBs number to 64 in 34Kf mips cpu model

[Khem Raj]

this looks good to me.

On Mon, Oct 19, 2020 at 3:22 PM Victor Kamensky <kamensky@cisco.com> wrote:
>
> Replace OE private qemu patch with one that got upstreamed
> and solves the same problem: increase qemumips CI performance
> by increasing number of TLBs in CPU model and reduce need to
> run software TLB refill code.
>
> Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> ---
>  meta/recipes-devtools/qemu/qemu.inc                |  1 +
>  ...-Increase-number-of-TLB-entries-on-the-34.patch | 59 ++++++++++++++++++++++
>  2 files changed, 60 insertions(+)
>  create mode 100644 meta/recipes-devtools/qemu/qemu/0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch
>
> diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc
> index bbb9038961..84f600cec0 100644
> --- a/meta/recipes-devtools/qemu/qemu.inc
> +++ b/meta/recipes-devtools/qemu/qemu.inc
> @@ -31,6 +31,7 @@ SRC_URI = "https://download.qemu.org/${BPN}-${PV}.tar.xz \
>             file://0001-qemu-Do-not-include-file-if-not-exists.patch \
>             file://find_datadir.patch \
>             file://usb-fix-setup_len-init.patch \
> +           file://0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch \
>             "
>  UPSTREAM_CHECK_REGEX = "qemu-(?P<pver>\d+(\.\d+)+)\.tar"
>
> diff --git a/meta/recipes-devtools/qemu/qemu/0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch b/meta/recipes-devtools/qemu/qemu/0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch
> new file mode 100644
> index 0000000000..5227b7cbd2
> --- /dev/null
> +++ b/meta/recipes-devtools/qemu/qemu/0001-target-mips-Increase-number-of-TLB-entries-on-the-34.patch
> @@ -0,0 +1,59 @@
> +From 68fa519a6cb455005317bd61f95214b58b2f1e69 Mon Sep 17 00:00:00 2001
> +From: =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= <f4bug@amsat.org>
> +Date: Fri, 16 Oct 2020 15:20:37 +0200
> +Subject: [PATCH] target/mips: Increase number of TLB entries on the 34Kf core
> + (16 -> 64)
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +Per "MIPS32 34K Processor Core Family Software User's Manual,
> +Revision 01.13" page 8 in "Joint TLB (JTLB)" section:
> +
> +  "The JTLB is a fully associative TLB cache containing 16, 32,
> +   or 64-dual-entries mapping up to 128 virtual pages to their
> +   corresponding physical addresses."
> +
> +There is no particular reason to restrict the 34Kf core model to
> +16 TLB entries, so raise its config to 64.
> +
> +This is helpful for other projects, in particular the Yocto Project:
> +
> +  Yocto Project uses qemu-system-mips 34Kf cpu model, to run 32bit
> +  MIPS CI loop. It was observed that in this case CI test execution
> +  time was almost twice longer than 64bit MIPS variant that runs
> +  under MIPS64R2-generic model. It was investigated and concluded
> +  that the difference in number of TLBs 16 in 34Kf case vs 64 in
> +  MIPS64R2-generic is responsible for most of CI real time execution
> +  difference. Because with 16 TLBs linux user-land trashes TLB more
> +  and it needs to execute more instructions in TLB refill handler
> +  calls, as result it runs much longer.
> +
> +(https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg03428.html)
> +
> +Buglink: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13992
> +Reported-by: Victor Kamensky <kamensky@cisco.com>
> +Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> +Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> +Message-Id: <20201016133317.553068-1-f4bug@amsat.org>
> +
> +Upstream-Status: Backport [https://github.com/qemu/qemu/commit/68fa519a6cb455005317bd61f95214b58b2f1e69]
> +Signed-off-by: Victor Kamensky <kamensky@cisco.com>
> +
> +---
> + target/mips/translate_init.c.inc | 2 +-
> + 1 file changed, 1 insertion(+), 1 deletion(-)
> +
> +Index: qemu-5.1.0/target/mips/translate_init.inc.c
> +===================================================================
> +--- qemu-5.1.0.orig/target/mips/translate_init.inc.c
> ++++ qemu-5.1.0/target/mips/translate_init.inc.c
> +@@ -254,7 +254,7 @@ const mips_def_t mips_defs[] =
> +         .CP0_PRid = 0x00019500,
> +         .CP0_Config0 = MIPS_CONFIG0 | (0x1 << CP0C0_AR) |
> +                        (MMU_TYPE_R4000 << CP0C0_MT),
> +-        .CP0_Config1 = MIPS_CONFIG1 | (1 << CP0C1_FP) | (15 << CP0C1_MMU) |
> ++        .CP0_Config1 = MIPS_CONFIG1 | (1 << CP0C1_FP) | (63 << CP0C1_MMU) |
> +                        (0 << CP0C1_IS) | (3 << CP0C1_IL) | (1 << CP0C1_IA) |
> +                        (0 << CP0C1_DS) | (3 << CP0C1_DL) | (1 << CP0C1_DA) |
> +                        (1 << CP0C1_CA),
> --
> 2.14.5
>

Re: [PATCH 3/3] qemu: change TLBs number to 64 in 34Kf mips cpu model

[Richard Purdie]

On Mon, 2020-10-19 at 15:21 -0700, Victor Kamensky wrote:
> Replace OE private qemu patch with one that got upstreamed
> and solves the same problem: increase qemumips CI performance
> by increasing number of TLBs in CPU model and reduce need to
> run software TLB refill code.
> 
> Signed-off-by: Victor Kamensky <kamensky@cisco.com>

Thanks, its amazing how simple this is in the end but very worthwhile!

Steve: This one should backport to dunfell without the rest of the
history once it makes master.

Cheers,

Richard