[PATCH v2] riscv: enable THP_SWAP for RV64

[PATCH v2] riscv: enable THP_SWAP for RV64

[Jisheng Zhang]

I have a Sipeed Lichee RV dock board which only has 512MB DDR, so
memory optimizations such as swap on zram are helpful. As is seen
in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and
commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after
swapped out"), THP_SWAP can improve the swap throughput significantly.

Enable THP_SWAP for RV64, testing the micro-benchmark which is
introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64")
shows below numbers on the Lichee RV dock board:

thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests)
thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)

Improved by 382%!

Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
---
Since v1:
 - collect reviewed-by tag
 - make ARCH_WANTS_THP_SWAP rely on HAVE_ARCH_TRANSPARENT_HUGEPAGE
   instead

 arch/riscv/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index ed66c31e4655..79e52441e18b 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -45,6 +45,7 @@ config RISCV
 	select ARCH_WANT_FRAME_POINTERS
 	select ARCH_WANT_GENERAL_HUGETLB
 	select ARCH_WANT_HUGE_PMD_SHARE if 64BIT
+	select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
 	select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU
 	select BUILDTIME_TABLE_SORT if MMU
 	select CLONE_BACKWARDS
-- 
2.34.1

Re: [PATCH v2] riscv: enable THP_SWAP for RV64

[Conor.Dooley]

Hey Jisheng,
On 27/08/2022 10:58, Jisheng Zhang wrote:
> I have a Sipeed Lichee RV dock board which only has 512MB DDR, so
> memory optimizations such as swap on zram are helpful. As is seen
> in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and
> commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after
> swapped out"), THP_SWAP can improve the swap throughput significantly.
> 
> Enable THP_SWAP for RV64, testing the micro-benchmark which is
> introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64")
> shows below numbers on the Lichee RV dock board:
> 
> thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests)
> thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)

I know the original commit message contains this, but it's a little
odd. If the patch /enables/ THP then how would there be THP swap
prior to the patch?

> 
> Improved by 382%!

I could not replicate the after numbers on my nezha, so I suspect
I am missing something in my config/setup. zswap is enabled and is
working, TRANSPARENT_HUGEPAGE is enabled etc. Not that it matters
for acceptance of the patch though.

I gave it a try and nothing went up in flames while using zswap so:
Reviewed-by: Conor Dooley <conor.dooley@microchip.com>

> 
> Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
> ---
> Since v1:
>  - collect reviewed-by tag
>  - make ARCH_WANTS_THP_SWAP rely on HAVE_ARCH_TRANSPARENT_HUGEPAGE
>    instead
> 
>  arch/riscv/Kconfig | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index ed66c31e4655..79e52441e18b 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -45,6 +45,7 @@ config RISCV
>  	select ARCH_WANT_FRAME_POINTERS
>  	select ARCH_WANT_GENERAL_HUGETLB
>  	select ARCH_WANT_HUGE_PMD_SHARE if 64BIT
> +	select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
>  	select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU
>  	select BUILDTIME_TABLE_SORT if MMU
>  	select CLONE_BACKWARDS

Re: [PATCH v2] riscv: enable THP_SWAP for RV64

[Jisheng Zhang]

On Sat, Aug 27, 2022 at 09:13:03PM +0000, Conor.Dooley@microchip.com wrote:
> Hey Jisheng,

Hi Conor,

> On 27/08/2022 10:58, Jisheng Zhang wrote:
> > I have a Sipeed Lichee RV dock board which only has 512MB DDR, so
> > memory optimizations such as swap on zram are helpful. As is seen
> > in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and
> > commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after
> > swapped out"), THP_SWAP can improve the swap throughput significantly.
> > 
> > Enable THP_SWAP for RV64, testing the micro-benchmark which is
> > introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64")
> > shows below numbers on the Lichee RV dock board:
> > 
> > thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests)
> > thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
> 
> I know the original commit message contains this, but it's a little
> odd. If the patch /enables/ THP then how would there be THP swap
> prior to the patch?

hmm, it's swap I'll send a v3 to correct the description.

> 
> > 
> > Improved by 382%!
> 
> I could not replicate the after numbers on my nezha, so I suspect
> I am missing something in my config/setup. zswap is enabled and is

swap on zram rather than zswap ;)

> working, TRANSPARENT_HUGEPAGE is enabled etc. Not that it matters
> for acceptance of the patch though.
> 
> I gave it a try and nothing went up in flames while using zswap so:
> Reviewed-by: Conor Dooley <conor.dooley@microchip.com>
> 
> > 
> > Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
> > Reviewed-by: Andrew Jones <ajones@ventanamicro.com>
> > ---
> > Since v1:
> >  - collect reviewed-by tag
> >  - make ARCH_WANTS_THP_SWAP rely on HAVE_ARCH_TRANSPARENT_HUGEPAGE
> >    instead
> > 
> >  arch/riscv/Kconfig | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> > index ed66c31e4655..79e52441e18b 100644
> > --- a/arch/riscv/Kconfig
> > +++ b/arch/riscv/Kconfig
> > @@ -45,6 +45,7 @@ config RISCV
> >  	select ARCH_WANT_FRAME_POINTERS
> >  	select ARCH_WANT_GENERAL_HUGETLB
> >  	select ARCH_WANT_HUGE_PMD_SHARE if 64BIT
> > +	select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
> >  	select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU
> >  	select BUILDTIME_TABLE_SORT if MMU
> >  	select CLONE_BACKWARDS

Re: [PATCH v2] riscv: enable THP_SWAP for RV64

[Conor.Dooley]

On 29/08/2022 15:10, Jisheng Zhang wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> 
> On Sat, Aug 27, 2022 at 09:13:03PM +0000, Conor.Dooley@microchip.com wrote:
>> Hey Jisheng,
> 
> Hi Conor,
> 
>> On 27/08/2022 10:58, Jisheng Zhang wrote:
>>> I have a Sipeed Lichee RV dock board which only has 512MB DDR, so
>>> memory optimizations such as swap on zram are helpful. As is seen
>>> in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and
>>> commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after
>>> swapped out"), THP_SWAP can improve the swap throughput significantly.
>>>
>>> Enable THP_SWAP for RV64, testing the micro-benchmark which is
>>> introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64")
>>> shows below numbers on the Lichee RV dock board:
>>>
>>> thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests)
>>> thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
>>
>> I know the original commit message contains this, but it's a little
>> odd. If the patch /enables/ THP then how would there be THP swap
>> prior to the patch?
> 
> hmm, it's swap I'll send a v3 to correct the description.
> 
>>
>>>
>>> Improved by 382%!
>>
>> I could not replicate the after numbers on my nezha, so I suspect
>> I am missing something in my config/setup. zswap is enabled and is
> 
> swap on zram rather than zswap ;)

I think I tried about 30 different config variations, initially not
using zswap and later using it.
My zramctl looks like so (although I did try zstd too) after running
the demo application from that commit:

NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle       241M  22M  8.4M  9.1M       1 [SWAP]

I am using the default riscv defconfig + the following:
CONFIG_ZRAM=y
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_ZSTD=y
CONFIG_ZRAM_MEMORY_TRACKING=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
CONFIG_THP_SWAP=y

Am I just missing something obvious here?
Sorry,
Conor.

Re: [PATCH v2] riscv: enable THP_SWAP for RV64

[Jisheng Zhang]

On Mon, Aug 29, 2022 at 05:27:48PM +0000, Conor.Dooley@microchip.com wrote:
> On 29/08/2022 15:10, Jisheng Zhang wrote:
> > EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> > 
> > On Sat, Aug 27, 2022 at 09:13:03PM +0000, Conor.Dooley@microchip.com wrote:
> >> Hey Jisheng,
> > 
> > Hi Conor,
> > 
> >> On 27/08/2022 10:58, Jisheng Zhang wrote:
> >>> I have a Sipeed Lichee RV dock board which only has 512MB DDR, so
> >>> memory optimizations such as swap on zram are helpful. As is seen
> >>> in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and
> >>> commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after
> >>> swapped out"), THP_SWAP can improve the swap throughput significantly.
> >>>
> >>> Enable THP_SWAP for RV64, testing the micro-benchmark which is
> >>> introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64")
> >>> shows below numbers on the Lichee RV dock board:
> >>>
> >>> thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests)
> >>> thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
> >>
> >> I know the original commit message contains this, but it's a little
> >> odd. If the patch /enables/ THP then how would there be THP swap
> >> prior to the patch?
> > 
> > hmm, it's swap I'll send a v3 to correct the description.
> > 
> >>
> >>>
> >>> Improved by 382%!
> >>
> >> I could not replicate the after numbers on my nezha, so I suspect
> >> I am missing something in my config/setup. zswap is enabled and is
> > 
> > swap on zram rather than zswap ;)
> 
> I think I tried about 30 different config variations, initially not
> using zswap and later using it.
> My zramctl looks like so (although I did try zstd too) after running
> the demo application from that commit:
> 
> NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
> /dev/zram0 lzo-rle       241M  22M  8.4M  9.1M       1 [SWAP]
> 
> I am using the default riscv defconfig + the following:
> CONFIG_ZRAM=y
> CONFIG_CRYPTO_DEFLATE=y
> CONFIG_CRYPTO_LZO=y
> CONFIG_CRYPTO_ZSTD=y
> CONFIG_ZRAM_MEMORY_TRACKING=y
> CONFIG_TRANSPARENT_HUGEPAGE=y
> CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
> CONFIG_THP_SWAP=y
> 
> Am I just missing something obvious here?

similar config options here. what's your rootfs? Is your board busy
with something? I used a minimal rootfs built from buildroot.
can you plz show your numbers w/ and w/o the patch?

I also tried the simple benchmark on qemu(just for reference, since
I have no other riscv boards except the lichee RV dock board):
swp out w/o patch: 30066 bytes/ms (mean of 10 tests)
swp out w/ patch: 130055 bytes/ms (mean of 10 tests)
so improved by 332.7% 

Re: [PATCH v2] riscv: enable THP_SWAP for RV64

[Conor.Dooley]

On 30/08/2022 14:59, Jisheng Zhang wrote:
> On Mon, Aug 29, 2022 at 05:27:48PM +0000, Conor.Dooley@microchip.com wrote:
>> On 29/08/2022 15:10, Jisheng Zhang wrote:
>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>
>>> On Sat, Aug 27, 2022 at 09:13:03PM +0000, Conor.Dooley@microchip.com wrote:
>>>> Hey Jisheng,
>>>
>>> Hi Conor,
>>>
>>>> On 27/08/2022 10:58, Jisheng Zhang wrote:
>>>>> I have a Sipeed Lichee RV dock board which only has 512MB DDR, so
>>>>> memory optimizations such as swap on zram are helpful. As is seen
>>>>> in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and
>>>>> commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after
>>>>> swapped out"), THP_SWAP can improve the swap throughput significantly.
>>>>>
>>>>> Enable THP_SWAP for RV64, testing the micro-benchmark which is
>>>>> introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64")
>>>>> shows below numbers on the Lichee RV dock board:
>>>>>
>>>>> thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests)
>>>>> thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
>>>>
>>>> I know the original commit message contains this, but it's a little
>>>> odd. If the patch /enables/ THP then how would there be THP swap
>>>> prior to the patch?
>>>
>>> hmm, it's swap I'll send a v3 to correct the description.
>>>
>>>>
>>>>>
>>>>> Improved by 382%!
>>>>
>>>> I could not replicate the after numbers on my nezha, so I suspect
>>>> I am missing something in my config/setup. zswap is enabled and is
>>>
>>> swap on zram rather than zswap ;)
>>
>> I think I tried about 30 different config variations, initially not
>> using zswap and later using it.
>> My zramctl looks like so (although I did try zstd too) after running
>> the demo application from that commit:
>>
>> NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
>> /dev/zram0 lzo-rle       241M  22M  8.4M  9.1M       1 [SWAP]
>>
>> I am using the default riscv defconfig + the following:
>> CONFIG_ZRAM=y
>> CONFIG_CRYPTO_DEFLATE=y
>> CONFIG_CRYPTO_LZO=y
>> CONFIG_CRYPTO_ZSTD=y
>> CONFIG_ZRAM_MEMORY_TRACKING=y
>> CONFIG_TRANSPARENT_HUGEPAGE=y
>> CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
>> CONFIG_THP_SWAP=y
>>
>> Am I just missing something obvious here?
> 
> similar config options here. what's your rootfs? Is your board busy
> with something? I used a minimal rootfs built from buildroot.
> can you plz show your numbers w/ and w/o the patch?

I was using fedora for the testing, downloaded directly from
koji. My before/after numbers varied, but were around 80,000
bytes/ms most of the time.

If I increased the size to 500 * 1024 * 1024 I got around 130k.

Before/after the patch, the numbers did not really change, but
things did fluctuate quite wildly - from about 50k to 90k using
the 400 size.

> 
> I also tried the simple benchmark on qemu(just for reference, since
> I have no other riscv boards except the lichee RV dock board):
> swp out w/o patch: 30066 bytes/ms (mean of 10 tests)
> swp out w/ patch: 130055 bytes/ms (mean of 10 tests)
> so improved by 332.7% 

I'll give QEMU a go so :)

Re: [PATCH v2] riscv: enable THP_SWAP for RV64

[Conor.Dooley]

On 30/08/2022 15:15, Conor Dooley - M52691 wrote:
> On 30/08/2022 14:59, Jisheng Zhang wrote:
>> On Mon, Aug 29, 2022 at 05:27:48PM +0000, Conor.Dooley@microchip.com wrote:
>>> On 29/08/2022 15:10, Jisheng Zhang wrote:
>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>>>
>>>> On Sat, Aug 27, 2022 at 09:13:03PM +0000, Conor.Dooley@microchip.com wrote:
>>>>> Hey Jisheng,
>>>>
>>>> Hi Conor,
>>>>
>>>>> On 27/08/2022 10:58, Jisheng Zhang wrote:
>>>>>> I have a Sipeed Lichee RV dock board which only has 512MB DDR, so
>>>>>> memory optimizations such as swap on zram are helpful. As is seen
>>>>>> in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and
>>>>>> commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after
>>>>>> swapped out"), THP_SWAP can improve the swap throughput significantly.
>>>>>>
>>>>>> Enable THP_SWAP for RV64, testing the micro-benchmark which is
>>>>>> introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64")
>>>>>> shows below numbers on the Lichee RV dock board:
>>>>>>
>>>>>> thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests)
>>>>>> thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
>>>>>
>>>>> I know the original commit message contains this, but it's a little
>>>>> odd. If the patch /enables/ THP then how would there be THP swap
>>>>> prior to the patch?
>>>>
>>>> hmm, it's swap I'll send a v3 to correct the description.
>>>>
>>>>>
>>>>>>
>>>>>> Improved by 382%!
>>>>>
>>>>> I could not replicate the after numbers on my nezha, so I suspect
>>>>> I am missing something in my config/setup. zswap is enabled and is
>>>>
>>>> swap on zram rather than zswap ;)
>>>
>>> I think I tried about 30 different config variations, initially not
>>> using zswap and later using it.
>>> My zramctl looks like so (although I did try zstd too) after running
>>> the demo application from that commit:
>>>
>>> NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
>>> /dev/zram0 lzo-rle       241M  22M  8.4M  9.1M       1 [SWAP]
>>>
>>> I am using the default riscv defconfig + the following:
>>> CONFIG_ZRAM=y
>>> CONFIG_CRYPTO_DEFLATE=y
>>> CONFIG_CRYPTO_LZO=y
>>> CONFIG_CRYPTO_ZSTD=y
>>> CONFIG_ZRAM_MEMORY_TRACKING=y
>>> CONFIG_TRANSPARENT_HUGEPAGE=y
>>> CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
>>> CONFIG_THP_SWAP=y
>>>
>>> Am I just missing something obvious here?
>>
>> similar config options here. what's your rootfs? Is your board busy
>> with something? I used a minimal rootfs built from buildroot.
>> can you plz show your numbers w/ and w/o the patch?
> 
> I was using fedora for the testing, downloaded directly from
> koji. My before/after numbers varied, but were around 80,000
> bytes/ms most of the time.
> 
> If I increased the size to 500 * 1024 * 1024 I got around 130k.

130k before & after.**

> 
> Before/after the patch, the numbers did not really change, but
> things did fluctuate quite wildly - from about 50k to 90k using
> the 400 size.

What I mean is: before/after the patch had visible performance
difference because it was always flucuating in the same range.

> 
>>
>> I also tried the simple benchmark on qemu(just for reference, since
>> I have no other riscv boards except the lichee RV dock board):
>> swp out w/o patch: 30066 bytes/ms (mean of 10 tests)
>> swp out w/ patch: 130055 bytes/ms (mean of 10 tests)
>> so improved by 332.7% 
> 
> I'll give QEMU a go so :)

Re: [PATCH v2] riscv: enable THP_SWAP for RV64

[Jisheng Zhang]

On Tue, Aug 30, 2022 at 02:26:38PM +0000, Conor.Dooley@microchip.com wrote:
> On 30/08/2022 15:15, Conor Dooley - M52691 wrote:
> > On 30/08/2022 14:59, Jisheng Zhang wrote:
> >> On Mon, Aug 29, 2022 at 05:27:48PM +0000, Conor.Dooley@microchip.com wrote:
> >>> On 29/08/2022 15:10, Jisheng Zhang wrote:
> >>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> >>>>
> >>>> On Sat, Aug 27, 2022 at 09:13:03PM +0000, Conor.Dooley@microchip.com wrote:
> >>>>> Hey Jisheng,
> >>>>
> >>>> Hi Conor,
> >>>>
> >>>>> On 27/08/2022 10:58, Jisheng Zhang wrote:
> >>>>>> I have a Sipeed Lichee RV dock board which only has 512MB DDR, so
> >>>>>> memory optimizations such as swap on zram are helpful. As is seen
> >>>>>> in commit d0637c505f8a ("arm64: enable THP_SWAP for arm64") and
> >>>>>> commit bd4c82c22c367e ("mm, THP, swap: delay splitting THP after
> >>>>>> swapped out"), THP_SWAP can improve the swap throughput significantly.
> >>>>>>
> >>>>>> Enable THP_SWAP for RV64, testing the micro-benchmark which is
> >>>>>> introduced by commit d0637c505f8a ("arm64: enable THP_SWAP for arm64")
> >>>>>> shows below numbers on the Lichee RV dock board:
> >>>>>>
> >>>>>> thp swp throughput w/o patch: 66908 bytes/ms (mean of 10 tests)
> >>>>>> thp swp throughput w/ patch: 322638 bytes/ms (mean of 10 tests)
> >>>>>
> >>>>> I know the original commit message contains this, but it's a little
> >>>>> odd. If the patch /enables/ THP then how would there be THP swap
> >>>>> prior to the patch?
> >>>>
> >>>> hmm, it's swap I'll send a v3 to correct the description.
> >>>>
> >>>>>
> >>>>>>
> >>>>>> Improved by 382%!
> >>>>>
> >>>>> I could not replicate the after numbers on my nezha, so I suspect
> >>>>> I am missing something in my config/setup. zswap is enabled and is
> >>>>
> >>>> swap on zram rather than zswap ;)
> >>>
> >>> I think I tried about 30 different config variations, initially not
> >>> using zswap and later using it.
> >>> My zramctl looks like so (although I did try zstd too) after running
> >>> the demo application from that commit:
> >>>
> >>> NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
> >>> /dev/zram0 lzo-rle       241M  22M  8.4M  9.1M       1 [SWAP]
> >>>
> >>> I am using the default riscv defconfig + the following:
> >>> CONFIG_ZRAM=y
> >>> CONFIG_CRYPTO_DEFLATE=y
> >>> CONFIG_CRYPTO_LZO=y
> >>> CONFIG_CRYPTO_ZSTD=y
> >>> CONFIG_ZRAM_MEMORY_TRACKING=y
> >>> CONFIG_TRANSPARENT_HUGEPAGE=y
> >>> CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
> >>> CONFIG_THP_SWAP=y
> >>>
> >>> Am I just missing something obvious here?
> >>
> >> similar config options here. what's your rootfs? Is your board busy
> >> with something? I used a minimal rootfs built from buildroot.
> >> can you plz show your numbers w/ and w/o the patch?
> > 
> > I was using fedora for the testing, downloaded directly from
> > koji. My before/after numbers varied, but were around 80,000
> > bytes/ms most of the time.
> > 
> > If I increased the size to 500 * 1024 * 1024 I got around 130k.
> 
> 130k before & after.**
> 
> > 
> > Before/after the patch, the numbers did not really change, but
> > things did fluctuate quite wildly - from about 50k to 90k using
> > the 400 size.
> 
> What I mean is: before/after the patch had visible performance
> difference because it was always flucuating in the same range.

I see the difference -- w/ minial buildroot rootfs, the numbers isn't
kept the same but the difference is trivial, I even got two or three
the same numbers during 10 times of testing. But your numbers were
always flucuating, so I guess your system maybe busy with with
something in a shore period, I.E the os env is full of noise. 

I guess you may get similar improvement percentage when trying buildroot
> 
> > 
> >>
> >> I also tried the simple benchmark on qemu(just for reference, since
> >> I have no other riscv boards except the lichee RV dock board):
> >> swp out w/o patch: 30066 bytes/ms (mean of 10 tests)
> >> swp out w/ patch: 130055 bytes/ms (mean of 10 tests)
> >> so improved by 332.7% 
> > 
> > I'll give QEMU a go so :)
>