[REGRESSION]: mmap performance regression starting with k-6.1

[REGRESSION]: mmap performance regression starting with k-6.1

[Chun Ng]

Hi,

Recently I observed there is performance regression on system call mmap(..). I tried both vanilla kernels and Raspberry Pi kernels on a Raspberry Pi 4 box and the results are pretty consistent among them.

Bisection showed that the regression starts from k-6.1, and the latest vanilla k-6.7 is still showing the same regression.

The test program calls mmap/munmap for a 4K page with MAP_ANON and MAP_PRIVATE flags, and ftrace is used to measure the time spent on the do_mmap(..) call.  Measured time of a sample run with different vanilla kernel versions are:
k-5.10 and k-6.0: ~157us
k-6.1: ~194us
k-6.7: ~214us
Results are pretty consistent across multiple runs with a small percentage variance.  Ftrace shows that latency of mmap_region(...) has increased since k-6.1.  An application that makes frequent mmap(..) calls the accumulated extra latency is very noticeable. 

Please find the ftrace results and kernel config files in this folder:
https://drive.google.com/drive/folders/1qy8YTBqxu8Gdbs7IigYbSd4FXldId5sd?usp=drive_link

The test program can be found in here:
https://drive.google.com/file/d/1tG6_BbQMCHwfKebvAIAg_xqbM_lpPcuM/view?usp=sharing

Info on the testing environment:
cpufreq_governor: performance
Test machine: Raspberry Pi 4, 8GB DDR
SCHED_FIFO with priority 99 for running the test program

Vanilla kernels are not tainted. However on k-6.0 and k-6.7, I have to patch the drivers/clk/bcm/clk-raspberrypi.c file with the version in Raspberry Pi kernel tree for the CPU frequency governor to work.

Best,
Chun
[nvpublic]

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

On Wed, Nov 22, 2023 at 08:03:19PM +0000, Chun Ng wrote:
> Hi,
> 
> Recently I observed there is performance regression on system call mmap(..). I tried both vanilla kernels and Raspberry Pi kernels on a Raspberry Pi 4 box and the results are pretty consistent among them.
> 
> Bisection showed that the regression starts from k-6.1, and the latest vanilla k-6.7 is still showing the same regression.
> 
> The test program calls mmap/munmap for a 4K page with MAP_ANON and MAP_PRIVATE flags, and ftrace is used to measure the time spent on the do_mmap(..) call.  Measured time of a sample run with different vanilla kernel versions are:
> k-5.10 and k-6.0: ~157us
> k-6.1: ~194us
> k-6.7: ~214us
> Results are pretty consistent across multiple runs with a small percentage variance.  Ftrace shows that latency of mmap_region(...) has increased since k-6.1.  An application that makes frequent mmap(..) calls the accumulated extra latency is very noticeable. 

Did you mean that v6.0 doesn't have this regression?

Confused...

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Chun Ng]

>> Did you mean that v6.0 doesn't have this regression?

No, k-6.0 does NOT have this regression. The regression starts from k-6.1.

Best,
Chun

From: Bagas Sanjaya
Sent: Wednesday, November 22, 2023 5:18 PM
To: Chun Ng; Linux Kernel Mailing List
Cc: Linux Regressions; Andrew Morton; Linux Memory Management List; Liam R. Howlett; Ankita Garg
Subject: Re: [REGRESSION]: mmap performance regression starting with k-6.1


On Wed, Nov 22, 2023 at 08:03:19PM +0000, Chun Ng wrote:

> Hi,

>

> Recently I observed there is performance regression on system call mmap(..). I tried both vanilla kernels and Raspberry Pi kernels on a Raspberry Pi 4 box and the results are pretty consistent among them.

>

> Bisection showed that the regression starts from k-6.1, and the latest vanilla k-6.7 is still showing the same regression.

>

> The test program calls mmap/munmap for a 4K page with MAP_ANON and MAP_PRIVATE flags, and ftrace is used to measure the time spent on the do_mmap(..) call.  Measured time of a sample run with different vanilla kernel versions are:

> k-5.10 and k-6.0: ~157us

> k-6.1: ~194us

> k-6.7: ~214us

> Results are pretty consistent across multiple runs with a small percentage variance.  Ftrace shows that latency of mmap_region(...) has increased since k-6.1.  An application that makes frequent mmap(..) calls the accumulated extra latency is very noticeable.



Did you mean that v6.0 doesn't have this regression?



Confused...



--

An old man doll... just what I always wanted! - Clara

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Bagas Sanjaya]

On 11/23/23 10:06, Chun Ng wrote:
>>> Did you mean that v6.0 doesn't have this regression?
> 
> No, k-6.0 does NOT have this regression. The regression starts from k-6.1.
> 

Thanks for confirmation.

-- 
An old man doll... just what I always wanted! - Clara

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 2004 bytes --]

On Wed, Nov 22, 2023 at 08:03:19PM +0000, Chun Ng wrote:
> Hi,
> 
> Recently I observed there is performance regression on system call mmap(..). I tried both vanilla kernels and Raspberry Pi kernels on a Raspberry Pi 4 box and the results are pretty consistent among them.
> 
> Bisection showed that the regression starts from k-6.1, and the latest vanilla k-6.7 is still showing the same regression.
> 
> The test program calls mmap/munmap for a 4K page with MAP_ANON and MAP_PRIVATE flags, and ftrace is used to measure the time spent on the do_mmap(..) call.  Measured time of a sample run with different vanilla kernel versions are:
> k-5.10 and k-6.0: ~157us
> k-6.1: ~194us
> k-6.7: ~214us
> Results are pretty consistent across multiple runs with a small percentage variance.  Ftrace shows that latency of mmap_region(...) has increased since k-6.1.  An application that makes frequent mmap(..) calls the accumulated extra latency is very noticeable. 
> 
> Please find the ftrace results and kernel config files in this folder:
> https://drive.google.com/drive/folders/1qy8YTBqxu8Gdbs7IigYbSd4FXldId5sd?usp=drive_link
> 
> The test program can be found in here:
> https://drive.google.com/file/d/1tG6_BbQMCHwfKebvAIAg_xqbM_lpPcuM/view?usp=sharing
> 
> Info on the testing environment:
> cpufreq_governor: performance
> Test machine: Raspberry Pi 4, 8GB DDR
> SCHED_FIFO with priority 99 for running the test program
> 
> Vanilla kernels are not tainted. However on k-6.0 and k-6.7, I have to patch the drivers/clk/bcm/clk-raspberrypi.c file with the version in Raspberry Pi kernel tree for the CPU frequency governor to work.
> 

The next step is to find the commit that introduces your regression with
`git bisect`. If you haven't done so, see
Documentation/admin-guide/bug-bisect.rst for instructions.

Anyway, I'm adding this regression to regzbot:

#regzbot ^introduced: v6.0..v6.1

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Liam R. Howlett]

* Bagas Sanjaya <bagasdotme@gmail.com> [231123 00:07]:
> On Wed, Nov 22, 2023 at 08:03:19PM +0000, Chun Ng wrote:
> > Hi,
> > 
> > Recently I observed there is performance regression on system call mmap(..). I tried both vanilla kernels and Raspberry Pi kernels on a Raspberry Pi 4 box and the results are pretty consistent among them.
> > 
> > Bisection showed that the regression starts from k-6.1, and the latest vanilla k-6.7 is still showing the same regression.

This is almost certainly the maple tree.  The tree is slower on writes
than the rbtree and so if the benchmark mmaps/munmaps in a tight loop
you will see this slow down.  What you are doing is measuring the speed
of inserting and removing a VMA with this benchmark, so it's not really
something that happens - we usually use the mapping between adding and
removing it.

What this gains us is the ability to remove contention on the mmap lock
during page faults.  If you were to test contention around that lock,
you will see a slowdown until you reach v6.4, where per-vma locking
started to show up.  More benchmarking will show different types of
fault handling outside of the mmap lock until (I believe) 6.6, where
most (or all?) types are supported.

Although this is expected, I am still looking to reduce any real
workloads that may suffer.  I've been reducing the allocations, for
example.

> > 
> > The test program calls mmap/munmap for a 4K page with MAP_ANON and MAP_PRIVATE flags, and ftrace is used to measure the time spent on the do_mmap(..) call.  Measured time of a sample run with different vanilla kernel versions are:
> > k-5.10 and k-6.0: ~157us
> > k-6.1: ~194us
> > k-6.7: ~214us

I would have expected v6.7 to remain closer to v6.1, but that may depend
on the minor versions you have been testing and what fixes have landed
there.


> > Results are pretty consistent across multiple runs with a small percentage variance.  Ftrace shows that latency of mmap_region(...) has increased since k-6.1.  An application that makes frequent mmap(..) calls the accumulated extra latency is very noticeable. 
> > 
> > Please find the ftrace results and kernel config files in this folder:
> > https://drive.google.com/drive/folders/1qy8YTBqxu8Gdbs7IigYbSd4FXldId5sd?usp=drive_link
> > 
> > The test program can be found in here:
> > https://drive.google.com/file/d/1tG6_BbQMCHwfKebvAIAg_xqbM_lpPcuM/view?usp=sharing
> > 
> > Info on the testing environment:
> > cpufreq_governor: performance
> > Test machine: Raspberry Pi 4, 8GB DDR
> > SCHED_FIFO with priority 99 for running the test program
> > 
> > Vanilla kernels are not tainted. However on k-6.0 and k-6.7, I have to patch the drivers/clk/bcm/clk-raspberrypi.c file with the version in Raspberry Pi kernel tree for the CPU frequency governor to work.
> > 
> 
> The next step is to find the commit that introduces your regression with
> `git bisect`. If you haven't done so, see
> Documentation/admin-guide/bug-bisect.rst for instructions.
> 
> Anyway, I'm adding this regression to regzbot:
> 
> #regzbot ^introduced: v6.0..v6.1
> 
> Thanks.
> 
> -- 
> An old man doll... just what I always wanted! - Clara

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Matthew Wilcox]

On Thu, Nov 23, 2023 at 12:07:40PM +0700, Bagas Sanjaya wrote:
> Anyway, I'm adding this regression to regzbot:
> 
> #regzbot ^introduced: v6.0..v6.1

this is not a regression.  close it, you idiot.

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 375 bytes --]

On Thu, Nov 23, 2023 at 03:38:18PM +0000, Matthew Wilcox wrote:
> On Thu, Nov 23, 2023 at 12:07:40PM +0700, Bagas Sanjaya wrote:
> > Anyway, I'm adding this regression to regzbot:
> > 
> > #regzbot ^introduced: v6.0..v6.1
> 
> this is not a regression.  close it, you idiot.
> 
> 

why?

Confused...

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Matthew Wilcox]

On Fri, Nov 24, 2023 at 07:15:34AM +0700, Bagas Sanjaya wrote:
> On Thu, Nov 23, 2023 at 03:38:18PM +0000, Matthew Wilcox wrote:
> > On Thu, Nov 23, 2023 at 12:07:40PM +0700, Bagas Sanjaya wrote:
> > > Anyway, I'm adding this regression to regzbot:
> > > 
> > > #regzbot ^introduced: v6.0..v6.1
> > 
> > this is not a regression.  close it, you idiot.
> > 
> > 
> 
> why?
> 
> Confused...

yes.  you're perpetually confused.  stop trying to help, you only make
things worse.  learn about the things you work on, or give up.

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Linux regression tracking #update (Thorsten Leemhuis)]

[TLDR: This mail in primarily relevant for Linux kernel regression
tracking. See link in footer if these mails annoy you.]

On 23.11.23 15:34, Liam R. Howlett wrote:
> * Bagas Sanjaya <bagasdotme@gmail.com> [231123 00:07]:
>> On Wed, Nov 22, 2023 at 08:03:19PM +0000, Chun Ng wrote:
>>>
>>> Recently I observed there is performance regression on system call mmap(..). I tried both vanilla kernels and Raspberry Pi kernels on a Raspberry Pi 4 box and the results are pretty consistent among them.
>>>
>>> Bisection showed that the regression starts from k-6.1, and the latest vanilla k-6.7 is still showing the same regression.
> 
> This is almost certainly the maple tree.  The tree is slower on writes
> than the rbtree and so if the benchmark mmaps/munmaps in a tight loop
> you will see this slow down. [...]
>
>> Anyway, I'm adding this regression to regzbot:
>> #regzbot ^introduced: v6.0..v6.1

Liam, many thx for your reply. I known that you are still working on
optimizing things in this area again, so I don't think this is worth
tracking this as a regression: that doesn't buy us much afaics. And it
might not be a regression at all anyway (not totally sure, didn't look
into the details due to the former aspect; sounded a bit like the
problem only can be seen in a microbenchmark; whatever).

#regzbot resolve: not worth tracking

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Greg KH]

On Fri, Nov 24, 2023 at 01:04:28AM +0000, Matthew Wilcox wrote:
> On Fri, Nov 24, 2023 at 07:15:34AM +0700, Bagas Sanjaya wrote:
> > On Thu, Nov 23, 2023 at 03:38:18PM +0000, Matthew Wilcox wrote:
> > > On Thu, Nov 23, 2023 at 12:07:40PM +0700, Bagas Sanjaya wrote:
> > > > Anyway, I'm adding this regression to regzbot:
> > > > 
> > > > #regzbot ^introduced: v6.0..v6.1
> > > 
> > > this is not a regression.  close it, you idiot.
> > > 
> > > 
> > 
> > why?
> > 
> > Confused...
> 
> yes.  you're perpetually confused.  stop trying to help, you only make
> things worse.  learn about the things you work on, or give up.

Um, is this really called for?  Bagas is trying to help track
regressions, and if something isn't a regression like you say here, a
simple "This is not a regression, it's already resolved in newer
kernels" is fine.

Resorting to insults on the reporter is not ok.

greg k-h

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Matthew Wilcox]

On Fri, Nov 24, 2023 at 11:52:06AM +0000, Greg KH wrote:
> On Fri, Nov 24, 2023 at 01:04:28AM +0000, Matthew Wilcox wrote:
> > On Fri, Nov 24, 2023 at 07:15:34AM +0700, Bagas Sanjaya wrote:
> > > On Thu, Nov 23, 2023 at 03:38:18PM +0000, Matthew Wilcox wrote:
> > > > On Thu, Nov 23, 2023 at 12:07:40PM +0700, Bagas Sanjaya wrote:
> > > > > Anyway, I'm adding this regression to regzbot:
> > > > > 
> > > > > #regzbot ^introduced: v6.0..v6.1
> > > > 
> > > > this is not a regression.  close it, you idiot.
> > > > 
> > > > 
> > > 
> > > why?
> > > 
> > > Confused...
> > 
> > yes.  you're perpetually confused.  stop trying to help, you only make
> > things worse.  learn about the things you work on, or give up.
> 
> Um, is this really called for?  Bagas is trying to help track
> regressions, and if something isn't a regression like you say here, a
> simple "This is not a regression, it's already resolved in newer
> kernels" is fine.

Bagas has a long history of inappropriately attempting to "help" and due
to a lack of understanding wasting peoples time.  He's not too dissimilar
to the various wrongbots we've had over the years, including Richard B
Johnson, Markus Elfring, etc.  I've tried to help guide him towards being
a more productive contributor, but'have failed.  Mostly I ignore him now,
but when he's instructing a bot to harass me, that crosses a line.

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Greg KH]

On Fri, Nov 24, 2023 at 03:06:44PM +0000, Matthew Wilcox wrote:
> On Fri, Nov 24, 2023 at 11:52:06AM +0000, Greg KH wrote:
> > On Fri, Nov 24, 2023 at 01:04:28AM +0000, Matthew Wilcox wrote:
> > > On Fri, Nov 24, 2023 at 07:15:34AM +0700, Bagas Sanjaya wrote:
> > > > On Thu, Nov 23, 2023 at 03:38:18PM +0000, Matthew Wilcox wrote:
> > > > > On Thu, Nov 23, 2023 at 12:07:40PM +0700, Bagas Sanjaya wrote:
> > > > > > Anyway, I'm adding this regression to regzbot:
> > > > > > 
> > > > > > #regzbot ^introduced: v6.0..v6.1
> > > > > 
> > > > > this is not a regression.  close it, you idiot.
> > > > > 
> > > > > 
> > > > 
> > > > why?
> > > > 
> > > > Confused...
> > > 
> > > yes.  you're perpetually confused.  stop trying to help, you only make
> > > things worse.  learn about the things you work on, or give up.
> > 
> > Um, is this really called for?  Bagas is trying to help track
> > regressions, and if something isn't a regression like you say here, a
> > simple "This is not a regression, it's already resolved in newer
> > kernels" is fine.
> 
> Bagas has a long history of inappropriately attempting to "help" and due
> to a lack of understanding wasting peoples time.  He's not too dissimilar
> to the various wrongbots we've had over the years, including Richard B
> Johnson, Markus Elfring, etc.  I've tried to help guide him towards being
> a more productive contributor, but'have failed.  Mostly I ignore him now,
> but when he's instructing a bot to harass me, that crosses a line.

Nope, still not justification to lash out at an individual, sorry.

Please be more careful, and kind, in the future.

greg k-h

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Linux regression tracking (Thorsten Leemhuis)]

On 24.11.23 16:06, Matthew Wilcox wrote:
>  Mostly I ignore him now, but when he's instructing a bot
> to harass me, that crosses a line.

I'm curious: How is regzbot able to harass you? It as of now and likely
for at least another year is not able to send mails on its own -- by
design, as I wanted to ensure it doesn't harass anyone.

Sure, I might manually send a mail if something looks stalled in
regzbot. But before I do that I always do a sanity check to avoid
annoying people. Do I sometimes make mistakes or miss something in that
process? Sure. But that happens to all of us.

Ciao, Thorsten

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Chun Ng]

[-- Attachment #1: Type: text/plain, Size: 55 bytes --]


Thanks for everyone's reply and Liam's explanation!

[-- Attachment #2: Type: text/html, Size: 667 bytes --]

Re: [REGRESSION]: mmap performance regression starting with k-6.1

[Matthew Wilcox]

On Sun, Nov 26, 2023 at 03:18:54PM +0800, David Wang wrote:
> I add memory access between mmap and munmap to the simple stress, and timeit.

It's still not a very good benchmark ...

> My test code now is:
> 
> 	#define MAXN 1024
> 	struct { void* addr; size_t n; } maps[MAXN];
> 	void accessit(char *addr, size_t n) {
> 		for (int i=0; i<n; i+=128) addr[i]=i;
> 	}
> 	int main() {
> 		int i, n, k, r;
> 		void *p;
> 		for (i=0; i<MAXN; i++) {
> 			n = 1024*((rand()%32)+1);
> 			p = mmap(NULL, n, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);

So 'n' is now a number between 1kB and 32kB.  That's not terribly
realistic; I'd say you want to be more like

			n = 4096 * ((rand() % 512) + 1));

> 		for (i=0; i<10000000; i++) {
> 			k = rand()%MAXN;
> 	#ifdef PAGE_FAULT
> 			accessit((char*)maps[k].addr, maps[k].n);
> 	#endif
> 			r = munmap(maps[k].addr, maps[k].n);
> 			if (r) {
> 				perror("fail to munmap");
> 				return -1;
> 			}
> 			n = 1024*((rand()%32)+1);
> 			p = mmap(NULL, n, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);

Are you simulating something a real application actually does?
Because this all seems very weird and micro-benchmark to me.  The real
applications we've benchmarked see a speedup so I'm not thrilled about
chasing down something that no real application does.

In terms of what's going on in the kernel, for each loop, you're calling
munmap(), taking between 1 and 8 page faults, then calling mmap().
That may just be too few page faults to see the benefit of the maple tree.