[PATCH] mm: fix ununiform page status when writing new file with small buffer

[PATCH] mm: fix ununiform page status when writing new file with small buffer

[Robin Dong]

From: Robin Dong <sanbai@taobao.com>

When writing a new file with 2048 bytes buffer, such as write(fd, buffer, 2048), it will
call generic_perform_write() twice for every page:

	write_begin
	mark_page_accessed(page) 
	write_end

	write_begin
	mark_page_accessed(page) 
	write_end

The page 1~13th will be added to lru_add_pvecs in write_begin() and will *NOT* be added to
active_list even they have be accessed twice because they are not PageLRU(page).
But when page 14th comes, all pages will be moved from lru_add_pvecs to active_list
(by __lru_cache_add() ) in first write_begin(), now page 14th *is* PageLRU(page) and after
second write_end() it will be in active_list.

In Hadoop environment, we do comes to this situation: after writing a file, we find
out that only 14th, 28th, 42th... page are in active_list and others in inactive_list. Now
kswaped works, shrinks the inactive_list, the file only have 14th, 28th...pages in memory,
the readahead request size will be broken to only 52k (13*4k), system's performance falls
dramatically.

This problem can also replay by below steps (the machine has 8G memory):

	1. dd if=/dev/zero of=/test/file.out bs=1024 count=1048576
	2. cat another 7.5G file to /dev/null
	3. vmtouch -m 1G -v /test/file.out, it will show:

	/test/file.out
	[oooooooooooooooooooOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO] 187847/262144

	the 'o' means same pages are in memory but same are not.


The solution for this problem is simple: the 14th page should be added to lru_add_pvecs
before mark_page_accessed() just as other pages.

Signed-off-by: Robin Dong <sanbai@taobao.com>
---
 mm/swap.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/swap.c b/mm/swap.c
index 4e7e2ec..0874d44 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -399,8 +399,9 @@ void __lru_cache_add(struct page *page, enum lru_list lru)
 	struct pagevec *pvec = &get_cpu_var(lru_add_pvecs)[lru];
 
 	page_cache_get(page);
-	if (!pagevec_add(pvec, page))
+	if (!pagevec_space(pvec))
 		__pagevec_lru_add(pvec, lru);
+	pagevec_add(pvec, page);
 	put_cpu_var(lru_add_pvecs);
 }
 EXPORT_SYMBOL(__lru_cache_add);
-- 
1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: fix ununiform page status when writing new file with small buffer

[KOSAKI Motohiro]

(6/6/12 7:39 AM), Robin Dong wrote:
> From: Robin Dong<sanbai@taobao.com>
> 
> When writing a new file with 2048 bytes buffer, such as write(fd, buffer, 2048), it will
> call generic_perform_write() twice for every page:
> 
> 	write_begin
> 	mark_page_accessed(page)
> 	write_end
> 
> 	write_begin
> 	mark_page_accessed(page)
> 	write_end
> 
> The page 1~13th will be added to lru_add_pvecs in write_begin() and will *NOT* be added to
> active_list even they have be accessed twice because they are not PageLRU(page).
> But when page 14th comes, all pages will be moved from lru_add_pvecs to active_list
> (by __lru_cache_add() ) in first write_begin(), now page 14th *is* PageLRU(page) and after
> second write_end() it will be in active_list.
> 
> In Hadoop environment, we do comes to this situation: after writing a file, we find
> out that only 14th, 28th, 42th... page are in active_list and others in inactive_list. Now
> kswaped works, shrinks the inactive_list, the file only have 14th, 28th...pages in memory,
> the readahead request size will be broken to only 52k (13*4k), system's performance falls
> dramatically.
> 
> This problem can also replay by below steps (the machine has 8G memory):
> 
> 	1. dd if=/dev/zero of=/test/file.out bs=1024 count=1048576
> 	2. cat another 7.5G file to /dev/null
> 	3. vmtouch -m 1G -v /test/file.out, it will show:
> 
> 	/test/file.out
> 	[oooooooooooooooooooOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO] 187847/262144
> 
> 	the 'o' means same pages are in memory but same are not.
> 
> 
> The solution for this problem is simple: the 14th page should be added to lru_add_pvecs
> before mark_page_accessed() just as other pages.
> 
> Signed-off-by: Robin Dong<sanbai@taobao.com>
> ---
>   mm/swap.c |    3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/swap.c b/mm/swap.c
> index 4e7e2ec..0874d44 100644
> --- a/mm/swap.c
> +++ b/mm/swap.c
> @@ -399,8 +399,9 @@ void __lru_cache_add(struct page *page, enum lru_list lru)
>   	struct pagevec *pvec =&get_cpu_var(lru_add_pvecs)[lru];
> 
>   	page_cache_get(page);
> -	if (!pagevec_add(pvec, page))
> +	if (!pagevec_space(pvec))
>   		__pagevec_lru_add(pvec, lru);
> +	pagevec_add(pvec, page);
>   	put_cpu_var(lru_add_pvecs);
>   }
>   EXPORT_SYMBOL(__lru_cache_add);

Please remove pagevec completely instead of insane hacking.




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: fix ununiform page status when writing new file with small buffer

[Minchan Kim]

On 06/06/2012 08:39 PM, Robin Dong wrote:

> From: Robin Dong <sanbai@taobao.com>
> 
> When writing a new file with 2048 bytes buffer, such as write(fd, buffer, 2048), it will
> call generic_perform_write() twice for every page:
> 
> 	write_begin
> 	mark_page_accessed(page) 
> 	write_end
> 
> 	write_begin
> 	mark_page_accessed(page) 
> 	write_end
> 
> The page 1~13th will be added to lru_add_pvecs in write_begin() and will *NOT* be added to
> active_list even they have be accessed twice because they are not PageLRU(page).

> But when page 14th comes, all pages will be moved from lru_add_pvecs to active_list
                                                                          ^^^^
                                                                          inactive list

> (by __lru_cache_add() ) in first write_begin(), now page 14th *is* PageLRU(page) and after
> second write_end() it will be in active_list.

> 
> In Hadoop environment, we do comes to this situation: after writing a file, we find
> out that only 14th, 28th, 42th... page are in active_list and others in inactive_list. Now
> kswaped works, shrinks the inactive_list, the file only have 14th, 28th...pages in memory,

   ^^^^^
  kswapd

> the readahead request size will be broken to only 52k (13*4k), system's performance falls
> dramatically.


Good catch!

> 
> This problem can also replay by below steps (the machine has 8G memory):
> 
> 	1. dd if=/dev/zero of=/test/file.out bs=1024 count=1048576
> 	2. cat another 7.5G file to /dev/null
> 	3. vmtouch -m 1G -v /test/file.out, it will show:
> 
> 	/test/file.out
> 	[oooooooooooooooooooOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO] 187847/262144
> 
> 	the 'o' means same pages are in memory but same are not.
> 
> 
> The solution for this problem is simple: the 14th page should be added to lru_add_pvecs
> before mark_page_accessed() just as other pages.
> 
> Signed-off-by: Robin Dong <sanbai@taobao.com>


Reviewed-by: Minchan Kim <minchan@kernel.org>

Nitpick:
Please comment in function as well as description.
It will prevent some guy restore original code for the consistency with other pagevec_add call sites.

Thanks.

-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] mm: fix ununiform page status when writing new file with small buffer

[Robin Dong]

2012/6/11 Minchan Kim <minchan@kernel.org>:
> On 06/06/2012 08:39 PM, Robin Dong wrote:
>
>> From: Robin Dong <sanbai@taobao.com>
>>
>> When writing a new file with 2048 bytes buffer, such as write(fd, buffer, 2048), it will
>> call generic_perform_write() twice for every page:
>>
>>       write_begin
>>       mark_page_accessed(page)
>>       write_end
>>
>>       write_begin
>>       mark_page_accessed(page)
>>       write_end
>>
>> The page 1~13th will be added to lru_add_pvecs in write_begin() and will *NOT* be added to
>> active_list even they have be accessed twice because they are not PageLRU(page).
>
>> But when page 14th comes, all pages will be moved from lru_add_pvecs to active_list
>                                                                          ^^^^
>                                                                          inactive list
>
>> (by __lru_cache_add() ) in first write_begin(), now page 14th *is* PageLRU(page) and after
>> second write_end() it will be in active_list.
>
>>
>> In Hadoop environment, we do comes to this situation: after writing a file, we find
>> out that only 14th, 28th, 42th... page are in active_list and others in inactive_list. Now
>> kswaped works, shrinks the inactive_list, the file only have 14th, 28th...pages in memory,
>
>   ^^^^^
>  kswapd
>
>> the readahead request size will be broken to only 52k (13*4k), system's performance falls
>> dramatically.
>
>
> Good catch!
>
>>
>> This problem can also replay by below steps (the machine has 8G memory):
>>
>>       1. dd if=/dev/zero of=/test/file.out bs=1024 count=1048576
>>       2. cat another 7.5G file to /dev/null
>>       3. vmtouch -m 1G -v /test/file.out, it will show:
>>
>>       /test/file.out
>>       [oooooooooooooooooooOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO] 187847/262144
>>
>>       the 'o' means same pages are in memory but same are not.
>>
>>
>> The solution for this problem is simple: the 14th page should be added to lru_add_pvecs
>> before mark_page_accessed() just as other pages.
>>
>> Signed-off-by: Robin Dong <sanbai@taobao.com>
>
>
> Reviewed-by: Minchan Kim <minchan@kernel.org>
>
> Nitpick:
> Please comment in function as well as description.
> It will prevent some guy restore original code for the consistency with other pagevec_add call sites.
>
> Thanks.

Thanks for your suggestion, I will send the second version ASAP
>
> --
> Kind regards,
> Minchan Kim



-- 
--
Best Regard
Robin Dong

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>