Re: [RFC] mm: add pad in sturct page

From: Matthew Wilcox <willy@infradead.org>
To: Kehong Zhang <zhangkehong@hisilicon.com>
Cc: Michal Hocko <mhocko@suse.com>, Rik van Riel <riel@surriel.com>,
	Marc Zyngier <marc.zyngier@arm.com>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Vlastimil Babka <vbabka@suse.cz>,
	Will Deacon <will.deacon@arm.com>,
	Masahiro Yamada <yamada.masahiro@socionext.com>,
	"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
	Andrey Konovalov <andreyknvl@google.com>,
	Andrey Ryabinin <aryabinin@virtuozzo.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	linux-arm-kernel@lists.infradead.org,
	Souptick Joarder <jrdr.linux@gmail.com>
Subject: Re: [RFC] mm: add pad in sturct page
Date: Tue, 25 Dec 2018 05:59:40 -0800	[thread overview]
Message-ID: <20181225135940.GA20878@bombadil.infradead.org> (raw)
In-Reply-To: <1545703503-20939-1-git-send-email-zhangkehong@hisilicon.com>

On Tue, Dec 25, 2018 at 10:05:03AM +0800, Kehong Zhang wrote:
>         When I analysis the performance of using nginx for static websites.
> 	I found a high cpu workload in the code below:
> 
> 	mm/filemap.c
> 	@@ 1425,3 @@ struct page *find_get_entry(struct address_space 
> 						*mapping, pgoff_t offset)
> 	head = compound_head(page);
> 	if (!page_cache_get_speculative(head))
> 		goto repeat;
>         
> 	The code will read page->compound_head and atomic add page->_refcount,
> 	which two variables are in the same cache-line. As multiple core 
> 	read/write two variables in the same cache-line	will cause serious
> 	slowdown,

page->compound is written at allocation.  It's not modified during use,
so your analysis is wrong here.  What you're seeing is the first attempt
to access this struct page, so it's a cache-miss.  If you take away the
access to page->compound, you'll instead see all of the penalty incurred
on the next attempt to access the struct page.

>       I test the performance of using nginx for static websites
> 	in two cases, add patch below and original.
> 
> 	core num\Scenes        add pad        no pad      
> 	8core                  317654         344414

These numbers aren't particularly useful.  How stable are they from run
to run?  That is, if you do five runs, what range do they fall in?  Also,
is higher better or is lower better?  I can't tell if this is requests
per second or latency or something else being measured.

> 	As we know, struct page is the most used struct and should be as
> 	small as possible. And I don't have a better way to solve it, if
> 	anybody have any idea to solve it?

Obviously we're not going to grow struct page from 64 bytes to 256 bytes.
I have briefly investigated an idea to eliminate the reference counting
for page cache accesses that don't need to sleep, but it's a long way
down my todo list.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel