From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx139.postini.com [74.125.245.139]) by kanga.kvack.org (Postfix) with SMTP id 434FF6B0031 for ; Fri, 9 Aug 2013 21:39:21 -0400 (EDT) Received: by mail-ve0-f171.google.com with SMTP id pa12so4462013veb.16 for ; Fri, 09 Aug 2013 18:39:20 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20130809155309.71d93380425ef8e19c0ff44c@linux-foundation.org> References: <1375829050-12654-1-git-send-email-hannes@cmpxchg.org> <20130809155309.71d93380425ef8e19c0ff44c@linux-foundation.org> Date: Fri, 9 Aug 2013 18:39:20 -0700 Message-ID: Subject: Re: [patch 0/9] mm: thrash detection-based file cache sizing v3 From: Ozgun Erdogan Content-Type: multipart/alternative; boundary=047d7b6d88c8c69c9c04e38df58f Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Johannes Weiner , linux-mm@kvack.org, Andi Kleen , Andrea Arcangeli , Greg Thelen , Christoph Hellwig , Hugh Dickins , Jan Kara , KOSAKI Motohiro , Mel Gorman , Minchan Kim , Peter Zijlstra , Rik van Riel , Michel Lespinasse , Seth Jennings , Roman Gushchin , Metin Doslu , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org --047d7b6d88c8c69c9c04e38df58f Content-Type: text/plain; charset=ISO-8859-1 Hi Andrew, One common use case where this is really helpful is in data analytics. Assume that you regularly analyze some chunk of data, say one month's worth, and you run SQL queries or MapReduce jobs on this data. Let's also assume you want to serve the current month's data from memory. Going with an example, let's say data for March takes 60% of total memory. You run queries over that data, and it gets pulled into the active list. Comes next month, you want to query April's data (which again holds 60% of memory). Since analytic queries sequentially walk over data, April's data never becomes active, doesn't get pulled into memory, and you're stuck with serving queries from disk. To overcome this issue, you could regularly drop the page cache, or advise customers to provision clusters whose cumulative memory is 2x the working set. Neither are that ideal. My understanding is that this patch resolves this issue, but then again my knowledge of the Linux memory manager is pretty limited. So please call off if I'm off here. Thanks, Ozgun On Fri, Aug 9, 2013 at 3:53 PM, Andrew Morton wrote: > On Tue, 6 Aug 2013 18:44:01 -0400 Johannes Weiner > wrote: > > > This series solves the problem by maintaining a history of pages > > evicted from the inactive list, enabling the VM to tell streaming IO > > from thrashing and rebalance the page cache lists when appropriate. > > Looks nice. The lack of testing results is conspicuous ;) > > It only really solves the problem in the case where > > size-of-inactive-list < size-of-working-set < size-of-total-memory > > yes? In fact less than that, because the active list presumably > doesn't get shrunk to zero (how far *can* it go?). I wonder how many > workloads fit into those constraints in the real world. > > --047d7b6d88c8c69c9c04e38df58f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Andrew,

One common use case where thi= s is really helpful is in data analytics. Assume that you regularly analyze= some chunk of data, say one month's worth, and you run SQL queries or = MapReduce jobs on this data. Let's also assume you want to serve the cu= rrent month's data from memory.

Going with an example, let's say data for March takes 60%= of total memory. You run queries over that data, and it gets pulled into t= he active list. Comes next month, you want to query April's data (which= again holds 60% of memory). Since analytic queries sequentially walk over = data, April's data never becomes active, doesn't get pulled into me= mory, and you're stuck with serving queries from disk.

To overcome this issue, you could regularly drop the page cache, or adv= ise customers to provision clusters whose cumulative memory is 2x the worki= ng set. Neither are that ideal. My understanding is that this patch resolve= s this issue, but then again my knowledge of the Linux memory manager is pr= etty limited. So please call off if I'm off here.

Thanks,
Ozgun


On Fri, Aug 9, 2013 at = 3:53 PM, Andrew Morton <akpm@linux-foundation.org> w= rote:
On Tue, =A06 Aug 2013 18:4= 4:01 -0400 Johannes Weiner <hannes= @cmpxchg.org> wrote:

> This series solves the problem by maintaining a history of pages
> evicted from the inactive list, enabling the VM to tell streaming IO > from thrashing and rebalance the page cache lists when appropriate.
Looks nice. The lack of testing results is conspicuous ;)

It only really solves the problem in the case where

=A0 =A0 =A0 =A0 size-of-inactive-list < size-of-working-set < size-of= -total-memory

yes? =A0In fact less than that, because the active list presumably
doesn't get shrunk to zero (how far *can* it go?). =A0I wonder how many=
workloads fit into those constraints in the real world.


--047d7b6d88c8c69c9c04e38df58f-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org