From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f172.google.com ([209.85.128.172]:35272 "EHLO mail-wr0-f172.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751273AbeA3I6f (ORCPT ); Tue, 30 Jan 2018 03:58:35 -0500 Received: by mail-wr0-f172.google.com with SMTP id w50so10228646wrc.2 for ; Tue, 30 Jan 2018 00:58:34 -0800 (PST) Subject: Re: xfs_extent_busy_flush vs. aio References: <20180123161120.GC32478@bfoster.bfoster> <5f67219e-e48b-a954-69d4-318268645377@scylladb.com> <20180123164718.GE32478@bfoster.bfoster> <20180123173902.GF32478@bfoster.bfoster> <20180125130830.GD43198@bfoster.bfoster> <20180129113556.fd4y3t5yupeqtn5t@destitution> <6eddde50-ad81-f2ce-c273-86442598a934@scylladb.com> <20180129215645.c5dfa3mmoio2iovo@destitution> From: Avi Kivity Message-ID: Date: Tue, 30 Jan 2018 10:58:31 +0200 MIME-Version: 1.0 In-Reply-To: <20180129215645.c5dfa3mmoio2iovo@destitution> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Sender: linux-xfs-owner@vger.kernel.org List-ID: List-Id: xfs To: Dave Chinner Cc: Brian Foster , linux-xfs@vger.kernel.org On 01/29/2018 11:56 PM, Dave Chinner wrote: > On Mon, Jan 29, 2018 at 01:44:14PM +0200, Avi Kivity wrote: >> >> On 01/29/2018 01:35 PM, Dave Chinner wrote: >>> On Mon, Jan 29, 2018 at 11:40:27AM +0200, Avi Kivity wrote: >>>> On 01/25/2018 03:08 PM, Brian Foster wrote: >>>>> On Thu, Jan 25, 2018 at 10:50:40AM +0200, Avi Kivity wrote: >>>>>> On 01/23/2018 07:39 PM, Brian Foster wrote: >>>>>>> Yeah, could be.. perhaps the issue is that despite the large amount of >>>>>>> total free space, the free space is too fragmented to satisfy a >>>>>>> particular allocation request..? >>>>>>    from      to extents  blocks    pct >>>>>>       1       1    2702    2702   0.00 >>>>>>       2       3     690    1547   0.00 >>>>>>       4       7     115     568   0.00 >>>>>>       8      15      60     634   0.00 >>>>>>      16      31      63    1457   0.00 >>>>>>      32      63     102    4751   0.00 >>>>>>      64     127    7940  895365   0.19 >>>>>>     128     255   49680 12422100   2.67 >>>>>>     256     511    1025  417078   0.09 >>>>>>     512    1023    4170 3660771   0.79 >>>>>>    1024    2047    2168 3503054   0.75 >>>>>>    2048    4095    2567 7729442   1.66 >>>>>>    4096    8191    8688 59394413  12.76 >>>>>>    8192   16383     310 3100186   0.67 >>>>>>   16384   32767     112 2339935   0.50 >>>>>>   32768   65535      35 1381122   0.30 >>>>>>   65536  131071       8  651391   0.14 >>>>>>  131072  262143       2  344196   0.07 >>>>>>  524288 1048575       4 2909925   0.62 >>>>>> 1048576 2097151       3 3550680   0.76 >>>>>> 4194304 8388607      10 82497658  17.72 >>>>>> 8388608 16777215      10 158022653  33.94 >>>>>> 16777216 24567552       5 122778062  26.37 >>>>>> total free extents 80469 >>>>>> total free blocks 465609690 >>>>>> average free extent size 5786.2 >>>>>> >>>>>> Looks like plenty of free large extents, with most of the free space >>>>>> completely, unfragmented. >>>>>> >>>>> Indeed.. >>> You need to look at each AG, not the overall summary. You could have >>> a suboptimal AG hidden in amongst that (e.g. near ENOSPC) and it's >>> that one AG that is causing all your problems. >>> >>> There's many reasons this can happen, but the most common is the >>> working files in a directory (or subset of directories in the same >>> AG) have a combined space usage of larger than an AG .... >> That's certainly possible, even likely (one huge directory with all of the >> files). >> >> This layout is imposed on us by the compatibility gods. Is there a way to >> tell XFS to change its policy of on-ag-per-directory? > mount with inode32. That rotors files around all AGs in a round > robin fashion instead of trying to keep directory locality for a > working set. i.e. it distributes the files evenly across the > filesystem. I remember you recommending it a couple of years ago, with the qualification to only do it if we see a problem. I think we qualify now. > > However, this can have substantial impact on performance if the > workload requires locality of files for performance and you're > running on spinning rust. This is not a problem. We do have a few users on spinning disks, but they're a small minority, and our installer can avoid the inode32 option if it sees a rotational device. > OTOH, if locality is not needed then > distributing all files across all AGs evenly should give more even > capacity usage. > > And, based on what was discussed on #xfs overnight, don't bother > with the filestreams allocator - it will not solve the "full AG" > problem, but it will introduce a whole bunch of new problems for > you. > >> That system has since been recycled, but we'll keep it in mind the next time >> we see it. Also, we'll have to fix the one-huge-directory problem for sure, >> one way or another. > If inode32 doesn't work for you, the easiest way is to use a small > directory hash - create top level directory with ~2x AG count child > directories and hash your files into those. This will distribute the > files roughly evenly across all AGs in the filesystem whilst still > maintaining locality within directories. It's kind of half way > between inode64 and inode32.... > Yeah. Directory layout is sorta imposed on us, but if we have to break it, we will.