Re: ENSOPC on a 10% used disk

[Avi Kivity <avi@scylladb.com>]

On 19/10/2018 10.51, Dave Chinner wrote:
> On Thu, Oct 18, 2018 at 04:36:42PM +0300, Avi Kivity wrote:
>> On 18/10/2018 14.00, Avi Kivity wrote:
>>>> Can I get access to the metadump to dig around in the filesystem
>>>> directly so I can see how everything has ended up laid out? that
>>>> will help me work out what is actually occurring and determine if
>>>> mkfs/mount options can address the problem or whether deeper
>>>> allocator algorithm changes may be necessary....
>>> I will ask permission to share the dump.
>> I'll send you a link privately.
> Thanks - I've started looking at this - the information here is
> just layout stuff - I'm omitted filenames and anything else that
> might be identifying from the output.
>
> Looking at a commit log file:
>
> stat.size = 33554432
> stat.blocks = 34720
> fsxattr.xflags = 0x800 [----------e-----]
> fsxattr.projid = 0
> fsxattr.extsize = 33554432
> fsxattr.cowextsize = 0
> fsxattr.nextents = 14
>
>
> and the layout:
>
> EXT: FILE-OFFSET      BLOCK-RANGE            AG AG-OFFSET            TOTAL FLAGS
>    0: [0..4079]:       2646677520..2646681599 22 (95606800..95610879)  4080 001010
>    1: [4080..8159]:    2643130384..2643134463 22 (92059664..92063743)  4080 001010
>    2: [8160..12239]:   2642124816..2642128895 22 (91054096..91058175)  4080 001010
>    3: [12240..16319]:  2640666640..2640670719 22 (89595920..89599999)  4080 001010
>    4: [16320..18367]:  2640523264..2640525311 22 (89452544..89454591)  2048 000000
>    5: [18368..20415]:  2640119808..2640121855 22 (89049088..89051135)  2048 000000
>    6: [20416..21287]:  2639874064..2639874935 22 (88803344..88804215)   872 001111
>    7: [21288..21295]:  2639874936..2639874943 22 (88804216..88804223)     8 011111
>    8: [21296..24495]:  2639874944..2639878143 22 (88804224..88807423)  3200 001010
>    9: [24496..26543]:  2639427584..2639429631 22 (88356864..88358911)  2048 000000
>   10: [26544..28591]:  2638981120..2638983167 22 (87910400..87912447)  2048 000000
>   11: [28592..30639]:  2638770176..2638772223 22 (87699456..87701503)  2048 000000
>   12: [30640..31279]:  2638247952..2638248591 22 (87177232..87177871)   640 001111
>   13: [31280..34719]:  2638248592..2638252031 22 (87177872..87181311)  3440 011010
>   14: [34720..65535]:  hole                                           30816
>
> The first thing I note is the initial allocations are just short of
> 2MB and so the extent size hint is, indeed, being truncated here
> according to contiguous free space limitations. I had thought that
> should occur from reading the code, but it's complex and I wasn't
> 100% certain what minimum allocation length would be used.
>
> Looking at the system batchlog files, I'm guessing the filesystem
> ran out of contiguous 32MB free space extents some time around
> September 25. The *Data.db files from 24 Sep and earlier then are
> all nice 32MB extents, from 25 sep onwards they never make the full
> 32MB (30-31MB max). eg, good:
>
>   EXT: FILE-OFFSET       BLOCK-RANGE          AG AG-OFFSET            TOTAL FLAGS
>     0: [0..65535]:       350524552..350590087  3 (2651272..2716807)   65536 001111
>     1: [65536..131071]:  353378024..353443559  3 (5504744..5570279)   65536 001111
>     2: [131072..196607]: 355147016..355212551  3 (7273736..7339271)   65536 001111
>     3: [196608..262143]: 360029416..360094951  3 (12156136..12221671) 65536 001111
>     4: [262144..327679]: 362244144..362309679  3 (14370864..14436399) 65536 001111
>     5: [327680..343415]: 365809456..365825191  3 (17936176..17951911) 15736 001111
>
> bad:
>
> EXT: FILE-OFFSET      BLOCK-RANGE            AG AG-OFFSET            TOTAL FLAGS
>    0: [0..64127]:       512855496..512919623  4 (49024456..49088583) 64128 001111
>    1: [64128..128247]:  266567048..266631167  2 (34651528..34715647) 64120 001010
>    2: [128248..142327]: 264401888..264415967  2 (32486368..32500447) 14080 001111
>   


So extent size is a hint but the extent alignment is a hard requirement. 
Since eventually the ENOSPC happened due to the alignment restriction, I 
think the alignment requirement should be made a hint too.


> Hmmm - there's 2 million files in this filesystem. that is quite a
> lot...
>
> Ok... I see where all the files are - there's a db that was
> snapshotted every half hour going back to December 19 2017. There's
> 55GB of snapshot data there 14362 snapshots holding in 1.8million
> files.
>
> Ok, now I understand how the filesystem got into this mess. It has
> nothing really to do with the filesystem allocator, geometry, extent
> size hints, etc. It isn't really even an XFS specific problem - I
> think most filesystems would be in trouble if you did this to them.


Well, if you create snapshots and never delete them you'd run into a 
real ENOSPC sooner or later, so the main problem was lack of snapshot 
hygene. But it did trigger a premature ENOSPC due to the alignment 
restriction on those small files with hints (which I'm going to remove).


>
> First, let me demonstrate that the freespace fragmentation is caused
> by these snapshots by removing them all:
>
> before:
>     from      to extents  blocks    pct
>        1       1    5916    5916   0.00
>        2       3   10235   22678   0.01
>        4       7   12251   66829   0.02
>        8      15    5521   59556   0.01
>       16      31    5703  132031   0.03
>       32      63    9754  463825   0.11
>       64     127   16742 1590339   0.37
>      128     255 1550511 390108625  89.87
>      256     511   71516 29178504   6.72
>      512    1023      19   15355   0.00
>     1024    2047     287  461824   0.11
>     2048    4095     528 1611413   0.37
>     4096    8191    1537 10352304   2.38
>     8192   16383       2   19015   0.00
>
> Run a delete:
>
> for d in snapshots/*; do
> 	rm -rf $d &
> done
>
> <cranking along at ~12,000 write iops>
>
> # uptime
> 17:41:08 up 22:07,  1 user,  load average: 14293.17, 13840.37, 9517.14
> #
>
> 500,000 files removed:
>     from      to extents  blocks    pct
>       64     127   22564 2054234   0.47
>      128     255  900480 226428059  51.43
>      256     511  189904 91033237  20.68
>      512    1023   68304 54958788  12.48
>     1024    2047   25187 38284024   8.70
>     2048    4095    5508 15204528   3.45
>     4096    8191    1665 10999789   2.50
>     8192   16383      15  139424   0.03
>
> 1m files removed:
>    from      to extents  blocks    pct
>       64     127   21940 1991685   0.45
>      128     255  536985 134731402  30.35
>      256     511  152092 73465972  16.55
>      512    1023  100471 82971130  18.69
>     1024    2047   48519 74016490  16.67
>     2048    4095   17272 49209538  11.09
>     4096    8191    4307 25135374   5.66
>     8192   16383     135 1254037   0.28
>
> 1.5m files removed:
>    from      to extents  blocks    pct
>       64     127    9851  924782   0.20
>      128     255  227945 57079302  12.32
>      256     511   38723 18129086   3.91
>      512    1023   33547 28027554   6.05
>     1024    2047   31904 50171699  10.83
>     2048    4095   25263 75381887  16.27
>     4096    8191   16885 102836365  22.19
>     8192   16383    6367 68809645  14.85
>    16384   32767    1862 40183775   8.67
>    32768   65535     385 16228869   3.50
>    65536  131071      51 4213237   0.91
>   131072  262143       6  958528   0.21
>
> after:
>    from      to extents  blocks    pct
>      128     255  154063 38785829   8.64
>      256     511   11037 4942114   1.10
>      512    1023    8576 6930035   1.54
>     1024    2047    8496 13464298   3.00
>     2048    4095    7664 23034455   5.13
>     4096    8191    8497 55217061  12.31
>     8192   16383    4233 45867691  10.22
>    16384   32767    1533 33488995   7.46
>    32768   65535     520 23924895   5.33
>    65536  131071     305 28675646   6.39
>   131072  262143     230 42411732   9.45
>   262144  524287      98 37213190   8.29
>   524288 1048575      41 29163579   6.50
> 1048576 2097151      27 40502889   9.03
> 2097152 4194303       5 14576157   3.25
> 4194304 8388607       2 10005670   2.23
>
> Ok, so the results is not perfect, but there are now huge contiguous
> free space extents available again - ~70% of the free space is now
> contiguous extents >=32MB in length. There's every chance that the
> fs would confinue to help reform large contiguous free spaces as the
> database files come and go now, as long as the snapshot problem is
> dealt with.
>
> So, what's the problem? Well, it's simply that the workload is
> mixing data with vastly different temporal characteristics in the
> same physical locality. Every half an hour, a set of ~100 smallish
> files are written into a new directory which lands them at the low
> endof the largest free space extent in that AG. Each new snapshot
> directory ends up in a different AG, so it slowly spreads the
> snapshots across all the AGs in the filesystem.


Not exactly - those snapshots are hard links into the live database 
files, which eventually get removed. Usually, small files get removed 
early, but with the snapshots they get to live forever.


> Each snapshot effective appends to the current working area in the
> AG, chopping it out of the largest contiguous free space. By the
> time the next snapshot in that AG comes around, there's other new
> short term data between the old snapshot and the new one. The new
> snapshot chops up the largest freespace, and on goes the cycle.
>
> Eventually the short term data between the snapshots gets removed,
> but this doesn't reform large contiguous free spaces because the
> snapshot data is in the way. And so this cycle continues with the
> snapshot data chopping up the largest freespace extents in the
> filesystem until there's not more large free space extents to be
> found.
>
> The solution is to manage the snapshot data better. We need to keep
> all the long term data physically isolated from the short term data
> so they don't fragment free space. A short term application level
> solution would require migrating the snapshot data out of the
> filesystem to somewhere else and point to it with symlinks.


Snapshots should not live forever on the disk. The procedure is to 
create a snapshot, copy it away, and then delete the snapshot. It's okay 
to let snapshots live for a while, but not all of them and not without a 
bound on their lifetime.


The filesystem did have a role in this, by requiring alignment of the 
extent to the RAID stripe size. Now, given that this was a RAID with one 
member, alignment is pointless, but most of our deployments are to RAID 
arrays with >1 members, and alignment does save 12.5% of IOPS compared 
to un-aligned extents for compactions and writes (our scans/writes use 
128k buffers, and the alignment is to 1MB). The database caused the 
problem by indirectly requiring 1MB alignment for files that are much 
smaller than 1MB, and the user contributed to the problem by causing 
millions of such small files to be kept.


>
>  From the filesystem POV, I'm not sure that there is much we can do
> about this directly - we have no idea what the lifetime of the data
> is going to be....
>
> <ding>
>
> Hold on....
>
> <rummage in code>
>
> ....we already have an interface so setting those sorts of hints.
>
> fcntl(F_SET_RW_HINT, rw_hint)
>
> /*
>   * Valid hint values for F_{GET,SET}_RW_HINT. 0 is "not set", or can be
>   * used to clear any hints previously set.
>   */
> #define RWF_WRITE_LIFE_NOT_SET  0
> #define RWH_WRITE_LIFE_NONE     1
> #define RWH_WRITE_LIFE_SHORT    2
> #define RWH_WRITE_LIFE_MEDIUM   3
> #define RWH_WRITE_LIFE_LONG     4
> #define RWH_WRITE_LIFE_EXTREME  5
>
> Avi, does this sound like something that you could use to
> classify the different types of data the data base writes out?


So long as the penalty for a mis-classification is not too large, we can 
for sure. Commitlog files have short lifespan, and so do newly born 
small data files. Those small data files are compacted into increasingly 
larger and long-lived files, and this information is known at the time 
of creation.


Even without the filesystem altering its allocation according to the 
hint, this is still useful, since the disk will alter its internal 
allocation and maybe do something useful with it (as long as the 
filesystem passes the hint to the disk).


>
> I'll need to have a think about how to apply this to the allocator
> policy algorithms before going any further, but I suspect making use
> of this hint interface will allow us prevent interleaving of short
> and long term data so avoid the freespace fragmentation it is
> causing here....


IIUC, the problem (of having ENOSPC on a 10% used disk) is not 
fragmentation per se, it's the alignment requirement. To take it to 
extreme, a 1TB disk can only hold a million files if those files must be 
aligned to 1MB, even if everything is perfectly laid out. For sure 
fragmentation would have degraded performance sooner or later, but 
that's not as bad as that ENOSPC.


I'm addressing the ENOSPC by removing the extent allocation hint on 
files that are known small (and increasing their application buffer 
sizes). In fact that will increase fragmentation as the filesystem will 
allocate on extent per buffer, rather than one extent for the entire 
file. But I think that, given that the extent size is treated as a hint 
(or so I infer from the fact that we have <32MB extents), so should the 
alignment. Perhaps allocation with a hint should be performed in two 
passes, first trying to match size and alignment, and second relaxing 
both restrictions.