Re: price to pay for nocow file bit?

[Chris Mason <clm@fb.com>]

On Thu, Jan 8, 2015 at 11:53 AM, Lennart Poettering 
<lennart@poettering.net> wrote:
> On Thu, 08.01.15 10:56, Zygo Blaxell (ce3g8jdj@umail.furryterror.org) 
> wrote:
> 
>>  On Wed, Jan 07, 2015 at 06:43:15PM +0100, Lennart Poettering wrote:
>>  > Heya!
>>  >
>>  > Currently, systemd-journald's disk access patterns (appending to 
>> the
>>  > end of files, then updating a few pointers in the front) result in
>>  > awfully fragmented journal files on btrfs, which has a pretty
>>  > negative effect on performance when accessing them.
>>  >
>>  > Now, to improve things a bit, I yesterday made a change to 
>> journald,
>>  > to issue the btrfs defrag ioctl when a journal file is rotated,
>>  > i.e. when we know that no further writes will be ever done on the
>>  > file.
>>  >
>>  > However, I wonder now if I should go one step further even, and 
>> use
>>  > the equivalent of "chattr -C" (i.e. nocow) on all journal files. 
>> I am
>>  > wondering what price I would precisely have to pay for
>>  > that. Judging by this earlier thread:
>>  >
>>  >         http://www.spinics.net/lists/linux-btrfs/msg33134.html
>>  >
>>  > it's mostly about data integrity, which is something I can live 
>> with,
>>  > given the conservative write patterns of journald, and the fact 
>> that
>>  > we do our own checksumming and careful data validation. I mean, if
>>  > btrfs in this mode provides no worse data integrity semantics than
>>  > ext4 I am fully fine with losing this feature for these files.
>> 
>>  This sounds to me like a job for fallocate with FALLOC_FL_KEEP_SIZE.
> 
> We already use fallocate(), but this is not enough on cow file
> systems. With fallocate() you can certainly improve fragmentation when
> appending things to a file. But on a COW file system this will help
> little if we change things in the beginning of the file, since COW
> means that it will then make a copy of those blocks and alter the
> copy, but leave the original version unmodified. And if we do that all
> the time the files get heavily fragmented, even though all the blocks
> we modify have been fallocate()d initially...
> 
>>  This would work on ext4, xfs, and others, and provide the same 
>> benefit
>>  (or even better) without filesystem-specific code.  journald would
>>  preallocate a contiguous chunk past the end of the file for appends,
>>  and
> 
> That's precisely what we do. But journald's write pattern is not
> purely appending to files, it's "append something to the end, then
> link it up in the beginning". And for the "append" part we are
> fine with fallocate(). It's the "link up" part that completely fucks
> up fragmentation so far.

I think a per-file autodefrag flag would help a lot here.  We've made 
some improvements for autodefrag and slowly growing log files because 
we noticed that compression ratios on slowly growing files really 
weren't very good.  The problem was we'd never have more than a single 
block to compress, so the compression code would give up and write the 
raw data.

compression + autodefrag on the other hand would take 64-128K and recow 
it down, giving very good results.

The second problem we hit was with stable page writes.  If bdflush 
decides to write the last block in the file, it's really a wasted IO 
unless the block is fully filled.  We've been experimenting with a 
patch to leave the last block out of writepages unless its a 
fsync/O_SYNC.

I'll code up the per-file autodefrag, we've hit a few use cases that 
make sense.

-chris