Is there any way to determine fragmentation for compressed btrfs volumes?

Is there any way to determine fragmentation for compressed btrfs volumes?

[Clemens Eisserer]

Hi,

I've been running btrfs on my SSD powered laptop for about a year and
a half  (with force-compress=lzo and autodefrag) and it seems the
volume has degraded quite a lot although I am not using snapshot
functionality.
No matter what I do (scrub, defrag ob the whole volume), files seem to
stay highly fragmented - at least that is my guess.
When e.g. starting up X11+XFCE the hdd led is lid for about 10s (the
Samsung 830 can do up to 80k/iops random read), also starting
google-chrome results in an active SSD for ~5s (although the chrome
binary + libs is for sure < 200mb, so should be loaded in 500ms).

What I wonder is how can I diagnose whats happening and why the FS is
rather slow?
The only target value I know is the extend count,  which doesn't seem
to be meaningful when using fragmentation.

Thank you in advance, Clemens Eisserer

Re: Is there any way to determine fragmentation for compressed btrfs volumes?

[Duncan]

Clemens Eisserer posted on Tue, 08 Apr 2014 13:46:37 +0200 as excerpted:

> I've been running btrfs on my SSD powered laptop for about a year and a
> half  (with force-compress=lzo and autodefrag) and it seems the volume
> has degraded quite a lot although I am not using snapshot functionality.
> No matter what I do (scrub, defrag ob the whole volume), files seem to
> stay highly fragmented - at least that is my guess.
> When e.g. starting up X11+XFCE the hdd led is lid for about 10s (the
> Samsung 830 can do up to 80k/iops random read), also starting
> google-chrome results in an active SSD for ~5s (although the chrome
> binary + libs is for sure < 200mb, so should be loaded in 500ms).
> 
> What I wonder is how can I diagnose whats happening and why the FS is
> rather slow?
> The only target value I know is the extend count,  which doesn't seem to
> be meaningful when using fragmentation.

You're correct, filefrag doesn't yet understand btrfs compression, which 
breaks files into 128 MiB compression blocks, and thus reports that as 
fragmentation.  There's some work in the pipeline to teach filefrag about 
compression, but they're trying to do it in a generic/standardized way so 
other filesystems can use the same API once it's available, so they're 
obviously taking the "best" route rather than the "fast" route, and it's 
taking some time, but btrfs and Linux in general will be better for it in 
the end.

There's no other known/easy way to check filefrag, that understands btrfs 
compression, unfortunately.

You mention trying scrub and defragging the entire volume, but you don't 
mention balance.  Balance by default rewrites all chunks (tho you can add 
filters to rewrite only say data chunks, not metadata, if you like), so 
that's what I'd say to try, as it should defrag in the process.

Tho we've seen a few reports from people saying a full balance actually 
made for instance boot times longer instead of shorter, too.  I haven't 
seen and don't have an explanation for that. <shrug>

Meanwhile, have you tried a trim/discard (fstrim command), and/or do you 
run with the discard (or is it trim) mount option?  For SSDs that support 
trim, as anything half-modern should (there's a discussion in the 
archives about reading hdparm or smartctl output to tell, or just run an 
fstrim and see if it helps), a filesystem trim once in awhile could help 
a great deal, particularly if you're running close to full capacity on 
your SSD.  However, until very recently most consumer grade hardware (as 
opposed to expensive enterprise grade) implemented trim/discard as a non-
queued command, which is why it's not automatically enabled with the ssd 
mount option (which is enabled automatically on btrfs when the kernel 
detects a non-rotational media device).  For such hardware an fstrim once 
in awhile is preferred to the mount option.

Additionally, a general recommendation for SSDs is to leave 20-30% of the 
drive entirely unallocated -- create partitions smaller than the drive or 
if you're putting the filesystem on the drive directly without 
partitions, make it smaller than the drive, by 20-30%.  That leaves the 
SSD's firmware plenty of room to shuffle things around in ordered to 
manage erase-blocks and the like.

FWIW here I left over 40% free, mainly because I ended up buying bigger 
SSDs than I had intended due to price/availability when I went shopping.  
I don't use the trim/discard mount option, but I do fstrim once in 
awhile.  Additionally, I use compress=lzo (decided not to use force here) 
and noatime mount options as well as autodefrag, and the kernel detects 
ssd so adds that on its own.  I've seen no such slowdowns here and in 
fact some things seem to be faster now than they were some months ago.
I guess I'm following best-practices and thus potentially getting best-
case results, but I'm not going to complain! =:^)

One more thing to consider.  The btrfs of a year and a half ago was a 
rather less mature btrfs than we have today.  I recently booted to backup 
here, and did a brand new mkfs.btrfs on my working filesystems to take 
advantage of several of the newer features (like the larger 16k metadata 
nodes), restoring from backup afterward.  It may be that is actually part 
of the reason things seem to be faster now.  You might consider trying 
that, if you have backups around to restore from (as you should, since 
btrfs isn't yet fully mature and stable yet), and either a current kernel 
and btrfs rescue media, or can boot from the backups (as I did).

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Is there any way to determine fragmentation for compressed btrfs volumes?

[Clemens Eisserer]

Hi Ducan,

> You mention trying scrub and defragging the entire volume, but you don't
> mention balance.  Balance by default rewrites all chunks (tho you can add
> filters to rewrite only say data chunks, not metadata, if you like), so
> that's what I'd say to try, as it should defrag in the process.
>
> Tho we've seen a few reports from people saying a full balance actually
> made for instance boot times longer instead of shorter, too.  I haven't
> seen and don't have an explanation for that. <shrug>

Ah sorry, I confused scrub with balance.
I didn't do a scrub but actually I tried to balance the device as I
was told basically all data has to go through the allocator again
which most likely will improve on-disk layout. Unfourtunatly in my
case it made the situation a lot worse, when running this Linux system
in virtualbox (where disk access has a lot more overhead) it is now
even with the SSD super slow.


> Meanwhile, have you tried a trim/discard (fstrim command), and/or do you
> run with the discard (or is it trim) mount option?
I have tried with discard on and off - however this isn't a SSD issue.

It is slow even for read-only workload, and windows on the same SSD
works as expected (with CrystalDiskMark showing the dirve meets the
specs (~350mb/s seq. write, 500b/s sequential read, good 4k random
values).
So I doubt this is an SSD issue.


> One more thing to consider.  The btrfs of a year and a half ago was a
> rather less mature btrfs than we have today.  I recently booted to backup
> here, and did a brand new mkfs.btrfs on my working filesystems to take
> advantage of several of the newer features

I had hopes I could avoid that.
Having a FS in such a degraded state with the only option to re-create
it isn't that compelling ;)

Thanks & regards, Clemens

Re: Is there any way to determine fragmentation for compressed btrfs volumes?

[Clemens Eisserer]

Hi again,

So it seems performance issues caused by FS degredation are in the
current development stage of btrfs not getting a lot of attention.
Hopefully production use on facebook servers will expose one or
another issue and with the developers being employed there chances are
good btrfs will handle workload like mine more gracefully with these
issues fixed.

Thanks & best regards, Clemens

2014-04-08 21:41 GMT+02:00 Clemens Eisserer <linuxhippy@gmail.com>:
> Hi Ducan,
>
>> You mention trying scrub and defragging the entire volume, but you don't
>> mention balance.  Balance by default rewrites all chunks (tho you can add
>> filters to rewrite only say data chunks, not metadata, if you like), so
>> that's what I'd say to try, as it should defrag in the process.
>>
>> Tho we've seen a few reports from people saying a full balance actually
>> made for instance boot times longer instead of shorter, too.  I haven't
>> seen and don't have an explanation for that. <shrug>
>
> Ah sorry, I confused scrub with balance.
> I didn't do a scrub but actually I tried to balance the device as I
> was told basically all data has to go through the allocator again
> which most likely will improve on-disk layout. Unfourtunatly in my
> case it made the situation a lot worse, when running this Linux system
> in virtualbox (where disk access has a lot more overhead) it is now
> even with the SSD super slow.
>
>
>> Meanwhile, have you tried a trim/discard (fstrim command), and/or do you
>> run with the discard (or is it trim) mount option?
> I have tried with discard on and off - however this isn't a SSD issue.
>
> It is slow even for read-only workload, and windows on the same SSD
> works as expected (with CrystalDiskMark showing the dirve meets the
> specs (~350mb/s seq. write, 500b/s sequential read, good 4k random
> values).
> So I doubt this is an SSD issue.
>
>
>> One more thing to consider.  The btrfs of a year and a half ago was a
>> rather less mature btrfs than we have today.  I recently booted to backup
>> here, and did a brand new mkfs.btrfs on my working filesystems to take
>> advantage of several of the newer features
>
> I had hopes I could avoid that.
> Having a FS in such a degraded state with the only option to re-create
> it isn't that compelling ;)
>
> Thanks & regards, Clemens