What is the current status of defragmentation?

What is the current status of defragmentation?

[Adam Ryczkowski]

Today I booted my linux-mint 14 into the latest 3.10.1 kernel to
defragment the root btrfs filesystem on my ASUS N56VZ laptop with hybrid
Seagate Momentus XT disk. I did something like

   find / -exec btrfs filesystem defrag {}

To my amazement it didn't really made a lot of I/O (the HDD LED wasn't
blinking). Confident that at the beginning the defragmentation is
CPU-bound (like it sometimes is for instance in case of O&O Defrag on
Windows world), I let it go on, and I switched myself to my other
chores. After a few minutes my system froze. I was barely able to switch
to text terminal and login, but I couldn't reach bash prompt; it looked
like the OS couln't perform any I/O on the disk. At the end, the system
responded to SysRq REISUB combination though; Unfortunately I can't
reproduce exactly what I typed in, because the command was not logged to
the bash history...).

It clearly showed me, that defragmenting the filesystem is not that
trivial, as it is for ext4. So I have quesions:

* Is the defragmentation of the whole filesystem supported at all? I
can't find a single reference that it is, and a syntax of btrfs-progs
suggest that it isn't. If supported, under what conditions? Like what %
of free space should be available?

* How to check the level of defragmentation, and what are the reasonable
threshold values, that should indicate the desktop filesystem needs
defragmenting? I know, that everyone's millage my vary; I just want to
know some values as a point-of-reference.

* What is the recommended command, that would efficiently defragment the
whole file system, preferably with some sort of progress indication?
Does this command

find / -type f -o -type d -print0 | xargs --null --no-run-if-empty btrfs
filesystem defragment -cv

look like a reasonable idiom for defragmenting the whole filesystem?

Thank you for your help,

Adam Ryczkowski

Re: What is the current status of defragmentation?

[George Mitchell]

Hello Adam,  I routinely defrag my filesystems and here is how I do it:

find /home -type d -mtime -3 -o -type f -mtime -3 | egrep -v 
"Cache|cache" | while read file; do /usr/sbin/btrfs filesystem defrag -f 
-v "${file}"; done

The above is what I use to defrag my data (non-OS) files.  I use a while 
loop because it relieves pressure on the system by guaranteeing that 
each file and directory are processed one at a time.  I use "-mtime -3" 
because I run this script daily via anacron and I see no reason to 
defragment user files over and over again.  I specifically filter out 
"cache" directories because they take forever to defragment and are 
refragmented in no time anyway and they are simply low priority for me 
in terms of performance. I may revisit that policy again sometime in the 
future, but for now I have been skipping them.

The following is what I use for system files:


find / -type d -mtime -3 -o -type f | while read file; do 
/usr/sbin/btrfs filesystem  defrag -f -v "${file}"; done

This is very similar to what I use for user files except I go back only 
for 3 days on directories (meta data), but run defrag on every file in 
the system every day just in case.  This approach has worked very well 
for me, but your mileage may vary (no guarantees).  Also, in the case of 
the system files noted above, I defrag daily and ALWAYS OFFLINE, because 
at this point as I understand things, btrfs cannot defragment files when 
they are currently in use.  I get around this issue by doing system (/) 
defrags only offline.

That's how I do it.

- George



On 07/17/2013 03:18 PM, Adam Ryczkowski wrote:
> Today I booted my linux-mint 14 into the latest 3.10.1 kernel to
> defragment the root btrfs filesystem on my ASUS N56VZ laptop with hybrid
> Seagate Momentus XT disk. I did something like
>
>     find / -exec btrfs filesystem defrag {}
>
> To my amazement it didn't really made a lot of I/O (the HDD LED wasn't
> blinking). Confident that at the beginning the defragmentation is
> CPU-bound (like it sometimes is for instance in case of O&O Defrag on
> Windows world), I let it go on, and I switched myself to my other
> chores. After a few minutes my system froze. I was barely able to switch
> to text terminal and login, but I couldn't reach bash prompt; it looked
> like the OS couln't perform any I/O on the disk. At the end, the system
> responded to SysRq REISUB combination though; Unfortunately I can't
> reproduce exactly what I typed in, because the command was not logged to
> the bash history...).
>
> It clearly showed me, that defragmenting the filesystem is not that
> trivial, as it is for ext4. So I have quesions:
>
> * Is the defragmentation of the whole filesystem supported at all? I
> can't find a single reference that it is, and a syntax of btrfs-progs
> suggest that it isn't. If supported, under what conditions? Like what %
> of free space should be available?
>
> * How to check the level of defragmentation, and what are the reasonable
> threshold values, that should indicate the desktop filesystem needs
> defragmenting? I know, that everyone's millage my vary; I just want to
> know some values as a point-of-reference.
>
> * What is the recommended command, that would efficiently defragment the
> whole file system, preferably with some sort of progress indication?
> Does this command
>
> find / -type f -o -type d -print0 | xargs --null --no-run-if-empty btrfs
> filesystem defragment -cv
>
> look like a reasonable idiom for defragmenting the whole filesystem?
>
> Thank you for your help,
>
> Adam Ryczkowski
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: What is the current status of defragmentation?

[George Mitchell]

Indeed! That is a VERY good question and one I have not seen a good 
answer to thus far.  A traditional defragger in the Windows realm, which 
defrags on a global basis rather than directory by directory, file by 
file, also "restacks" the file and directory structure in a way that 
minimizes free space and the result is also close to a maximum amount of 
contiguous free space somewhere within the filesystem after the defrag 
process finishes.  As you correctly point out, no one has indicated a 
method as to how that can be accomplished with btrfs at this point.  I 
am not even sure that rebuilding the partition from scratch would do the 
job.  My solution so far is just to always make sure to have a 
ridiculous amount of free space available, but that cannot go on forever 
(and is very inefficient).  So it would be nice to know what the 
solution to this conundrum would be.  Thanks for pointing this out.

On 07/18/2013 12:12 AM, Adam Ryczkowski wrote:
> On 07/18/2013 02:17 AM, George Mitchell wrote:
>> find /home -type d -mtime -3 -o -type f -mtime -3 | egrep -v
>> "Cache|cache" | while read file; do /usr/sbin/btrfs filesystem defrag
>> -f -v "${file}"; done
> Thank you for your answer.
>
> This still defragments file-after-file. But what about a little
> consolidation of free space? As I understand, if there is no consecutive
> free space for a file's extents, it is impossible to defragment it, no
> matter how many times I run the above command, am I right?
>
> Adam Ryczkowski
>
> <http://www.google.com/>
>
>
>

Re: What is the current status of defragmentation?

[Duncan]

Adam Ryczkowski posted on Thu, 18 Jul 2013 00:18:23 +0200 as excerpted:

> Today I booted my linux-mint 14 into the latest 3.10.1 kernel to
> defragment the root btrfs filesystem on my ASUS N56VZ laptop with hybrid
> Seagate Momentus XT disk. I did something like
> 
>    find / -exec btrfs filesystem defrag {}

> [description of experience snipped] It clearly showed me, that
> defragmenting the filesystem is not that trivial, as it is for ext4.
> So I have quesions: [snipped]

> Does this command
> 
> find / -type f -o -type d -print0 | xargs --null --no-run-if-empty btrfs
> filesystem defragment -cv
> 
> look like a reasonable idiom for defragmenting the whole filesystem?

Given that you don't mention the btrfs wiki, and to a large extent your 
questions are covered there, I'll assume you're not familiar with it.

The short answer is that yes, a find of that nature, piped to xargs to 
run btrfs filesystem defrag, is the suggested solution.  However, you 
really need to read up on the wiki, and then ask any remaining questions 
you may have.  (I still had some questions myself.)

Bookmarking link:  https://btrfs.wiki.kernel.org/

Defrag is covered there in some detail, including a recommended find-
piped defrag command. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: What is the current status of defragmentation?

[Shridhar Daithankar]

On Thursday, July 18, 2013 12:18:23 AM you wrote:
> * Is the defragmentation of the whole filesystem supported at all? I
> can't find a single reference that it is, and a syntax of btrfs-progs
> suggest that it isn't. If supported, under what conditions? Like what %
> of free space should be available?

It works for me, I have done it few times on 3.9.9/arch. But I have kept 
plenty of free space handy to avoid any corner cases.

> * How to check the level of defragmentation, and what are the reasonable
> threshold values, that should indicate the desktop filesystem needs
> defragmenting? I know, that everyone's millage my vary; I just want to
> know some values as a point-of-reference.

Thats a hard question to answer but after you fully defrag the system(every 
btrfs mount/partition/filesystem), rebooting immediately with 
compress/autodefrag should do it automatically, since then.

Are you mounting with noatime? storing access time could lead to massive 
direcory level fragmentation which is hard to measure.

filefrag can help you but its per file and does not exactly give the level of 
fragmentation.
> 
> * What is the recommended command, that would efficiently defragment the
> whole file system, preferably with some sort of progress indication?
> Does this command
> 
> find / -type f -o -type d -print0 | xargs --null --no-run-if-empty btrfs
> filesystem defragment -cv
> 
> look like a reasonable idiom for defragmenting the whole filesystem?

again, I use this. no progress indicator but it works

----------
for dir in / /home ;
do

find $dir -mount -type d -exec btrfs fi defrag '{}' \; ;
find $dir -mount -type f -exec btrfs fi defrag '{}' \; ;

done
----------

HTH


-- 
Regards
 Shridhar

Re: What is the current status of defragmentation?

[Duncan]

Shridhar Daithankar posted on Thu, 18 Jul 2013 08:29:22 +0530 as
excerpted:

> [A]fter you fully defrag the system(every btrfs
> mount/partition/filesystem), rebooting immediately with
> compress/autodefrag should do it automatically, since then.

Indeed.

It's worth noting that the automated install system used by many distros 
leaves system files (obviously, no user files at that point) rather 
fragmented.  Since the install process may leave no opportunity to mount 
with autodefrag during the install, it's worth either pre-creating/pre-
mounting the system partitions and skipping the part of the install 
process that would create and mount them, or installing to a temporary 
location and copying the whole install to a final location once you have 
the final destinations created/mounted with autodefrag and optionally 
compress, etc.  That'll save quite some time of the initial defrag 
slowing down the system, as well as allow you to defrag otherwise in-use 
files that aren't "online defragged" otherwise.

> Are you mounting with noatime? storing access time could lead to massive
> direcory level fragmentation which is hard to measure.

The kernel's relatime default helps, but unless you're running something 
like mutt that really depends on atime, it's worth using noatime as a 
rule.  (The only volumes I don't use noatime on are virtual filesystems 
such as tmpfs and sysfs, where it's simply a memory access in any case 
and I've seen no qualified opinion or benchmarks arguing either way.)

> filefrag can help you but its per file and does not exactly give the
> level of fragmentation.

Someone did mention that btrfs compressed files larger than some size 
(the btrfs leaf size?) will always look fragmented to filefrag.  I do not 
personally understand enough about it to verify, however.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman