* defrag makes fragmentation worse
@ 2011-10-08 22:17 Chester
2011-10-11 6:19 ` Li Zefan
0 siblings, 1 reply; 9+ messages in thread
From: Chester @ 2011-10-08 22:17 UTC (permalink / raw)
To: linux-btrfs
Kernel 3.1-rc8
btrfs-progs-0.19
mount options: noatime,autodefrag (space_cache is enabled)
There are snapshots present on the filesystem.
When I do a btrfs fi defrag on a file, the file becomes much more
fragmented. The end result can be a file with 20k times more fragments
than before. Initially I thought the extents were just smaller but
were next to each other, so I checked with both 'filefrag' as well as
'filefrag -v'. Both reported the same number. I don't know if this has
anything to do with having snapshots at all because files that haven't
been snapshotted yet are affected by this. Disk space isn't an issue
since my 1TB disk isn't even halfway filled, and moving the file
around actually gives me fairly contiguous files.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: defrag makes fragmentation worse
2011-10-08 22:17 defrag makes fragmentation worse Chester
@ 2011-10-11 6:19 ` Li Zefan
2011-10-11 20:47 ` Chester
0 siblings, 1 reply; 9+ messages in thread
From: Li Zefan @ 2011-10-11 6:19 UTC (permalink / raw)
To: Chester; +Cc: linux-btrfs
Chester wrote:
> Kernel 3.1-rc8
> btrfs-progs-0.19
> mount options: noatime,autodefrag (space_cache is enabled)
> There are snapshots present on the filesystem.
>
> When I do a btrfs fi defrag on a file, the file becomes much more
> fragmented. The end result can be a file with 20k times more fragments
> than before. Initially I thought the extents were just smaller but
> were next to each other, so I checked with both 'filefrag' as well as
> 'filefrag -v'. Both reported the same number. I don't know if this has
> anything to do with having snapshots at all because files that haven't
> been snapshotted yet are affected by this. Disk space isn't an issue
> since my 1TB disk isn't even halfway filled, and moving the file
> around actually gives me fairly contiguous files.
Was any process doing heavy I/O work while you were defragging the file?
Could you try to remount the fs without autodefrag, and defrag
the file again? Firstly let's see if autodefrag makes things worse.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: defrag makes fragmentation worse
2011-10-11 6:19 ` Li Zefan
@ 2011-10-11 20:47 ` Chester
[not found] ` <CAAE6i0j=7ZoYnXsmNdWewmszSpaq4HLcy+M7--3s2hiy2p4TBA@mail.gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Chester @ 2011-10-11 20:47 UTC (permalink / raw)
To: Li Zefan; +Cc: linux-btrfs
On Tue, Oct 11, 2011 at 1:19 AM, Li Zefan <lizf@cn.fujitsu.com> wrote:
> Was any process doing heavy I/O work while you were defragging the file?
>
> Could you try to remount the fs without autodefrag, and defrag
> the file again? Firstly let's see if autodefrag makes things worse.
>
As far as I can tell, autodefrag seems to be making it worse.
I turned off autodefrag but btrfs fi defrag is still having some
trouble defragmenting highly fragmented files. For example, when I
defragment a ~400mb file with 3 extents, it does a nice job and turns
it into 1 extent.
However, for more fragmented files (of the same size) say with 1k-2k
extents, the defrag utility has a hard time. It either reduces the
number of extents by a very small amount or it doesn't change at all.
I decided to run btrfs fi defrag on the same file again just to make
sure nothing else can be done, but after running it again, the number
of extents in that file actually DOUBLED! But this isn't as bad as
when autodefrag is on, where extents can increase by 20x
tl;dr: with autodefrag off, btrfs fi defrag isn't nearly as
counter-productive as it is with autodefrag on.
^ permalink raw reply [flat|nested] 9+ messages in thread
* defrag makes fragmentation worse
@ 2011-06-09 22:48 Johannes Hirte
2011-06-09 23:53 ` David Sterba
0 siblings, 1 reply; 9+ messages in thread
From: Johannes Hirte @ 2011-06-09 22:48 UTC (permalink / raw)
To: linux-btrfs
I've observed several times that after a btrfs filesystem defrag a file was way
more fragmented than before. For example, a file that was recently written, had
10 extents (output from filefrag). After a defrag filefrag showed more than 1900
extents. For curiosity, a simple copy of this "defragmented" file reduced the
number of fragments to 1. With a different file I got 63 extents before and over
3000 extents after defrag.
It's no problem if defrag can't reduce the fragmentation. But in this case it
shouldn't be done at all.
regards,
Johannes
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: defrag makes fragmentation worse
2011-06-09 22:48 Johannes Hirte
@ 2011-06-09 23:53 ` David Sterba
2011-06-23 0:12 ` Johannes Hirte
0 siblings, 1 reply; 9+ messages in thread
From: David Sterba @ 2011-06-09 23:53 UTC (permalink / raw)
To: Johannes Hirte; +Cc: linux-btrfs
On Fri, Jun 10, 2011 at 12:48:36AM +0200, Johannes Hirte wrote:
> I've observed several times that after a btrfs filesystem defrag a file was way
> more fragmented than before. For example, a file that was recently written, had
> 10 extents (output from filefrag). After a defrag filefrag showed more than 1900
> extents.
> For curiosity, a simple copy of this "defragmented" file reduced the
> number of fragments to 1. With a different file I got 63 extents before and over
> 3000 extents after defrag.
Do you have compression enabled? Or autodefrag mount option?
'filefrag -v' will tell you size of the extents, would be interesting
to see.
> It's no problem if defrag can't reduce the fragmentation. But in this case it
> shouldn't be done at all.
AFAIK defragmentation just reads the file, marks all pages dirty and
lets it be written back. If the free space is fragmented, so will be
the newly written copy. I do not know if there is some logic comparing
old and new extent layout (or if it's even possible).
david
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: defrag makes fragmentation worse
2011-06-09 23:53 ` David Sterba
@ 2011-06-23 0:12 ` Johannes Hirte
0 siblings, 0 replies; 9+ messages in thread
From: Johannes Hirte @ 2011-06-23 0:12 UTC (permalink / raw)
To: dave; +Cc: linux-btrfs
On Friday 10 June 2011 01:53:41 David Sterba wrote:
> On Fri, Jun 10, 2011 at 12:48:36AM +0200, Johannes Hirte wrote:
> > I've observed several times that after a btrfs filesystem defrag a file
> > was way more fragmented than before. For example, a file that was
> > recently written, had 10 extents (output from filefrag). After a defrag
> > filefrag showed more than 1900 extents.
> > For curiosity, a simple copy of this "defragmented" file reduced the
> > number of fragments to 1. With a different file I got 63 extents before
> > and over 3000 extents after defrag.
>
> Do you have compression enabled? Or autodefrag mount option?
No compression, only autodefrag was enabled this time. The last time before I
saw this, autodefrag didn't exist.
> 'filefrag -v' will tell you size of the extents, would be interesting
> to see.
Needed some tries but now I have one. Before defrag the file consisted of 174
extents. Now after defrag there are 786 extents. filefrag -v shows:
Filesystem type is: 9123683e
File size of test1 is 106857600 (26089 blocks, blocksize 4096)
ext logical physical expected length flags
0 0 7037185 299
1 299 7989102 7037483 64
2 363 7037548 7989165 1
3 364 7989538 7037548 64
4 428 7037613 7989601 1
5 429 7990288 7037613 64
6 493 7037678 7990351 1
7 494 7992819 7037678 64
8 558 7037743 7992882 1
9 559 7993037 7037743 64
10 623 7037809 7993100 1
11 624 7993171 7037809 64
12 688 9547471 7993234 1
13 689 9547947 9547471 64
14 753 9547536 9548010 1
...
89 16159 9696920 9696590 64
490 16223 9696655 9696983 1
491 16224 9700654 9696655 64
492 16288 9697127 9700717 276
493 16564 9700718 9697402 64
494 16628 9697467 9700781 1
...
781 25924 12575962 12535459 64
782 25988 12535524 12576025 1
783 25989 12576026 12535524 64
784 26053 12531601 12576089 1
785 26054 12541596 12531601 35 eof
test1: 786 extents found
> > It's no problem if defrag can't reduce the fragmentation. But in this
> > case it shouldn't be done at all.
>
> AFAIK defragmentation just reads the file, marks all pages dirty and
> lets it be written back. If the free space is fragmented, so will be
> the newly written copy. I do not know if there is some logic comparing
> old and new extent layout (or if it's even possible).
If there is some comparison between old and new, it seems to be broken (or the
fiemap from btrfs).
regards,
Johannes
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-10-12 15:14 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-10-08 22:17 defrag makes fragmentation worse Chester
2011-10-11 6:19 ` Li Zefan
2011-10-11 20:47 ` Chester
[not found] ` <CAAE6i0j=7ZoYnXsmNdWewmszSpaq4HLcy+M7--3s2hiy2p4TBA@mail.gmail.com>
2011-10-12 2:37 ` Li Zefan
2011-10-12 4:14 ` Chester
2011-10-12 15:14 ` Chris Mason
-- strict thread matches above, loose matches on Subject: below --
2011-06-09 22:48 Johannes Hirte
2011-06-09 23:53 ` David Sterba
2011-06-23 0:12 ` Johannes Hirte
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox