defrag makes fragmentation worse

defrag makes fragmentation worse

[Johannes Hirte]

I've observed several times that after a btrfs filesystem defrag a file was way 
more fragmented than before. For example, a file that was recently written, had 
10 extents (output from filefrag). After a defrag filefrag showed more than 1900 
extents. For curiosity, a simple copy of this "defragmented" file reduced the 
number of fragments to 1. With a different file I got 63 extents before and over 
3000 extents after defrag. 
It's no problem if defrag can't reduce the fragmentation. But in this case it 
shouldn't be done at all. 

regards,
  Johannes

Re: defrag makes fragmentation worse

[David Sterba]

On Fri, Jun 10, 2011 at 12:48:36AM +0200, Johannes Hirte wrote:
> I've observed several times that after a btrfs filesystem defrag a file was way 
> more fragmented than before. For example, a file that was recently written, had 
> 10 extents (output from filefrag). After a defrag filefrag showed more than 1900 
> extents.
> For curiosity, a simple copy of this "defragmented" file reduced the 
> number of fragments to 1. With a different file I got 63 extents before and over 
> 3000 extents after defrag.

Do you have compression enabled? Or autodefrag mount option?

'filefrag -v' will tell you size of the extents, would be interesting
to see.

> It's no problem if defrag can't reduce the fragmentation. But in this case it 
> shouldn't be done at all. 

AFAIK defragmentation just reads the file, marks all pages dirty and
lets it be written  back. If the free space is fragmented, so will be
the newly written copy. I do not know if there is some logic comparing
old and new extent layout (or if it's even possible).


david

Re: defrag makes fragmentation worse

[Johannes Hirte]

On Friday 10 June 2011 01:53:41 David Sterba wrote:
> On Fri, Jun 10, 2011 at 12:48:36AM +0200, Johannes Hirte wrote:
> > I've observed several times that after a btrfs filesystem defrag a file
> > was way more fragmented than before. For example, a file that was
> > recently written, had 10 extents (output from filefrag). After a defrag
> > filefrag showed more than 1900 extents.
> > For curiosity, a simple copy of this "defragmented" file reduced the
> > number of fragments to 1. With a different file I got 63 extents before
> > and over 3000 extents after defrag.
> 
> Do you have compression enabled? Or autodefrag mount option?

No compression, only autodefrag was enabled this time. The last time before I 
saw this, autodefrag didn't exist. 

> 'filefrag -v' will tell you size of the extents, would be interesting
> to see.

Needed some tries but now I have one. Before defrag the file consisted of 174 
extents. Now after defrag there are 786 extents. filefrag -v shows:

Filesystem type is: 9123683e
File size of test1 is 106857600 (26089 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0  7037185             299 
   1     299  7989102  7037483     64 
   2     363  7037548  7989165      1 
   3     364  7989538  7037548     64 
   4     428  7037613  7989601      1 
   5     429  7990288  7037613     64 
   6     493  7037678  7990351      1 
   7     494  7992819  7037678     64 
   8     558  7037743  7992882      1 
   9     559  7993037  7037743     64 
  10     623  7037809  7993100      1 
  11     624  7993171  7037809     64 
  12     688  9547471  7993234      1 
  13     689  9547947  9547471     64 
  14     753  9547536  9548010      1 
...
   89   16159  9696920  9696590     64 
 490   16223  9696655  9696983      1 
 491   16224  9700654  9696655     64 
 492   16288  9697127  9700717    276 
 493   16564  9700718  9697402     64 
 494   16628  9697467  9700781      1 
...
 781   25924 12575962 12535459     64 
 782   25988 12535524 12576025      1 
 783   25989 12576026 12535524     64 
 784   26053 12531601 12576089      1 
 785   26054 12541596 12531601     35 eof
test1: 786 extents found

> > It's no problem if defrag can't reduce the fragmentation. But in this
> > case it shouldn't be done at all.
> 
> AFAIK defragmentation just reads the file, marks all pages dirty and
> lets it be written  back. If the free space is fragmented, so will be
> the newly written copy. I do not know if there is some logic comparing
> old and new extent layout (or if it's even possible).

If there is some comparison between old and new, it seems to be broken (or the 
fiemap from btrfs).

regards,
  Johannes

defrag makes fragmentation worse

[Chester]

Kernel 3.1-rc8
btrfs-progs-0.19
mount options: noatime,autodefrag (space_cache is enabled)
There are snapshots present on the filesystem.

When I do a btrfs fi defrag on a file, the file becomes much more
fragmented. The end result can be a file with 20k times more fragments
than before. Initially I thought the extents were just smaller but
were next to each other, so I checked with both 'filefrag' as well as
'filefrag -v'. Both reported the same number. I don't know if this has
anything to do with having snapshots at all because files that haven't
been snapshotted yet are affected by this. Disk space isn't an issue
since my 1TB disk isn't even halfway filled, and moving the file
around actually gives me fairly contiguous files.

Re: defrag makes fragmentation worse

[Li Zefan]

Chester wrote:
> Kernel 3.1-rc8
> btrfs-progs-0.19
> mount options: noatime,autodefrag (space_cache is enabled)
> There are snapshots present on the filesystem.
> 
> When I do a btrfs fi defrag on a file, the file becomes much more
> fragmented. The end result can be a file with 20k times more fragments
> than before. Initially I thought the extents were just smaller but
> were next to each other, so I checked with both 'filefrag' as well as
> 'filefrag -v'. Both reported the same number. I don't know if this has
> anything to do with having snapshots at all because files that haven't
> been snapshotted yet are affected by this. Disk space isn't an issue
> since my 1TB disk isn't even halfway filled, and moving the file
> around actually gives me fairly contiguous files.

Was any process doing heavy I/O work while you were defragging the file?

Could you try to remount the fs without autodefrag, and defrag
the file again? Firstly let's see if autodefrag makes things worse.

Re: defrag makes fragmentation worse

[Chester]

On Tue, Oct 11, 2011 at 1:19 AM, Li Zefan <lizf@cn.fujitsu.com> wrote:
> Was any process doing heavy I/O work while you were defragging the file?
>
> Could you try to remount the fs without autodefrag, and defrag
> the file again? Firstly let's see if autodefrag makes things worse.
>

As far as I can tell, autodefrag seems to be making it worse.

I turned off autodefrag but btrfs fi defrag is still having some
trouble defragmenting highly fragmented files. For example, when I
defragment a ~400mb file with 3 extents, it does a nice job and turns
it into 1 extent.

However, for more fragmented files (of the same size) say with 1k-2k
extents, the defrag utility has a hard time. It either reduces the
number of extents by a very small amount or it doesn't change at all.
I decided to run btrfs fi defrag on the same file again just to make
sure nothing else can be done, but after running it again, the number
of extents in that file actually DOUBLED! But this isn't as bad as
when autodefrag is on, where extents can increase by 20x

tl;dr: with autodefrag off, btrfs fi defrag isn't nearly as
counter-productive as it is with autodefrag on.

Re: defrag makes fragmentation worse

[Li Zefan]

05:15, Chester wrote:
> Okay, this pretty much confirms that autodefrag makes defrag totally
> useless.. The above email was with autodefrag turned off..
> 
> With autodefrag on:
> 339 MB file
> initially 4 extents
> 
> btrfs fi defrag filename
> filefrag filename
> 
> about 900 extents
> 
> I did btrfs defrag two more times and ended up with 2048 extents
> 
> I attached the output of filefrag -v, which shows a pretty peculiar pattern.

This confirmed my speculation. I've fixed this bug a month ago, but
the patch hasn't hitted mainline.

You can try it out:

http://marc.info/?l=linux-btrfs&m=131495014823121&w=2

With this bug fixed, I think autodefrag won't be problematic.

--
Li Zefan

Re: defrag makes fragmentation worse

[Chester]

On Tue, Oct 11, 2011 at 9:37 PM, Li Zefan <lizf@cn.fujitsu.com> wrote:
>
>
> This confirmed my speculation. I've fixed this bug a month ago, but
> the patch hasn't hitted mainline.
>
> You can try it out:
>
> http://marc.info/?l=linux-btrfs&m=131495014823121&w=2
>
> With this bug fixed, I think autodefrag won't be problematic.
>
> --
> Li Zefan

Yup. That patch fixes it for me. Thanks.

Re: defrag makes fragmentation worse

[Chris Mason]

Excerpts from Chester's message of 2011-10-12 00:14:21 -0400:
> On Tue, Oct 11, 2011 at 9:37 PM, Li Zefan <lizf@cn.fujitsu.com> wrote:
> >
> >
> > This confirmed my speculation. I've fixed this bug a month ago, but
> > the patch hasn't hitted mainline.
> >
> > You can try it out:
> >
> > http://marc.info/?l=linux-btrfs&m=131495014823121&w=2
> >
> > With this bug fixed, I think autodefrag won't be problematic.
> >
> > --
> > Li Zefan
> 
> Yup. That patch fixes it for me. Thanks.

Thanks guys,

This patch is queued up now for the next release (3.2)

-chris