Strange fragmentation in nearly empty filesystem

Strange fragmentation in nearly empty filesystem

[Carsten Oberscheid]

Hi there,

I am experiencing my XFS filesystem degrading over time in quite a
strange and annoying way. Googling "XFS fragmenation" tells me either
that this does not happen or to use xfs_fsr, which doesn't really help
me anymore -- see below. I'd appreciate any help on this.

Background: I am using two VMware virtual machines on my Linux
desktop. These virtual machines store images of their main memory in
.vmem files, which are about half a gigabyte in size for each of my
VMs. The .vmem files are created when starting the VM, written when
suspending it and read when resuming. I prefer suspendig and resuming
over shutting down and booting again, so with my VMs these files can
have a lifetime of several weeks.

On Ubuntu's default ext3 filesystem the vmem files showed heavy
fragmentation problems right from the start, so I switched over to
XFS. At first everything seemed fine, but after some time (about two
weeks, in the beginning) it took longer and longer to suspend the VM
in the evening and to resume it in the morning. 'sudo filefrag *'
showed heavy fragmentation (up to 50.000 extents and more) of the
.vmem files. Reading a 512M file thus fragmented takes several
minutes, and writing it takes at least twice as long.

For some time, 'sudo xfs_fsr *.vmem' used to fix this (back down to
one single extent). Replacing the files by a copy of themselves or
rebooting the VM (thus creating new .vmem files) fixed fragmentation
as well.

After some months I observed that the time it took to reach a
perceptible level of fragmentation (slow reading/writing) got shorter
and shorter. Instead of, say, two weeks it now took only a few days to
reach 10.000 extents or more.

Again some months later, xfs_fsr was unable to get the files back to
one single extent, and even new files started with at least a handful
of extents right away.

Today, a formerly new .vmem file is already after the first VM suspend
badly fragmented (about 20.000 extents) and accordingly slow to read
and write. Suspending the VM can take 15 minutes or more.

Some more facts:

 -  The filesystem is nearly empty (8% of about 500GB used, see
    below).

 -  System information, xfs_info output appended below.

 -  Other large files created in this filesystem show fragmentation
    from start as well, but they are not rewritten as often as the
    .vmem files, so they don't deteriorate as much and as quickly. So
    I don't think this to be a VMware specific problem. I think it
    just shows more obviously there.

 -  A few weeks ago, I did a fresh mkfs.xfs on the filesystem and
    restored the contents from a tar backup -- observing the same
    heavy fragmentation as before. Perhaps this did not really
    create a fresh filesystem structure?

What is happening here? I thought fragmentation would become serious
only on nearly full filesystems when there's not enough continuous
free space left. Or can free space also be fragmented over time by
some patterns of usage? Are there any XFS parameters I could try to
make this more robust against fragmentation?

Thanks in advance


Carsten Oberscheid



Running Ubuntu 8.10 with current XFS package (where can I find the XFS
version?)

[co@tangchai]~ uname -a
Linux tangchai 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:06 UTC 2008 x86_64 GNU/Linux

[co@tangchai]~ df
Filesystem           1K-blocks      Used Available Use% Mounted on
...
/dev/sdb5            488252896  40028724 448224172   9% /home

[co@tangchai]~ xfs_info /home
meta-data=/dev/sdb5              isize=256    agcount=4, agsize=30523998 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=122095992, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096  
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

[co@tangchai]~ cat /etc/mtab
...
/dev/sdb5 /home xfs rw,inode64 0 0

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Eric Sandeen]

Carsten Oberscheid wrote:
> Hi there,
> 
> I am experiencing my XFS filesystem degrading over time in quite a
> strange and annoying way. Googling "XFS fragmenation" tells me either
> that this does not happen or to use xfs_fsr, which doesn't really help
> me anymore -- see below. I'd appreciate any help on this.
> 
> Background: I am using two VMware virtual machines on my Linux
> desktop. These virtual machines store images of their main memory in
> .vmem files, which are about half a gigabyte in size for each of my
> VMs. The .vmem files are created when starting the VM, written when
> suspending it and read when resuming. I prefer suspendig and resuming
> over shutting down and booting again, so with my VMs these files can
> have a lifetime of several weeks.

I would suggest using xfs_bmap to look at the file layout, it will be
much more informative than filefrag.

I don't know how vmware is writing the files out (I don't use vmware, so
can't test) but I'd suggest looking at xfs_bmap output when it's first
created, and then also when it's first written (but perhaps before it
gets too horribly fragmented).

The first question I'd ask is whether it's a swiss-cheese looking sparse
file with many holes.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Dave Chinner]

On Fri, Jan 23, 2009 at 11:21:30AM +0100, Carsten Oberscheid wrote:
> Hi there,
> 
> I am experiencing my XFS filesystem degrading over time in quite a
> strange and annoying way. Googling "XFS fragmenation" tells me either
> that this does not happen or to use xfs_fsr, which doesn't really help
> me anymore -- see below. I'd appreciate any help on this.
> 
> Background: I am using two VMware virtual machines on my Linux
> desktop. These virtual machines store images of their main memory in
> .vmem files, which are about half a gigabyte in size for each of my
> VMs. The .vmem files are created when starting the VM, written when
> suspending it and read when resuming. I prefer suspendig and resuming
> over shutting down and booting again, so with my VMs these files can
> have a lifetime of several weeks.

Oh, that's vmware being incredibly stupid about how they write
out the memory images. They only write pages that are allocated
and it's sparse file full of holes. Effectively this guarantees
file fragmentation over time as random holes are filled. For
example, a .vmem file on a recent VM I built:

$ xfs_bmap -vvp foo.vmem |grep hole |wc -l
675
$ xfs_bmap -vvp foo.vmem |grep -v hole |wc -l
885
$

Contains 675 holes and almost 900 real extents in a 512MB memory
image that has only 160MB of data blocks allocated.

In reality, this is a classic case of the application doing a "smart
optimisation" that looks good in the short term (i.e.  saves some
disk space), but that has very bad long term side effects (i.e.
guaranteed fragmentation of the file in the long term).

You might be able to pre-allocate the .vmem files after the file
is created with an xfs_io hack prior to it being badly fragmented;
that should avoid the worse case fragmentation caused by writing
randomly to a sparse file.

In summary, this is an application problem, not a filesystem
issue.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Carsten Oberscheid]

On Sat, Jan 24, 2009 at 11:33:29AM +1100, Dave Chinner wrote:
> Oh, that's vmware being incredibly stupid about how they write
> out the memory images. They only write pages that are allocated
> and it's sparse file full of holes. Effectively this guarantees
> file fragmentation over time as random holes are filled. For
> example, a .vmem file on a recent VM I built:
> 
> $ xfs_bmap -vvp foo.vmem |grep hole |wc -l
> 675
> $ xfs_bmap -vvp foo.vmem |grep -v hole |wc -l
> 885
> $
> 
> Contains 675 holes and almost 900 real extents in a 512MB memory
> image that has only 160MB of data blocks allocated.

Well, things look a bit different over here:


[co@tangchai]~/vmware/foo ls -la *.vmem
-rw------- 1 co co 536870912 2009-01-23 10:42 foo.vmem

[co@tangchai]~/vmware/foo xfs_bmap -vvp  voo.vmem | grep hole | wc -l
28

[co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep -v hole | wc -l
98644


The hole/extent ratio cannot really be compared with your example. The
vmem file has been written about three or four times to reach this
state.

Now rebooting the VM to create a new vmem file:


[co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep hole | wc -l
3308

[co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep -v hole | wc -l
3327


That looks more like swiss cheese to me. And remember, it is a new file.

Now suspending the fresh VM for the first time, causing the vmem file
to be written again:


[co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep hole | wc -l
38

[co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep -v hole | wc -l
6678


Hmmm.

Now one more thing:


[co@tangchai]~/vmware/foo sudo xfs_fsr -v *vmem
foo.vmem
extents before:6708 after:77 DONE foo.vmem


I happily accept your point about vmware writing the vmem file in a
clumsy way that guarantees fragmentation. What bothers me is that
today these files get fragmented *much* faster than they did about a
year ago. Back then the vmem files used to start with one extent,
stayed between one and a handful for a week (being written 6-10 times)
and then rose to several thousand, maybe 10k or 20k during one or two
more weeks. Applying xfs_fsr to the file then got it back to one
extent.

Today: see above. Heavy fragmentation right from the start, jumping to
90k and more within 2 or 3 writes. No chance to defragment the file
completely with xfs_fsr.

All this on the same disk with the same filesystem which is and always
has been more than 90% empty.

So even vmware's way of writing the vmem files causes fragmentation,
something has happened affecting the way fragmentation takes
place. Can this really be an application problem, or is the
application just making something obvious that happens on the
filesystem level?

Best regards


Carsten Oberscheid

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Eric Sandeen]

Carsten Oberscheid wrote:
> On Sat, Jan 24, 2009 at 11:33:29AM +1100, Dave Chinner wrote:
>> Oh, that's vmware being incredibly stupid about how they write
>> out the memory images. They only write pages that are allocated
>> and it's sparse file full of holes. Effectively this guarantees
>> file fragmentation over time as random holes are filled. For
>> example, a .vmem file on a recent VM I built:
>>
>> $ xfs_bmap -vvp foo.vmem |grep hole |wc -l
>> 675
>> $ xfs_bmap -vvp foo.vmem |grep -v hole |wc -l
>> 885
>> $
>>
>> Contains 675 holes and almost 900 real extents in a 512MB memory
>> image that has only 160MB of data blocks allocated.
> 
> Well, things look a bit different over here:
> 
> 
> [co@tangchai]~/vmware/foo ls -la *.vmem
> -rw------- 1 co co 536870912 2009-01-23 10:42 foo.vmem
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp  voo.vmem | grep hole | wc -l
> 28
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep -v hole | wc -l
> 98644
> 
> 
> The hole/extent ratio cannot really be compared with your example. The
> vmem file has been written about three or four times to reach this
> state.

It could still be being written backwards & synchronously, or some other
way which doesn't play well with the allocator in xfs....

> Now rebooting the VM to create a new vmem file:
> 
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep hole | wc -l
> 3308
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep -v hole | wc -l
> 3327
> 
> 
> That looks more like swiss cheese to me. And remember, it is a new file.
> 
> Now suspending the fresh VM for the first time, causing the vmem file
> to be written again:
> 
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep hole | wc -l
> 38
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp  foo.vmem | grep -v hole | wc -l
> 6678
> 
> 
> Hmmm.
> 
> Now one more thing:
> 
> 
> [co@tangchai]~/vmware/foo sudo xfs_fsr -v *vmem
> foo.vmem
> extents before:6708 after:77 DONE foo.vmem

ok, so now it's reasonably rearranged; if you had 38 holes that means
around 38 extents, 77 in the end seems about right.  How many holes are
left, then?

> I happily accept your point about vmware writing the vmem file in a
> clumsy way that guarantees fragmentation. What bothers me is that
> today these files get fragmented *much* faster than they did about a
> year ago. Back then the vmem files used to start with one extent,
> stayed between one and a handful for a week (being written 6-10 times)
> and then rose to several thousand, maybe 10k or 20k during one or two
> more weeks. Applying xfs_fsr to the file then got it back to one
> extent.

It's possible that vmware changed too, I suppose.  If it's leaving holes
now, you won't get back to one extent.

> Today: see above. Heavy fragmentation right from the start, jumping to
> 90k and more within 2 or 3 writes. No chance to defragment the file
> completely with xfs_fsr.

Probably due to holes left by the application.

How hard would it be to boot a kernel from a year ago, with your current
vmware, and see how that goes - it might be an interesting test.

> All this on the same disk with the same filesystem which is and always
> has been more than 90% empty.
> 
> So even vmware's way of writing the vmem files causes fragmentation,
> something has happened affecting the way fragmentation takes
> place. Can this really be an application problem, or is the
> application just making something obvious that happens on the
> filesystem level?

I'd try to sort out the 2 moving parts you have, vmware & kernel.  See
which one seems to have affected this behavior the most; downgrade one
of the 2 pieces, and see how it behaves.

-Eric

> Best regards
> 
> 
> Carsten Oberscheid
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Carsten Oberscheid]

On Mon, Jan 26, 2009 at 12:37:01PM -0600, Eric Sandeen wrote:
> 
> It's possible that vmware changed too, I suppose.  If it's leaving holes
> now, you won't get back to one extent.

Point taken.

> How hard would it be to boot a kernel from a year ago, with your current
> vmware, and see how that goes - it might be an interesting test.

Indeed. Unfortunately, since this is my office desktop PC, I'm quite
reluctant to fiddle around with it too much. But perhaps I find an
older live CD to try this with.

> I'd try to sort out the 2 moving parts you have, vmware & kernel.  See
> which one seems to have affected this behavior the most; downgrade one
> of the 2 pieces, and see how it behaves.

Sounds perfectly reasonable.

So I understand that there are no known issues with XFS that might
explain this long term fragmentation behaviour, and that there are no
magic XFS options to improve it.

I'll see what tests I can do and report back about the findings.

Thank you & best regards


Carsten Oberscheid

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Carsten Oberscheid]

On Tue, Jan 27, 2009 at 08:10:23AM +0100, Carsten Oberscheid wrote:
> 
> I'll see what tests I can do and report back about the findings.

Just booted an Ubuntu live CD from October 2007 and mounted the
filesystem in question. Could not run vmware from there easily, so I
tried just a copy of the vmem file:


root@ubuntu# uname -a
Linux tangchai 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:06 UTC 2008 x86_64 GNU/Linux

root@ubuntu# xfs_bmap -vvp foo.vmem | grep hole | wc -l
34
root@ubuntu# xfs_bmap -vvp foo.vmem | grep -v hole | wc -l
38

root@ubuntu# cp foo.vmem test

root@ubuntu# xfs_bmap -vvp test | grep hole | wc -l
27078
root@ubuntu# xfs_bmap -vvp test | grep -v hole | wc -l
27081


So a simple copy of a hardly fragmented vmem file gets very badly
fragmented. If we assume the vmem file fragmentation to be caused by
vmware writing this file inefficiently, does this mean that cp is even
worse?

For comparison, I created a new clean dummy file:


root@ubuntu# dd if=/dev/zero of=ztest bs=1000 count=500000
500000+0 records in
500000+0 records out
500000000 bytes (500 MB) copied, 6.52903 seconds, 76.6 MB/s

root@ubuntu# xfs_bmap -vvp ztest | grep hole | wc -l 
0

root@ubuntu# xfs_bmap -vvp ztest | grep -v hole | wc -l 
14

root@ubuntu# cp ztest ztest2

root@ubuntu# xfs_bmap -vvp ztest2 | grep hole | wc -l 
0

root@ubuntu# xfs_bmap -vvp ztest2 | grep -v hole | wc -l 
3


No problem here. I repeated all this after rebooting my current
kernel, with the same results. Copying the vmem file to an etx3
filesystem gives about 1,700 extents, which is also bad, but not as
bad as on the XFS disk.

While this test says nothing about the interaction of old/new kernel
and old/new VMware, for me it raises some questions about
file-specific properties affecting fragmentation which appear to be
independent of recent kernel changes. Please bear with me if I miss
something obvious, I'm just a user.

Regards


Carsten Oberscheid





-- 
carsten oberscheid                  d  o  c  t  r  o  n  i  c
email oberscheid@doctronic.de       information publishing + retrieval
phone +49 228 92 682 00             http://www.doctronic.de

doctronic GmbH & Co. KG, Bonn
Handelsregister: HRA 4685 (AG Bonn); USt-ID: DE 210 898 186
Komplementärin:  doctronic Verwaltungsges. mbH, Bonn, HRB 8926 (AG Bonn)
Geschäftsführer: Holger Flörke, Ingo Küper, Carsten Oberscheid

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Michael Monnerie]

[-- Attachment #1.1: Type: text/plain, Size: 559 bytes --]

On Dienstag 27 Januar 2009 Carsten Oberscheid wrote:
> root@ubuntu# cp foo.vmem test

What if you dd if=foo.vmem of=test bs=1000 ? That would do nearly the 
same as your "create empty file" test.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4


[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Eric Sandeen]

Carsten Oberscheid wrote:
> On Tue, Jan 27, 2009 at 08:10:23AM +0100, Carsten Oberscheid wrote:
>> I'll see what tests I can do and report back about the findings.
> 
> Just booted an Ubuntu live CD from October 2007 and mounted the
> filesystem in question. Could not run vmware from there easily, so I
> tried just a copy of the vmem file:

It'd be best to run vmware under some other kernel, and observe its
behavior, not just mount some existing filesystem and look at existing
files and do other non-vmware-related tests.

> 
> root@ubuntu# uname -a
> Linux tangchai 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:06 UTC 2008 x86_64 GNU/Linux
> 
> root@ubuntu# xfs_bmap -vvp foo.vmem | grep hole | wc -l
> 34
> root@ubuntu# xfs_bmap -vvp foo.vmem | grep -v hole | wc -l
> 38
> 
> root@ubuntu# cp foo.vmem test
> 
> root@ubuntu# xfs_bmap -vvp test | grep hole | wc -l
> 27078
> root@ubuntu# xfs_bmap -vvp test | grep -v hole | wc -l
> 27081

You went from a file with 34 holes to one with 27k holes by copying it?
 Perhaps this is cp's sparse file detection in action, seeking over
swaths of zeros.
> 
> So a simple copy of a hardly fragmented vmem file gets very badly
> fragmented. If we assume the vmem file fragmentation to be caused by
> vmware writing this file inefficiently, does this mean that cp is even
> worse?

Perhaps, if by "worse" you mean "leaves holes for regions with zeros".
Try cp --sparse=never and see how that goes.

> For comparison, I created a new clean dummy file:
> 
> 
> root@ubuntu# dd if=/dev/zero of=ztest bs=1000 count=500000
> 500000+0 records in
> 500000+0 records out
> 500000000 bytes (500 MB) copied, 6.52903 seconds, 76.6 MB/s
> 
> root@ubuntu# xfs_bmap -vvp ztest | grep hole | wc -l 
> 0

of course, I'd hope you have no holes here ;)

> root@ubuntu# xfs_bmap -vvp ztest | grep -v hole | wc -l 
> 14
> 
> root@ubuntu# cp ztest ztest2
> 
> root@ubuntu# xfs_bmap -vvp ztest2 | grep hole | wc -l 
> 0
> 
> root@ubuntu# xfs_bmap -vvp ztest2 | grep -v hole | wc -l 
> 3
> 
> 
> No problem here. I repeated all this after rebooting my current
> kernel, with the same results. Copying the vmem file to an etx3
> filesystem gives about 1,700 extents, which is also bad, but not as
> bad as on the XFS disk.
> 
> While this test says nothing about the interaction of old/new kernel
> and old/new VMware, for me it raises some questions about
> file-specific properties affecting fragmentation which appear to be
> independent of recent kernel changes. Please bear with me if I miss
> something obvious, I'm just a user.

My best guess is that your cp test is making the file even more sparse
by detecting blocks full of zeros and seeking over them, leaving more
holes.  Not really related to vmware behavior, though.

-Eric

> Regards
> 
> 
> Carsten Oberscheid

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Carsten Oberscheid]

On Tue, Jan 27, 2009 at 07:30:32AM -0600, Eric Sandeen wrote:
> It'd be best to run vmware under some other kernel, and observe its
> behavior, not just mount some existing filesystem and look at existing
> files and do other non-vmware-related tests.

If this really is just a vmware and/or kernel problem that has nothing
to do with the filesystem, then I agree.

> You went from a file with 34 holes to one with 27k holes by copying it?

Yep.

>  Perhaps this is cp's sparse file detection in action, seeking over
> swaths of zeros.
...
> Perhaps, if by "worse" you mean "leaves holes for regions with zeros".
> Try cp --sparse=never and see how that goes.

Didn't know this one.


[co@tangchai]~/vmware/foo cp --sparse=never foo.vmem test_nosparse

[co@tangchai]~/vmware/foo xfs_bmap -vvp test_ | grep hole | wc -l
test_livecd    test_nosparse  

[co@tangchai]~/vmware/foo xfs_bmap -vvp test_nosparse | grep hole | wc -l
0

[co@tangchai]~/vmware/foo xfs_bmap -vvp test_nosparse | grep -v hole | wc -l
9


You win.

> My best guess is that your cp test is making the file even more sparse
> by detecting blocks full of zeros and seeking over them, leaving more
> holes.  Not really related to vmware behavior, though.

All right. So next I'll try and downgrade vmplayer.

Just out of couriosity (and stubbornness): Are there any XFS
parameters that might influence fragmentation for the better, in case
I have to put up with a stupid application?

Thanks for your time & thoughts & best regards


Carsten Oberscheid

-- 
carsten oberscheid                  d  o  c  t  r  o  n  i  c
email oberscheid@doctronic.de       information publishing + retrieval
phone +49 228 92 682 00             http://www.doctronic.de

doctronic GmbH & Co. KG, Bonn
Handelsregister: HRA 4685 (AG Bonn); USt-ID: DE 210 898 186
Komplementärin:  doctronic Verwaltungsges. mbH, Bonn, HRB 8926 (AG Bonn)
Geschäftsführer: Holger Flörke, Ingo Küper, Carsten Oberscheid

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Carsten Oberscheid]

On Tue, Jan 27, 2009 at 10:30:38AM +0100, Michael Monnerie wrote:
> What if you dd if=foo.vmem of=test bs=1000 ? That would do nearly the 
> same as your "create empty file" test.

[co@tangchai]~/vmware/foo dd if=foo.vmem of=test_dd bs=1000
536870+1 Datensätze ein
536870+1 Datensätze aus
536870912 Bytes (537 MB) kopiert, 3,76026 s, 143 MB/s

[co@tangchai]~/vmware/foo xfs_bmap -vvp test_dd | grep hole | wc -l0

[co@tangchai]~/vmware/foo xfs_bmap -vvp test_dd | grep -v hole | wc -l 
10


Looks good. Seems like I'm ready to accept that XFS is innocent. See
other post.

Thank you & best regards


Carsten Oberscheid

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Felix Blyakher]

On Jan 27, 2009, at 8:37 AM, Carsten Oberscheid wrote:

> Just out of couriosity (and stubbornness): Are there any XFS
> parameters that might influence fragmentation for the better, in case
> I have to put up with a stupid application?

Not much could be tuned in XFS for random writes over the holes.
Ideally such file should be preallocated. Can you provide
vmware the zero filled file by doing

dd if=/dev/zero of=foo.mem count=512 bs=1024k

instead of letting vmware populate it over the time and thus
badly fragmenting it.

Felix

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Eric Sandeen]

Carsten Oberscheid wrote:
> On Tue, Jan 27, 2009 at 07:30:32AM -0600, Eric Sandeen wrote:
>> It'd be best to run vmware under some other kernel, and observe its
>> behavior, not just mount some existing filesystem and look at existing
>> files and do other non-vmware-related tests.
> 
> If this really is just a vmware and/or kernel problem that has nothing
> to do with the filesystem, then I agree.

well when I say "kernel" I include the filesystem in that kernel.  :)

>> You went from a file with 34 holes to one with 27k holes by copying it?
> 
> Yep.
> 
>>  Perhaps this is cp's sparse file detection in action, seeking over
>> swaths of zeros.
> ...
>> Perhaps, if by "worse" you mean "leaves holes for regions with zeros".
>> Try cp --sparse=never and see how that goes.
> 
> Didn't know this one.
> 
> 
> [co@tangchai]~/vmware/foo cp --sparse=never foo.vmem test_nosparse
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp test_ | grep hole | wc -l
> test_livecd    test_nosparse  
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp test_nosparse | grep hole | wc -l
> 0
> 
> [co@tangchai]~/vmware/foo xfs_bmap -vvp test_nosparse | grep -v hole | wc -l
> 9
> 
> 
> You win.

\o/  :)

>> My best guess is that your cp test is making the file even more sparse
>> by detecting blocks full of zeros and seeking over them, leaving more
>> holes.  Not really related to vmware behavior, though.
> 
> All right. So next I'll try and downgrade vmplayer.
> 
> Just out of couriosity (and stubbornness): Are there any XFS
> parameters that might influence fragmentation for the better, in case
> I have to put up with a stupid application?
> 
> Thanks for your time & thoughts & best regards

There is an -o allocsize=<number> which controls how much is
speculatively allocated off the end of a file; in some cases it could
help but I'm not sure it would in this case.  As Dave said a while ago,
it's really an issue with how vmware is writing the files out.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange fragmentation in nearly empty filesystem

[Felix Blyakher]

On Jan 27, 2009, at 11:26 AM, Eric Sandeen wrote:

> There is an -o allocsize=<number> which controls how much is
> speculatively allocated off the end of a file; in some cases it could
> help but I'm not sure it would in this case.

"the end of a file" is the key for this option.
Wouldn't help in case of writing into the sparse file.

Felix

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs