Defragging XFS File Systems

Defragging XFS File Systems

[Kenneth Emerson]

[-- Attachment #1.1: Type: text/plain, Size: 991 bytes --]

I hadn't given much thought to fragmentation of my TV recordings volume
(XFS) until reading through some MythTV-users threads recently that
mentioned how fragmented an XFS file system could become.  After running
xfs_db, I found out that my fs appeared to be quite bad:

$ sudo xfs_db -c frag -r /dev/mapper/appl_vg-appl_lv
 actual 1138668, ideal 11023, fragmentation factor 99.03%

I then ran xfs_fsr with all defaults (ran for two hours) and then re-ran
xfs_db and got the following results:

$ sudo xfs_db -c frag -r /dev/mapper/appl_vg-appl_lv
invalid numrecs (27111) in bmapbtd block
invalid numrecs (4716) in bmapbtd block
invalid numrecs (58978) in bmapbtd block
actual 1034793, ideal 11024, fragmentation factor 98.93%

The fragmentation level was reduced, but I was concerned about the error
messages.  Before I go any further, am I corrupting my file system with the
defragging or are these "invalid numrecs" messages unimportant?

Google didn't offer much help.

Regards,

Ken E.

[-- Attachment #1.2: Type: text/html, Size: 1282 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Stewart Smith]

On Mon, 6 Jun 2011 22:52:48 -0500, Kenneth Emerson <kenneth.emerson@gmail.com> wrote:
> I hadn't given much thought to fragmentation of my TV recordings volume
> (XFS) until reading through some MythTV-users threads recently that
> mentioned how fragmented an XFS file system could become.  After running
> xfs_db, I found out that my fs appeared to be quite bad:

MythTV can end up with fragmentation on XFS due to an fsync() call that
attempts to work raound limitations in ext3.

Workarounds include:
- allocsize mount parameter
- patch mythtv source not to fsync (you could easily write a patch that
only did fsync if not xfs... I've been meaning to do this for
years... not enough hours in day).
- run mythbackend with libeatmydata, thus disabling the fsync

-- 
Stewart Smith

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Stan Hoeppner]

On 6/6/2011 10:52 PM, Kenneth Emerson wrote:
> I hadn't given much thought to fragmentation of my TV recordings volume
> (XFS) until reading through some MythTV-users threads recently that
> mentioned how fragmented an XFS file system could become.  After running
> xfs_db, I found out that my fs appeared to be quite bad:
> 
> $ sudo xfs_db -c frag -r /dev/mapper/appl_vg-appl_lv
>  actual 1138668, ideal 11023, fragmentation factor 99.03%
> 
> I then ran xfs_fsr with all defaults (ran for two hours) and then re-ran

>From man xfs_fsr:

       It runs for up to two hours after which it records the
       filesystem where it left off, so it can start there the next
       time.  This  information  is stored in the file
       /var/tmp/.fsrlast_xfs.  If the information found here is somehow
       inconsistent or out of date it is ignored and reorganization
       starts at the beginning of the first filesystem found in
       /etc/mtab.

If xfs_fsr stopped at 2 hours, multiple additional runs will likely be
required to get good defragmentation.

> xfs_db and got the following results:
> 
> $ sudo xfs_db -c frag -r /dev/mapper/appl_vg-appl_lv

The -r above suggests you created a large realtime section for your
MythTV storage.  It may be helpful for you to provide xfs_info output
for the heavily fragmented filesystem.

> invalid numrecs (27111) in bmapbtd block
> invalid numrecs (4716) in bmapbtd block
> invalid numrecs (58978) in bmapbtd block

I'll leave these errors for one of the devs to tackle.

> actual 1034793, ideal 11024, fragmentation factor 98.93%
> 
> The fragmentation level was reduced, 

It was likely reduced much more than this.  Dropping caches or
unmounting and remounting the filesystem is often necessary after
running xfs_fsr in order to show the actual fragmentation level.  Try:

# echo 3 > /proc/sys/vm/drop_caches

and then run xfs_db again.

> but I was concerned about the error
> messages.  Before I go any further, am I corrupting my file system with the
> defragging or are these "invalid numrecs" messages unimportant?

Run 'xfs_check' or 'xfs_repair -n' and post the results.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Stan Hoeppner]

On 6/8/2011 2:55 PM, Kenneth Emerson wrote:
> On Tue, Jun 7, 2011 at 7:05 AM, Stan Hoeppner <stan@hardwarefreak.com>wrote:
> 
>> On 6/6/2011 10:52 PM, Kenneth Emerson wrote:
>>> I hadn't given much thought to fragmentation of my TV recordings volume
>>> (XFS) until reading through some MythTV-users threads recently that
>>> mentioned how fragmented an XFS file system could become.  After running
>>> xfs_db, I found out that my fs appeared to be quite bad:
>>>
>>> $ sudo xfs_db -c frag -r /dev/mapper/appl_vg-appl_lv
>>>  actual 1138668, ideal 11023, fragmentation factor 99.03%
>>>
>>> I then ran xfs_fsr with all defaults (ran for two hours) and then re-ran
>>
>> From man xfs_fsr:
>>
>>       It runs for up to two hours after which it records the
>>       filesystem where it left off, so it can start there the next
>>       time.  This  information  is stored in the file
>>       /var/tmp/.fsrlast_xfs.  If the information found here is somehow
>>       inconsistent or out of date it is ignored and reorganization
>>       starts at the beginning of the first filesystem found in
>>       /etc/mtab.
>>
>> If xfs_fsr stopped at 2 hours, multiple additional runs will likely be
>> required to get good defragmentation.
>>
> 
> After some more research, I don't think I would ever be able to defrag this
> volume.  It is too badly fragmented and too full.

Then you have a couple of options:

1.  Delete files to free up space for xfs_fsr to work properly
2.  Dump, delete, re-create, and restore the filesystem

The first will likely leave you with fragmentation.  The second will
eliminate all fragmentation.

>>> xfs_db and got the following results:
>>>
>>> $ sudo xfs_db -c frag -r /dev/mapper/appl_vg-appl_lv
>>
>> The -r above suggests you created a large realtime section for your
>> MythTV storage.  It may be helpful for you to provide xfs_info output
>> for the heavily fragmented filesystem.

> I thought the -r was just for read only so that I didn't have to un-mount
> it before running the report.

According to man xfs_db, '-r' immediately following the frag command:
...
	-r  enables processing of realtime file data

>>> invalid numrecs (27111) in bmapbtd block
>>> invalid numrecs (4716) in bmapbtd block
>>> invalid numrecs (58978) in bmapbtd block
>>>
>>> I'll leave these errors for one of the devs to tackle.
>
> These errors 'disappeared' when I ran xfs_db again later.

Maybe a cache consistency issue.

>>> actual 1034793, ideal 11024, fragmentation factor 98.93%
>>>
>>> The fragmentation level was reduced,
>>
>> It was likely reduced much more than this.  Dropping caches or
>> unmounting and remounting the filesystem is often necessary after
>> running xfs_fsr in order to show the actual fragmentation level.  Try:
>>
>> # echo 3 > /proc/sys/vm/drop_caches
>>
>> and then run xfs_db again.

> I did this, but the numbers did not change much.  Many of the files are >
> 4GiB and have over a hundred extents.  When I tried to defrag a single file,
> it reported that it wan't possible.

Looks like a dump/remake/restore is in your future.

>>> but I was concerned about the error
>>> messages.  Before I go any further, am I corrupting my file system with
>> the
>>> defragging or are these "invalid numrecs" messages unimportant?
>>
>> Run 'xfs_check' or 'xfs_repair -n' and post the results.

> Not necessary since the errors are now gone.  I have ordered a 3TB disk
> that I can connect via e-sata and copy the entire volume off of the RAID
> set.  I will then reformat and put the files back.  After that, I will make
> a cron job to run on a regular basis to keep the volume from getting so
> fragmented again.

Bang on that eSATA drive/interface/driver thoroughly before relying on
it for your intended purpose.  Look in dmesg regularly for errors while
burning it in.  Be mindful of partition alignment if he 3TB drive is an
"Advanced Format" hybrid 512/4096 byte drive.

> Thanks for your input.

You're welcome.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 781 bytes --]

On Mittwoch, 8. Juni 2011 Kenneth wrote:
> > After that, I will make
> > a cron job to run on a regular basis to keep the volume from
> > getting so fragmented again.

How will you do that? Running xfs_fsr on a regular basis is *not* 
recommended, as it will age your filesystem and defrag free space. In 
the beginning, and as long as your filesystem is maybe 50% full it will 
be fine, but there's a point where it starts to get worse - much worse. 
Just run a recent kernel, it has all the naughty stuff to help keep the 
performance high.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// Haus zu verkaufen: http://zmi.at/langegg/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Stan Hoeppner]

On 6/9/2011 12:49 AM, Michael Monnerie wrote:
> On Mittwoch, 8. Juni 2011 Kenneth wrote:
>>> After that, I will make
>>> a cron job to run on a regular basis to keep the volume from
>>> getting so fragmented again.
> 
> How will you do that? Running xfs_fsr on a regular basis is *not* 
> recommended, as it will age your filesystem and defrag free space. In 
> the beginning, and as long as your filesystem is maybe 50% full it will 
> be fine, but there's a point where it starts to get worse - much worse. 
> Just run a recent kernel, it has all the naughty stuff to help keep the 
> performance high.

When *is* running xfs_fsr recommended?

I scheduled it twice a week some time ago due to a filesystem containing
active mbox files.  I did so because they became so heavily fragmented
in short order, especially those swallowing copious amounts of list
mail.  Before cron'ing xfs_fsr I was seeing mbox files with over 1000
fragmented extents, and increasing MUA latency as the files became more
fragmented.  The filesystem is currently 90% free.

-- 
Stan


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1432 bytes --]

On Donnerstag, 9. Juni 2011 Stan Hoeppner wrote:
> When is running xfs_fsr recommended?

Good question. One case that comes to my mind is a filesystem that was 
used a long time when filled >85%, which has now either been expanded or 
files removed so you have a lot of space again, and you want to defrag 
all those files that have been badly fragmented.
 
> I scheduled it twice a week some time ago due to a filesystem
> containing active mbox files.  I did so because they became so
> heavily fragmented in short order, especially those swallowing
> copious amounts of list mail.  Before cron'ing xfs_fsr I was seeing
> mbox files with over 1000 fragmented extents, and increasing MUA
> latency as the files became more fragmented.  The filesystem is
> currently 90% free.

This is also an example where defrag may help. You have 10% usage, so 
there's enough space. Maybe your usage fits the mount option 
"allocsize", so that you keep room for file append. But newer kernels 
changed behaviour of XFS, so I'm not sure up to which kernel version its 
good. Maybe Dave Chinner can chime in here. BTW, maybe this behaviour 
should be documented in the FAQ so we can reference it.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// Haus zu verkaufen: http://zmi.at/langegg/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Stan Hoeppner]

On 6/9/2011 5:12 AM, Michael Monnerie wrote:
> On Donnerstag, 9. Juni 2011 Stan Hoeppner wrote:
>> When is running xfs_fsr recommended?
> 
> Good question. One case that comes to my mind is a filesystem that was 
> used a long time when filled >85%, which has now either been expanded or 
> files removed so you have a lot of space again, and you want to defrag 
> all those files that have been badly fragmented.
>  
>> I scheduled it twice a week some time ago due to a filesystem
>> containing active mbox files.  I did so because they became so
>> heavily fragmented in short order, especially those swallowing
>> copious amounts of list mail.  Before cron'ing xfs_fsr I was seeing
>> mbox files with over 1000 fragmented extents, and increasing MUA
>> latency as the files became more fragmented.  The filesystem is
>> currently 90% free.
> 
> This is also an example where defrag may help. You have 10% usage, so 
> there's enough space. Maybe your usage fits the mount option 
> "allocsize", 

I tried allocsize=1m but it didn't seem to help already existing files.
 I simply don't think there's much that can be done in filesystem logic
to keep long lived constantly appended files from fragmenting, short of
the log rotation method, which isn't really suitable for large mbox
files.  Maildir is obviously an option to combat fragmentation, but it
has downsides as well, mainly much higher small file IOPS during normal
operation.  Using mbox I can 'hide the IO' by running xfs_fsr at night
and/or on weekends.

> so that you keep room for file append. But newer kernels 
> changed behaviour of XFS, so I'm not sure up to which kernel version its 
> good. Maybe Dave Chinner can chime in here. BTW, maybe this behaviour 
> should be documented in the FAQ so we can reference it.

My current method seems to work well for now.  This filesystem will
likely not even be half full by the time this server is replaced in a
few years, so I probably don't need to worry about xfs_fsr efficiency
WRT free space.  Due to the low cost of today's gargantuan platters, I'm
guessing there are likely many admins in a similar position.  Four 600GB
SAS drives in RAID10 yields ~1.2TB of mail storage.  That's a lot of
space, and performance, for small/medium organization's mailboxes...

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Dave Chinner]

On Thu, Jun 09, 2011 at 01:24:31PM -0500, Stan Hoeppner wrote:
> On 6/9/2011 5:12 AM, Michael Monnerie wrote:
> > On Donnerstag, 9. Juni 2011 Stan Hoeppner wrote:
> >> When is running xfs_fsr recommended?
> > 
> > Good question. One case that comes to my mind is a filesystem that was 
> > used a long time when filled >85%, which has now either been expanded or 
> > files removed so you have a lot of space again, and you want to defrag 
> > all those files that have been badly fragmented.
> >  
> >> I scheduled it twice a week some time ago due to a filesystem
> >> containing active mbox files.  I did so because they became so
> >> heavily fragmented in short order, especially those swallowing
> >> copious amounts of list mail.  Before cron'ing xfs_fsr I was seeing
> >> mbox files with over 1000 fragmented extents, and increasing MUA
> >> latency as the files became more fragmented.  The filesystem is
> >> currently 90% free.
> > 
> > This is also an example where defrag may help. You have 10% usage, so 
> > there's enough space. Maybe your usage fits the mount option 
> > "allocsize", 
> 
> I tried allocsize=1m but it didn't seem to help already existing files.
>  I simply don't think there's much that can be done in filesystem logic
> to keep long lived constantly appended files from fragmenting, short of

YOu can stop XFS from truncating speculative preallocation beyond
EOF by either telling the inode it has preallocated space or
or turning it into a an append-only file. e.g.

$ xfs_io -f -c "resvsp 0 4k" <file>

or

$ sudo chattr +a <file>

Either way, XFS won't truncate extents beyond EOF on file close for
such a file and that should prevent most future fragmentation of the
file.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Defragging XFS File Systems

[Stan Hoeppner]

On 6/10/2011 12:56 PM, Dave Chinner wrote:
> On Thu, Jun 09, 2011 at 01:24:31PM -0500, Stan Hoeppner wrote:
>> On 6/9/2011 5:12 AM, Michael Monnerie wrote:
>>> On Donnerstag, 9. Juni 2011 Stan Hoeppner wrote:
>>>> When is running xfs_fsr recommended?
>>>
>>> Good question. One case that comes to my mind is a filesystem that was 
>>> used a long time when filled >85%, which has now either been expanded or 
>>> files removed so you have a lot of space again, and you want to defrag 
>>> all those files that have been badly fragmented.
>>>  
>>>> I scheduled it twice a week some time ago due to a filesystem
>>>> containing active mbox files.  I did so because they became so
>>>> heavily fragmented in short order, especially those swallowing
>>>> copious amounts of list mail.  Before cron'ing xfs_fsr I was seeing
>>>> mbox files with over 1000 fragmented extents, and increasing MUA
>>>> latency as the files became more fragmented.  The filesystem is
>>>> currently 90% free.
>>>
>>> This is also an example where defrag may help. You have 10% usage, so 
>>> there's enough space. Maybe your usage fits the mount option 
>>> "allocsize", 
>>
>> I tried allocsize=1m but it didn't seem to help already existing files.
>>  I simply don't think there's much that can be done in filesystem logic
>> to keep long lived constantly appended files from fragmenting, short of
> 
> YOu can stop XFS from truncating speculative preallocation beyond
> EOF by either telling the inode it has preallocated space or
> or turning it into a an append-only file. e.g.
> 
> $ xfs_io -f -c "resvsp 0 4k" <file>
> 
> or
> 
> $ sudo chattr +a <file>
> 
> Either way, XFS won't truncate extents beyond EOF on file close for
> such a file and that should prevent most future fragmentation of the
> file.

Given the way in which individual mails are deleted from an mbox
file[1], how would changing the file to append-only affect a user's
ability to delete mail from that mbox?  Is this something I could use
generically on all mbox files, or is this only applicable to files for
which there is some certainty that they'll only be appended in the future?

[1] http://www.linuxmail.info/mbox-maildir-mail-storage-formats/
1.  Lock the mailbox.
2.  Move the contents of the mailbox, beginning from the position right
    after the mail to be deleted until the end of the mailbox, into the
    position of the mail to be deleted.
3.  Reduce the size of the mailbox file by the size of the deleted mail.
4.  Unlock the mailbox.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs