2 question about XFS fragmentation and _fsr

2 question about XFS fragmentation and _fsr

[Janos Haar]

Hello list,

First, please CC me because i am not on the list.

Question 1:
I am working in a data recovery company, and we are using one 3TB RAID 
storage with XFS to store the recovery cases in image files.
For spare a lot of space, usually we are imaging only the used spaces of the 
filesystems. (for example 1TB system drive wich have only 80GB data inside 
but needs to be bootable, the image is similar like Ghost images)
But this makes a lot of sparse files, wich right should be this way.
Some cases are done and we are closed (image can be deleted), and some needs 
to store for long time.
In the result, actually we have >6TB images on the 3TB disk, wich is 97.9% 
fragmented.
Basically the sparse RAW disk images should be more faster accessible than 
the original drive, because this is 4disk raid, instead of one, AND the head 
don't need to travel through the empty space of the drive...
But actually we are proud if we can read or write in 8-10MB/s! :-D
I have started to read about xfs_fsr but there is almost none information, 
how it is _really_ working, and i am doubt about this will try to fill up 
all of my images, and therefore fail to do any improvement, or this will do 
the right job and will re-organize the sparse files again.
The XFS_FSR can be good for me or not?
If not, i suggest to implement an option to do sparse fles on the result or 
not in the next releases...
I will say a huge thanks in the end. :-)

Question 2:
One of our customers have one storage wich is exactly the same like the one 
wich is described on the Q1, but only used for samba storage for storing 
media files (big files.)
I am sure, there is no torrent or similar, and i have told to the customers 
on the beginning "don't write more files parallel, to avoid fragmentation", 
but today the storage is >95% fragmented.
The customer sad, he only does file write one by one, and nothing more.
How can this be?

Thanks for your attention,

Best regards,
Janos Haar 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr

[Stan Hoeppner]

Janos Haar put forth on 4/11/2011 12:39 PM:

> In the result, actually we have >6TB images on the 3TB disk, wich is
> 97.9% fragmented.

How much free space does the filesystem have?  How big is each image
file?  For xfs_fsr to work properly it must have sufficient free space
in the filesystem.

> Basically the sparse RAW disk images should be more faster accessible
> than the original drive, because this is 4disk raid, instead of one, AND
> the head don't need to travel through the empty space of the drive...

It sounds like you may have some other issues besides filesystem
fragmentation.

> The XFS_FSR can be good for me or not?

If you have plenty of free space.

> Question 2:
> One of our customers have one storage wich is exactly the same like the
> one wich is described on the Q1, but only used for samba storage for
> storing media files (big files.)

Writing large files sequentially shouldn't cause fragmentation.

> I am sure, there is no torrent or similar, and i have told to the
> customers on the beginning "don't write more files parallel, to avoid
> fragmentation", but today the storage is >95% fragmented.
> The customer sad, he only does file write one by one, and nothing more.
> How can this be?

What were the mkfs.xfs arguments you used when creating these
filesystem?  Please share xfs_info output for the filesystems in
question, and details of the underlying hardware RAID storage.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr: SPLITTED Q2:normal files, samba share

[Janos Haar]

Note: I have splitted my previous letter for easyer discusion...
This have only Question-2 the storage with only normal files and samba 
share.

----- Original Message ----- 
From: "Stan Hoeppner" <stan@hardwarefreak.com>
To: "Janos Haar" <janos.haar@netcenter.hu>
Cc: <linux-xfs@oss.sgi.com>
Sent: Monday, April 11, 2011 10:37 PM
Subject: Re: 2 question about XFS fragmentation and _fsr


> Janos Haar put forth on 4/11/2011 12:39 PM:
>> Question 2:
>> One of our customers have one storage wich is exactly the same like the
>> one wich is described on the Q1, but only used for samba storage for
>> storing media files (big files.)
>
> Writing large files sequentially shouldn't cause fragmentation.
>
>> I am sure, there is no torrent or similar, and i have told to the
>> customers on the beginning "don't write more files parallel, to avoid
>> fragmentation", but today the storage is >95% fragmented.
>> The customer sad, he only does file write one by one, and nothing more.
>> How can this be?
>
> What were the mkfs.xfs arguments you used when creating these
> filesystem?  Please share xfs_info output for the filesystems in
> question, and details of the underlying hardware RAID storage.

This was the default.

 xfs_info /dev/loop0
meta-data=/dev/loop0             isize=256    agcount=32, agsize=114269833 
blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=3656634656, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

>
> -- 
> Stan

Thanks,
Janos 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr: SPLITTED Q1:sparese files

[Janos Haar]

Note: I have splitted my previous letter for easyer discusion...
This have only Question-1 the storage with only sparse files.

----- Original Message ----- 
From: "Stan Hoeppner" <stan@hardwarefreak.com>
To: "Janos Haar" <janos.haar@netcenter.hu>
Cc: <linux-xfs@oss.sgi.com>
Sent: Monday, April 11, 2011 10:37 PM
Subject: Re: 2 question about XFS fragmentation and _fsr


> Janos Haar put forth on 4/11/2011 12:39 PM:
>
>> In the result, actually we have >6TB images on the 3TB disk, wich is
>> 97.9% fragmented.
>
> How much free space does the filesystem have?  How big is each image
> file?  For xfs_fsr to work properly it must have sufficient free space
> in the filesystem.

Actually less than 1%.
I know, this is too less for xfs_fsr, but i am only thinking actually to try 
because i am unsure this is useful for me.

>
>> Basically the sparse RAW disk images should be more faster accessible
>> than the original drive, because this is 4disk raid, instead of one, AND
>> the head don't need to travel through the empty space of the drive...
>
> It sounds like you may have some other issues besides filesystem
> fragmentation.
>
>> The XFS_FSR can be good for me or not?
>
> If you have plenty of free space.
>
> -- 
> Stan

Thanks,
Janos 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr

[Dave Chinner]

On Mon, Apr 11, 2011 at 07:39:07PM +0200, Janos Haar wrote:
> Hello list,
> 
> First, please CC me because i am not on the list.
> 
> Question 1:
> I am working in a data recovery company, and we are using one 3TB
> RAID storage with XFS to store the recovery cases in image files.
> For spare a lot of space, usually we are imaging only the used
> spaces of the filesystems. (for example 1TB system drive wich have
> only 80GB data inside but needs to be bootable, the image is similar
> like Ghost images)
> But this makes a lot of sparse files, wich right should be this way.
> Some cases are done and we are closed (image can be deleted), and
> some needs to store for long time.
> In the result, actually we have >6TB images on the 3TB disk, wich is
> 97.9% fragmented.

How are you determining that figure?

> Basically the sparse RAW disk images should be more faster
> accessible than the original drive, because this is 4disk raid,
> instead of one, AND the head don't need to travel through the empty
> space of the drive...
> But actually we are proud if we can read or write in 8-10MB/s! :-D
> I have started to read about xfs_fsr but there is almost none
> information, how it is _really_ working, and i am doubt about this
> will try to fill up all of my images, and therefore fail to do any
> improvement, or this will do the right job and will re-organize the
> sparse files again.
> The XFS_FSR can be good for me or not?

xfs_fsr preserves the layout of sparse files, so it is unlikely to
help you at all here. Remember, sparse files only reduce the amount
of space used by a file - they don't magically reduce the number of
seeks needed to read data from the files...

> If not, i suggest to implement an option to do sparse fles on the
> result or not in the next releases...
> I will say a huge thanks in the end. :-)
> 
> Question 2:
> One of our customers have one storage wich is exactly the same like
> the one wich is described on the Q1, but only used for samba storage
> for storing media files (big files.)
> I am sure, there is no torrent or similar, and i have told to the
> customers on the beginning "don't write more files parallel, to
> avoid fragmentation", but today the storage is >95% fragmented.

Once again, what does 95% fragmented mean?

> The customer sad, he only does file write one by one, and nothing more.
> How can this be?

Depends on your samba configuration to how it does writes. If it is
conigured to do sync writes then it will have all sorts of
fragmentation problems. So without more details about the workload,
the kernel that is being used, the fragmentation occurring (xfs_bmap
is your friend) and the samba config, there's little that can be
suggested here.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr

[Christoph Hellwig]

On Tue, Apr 12, 2011 at 07:42:38AM +1000, Dave Chinner wrote:
> Depends on your samba configuration to how it does writes. If it is
> conigured to do sync writes then it will have all sorts of
> fragmentation problems. So without more details about the workload,
> the kernel that is being used, the fragmentation occurring (xfs_bmap
> is your friend) and the samba config, there's little that can be
> suggested here.

Very few people configure Samba to do sync writes, but even without
that I see files written by Samba heavily fragmented when the systems
operates under intense memory pressure.  The CIFS protocol has very
small write sizes, and it seems like they do not always arrive in order.
If the system is under enough memory pressure to write them out already
before the previous fragment arrive it can cause very fragmented
extent maps.  Enabling the "strict allocate" option in newer Samba
version fixes that problem by doing fallocates in larger chunks, which
from my observation always happen in order.  The downside is that
current fallocate in XFS is not designed to be used in a fastpath,
and thus causes performance degradations during the actual write.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr: Q2

[Janos Haar]

----- Original Message ----- 
From: "Christoph Hellwig" <hch@infradead.org>
To: "Dave Chinner" <david@fromorbit.com>
Cc: "Janos Haar" <janos.haar@netcenter.hu>; <linux-xfs@oss.sgi.com>
Sent: Tuesday, April 12, 2011 1:37 AM
Subject: Re: 2 question about XFS fragmentation and _fsr


> On Tue, Apr 12, 2011 at 07:42:38AM +1000, Dave Chinner wrote:
>> Depends on your samba configuration to how it does writes. If it is
>> conigured to do sync writes then it will have all sorts of
>> fragmentation problems. So without more details about the workload,
>> the kernel that is being used, the fragmentation occurring (xfs_bmap
>> is your friend) and the samba config, there's little that can be
>> suggested here.
>
> Very few people configure Samba to do sync writes, but even without
> that I see files written by Samba heavily fragmented when the systems
> operates under intense memory pressure.  The CIFS protocol has very
> small write sizes, and it seems like they do not always arrive in order.
> If the system is under enough memory pressure to write them out already
> before the previous fragment arrive it can cause very fragmented
> extent maps.  Enabling the "strict allocate" option in newer Samba
> version fixes that problem by doing fallocates in larger chunks, which
> from my observation always happen in order.  The downside is that
> current fallocate in XFS is not designed to be used in a fastpath,
> and thus causes performance degradations during the actual write.

Is there anything what i can do in proc/sys/vm configuration to solve the 
fragment write problem?

Thanks,
Janos 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr: SPLITTED Q1:sparse files

[Janos Haar]

Note: I have splitted my previous letter for easyer discusion...
This have only Question-1 the storage with only sparse files.

----- Original Message ----- 
From: "Dave Chinner" <david@fromorbit.com>
To: "Janos Haar" <janos.haar@netcenter.hu>
Cc: <linux-xfs@oss.sgi.com>
Sent: Monday, April 11, 2011 11:42 PM
Subject: Re: 2 question about XFS fragmentation and _fsr


> On Mon, Apr 11, 2011 at 07:39:07PM +0200, Janos Haar wrote:
>> Hello list,
>> 
>> First, please CC me because i am not on the list.
>> 
>> Question 1:
>> I am working in a data recovery company, and we are using one 3TB
>> RAID storage with XFS to store the recovery cases in image files.
>> For spare a lot of space, usually we are imaging only the used
>> spaces of the filesystems. (for example 1TB system drive wich have
>> only 80GB data inside but needs to be bootable, the image is similar
>> like Ghost images)
>> But this makes a lot of sparse files, wich right should be this way.
>> Some cases are done and we are closed (image can be deleted), and
>> some needs to store for long time.
>> In the result, actually we have >6TB images on the 3TB disk, wich is
>> 97.9% fragmented.
> 
> How are you determining that figure?

[root@UNISTORE admin]# cat xfs_get_frat_ratio
echo loop0
xfs_db -c frag -r /dev/loop0
[root@UNISTORE admin]# ./xfs_get_frat_ratio
loop0
actual 7650952, ideal 752501, fragmentation factor 90.16%

> 
>> Basically the sparse RAW disk images should be more faster
>> accessible than the original drive, because this is 4disk raid,
>> instead of one, AND the head don't need to travel through the empty
>> space of the drive...
>> But actually we are proud if we can read or write in 8-10MB/s! :-D
>> I have started to read about xfs_fsr but there is almost none
>> information, how it is _really_ working, and i am doubt about this
>> will try to fill up all of my images, and therefore fail to do any
>> improvement, or this will do the right job and will re-organize the
>> sparse files again.
>> The XFS_FSR can be good for me or not?
> 
> xfs_fsr preserves the layout of sparse files, so it is unlikely to
> help you at all here. Remember, sparse files only reduce the amount
> of space used by a file - they don't magically reduce the number of
> seeks needed to read data from the files...

Yes of course, but heads needs move more closer from one point to another.
I think this should reduce the seeking time.
Or i have missed something? :)

> 
>> If not, i suggest to implement an option to do sparse fles on the
>> result or not in the next releases...
>> I will say a huge thanks in the end. :-)
>> 

> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Thanks,
Janos

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr: SPLITTED Q1:sparse files

[Eric Sandeen]

On 4/26/11 4:51 PM, Janos Haar wrote:

<snip>

>>> In the result, actually we have >6TB images on the 3TB disk, wich is
>>> 97.9% fragmented.
>>
>> How are you determining that figure?
> 
> [root@UNISTORE admin]# cat xfs_get_frat_ratio
> echo loop0
> xfs_db -c frag -r /dev/loop0
> [root@UNISTORE admin]# ./xfs_get_frat_ratio
> loop0
> actual 7650952, ideal 752501, fragmentation factor 90.16%

so you had 7650952 extents on the fs, ideally you'd have 752501, or so xfs_db says...

Another way of looking at this is that you have about 10 extents per file on average.

Depending on the size of the files, this may be perfectly fine.
Is, for example, a 10G file in ten 1G extents really a problem?

I doubt that fragmentation is your performance problem here.

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr: SPLITTED Q1:sparse files

[Eric Sandeen]

On 4/27/11 3:18 PM, Eric Sandeen wrote:
> On 4/26/11 4:51 PM, Janos Haar wrote:
> 
> <snip>
> 
>>>> In the result, actually we have >6TB images on the 3TB disk, wich is
>>>> 97.9% fragmented.
>>>
>>> How are you determining that figure?
>>
>> [root@UNISTORE admin]# cat xfs_get_frat_ratio
>> echo loop0
>> xfs_db -c frag -r /dev/loop0
>> [root@UNISTORE admin]# ./xfs_get_frat_ratio
>> loop0
>> actual 7650952, ideal 752501, fragmentation factor 90.16%

btw we should just nuke that stupid command ;)

> so you had 7650952 extents on the fs, ideally you'd have 752501, or so xfs_db says...
> 
> Another way of looking at this is that you have about 10 extents per file on average.
> 
> Depending on the size of the files, this may be perfectly fine.
> Is, for example, a 10G file in ten 1G extents really a problem?
> 
> I doubt that fragmentation is your performance problem here.

... note however that you can use xfs_bmap to examine the layout of any particular file.

10 on average may be 1 for most, and 10,000 for the last one.   :)

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2 question about XFS fragmentation and _fsr: SPLITTED Q2:normal files, samba share

[Janos Haar]

Note: I have splitted my previous letter for easyer discusion...
This have only Question-2 the storage with only normal files and samba
share.


----- Original Message ----- 
From: "Dave Chinner" <david@fromorbit.com>
To: "Janos Haar" <janos.haar@netcenter.hu>
Cc: <linux-xfs@oss.sgi.com>
Sent: Monday, April 11, 2011 11:42 PM
Subject: Re: 2 question about XFS fragmentation and _fsr


> On Mon, Apr 11, 2011 at 07:39:07PM +0200, Janos Haar wrote:
>> Hello list,

>> Question 2:
>> One of our customers have one storage wich is exactly the same like
>> the one wich is described on the Q1, but only used for samba storage
>> for storing media files (big files.)
>> I am sure, there is no torrent or similar, and i have told to the
>> customers on the beginning "don't write more files parallel, to
>> avoid fragmentation", but today the storage is >95% fragmented.
>
> Once again, what does 95% fragmented mean?

[root@Clarus-gl2k10-2 admin]# cat xfs_get_frag_ratio
echo loop0
xfs_db -c frag -r /dev/loop0
[root@Clarus-gl2k10-2 admin]# ./xfs_get_frag_ratio
loop0
actual 51179670, ideal 5914, fragmentation factor 99.99%

>
>> The customer sad, he only does file write one by one, and nothing more.
>> How can this be?
>
> Depends on your samba configuration to how it does writes. If it is
> conigured to do sync writes then it will have all sorts of
> fragmentation problems. So without more details about the workload,
> the kernel that is being used, the fragmentation occurring (xfs_bmap
> is your friend) and the samba config, there's little that can be
> suggested here.

[root@Clarus-gl2k10-2 ~]# uname -a
Linux Clarus-gl2k10-2 2.6.28.10 #6 SMP Thu Apr 29 01:00:58 CEST 2010 x86_64 
x86_64 x86_64 GNU/Linux

In kernel, i have tried out the pdflush-patch wich let me set the number of 
pdflushd-s to 8-64.
This was not a good idea i know that, but if i am right this have no effet 
to the fragmentation because pdflush not works on the FS layer.
Or i have missed something?

[root@Clarus-gl2k10-2 ~]# smbd -V
Version 3.0.23c-2

smb.conf (filtered the # lines and the ; lines)

[global]
   workgroup = WORKGROUP
   server string = Gladiator-StreamLine-gl2k10-2
   security = user
   load printers = yes
   cups options = raw
   log file = /var/log/samba/%m.log
   max log size = 50
   dns proxy = no
[store]
......

The workload is one PC for "administration" namely filling up the storage, 
and 2 client wich can reach the media files in RO mount point.
The customer knows about "only write on file at one time".

>
> Cheers,
>
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Thanks,
Janos 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs