mkfs.xfs error creating large agcount an raid

mkfs.xfs error creating large agcount an raid

[Marcus Pereira]

I have an issue when creating xfs volume using large agcounts on raid 
volumes.

/dev/md0 is a 4 disks raid 0 array:

----------------------------------------
# mkfs.xfs -V
mkfs.xfs version 3.1.4

# mkfs.xfs -d agcount=1872 -b size=4096 /dev/md0 -f
Warning: AG size is a multiple of stripe width.  This can cause performance
problems by aligning all AGs on the same disk.  To avoid this, run mkfs with
an AG size that is one stripe unit smaller, for example 147840.
log stripe unit (524288 bytes) is too large (maximum is 256KiB)
log stripe unit adjusted to 32KiB
meta-data=/dev/md0               isize=256    agcount=1872, 
agsize=147968 blks
          =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=276810752, imaxpct=5
          =                       sunit=128    swidth=512 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=135168, version=2
          =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
mkfs.xfs: pwrite64 failed: No space left on device

# mkfs.xfs -d agcount=1871 -b size=4096 /dev/md0 -f
Warning: AG size is a multiple of stripe width.  This can cause performance
problems by aligning all AGs on the same disk.  To avoid this, run mkfs with
an AG size that is one stripe unit smaller, for example 147840.
log stripe unit (524288 bytes) is too large (maximum is 256KiB)
log stripe unit adjusted to 32KiB
meta-data=/dev/md0               isize=256    agcount=1871, 
agsize=147968 blks
          =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=276810752, imaxpct=5
          =                       sunit=128    swidth=512 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=135168, version=2
          =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

----------------------------------------------

Any agcount greater then 1871 will lead an error, below that is OK.
I have the same issue when creating xfs volumes on a lvm stripe but with 
different agcounts.

When the volume is not on an raid array any number of agcount is OK, so 
seems the problem is when sunit/swidth is used.

Marcus

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Stan Hoeppner]

On 6/25/2011 2:49 PM, Marcus Pereira wrote:
> I have an issue when creating xfs volume using large agcounts on raid
> volumes.

Yes, you do have an issue, but not the one you think.

> /dev/md0 is a 4 disks raid 0 array:
> 
> ----------------------------------------
> # mkfs.xfs -V
> mkfs.xfs version 3.1.4
> 
> # mkfs.xfs -d agcount=1872 -b size=4096 /dev/md0 -f

mkfs.xfs queries mdraid for its parameters and creates close to the
optimal number of AGs, sets the stripe width, etc, all automatically.
The default number of AGs for striped mdraid devices is 16 IIRC, and
even that is probably a tad too high for a 4 spindle stripe.  Four or
eight AGs would probably be better here, depending on your workload,
which you did not state.  Please state your target workload.

At 1872 you have 117 times the number of default AGs.  The two main
downsides to doing this are:

1. Abysmal performance due to excessive head seeking on an epic scale
2. Premature drive failure due to head actuator failure

Now, the above assumes your "4 disks" are mechanical drives.  If these
are actually SSDs then the hardware won't suffer failures, but
performance will likely be far less than optimal.

Why are you attempting to create an insane number of allocation groups?
 What benefit do you expect to gain from doing so?

Regardless of your answer, the correct answer is that such high AG
counts only have downsides, and zero upside.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Marcus Pereira]

Em 25-06-2011 23:09, Stan Hoeppner escreveu:
> On 6/25/2011 2:49 PM, Marcus Pereira wrote:
>> I have an issue when creating xfs volume using large agcounts on raid
>> volumes.
> Yes, you do have an issue, but not the one you think.
Ok, but seems something that should be corrected. Isn't that?

>> /dev/md0 is a 4 disks raid 0 array:
>>
>> ----------------------------------------
>> # mkfs.xfs -V
>> mkfs.xfs version 3.1.4
>>
>> # mkfs.xfs -d agcount=1872 -b size=4096 /dev/md0 -f
> mkfs.xfs queries mdraid for its parameters and creates close to the
> optimal number of AGs, sets the stripe width, etc, all automatically.
> The default number of AGs for striped mdraid devices is 16 IIRC, and
> even that is probably a tad too high for a 4 spindle stripe.  Four or
> eight AGs would probably be better here, depending on your workload,
> which you did not state.  Please state your target workload.
The system is a heavy loaded email server.
> At 1872 you have 117 times the number of default AGs.  The two main
> downsides to doing this are:
The default agcount was 32 at this system.
> 1. Abysmal performance due to excessive head seeking on an epic scale
> 2. Premature drive failure due to head actuator failure
There is already insane head seeking at this server, hundreds of 
simultaneous users reading their mailboxes. In fact I was trying to 
reduce the head seeking with larger agcounts.

> Now, the above assumes your "4 disks" are mechanical drives.  If these
> are actually SSDs then the hardware won't suffer failures, but
> performance will likely be far less than optimal.
The 4 disks are mechanical, in fact each of them are 2 SCSI HD raid 1 
hardware raid 0 array but the OS sees it as a single device.
So its a raid 10 with hardware raid 1 and software raid 0.

> Why are you attempting to create an insane number of allocation groups?
>   What benefit do you expect to gain from doing so?
>
> Regardless of your answer, the correct answer is that such high AG
> counts only have downsides, and zero upside.
It is still a test to find an optimal agcount, there are several of this 
servers and each of them would be with a different agcount. I was trying 
to build an even larger agcount something like 20000 to 30000. :-)
The goal is to try to keep less or even 1 mailboxes per AG so more 
sequential reading at each mailbox access and less random seek at the 
volume. I dont know if it was going to work like I was thinking.
I got this idea at this post and was giving it a try: 
http://www.techforce.com.br/news/linux_blog/lvm_raid_xfs_ext3_tuning_for_small_files_parallel_i_o_on_debian

-- 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Stan Hoeppner]

On 6/26/2011 12:53 AM, Marcus Pereira wrote:
> Em 25-06-2011 23:09, Stan Hoeppner escreveu:
>> On 6/25/2011 2:49 PM, Marcus Pereira wrote:
>>> I have an issue when creating xfs volume using large agcounts on raid
>>> volumes.
>> Yes, you do have an issue, but not the one you think.
> Ok, but seems something that should be corrected. Isn't that?

No.  The error you received had nothing directly to do with the insane
AG count.  You received the error because with that specific AG count
you end up with the alignment issue that was stated in the error message
itself.

>>> /dev/md0 is a 4 disks raid 0 array:
>>>
>>> ----------------------------------------
>>> # mkfs.xfs -V
>>> mkfs.xfs version 3.1.4
>>>
>>> # mkfs.xfs -d agcount=1872 -b size=4096 /dev/md0 -f
>> mkfs.xfs queries mdraid for its parameters and creates close to the
>> optimal number of AGs, sets the stripe width, etc, all automatically.
>> The default number of AGs for striped mdraid devices is 16 IIRC, and
>> even that is probably a tad too high for a 4 spindle stripe.  Four or
>> eight AGs would probably be better here, depending on your workload,
>> which you did not state.  Please state your target workload.

> The system is a heavy loaded email server.

Maildir is much more metadata intensive than mbox, generating many more
small IOs, and thus head movement.  With a large number of allocation
groups this will exacerbate the head seeking problem.

>> At 1872 you have 117 times the number of default AGs.  The two main
>> downsides to doing this are:

> The default agcount was 32 at this system.

That seems high.  IIRC the default for mdraid stripes is 16 AGs.  Maybe
the default is higher for RAID0 (which I never use).

>> 1. Abysmal performance due to excessive head seeking on an epic scale
>> 2. Premature drive failure due to head actuator failure
> There is already insane head seeking at this server, hundreds of
> simultaneous users reading their mailboxes. In fact I was trying to
> reduce the head seeking with larger agcounts.
> 
>> Now, the above assumes your "4 disks" are mechanical drives.  If these
>> are actually SSDs then the hardware won't suffer failures, but
>> performance will likely be far less than optimal.

> The 4 disks are mechanical, in fact each of them are 2 SCSI HD raid 1
> hardware raid 0 array but the OS sees it as a single device.
> So its a raid 10 with hardware raid 1 and software raid 0.

Please always provide this level of detail up front.  Until now you had
us believing this was a straight RAID0 stripe for storing mail.

>> Why are you attempting to create an insane number of allocation groups?
>>   What benefit do you expect to gain from doing so?
>>
>> Regardless of your answer, the correct answer is that such high AG
>> counts only have downsides, and zero upside.

> It is still a test to find an optimal agcount, there are several of this
> servers and each of them would be with a different agcount. I was trying
> to build an even larger agcount something like 20000 to 30000. :-)

You have no idea what you are doing.  You have no understanding of XFS
allocation groups.  See 'man 5 xfs' and search this list's archives for
threads discussing agcount.

> The goal is to try to keep less or even 1 mailboxes per AG so more
> sequential reading at each mailbox access and less random seek at the
> volume. 

The logic behind your goal is flawed.  Each AG contains its own metadata
section which contains btrees for inodes and freespace.  When new mail
is written into a user maildir the btrees for that AG are read from
disk, unless cached.  With the numbers of AGs you're talking about,
you're increasing your head seeks for metadata reads by several orders
of magnitude as you now have 1872 metadata sections to read/write
instead of something sane like 16.

> I dont know if it was going to work like I was thinking.

It won't.

> I got this idea at this post and was giving it a try:
> http://www.techforce.com.br/news/linux_blog/lvm_raid_xfs_ext3_tuning_for_small_files_parallel_i_o_on_debian

Did you happen to notice that configuration has an IBM DS8300 SAN head
with tons of BBWC and *512* fiber channel disks?  You have 8 disks.

You are attempting to duplicate a filesystem configuration, that may
work well on that specific high end platform, but is never going to work
on your 8 disk machine.  As is stated in that article, they tuned and
re-tuned that system over a very long period of time before arriving
where they are.  They have tuned XFS to that specific machine/storage
environment.

Those lessons are not directly applicable to your system.  In fact
they're not applicable at all.

Stick with a sane agcount of 8 or 16.  Also, for a maildir server with
XFS you'd be better off concatenating those 4 RAID1 pairs instead of
striping them, due to the fact that mail files are so small, typically
4-16KB, which can cause many partial width stripes, decreasing overall
performance.

Using concatenation (mdadm --linear) you can take more advantage of
allocation group parallelism and achieve better overall throughput vs
the md RAID0 over hardware RAID1 setup.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Stan Hoeppner]

On 6/26/2011 4:26 PM, Stan Hoeppner wrote:
> On 6/26/2011 12:53 AM, Marcus Pereira wrote:

>> I got this idea at this post and was giving it a try:
>> http://www.techforce.com.br/news/linux_blog/lvm_raid_xfs_ext3_tuning_for_small_files_parallel_i_o_on_debian
> 
> Did you happen to notice that configuration has an IBM DS8300 SAN head
> with tons of BBWC and *512* fiber channel disks?  You have 8 disks.

The DS8300 has up to 256GB of read/write cache:
http://publib.boulder.ibm.com/infocenter/dsichelp/ds8000ic/index.jsp?topic=%2Fcom.ibm.storage.ssic.help.doc%2Ff2c_ds8300models922and9a2_1w2zq9.html

It's difficult to convey how critical the DS8300, its 8 processor cores,
and its gargantuan cache are in allowing the author to get away with
using so many small XFS allocation groups.  If his storage back end had
consisted of plain HBAs or relatively small cache RAID cards connected
to JBODs w/expanders, using that many small AGs would likely have caused
serious performance degradation instead of a performance increase.

Performance tuning is system specific.  You read a Ferrari tuning manual
and are attempting to apply that to tuning your Volkswagon.  That's
never going to work.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Dave Chinner]

On Sun, Jun 26, 2011 at 02:53:43AM -0300, Marcus Pereira wrote:
> Em 25-06-2011 23:09, Stan Hoeppner escreveu:
> >On 6/25/2011 2:49 PM, Marcus Pereira wrote:
> >>I have an issue when creating xfs volume using large agcounts on raid
> >>volumes.
> >Yes, you do have an issue, but not the one you think.
> Ok, but seems something that should be corrected. Isn't that?
> 
> >>/dev/md0 is a 4 disks raid 0 array:
> >>
> >>----------------------------------------
> >># mkfs.xfs -V
> >>mkfs.xfs version 3.1.4
> >>
> >># mkfs.xfs -d agcount=1872 -b size=4096 /dev/md0 -f
> >mkfs.xfs queries mdraid for its parameters and creates close to the
> >optimal number of AGs, sets the stripe width, etc, all automatically.
> >The default number of AGs for striped mdraid devices is 16 IIRC, and
> >even that is probably a tad too high for a 4 spindle stripe.  Four or
> >eight AGs would probably be better here, depending on your workload,
> >which you did not state.  Please state your target workload.
> The system is a heavy loaded email server.
> >At 1872 you have 117 times the number of default AGs.  The two main
> >downsides to doing this are:
> The default agcount was 32 at this system.
> >1. Abysmal performance due to excessive head seeking on an epic scale
> >2. Premature drive failure due to head actuator failure
> There is already insane head seeking at this server, hundreds of
> simultaneous users reading their mailboxes.

Perhaps you should just usethe defaults first and only consider
changes if there is an obvious problem,?

> In fact I was trying to
> reduce the head seeking with larger agcounts.

AGs are not for reducing seeking - they are for increasing
allocation parallelism and scaling freespace indexes to extremely
large filesystem sizes.

In fact, trying to use more than a few hundred AGs will hit internal
AG indexing scalability limitations, especially as you start to fill
up AGs and have to scan for AGs with free space in them.

IOWs, using large numbers of AGs are inadvisable for many reasons.

> >are actually SSDs then the hardware won't suffer failures, but
> >performance will likely be far less than optimal.
> The 4 disks are mechanical, in fact each of them are 2 SCSI HD raid
> 1 hardware raid 0 array but the OS sees it as a single device.
> So its a raid 10 with hardware raid 1 and software raid 0.
> 
> >Why are you attempting to create an insane number of allocation groups?
> >  What benefit do you expect to gain from doing so?
> >
> >Regardless of your answer, the correct answer is that such high AG
> >counts only have downsides, and zero upside.
> It is still a test to find an optimal agcount, there are several of
> this servers and each of them would be with a different agcount. I
> was trying to build an even larger agcount something like 20000 to
> 30000. :-)

IOWs, you truly do not understand how AGs are used to scale
filesystem performance and therefore you should be using the
defaults.

> The goal is to try to keep less or even 1 mailboxes per AG so more
> sequential reading at each mailbox access and less random seek at
> the volume.

You have no direct control over the placement of directories and
files in AGs, so it doesn't matter how many AGs you create, you
aren't going to be able to achieve this....

> I dont know if it was going to work like I was thinking.
> I got this idea at this post and was giving it a try:
> http://www.techforce.com.br/news/linux_blog/lvm_raid_xfs_ext3_tuning_for_small_files_parallel_i_o_on_debian

<sigh>

http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E

Cheers,

Dave.
> 
> -- 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Stan Hoeppner]

On 6/26/2011 6:59 PM, Dave Chinner wrote:
> On Sun, Jun 26, 2011 at 02:53:43AM -0300, Marcus Pereira wrote:

>> There is already insane head seeking at this server, hundreds of
>> simultaneous users reading their mailboxes.

I missed this part of the reply probably due to the reply formatting
style used.

> Perhaps you should just usethe defaults first and only consider
> changes if there is an obvious problem,?
> 
>> In fact I was trying to
>> reduce the head seeking with larger agcounts.
> 
> AGs are not for reducing seeking - they are for increasing
> allocation parallelism and scaling freespace indexes to extremely
> large filesystem sizes.
> 
> In fact, trying to use more than a few hundred AGs will hit internal
> AG indexing scalability limitations, especially as you start to fill
> up AGs and have to scan for AGs with free space in them.
> 
> IOWs, using large numbers of AGs are inadvisable for many reasons.

Marcus, if you are seeing excessive head seeking with this maildir
workload, the problem isn't the number of AGs.  The problem is that
you're using striping for a small file high random IOPS workload, and
you don't have enough spindles, thus not enough seek bandwidth.

I recommend 3 changes, one of which I previously mentioned:

1.  Use 8 mirror pairs instead of 4
2.  Don't use striping.  Make an mdraid --linear device of the 8 mirrors
3.  Format with '-d agcount=32' which will give you 4 AGs per spindle

With striping, each file IO will generate multiple head seeks per
spindle across all spindles.  With a linear array each file IO will
typically only generate seeks on a single spindle, as the metadata and
file data should be within the same AG.

Test this configuration and post your results.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Marcus Pereira]

Em 27-06-2011 00:33, Stan Hoeppner escreveu:
>
> I recommend 3 changes, one of which I previously mentioned:
>
> 1.  Use 8 mirror pairs instead of 4
> 2.  Don't use striping.  Make an mdraid --linear device of the 8 mirrors
> 3.  Format with '-d agcount=32' which will give you 4 AGs per spindle
>
> Test this configuration and post your results.

I am thanks for all advices. I will make the tests and post, may take 
some time.

About all other messages. My system may not be a Ferrari but its not a 
Volks. I certainly do not have that many HDs in fiber channel, but the 
sever is a dual core Xeon 6 cores with HT. Linux sees a total of 24 
cores, total RAM is 24GB. The HDs are all SAS 15Krpm and the system runs 
on SSD. They are dedicated to handle the maildir files and I have 
several of those servers running nicely.
But I don’t want to make the thread about my system larger.

Yes, I don’t know much about XFS and Allocation groups, thanks for you 
all to help me a bit.

At the end the reason why I opened the thread it the error and the 
developers should take some care about that.

Ok, no reason to use that many agcount but giving a "mkfs.xfs: pwrite64 
failed: No space left on device" error for me stills seems a bug.
I manage to create a XFS volume with agcount=30000 on a normal device, 
no error or warning.
On md or lvm arrays I got that error at some point.

Marcus


-- 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Stan Hoeppner]

On 6/26/2011 11:14 PM, Marcus Pereira wrote:
> Em 27-06-2011 00:33, Stan Hoeppner escreveu:
>>
>> I recommend 3 changes, one of which I previously mentioned:
>>
>> 1.  Use 8 mirror pairs instead of 4
>> 2.  Don't use striping.  Make an mdraid --linear device of the 8 mirrors
>> 3.  Format with '-d agcount=32' which will give you 4 AGs per spindle
>>
>> Test this configuration and post your results.
> 
> I am thanks for all advices. I will make the tests and post, may take
> some time.
> 
> About all other messages. My system may not be a Ferrari but its not a
> Volks. I certainly do not have that many HDs in fiber channel, but the
> sever is a dual core Xeon 6 cores with HT. Linux sees a total of 24
> cores, total RAM is 24GB. The HDs are all SAS 15Krpm and the system runs
> on SSD. They are dedicated to handle the maildir files and I have
> several of those servers running nicely.
> But I don’t want to make the thread about my system larger.

So you do or don't have the excessive head seek problem you previously
mentioned?  If not then use the mkfs.xfs defaults.

> Yes, I don’t know much about XFS and Allocation groups, thanks for you
> all to help me a bit.

You're welcome.  Google should turn up a decent amount of information
about XFS allocation groups if you're interested in further reading.

> At the end the reason why I opened the thread it the error and the
> developers should take some care about that.

> Ok, no reason to use that many agcount but giving a "mkfs.xfs: pwrite64
> failed: No space left on device" error for me stills seems a bug.

The definition of a software bug stipulates incorrect or unexpected
program behavior.  Error messages aren't bugs unless the wrong error
message is returned for a given fault condition, or no error is returned
when one should be.

Are you stipulating that the above isn't the correct error message for
the fault condition?  Or do you simply not understand the error message?
 If the latter, maybe you should simply ask what that error means before
saying the error message is a bug. :)

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Paul Anderson]

One thing this thread indicates is the need for a warning in mkfs.xfs
- according to several developers, there is, I think, linear increase
in allocation time to number of allocation groups.

It would be helpful for the end user to simply issue a warning stating
this when the AG count seems high with a brief explanation as to why
it seems high.  I would allow it, but print the warning.  Even a
simple linear check like agroups>500 should suffice for "a while".

Paul

On Mon, Jun 27, 2011 at 4:55 AM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 6/26/2011 11:14 PM, Marcus Pereira wrote:
>> Em 27-06-2011 00:33, Stan Hoeppner escreveu:
>>>
>>> I recommend 3 changes, one of which I previously mentioned:
>>>
>>> 1.  Use 8 mirror pairs instead of 4
>>> 2.  Don't use striping.  Make an mdraid --linear device of the 8 mirrors
>>> 3.  Format with '-d agcount=32' which will give you 4 AGs per spindle
>>>
>>> Test this configuration and post your results.
>>
>> I am thanks for all advices. I will make the tests and post, may take
>> some time.
>>
>> About all other messages. My system may not be a Ferrari but its not a
>> Volks. I certainly do not have that many HDs in fiber channel, but the
>> sever is a dual core Xeon 6 cores with HT. Linux sees a total of 24
>> cores, total RAM is 24GB. The HDs are all SAS 15Krpm and the system runs
>> on SSD. They are dedicated to handle the maildir files and I have
>> several of those servers running nicely.
>> But I don’t want to make the thread about my system larger.
>
> So you do or don't have the excessive head seek problem you previously
> mentioned?  If not then use the mkfs.xfs defaults.
>
>> Yes, I don’t know much about XFS and Allocation groups, thanks for you
>> all to help me a bit.
>
> You're welcome.  Google should turn up a decent amount of information
> about XFS allocation groups if you're interested in further reading.
>
>> At the end the reason why I opened the thread it the error and the
>> developers should take some care about that.
>
>> Ok, no reason to use that many agcount but giving a "mkfs.xfs: pwrite64
>> failed: No space left on device" error for me stills seems a bug.
>
> The definition of a software bug stipulates incorrect or unexpected
> program behavior.  Error messages aren't bugs unless the wrong error
> message is returned for a given fault condition, or no error is returned
> when one should be.
>
> Are you stipulating that the above isn't the correct error message for
> the fault condition?  Or do you simply not understand the error message?
>  If the latter, maybe you should simply ask what that error means before
> saying the error message is a bug. :)
>
> --
> Stan
>
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Eric Sandeen]

On 6/27/11 8:04 AM, Paul Anderson wrote:
> One thing this thread indicates is the need for a warning in mkfs.xfs
> - according to several developers, there is, I think, linear increase
> in allocation time to number of allocation groups.
> 
> It would be helpful for the end user to simply issue a warning stating
> this when the AG count seems high with a brief explanation as to why
> it seems high.  I would allow it, but print the warning.  Even a
> simple linear check like agroups>500 should suffice for "a while".

I disagree.

There are all sorts of ways a user can shoot themselves in the foot with
unix commands.  Detecting and warning about all of them is a fool's errand.

======================================
= Warning!  mkfs.xfs detected insane =
=   option specification.  Cancel?   =
=                                    =
=      [   OK   ]     [ Cancel ]     =
======================================

-Eric

> Paul
> 
> On Mon, Jun 27, 2011 at 4:55 AM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 6/26/2011 11:14 PM, Marcus Pereira wrote:
>>> Em 27-06-2011 00:33, Stan Hoeppner escreveu:
>>>>
>>>> I recommend 3 changes, one of which I previously mentioned:
>>>>
>>>> 1.  Use 8 mirror pairs instead of 4
>>>> 2.  Don't use striping.  Make an mdraid --linear device of the 8 mirrors
>>>> 3.  Format with '-d agcount=32' which will give you 4 AGs per spindle
>>>>
>>>> Test this configuration and post your results.
>>>
>>> I am thanks for all advices. I will make the tests and post, may take
>>> some time.
>>>
>>> About all other messages. My system may not be a Ferrari but its not a
>>> Volks. I certainly do not have that many HDs in fiber channel, but the
>>> sever is a dual core Xeon 6 cores with HT. Linux sees a total of 24
>>> cores, total RAM is 24GB. The HDs are all SAS 15Krpm and the system runs
>>> on SSD. They are dedicated to handle the maildir files and I have
>>> several of those servers running nicely.
>>> But I don’t want to make the thread about my system larger.
>>
>> So you do or don't have the excessive head seek problem you previously
>> mentioned?  If not then use the mkfs.xfs defaults.
>>
>>> Yes, I don’t know much about XFS and Allocation groups, thanks for you
>>> all to help me a bit.
>>
>> You're welcome.  Google should turn up a decent amount of information
>> about XFS allocation groups if you're interested in further reading.
>>
>>> At the end the reason why I opened the thread it the error and the
>>> developers should take some care about that.
>>
>>> Ok, no reason to use that many agcount but giving a "mkfs.xfs: pwrite64
>>> failed: No space left on device" error for me stills seems a bug.
>>
>> The definition of a software bug stipulates incorrect or unexpected
>> program behavior.  Error messages aren't bugs unless the wrong error
>> message is returned for a given fault condition, or no error is returned
>> when one should be.
>>
>> Are you stipulating that the above isn't the correct error message for
>> the fault condition?  Or do you simply not understand the error message?
>>  If the latter, maybe you should simply ask what that error means before
>> saying the error message is a bug. :)
>>
>> --
>> Stan
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
>>
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Paul Anderson]

On Mon, Jun 27, 2011 at 11:10 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> On 6/27/11 8:04 AM, Paul Anderson wrote:
>> One thing this thread indicates is the need for a warning in mkfs.xfs
>> - according to several developers, there is, I think, linear increase
>> in allocation time to number of allocation groups.
>>
>> It would be helpful for the end user to simply issue a warning stating
>> this when the AG count seems high with a brief explanation as to why
>> it seems high.  I would allow it, but print the warning.  Even a
>> simple linear check like agroups>500 should suffice for "a while".
>
> I disagree.
>
> There are all sorts of ways a user can shoot themselves in the foot with
> unix commands.  Detecting and warning about all of them is a fool's errand.

Clearly a philosophical difference.

In managing complex software, it is far better for users if the
software itself can simply report why something is a problem, without
resorting to expecting users to read source code or ask developers
why.

There is nothing in the man page I see indicating what is good or bad
regarding allocation groups - either document it there or warn in the
software.  If allocation algorithms are linear with respect to
allocation groups, the something like this should be stated in the man
pages.

Doing neither leads to frustrated end users.  If you answer is "use
the defaults" then explain why and which parameters is applies to
(again in the documentation).

Also, it is not hard to do, and would have in this instance saved
developer time.  Since the issue has come up a few times the last
month or so, it seems worthwhile to deal with.

It's sort of like the story about giving a person a fish versus
teaching them how to fish.

Paul


>
> ======================================
> = Warning!  mkfs.xfs detected insane =
> =   option specification.  Cancel?   =
> =                                    =
> =      [   OK   ]     [ Cancel ]     =
> ======================================
>
> -Eric
>
>> Paul
>>
>> On Mon, Jun 27, 2011 at 4:55 AM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>> On 6/26/2011 11:14 PM, Marcus Pereira wrote:
>>>> Em 27-06-2011 00:33, Stan Hoeppner escreveu:
>>>>>
>>>>> I recommend 3 changes, one of which I previously mentioned:
>>>>>
>>>>> 1.  Use 8 mirror pairs instead of 4
>>>>> 2.  Don't use striping.  Make an mdraid --linear device of the 8 mirrors
>>>>> 3.  Format with '-d agcount=32' which will give you 4 AGs per spindle
>>>>>
>>>>> Test this configuration and post your results.
>>>>
>>>> I am thanks for all advices. I will make the tests and post, may take
>>>> some time.
>>>>
>>>> About all other messages. My system may not be a Ferrari but its not a
>>>> Volks. I certainly do not have that many HDs in fiber channel, but the
>>>> sever is a dual core Xeon 6 cores with HT. Linux sees a total of 24
>>>> cores, total RAM is 24GB. The HDs are all SAS 15Krpm and the system runs
>>>> on SSD. They are dedicated to handle the maildir files and I have
>>>> several of those servers running nicely.
>>>> But I don’t want to make the thread about my system larger.
>>>
>>> So you do or don't have the excessive head seek problem you previously
>>> mentioned?  If not then use the mkfs.xfs defaults.
>>>
>>>> Yes, I don’t know much about XFS and Allocation groups, thanks for you
>>>> all to help me a bit.
>>>
>>> You're welcome.  Google should turn up a decent amount of information
>>> about XFS allocation groups if you're interested in further reading.
>>>
>>>> At the end the reason why I opened the thread it the error and the
>>>> developers should take some care about that.
>>>
>>>> Ok, no reason to use that many agcount but giving a "mkfs.xfs: pwrite64
>>>> failed: No space left on device" error for me stills seems a bug.
>>>
>>> The definition of a software bug stipulates incorrect or unexpected
>>> program behavior.  Error messages aren't bugs unless the wrong error
>>> message is returned for a given fault condition, or no error is returned
>>> when one should be.
>>>
>>> Are you stipulating that the above isn't the correct error message for
>>> the fault condition?  Or do you simply not understand the error message?
>>>  If the latter, maybe you should simply ask what that error means before
>>> saying the error message is a bug. :)
>>>
>>> --
>>> Stan
>>>
>>> _______________________________________________
>>> xfs mailing list
>>> xfs@oss.sgi.com
>>> http://oss.sgi.com/mailman/listinfo/xfs
>>>
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
>>
>
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Eric Sandeen]

On 6/27/11 10:27 AM, Paul Anderson wrote:
> On Mon, Jun 27, 2011 at 11:10 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
>> On 6/27/11 8:04 AM, Paul Anderson wrote:
>>> One thing this thread indicates is the need for a warning in mkfs.xfs
>>> - according to several developers, there is, I think, linear increase
>>> in allocation time to number of allocation groups.
>>>
>>> It would be helpful for the end user to simply issue a warning stating
>>> this when the AG count seems high with a brief explanation as to why
>>> it seems high.  I would allow it, but print the warning.  Even a
>>> simple linear check like agroups>500 should suffice for "a while".
>>
>> I disagree.
>>
>> There are all sorts of ways a user can shoot themselves in the foot with
>> unix commands.  Detecting and warning about all of them is a fool's errand.
> 
> Clearly a philosophical difference.
> 
> In managing complex software, it is far better for users if the
> software itself can simply report why something is a problem, without
> resorting to expecting users to read source code or ask developers
> why.
> 
> There is nothing in the man page I see indicating what is good or bad
> regarding allocation groups - either document it there or warn in the
> software.  If allocation algorithms are linear with respect to
> allocation groups, the something like this should be stated in the man
> pages.
> 
> Doing neither leads to frustrated end users.  If you answer is "use
> the defaults" then explain why and which parameters is applies to
> (again in the documentation).
> 
> Also, it is not hard to do, and would have in this instance saved
> developer time.  Since the issue has come up a few times the last
> month or so, it seems worthwhile to deal with.

This one instance would not be hard to do, agreed.

To point out every potential pitfall in every bad option or combination
of options would be impossible.

I'd be happy with a version of the FAQ entry Dave pointed to in
the mkfs.xfs manpage, though, basically "don't change the defaults
unless you know for sure that it will address a shortcoming you have
seen in testing."

> It's sort of like the story about giving a person a fish versus
> teaching them how to fish.

Or a little like performing neurosurgery on a person vs. teaching
them how to do it themselves.  ;)

There is lots of "teaching to fish" in the documentation and the FAQ;
until you are really able to delve into the technical complexities
of XFS you probably should not try to fish in water that is too deep.

Just because a knob is there doesn't mean you should turn it as
far as it can go, and I don't think it's our job to warn against
that in every instance, either...

Those who wish to learn would be well advised to read up on the
many detailed technical docs available; I don't think that the
mkfs.xfs code is the right place to do this teaching, though.

But I guess it is a philosophical difference.

-Eric

> Paul
> 
> 
>>
>> ======================================
>> = Warning!  mkfs.xfs detected insane =
>> =   option specification.  Cancel?   =
>> =                                    =
>> =      [   OK   ]     [ Cancel ]     =
>> ======================================
>>
>> -Eric
>>
>>> Paul
>>>
>>> On Mon, Jun 27, 2011 at 4:55 AM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>>>> On 6/26/2011 11:14 PM, Marcus Pereira wrote:
>>>>> Em 27-06-2011 00:33, Stan Hoeppner escreveu:
>>>>>>
>>>>>> I recommend 3 changes, one of which I previously mentioned:
>>>>>>
>>>>>> 1.  Use 8 mirror pairs instead of 4
>>>>>> 2.  Don't use striping.  Make an mdraid --linear device of the 8 mirrors
>>>>>> 3.  Format with '-d agcount=32' which will give you 4 AGs per spindle
>>>>>>
>>>>>> Test this configuration and post your results.
>>>>>
>>>>> I am thanks for all advices. I will make the tests and post, may take
>>>>> some time.
>>>>>
>>>>> About all other messages. My system may not be a Ferrari but its not a
>>>>> Volks. I certainly do not have that many HDs in fiber channel, but the
>>>>> sever is a dual core Xeon 6 cores with HT. Linux sees a total of 24
>>>>> cores, total RAM is 24GB. The HDs are all SAS 15Krpm and the system runs
>>>>> on SSD. They are dedicated to handle the maildir files and I have
>>>>> several of those servers running nicely.
>>>>> But I don’t want to make the thread about my system larger.
>>>>
>>>> So you do or don't have the excessive head seek problem you previously
>>>> mentioned?  If not then use the mkfs.xfs defaults.
>>>>
>>>>> Yes, I don’t know much about XFS and Allocation groups, thanks for you
>>>>> all to help me a bit.
>>>>
>>>> You're welcome.  Google should turn up a decent amount of information
>>>> about XFS allocation groups if you're interested in further reading.
>>>>
>>>>> At the end the reason why I opened the thread it the error and the
>>>>> developers should take some care about that.
>>>>
>>>>> Ok, no reason to use that many agcount but giving a "mkfs.xfs: pwrite64
>>>>> failed: No space left on device" error for me stills seems a bug.
>>>>
>>>> The definition of a software bug stipulates incorrect or unexpected
>>>> program behavior.  Error messages aren't bugs unless the wrong error
>>>> message is returned for a given fault condition, or no error is returned
>>>> when one should be.
>>>>
>>>> Are you stipulating that the above isn't the correct error message for
>>>> the fault condition?  Or do you simply not understand the error message?
>>>>  If the latter, maybe you should simply ask what that error means before
>>>> saying the error message is a bug. :)
>>>>
>>>> --
>>>> Stan
>>>>
>>>> _______________________________________________
>>>> xfs mailing list
>>>> xfs@oss.sgi.com
>>>> http://oss.sgi.com/mailman/listinfo/xfs
>>>>
>>>
>>> _______________________________________________
>>> xfs mailing list
>>> xfs@oss.sgi.com
>>> http://oss.sgi.com/mailman/listinfo/xfs
>>>
>>
>>
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Stan Hoeppner]

On 6/27/2011 10:27 AM, Paul Anderson wrote:

> There is nothing in the man page I see indicating what is good or bad
> regarding allocation groups - either document it there or warn in the
> software.  If allocation algorithms are linear with respect to
> allocation groups, the something like this should be stated in the man
> pages.

It is.  From the 2nd paragraph of 'man 5 xfs':

...The data section is divided into a number of allocation groups. The
number and size of the allocation groups are chosen by mkfs.xfs(8) so
that there is normally a small number of equal-sized groups. The number
of allocation groups controls the amount of parallelism available in
file and block allocation. It should be increased from the default if
there is sufficient memory and a lot of allocation activity. The number
of allocation groups should not be set very high, since this can cause
large amounts of CPU time to be used by the filesystem, especially when
the filesystem is nearly full. More allocation groups are added (of the
original size) when xfs_growfs(8) is run.

Maybe some of this information could/should be moved to the agcount
section of 'man mkfs.xfs'.  I'll concede that "should not be set very
high" is subjective for novice XFS users.  To Marcus 20,000 may not be
"very high". :)

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: mkfs.xfs error creating large agcount an raid

[Dave Chinner]

On Mon, Jun 27, 2011 at 11:27:00AM -0400, Paul Anderson wrote:
> On Mon, Jun 27, 2011 at 11:10 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> > On 6/27/11 8:04 AM, Paul Anderson wrote:
> >> One thing this thread indicates is the need for a warning in mkfs.xfs
> >> - according to several developers, there is, I think, linear increase
> >> in allocation time to number of allocation groups.
> >>
> >> It would be helpful for the end user to simply issue a warning stating
> >> this when the AG count seems high with a brief explanation as to why
> >> it seems high.  I would allow it, but print the warning.  Even a
> >> simple linear check like agroups>500 should suffice for "a while".
> >
> > I disagree.
> >
> > There are all sorts of ways a user can shoot themselves in the foot with
> > unix commands.  Detecting and warning about all of them is a fool's errand.
> 
> Clearly a philosophical difference.
> 
> In managing complex software, it is far better for users if the
> software itself can simply report why something is a problem, without
> resorting to expecting users to read source code or ask developers
> why.

I don't expect users to read code. We have documentation, we expect
users to read it first and have some understanding of how stuff works.
If you don't understand it, then I expect that users will ask
questions on the mailing list. That's -exactly- how the OSS world
works and one of it's great strengths - if you don't understand how
something works you can ask the developers directly!

You seem to be indicating that the XFS developers should be handing
everything to users on a silver platter so they don't have to think
about anything. i.e. that they need *no prior knowledge* to use XFS
effectively and everything should just work.

Welcome to the real world, buddy.

XFS is aimed at high-end, large-scale, high-performance,
high-reliability, 24x7 production workloads.  Any user who ticks
even one of those boxes tends to have at least some knowledge of
their storage and filesystems. If they don't have the necessary
knowledge, they generally have a process via which to evaluate new
technologies for their environment. If they have neither, then they
aren't qualified for the job they've been asked to do.....

Hence we assume that users - at minimum - have read the various man
pages and some of the user/admin documentation on xfs.org before
starting out.  Most of these people tend to heed the "use the
defaults" recommendations, and we typically never hear from them
because Stuff Just Works The Way It Should.

In this particular case, the user had obviously read some of the
documentation, but was applying scalability concepts inappropriately
for the problem he was trying to solve. That is not a bad thing - it
happens all the time - and asking questions is the quickest and
easiest way to understand why something didn't work as expected.
e.g. Just look at Stan's responses about reconfiguring the entire
storage stack to be more optimal for the specific workload the user
was running.

The result of this thread is that the user had a problem, and has
come away with a greatly improved storage configuration from top to
bottom, along with a better understanding of how XFS works and a
better idea of the process needed to evaluate the benefit of
changes.

In the end, that's a user who is much more likely to be happy with
XFS, and a user that knows he'll get good support from the community
so is more likely to report problems in future. That's a win-win
situation as far as I'm concerned, and it helps keeps a high
signal-to-noise ratio on the list.

> There is nothing in the man page I see indicating what is good or bad
> regarding allocation groups - either document it there or warn in the
> software.  If allocation algorithms are linear with respect to
> allocation groups, the something like this should be stated in the man
> pages.

It is, in this man page:

$ man 5 xfs
xfs(5)

NAME
       xfs - layout of the XFS filesystem
.....

It warns that too many AGs is bad.  So really what you are
complaining about is that it doesn't define "too many", right?

Well, IMO, changing a config value by more than an order of
magnitude should ring alarm bells, let alone a change of 3-5 orders
of magnitude.  i.e. When your default is 4 ("small"), then 400 is
large and 4000 is clearly "very high". That's common sense, yes?

However, there may very well be a use case for having 4000 AGs in a
small filesystem - just because I haven't seen one doesn't mean it
doesn't exist, and therefore the capability to do this remains.

IOWs, we cater for people that need to do crazy (good) things, but
we also rely on people having the knoweldge and common sense to
determine whether they really should be doing something crazy or
not...

> Doing neither leads to frustrated end users.  If you answer is "use
> the defaults" then explain why and which parameters is applies to
> (again in the documentation).

As a developer, I always take the time to explain why something
is bad before pointing to the "use the defaults" FAQ entry. I do
that to help the education of everyone on the list that is reading,
and also to get the "don't do this" reason into google....

We've been doing this often enough now that often it is the long
term users that are making such responses and pointing out why
something is bad, not the developers. They've learnt enough from us
through the same process, and now they are helping educate the new
users in turn.

This is also a good feedback loop, because when someone else
explains the problems in their own words, I get to see how well they
understood the previous explainations. As such I'll often let such
threads run until I see some small misunderstanding in an
explanation - at that point I'll jump in to help these expert-users
improve their knowledge further.

Users with in-depth knowledge of the systems are a very valuable
resource - I often learn from their experience solving problems just
like they learn from mine. That's another big win-win situation from
my POV. ;)

> Also, it is not hard to do, and would have in this instance saved
> developer time.  Since the issue has come up a few times the last
> month or so, it seems worthwhile to deal with.

We cannot force people to read the documentation before they twiddle
knobs.

While I fully agree that user friendly interfaces are needed to make
storage administration easy for the technologically challenged, I
think that mkfs is not the place for such refinement.  User friendly
storage tools need to sit above mkfs, fsck, growfs, lvm, md,
dm-crypt, etc and hide *all* the deep, dark, nasty corners of the
storage stack from view. This is something the Sun engineers got
right when designing the ZFS UI.

IOWs, if such a UI is well designed, then it can drive mkfs in
different directions according to the user's needs without the user
needing to know a single thing about how the filesystem actually
works. Hell, done right the user won't even know (or need to care)
what filesystem they are using....

> It's sort of like the story about giving a person a fish versus
> teaching them how to fish.

"Give a man a fish and you feed him for a day. Teach a man to fish
and you feed him for a lifetime."
	- Chinese Proverb.

If you want to use that analogy, then consider that it takes months
to teach someone how to "fish XFS" properly. So they will have
starved before they can catch enough fish to be self-sufficient.
Yes, you can learn the basics of how to "fish XFS" from the
documentation, but it's not enough by itself to be a self-sufficient
expert....

If you think about what I've written above, you'll see that we do
indeed have a (long) process for teaching people how to "fish XFS".
You'll also see that asking questions or even doing silly things
that result in discussion threads like this is a core part of that
education process...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs