Warning: AG size is a multiple of stripe width?

Warning: AG size is a multiple of stripe width?

[Gim Leong Chin]

Hi,


I have a Sun workstation with eight Cheetah 15K.5 SAS 300 GB on RAID 1E (RAID 10) on LSI SAS3081E-R.

I am installing SLED 11 SP1 on it and I thought I will do a thorough optimization right down to the partition boundaries.

Since the default for XFS is to create four aggregation groups, and with the reasoning that Cheetah can do double the seeks of normal 7200 RPM drives, I have four aggregation groups per drive for a total of 16 for 70 GB /dev/sda2 partition, and eight per drive for total of 32 for /dev/sda3 partition (1011 GB).

I have aligned the partition start and end with the stripe width boundaries.
The stripe size is 64 kB, stripe width is 4*64 kB = 256 kB, in terms of 512 byte sectors:

70 GB /

No      Start           End             Number
1       512             67109375        32 GB = 67108864 sectors = 131072 stripe sets

2       67109376        213910015       70 GB = 146800640 sectors = 286720 stripe sets

3       213910016       2335932415      Left = 2335932416 - 213910016 = 2122022400 sectors = 4144575 stripe sets


When I do the following:

mkfs.xfs -f -b size=4k -d agcount=16,su=64k,sw=4 -i size=256,align=1,attr=2 -l version=2,su=64k,lazy-count=1 -n version=2 -s size=512 -L / /dev/sda2

Warning: AG size is a multiple of stripe width.  This can cause performance problems by aligning all AGs on the same disk.  To avoid this, run mkfs with an AG size that is one stripe unit smaller, for example 1146864

agcount=16 agsize=1146880 blks
bsize=4096
sunit=16 swidth=64 blks

mkfs.xfs -f -b size=4k -d agcount=32,su=64k,sw=4 -i size=256,align=1,attr=2 -l version=2,su=64k,lazy-count=1 -n version=2 -s size=512 -L /home /dev/sda3

Warning: AG size is a multiple of stripe width.  This can cause performance problems by aligning all AGs on the same disk.  To avoid this, run mkfs with an AG size that is one stripe unit smaller, for example 8289136

agcount=32 agsize=8289152 blks
bsize=4096
sunit=16 swidth=64 blks

I am really puzzled since I thought all I am doing is distributing 4 aggregation groups per drive for sda2 and 8 per drive for sda3.

What have I done wrong and what is the flaw with my understanding?

Thank you!


Chin Gim Leong


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Warning: AG size is a multiple of stripe width?

[Eric Sandeen]

On 6/17/11 6:36 AM, Gim Leong Chin wrote:
> Hi,
> 
> 
> I have a Sun workstation with eight Cheetah 15K.5 SAS 300 GB on RAID
> 1E (RAID 10) on LSI SAS3081E-R.
> 
> I am installing SLED 11 SP1 on it and I thought I will do a thorough
> optimization right down to the partition boundaries.
> 
> Since the default for XFS is to create four aggregation groups, and
> with the reasoning that Cheetah can do double the seeks of normal
> 7200 RPM drives, I have four aggregation groups per drive for a total

"Allocation groups" just FWIW :)

Probably no real reason to try to outfox the defaults by doubling
AGs though, at least at this point.

> of 16 for 70 GB /dev/sda2 partition, and eight per drive for total of
> 32 for /dev/sda3 partition (1011 GB).
> 
> I have aligned the partition start and end with the stripe width
> boundaries. The stripe size is 64 kB, stripe width is 4*64 kB = 256
> kB, in terms of 512 byte sectors:

> 
> 70 GB /
> 
> No      Start           End             Number
> 1       512             67109375        32 GB = 67108864 sectors = 131072 stripe sets
> 
> 2       67109376        213910015       70 GB = 146800640 sectors = 286720 stripe sets
> 
> 3       213910016       2335932415      Left = 2335932416 - 213910016 = 2122022400 sectors = 4144575 stripe sets
> 
> 
> When I do the following:
> 
> mkfs.xfs -f -b size=4k -d agcount=16,su=64k,sw=4 -i size=256,align=1,attr=2 -l version=2,su=64k,lazy-count=1 -n version=2 -s size=512 -L / /dev/sda2

You are restating many defaults here, I'm not sure why...

I would probably just drop the agcount specification and let mkfs do its own thing here; left to its own devices it will choose 4 AGs.

> Warning: AG size is a multiple of stripe width. This can cause
> performance problems by aligning all AGs on the same disk. To avoid
> this, run mkfs with an AG size that is one stripe unit smaller, for
> example 1146864

mkfs.xfs -f -b size=4k -d agsize=1146864b,su=64k,sw=4  ...

does work too, if you really want that many AGs for some reason:

meta-data=testfile               isize=256    agcount=16, agsize=1146864 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=18349824, imaxpct=25
         =                       sunit=16     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=8960, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Warning: AG size is a multiple of stripe width?

[Lachlan McIlroy]

----- Original Message -----
> Hi,
> 
> 
> I have a Sun workstation with eight Cheetah 15K.5 SAS 300 GB on RAID
> 1E (RAID 10) on LSI SAS3081E-R.
> 
> I am installing SLED 11 SP1 on it and I thought I will do a thorough
> optimization right down to the partition boundaries.
> 
> Since the default for XFS is to create four aggregation groups, and
> with the reasoning that Cheetah can do double the seeks of normal 7200
> RPM drives, I have four aggregation groups per drive for a total of 16
> for 70 GB /dev/sda2 partition, and eight per drive for total of 32 for
> /dev/sda3 partition (1011 GB).
> 
> I have aligned the partition start and end with the stripe width
> boundaries.
> The stripe size is 64 kB, stripe width is 4*64 kB = 256 kB, in terms
> of 512 byte sectors:
> 
> 70 GB /
> 
> No Start End Number
> 1 512 67109375 32 GB = 67108864 sectors = 131072 stripe sets
> 
> 2 67109376 213910015 70 GB = 146800640 sectors = 286720 stripe sets
> 
> 3 213910016 2335932415 Left = 2335932416 - 213910016 = 2122022400
> sectors = 4144575 stripe sets
> 
> 
> When I do the following:
> 
> mkfs.xfs -f -b size=4k -d agcount=16,su=64k,sw=4 -i
> size=256,align=1,attr=2 -l version=2,su=64k,lazy-count=1 -n version=2
> -s size=512 -L / /dev/sda2
> 
> Warning: AG size is a multiple of stripe width. This can cause
> performance problems by aligning all AGs on the same disk. To avoid
> this, run mkfs with an AG size that is one stripe unit smaller, for
> example 1146864
> 
> agcount=16 agsize=1146880 blks
> bsize=4096
> sunit=16 swidth=64 blks
> 
> mkfs.xfs -f -b size=4k -d agcount=32,su=64k,sw=4 -i
> size=256,align=1,attr=2 -l version=2,su=64k,lazy-count=1 -n version=2
> -s size=512 -L /home /dev/sda3
> 
> Warning: AG size is a multiple of stripe width. This can cause
> performance problems by aligning all AGs on the same disk. To avoid
> this, run mkfs with an AG size that is one stripe unit smaller, for
> example 8289136

Each allocation group has some headers that are updated frequently and if
the allocation group size is a multiple of the AG size then all of the AG
headers will land on the same disk.  This may cause congestion on this
disk and reduce the performance of the rest of the stripe.

> 
> agcount=32 agsize=8289152 blks
> bsize=4096
> sunit=16 swidth=64 blks
> 
> I am really puzzled since I thought all I am doing is distributing 4
> aggregation groups per drive for sda2 and 8 per drive for sda3.
> 
> What have I done wrong and what is the flaw with my understanding?

If you had concatenated the disks then your logic would make sense but
since you've striped the disks then all of the allocations groups will be
represented in part on each of the disks.  So if you have 32 AGs then each
disk will need to accommodate (part of) each of the 32 AGs instead of
the recommended 4 AGs which will probably make the disks seek a lot more.

> 
> Thank you!
> 
> 
> Chin Gim Leong
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs