agcount 33 by default for a single HDD?

agcount 33 by default for a single HDD?

[Chris Murphy]

This seems suboptimal. Basically this is a 750G thin volume. I don't
have a plain partition on a device handy to try this out but I'm
pretty certain the default is 4 AG's in that case, so I'm confused why
by default 33 AGs are created on a thin volume. The LVM volume group
is on a dmcrypt PV.


xfsprogs-4.15.1-1.fc28.x86_64
lvm2-2.02.177-5.fc28.x86_64
kernel-4.17.11-200.fc28.x86_64




[chris@f28s ~]$ sudo lvdisplay 4vg/nukeit
  --- Logical volume ---
  LV Path                /dev/4vg/nukeit
  LV Name                nukeit
  VG Name                4vg
  LV UUID                T21akN-RZeV-CfpZ-br70-6A1e-IkFI-VYOhNj
  LV Write Access        read/write
  LV Creation host, time f28s.local, 2018-07-31 19:28:21 -0600
  LV Pool name           4pool
  LV Status              available
  # open                 0
  LV Size                750.00 GiB
  Mapped size            0.00%
  Current LE             192000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:10

[chris@f28s ~]$ sudo mkfs.xfs /dev/mapper/4vg-nukeit
meta-data=/dev/mapper/4vg-nukeit isize=512    agcount=33, agsize=6143872 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0,
rmapbt=0, reflink=0
data     =                       bsize=4096   blocks=196608000, imaxpct=25
         =                       sunit=128    swidth=1024 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=96000, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[chris@f28s ~]$


-- 
Chris Murphy

Re: agcount 33 by default for a single HDD?

[Chris Murphy]

On Tue, Jul 31, 2018 at 7:32 PM, Chris Murphy <lists@colorremedies.com> wrote:
> This seems suboptimal. Basically this is a 750G thin volume. I don't
> have a plain partition on a device handy to try this out but I'm
> pretty certain the default is 4 AG's in that case, so I'm confused why
> by default 33 AGs are created on a thin volume. The LVM volume group
> is on a dmcrypt PV.
>
>
> xfsprogs-4.15.1-1.fc28.x86_64
> lvm2-2.02.177-5.fc28.x86_64
> kernel-4.17.11-200.fc28.x86_64
>
>
>          =                       sunit=128    swidth=1024 blks

This also seems screwy for a single non-raid device. I wonder if
there's something goofy happening with device mapper that's giving the
wrong information to XFS.

I just built xfsprogs 4.17 from git and I get different unexpected results:


[chris@f28s mkfs]$ sudo ./mkfs.xfs -V
mkfs.xfs version 4.17.0
[chris@f28s mkfs]$ sudo ./mkfs.xfs /dev/mapper/4vg-nukeit
mkfs.xfs: Use the -f option to force overwrite.
[chris@f28s mkfs]$ sudo ./mkfs.xfs -f /dev/mapper/4vg-nukeit
meta-data=/dev/mapper/4vg-nukeit isize=512    agcount=4, agsize=49152000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=196608000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=96000, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[chris@f28s mkfs]$



The agcount, sunit and swidth are what I expect, but sectsz=512 is
not. The device has a 4096 byte physical sector. And this is being
reported by 'lsblk' as a 4096 byte physical sector with 512 byte
logical sector (it is a 512e drive). The 4.15.0 Fedora built xfsprogs
got this right, but 4.17.0 as built is not.

:shrug:


-- 
Chris Murphy

Re: agcount 33 by default for a single HDD?

[Dave Chinner]

On Tue, Jul 31, 2018 at 07:32:50PM -0600, Chris Murphy wrote:
> This seems suboptimal.

It's actually a very useful optimisation to make on thin devices.

> Basically this is a 750G thin volume. I don't
> have a plain partition on a device handy to try this out but I'm
> pretty certain the default is 4 AG's in that case, so I'm confused why
> by default 33 AGs are created on a thin volume. The LVM volume group
> is on a dmcrypt PV.

It's a thin volume, therefore it advertises an optimal IO size and
alignment setting (i.e. the thin volume allocation chunk size).
Hence mkfs.xfs treats it as a "multi-disk device" and sets up
alignment and AG count appropriately.

This is actually the right optimisation to make for sparse devices -
more AGs increase filesystem concurrency but we normally restrict it
on single spindles because each AG adds more seeks into typical
workloads and slows them down. However, the thin volume already adds
that penalty to the storage stack for us because they don't have a
linear LBA-to-physical location characteristic.  Hence we can
increase filesystem concurrency without addition performance
penalties being incurred.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: agcount 33 by default for a single HDD?

[Chris Murphy]

On Tue, Jul 31, 2018 at 10:12 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Tue, Jul 31, 2018 at 07:32:50PM -0600, Chris Murphy wrote:
>> This seems suboptimal.
>
> It's actually a very useful optimisation to make on thin devices.
>
>> Basically this is a 750G thin volume. I don't
>> have a plain partition on a device handy to try this out but I'm
>> pretty certain the default is 4 AG's in that case, so I'm confused why
>> by default 33 AGs are created on a thin volume. The LVM volume group
>> is on a dmcrypt PV.
>
> It's a thin volume, therefore it advertises an optimal IO size and
> alignment setting (i.e. the thin volume allocation chunk size).
> Hence mkfs.xfs treats it as a "multi-disk device" and sets up
> alignment and AG count appropriately.
>
> This is actually the right optimisation to make for sparse devices -
> more AGs increase filesystem concurrency but we normally restrict it
> on single spindles because each AG adds more seeks into typical
> workloads and slows them down. However, the thin volume already adds
> that penalty to the storage stack for us because they don't have a
> linear LBA-to-physical location characteristic.  Hence we can
> increase filesystem concurrency without addition performance
> penalties being incurred.

OK so why 33 AG's with xfsprogs 4.15, but 4 AG's with xfsprogs 4.17,
when directed to the same thin LV? And also the difference in sunit
and swidth?

-- 
Chris Murphy

Re: agcount 33 by default for a single HDD?

[Eric Sandeen]

On 7/31/18 11:38 PM, Chris Murphy wrote:
> On Tue, Jul 31, 2018 at 10:12 PM, Dave Chinner <david@fromorbit.com> wrote:
>> On Tue, Jul 31, 2018 at 07:32:50PM -0600, Chris Murphy wrote:
>>> This seems suboptimal.
>>
>> It's actually a very useful optimisation to make on thin devices.
>>
>>> Basically this is a 750G thin volume. I don't
>>> have a plain partition on a device handy to try this out but I'm
>>> pretty certain the default is 4 AG's in that case, so I'm confused why
>>> by default 33 AGs are created on a thin volume. The LVM volume group
>>> is on a dmcrypt PV.
>>
>> It's a thin volume, therefore it advertises an optimal IO size and
>> alignment setting (i.e. the thin volume allocation chunk size).
>> Hence mkfs.xfs treats it as a "multi-disk device" and sets up
>> alignment and AG count appropriately.
>>
>> This is actually the right optimisation to make for sparse devices -
>> more AGs increase filesystem concurrency but we normally restrict it
>> on single spindles because each AG adds more seeks into typical
>> workloads and slows them down. However, the thin volume already adds
>> that penalty to the storage stack for us because they don't have a
>> linear LBA-to-physical location characteristic.  Hence we can
>> increase filesystem concurrency without addition performance
>> penalties being incurred.
> 
> OK so why 33 AG's with xfsprogs 4.15, but 4 AG's with xfsprogs 4.17,
> when directed to the same thin LV? And also the difference in sunit
> and swidth?
> 

Did you build 4.17 with --disable-blkid?

-Eric

Re: agcount 33 by default for a single HDD?

[Chris Murphy]

On Tue, Jul 31, 2018 at 10:41 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>
>
> On 7/31/18 11:38 PM, Chris Murphy wrote:
>> On Tue, Jul 31, 2018 at 10:12 PM, Dave Chinner <david@fromorbit.com> wrote:
>>> On Tue, Jul 31, 2018 at 07:32:50PM -0600, Chris Murphy wrote:
>>>> This seems suboptimal.
>>>
>>> It's actually a very useful optimisation to make on thin devices.
>>>
>>>> Basically this is a 750G thin volume. I don't
>>>> have a plain partition on a device handy to try this out but I'm
>>>> pretty certain the default is 4 AG's in that case, so I'm confused why
>>>> by default 33 AGs are created on a thin volume. The LVM volume group
>>>> is on a dmcrypt PV.
>>>
>>> It's a thin volume, therefore it advertises an optimal IO size and
>>> alignment setting (i.e. the thin volume allocation chunk size).
>>> Hence mkfs.xfs treats it as a "multi-disk device" and sets up
>>> alignment and AG count appropriately.
>>>
>>> This is actually the right optimisation to make for sparse devices -
>>> more AGs increase filesystem concurrency but we normally restrict it
>>> on single spindles because each AG adds more seeks into typical
>>> workloads and slows them down. However, the thin volume already adds
>>> that penalty to the storage stack for us because they don't have a
>>> linear LBA-to-physical location characteristic.  Hence we can
>>> increase filesystem concurrency without addition performance
>>> penalties being incurred.
>>
>> OK so why 33 AG's with xfsprogs 4.15, but 4 AG's with xfsprogs 4.17,
>> when directed to the same thin LV? And also the difference in sunit
>> and swidth?
>>
>
> Did you build 4.17 with --disable-blkid?

Yep, that's what doc/INSTALL recommends. Removing that and rebuilding,
sure enough 33 AGs, sunit 128, swidth 1024, and sectsz 4096. I do get
a build time warning.

Building mkfs
    [CC]     proto.o
    [CC]     xfs_mkfs.o
In function ‘finish_superblock_setup’,
    inlined from ‘main’ at xfs_mkfs.c:3937:2:
xfs_mkfs.c:3188:3: warning: ‘strncpy’ specified bound 12 equals
destination size [-Wstringop-truncation]
   strncpy(sbp->sb_fname, cfg->label, sizeof(sbp->sb_fname));
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    [LD]     mkfs.xfs



-- 
Chris Murphy

Re: agcount 33 by default for a single HDD?

[Eric Sandeen]

On 8/1/18 12:04 AM, Chris Murphy wrote:
> On Tue, Jul 31, 2018 at 10:41 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>>
>>
>> On 7/31/18 11:38 PM, Chris Murphy wrote:
>>> On Tue, Jul 31, 2018 at 10:12 PM, Dave Chinner <david@fromorbit.com> wrote:
>>>> On Tue, Jul 31, 2018 at 07:32:50PM -0600, Chris Murphy wrote:
>>>>> This seems suboptimal.
>>>>
>>>> It's actually a very useful optimisation to make on thin devices.
>>>>
>>>>> Basically this is a 750G thin volume. I don't
>>>>> have a plain partition on a device handy to try this out but I'm
>>>>> pretty certain the default is 4 AG's in that case, so I'm confused why
>>>>> by default 33 AGs are created on a thin volume. The LVM volume group
>>>>> is on a dmcrypt PV.
>>>>
>>>> It's a thin volume, therefore it advertises an optimal IO size and
>>>> alignment setting (i.e. the thin volume allocation chunk size).
>>>> Hence mkfs.xfs treats it as a "multi-disk device" and sets up
>>>> alignment and AG count appropriately.
>>>>
>>>> This is actually the right optimisation to make for sparse devices -
>>>> more AGs increase filesystem concurrency but we normally restrict it
>>>> on single spindles because each AG adds more seeks into typical
>>>> workloads and slows them down. However, the thin volume already adds
>>>> that penalty to the storage stack for us because they don't have a
>>>> linear LBA-to-physical location characteristic.  Hence we can
>>>> increase filesystem concurrency without addition performance
>>>> penalties being incurred.
>>>
>>> OK so why 33 AG's with xfsprogs 4.15, but 4 AG's with xfsprogs 4.17,
>>> when directed to the same thin LV? And also the difference in sunit
>>> and swidth?
>>>
>>
>> Did you build 4.17 with --disable-blkid?
> 
> Yep, that's what doc/INSTALL recommends. Removing that and rebuilding,
> sure enough 33 AGs, sunit 128, swidth 1024, and sectsz 4096. I do get
> a build time warning.
> 
> Building mkfs
>     [CC]     proto.o
>     [CC]     xfs_mkfs.o
> In function ‘finish_superblock_setup’,
>     inlined from ‘main’ at xfs_mkfs.c:3937:2:
> xfs_mkfs.c:3188:3: warning: ‘strncpy’ specified bound 12 equals
> destination size [-Wstringop-truncation]
>    strncpy(sbp->sb_fname, cfg->label, sizeof(sbp->sb_fname));

yeah, the on-disk label is only 12 bytes and is not null-terminated.

I guess we could change it to a memcpy or something.  o_O

-Eric

Re: agcount 33 by default for a single HDD?

[Carlos Maiolino]

On Tue, Jul 31, 2018 at 11:04:05PM -0600, Chris Murphy wrote:
> On Tue, Jul 31, 2018 at 10:41 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> >
> >
> > On 7/31/18 11:38 PM, Chris Murphy wrote:
> >> On Tue, Jul 31, 2018 at 10:12 PM, Dave Chinner <david@fromorbit.com> wrote:
> >>> On Tue, Jul 31, 2018 at 07:32:50PM -0600, Chris Murphy wrote:
> >>>> This seems suboptimal.
> >>>
> >>> It's actually a very useful optimisation to make on thin devices.
> >>>
> >>>> Basically this is a 750G thin volume. I don't
> >>>> have a plain partition on a device handy to try this out but I'm
> >>>> pretty certain the default is 4 AG's in that case, so I'm confused why
> >>>> by default 33 AGs are created on a thin volume. The LVM volume group
> >>>> is on a dmcrypt PV.
> >>>
> >>> It's a thin volume, therefore it advertises an optimal IO size and
> >>> alignment setting (i.e. the thin volume allocation chunk size).
> >>> Hence mkfs.xfs treats it as a "multi-disk device" and sets up
> >>> alignment and AG count appropriately.
> >>>
> >>> This is actually the right optimisation to make for sparse devices -
> >>> more AGs increase filesystem concurrency but we normally restrict it
> >>> on single spindles because each AG adds more seeks into typical
> >>> workloads and slows them down. However, the thin volume already adds
> >>> that penalty to the storage stack for us because they don't have a
> >>> linear LBA-to-physical location characteristic.  Hence we can
> >>> increase filesystem concurrency without addition performance
> >>> penalties being incurred.
> >>
> >> OK so why 33 AG's with xfsprogs 4.15, but 4 AG's with xfsprogs 4.17,
> >> when directed to the same thin LV? And also the difference in sunit
> >> and swidth?
> >>
> >
> > Did you build 4.17 with --disable-blkid?
> 
> Yep, that's what doc/INSTALL recommends. Removing that and rebuilding,
> sure enough 33 AGs, sunit 128, swidth 1024, and sectsz 4096. I do get
> a build time warning.

Where did you see that --disable-blkid is recommended? Are you using MacOS to
build it?

> 
> Building mkfs
>     [CC]     proto.o
>     [CC]     xfs_mkfs.o
> In function ‘finish_superblock_setup’,
>     inlined from ‘main’ at xfs_mkfs.c:3937:2:
> xfs_mkfs.c:3188:3: warning: ‘strncpy’ specified bound 12 equals
> destination size [-Wstringop-truncation]
>    strncpy(sbp->sb_fname, cfg->label, sizeof(sbp->sb_fname));
>    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>     [LD]     mkfs.xfs
> 
> 
> 
> -- 
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Carlos

Re: agcount 33 by default for a single HDD?

[Chris Murphy]

[-- Attachment #1: Type: text/plain, Size: 2415 bytes --]

On Wed, Aug 1, 2018, 5:45 AM Carlos Maiolino <cmaiolino@redhat.com> wrote:

> On Tue, Jul 31, 2018 at 11:04:05PM -0600, Chris Murphy wrote:
> > On Tue, Jul 31, 2018 at 10:41 PM, Eric Sandeen <sandeen@sandeen.net>
> wrote:
> > >
> > >
> > > On 7/31/18 11:38 PM, Chris Murphy wrote:
> > >> On Tue, Jul 31, 2018 at 10:12 PM, Dave Chinner <david@fromorbit.com>
> wrote:
> > >>> On Tue, Jul 31, 2018 at 07:32:50PM -0600, Chris Murphy wrote:
> > >>>> This seems suboptimal.
> > >>>
> > >>> It's actually a very useful optimisation to make on thin devices.
> > >>>
> > >>>> Basically this is a 750G thin volume. I don't
> > >>>> have a plain partition on a device handy to try this out but I'm
> > >>>> pretty certain the default is 4 AG's in that case, so I'm confused
> why
> > >>>> by default 33 AGs are created on a thin volume. The LVM volume group
> > >>>> is on a dmcrypt PV.
> > >>>
> > >>> It's a thin volume, therefore it advertises an optimal IO size and
> > >>> alignment setting (i.e. the thin volume allocation chunk size).
> > >>> Hence mkfs.xfs treats it as a "multi-disk device" and sets up
> > >>> alignment and AG count appropriately.
> > >>>
> > >>> This is actually the right optimisation to make for sparse devices -
> > >>> more AGs increase filesystem concurrency but we normally restrict it
> > >>> on single spindles because each AG adds more seeks into typical
> > >>> workloads and slows them down. However, the thin volume already adds
> > >>> that penalty to the storage stack for us because they don't have a
> > >>> linear LBA-to-physical location characteristic.  Hence we can
> > >>> increase filesystem concurrency without addition performance
> > >>> penalties being incurred.
> > >>
> > >> OK so why 33 AG's with xfsprogs 4.15, but 4 AG's with xfsprogs 4.17,
> > >> when directed to the same thin LV? And also the difference in sunit
> > >> and swidth?
> > >>
> > >
> > > Did you build 4.17 with --disable-blkid?
> >
> > Yep, that's what doc/INSTALL recommends. Removing that and rebuilding,
> > sure enough 33 AGs, sunit 128, swidth 1024, and sectsz 4096. I do get
> > a build time warning.
>
> Where did you see that --disable-blkid is recommended? Are you using MacOS
> to
> build it?
>

In the install instructions under the Mac OS X section. The Linux and Mac
OS X sections are obviously titled, nevertheless I totally missed those
titles.

Sorry for the noise.


Chris Murphy

>

[-- Attachment #2: Type: text/html, Size: 3577 bytes --]