Re: XFS Mount need ages

[Dave Chinner <david@fromorbit.com>]

On Fri, Jun 25, 2021 at 02:05:29PM +0200, Ml Ml wrote:
> After a loong time it mounted now. Here is some more info:
> 
> xfs_info /mnt/backup-cluster5
> meta-data=/dev/rbd6              isize=512    agcount=65536, agsize=32768 blks
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>          =                       sectsz=512   attr=2, projid32bit=1
>          =                       crc=1        finobt=1, sparse=1, rmapbt=0
>          =                       reflink=0
> data     =                       bsize=4096   blocks=2147483648, imaxpct=25
>          =                       sunit=16     swidth=16 blks
> naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
> log      =internal log           bsize=4096   blocks=2560, version=2
                                                ^^^^^^^^^^^
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote:
> >
> > Hello List,
> >
> > i have a rbd block device with xfs on it. After resizing it (from 6TB
> > to 8TB i think) the mount need hours to complete:
> >
> > I started the mount 15mins ago.:
> >   mount -nv /dev/rbd6 /mnt/backup-cluster5
> >
> > ps:
> > root      1143  0.2  0.0   8904  3088 pts/0    D+   12:17   0:03  |
> >    \_ mount -nv /dev/rbd6 /mnt/backup-cluster5
> >
> >
> > There is no timeout or ANY msg in dmesg until now.
> >
> > strace -p 1143  :  seems to do nothing.
> > iotop --pid=1143: uses about 50KB/sec
> >
> > dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec
> >
> >
> > Any idea what´s the problem here?

You started with a ~10GB sized filesystem (based on sunit, agsize
and log size) and have grown it almost 3 orders of magnitude. This
is way beyond recommendations - problems with fs layout tend to
expose themselves before the filesystem has been grown by a factor
of 10x, let alone ~1000x.

What I'd say is happening here is that if mount needed to iterate
all the AGs for some reason (e.g. an unclean shutdown), and it now
has to read all the AGF and AGI headers.  That's 130,000 IOs it now
needs to do.  At 100 IOPS, that's going to take 1300 seconds - a bit
over 20 minutes. And mount may have to do more than this (read AG
btrees to count blocks), so it could be doing thousands of IOs per
AG, which because of the sub-optimal layout, is multiplied by tens
of thousands of times...

Ceph RBDs ain't the fastest devices around, so this is very likely.

When you deploy a TB scale device, start with a TB-scale device, not
a device that is <10GB. Starting with a TB scale device will give
you TB-scale AGs, not 128MB sized AGs. Scanning a couple of dozen
AGs takes very little time, scanning tens of thousands of tiny AGs
takes a long time...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com