Re: XFS Mount need ages

["Darrick J. Wong" <djwong@kernel.org>]

On Sat, Jun 26, 2021 at 07:12:42AM +1000, Dave Chinner wrote:
> On Fri, Jun 25, 2021 at 02:05:29PM +0200, Ml Ml wrote:
> > After a loong time it mounted now. Here is some more info:
> > 
> > xfs_info /mnt/backup-cluster5
> > meta-data=/dev/rbd6              isize=512    agcount=65536, agsize=32768 blks
>                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >          =                       sectsz=512   attr=2, projid32bit=1
> >          =                       crc=1        finobt=1, sparse=1, rmapbt=0
> >          =                       reflink=0
> > data     =                       bsize=4096   blocks=2147483648, imaxpct=25
> >          =                       sunit=16     swidth=16 blks
> > naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
> > log      =internal log           bsize=4096   blocks=2560, version=2
>                                                 ^^^^^^^^^^^
> >          =                       sectsz=512   sunit=0 blks, lazy-count=1
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
> > 
> > On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote:
> > >
> > > Hello List,
> > >
> > > i have a rbd block device with xfs on it. After resizing it (from 6TB
> > > to 8TB i think) the mount need hours to complete:
> > >
> > > I started the mount 15mins ago.:
> > >   mount -nv /dev/rbd6 /mnt/backup-cluster5
> > >
> > > ps:
> > > root      1143  0.2  0.0   8904  3088 pts/0    D+   12:17   0:03  |
> > >    \_ mount -nv /dev/rbd6 /mnt/backup-cluster5
> > >
> > >
> > > There is no timeout or ANY msg in dmesg until now.
> > >
> > > strace -p 1143  :  seems to do nothing.
> > > iotop --pid=1143: uses about 50KB/sec
> > >
> > > dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec
> > >
> > >
> > > Any idea what´s the problem here?
> 
> You started with a ~10GB sized filesystem (based on sunit, agsize
> and log size) and have grown it almost 3 orders of magnitude. This
> is way beyond recommendations - problems with fs layout tend to
> expose themselves before the filesystem has been grown by a factor
> of 10x, let alone ~1000x.
> 
> What I'd say is happening here is that if mount needed to iterate
> all the AGs for some reason (e.g. an unclean shutdown), and it now
> has to read all the AGF and AGI headers.  That's 130,000 IOs it now
> needs to do.  At 100 IOPS, that's going to take 1300 seconds - a bit
> over 20 minutes. And mount may have to do more than this (read AG
> btrees to count blocks), so it could be doing thousands of IOs per
> AG, which because of the sub-optimal layout, is multiplied by tens
> of thousands of times...

finobt=1 and no mention of inobtcount, which means it's going to mount
verrrry slowly as it walks the free inode btree in each AG to figure out
how much space to reserve for that index.

--D

> 
> Ceph RBDs ain't the fastest devices around, so this is very likely.
> 
> When you deploy a TB scale device, start with a TB-scale device, not
> a device that is <10GB. Starting with a TB scale device will give
> you TB-scale AGs, not 128MB sized AGs. Scanning a couple of dozen
> AGs takes very little time, scanning tens of thousands of tiny AGs
> takes a long time...
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com