XFS Mount need ages

XFS Mount need ages

[Ml Ml]

Hello List,

i have a rbd block device with xfs on it. After resizing it (from 6TB
to 8TB i think) the mount need hours to complete:

I started the mount 15mins ago.:
  mount -nv /dev/rbd6 /mnt/backup-cluster5

ps:
root      1143  0.2  0.0   8904  3088 pts/0    D+   12:17   0:03  |
   \_ mount -nv /dev/rbd6 /mnt/backup-cluster5


There is no timeout or ANY msg in dmesg until now.

strace -p 1143  :  seems to do nothing.
iotop --pid=1143: uses about 50KB/sec

dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec


Any idea what´s the problem here?

Cheers,
Michael

Re: XFS Mount need ages

[Ml Ml]

After a loong time it mounted now. Here is some more info:

xfs_info /mnt/backup-cluster5
meta-data=/dev/rbd6              isize=512    agcount=65536, agsize=32768 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=2147483648, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote:
>
> Hello List,
>
> i have a rbd block device with xfs on it. After resizing it (from 6TB
> to 8TB i think) the mount need hours to complete:
>
> I started the mount 15mins ago.:
>   mount -nv /dev/rbd6 /mnt/backup-cluster5
>
> ps:
> root      1143  0.2  0.0   8904  3088 pts/0    D+   12:17   0:03  |
>    \_ mount -nv /dev/rbd6 /mnt/backup-cluster5
>
>
> There is no timeout or ANY msg in dmesg until now.
>
> strace -p 1143  :  seems to do nothing.
> iotop --pid=1143: uses about 50KB/sec
>
> dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec
>
>
> Any idea what´s the problem here?
>
> Cheers,
> Michael

Re: XFS Mount need ages

[Eric Sandeen]

What's in dmesg now? Was it doing log recovery? Which kernel version is this?
More info please.

Also for your dd speed test, please use iflag=direct to ensure you aren't
reading cached data.

-Eric

On 6/25/21 7:05 AM, Ml Ml wrote:
> After a loong time it mounted now. Here is some more info:
> 
> xfs_info /mnt/backup-cluster5
> meta-data=/dev/rbd6              isize=512    agcount=65536, agsize=32768 blks
>          =                       sectsz=512   attr=2, projid32bit=1
>          =                       crc=1        finobt=1, sparse=1, rmapbt=0
>          =                       reflink=0
> data     =                       bsize=4096   blocks=2147483648, imaxpct=25
>          =                       sunit=16     swidth=16 blks
> naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
> log      =internal log           bsize=4096   blocks=2560, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote:
>>
>> Hello List,
>>
>> i have a rbd block device with xfs on it. After resizing it (from 6TB
>> to 8TB i think) the mount need hours to complete:
>>
>> I started the mount 15mins ago.:
>>   mount -nv /dev/rbd6 /mnt/backup-cluster5
>>
>> ps:
>> root      1143  0.2  0.0   8904  3088 pts/0    D+   12:17   0:03  |
>>    \_ mount -nv /dev/rbd6 /mnt/backup-cluster5
>>
>>
>> There is no timeout or ANY msg in dmesg until now.
>>
>> strace -p 1143  :  seems to do nothing.
>> iotop --pid=1143: uses about 50KB/sec
>>
>> dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec
>>
>>
>> Any idea what´s the problem here?
>>
>> Cheers,
>> Michael
> 

Re: XFS Mount need ages

[Dave Chinner]

On Fri, Jun 25, 2021 at 02:05:29PM +0200, Ml Ml wrote:
> After a loong time it mounted now. Here is some more info:
> 
> xfs_info /mnt/backup-cluster5
> meta-data=/dev/rbd6              isize=512    agcount=65536, agsize=32768 blks
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>          =                       sectsz=512   attr=2, projid32bit=1
>          =                       crc=1        finobt=1, sparse=1, rmapbt=0
>          =                       reflink=0
> data     =                       bsize=4096   blocks=2147483648, imaxpct=25
>          =                       sunit=16     swidth=16 blks
> naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
> log      =internal log           bsize=4096   blocks=2560, version=2
                                                ^^^^^^^^^^^
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote:
> >
> > Hello List,
> >
> > i have a rbd block device with xfs on it. After resizing it (from 6TB
> > to 8TB i think) the mount need hours to complete:
> >
> > I started the mount 15mins ago.:
> >   mount -nv /dev/rbd6 /mnt/backup-cluster5
> >
> > ps:
> > root      1143  0.2  0.0   8904  3088 pts/0    D+   12:17   0:03  |
> >    \_ mount -nv /dev/rbd6 /mnt/backup-cluster5
> >
> >
> > There is no timeout or ANY msg in dmesg until now.
> >
> > strace -p 1143  :  seems to do nothing.
> > iotop --pid=1143: uses about 50KB/sec
> >
> > dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec
> >
> >
> > Any idea what´s the problem here?

You started with a ~10GB sized filesystem (based on sunit, agsize
and log size) and have grown it almost 3 orders of magnitude. This
is way beyond recommendations - problems with fs layout tend to
expose themselves before the filesystem has been grown by a factor
of 10x, let alone ~1000x.

What I'd say is happening here is that if mount needed to iterate
all the AGs for some reason (e.g. an unclean shutdown), and it now
has to read all the AGF and AGI headers.  That's 130,000 IOs it now
needs to do.  At 100 IOPS, that's going to take 1300 seconds - a bit
over 20 minutes. And mount may have to do more than this (read AG
btrees to count blocks), so it could be doing thousands of IOs per
AG, which because of the sub-optimal layout, is multiplied by tens
of thousands of times...

Ceph RBDs ain't the fastest devices around, so this is very likely.

When you deploy a TB scale device, start with a TB-scale device, not
a device that is <10GB. Starting with a TB scale device will give
you TB-scale AGs, not 128MB sized AGs. Scanning a couple of dozen
AGs takes very little time, scanning tens of thousands of tiny AGs
takes a long time...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS Mount need ages

[Darrick J. Wong]

On Sat, Jun 26, 2021 at 07:12:42AM +1000, Dave Chinner wrote:
> On Fri, Jun 25, 2021 at 02:05:29PM +0200, Ml Ml wrote:
> > After a loong time it mounted now. Here is some more info:
> > 
> > xfs_info /mnt/backup-cluster5
> > meta-data=/dev/rbd6              isize=512    agcount=65536, agsize=32768 blks
>                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >          =                       sectsz=512   attr=2, projid32bit=1
> >          =                       crc=1        finobt=1, sparse=1, rmapbt=0
> >          =                       reflink=0
> > data     =                       bsize=4096   blocks=2147483648, imaxpct=25
> >          =                       sunit=16     swidth=16 blks
> > naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
> > log      =internal log           bsize=4096   blocks=2560, version=2
>                                                 ^^^^^^^^^^^
> >          =                       sectsz=512   sunit=0 blks, lazy-count=1
> > realtime =none                   extsz=4096   blocks=0, rtextents=0
> > 
> > On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote:
> > >
> > > Hello List,
> > >
> > > i have a rbd block device with xfs on it. After resizing it (from 6TB
> > > to 8TB i think) the mount need hours to complete:
> > >
> > > I started the mount 15mins ago.:
> > >   mount -nv /dev/rbd6 /mnt/backup-cluster5
> > >
> > > ps:
> > > root      1143  0.2  0.0   8904  3088 pts/0    D+   12:17   0:03  |
> > >    \_ mount -nv /dev/rbd6 /mnt/backup-cluster5
> > >
> > >
> > > There is no timeout or ANY msg in dmesg until now.
> > >
> > > strace -p 1143  :  seems to do nothing.
> > > iotop --pid=1143: uses about 50KB/sec
> > >
> > > dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec
> > >
> > >
> > > Any idea what´s the problem here?
> 
> You started with a ~10GB sized filesystem (based on sunit, agsize
> and log size) and have grown it almost 3 orders of magnitude. This
> is way beyond recommendations - problems with fs layout tend to
> expose themselves before the filesystem has been grown by a factor
> of 10x, let alone ~1000x.
> 
> What I'd say is happening here is that if mount needed to iterate
> all the AGs for some reason (e.g. an unclean shutdown), and it now
> has to read all the AGF and AGI headers.  That's 130,000 IOs it now
> needs to do.  At 100 IOPS, that's going to take 1300 seconds - a bit
> over 20 minutes. And mount may have to do more than this (read AG
> btrees to count blocks), so it could be doing thousands of IOs per
> AG, which because of the sub-optimal layout, is multiplied by tens
> of thousands of times...

finobt=1 and no mention of inobtcount, which means it's going to mount
verrrry slowly as it walks the free inode btree in each AG to figure out
how much space to reserve for that index.

--D

> 
> Ceph RBDs ain't the fastest devices around, so this is very likely.
> 
> When you deploy a TB scale device, start with a TB-scale device, not
> a device that is <10GB. Starting with a TB scale device will give
> you TB-scale AGs, not 128MB sized AGs. Scanning a couple of dozen
> AGs takes very little time, scanning tens of thousands of tiny AGs
> takes a long time...
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com