* XFS Mount need ages @ 2021-06-25 10:49 Ml Ml 2021-06-25 12:05 ` Ml Ml 0 siblings, 1 reply; 5+ messages in thread From: Ml Ml @ 2021-06-25 10:49 UTC (permalink / raw) To: linux-xfs Hello List, i have a rbd block device with xfs on it. After resizing it (from 6TB to 8TB i think) the mount need hours to complete: I started the mount 15mins ago.: mount -nv /dev/rbd6 /mnt/backup-cluster5 ps: root 1143 0.2 0.0 8904 3088 pts/0 D+ 12:17 0:03 | \_ mount -nv /dev/rbd6 /mnt/backup-cluster5 There is no timeout or ANY msg in dmesg until now. strace -p 1143 : seems to do nothing. iotop --pid=1143: uses about 50KB/sec dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec Any idea what´s the problem here? Cheers, Michael ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: XFS Mount need ages 2021-06-25 10:49 XFS Mount need ages Ml Ml @ 2021-06-25 12:05 ` Ml Ml 2021-06-25 14:55 ` Eric Sandeen 2021-06-25 21:12 ` Dave Chinner 0 siblings, 2 replies; 5+ messages in thread From: Ml Ml @ 2021-06-25 12:05 UTC (permalink / raw) To: linux-xfs After a loong time it mounted now. Here is some more info: xfs_info /mnt/backup-cluster5 meta-data=/dev/rbd6 isize=512 agcount=65536, agsize=32768 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 data = bsize=4096 blocks=2147483648, imaxpct=25 = sunit=16 swidth=16 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote: > > Hello List, > > i have a rbd block device with xfs on it. After resizing it (from 6TB > to 8TB i think) the mount need hours to complete: > > I started the mount 15mins ago.: > mount -nv /dev/rbd6 /mnt/backup-cluster5 > > ps: > root 1143 0.2 0.0 8904 3088 pts/0 D+ 12:17 0:03 | > \_ mount -nv /dev/rbd6 /mnt/backup-cluster5 > > > There is no timeout or ANY msg in dmesg until now. > > strace -p 1143 : seems to do nothing. > iotop --pid=1143: uses about 50KB/sec > > dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec > > > Any idea what´s the problem here? > > Cheers, > Michael ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: XFS Mount need ages 2021-06-25 12:05 ` Ml Ml @ 2021-06-25 14:55 ` Eric Sandeen 2021-06-25 21:12 ` Dave Chinner 1 sibling, 0 replies; 5+ messages in thread From: Eric Sandeen @ 2021-06-25 14:55 UTC (permalink / raw) To: Ml Ml, linux-xfs What's in dmesg now? Was it doing log recovery? Which kernel version is this? More info please. Also for your dd speed test, please use iflag=direct to ensure you aren't reading cached data. -Eric On 6/25/21 7:05 AM, Ml Ml wrote: > After a loong time it mounted now. Here is some more info: > > xfs_info /mnt/backup-cluster5 > meta-data=/dev/rbd6 isize=512 agcount=65536, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=1, sparse=1, rmapbt=0 > = reflink=0 > data = bsize=4096 blocks=2147483648, imaxpct=25 > = sunit=16 swidth=16 blks > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote: >> >> Hello List, >> >> i have a rbd block device with xfs on it. After resizing it (from 6TB >> to 8TB i think) the mount need hours to complete: >> >> I started the mount 15mins ago.: >> mount -nv /dev/rbd6 /mnt/backup-cluster5 >> >> ps: >> root 1143 0.2 0.0 8904 3088 pts/0 D+ 12:17 0:03 | >> \_ mount -nv /dev/rbd6 /mnt/backup-cluster5 >> >> >> There is no timeout or ANY msg in dmesg until now. >> >> strace -p 1143 : seems to do nothing. >> iotop --pid=1143: uses about 50KB/sec >> >> dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec >> >> >> Any idea what´s the problem here? >> >> Cheers, >> Michael > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: XFS Mount need ages 2021-06-25 12:05 ` Ml Ml 2021-06-25 14:55 ` Eric Sandeen @ 2021-06-25 21:12 ` Dave Chinner 2021-06-25 21:27 ` Darrick J. Wong 1 sibling, 1 reply; 5+ messages in thread From: Dave Chinner @ 2021-06-25 21:12 UTC (permalink / raw) To: Ml Ml; +Cc: linux-xfs On Fri, Jun 25, 2021 at 02:05:29PM +0200, Ml Ml wrote: > After a loong time it mounted now. Here is some more info: > > xfs_info /mnt/backup-cluster5 > meta-data=/dev/rbd6 isize=512 agcount=65536, agsize=32768 blks ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=1, sparse=1, rmapbt=0 > = reflink=0 > data = bsize=4096 blocks=2147483648, imaxpct=25 > = sunit=16 swidth=16 blks > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > log =internal log bsize=4096 blocks=2560, version=2 ^^^^^^^^^^^ > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote: > > > > Hello List, > > > > i have a rbd block device with xfs on it. After resizing it (from 6TB > > to 8TB i think) the mount need hours to complete: > > > > I started the mount 15mins ago.: > > mount -nv /dev/rbd6 /mnt/backup-cluster5 > > > > ps: > > root 1143 0.2 0.0 8904 3088 pts/0 D+ 12:17 0:03 | > > \_ mount -nv /dev/rbd6 /mnt/backup-cluster5 > > > > > > There is no timeout or ANY msg in dmesg until now. > > > > strace -p 1143 : seems to do nothing. > > iotop --pid=1143: uses about 50KB/sec > > > > dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec > > > > > > Any idea what´s the problem here? You started with a ~10GB sized filesystem (based on sunit, agsize and log size) and have grown it almost 3 orders of magnitude. This is way beyond recommendations - problems with fs layout tend to expose themselves before the filesystem has been grown by a factor of 10x, let alone ~1000x. What I'd say is happening here is that if mount needed to iterate all the AGs for some reason (e.g. an unclean shutdown), and it now has to read all the AGF and AGI headers. That's 130,000 IOs it now needs to do. At 100 IOPS, that's going to take 1300 seconds - a bit over 20 minutes. And mount may have to do more than this (read AG btrees to count blocks), so it could be doing thousands of IOs per AG, which because of the sub-optimal layout, is multiplied by tens of thousands of times... Ceph RBDs ain't the fastest devices around, so this is very likely. When you deploy a TB scale device, start with a TB-scale device, not a device that is <10GB. Starting with a TB scale device will give you TB-scale AGs, not 128MB sized AGs. Scanning a couple of dozen AGs takes very little time, scanning tens of thousands of tiny AGs takes a long time... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: XFS Mount need ages 2021-06-25 21:12 ` Dave Chinner @ 2021-06-25 21:27 ` Darrick J. Wong 0 siblings, 0 replies; 5+ messages in thread From: Darrick J. Wong @ 2021-06-25 21:27 UTC (permalink / raw) To: Dave Chinner; +Cc: Ml Ml, linux-xfs On Sat, Jun 26, 2021 at 07:12:42AM +1000, Dave Chinner wrote: > On Fri, Jun 25, 2021 at 02:05:29PM +0200, Ml Ml wrote: > > After a loong time it mounted now. Here is some more info: > > > > xfs_info /mnt/backup-cluster5 > > meta-data=/dev/rbd6 isize=512 agcount=65536, agsize=32768 blks > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > = sectsz=512 attr=2, projid32bit=1 > > = crc=1 finobt=1, sparse=1, rmapbt=0 > > = reflink=0 > > data = bsize=4096 blocks=2147483648, imaxpct=25 > > = sunit=16 swidth=16 blks > > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > > log =internal log bsize=4096 blocks=2560, version=2 > ^^^^^^^^^^^ > > = sectsz=512 sunit=0 blks, lazy-count=1 > > realtime =none extsz=4096 blocks=0, rtextents=0 > > > > On Fri, Jun 25, 2021 at 12:49 PM Ml Ml <mliebherr99@googlemail.com> wrote: > > > > > > Hello List, > > > > > > i have a rbd block device with xfs on it. After resizing it (from 6TB > > > to 8TB i think) the mount need hours to complete: > > > > > > I started the mount 15mins ago.: > > > mount -nv /dev/rbd6 /mnt/backup-cluster5 > > > > > > ps: > > > root 1143 0.2 0.0 8904 3088 pts/0 D+ 12:17 0:03 | > > > \_ mount -nv /dev/rbd6 /mnt/backup-cluster5 > > > > > > > > > There is no timeout or ANY msg in dmesg until now. > > > > > > strace -p 1143 : seems to do nothing. > > > iotop --pid=1143: uses about 50KB/sec > > > > > > dd bs=1M count=2048 if=/dev/rbd6 of=/dev/null => gives me 50MB/sec > > > > > > > > > Any idea what´s the problem here? > > You started with a ~10GB sized filesystem (based on sunit, agsize > and log size) and have grown it almost 3 orders of magnitude. This > is way beyond recommendations - problems with fs layout tend to > expose themselves before the filesystem has been grown by a factor > of 10x, let alone ~1000x. > > What I'd say is happening here is that if mount needed to iterate > all the AGs for some reason (e.g. an unclean shutdown), and it now > has to read all the AGF and AGI headers. That's 130,000 IOs it now > needs to do. At 100 IOPS, that's going to take 1300 seconds - a bit > over 20 minutes. And mount may have to do more than this (read AG > btrees to count blocks), so it could be doing thousands of IOs per > AG, which because of the sub-optimal layout, is multiplied by tens > of thousands of times... finobt=1 and no mention of inobtcount, which means it's going to mount verrrry slowly as it walks the free inode btree in each AG to figure out how much space to reserve for that index. --D > > Ceph RBDs ain't the fastest devices around, so this is very likely. > > When you deploy a TB scale device, start with a TB-scale device, not > a device that is <10GB. Starting with a TB scale device will give > you TB-scale AGs, not 128MB sized AGs. Scanning a couple of dozen > AGs takes very little time, scanning tens of thousands of tiny AGs > takes a long time... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-06-25 21:27 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-06-25 10:49 XFS Mount need ages Ml Ml 2021-06-25 12:05 ` Ml Ml 2021-06-25 14:55 ` Eric Sandeen 2021-06-25 21:12 ` Dave Chinner 2021-06-25 21:27 ` Darrick J. Wong
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox