* mkfs.xfs fails with raid5 and smaller chunk sizes
@ 2014-09-16 22:03 Brian Hemme
2014-09-16 22:17 ` Dave Chinner
0 siblings, 1 reply; 5+ messages in thread
From: Brian Hemme @ 2014-09-16 22:03 UTC (permalink / raw)
To: xfs
Hello all,
I am having some odd problems with mkfs.xfs when used on a raid 5
array. The array is built from 6 960GB SSDs all connected to SATA ports
on the MB and created with mdadm. If I use a chunk size any smaller
then 512K mkfs.xfs just hangs forever. It continues to use CPU and so
does the raid array but never completes. If the system is just left
running for an extended length of time the whole OS eventually locks
up. I have tried this on three different systems with the same
results. I have searched all over for someone with similar issues
without success. I am hoping I am just doing something clearly wrong
and you all can set me straight quickly.
Some specifics:
Arch linux with 3.14.1 kernel
mkfs.xfs version 3.1.11
mdadm - v3.3 - 3rd September 2013
Commands:
> mdadm --create /dev/md0 --chunk=64K --level=5 --raid-devices=6
/dev/sd[a-f]
> mkfs.xfs /dev/md0
** This command fails and locks up
I have tried specifying the arguments to mkfs.xfs with the same
results. Building a 4 drive array seems to require a chunk size of 1M
or greater to work. Same results if I make a partition on the array and
make the fs there.
Any help would be appreciated
Thanks
Brian
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: mkfs.xfs fails with raid5 and smaller chunk sizes 2014-09-16 22:03 mkfs.xfs fails with raid5 and smaller chunk sizes Brian Hemme @ 2014-09-16 22:17 ` Dave Chinner 2014-09-16 22:47 ` Brian Hemme 0 siblings, 1 reply; 5+ messages in thread From: Dave Chinner @ 2014-09-16 22:17 UTC (permalink / raw) To: Brian Hemme; +Cc: xfs On Tue, Sep 16, 2014 at 03:03:08PM -0700, Brian Hemme wrote: > Hello all, > > I am having some odd problems with mkfs.xfs when used on a raid 5 > array. The array is built from 6 960GB SSDs all connected to SATA > ports on the MB and created with mdadm. If I use a chunk size any > smaller then 512K mkfs.xfs just hangs forever. It continues to use > CPU and so does the raid array but never completes. If the system > is just left running for an extended length of time the whole OS > eventually locks up. I have tried this on three different systems > with the same results. I have searched all over for someone with > similar issues without success. I am hoping I am just doing > something clearly wrong and you all can set me straight quickly. > > Some specifics: > Arch linux with 3.14.1 kernel > mkfs.xfs version 3.1.11 > mdadm - v3.3 - 3rd September 2013 > > Commands: > > mdadm --create /dev/md0 --chunk=64K --level=5 --raid-devices=6 > /dev/sd[a-f] > > mkfs.xfs /dev/md0 > ** This command fails and locks up > > I have tried specifying the arguments to mkfs.xfs with the same > results. Building a 4 drive array seems to require a chunk size of > 1M or greater to work. Same results if I make a partition on the > array and make the fs there. mkfs.xfs really should only take a couple of seconds to complete. Seeing as you are using SSDs, my first suspicion is that md or the SSDs are having problems with discard. Hence you should first try 'mkfs.xfs -K /dev/md0' and see if that completes quickly. Otherwise, output of 'echo w > sysrq-trigger' from dmesg would be a good start, as would a 'perf top -G -U' snapshot (run for 30s at least a minute after mkfs.xfs starts) to tell us what is burning CPU. Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mkfs.xfs fails with raid5 and smaller chunk sizes 2014-09-16 22:17 ` Dave Chinner @ 2014-09-16 22:47 ` Brian Hemme 2014-09-17 6:02 ` Dave Chinner 0 siblings, 1 reply; 5+ messages in thread From: Brian Hemme @ 2014-09-16 22:47 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs On 09/16/2014 03:17 PM, Dave Chinner wrote: > On Tue, Sep 16, 2014 at 03:03:08PM -0700, Brian Hemme wrote: >> Hello all, >> >> I am having some odd problems with mkfs.xfs when used on a raid 5 >> array. The array is built from 6 960GB SSDs all connected to SATA >> ports on the MB and created with mdadm. If I use a chunk size any >> smaller then 512K mkfs.xfs just hangs forever. It continues to use >> CPU and so does the raid array but never completes. If the system >> is just left running for an extended length of time the whole OS >> eventually locks up. I have tried this on three different systems >> with the same results. I have searched all over for someone with >> similar issues without success. I am hoping I am just doing >> something clearly wrong and you all can set me straight quickly. >> >> Some specifics: >> Arch linux with 3.14.1 kernel >> mkfs.xfs version 3.1.11 >> mdadm - v3.3 - 3rd September 2013 >> >> Commands: >>> mdadm --create /dev/md0 --chunk=64K --level=5 --raid-devices=6 >> /dev/sd[a-f] >>> mkfs.xfs /dev/md0 >> ** This command fails and locks up >> >> I have tried specifying the arguments to mkfs.xfs with the same >> results. Building a 4 drive array seems to require a chunk size of >> 1M or greater to work. Same results if I make a partition on the >> array and make the fs there. > mkfs.xfs really should only take a couple of seconds to complete. > Seeing as you are using SSDs, my first suspicion is that md or the > SSDs are having problems with discard. Hence you should first > try 'mkfs.xfs -K /dev/md0' and see if that completes quickly. > > Otherwise, output of 'echo w> sysrq-trigger' from dmesg would be a > good start, as would a 'perf top -G -U' snapshot (run for 30s at > least a minute after mkfs.xfs starts) to tell us what is burning > CPU. > > Cheers, > > Dave. Thanks for the quick response! Adding the -K seemed to do the trick. However, for my education, why is this needed in this case? It seems to work without it for larger chunk sizes or for raid 0 instead of 5. It also worked on our old install with a 3.1.6 kernel. Any why would not using the -K cause enough of a problem that the whole machine hangs? Just trying to understand this enough to make sure I don't run into problems down the road. Thanks again, Brian _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mkfs.xfs fails with raid5 and smaller chunk sizes 2014-09-16 22:47 ` Brian Hemme @ 2014-09-17 6:02 ` Dave Chinner 2014-09-17 15:06 ` Brian Hemme 0 siblings, 1 reply; 5+ messages in thread From: Dave Chinner @ 2014-09-17 6:02 UTC (permalink / raw) To: Brian Hemme; +Cc: xfs On Tue, Sep 16, 2014 at 03:47:43PM -0700, Brian Hemme wrote: > On 09/16/2014 03:17 PM, Dave Chinner wrote: > >On Tue, Sep 16, 2014 at 03:03:08PM -0700, Brian Hemme wrote: > >>Hello all, > >> > >>I am having some odd problems with mkfs.xfs when used on a raid 5 > >>array. The array is built from 6 960GB SSDs all connected to SATA > >>ports on the MB and created with mdadm. If I use a chunk size any > >>smaller then 512K mkfs.xfs just hangs forever. It continues to use > >>CPU and so does the raid array but never completes. If the system > >>is just left running for an extended length of time the whole OS > >>eventually locks up. I have tried this on three different systems > >>with the same results. I have searched all over for someone with > >>similar issues without success. I am hoping I am just doing > >>something clearly wrong and you all can set me straight quickly. > >> > >>Some specifics: > >> Arch linux with 3.14.1 kernel > >> mkfs.xfs version 3.1.11 > >> mdadm - v3.3 - 3rd September 2013 > >> > >>Commands: > >>>mdadm --create /dev/md0 --chunk=64K --level=5 --raid-devices=6 > >>/dev/sd[a-f] > >>>mkfs.xfs /dev/md0 > >> ** This command fails and locks up > >> > >>I have tried specifying the arguments to mkfs.xfs with the same > >>results. Building a 4 drive array seems to require a chunk size of > >>1M or greater to work. Same results if I make a partition on the > >>array and make the fs there. > >mkfs.xfs really should only take a couple of seconds to complete. > >Seeing as you are using SSDs, my first suspicion is that md or the > >SSDs are having problems with discard. Hence you should first > >try 'mkfs.xfs -K /dev/md0' and see if that completes quickly. > > > >Otherwise, output of 'echo w> sysrq-trigger' from dmesg would be a > >good start, as would a 'perf top -G -U' snapshot (run for 30s at > >least a minute after mkfs.xfs starts) to tell us what is burning > >CPU. > > > >Cheers, > > > >Dave. > > Thanks for the quick response! > > Adding the -K seemed to do the trick. However, for my education, > why is this needed in this case? It seems to work without it for > larger chunk sizes or for raid 0 instead of 5. Discard on RAID 5 can require parity recalculation if the discard sizes are small which means RMW operations. I'd say you probably need to ask the linux-raid list to debug whatever issue you are having with the RAID5 code. > It also worked on > our old install with a 3.1.6 kernel. RAID5 discard support was added in 3.1.7.... > Any why would not using the -K > cause enough of a problem that the whole machine hangs? Just trying > to understand this enough to make sure I don't run into problems > down the road. If you cause the IO subsystem to choke up, the system can hang because it can't clean dirty pages of memory and hence you can get ENOMEM situations that can hang the machine. Again, i'd first talk to the linux-raid folk to find out what is causing the RAID5 to be so slow in this case as it's really nothing to do with XFS at this point... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mkfs.xfs fails with raid5 and smaller chunk sizes 2014-09-17 6:02 ` Dave Chinner @ 2014-09-17 15:06 ` Brian Hemme 0 siblings, 0 replies; 5+ messages in thread From: Brian Hemme @ 2014-09-17 15:06 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs On 09/16/2014 11:02 PM, Dave Chinner wrote: > On Tue, Sep 16, 2014 at 03:47:43PM -0700, Brian Hemme wrote: >> On 09/16/2014 03:17 PM, Dave Chinner wrote: >>> On Tue, Sep 16, 2014 at 03:03:08PM -0700, Brian Hemme wrote: >>>> Hello all, >>>> >>>> I am having some odd problems with mkfs.xfs when used on a raid 5 >>>> array. The array is built from 6 960GB SSDs all connected to SATA >>>> ports on the MB and created with mdadm. If I use a chunk size any >>>> smaller then 512K mkfs.xfs just hangs forever. It continues to use >>>> CPU and so does the raid array but never completes. If the system >>>> is just left running for an extended length of time the whole OS >>>> eventually locks up. I have tried this on three different systems >>>> with the same results. I have searched all over for someone with >>>> similar issues without success. I am hoping I am just doing >>>> something clearly wrong and you all can set me straight quickly. >>>> >>>> Some specifics: >>>> Arch linux with 3.14.1 kernel >>>> mkfs.xfs version 3.1.11 >>>> mdadm - v3.3 - 3rd September 2013 >>>> >>>> Commands: >>>>> mdadm --create /dev/md0 --chunk=64K --level=5 --raid-devices=6 >>>> /dev/sd[a-f] >>>>> mkfs.xfs /dev/md0 >>>> ** This command fails and locks up >>>> >>>> I have tried specifying the arguments to mkfs.xfs with the same >>>> results. Building a 4 drive array seems to require a chunk size of >>>> 1M or greater to work. Same results if I make a partition on the >>>> array and make the fs there. >>> mkfs.xfs really should only take a couple of seconds to complete. >>> Seeing as you are using SSDs, my first suspicion is that md or the >>> SSDs are having problems with discard. Hence you should first >>> try 'mkfs.xfs -K /dev/md0' and see if that completes quickly. >>> >>> Otherwise, output of 'echo w> sysrq-trigger' from dmesg would be a >>> good start, as would a 'perf top -G -U' snapshot (run for 30s at >>> least a minute after mkfs.xfs starts) to tell us what is burning >>> CPU. >>> >>> Cheers, >>> >>> Dave. >> Thanks for the quick response! >> >> Adding the -K seemed to do the trick. However, for my education, >> why is this needed in this case? It seems to work without it for >> larger chunk sizes or for raid 0 instead of 5. > Discard on RAID 5 can require parity recalculation if the discard > sizes are small which means RMW operations. I'd say you probably > need to ask the linux-raid list to debug whatever issue you are > having with the RAID5 code. > >> It also worked on >> our old install with a 3.1.6 kernel. > RAID5 discard support was added in 3.1.7.... > >> Any why would not using the -K >> cause enough of a problem that the whole machine hangs? Just trying >> to understand this enough to make sure I don't run into problems >> down the road. > If you cause the IO subsystem to choke up, the system can hang > because it can't clean dirty pages of memory and hence you can get > ENOMEM situations that can hang the machine. Again, i'd first talk > to the linux-raid folk to find out what is causing the RAID5 to be > so slow in this case as it's really nothing to do with XFS at this > point... > > Cheers, > > Dave. Thanks very much Dave. You got me everything I needed. Brian _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-09-17 15:07 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-09-16 22:03 mkfs.xfs fails with raid5 and smaller chunk sizes Brian Hemme 2014-09-16 22:17 ` Dave Chinner 2014-09-16 22:47 ` Brian Hemme 2014-09-17 6:02 ` Dave Chinner 2014-09-17 15:06 ` Brian Hemme
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox