* mdadm "hang", 100% CPU usage when trying to create RAID-1 array with external bitmap
@ 2010-07-14 10:53 Tomasz Chmielewski
2010-07-15 5:45 ` Neil Brown
0 siblings, 1 reply; 6+ messages in thread
From: Tomasz Chmielewski @ 2010-07-14 10:53 UTC (permalink / raw)
To: linux-raid
I'm trying to create RAID-1 with external bitmap with this command:
mdadm --create /dev/md11 --level=1 --bitmap=/mnt/src/bitmap --raid-devices 2 /dev/storage/origin missing
/dev/storage/origin is 1.2 TB; is placed on "stacked" block devices like this:
RAID-5 -> dm-crypt -> LVM-2 -> /dev/storage/origin
/mnt/src/bitmap is technically the same stacked device, except the last LVM volume:
RAID-5 -> dm-crypt -> LVM-2 -> /dev/storage/src -> /mnt/src/bitmap
(although I tried to keep bitmap outside of this RAID-5 device, and it was still hanging).
When started with "strace", mdadm hangs with 100% CPU usage after opening /mnt/src/bitmap:
(...)
rename("/var/run/map.new", "/var/run/map") = 0
flock(3, LOCK_UN) = 0
close(3) = 0
munmap(0x6ff40000, 4096) = 0
unlink("/var/run/map.lock") = 0
ioctl(4, 0x800c0910, 0x77f5f89c) = 0
fstat64(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(9, 1), ...}) = 0
fstat64(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(9, 1), ...}) = 0
ioctl(4, 0x800c0910, 0x77f5f7cc) = 0
ioctl(4, 0x40480923, 0x77f5f854) = 0
access("/mnt/src/bitmap", F_OK) = -1 ENOENT (No such file or directory)
open("/mnt/src/bitmap", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
I see no disk activity when this happens.
# mdadm -V
mdadm - v3.1.2 - 10th March 2010
Kernel is 2.6.34.1 (tried also 2.6.28.7), 32 bit x86 system.
Or, I didn't wait long enough? So far, it takes about an hour to create this array...
Creating smaller arrays (i.e. 8 GB) in a similar configuration works fine.
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm "hang", 100% CPU usage when trying to create RAID-1 array with external bitmap
2010-07-14 10:53 mdadm "hang", 100% CPU usage when trying to create RAID-1 array with external bitmap Tomasz Chmielewski
@ 2010-07-15 5:45 ` Neil Brown
2010-07-15 10:35 ` Tomasz Chmielewski
2010-07-18 19:09 ` John Stoffel
0 siblings, 2 replies; 6+ messages in thread
From: Neil Brown @ 2010-07-15 5:45 UTC (permalink / raw)
To: Tomasz Chmielewski; +Cc: linux-raid
On Wed, 14 Jul 2010 12:53:45 +0200
Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> I'm trying to create RAID-1 with external bitmap with this command:
>
> mdadm --create /dev/md11 --level=1 --bitmap=/mnt/src/bitmap --raid-devices 2 /dev/storage/origin missing
>
>
> /dev/storage/origin is 1.2 TB; is placed on "stacked" block devices like this:
>
> RAID-5 -> dm-crypt -> LVM-2 -> /dev/storage/origin
>
>
> /mnt/src/bitmap is technically the same stacked device, except the last LVM volume:
>
> RAID-5 -> dm-crypt -> LVM-2 -> /dev/storage/src -> /mnt/src/bitmap
>
> (although I tried to keep bitmap outside of this RAID-5 device, and it was still hanging).
>
>
> When started with "strace", mdadm hangs with 100% CPU usage after opening /mnt/src/bitmap:
>
> (...)
> rename("/var/run/map.new", "/var/run/map") = 0
> flock(3, LOCK_UN) = 0
> close(3) = 0
> munmap(0x6ff40000, 4096) = 0
> unlink("/var/run/map.lock") = 0
> ioctl(4, 0x800c0910, 0x77f5f89c) = 0
> fstat64(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(9, 1), ...}) = 0
> fstat64(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(9, 1), ...}) = 0
> ioctl(4, 0x800c0910, 0x77f5f7cc) = 0
> ioctl(4, 0x40480923, 0x77f5f854) = 0
> access("/mnt/src/bitmap", F_OK) = -1 ENOENT (No such file or directory)
> open("/mnt/src/bitmap", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
>
>
Hmm... that's rather embarrassing.
This patch should fit it. Or you can explicitly set a bitmap chunk size -
aim for several megabyte.
Thanks for the report,
NeilBrown
diff --git a/bitmap.c b/bitmap.c
index beef2dc..44a8677 100644
--- a/bitmap.c
+++ b/bitmap.c
@@ -373,7 +373,7 @@ int CreateBitmap(char *filename, int force, char uuid[16],
*/
chunksize = DEFAULT_BITMAP_CHUNK;
/* <<20 for 2^20 chunks, >>9 to convert bytes to sectors */
- while (array_size > (chunksize << (20-9)))
+ while (array_size > ((unsigned long long)chunksize << (20-9)))
chunksize <<= 1;
}
> I see no disk activity when this happens.
>
>
> # mdadm -V
> mdadm - v3.1.2 - 10th March 2010
>
> Kernel is 2.6.34.1 (tried also 2.6.28.7), 32 bit x86 system.
>
> Or, I didn't wait long enough? So far, it takes about an hour to create this array...
>
> Creating smaller arrays (i.e. 8 GB) in a similar configuration works fine.
>
>
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: mdadm "hang", 100% CPU usage when trying to create RAID-1 array with external bitmap
2010-07-15 5:45 ` Neil Brown
@ 2010-07-15 10:35 ` Tomasz Chmielewski
2010-07-18 19:09 ` John Stoffel
1 sibling, 0 replies; 6+ messages in thread
From: Tomasz Chmielewski @ 2010-07-15 10:35 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On 15.07.2010 07:45, Neil Brown wrote:
(...)
>> access("/mnt/src/bitmap", F_OK) = -1 ENOENT (No such file or directory)
>> open("/mnt/src/bitmap", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
>>
>>
>
> Hmm... that's rather embarrassing.
> This patch should fit it. Or you can explicitly set a bitmap chunk size -
> aim for several megabyte.
>
> Thanks for the report,
> NeilBrown
>
> diff --git a/bitmap.c b/bitmap.c
> index beef2dc..44a8677 100644
> --- a/bitmap.c
> +++ b/bitmap.c
> @@ -373,7 +373,7 @@ int CreateBitmap(char *filename, int force, char uuid[16],
> */
> chunksize = DEFAULT_BITMAP_CHUNK;
> /*<<20 for 2^20 chunks,>>9 to convert bytes to sectors */
> - while (array_size> (chunksize<< (20-9)))
> + while (array_size> ((unsigned long long)chunksize<< (20-9)))
It works fine with that patch applied, thanks a lot!
--
Tomasz Chmielewski
http://syneticon.net
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm "hang", 100% CPU usage when trying to create RAID-1 array with external bitmap
2010-07-15 5:45 ` Neil Brown
2010-07-15 10:35 ` Tomasz Chmielewski
@ 2010-07-18 19:09 ` John Stoffel
2010-07-19 23:26 ` Neil Brown
1 sibling, 1 reply; 6+ messages in thread
From: John Stoffel @ 2010-07-18 19:09 UTC (permalink / raw)
To: Neil Brown; +Cc: Tomasz Chmielewski, linux-raid
>>>>> "Neil" == Neil Brown <neilb@suse.de> writes:
Neil> On Wed, 14 Jul 2010 12:53:45 +0200
Neil> Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>> I'm trying to create RAID-1 with external bitmap with this command:
>>
>> mdadm --create /dev/md11 --level=1 --bitmap=/mnt/src/bitmap --raid-devices 2 /dev/storage/origin missing
>>
>>
>> /dev/storage/origin is 1.2 TB; is placed on "stacked" block devices like this:
>>
>> RAID-5 -> dm-crypt -> LVM-2 -> /dev/storage/origin
>>
>>
>> /mnt/src/bitmap is technically the same stacked device, except the last LVM volume:
>>
>> RAID-5 -> dm-crypt -> LVM-2 -> /dev/storage/src -> /mnt/src/bitmap
>>
>> (although I tried to keep bitmap outside of this RAID-5 device, and it was still hanging).
>>
>>
>> When started with "strace", mdadm hangs with 100% CPU usage after opening /mnt/src/bitmap:
>>
>> (...)
>> rename("/var/run/map.new", "/var/run/map") = 0
>> flock(3, LOCK_UN) = 0
>> close(3) = 0
>> munmap(0x6ff40000, 4096) = 0
>> unlink("/var/run/map.lock") = 0
>> ioctl(4, 0x800c0910, 0x77f5f89c) = 0
>> fstat64(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(9, 1), ...}) = 0
>> fstat64(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(9, 1), ...}) = 0
>> ioctl(4, 0x800c0910, 0x77f5f7cc) = 0
>> ioctl(4, 0x40480923, 0x77f5f854) = 0
>> access("/mnt/src/bitmap", F_OK) = -1 ENOENT (No such file or directory)
>> open("/mnt/src/bitmap", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
>>
>>
Neil> Hmm... that's rather embarrassing.
Neil> This patch should fit it. Or you can explicitly set a bitmap chunk size -
Neil> aim for several megabyte.
Neil> diff --git a/bitmap.c b/bitmap.c
Neil> index beef2dc..44a8677 100644
Neil> --- a/bitmap.c
Neil> +++ b/bitmap.c
Neil> @@ -373,7 +373,7 @@ int CreateBitmap(char *filename, int force, char uuid[16],
Neil> */
Neil> chunksize = DEFAULT_BITMAP_CHUNK;
Neil> /* <<20 for 2^20 chunks, >>9 to convert bytes to sectors */
Neil> - while (array_size > (chunksize << (20-9)))
Neil> + while (array_size > ((unsigned long long)chunksize << (20-9)))
Neil> chunksize <<= 1;
Neil> }
How about also replacing the magic numbers 20 and 9 with usefully
named constants as well? I realize the comment tells you what's
going on here, but I just boggle a bit when I see 20-9 in the source.
:-)
John
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm "hang", 100% CPU usage when trying to create RAID-1 array with external bitmap
2010-07-18 19:09 ` John Stoffel
@ 2010-07-19 23:26 ` Neil Brown
2010-07-20 14:14 ` John Stoffel
0 siblings, 1 reply; 6+ messages in thread
From: Neil Brown @ 2010-07-19 23:26 UTC (permalink / raw)
To: John Stoffel; +Cc: Tomasz Chmielewski, linux-raid
On Sun, 18 Jul 2010 15:09:38 -0400
"John Stoffel" <john@stoffel.org> wrote:
> >>>>> "Neil" == Neil Brown <neilb@suse.de> writes:
>
> Neil> On Wed, 14 Jul 2010 12:53:45 +0200
> Neil> Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>
> >> I'm trying to create RAID-1 with external bitmap with this command:
> >>
> >> mdadm --create /dev/md11 --level=1 --bitmap=/mnt/src/bitmap --raid-devices 2 /dev/storage/origin missing
> >>
> >>
> >> /dev/storage/origin is 1.2 TB; is placed on "stacked" block devices like this:
> >>
> >> RAID-5 -> dm-crypt -> LVM-2 -> /dev/storage/origin
> >>
> >>
> >> /mnt/src/bitmap is technically the same stacked device, except the last LVM volume:
> >>
> >> RAID-5 -> dm-crypt -> LVM-2 -> /dev/storage/src -> /mnt/src/bitmap
> >>
> >> (although I tried to keep bitmap outside of this RAID-5 device, and it was still hanging).
> >>
> >>
> >> When started with "strace", mdadm hangs with 100% CPU usage after opening /mnt/src/bitmap:
> >>
> >> (...)
> >> rename("/var/run/map.new", "/var/run/map") = 0
> >> flock(3, LOCK_UN) = 0
> >> close(3) = 0
> >> munmap(0x6ff40000, 4096) = 0
> >> unlink("/var/run/map.lock") = 0
> >> ioctl(4, 0x800c0910, 0x77f5f89c) = 0
> >> fstat64(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(9, 1), ...}) = 0
> >> fstat64(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(9, 1), ...}) = 0
> >> ioctl(4, 0x800c0910, 0x77f5f7cc) = 0
> >> ioctl(4, 0x40480923, 0x77f5f854) = 0
> >> access("/mnt/src/bitmap", F_OK) = -1 ENOENT (No such file or directory)
> >> open("/mnt/src/bitmap", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
> >>
> >>
>
> Neil> Hmm... that's rather embarrassing.
> Neil> This patch should fit it. Or you can explicitly set a bitmap chunk size -
> Neil> aim for several megabyte.
>
> Neil> diff --git a/bitmap.c b/bitmap.c
> Neil> index beef2dc..44a8677 100644
> Neil> --- a/bitmap.c
> Neil> +++ b/bitmap.c
> Neil> @@ -373,7 +373,7 @@ int CreateBitmap(char *filename, int force, char uuid[16],
> Neil> */
> Neil> chunksize = DEFAULT_BITMAP_CHUNK;
> Neil> /* <<20 for 2^20 chunks, >>9 to convert bytes to sectors */
> Neil> - while (array_size > (chunksize << (20-9)))
> Neil> + while (array_size > ((unsigned long long)chunksize << (20-9)))
> Neil> chunksize <<= 1;
> Neil> }
>
> How about also replacing the magic numbers 20 and 9 with usefully
> named constants as well? I realize the comment tells you what's
> going on here, but I just boggle a bit when I see 20-9 in the source.
> :-)
If you sent that suggestion as a patch it would be much more likely to be
applied (it might anyway, but still...)
:-)
NeilBrown
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mdadm "hang", 100% CPU usage when trying to create RAID-1 array with external bitmap
2010-07-19 23:26 ` Neil Brown
@ 2010-07-20 14:14 ` John Stoffel
0 siblings, 0 replies; 6+ messages in thread
From: John Stoffel @ 2010-07-20 14:14 UTC (permalink / raw)
To: Neil Brown; +Cc: John Stoffel, Tomasz Chmielewski, linux-raid
>>>>> "Neil" == Neil Brown <neilb@suse.de> writes:
Neil> If you sent that suggestion as a patch it would be much more likely to be
Neil> applied (it might anyway, but still...)
Neil> :-)
touche! I'll see about whipping up a patch today/tonight. Thanks for
taking my nit-picking so well....
John
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-07-20 14:14 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-07-14 10:53 mdadm "hang", 100% CPU usage when trying to create RAID-1 array with external bitmap Tomasz Chmielewski
2010-07-15 5:45 ` Neil Brown
2010-07-15 10:35 ` Tomasz Chmielewski
2010-07-18 19:09 ` John Stoffel
2010-07-19 23:26 ` Neil Brown
2010-07-20 14:14 ` John Stoffel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).