equal size not large enough for RAID1?

equal size not large enough for RAID1?

[Lars Täuber]

Hallo list,

I'm new to this list. But I came across some strange problems I nowhere found a solution for on the net.
This is our situation:

monosan + duosan:
Linux duosan 2.6.22.17-0.1-default #1 SMP 2008/02/10 20:01:04 UTC x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
[...]
processor       : 3
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 65
model name      : Dual-Core AMD Opteron(tm) Processor 2216
stepping        : 3
cpu MHz         : 2412.402
cache size      : 1024 KB
physical id     : 1
siblings        : 2
core id         : 1
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips        : 4825.08
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

SAS controller:
07:00.0 Ethernet controller: MYRICOM Inc. Myri-10G Dual-Protocol NIC (10G-PCIE-8A)


There are 16 SATA drives multipathed connected to each server.
duosan exports its RAID 6 over AoE via qaoed
monosan sees this as etherd/e22.0

monosan# cat /proc/partitions:
   9     4 12697912448 md4
   9     9 12697912312 md9
 152  5632 12697912448 etherd/e22.0


monosan:~ # cat /proc/mdstat 
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] 
md9 : active raid1 md4[0]
      12697912312 blocks super 1.0 [2/1] [U_]
      
md4 : active raid6 dm-0[0] dm-8[14] dm-7[13] dm-6[12] dm-5[11] dm-4[10] dm-3[9] dm-2[8] dm-14[7] dm-13[6] dm-12[5] dm-11[4] dm-10[3] dm-9[2] dm-1[1]
      12697912448 blocks level 6, 64k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]



But then this:
monosan:~ # mdadm /dev/md9 -a /dev/etherd/e22.0 
mdadm: /dev/etherd/e22.0 not large enough to join array

The md9 RAID1 was originally built with e22.0 as second drive. I just simulated a connection loss.

Why is this happening? As you can see md9 consists of md4 and e22.0 and both are equal in size: 12697912448 .
How can I debug this? Are there detailed logs anywhere?

Many thanks for any hint.
Lars



-- 
                            Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstrasse 22-23                     10117 Berlin
Tel.: +49 30 20370-352           http://www.bbaw.de
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: equal size not large enough for RAID1?

[Lars Täuber]

Hi Neil,

Neil Brown <neilb@suse.de> schrieb:
> On Wednesday March 5, taeuber@bbaw.de wrote:
> > Hallo list,
> 
> Hi.
> 
> > 
> > monosan# cat /proc/partitions:
> >    9     4 12697912448 md4
> >    9     9 12697912312 md9
> >  152  5632 12697912448 etherd/e22.0
> > 
> > 
> > monosan:~ # cat /proc/mdstat 
> > Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] 
> > md9 : active raid1 md4[0]
> >       12697912312 blocks super 1.0 [2/1] [U_]
> >       
> > md4 : active raid6 dm-0[0] dm-8[14] dm-7[13] dm-6[12] dm-5[11] dm-4[10] dm-3[9] dm-2[8] dm-14[7] dm-13[6] dm-12[5] dm-11[4] dm-10[3] dm-9[2] dm-1[1]
> >       12697912448 blocks level 6, 64k chunk, algorithm 2 [15/15] [UUUUUUUUUUUUUUU]
> > 
> > 
> > 
> > But then this:
> > monosan:~ # mdadm /dev/md9 -a /dev/etherd/e22.0 
> > mdadm: /dev/etherd/e22.0 not large enough to join array
> 
> This would be because mdadm is reserving a bit of space in case you
> want to add an internal bitmap one day.  And the version of mdadm you
> are now using is reserving more space than the version that was used
> to create the array does.

we are using SuSE 10.3 here. And I don't remember if mdadm was updated due to an online update. But I think it should be consistent/compatible with the version shipped in the original distribution.

What is this internal bitmap good for? Is there a documentation somewhere on the net about this?


> 
> I should fix that...
> 
> If you want a quick fix and are happy to compile your own mdadm, then
> edit super1.c and remove the line
> 
> 	devsize -= choose_bm_space(devsize);
> 
> in avail_size1().

I could compile, but on our productive server we don't compile anything that is shipped with the distribution due to the possibility to have security updates from SuSE.

I just repeated the following:

monosan:~ # mdadm -C /dev/md9 -l1 -n2 -x0 /dev/md4 /dev/etherd/e22.0 
mdadm: Defaulting to verion 1.0 metadata
mdadm: /dev/md4 appears to be part of a raid array:
    level=raid1 devices=2 ctime=Thu Mar  6 09:39:37 2008
mdadm: /dev/etherd/e22.0 appears to be part of a raid array:
    level=raid1 devices=2 ctime=Thu Mar  6 09:39:37 2008
Continue creating array? y
mdadm: array /dev/md9 started.

monosan:~ # mdadm -V
mdadm - v2.6.2 - 21st May 2007

monosan:~ # mdadm /dev/md9 -f /dev/etherd/e22.0 
mdadm: set /dev/etherd/e22.0 faulty in /dev/md9

monosan:~ # mdadm /dev/md9 -r /dev/etherd/e22.0 
mdadm: hot removed /dev/etherd/e22.0

monosan:~ # mdadm /dev/md9 -a /dev/etherd/e22.0 
mdadm: /dev/etherd/e22.0 not large enough to join array

monosan:~ # cat /etc/SuSE-release 
openSUSE 10.3 (X86-64)
VERSION = 10.3

How come?

Thanks
Lars
-- 
                            Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstrasse 22-23                     10117 Berlin
Tel.: +49 30 20370-352           http://www.bbaw.de
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: equal size not large enough for RAID1?

[Neil Brown]

On Thursday March 6, taeuber@bbaw.de wrote:
> 
> What is this internal bitmap good for? Is there a documentation
> somewhere on the net about this? 

  man md
and search for BITMAP.  There is a section of BITMAP WRITE-INTENT
LOGGING.

> 
> I just repeated the following:
> 
> monosan:~ # mdadm -C /dev/md9 -l1 -n2 -x0 /dev/md4 /dev/etherd/e22.0 
> mdadm: Defaulting to verion 1.0 metadata
> mdadm: /dev/md4 appears to be part of a raid array:
>     level=raid1 devices=2 ctime=Thu Mar  6 09:39:37 2008
> mdadm: /dev/etherd/e22.0 appears to be part of a raid array:
>     level=raid1 devices=2 ctime=Thu Mar  6 09:39:37 2008
> Continue creating array? y
> mdadm: array /dev/md9 started.
> 
> monosan:~ # mdadm -V
> mdadm - v2.6.2 - 21st May 2007
> 
> monosan:~ # mdadm /dev/md9 -f /dev/etherd/e22.0 
> mdadm: set /dev/etherd/e22.0 faulty in /dev/md9
> 
> monosan:~ # mdadm /dev/md9 -r /dev/etherd/e22.0 
> mdadm: hot removed /dev/etherd/e22.0
> 
> monosan:~ # mdadm /dev/md9 -a /dev/etherd/e22.0 
> mdadm: /dev/etherd/e22.0 not large enough to join array

Hmm..  I just read through the change logs and I see there was a bug
here that was fixed in 2.6.3.
If you want an update from SuSE you would need to log a bug report in
the Novell bugzilla.

Alternately request v0.90 metadata (--metadata=0.90)  That will avoid
the bug.

NeilBrown

Re: equal size not large enough for RAID1?

[Lars Täuber]

Hallo Neil,

> Hmm..  I just read through the change logs and I see there was a bug
> here that was fixed in 2.6.3.
> If you want an update from SuSE you would need to log a bug report in
> the Novell bugzilla.
> 
> Alternately request v0.90 metadata (--metadata=0.90)  That will avoid
> the bug.
> 
> NeilBrown

thank you for this info. I'll try to get SuSE to correct this.
I just created a bugzilla account.

Thanks again.

Lars