* Array of disks attached to multiple controllers
@ 2004-09-14 13:21 Lukas Kubin
2004-09-14 13:57 ` Guy
0 siblings, 1 reply; 7+ messages in thread
From: Lukas Kubin @ 2004-09-14 13:21 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1.1: Type: text/plain, Size: 810 bytes --]
I'm having troubles creating Raid-5 array consisted of 16 (+1 spare)
disks attached to two 3ware (not same version) SATA controllers.
Mkraid creates the array without problems, however the array's total
size only equals the sum of space on drives attached to the first of 2
controllers. Ie, sd{b..f} are on scsi0 and sd{g..r} on scsi1. The space
of created array is close to the sum of drives attechted to scsi0 only.
Is it possible to combine drives on 2 controllers into one array? If
not, what solution should I use to have one filesystem containing whole
the drivespace on Raid-5?
Thank you.
lukas
--
Lukas Kubin
phone: +420596398275
email: kubin@opf.slu.cz
Information centre
The School of Business Administration in Karvina
Silesian University in Opava
Czech Republic
http://www.opf.slu.cz
[-- Attachment #1.2: kubin.vcf --]
[-- Type: text/x-vcard, Size: 329 bytes --]
begin:vcard
fn:Lukas Kubin
n:Kubin;Lukas
org:The School of Business Administration in Karvina;Information centre
adr:;;Univerzitni namesti 1934/3;Karvina;;73340;Czech Republic
email;internet:kubin@opf.slu.cz
title:Network administrator
tel;work:+420596398275
x-mozilla-html:FALSE
url:http://www.opf.slu.cz
version:2.1
end:vcard
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 2257 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread* RE: Array of disks attached to multiple controllers
2004-09-14 13:21 Array of disks attached to multiple controllers Lukas Kubin
@ 2004-09-14 13:57 ` Guy
2004-09-14 14:07 ` Lukas Kubin
0 siblings, 1 reply; 7+ messages in thread
From: Guy @ 2004-09-14 13:57 UTC (permalink / raw)
To: 'Lukas Kubin', linux-raid
Give us more info.
Output from mdadm -D /dev/md?
Guy
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Lukas Kubin
Sent: Tuesday, September 14, 2004 9:22 AM
To: linux-raid@vger.kernel.org
Subject: Array of disks attached to multiple controllers
I'm having troubles creating Raid-5 array consisted of 16 (+1 spare)
disks attached to two 3ware (not same version) SATA controllers.
Mkraid creates the array without problems, however the array's total
size only equals the sum of space on drives attached to the first of 2
controllers. Ie, sd{b..f} are on scsi0 and sd{g..r} on scsi1. The space
of created array is close to the sum of drives attechted to scsi0 only.
Is it possible to combine drives on 2 controllers into one array? If
not, what solution should I use to have one filesystem containing whole
the drivespace on Raid-5?
Thank you.
lukas
--
Lukas Kubin
phone: +420596398275
email: kubin@opf.slu.cz
Information centre
The School of Business Administration in Karvina
Silesian University in Opava
Czech Republic
http://www.opf.slu.cz
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Array of disks attached to multiple controllers
2004-09-14 13:57 ` Guy
@ 2004-09-14 14:07 ` Lukas Kubin
2004-09-14 14:46 ` Sebastien Koechlin
2004-09-14 14:59 ` Guy
0 siblings, 2 replies; 7+ messages in thread
From: Lukas Kubin @ 2004-09-14 14:07 UTC (permalink / raw)
To: Guy; +Cc: linux-raid
[-- Attachment #1.1: Type: text/plain, Size: 3284 bytes --]
Ok, the output follows:
=====
/dev/md0:
Version : 00.90.00
Creation Time : Tue Sep 14 16:02:42 2004
Raid Level : raid5
Array Size : 782929792 (746.66 GiB 801.72 GB)
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 16
Total Devices : 17
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Sep 14 16:02:42 2004
State : dirty, recovering
Active Devices : 16
Working Devices : 17
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 128K
Rebuild Status : 0% complete
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
3 8 64 3 active sync /dev/sde
4 8 80 4 active sync /dev/sdf
5 8 96 5 active sync /dev/sdg
6 8 112 6 active sync /dev/sdh
7 8 128 7 active sync /dev/sdi
8 8 144 8 active sync /dev/sdj
9 8 160 9 active sync /dev/sdk
10 8 176 10 active sync /dev/sdl
11 8 192 11 active sync /dev/sdm
12 8 208 12 active sync /dev/sdn
13 8 224 13 active sync /dev/sdo
14 8 240 14 active sync /dev/sdp
15 65 0 15 active sync /dev/sdq
16 65 16 16 spare /dev/sdr
UUID : c18d1ba9:d7d4f338:6d83a4e1:023cbbdb
Events : 0.1
=====
All the disks are SATA 200GB, the first controller (for drives sd{a..f})
is 3ware 8xxx, the second is 3ware 9000 ATA Raid controller. The first
controller uses 3w-xxxx kernel module, the second one 3x-9xxx on linux
2.4.27.
Thank you for any help.
lukas
Guy wrote:
> Give us more info.
> Output from mdadm -D /dev/md?
>
> Guy
>
>
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Lukas Kubin
> Sent: Tuesday, September 14, 2004 9:22 AM
> To: linux-raid@vger.kernel.org
> Subject: Array of disks attached to multiple controllers
>
> I'm having troubles creating Raid-5 array consisted of 16 (+1 spare)
> disks attached to two 3ware (not same version) SATA controllers.
>
> Mkraid creates the array without problems, however the array's total
> size only equals the sum of space on drives attached to the first of 2
> controllers. Ie, sd{b..f} are on scsi0 and sd{g..r} on scsi1. The space
> of created array is close to the sum of drives attechted to scsi0 only.
>
> Is it possible to combine drives on 2 controllers into one array? If
> not, what solution should I use to have one filesystem containing whole
> the drivespace on Raid-5?
>
> Thank you.
>
> lukas
>
--
Lukas Kubin
phone: +420596398275
email: kubin@opf.slu.cz
Information centre
The School of Business Administration in Karvina
Silesian University in Opava
Czech Republic
http://www.opf.slu.cz
[-- Attachment #1.2: kubin.vcf --]
[-- Type: text/x-vcard, Size: 329 bytes --]
begin:vcard
fn:Lukas Kubin
n:Kubin;Lukas
org:The School of Business Administration in Karvina;Information centre
adr:;;Univerzitni namesti 1934/3;Karvina;;73340;Czech Republic
email;internet:kubin@opf.slu.cz
title:Network administrator
tel;work:+420596398275
x-mozilla-html:FALSE
url:http://www.opf.slu.cz
version:2.1
end:vcard
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 2257 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Array of disks attached to multiple controllers
2004-09-14 14:07 ` Lukas Kubin
@ 2004-09-14 14:46 ` Sebastien Koechlin
2004-09-14 15:22 ` Guy
2004-09-14 14:59 ` Guy
1 sibling, 1 reply; 7+ messages in thread
From: Sebastien Koechlin @ 2004-09-14 14:46 UTC (permalink / raw)
To: Lukas Kubin; +Cc: Guy, linux-raid
On Tue, Sep 14, 2004 at 04:07:43PM +0200, Lukas Kubin wrote:
> Chunk Size : 128K
(...)
> Number Major Minor RaidDevice State
> 0 8 16 0 active sync /dev/sdb
> 1 8 32 1 active sync /dev/sdc
> 2 8 48 2 active sync /dev/sdd
> 3 8 64 3 active sync /dev/sde
> 4 8 80 4 active sync /dev/sdf
> 5 8 96 5 active sync /dev/sdg
> 6 8 112 6 active sync /dev/sdh
> 7 8 128 7 active sync /dev/sdi
> 8 8 144 8 active sync /dev/sdj
> 9 8 160 9 active sync /dev/sdk
> 10 8 176 10 active sync /dev/sdl
> 11 8 192 11 active sync /dev/sdm
> 12 8 208 12 active sync /dev/sdn
> 13 8 224 13 active sync /dev/sdo
> 14 8 240 14 active sync /dev/sdp
> 15 65 0 15 active sync /dev/sdq
> 16 65 16 16 spare /dev/sdr
I have a question about performance: What is the cost of writing a
'data-unit' in such an array?
- Write data: 1 write
- Calculate new checksum: 14 reads
- Write checksum: 1 write
Right or wrong?
What is the granularity of thoses ckecksum updates? 512 bytes (sector size)?
4k (page size on i386)? chunk-size?
Does Linux do read-ahead on thoses 14 disks reads?
Thanks
--
Seb, autocuiseur
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: Array of disks attached to multiple controllers
2004-09-14 14:46 ` Sebastien Koechlin
@ 2004-09-14 15:22 ` Guy
0 siblings, 0 replies; 7+ messages in thread
From: Guy @ 2004-09-14 15:22 UTC (permalink / raw)
To: 'Sebastien Koechlin', 'Lukas Kubin'; +Cc: linux-raid
I know this one! Got to go from memory since I can't find Neil's answer.
md will do 4K disk I/O. I am not sure about the size, but much smaller than
normal stripe size.
If you were to change 1 sector, md would read the 4K that contains the
sector, and read the 4K of parity. Factor out the old data, factor in the
new data. Then write the new 4K data and 4K parity.
- Get old data: 1 read (4K)
- Get current/old parity: 1 read (4K)
- factor out old data (XOR)
- compute new parity (XOR)
- Write data: 1 write (4k)
- Write parity: 1 write (4k)
There is some read-ahead, but I don't know what layer does that.
My array is over 3 times as fast as a single disk. Using dd I get 19M/sec
on any 1 disk, but 61M/sec on the array. My write speed is 30M/sec creating
a file on a filesystem.
Multi-threaded random read speed is much faster than 1 disk. Maybe 14 times
faster, but not sure.
Guy
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Sebastien Koechlin
Sent: Tuesday, September 14, 2004 10:46 AM
To: Lukas Kubin
Cc: Guy; linux-raid@vger.kernel.org
Subject: Re: Array of disks attached to multiple controllers
On Tue, Sep 14, 2004 at 04:07:43PM +0200, Lukas Kubin wrote:
> Chunk Size : 128K
(...)
> Number Major Minor RaidDevice State
> 0 8 16 0 active sync /dev/sdb
> 1 8 32 1 active sync /dev/sdc
> 2 8 48 2 active sync /dev/sdd
> 3 8 64 3 active sync /dev/sde
> 4 8 80 4 active sync /dev/sdf
> 5 8 96 5 active sync /dev/sdg
> 6 8 112 6 active sync /dev/sdh
> 7 8 128 7 active sync /dev/sdi
> 8 8 144 8 active sync /dev/sdj
> 9 8 160 9 active sync /dev/sdk
> 10 8 176 10 active sync /dev/sdl
> 11 8 192 11 active sync /dev/sdm
> 12 8 208 12 active sync /dev/sdn
> 13 8 224 13 active sync /dev/sdo
> 14 8 240 14 active sync /dev/sdp
> 15 65 0 15 active sync /dev/sdq
> 16 65 16 16 spare /dev/sdr
I have a question about performance: What is the cost of writing a
'data-unit' in such an array?
- Write data: 1 write
- Calculate new checksum: 14 reads
- Write checksum: 1 write
Right or wrong?
What is the granularity of thoses ckecksum updates? 512 bytes (sector size)?
4k (page size on i386)? chunk-size?
Does Linux do read-ahead on thoses 14 disks reads?
Thanks
--
Seb, autocuiseur
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* RE: Array of disks attached to multiple controllers
2004-09-14 14:07 ` Lukas Kubin
2004-09-14 14:46 ` Sebastien Koechlin
@ 2004-09-14 14:59 ` Guy
2004-09-14 20:31 ` Neil Brown
1 sibling, 1 reply; 7+ messages in thread
From: Guy @ 2004-09-14 14:59 UTC (permalink / raw)
To: 'Lukas Kubin'; +Cc: linux-raid
Well I have a clue!
<Device size> X (<Number of disks> - 1)
195360896 X (16-1) = 2930413440 (a very nice number!)
I think you have crossed a 2T limit.
Subtract 2T from the expected size and you get you size.
2930413440 - 2^31 = 782929792 (This is your size!)
2^31 = 2T. (we have 1K blocks)
You have exceeded a 2T limit somewhere. I don't know where the limit is. I
thought I read that devices were limited to 2T, but arrays could be larger.
Maybe you have an older kernel that only supports 2T. My 14 disk monster
array is just over 200Gig, so I have not had this problem. A new kernel may
help, but I don't know. Time for Neil to help! :)
I think you will be fine if you only use 11 disks!
195360896 * (11-1) = 1953608960 (this is under the 2T limit).
Guy
-----Original Message-----
From: Lukas Kubin [mailto:kubin@opf.slu.cz]
Sent: Tuesday, September 14, 2004 10:08 AM
To: Guy
Cc: linux-raid@vger.kernel.org
Subject: Re: Array of disks attached to multiple controllers
Ok, the output follows:
=====
/dev/md0:
Version : 00.90.00
Creation Time : Tue Sep 14 16:02:42 2004
Raid Level : raid5
Array Size : 782929792 (746.66 GiB 801.72 GB)
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 16
Total Devices : 17
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Sep 14 16:02:42 2004
State : dirty, recovering
Active Devices : 16
Working Devices : 17
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 128K
Rebuild Status : 0% complete
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
3 8 64 3 active sync /dev/sde
4 8 80 4 active sync /dev/sdf
5 8 96 5 active sync /dev/sdg
6 8 112 6 active sync /dev/sdh
7 8 128 7 active sync /dev/sdi
8 8 144 8 active sync /dev/sdj
9 8 160 9 active sync /dev/sdk
10 8 176 10 active sync /dev/sdl
11 8 192 11 active sync /dev/sdm
12 8 208 12 active sync /dev/sdn
13 8 224 13 active sync /dev/sdo
14 8 240 14 active sync /dev/sdp
15 65 0 15 active sync /dev/sdq
16 65 16 16 spare /dev/sdr
UUID : c18d1ba9:d7d4f338:6d83a4e1:023cbbdb
Events : 0.1
=====
All the disks are SATA 200GB, the first controller (for drives sd{a..f})
is 3ware 8xxx, the second is 3ware 9000 ATA Raid controller. The first
controller uses 3w-xxxx kernel module, the second one 3x-9xxx on linux
2.4.27.
Thank you for any help.
lukas
Guy wrote:
> Give us more info.
> Output from mdadm -D /dev/md?
>
> Guy
>
>
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org
> [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Lukas Kubin
> Sent: Tuesday, September 14, 2004 9:22 AM
> To: linux-raid@vger.kernel.org
> Subject: Array of disks attached to multiple controllers
>
> I'm having troubles creating Raid-5 array consisted of 16 (+1 spare)
> disks attached to two 3ware (not same version) SATA controllers.
>
> Mkraid creates the array without problems, however the array's total
> size only equals the sum of space on drives attached to the first of 2
> controllers. Ie, sd{b..f} are on scsi0 and sd{g..r} on scsi1. The space
> of created array is close to the sum of drives attechted to scsi0 only.
>
> Is it possible to combine drives on 2 controllers into one array? If
> not, what solution should I use to have one filesystem containing whole
> the drivespace on Raid-5?
>
> Thank you.
>
> lukas
>
--
Lukas Kubin
phone: +420596398275
email: kubin@opf.slu.cz
Information centre
The School of Business Administration in Karvina
Silesian University in Opava
Czech Republic
http://www.opf.slu.cz
^ permalink raw reply [flat|nested] 7+ messages in thread* RE: Array of disks attached to multiple controllers
2004-09-14 14:59 ` Guy
@ 2004-09-14 20:31 ` Neil Brown
0 siblings, 0 replies; 7+ messages in thread
From: Neil Brown @ 2004-09-14 20:31 UTC (permalink / raw)
To: Guy; +Cc: 'Lukas Kubin', linux-raid
On Tuesday September 14, bugzilla@watkins-home.com wrote:
> Well I have a clue!
> <Device size> X (<Number of disks> - 1)
> 195360896 X (16-1) = 2930413440 (a very nice number!)
>
> I think you have crossed a 2T limit.
> Subtract 2T from the expected size and you get you size.
> 2930413440 - 2^31 = 782929792 (This is your size!)
>
> 2^31 = 2T. (we have 1K blocks)
>
> You have exceeded a 2T limit somewhere. I don't know where the limit is. I
> thought I read that devices were limited to 2T, but arrays could be larger.
> Maybe you have an older kernel that only supports 2T. My 14 disk monster
> array is just over 200Gig, so I have not had this problem. A new kernel may
> help, but I don't know. Time for Neil to help! :)
2.4 kernels on 32bit processors cannot handle devices larger than 2^32
sectors (2T). Maybe not even bigger than 1T, I'm not sure.
2.6 kernels can if you select CONFIG_LBD (Large Block Devices).
>
> I think you will be fine if you only use 11 disks!
> 195360896 * (11-1) = 1953608960 (this is under the 2T limit).
Worth a try, as is using 2.6
NeilBrown
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2004-09-14 20:31 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-09-14 13:21 Array of disks attached to multiple controllers Lukas Kubin
2004-09-14 13:57 ` Guy
2004-09-14 14:07 ` Lukas Kubin
2004-09-14 14:46 ` Sebastien Koechlin
2004-09-14 15:22 ` Guy
2004-09-14 14:59 ` Guy
2004-09-14 20:31 ` Neil Brown
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).