Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Tyler]

I'd like to build a Raid-6 array using 31 drives, is there a patch 
available or an easy way to safely increase the *MD_SB_DISKS*=27 to a 
higher number?  My attempts at patching the source myself were 
unsuccessful, more than likely because i don't know what/why the limit 
is 27, and whether that effects other parts of the kernel.

Thanks,
Tyler.

PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm v2.0-devel, 
using the --metadata 1.0 option on the create line, and still get:

# mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd 
/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh 
/dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp 
/dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx 
/dev/sdy /dev/sdz /dev/sdaa /dev/sdab
mdadm: invalid number of raid devices: 31
# mdadm -V
mdadm - v2.0-devel-1 - DEVELOPMENT VERSION NOT FOR REGULAR USE - 18 
February 2005
# uname -a
Linux pink 2.6.12-rc3-mm2 #1 SMP Mon May 2 02:15:28 CEST 2005 i686 GNU/Linux
# dmesg |grep md
md: raid5 personality registered as nr 4
md: raid6 personality registered as nr 8
md: md driver 0.90.2 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 3.38

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Peter T. Breuer]

Tyler <pml@dtbb.net> wrote:
> I'd like to build a Raid-6 array using 31 drives, is there a patch 
> available or an easy way to safely increase the *MD_SB_DISKS*=27 to a 
> higher number?  My attempts at patching the source myself were 
> unsuccessful, more than likely because i don't know what/why the limit 
> is 27, and whether that effects other parts of the kernel.

It's 27 because that's how much space there is in some of the structs.

Look in include/linux/raid. md_p.h, for example.

   #define MD_SB_DISKS                     27

and it calculates how much space that needs:

   #define MD_SB_DISKS_WORDS (MD_SB_DISKS*MD_SB_DESCRIPTOR_WORDS)
   #define MD_SB_DESCRIPTOR_WORDS          32

and all that is part of what must fit in a "mdp_superblock_s" struct.

           /*
            * Disks information
            */
           mdp_disk_t disks[MD_SB_DISKS];

with the space at the tail of the struct calculated ...


        /*
         * Reserved
         */
        __u32 reserved[MD_SB_RESERVED_WORDS];

 ... by figuring how much space was used and subtracting from 1K:

       #define MD_SB_RESERVED_WORDS            (1024 - MD_SB_GENERIC_WORDS - MD_SB_PERSONALITY_WORDS - MD_SB_DISKS_WORDS - MD_SB_DESCRIPTOR_WORDS)

and the space used includes the MD_SB_DESCRIPTOR_WORDS conisting of
those 27 32-byte array elements.

To make this number bigger, you would first have to make more room in
the struct, by changing the 1K in the calculation to 4096 (thus
extending the struct by extending its reserved section), then you
could use more of that room by increasing the 27. I presume the max
would now be 3*32+27 = 123.


There might be another such struct in the includes. I haven't looked
 ...

Peter

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Tyler]

Peter, after looking at the code after i sent the email, I more or less 
figured out what you put below, i'm wondering though, will changing the 
structures here affect anything else? other than the mdadm tool.. what 
about mkfs.ext3/xfs and possibly mount?

Thanks,
Tyler.

Peter T. Breuer wrote:

>Tyler <pml@dtbb.net> wrote:
>  
>
>>I'd like to build a Raid-6 array using 31 drives, is there a patch 
>>available or an easy way to safely increase the *MD_SB_DISKS*=27 to a 
>>higher number?  My attempts at patching the source myself were 
>>unsuccessful, more than likely because i don't know what/why the limit 
>>is 27, and whether that effects other parts of the kernel.
>>    
>>
>
>It's 27 because that's how much space there is in some of the structs.
>
>Look in include/linux/raid. md_p.h, for example.
>
>   #define MD_SB_DISKS                     27
>
>and it calculates how much space that needs:
>
>   #define MD_SB_DISKS_WORDS (MD_SB_DISKS*MD_SB_DESCRIPTOR_WORDS)
>   #define MD_SB_DESCRIPTOR_WORDS          32
>
>and all that is part of what must fit in a "mdp_superblock_s" struct.
>
>           /*
>            * Disks information
>            */
>           mdp_disk_t disks[MD_SB_DISKS];
>
>with the space at the tail of the struct calculated ...
>
>
>        /*
>         * Reserved
>         */
>        __u32 reserved[MD_SB_RESERVED_WORDS];
>
> ... by figuring how much space was used and subtracting from 1K:
>
>       #define MD_SB_RESERVED_WORDS            (1024 - MD_SB_GENERIC_WORDS - MD_SB_PERSONALITY_WORDS - MD_SB_DISKS_WORDS - MD_SB_DESCRIPTOR_WORDS)
>
>and the space used includes the MD_SB_DESCRIPTOR_WORDS conisting of
>those 27 32-byte array elements.
>
>To make this number bigger, you would first have to make more room in
>the struct, by changing the 1K in the calculation to 4096 (thus
>extending the struct by extending its reserved section), then you
>could use more of that room by increasing the 27. I presume the max
>would now be 3*32+27 = 123.
>
>
>There might be another such struct in the includes. I haven't looked
> ...
>
>Peter
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>  
>

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Peter T. Breuer]

Tyler <pml@dtbb.net> wrote:
> figured out what you put below, i'm wondering though, will changing the 
> structures here affect anything else? other than the mdadm tool.. what 

I can't think of anything, offhand.  I would first check through the
kernel cde looking for associated gotchas before worrying about trivia
like applications, however!

> about mkfs.ext3/xfs and possibly mount?

Not affected at all. They have no knowledge of such things.

You want to look for anything that uses md_p.h. I assume that is
"nothing", since applications will use md_u.h.


Peter

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Peter T. Breuer]

Peter T. Breuer <ptb@lab.it.uc3m.es> wrote:
> Tyler <pml@dtbb.net> wrote:
> > figured out what you put below, i'm wondering though, will changing the 
> > structures here affect anything else? other than the mdadm tool.. what 

> I can't think of anything, offhand.  I would first check through the
> kernel cde looking for associated gotchas before worrying about trivia
> like applications, however!

mdp_super_t (the size of which you modified) is used throughout md.c
only.  You want to check if it is mapped to disk or not. I think it is
(offhand) ...

  mdp_super_t *sb;

  ...

  bdevname(rdev->bdev, b);
  sb = (mdp_super_t*)page_address(rdev->sb_page);


So it seems to me to be intended to be 1KB of (sb) info on disk. If the
blocksize is 1K, then reading/writing from disk may only deliver 1K.
...

     if (!sync_page_io(rdev->bdev, rdev->sb_offset<<1, MD_SB_BYTES, rdev->sb_page, READ))
              goto fail;

Hmm ... MD_SB_BYTES. But ....


  #define MD_SB_BYTES                     4096

So we read 4K off disk at a time! Phew! 

Arrgh, but the undimensioned counts in md_p.h are in units of 4B, so
when it says "1024 - ...", it really means "4KB - ...", and hence if
you are going to icrease the size of the superblock, what you should do
is

1) increase MD_SB_BYTES to 8KB.
2) change "1024 - ..." to "2048 - ..." in md_p.h.
3) keep looking. Can one get 2 pages at once off disk with sync_page_io?

Where is that rdev->sb_offset set? Great, in calc_dev_sboffset:

        /*
         * Calculate the position of the superblock,
         * it's at the end of the disk.
         *
         * It also happens to be a multiple of 4Kb.
         */
        sb_offset = calc_dev_sboffset(rdev->bdev);
        rdev->sb_offset = sb_offset;

Yes, but if we are going to increase the size to 8KB, we'd better check
that calc ...

inline static sector_t calc_dev_sboffset(struct block_device *bdev)
{
        sector_t size = bdev->bd_inode->i_size >> BLOCK_SIZE_BITS;
        return MD_NEW_SIZE_BLOCKS(size);
 }

Uh huhh... it counts the blocks in the device (I think those are 1KB
units, not device-dependent, but I may be wrong). Then it chooses the
index of a smaller number to return - 1 past the available array size:

  #define MD_NEW_SIZE_BLOCKS(x)           ((x & ~(MD_RESERVED_BLOCKS - 1)) - MD_RESERVED_BLOCKS)

Ummmm ... that appears to be an attempt to calculate (x / y) * y - y, where 
y is a power of two ...

   #define MD_RESERVED_BLOCKS      (MD_RESERVED_BYTES / BLOCK_SIZE)

and the clunker ...

   #define MD_RESERVED_BYTES       (64 * 1024)

OK. So AT LEAST 64KB are reserved at the end of the disk. We can only
count on 64KB, however. The good news is that it looks like the first
4KB of that is the superblock, and that we might be able to use the rest
for a bigger superblock (I suspect bitmap info will be kept there
eventually). Provided you can arrange to read 8KB of sb, you might be
OK.


Peter





Peter

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Tyler]

Hi Peter,

Thanks for all the great info, I had about half of it figured out :)  
I'm starting to wonder though, isn't the version 1 superblock supposed 
to be able to use many more drives than 27? .. If so, should this not 
already be patched in the kernel source, and in the mdadm source?  What 
am I missing? .. it seems like I am trying to do something that's 
already supposedly done.. am I not?

Thanks,
Tyler.

Peter T. Breuer wrote:

>Peter T. Breuer <ptb@lab.it.uc3m.es> wrote:
>  
>
>>Tyler <pml@dtbb.net> wrote:
>>    
>>
>>>figured out what you put below, i'm wondering though, will changing the 
>>>structures here affect anything else? other than the mdadm tool.. what 
>>>      
>>>
>
>  
>
>>I can't think of anything, offhand.  I would first check through the
>>kernel cde looking for associated gotchas before worrying about trivia
>>like applications, however!
>>    
>>
>
>mdp_super_t (the size of which you modified) is used throughout md.c
>only.  You want to check if it is mapped to disk or not. I think it is
>(offhand) ...
>
>  mdp_super_t *sb;
>
>  ...
>
>  bdevname(rdev->bdev, b);
>  sb = (mdp_super_t*)page_address(rdev->sb_page);
>
>
>So it seems to me to be intended to be 1KB of (sb) info on disk. If the
>blocksize is 1K, then reading/writing from disk may only deliver 1K.
>...
>
>     if (!sync_page_io(rdev->bdev, rdev->sb_offset<<1, MD_SB_BYTES, rdev->sb_page, READ))
>              goto fail;
>
>Hmm ... MD_SB_BYTES. But ....
>
>
>  #define MD_SB_BYTES                     4096
>
>So we read 4K off disk at a time! Phew! 
>
>Arrgh, but the undimensioned counts in md_p.h are in units of 4B, so
>when it says "1024 - ...", it really means "4KB - ...", and hence if
>you are going to icrease the size of the superblock, what you should do
>is
>
>1) increase MD_SB_BYTES to 8KB.
>2) change "1024 - ..." to "2048 - ..." in md_p.h.
>3) keep looking. Can one get 2 pages at once off disk with sync_page_io?
>
>Where is that rdev->sb_offset set? Great, in calc_dev_sboffset:
>
>        /*
>         * Calculate the position of the superblock,
>         * it's at the end of the disk.
>         *
>         * It also happens to be a multiple of 4Kb.
>         */
>        sb_offset = calc_dev_sboffset(rdev->bdev);
>        rdev->sb_offset = sb_offset;
>
>Yes, but if we are going to increase the size to 8KB, we'd better check
>that calc ...
>
>inline static sector_t calc_dev_sboffset(struct block_device *bdev)
>{
>        sector_t size = bdev->bd_inode->i_size >> BLOCK_SIZE_BITS;
>        return MD_NEW_SIZE_BLOCKS(size);
> }
>
>Uh huhh... it counts the blocks in the device (I think those are 1KB
>units, not device-dependent, but I may be wrong). Then it chooses the
>index of a smaller number to return - 1 past the available array size:
>
>  #define MD_NEW_SIZE_BLOCKS(x)           ((x & ~(MD_RESERVED_BLOCKS - 1)) - MD_RESERVED_BLOCKS)
>
>Ummmm ... that appears to be an attempt to calculate (x / y) * y - y, where 
>y is a power of two ...
>
>   #define MD_RESERVED_BLOCKS      (MD_RESERVED_BYTES / BLOCK_SIZE)
>
>and the clunker ...
>
>   #define MD_RESERVED_BYTES       (64 * 1024)
>
>OK. So AT LEAST 64KB are reserved at the end of the disk. We can only
>count on 64KB, however. The good news is that it looks like the first
>4KB of that is the superblock, and that we might be able to use the rest
>for a bigger superblock (I suspect bitmap info will be kept there
>eventually). Provided you can arrange to read 8KB of sb, you might be
>OK.
>
>
>Peter
>
>
>
>
>
>Peter
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>  
>

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Peter T. Breuer]

Tyler <pml@dtbb.net> wrote:
> Thanks for all the great info, I had about half of it figured out :)  
> I'm starting to wonder though, isn't the version 1 superblock supposed 
> to be able to use many more drives than 27? .. If so, should this not 

I don't know (nor do I know anything about superblocks, i.e.  which
format you should be using).  All I can see is that the struct you
wanted to modify (the one with the "27" in its standard definition) is
read and written to disk in the first 4K of the 64K just after the
array contents.

Maybe you don't want to modify that struct but something else.  Maybe
something slightly different is written to disk.  That's impossible to
tell without documetation or experiment!

According to Paul, all should just work hunky-dory if you just suppress
the check in mdadm itself!

> already be patched in the kernel source, and in the mdadm source?  What 
> am I missing? .. it seems like I am trying to do something that's 
> already supposedly done.. am I not?

It most definitely is not the case that a particular one of the structs
in md_p.h can take more than 27 entries in the array that appears in it.
What significance that has requires some strategic inverse engineering
to discover. The most obvious interpretation is that it is a real
limit, but maybe not one that comes into play because we are using a
different ondis struct.

Peter

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Paul Clements]

Tyler wrote:

> PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm v2.0-devel, 
> using the --metadata 1.0 option on the create line, and still get:

> # mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd 
> /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh 
> /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp 
> /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx 
> /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
> mdadm: invalid number of raid devices: 31

As far as I can tell, mdadm just shouldn't give this error when you're 
using a version 1 superblock. If you disable that check in mdadm.c, I 
think everything will work. Version 1 superblocks allow something like 
2000 disks in an array.

--
Paul

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Tyler]

Hi Paul,

Isn't the superblock still limited to 27 drives though? .. is there 
maybe a patch for version 1 superblocks that i don't have in 
2.6.12-rc3-mm2, that changes the max drives in the md_p.h and md.c 
driver?  There is only 4096 bytes earmarked for superblocks, and within 
that amount, there isn't enough room to list more than 27 drives that 
may be part of the same array.

Do I need some more patches for this to work, that aren't in the -mm 
tree yet?  I was under the impression (from the mdadm v2.0 announce 
file, that anything *after* 2.6.11-rc3 in the -mm tree would have the 
required patches.

Also, where can I disable the check in mdadm?

Thanks,
Tyler.

Paul Clements wrote:

> Tyler wrote:
>
>> PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm 
>> v2.0-devel, using the --metadata 1.0 option on the create line, and 
>> still get:
>
>
>> # mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd 
>> /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg 
>> /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn 
>> /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu 
>> /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
>> mdadm: invalid number of raid devices: 31
>
>
> As far as I can tell, mdadm just shouldn't give this error when you're 
> using a version 1 superblock. If you disable that check in mdadm.c, I 
> think everything will work. Version 1 superblocks allow something like 
> 2000 disks in an array.
>
> -- 
> Paul

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Paul Clements]

Tyler wrote:

> Isn't the superblock still limited to 27 drives though? .. is there 

The version 0.90 superblock is limited to 27 drives. Version 1 
superblock is limited to about 2000 drives, because Neil shrunk the 
amount of per-drive data stored in the superblock drastically. The check 
in mdadm, is in the file mdadm.c, just grep for the message that you got 
("invalid number of raid devices"). Comment out that code and recompile 
mdadm. Everything should work after that. The kernel you have already 
has support for version 1 superblocks.

--
Paul

> maybe a patch for version 1 superblocks that i don't have in 
> 2.6.12-rc3-mm2, that changes the max drives in the md_p.h and md.c 
> driver?  There is only 4096 bytes earmarked for superblocks, and within 
> that amount, there isn't enough room to list more than 27 drives that 
> may be part of the same array.
> 
> Do I need some more patches for this to work, that aren't in the -mm 
> tree yet?  I was under the impression (from the mdadm v2.0 announce 
> file, that anything *after* 2.6.11-rc3 in the -mm tree would have the 
> required patches.
> 
> Also, where can I disable the check in mdadm?
> 
> Thanks,
> Tyler.
> 
> Paul Clements wrote:
> 
>> Tyler wrote:
>>
>>> PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm 
>>> v2.0-devel, using the --metadata 1.0 option on the create line, and 
>>> still get:
>>
>>
>>
>>> # mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd 
>>> /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg 
>>> /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn 
>>> /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu 
>>> /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
>>> mdadm: invalid number of raid devices: 31
>>
>>
>>
>> As far as I can tell, mdadm just shouldn't give this error when you're 
>> using a version 1 superblock. If you disable that check in mdadm.c, I 
>> think everything will work. Version 1 superblocks allow something like 
>> 2000 disks in an array.
>>
>> -- 
>> Paul
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

[Tyler]

Okay, I will give that a try paul... but i've actually tryed using mdadm 
v2.0-devel to create an array with a version 1 superblock, using only 3 
drives in a raid5 array.  It hasn't worked so far, read my 2 bugreports 
to the mailing list, to see whats happening.. and maybe draw some 
insight from them?  I'm at a loss as to why mdadm is failing right now.. 
even without going over the 27 drive limit.  Neil hasn't responded to 
any emails I've sent regarding this either..

Thanks,
Tyler.

PS - Can you see any kind of order to the "applied" patches on Neil's 
site? .. I can't tell which ones are for the 2.0-devel version and which 
are for 1.9.x .. and one says 1.8 ... etc..

Paul Clements wrote:

> Tyler wrote:
>
>> Isn't the superblock still limited to 27 drives though? .. is there 
>
>
> The version 0.90 superblock is limited to 27 drives. Version 1 
> superblock is limited to about 2000 drives, because Neil shrunk the 
> amount of per-drive data stored in the superblock drastically. The 
> check in mdadm, is in the file mdadm.c, just grep for the message that 
> you got ("invalid number of raid devices"). Comment out that code and 
> recompile mdadm. Everything should work after that. The kernel you 
> have already has support for version 1 superblocks.
>
> -- 
> Paul
>
>> maybe a patch for version 1 superblocks that i don't have in 
>> 2.6.12-rc3-mm2, that changes the max drives in the md_p.h and md.c 
>> driver?  There is only 4096 bytes earmarked for superblocks, and 
>> within that amount, there isn't enough room to list more than 27 
>> drives that may be part of the same array.
>>
>> Do I need some more patches for this to work, that aren't in the -mm 
>> tree yet?  I was under the impression (from the mdadm v2.0 announce 
>> file, that anything *after* 2.6.11-rc3 in the -mm tree would have the 
>> required patches.
>>
>> Also, where can I disable the check in mdadm?
>>
>> Thanks,
>> Tyler.
>>
>> Paul Clements wrote:
>>
>>> Tyler wrote:
>>>
>>>> PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm 
>>>> v2.0-devel, using the --metadata 1.0 option on the create line, and 
>>>> still get:
>>>
>>>
>>>
>>>
>>>> # mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd 
>>>> /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg 
>>>> /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn 
>>>> /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu 
>>>> /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
>>>> mdadm: invalid number of raid devices: 31
>>>
>>>
>>>
>>>
>>> As far as I can tell, mdadm just shouldn't give this error when 
>>> you're using a version 1 superblock. If you disable that check in 
>>> mdadm.c, I think everything will work. Version 1 superblocks allow 
>>> something like 2000 disks in an array.
>>>
>>> -- 
>>> Paul
>>
>>
>>
>> -
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>