Re: problems with lots of arrays

[NeilBrown <nfbrown@novell.com>]

[-- Attachment #1: Type: text/plain, Size: 3581 bytes --]

On Fri, May 06 2016, Mike Lovell wrote:

> we have a number of systems that have a large number of software
> arrays running. its in the couple hundred range. we have been using a
> custom built kernel based on 3.4 but are wanting to update to a
> mainline kernel and have been experimenting with 4.4. the systems are
> running recent centos 6 releases but we have been downgrading the
> mdadm version from 3.3.2 in 6.7 to a custom build 3.2.6. we installed
> the downgraded version due to a problem with array numbering. i
> emailed the list a while ago explaining the issue and submitting a
> patch to fix [1]. i never heard anything back and since we had a
> simple fix i didn't follow up on it.
>
> unfortunately, when testing the 3.2.6 mdadm with linux kernel 4.4
> wasn't working. mdadm and the kernel would complain about the devices
> not having a valid v1.2 superblock and not start the array. testing
> with 3.3.2 from the current centos repos worked. i'd like to update
> but we still have the issue with lots of arrays mentioned previously.
>
> i spent some time checking to make sure that my patch rebases against
> master properly (and it does) but during testing i was unable to
> create an array with number larger than /dev/md511 when using the 4.4
> kernel we were testing as well as the 4.2 kernel i had on another test
> box. creating one larger than 511 on a system with a 3.16 kernel
> worked. it looks like something broke between kernel 3.16 and 4.2 that
> limited the number of arrays to 512 (/dev/md0 to /dev/md511). this was
> a problem regardless of mdadm version and i haven't yet done much
> digging into the problem.
>
> there are a couple things that could potentially be done. the easiest,
> would be to just modify find_free_devnm() in mdopen.c from wrapping to
> (1<<20)-1 and instead have it wrap around to (1<<9))-1. this would
> limit mdadm to 512 auto-generated array numbers. i'm guessing this
> would be sufficient for the vast majority of cases and would solve the
> problem i'm facing at work. the next option would be to apply the
> patch in my previous email and then figuring out why the newer
> versions of the kernel don't support more than 512 arrays. this would
> take more work but probably the better long term approach.
>

I know why newer kernels don't seem to support more than 512 array.

Commit: af5628f05db6 ("md: disable probing for md devices 512 and over.")


You can easily use many more md devices by using a newish mdadm and
setting

   CREATE names=yes

in /etc/mdadm.conf

You cannot use names like "md512" because that gets confusing, but any
name that isn't a string of digits is fine.  e.g. create /dev/md/foo
and the array will be named "md_foo" in the kernel rather than "md127".

I guess this qualifies as a regression and regressions are bad.....
But I really wanted to be able to have arrays that didn't get magically
created simply because you open a file in /dev.  That just leads to
races with udev.

The magic number "512" appears three times in the kernel.

                /* find an unused unit number */
                static int next_minor = 512;

and

        blk_register_region(MKDEV(MD_MAJOR, 0), 512, THIS_MODULE,
                            md_probe, NULL, NULL);
and
        blk_unregister_region(MKDEV(MD_MAJOR,0), 512);

A boot parameter which set that to something larger would probably be OK
and would solve your immediate problem.

But if you could transition to using named arrays instead of numbered
arrays - even if that are "/dev/md/X%d", that would be be good I think.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]