Scalability of MD raid 1 mirror devices

Scalability of MD raid 1 mirror devices

[Suresh Babu Kandukuru]

Dear Group,

We are doing scalability of MD raid 1  mirror devices on the Linux host running the 3.17.2 .   we see the number of RAID 1 devices limited to 128 in one case and 511 in another case .  we would like to know why this limitation ?. Appreciate any kind of inputs and pointers 

We can create the RAID 1 device in 3 ways.

1.  /dev/mdX   -> here X is the number, we can specify from 0 to 511.
2. /dev/md/X  -> here also X is a number from 0 to 511. It creates it as a link to the actual device /dev/mdX. ( similar to above but creates a link also).
3. /dev/md/”name”  -> This creates a link to actual device, whichever is free starting from 127 to 0. Below is the function which is responsible for it.

char *find_free_devnm(int use_partitions)
{
        static char devnm[32];
        int devnum;
        for (devnum = 127; devnum != 128;  devnum = devnum ? devnum-1 : (1<<20)-1) {

                if (use_partitions)
                        sprintf(devnm, "md_d%d", devnum);
                else
                        sprintf(devnm, "md%d", devnum);
                if (mddev_busy(devnm))
                        continue;
                if (!conf_name_is_free(devnm))
                        continue;
                if (!use_udev()) {
                        /* make sure it is new to /dev too, at least as a
                         * non-standard */
                        int devid = devnm2devid(devnm);
                        if (devid) {
                                char *dn = map_dev(major(devid),
                                                   minor(devid), 0);
                                if (dn && ! is_standard(dn, NULL))
                                        continue;
                        }
                }
                break;
        }
        if (devnum == 128)
                return NULL;
        return devnm;
}

So ideally we should not create a device which is more than 128 { The program may crash }.

Then we  tried to find how we are able to create up to 511 and why it is failing after that.

int dev_open(char *dev, int flags)

inside this function 

fd = open(dev, flags);  / this line is assigning fd to -1 , which is causing the program to fail. So I wrote a simple program to crosscheck it.

int main(){

        char devname[32] = "/dev/hello1";

// The flags I have set according to the code.

        int flags = O_RDWR;
        flags |= O_DIRECT;
        if (mknod(devname, S_IFBLK|0600, makedev(9,511)) == 0) {

                int  fd = open(devname, flags);
                cout<<fd<<endl;
                unlink(devname);

        }
}

So if the minor number is more than 511, the “fd” is assigned to -1, if it is in the range of 0 to 511 It is working fine.

So should we go with the range of 511 or stick to 128 ?

Thanks 
/Suresh
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Scalability of MD raid 1 mirror devices

[Ankur Bose]

Hi, it seems like this is very active group , I don't see any reason why anyone is not replying to the below mail.
 

Kindly reply we are in middle of something.

Thanks,
ankur.
-----Original Message-----
From: Suresh Babu Kandukuru 
Sent: 12 October 2015 17:34
To: linux-raid@vger.kernel.org; neilb@suse.de
Subject: Scalability of MD raid 1 mirror devices


Dear Group,

We are doing scalability of MD raid 1  mirror devices on the Linux host running the 3.17.2 .   we see the number of RAID 1 devices limited to 128 in one case and 511 in another case .  we would like to know why this limitation ?. Appreciate any kind of inputs and pointers 

We can create the RAID 1 device in 3 ways.

1.  /dev/mdX   -> here X is the number, we can specify from 0 to 511.
2. /dev/md/X  -> here also X is a number from 0 to 511. It creates it as a link to the actual device /dev/mdX. ( similar to above but creates a link also).
3. /dev/md/”name”  -> This creates a link to actual device, whichever is free starting from 127 to 0. Below is the function which is responsible for it.

char *find_free_devnm(int use_partitions) {
        static char devnm[32];
        int devnum;
        for (devnum = 127; devnum != 128;  devnum = devnum ? devnum-1 : (1<<20)-1) {

                if (use_partitions)
                        sprintf(devnm, "md_d%d", devnum);
                else
                        sprintf(devnm, "md%d", devnum);
                if (mddev_busy(devnm))
                        continue;
                if (!conf_name_is_free(devnm))
                        continue;
                if (!use_udev()) {
                        /* make sure it is new to /dev too, at least as a
                         * non-standard */
                        int devid = devnm2devid(devnm);
                        if (devid) {
                                char *dn = map_dev(major(devid),
                                                   minor(devid), 0);
                                if (dn && ! is_standard(dn, NULL))
                                        continue;
                        }
                }
                break;
        }
        if (devnum == 128)
                return NULL;
        return devnm;
}

So ideally we should not create a device which is more than 128 { The program may crash }.

Then we  tried to find how we are able to create up to 511 and why it is failing after that.

int dev_open(char *dev, int flags)

inside this function 

fd = open(dev, flags);  / this line is assigning fd to -1 , which is causing the program to fail. So I wrote a simple program to crosscheck it.

int main(){

        char devname[32] = "/dev/hello1";

// The flags I have set according to the code.

        int flags = O_RDWR;
        flags |= O_DIRECT;
        if (mknod(devname, S_IFBLK|0600, makedev(9,511)) == 0) {

                int  fd = open(devname, flags);
                cout<<fd<<endl;
                unlink(devname);

        }
}

So if the minor number is more than 511, the “fd” is assigned to -1, if it is in the range of 0 to 511 It is working fine.

So should we go with the range of 511 or stick to 128 ?

Thanks
/Suresh
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Scalability of MD raid 1 mirror devices

[Neil Brown]

[-- Attachment #1: Type: text/plain, Size: 4307 bytes --]

Ankur Bose <ankur.bose@oracle.com> writes:

> Hi, it seems like this is very active group , I don't see any reason why anyone is not replying to the below mail.
>  
>
> Kindly reply we are in middle of something.

We are all in the middle of something.


>
> Thanks,
> ankur.
> -----Original Message-----
> From: Suresh Babu Kandukuru 
> Sent: 12 October 2015 17:34
> To: linux-raid@vger.kernel.org; neilb@suse.de
> Subject: Scalability of MD raid 1 mirror devices
>
>
> Dear Group,
>
> We are doing scalability of MD raid 1  mirror devices on the Linux host running the 3.17.2 .   we see the number of RAID 1 devices limited to 128 in one case and 511 in another case .  we would like to know why this limitation ?. Appreciate any kind of inputs and pointers 
>
> We can create the RAID 1 device in 3 ways.
>
> 1.  /dev/mdX   -> here X is the number, we can specify from 0 to 511.
> 2. /dev/md/X  -> here also X is a number from 0 to 511. It creates it as a link to the actual device /dev/mdX. ( similar to above but creates a link also).
> 3. /dev/md/”name”  -> This creates a link to actual device, whichever is free starting from 127 to 0. Below is the function which is responsible for it.
>
> char *find_free_devnm(int use_partitions) {
>         static char devnm[32];
>         int devnum;
>         for (devnum = 127; devnum != 128;  devnum = devnum ? devnum-1 : (1<<20)-1) {
>
>                 if (use_partitions)
>                         sprintf(devnm, "md_d%d", devnum);
>                 else
>                         sprintf(devnm, "md%d", devnum);
>                 if (mddev_busy(devnm))
>                         continue;
>                 if (!conf_name_is_free(devnm))
>                         continue;
>                 if (!use_udev()) {
>                         /* make sure it is new to /dev too, at least as a
>                          * non-standard */
>                         int devid = devnm2devid(devnm);
>                         if (devid) {
>                                 char *dn = map_dev(major(devid),
>                                                    minor(devid), 0);
>                                 if (dn && ! is_standard(dn, NULL))
>                                         continue;
>                         }
>                 }
>                 break;
>         }
>         if (devnum == 128)
>                 return NULL;
>         return devnm;
> }
>
> So ideally we should not create a device which is more than 128 { The
> program may crash }.

Please explain why you think the program would crash?


>
> Then we  tried to find how we are able to create up to 511 and why it is failing after that.
>
> int dev_open(char *dev, int flags)
>
> inside this function 
>
> fd = open(dev, flags);  / this line is assigning fd to -1 , which is causing the program to fail. So I wrote a simple program to crosscheck it.
>
> int main(){
>
>         char devname[32] = "/dev/hello1";
>
> // The flags I have set according to the code.
>
>         int flags = O_RDWR;
>         flags |= O_DIRECT;
>         if (mknod(devname, S_IFBLK|0600, makedev(9,511)) == 0) {
>
>                 int  fd = open(devname, flags);
>                 cout<<fd<<endl;
>                 unlink(devname);
>
>         }
> }
>
> So if the minor number is more than 511, the “fd” is assigned to -1,
> if it is in the range of 0 to 511 It is working fine.

Hmmm... you are right.
Probably due to this:

	blk_register_region(MKDEV(MD_MAJOR, 0), 512, THIS_MODULE,
			    md_probe, NULL, NULL);

Try changing the "512" to "1<<MINORBITS".


NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

RE: Scalability of MD raid 1 mirror devices

[Neil Brown]

Neil Brown <neilb@suse.de> writes:
>> So if the minor number is more than 511, the “fd” is assigned to -1,
>> if it is in the range of 0 to 511 It is working fine.
>
> Hmmm... you are right.
> Probably due to this:
>
> 	blk_register_region(MKDEV(MD_MAJOR, 0), 512, THIS_MODULE,
> 			    md_probe, NULL, NULL);
>
> Try changing the "512" to "1<<MINORBITS".

Actually don't change that.  That is deliberate.

add "CREATE named=yes" to /etc/mdadm.conf, then create arrays
with names like
  /dev/md/whatever-you-like

and you can create as many as you like.

The arrays are create by effectively doing

  echo md_whatever-you-like > /sys/module/md_mod/parameters/new_array

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html