limit of maximum number of rbd devices that can be mapped

limit of maximum number of rbd devices that can be mapped

[Christian Huang]

Hi,
    what is the maximum number of rbd devices that can be mapped on a
single host?
    we recently did a test to mount a large number of rbd devices and
hit a wall around 230
    with the following message

with 230 rbd devices already mapped, the next one yielded this.
Console output
wistor@wistor-002:~$ sudo rbd map test299
add failed: (16) Device or resource busy

dmesg
[77962.346749] register_blkdev: failed to get major for rbd229

Chris.

Re: limit of maximum number of rbd devices that can be mapped

[Alex Elder]

On 10/17/2012 09:15 PM, Christian Huang wrote:
> Hi,
>     what is the maximum number of rbd devices that can be mapped on a
> single host?
>     we recently did a test to mount a large number of rbd devices and
> hit a wall around 230
>     with the following message

That sounds about right.  We were just discussing this earlier
this week.  It's limited at the moment by the number of major
device numbers available on the system, and I think you found
that limit.  We would obviously like to have a solution for
this but we've only just started considering options.

					-Alex

> with 230 rbd devices already mapped, the next one yielded this.
> Console output
> wistor@wistor-002:~$ sudo rbd map test299
> add failed: (16) Device or resource busy
> 
> dmesg
> [77962.346749] register_blkdev: failed to get major for rbd229
> 
> Chris.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: limit of maximum number of rbd devices that can be mapped

[Christian Huang]

Hi,
    i see,
    could this be relevant?
    SCSI disk limitations: Defining the number of disk devices on Linux
    http://publib.boulder.ibm.com/infocenter/dsichelp/ds6000ic/index.jsp?topic=%2Fcom.ibm.storage.smric.help.doc%2Ff2c_linuxscsilimit_2hsag9.html

    also from the description of the following 2 sections, it seems
this limit can be made much higher?
    http://www.makelinux.net/ldd3/chp-3-sect-2
    http://www.makelinux.net/ldd3/chp-16-sect-1

Best Regards.


On Thu, Oct 18, 2012 at 10:18 AM, Alex Elder <elder@inktank.com> wrote:
> On 10/17/2012 09:15 PM, Christian Huang wrote:
>> Hi,
>>     what is the maximum number of rbd devices that can be mapped on a
>> single host?
>>     we recently did a test to mount a large number of rbd devices and
>> hit a wall around 230
>>     with the following message
>
> That sounds about right.  We were just discussing this earlier
> this week.  It's limited at the moment by the number of major
> device numbers available on the system, and I think you found
> that limit.  We would obviously like to have a solution for
> this but we've only just started considering options.
>
>                                         -Alex
>
>> with 230 rbd devices already mapped, the next one yielded this.
>> Console output
>> wistor@wistor-002:~$ sudo rbd map test299
>> add failed: (16) Device or resource busy
>>
>> dmesg
>> [77962.346749] register_blkdev: failed to get major for rbd229
>>
>> Chris.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>

Re: limit of maximum number of rbd devices that can be mapped

[Danny Al-Gaaf]

Am 18.10.2012 04:18, schrieb Alex Elder:
> On 10/17/2012 09:15 PM, Christian Huang wrote:
>> Hi,
>>     what is the maximum number of rbd devices that can be mapped on a
>> single host?
>>     we recently did a test to mount a large number of rbd devices and
>> hit a wall around 230
>>     with the following message
> 
> That sounds about right.  We were just discussing this earlier
> this week.  It's limited at the moment by the number of major
> device numbers available on the system, and I think you found
> that limit.  We would obviously like to have a solution for
> this but we've only just started considering options.

Any news on this? We have seen the same issues here.

IMO the RBD code should to something similar to some other block drivers:

1) reserve a major block number for RBD in general (or use the number
returned from the register_blkdev call, if there is no number we can
reserve in general for RBD, see [1])

2) Use this major number for all RBDs on this machine and handle each
new RBD via different minor numbers.

If we can agree on this I would take a look into it and send a patch.

Regards

Danny

[1]
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob_plain;f=Documentation/devices.txt;hb=HEAD

Re: limit of maximum number of rbd devices that can be mapped

[Sage Weil]

On Fri, 2 Nov 2012, Danny Al-Gaaf wrote:
> Am 18.10.2012 04:18, schrieb Alex Elder:
> > On 10/17/2012 09:15 PM, Christian Huang wrote:
> >> Hi,
> >>     what is the maximum number of rbd devices that can be mapped on a
> >> single host?
> >>     we recently did a test to mount a large number of rbd devices and
> >> hit a wall around 230
> >>     with the following message
> > 
> > That sounds about right.  We were just discussing this earlier
> > this week.  It's limited at the moment by the number of major
> > device numbers available on the system, and I think you found
> > that limit.  We would obviously like to have a solution for
> > this but we've only just started considering options.
> 
> Any news on this? We have seen the same issues here.
> 
> IMO the RBD code should to something similar to some other block drivers:
> 
> 1) reserve a major block number for RBD in general (or use the number
> returned from the register_blkdev call, if there is no number we can
> reserve in general for RBD, see [1])
> 
> 2) Use this major number for all RBDs on this machine and handle each
> new RBD via different minor numbers.
> 
> If we can agree on this I would take a look into it and send a patch.

We should probably do what SCSI does, where the minor for each device 
jumps by 16 to leave room for partitions.

There was a thread on this a couple of weeks ago that mmy slow internet is 
preventing me from finding... :)

sage

Re: limit of maximum number of rbd devices that can be mapped

[Danny Al-Gaaf]

Am 02.11.2012 12:22, schrieb Sage Weil:
> On Fri, 2 Nov 2012, Danny Al-Gaaf wrote:
[...]
>> Any news on this? We have seen the same issues here.
>>
>> IMO the RBD code should to something similar to some other block drivers:
>>
>> 1) reserve a major block number for RBD in general (or use the number
>> returned from the register_blkdev call, if there is no number we can
>> reserve in general for RBD, see [1])
>>
>> 2) Use this major number for all RBDs on this machine and handle each
>> new RBD via different minor numbers.
>>
>> If we can agree on this I would take a look into it and send a patch.
> 
> We should probably do what SCSI does, where the minor for each device 
> jumps by 16 to leave room for partitions.

That's exactly what I would propose. I wasn't sure if we really need to
leave the room for the partitions, since I'm not sure if they really
need to have contiguous numbers, but I guess there are some tools which
expect it.

> There was a thread on this a couple of weeks ago that mmy slow internet is 
> preventing me from finding... :)

My internet connection isn't that slow here in Amsterdam ;-) ... but I
couldn't find another thread about this topic.

Do you remember if someone already is working on this?

Danny