Configuring large numbers of disks

Configuring large numbers of disks

[Mark Haverkamp]

Hello,

I have 4000 fibrechannel LUNs connected to my system.  There are two
paths to the devices and two ports on the host connected via a switch.
This gives 16000 when probed.  All of the devices are being configured
and appear in /sys/block.  I have had problems getting all of the
entries in /dev created.  I have been able to get udev to create all the
entries in /dev if I recompile udevd to increase the netlink setsockopt
SO_RCVBUF to 16384 * 1024 from the original 1024 * 1024.

I'd like some advice if that is the best way to go. Or is there some
other way to do this without recompiling?

I have a power arch machine with 2 processors and 7GB of memory.  I'm
running a suse sles9 distribution, 2.6.13-rc4,  and udev 070.

Thanks,
Mark.

-- 
Mark Haverkamp <markh@osdl.org>



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Configuring large numbers of disks

[Kay Sievers]

On Mon, Oct 10, 2005 at 01:14:51PM -0700, Mark Haverkamp wrote:
> I have 4000 fibrechannel LUNs connected to my system.  There are two
> paths to the devices and two ports on the host connected via a switch.
> This gives 16000 when probed.  All of the devices are being configured
> and appear in /sys/block.  I have had problems getting all of the
> entries in /dev created.  I have been able to get udev to create all the
> entries in /dev if I recompile udevd to increase the netlink setsockopt
> SO_RCVBUF to 16384 * 1024 from the original 1024 * 1024.

Did you allow such a big socket buffer? What does:
  grep . /proc/sys/net/core/*
print on that box?

> I'd like some advice if that is the best way to go. Or is there some
> other way to do this without recompiling?

2.6.13 got new sockopts to set a reasonable limit without making
it global with sysctrl:
  http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h°573dea1fb32ebc72ffa05980fd840df1d80860
We will probably use this in the next udev release.

Anyway, we may need to switch to a reliable transport instead of netlink,
if it still causes problems.

Thanks,
Kay


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Configuring large numbers of disks

[Mark Haverkamp]

On Tue, 2005-10-11 at 20:25 +0200, Kay Sievers wrote:
> On Mon, Oct 10, 2005 at 01:14:51PM -0700, Mark Haverkamp wrote:
> > I have 4000 fibrechannel LUNs connected to my system.  There are two
> > paths to the devices and two ports on the host connected via a switch.
> > This gives 16000 when probed.  All of the devices are being configured
> > and appear in /sys/block.  I have had problems getting all of the
> > entries in /dev created.  I have been able to get udev to create all the
> > entries in /dev if I recompile udevd to increase the netlink setsockopt
> > SO_RCVBUF to 16384 * 1024 from the original 1024 * 1024.
> 
> Did you allow such a big socket buffer? What does:
>   grep . /proc/sys/net/core/*
> print on that box?

# grep . /proc/sys/net/core/*
/proc/sys/net/core/dev_weight:64
/proc/sys/net/core/message_burst:10
/proc/sys/net/core/message_cost:5
/proc/sys/net/core/netdev_budget:300
/proc/sys/net/core/netdev_max_backlog:1000
/proc/sys/net/core/optmem_max:20480
/proc/sys/net/core/rmem_default:131072
/proc/sys/net/core/rmem_max:131071
/proc/sys/net/core/somaxconn:128
/proc/sys/net/core/wmem_default:131072
/proc/sys/net/core/wmem_max:131071

Although before I start udevd I issue sysctl commands to increase the
maximum buffer sizes.

sysctl -w net.core.rmem_max\x16777216
sysctl -w net.core.wmem_max\x16777216


 # grep . /proc/sys/net/core/*
/proc/sys/net/core/dev_weight:64
/proc/sys/net/core/message_burst:10
/proc/sys/net/core/message_cost:5
/proc/sys/net/core/netdev_budget:300
/proc/sys/net/core/netdev_max_backlog:1000
/proc/sys/net/core/optmem_max:20480
/proc/sys/net/core/rmem_default:131072
/proc/sys/net/core/rmem_max:16777216
/proc/sys/net/core/somaxconn:128
/proc/sys/net/core/wmem_default:131072
/proc/sys/net/core/wmem_max:16777216


> 
> > I'd like some advice if that is the best way to go. Or is there some
> > other way to do this without recompiling?
> 
> 2.6.13 got new sockopts to set a reasonable limit without making
> it global with sysctrl:
>   http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h°573dea1fb32ebc72ffa05980fd840df1d80860
> We will probably use this in the next udev release.
> 
> Anyway, we may need to switch to a reliable transport instead of netlink,
> if it still causes problems.
> 
> Thanks,
> Kay

Thanks,
Mark.

-- 
Mark Haverkamp <markh@osdl.org>



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Configuring large numbers of disks

[Kay Sievers]

On Tue, Oct 11, 2005 at 11:35:29AM -0700, Mark Haverkamp wrote:
> On Tue, 2005-10-11 at 20:25 +0200, Kay Sievers wrote:
> > On Mon, Oct 10, 2005 at 01:14:51PM -0700, Mark Haverkamp wrote:
> > > I have 4000 fibrechannel LUNs connected to my system.  There are two
> > > paths to the devices and two ports on the host connected via a switch.
> > > This gives 16000 when probed.  All of the devices are being configured
> > > and appear in /sys/block.  I have had problems getting all of the
> > > entries in /dev created.  I have been able to get udev to create all the
> > > entries in /dev if I recompile udevd to increase the netlink setsockopt
> > > SO_RCVBUF to 16384 * 1024 from the original 1024 * 1024.
> > 
> > Did you allow such a big socket buffer? What does:
> >   grep . /proc/sys/net/core/*
> > print on that box?
> 
> # grep . /proc/sys/net/core/*
> /proc/sys/net/core/dev_weight:64
> /proc/sys/net/core/message_burst:10
> /proc/sys/net/core/message_cost:5
> /proc/sys/net/core/netdev_budget:300
> /proc/sys/net/core/netdev_max_backlog:1000
> /proc/sys/net/core/optmem_max:20480
> /proc/sys/net/core/rmem_default:131072
> /proc/sys/net/core/rmem_max:131071
> /proc/sys/net/core/somaxconn:128
> /proc/sys/net/core/wmem_default:131072
> /proc/sys/net/core/wmem_max:131071
> 
> Although before I start udevd I issue sysctl commands to increase the
> maximum buffer sizes.
> 
> sysctl -w net.core.rmem_max\x16777216
> sysctl -w net.core.wmem_max\x16777216

I will make udevd to force it itself with the new option, so that will
no longer be needed.

Kay


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel