Re: [patch for playing] 2.5.65 patch to support > 256 disks

[Douglas Gilbert <dougg@torque.net>]

Badari Pulavarty wrote:
> Hi,
> 
> Andries Brouwer recently submitted 32 bit dev_t patches,
> which are in 2.5.65-mm2. This patch applies on those patches to support 
> more than 256 disks.  This is for playing only.
> 
> I tested this with 4000 disks using scsi_debug. I attached my actual
> disks (50) after 4000 scsi_debug disks. I am able to access my disks
> fine and do IO on them.
> 
> Problems (so far):
> 
> 1) sd.c -  sd_index_bits[] arrys became big - need to be fixed.
> 
> 2) 4000 disks eats up lots of low memory (~460 MB). Here is the
> /proc/meminfo output before & after insmod.
> 
> Before:
> MemTotal:      3883276 kB
> MemFree:       3808028 kB
> Buffers:          3240 kB
> Cached:          41860 kB
> SwapCached:          0 kB
> Active:          45360 kB
> Inactive:         7288 kB
> HighTotal:     3014616 kB
> HighFree:      2961856 kB
> LowTotal:       868660 kB
> LowFree:        846172 kB
> SwapTotal:     2040244 kB
> SwapFree:      2040244 kB
> Dirty:             192 kB
> Writeback:           0 kB
> Mapped:          14916 kB
> Slab:             7164 kB
> Committed_AS:    12952 kB
> PageTables:        312 kB
> ReverseMaps:      1895
> ====
> After:
> MemTotal:      3883276 kB
> MemFree:       3224140 kB
> Buffers:          3880 kB
> Cached:         140376 kB
> SwapCached:          0 kB
> Active:          47512 kB
> Inactive:       105508 kB
> HighTotal:     3014616 kB
> HighFree:      2838144 kB
> LowTotal:       868660 kB
> LowFree:        385996 kB
> SwapTotal:     2040244 kB
> SwapFree:      2040244 kB
> Dirty:              92 kB
> Writeback:           0 kB
> Mapped:          16172 kB
> Slab:           464364 kB
> Committed_AS:    14996 kB
> PageTables:        412 kB
> ReverseMaps:      2209

Badari,
I poked around looking for data on the size issue.
Here are the byte sizes for the per device and per host
structures in scsi_debug and the scsi mid level for
i386, non-smp in lk 2.5.65:
   sizeof(sdebug_dev_info)=60, sizeof(scsi_device)=376
     sizeof(sdebug_host_info)=24, sizeof(Scsi_Host)=224

So for 4000 disks they should be responsible for about
2 MB.

The scsi_cmd_cache slab info went from this before those
84 pseudo disks were added:
# cat slabinfo_pre.txt
scsi_cmd_cache         3     11    356    1    1    1 :   32   16 : 
22     301     6    5    0    0   43 :   3235     31   3264      0
to this afterwards:
# cat slabinfo_post.txt
scsi_cmd_cache        44     55    356    5    5    1 :   32   16 : 
66     398    12    7    0    0   43 :   5837     40   5833      0

I did notice a rather large growth of nodes
in sysfs. For 84 added scsi_debug pseudo disks the number
of sysfs nodes went from 686 to 3347.

Does anybody know what is the per node memory cost of sysfs?

Doug Gilbert