Re: [patch for playing] 2.5.65 patch to support > 256 disks

Douglas Gilbert
2003-03-22T11:00:33Z

Badari Pulavarty wrote:
> Hi,
>
> Andries Brouwer recently submitted 32 bit dev_t patches,
> which are in 2.5.65-mm2. This patch applies on those patches to support
> more than 256 disks. This is for playing only.
>
> I tested this with 4000 disks using scsi_debug. I attached my actual
> disks (50) after 4000 scsi_debug disks. I am able to access my disks
> fine and do IO on them.
>
> Problems (so far):
>
> 1) sd.c - sd_index_bits[] arrys became big - need to be fixed.
>
> 2) 4000 disks eats up lots of low memory (~460 MB). Here is the
> /proc/meminfo output before & after insmod.
>
> Before:
> MemTotal: 3883276 kB
> MemFree: 3808028 kB
> Buffers: 3240 kB
> Cached: 41860 kB
> SwapCached: 0 kB
> Active: 45360 kB
> Inactive: 7288 kB
> HighTotal: 3014616 kB
> HighFree: 2961856 kB
> LowTotal: 868660 kB
> LowFree: 846172 kB
> SwapTotal: 2040244 kB
> SwapFree: 2040244 kB
> Dirty: 192 kB
> Writeback: 0 kB
> Mapped: 14916 kB
> Slab: 7164 kB
> Committed_AS: 12952 kB
> PageTables: 312 kB
> ReverseMaps: 1895
> ====
> After:
> MemTotal: 3883276 kB
> MemFree: 3224140 kB
> Buffers: 3880 kB
> Cached: 140376 kB
> SwapCached: 0 kB
> Active: 47512 kB
> Inactive: 105508 kB
> HighTotal: 3014616 kB
> HighFree: 2838144 kB
> LowTotal: 868660 kB
> LowFree: 385996 kB
> SwapTotal: 2040244 kB
> SwapFree: 2040244 kB
> Dirty: 92 kB
> Writeback: 0 kB
> Mapped: 16172 kB
> Slab: 464364 kB
> Committed_AS: 14996 kB
> PageTables: 412 kB
> ReverseMaps: 2209

Badari,
I poked around looking for data on the size issue.
Here are the byte sizes for the per device and per host
structures in scsi_debug and the scsi mid level for
i386, non-smp in lk 2.5.65:
sizeof(sdebug_dev_info)=60, sizeof(scsi_device)=376
sizeof(sdebug_host_info)=24, sizeof(Scsi_Host)=224

So for 4000 disks they should be responsible for about
2 MB.

The scsi_cmd_cache slab info went from this before those
84 pseudo disks were added:
# cat slabinfo_pre.txt
scsi_cmd_cache 3 11 356 1 1 1 : 32 16 :
22 301 6 5 0 0 43 : 3235 31 3264 0
to this afterwards:
# cat slabinfo_post.txt
scsi_cmd_cache 44 55 356 5 5 1 : 32 16 :
66 398 12 7 0 0 43 : 5837 40 5833 0

I did notice a rather large growth of nodes
in sysfs. For 84 added scsi_debug pseudo disks the number
of sysfs nodes went from 686 to 3347.

Does anybody know what is the per node memory cost of sysfs?

Doug Gilbert