Disabling block layer

Disabling block layer

[Mark Lobo]

Hello!

I had a question about disabling the block layer for SCSI devices. We
have an embedded device, and it runs 2.4.30. We need to be able to
support a lot of SCSI devices (in the thousands) for our device, and we
talk to the devices via SG. We are facing a memory allocation problem
after discovering a few thousand devices. For every device,  there
seems to be a lot of memory allocated in the block layer. This memory
includes cache memory (which IIRC is reclaimable by the kernel memory
subsystem when it needs it) and also pages that are used for the
alloc_pages pool.



My questions were relating to disabling the block layer for the
devices. We always talk direct passthrough to the storage(except the
local hard disk),  and do not need the block layer at all. 

1. Is there a way to disable the block layer for specific devices?

2. If yes, how can that be done, and  are there any gotchas associated with that?



Thanks!

M


 
____________________________________________________________________________________
Never miss an email again!
Yahoo! Toolbar alerts you the instant new Mail arrives.
http://tools.search.yahoo.com/toolbar/features/mail/

Re: Disabling block layer

[Douglas Gilbert]

Mark Lobo wrote:
> Hello!
> 
> I had a question about disabling the block layer for SCSI devices. We
> have an embedded device, and it runs 2.4.30. We need to be able to
> support a lot of SCSI devices (in the thousands) for our device, and we
> talk to the devices via SG. We are facing a memory allocation problem
> after discovering a few thousand devices. For every device,  there
> seems to be a lot of memory allocated in the block layer. This memory
> includes cache memory (which IIRC is reclaimable by the kernel memory
> subsystem when it needs it) and also pages that are used for the
> alloc_pages pool.
> 
> 
> 
> My questions were relating to disabling the block layer for the
> devices. We always talk direct passthrough to the storage(except the
> local hard disk),  and do not need the block layer at all. 
> 
> 1. Is there a way to disable the block layer for specific devices?
> 
> 2. If yes, how can that be done, and  are there any gotchas associated with that?

Mark,
Tempting thought that: linux without a block layer.
I think you have no hope in the lk 2.4 series and
even less in the lk 2.6 series.

Now for some thoughts. If you don't need to mount any
SCSI disks, you could build a kernel with sd as a
module and remove/hide sd_mod.o . A more invasive method
would be to modify the sd driver so that it was no
longer interested in SCSI devices whose peripheral
device type was zero (i.e. disks).

On the sg driver side, if lots of sg file descriptors
are open to those thousands of SCSI devices, then
reducing the per fd SG_DEF_RESERVED_SIZE from 32 KB
may help. This could be reduced by editing
include/scsi/sg.h .

Doug Gilbert

Re: Disabling block layer

[James Bottomley]

On Mon, 2007-03-26 at 17:23 -0400, Douglas Gilbert wrote:
> Mark Lobo wrote:
> > I had a question about disabling the block layer for SCSI devices. We
> > have an embedded device, and it runs 2.4.30. We need to be able to
> > support a lot of SCSI devices (in the thousands) for our device, and we
> > talk to the devices via SG. We are facing a memory allocation problem
> > after discovering a few thousand devices. For every device,  there
> > seems to be a lot of memory allocated in the block layer. This memory
> > includes cache memory (which IIRC is reclaimable by the kernel memory
> > subsystem when it needs it) and also pages that are used for the
> > alloc_pages pool.
> > 
> > 
> > 
> > My questions were relating to disabling the block layer for the
> > devices. We always talk direct passthrough to the storage(except the
> > local hard disk),  and do not need the block layer at all. 
> > 
> > 1. Is there a way to disable the block layer for specific devices?
> > 
> > 2. If yes, how can that be done, and  are there any gotchas associated with that?
> 
> Mark,
> Tempting thought that: linux without a block layer.
> I think you have no hope in the lk 2.4 series and
> even less in the lk 2.6 series.
> 
> Now for some thoughts. If you don't need to mount any
> SCSI disks, you could build a kernel with sd as a
> module and remove/hide sd_mod.o . A more invasive method
> would be to modify the sd driver so that it was no
> longer interested in SCSI devices whose peripheral
> device type was zero (i.e. disks).
> 
> On the sg driver side, if lots of sg file descriptors
> are open to those thousands of SCSI devices, then
> reducing the per fd SG_DEF_RESERVED_SIZE from 32 KB
> may help. This could be reduced by editing
> include/scsi/sg.h .

If it's just a question of tearing down all the resources for a given
device, wasn't 

scsi remove-single-device <h> <c> <t> <L> > /proc/scsi/scsi

The accepted way of doing it, even in 2.4?

James

Re: Disabling block layer

[Jeremy Linton]

Mark Lobo wrote:
> I had a question about disabling the block layer for SCSI devices. We
> have an embedded device, and it runs 2.4.30. We need to be able to
> support a lot of SCSI devices (in the thousands) for our device, and we
> talk to the devices via SG. We are facing a memory allocation problem
> after discovering a few thousand devices. For every device,  there
> seems to be a lot of memory allocated in the block layer. This memory
> includes cache memory (which IIRC is reclaimable by the kernel memory
> subsystem when it needs it) and also pages that are used for the
> alloc_pages pool.
> 
> My questions were relating to disabling the block layer for the
> devices. We always talk direct passthrough to the storage(except the
> local hard disk),  and do not need the block layer at all.

	You may consider something we experimented with here (for
performance 
reasons).. We basically recompiled one of the scsi drivers to call our 
own version of scsi_host_alloc() and then made calls to the 
queuecommand() routine directly. You then allow the kernel probe 
routines to only discover the first target with the local disk. I assume 
you know ahead of time which scsi cards your using in your system. The 
point is that you could just build a heavily modified scsi driver with 
application specific hooks.

	BTW we aren't currently doing this because in the end we got most of

what we needed by writing a driver which replaces sg and bypasses most 
of the kernel without being as invasive. In the long run we may still 
use a modified LLDD since the interfaces we depend on are changing a 
little to fast for our liking and we are not running any system devices 
on the interface cards we need to directly access.

	I'm not sure how you would go about tearing down enough of the
system 
that the device doesn't consume any resources, yet leave enough of it 
around to be accessible. I will be interested to find out what you end 
up doing.

Re: Disabling block layer

[Jens Axboe]

On Mon, Mar 26 2007, Mark Lobo wrote:
> Hello!
> 
> I had a question about disabling the block layer for SCSI devices. We
> have an embedded device, and it runs 2.4.30. We need to be able to
> support a lot of SCSI devices (in the thousands) for our device, and we
> talk to the devices via SG. We are facing a memory allocation problem
> after discovering a few thousand devices. For every device,  there
> seems to be a lot of memory allocated in the block layer. This memory
> includes cache memory (which IIRC is reclaimable by the kernel memory
> subsystem when it needs it) and also pages that are used for the
> alloc_pages pool.

A much easier approach would be to limit the memory used for each device
in the block layer. Since SCSI uses the block layer as a transport for
commands, you cannot disable the block layer in any easy manner.

But your memory is likely being eated by the queue freelist. So edit
drivers/block/ll_rw_blk.c and hardcode nr_requests to a low number (like
2).

-- 
Jens Axboe