Re: cdrom: use kmalloced buffers instead of buffers on stack

Re: cdrom: use kmalloced buffers instead of buffers on stack

[Jeff Garzik]

Linux Kernel Mailing List wrote:
> Gitweb:     http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=22a9189fd073db3d03a4cf8b8c098aa207602de1
> Commit:     22a9189fd073db3d03a4cf8b8c098aa207602de1
> Parent:     0a0c4114df4a6903bccb65b06cabb6ddc968f877
> Author:     Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> AuthorDate: Wed Mar 26 12:09:38 2008 +0100
> Committer:  Jens Axboe <jens.axboe@oracle.com>
> CommitDate: Mon Apr 21 09:50:08 2008 +0200
> 
>     cdrom: use kmalloced buffers instead of buffers on stack
>     
>     If cdrom commands are issued to a scsi drive in most cases the buffer will be
>     filled via dma.  This leads to bad stack corruption on non coherent platforms,
>     because the buffers are neither cache line aligned nor is the size a multiple
>     of the cache line size.  Using kmalloced buffers avoids this.
>     
>     Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
>     Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> ---
>  drivers/cdrom/cdrom.c |  274 ++++++++++++++++++++++++++++++++-----------------
>  1 files changed, 181 insertions(+), 93 deletions(-)

Eh... AFAICS this is only really useful in two of the cases converted.

For all the other cases (<= 32 bytes), it is _far_ less complex, far 
less code to simply communicate the additional alignment requirements to 
the compiler.

What about __attribute__ __aligned__?  Was that tried?

	Jeff

Re: cdrom: use kmalloced buffers instead of buffers on stack

[David Miller]

From: Jeff Garzik <jeff@garzik.org>
Date: Mon, 21 Apr 2008 22:01:26 -0400

> Eh... AFAICS this is only really useful in two of the cases converted.
> 
> For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> less code to simply communicate the additional alignment requirements to 
> the compiler.
> 
> What about __attribute__ __aligned__?  Was that tried?

On some platforms the stack is virtually mapped instead of
from kmalloc or get_free_pages memory.

DMA on stack buffers is really not feasible.

Re: cdrom: use kmalloced buffers instead of buffers on stack

[Thomas Bogendoerfer]

On Mon, Apr 21, 2008 at 10:01:26PM -0400, Jeff Garzik wrote:
> >    cdrom: use kmalloced buffers instead of buffers on stack
> >    
> >    If cdrom commands are issued to a scsi drive in most cases the buffer 
> >    will be
> >    filled via dma.  This leads to bad stack corruption on non coherent 
> >    platforms,
> >    because the buffers are neither cache line aligned nor is the size a 
> >    multiple
> >    of the cache line size.  Using kmalloced buffers avoids this.
> >    
> >    Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> >    Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> >---
> > drivers/cdrom/cdrom.c |  274 
> > ++++++++++++++++++++++++++++++++-----------------
> > 1 files changed, 181 insertions(+), 93 deletions(-)
> 
> Eh... AFAICS this is only really useful in two of the cases converted.
> 
> For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> less code to simply communicate the additional alignment requirements to 
> the compiler.
> 
> What about __attribute__ __aligned__?  Was that tried?

I used that while narrowing down the bug. But not only the alignment is
important, but also size needs to be a multiple of the cache line size.
Which means it needs to be 128 bytes for most SGI machines. That
and the following in DMA-mapping.txt

"This rule also means that you may use neither kernel image addresses
(items in data/text/bss segments), nor module image addresses, nor
stack addresses for DMA."

let me choose the kmalloc() solution.

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessary a
good idea.                                                [ RFC1925, 2.3 ]

Re: cdrom: use kmalloced buffers instead of buffers on stack

[Jens Axboe]

On Tue, Apr 22 2008, Thomas Bogendoerfer wrote:
> On Mon, Apr 21, 2008 at 10:01:26PM -0400, Jeff Garzik wrote:
> > >    cdrom: use kmalloced buffers instead of buffers on stack
> > >    
> > >    If cdrom commands are issued to a scsi drive in most cases the buffer 
> > >    will be
> > >    filled via dma.  This leads to bad stack corruption on non coherent 
> > >    platforms,
> > >    because the buffers are neither cache line aligned nor is the size a 
> > >    multiple
> > >    of the cache line size.  Using kmalloced buffers avoids this.
> > >    
> > >    Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> > >    Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> > >---
> > > drivers/cdrom/cdrom.c |  274 
> > > ++++++++++++++++++++++++++++++++-----------------
> > > 1 files changed, 181 insertions(+), 93 deletions(-)
> > 
> > Eh... AFAICS this is only really useful in two of the cases converted.
> > 
> > For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> > less code to simply communicate the additional alignment requirements to 
> > the compiler.
> > 
> > What about __attribute__ __aligned__?  Was that tried?
> 
> I used that while narrowing down the bug. But not only the alignment is
> important, but also size needs to be a multiple of the cache line size.
> Which means it needs to be 128 bytes for most SGI machines. That
> and the following in DMA-mapping.txt
> 
> "This rule also means that you may use neither kernel image addresses
> (items in data/text/bss segments), nor module image addresses, nor
> stack addresses for DMA."
> 
> let me choose the kmalloc() solution.

Which is good, the patch has been due for years :-)

-- 
Jens Axboe

Re: cdrom: use kmalloced buffers instead of buffers on stack

[FUJITA Tomonori]

On Tue, 22 Apr 2008 07:48:58 +0200
tsbogend@alpha.franken.de (Thomas Bogendoerfer) wrote:

> On Mon, Apr 21, 2008 at 10:01:26PM -0400, Jeff Garzik wrote:
> > >    cdrom: use kmalloced buffers instead of buffers on stack
> > >    
> > >    If cdrom commands are issued to a scsi drive in most cases the buffer 
> > >    will be
> > >    filled via dma.  This leads to bad stack corruption on non coherent 
> > >    platforms,
> > >    because the buffers are neither cache line aligned nor is the size a 
> > >    multiple
> > >    of the cache line size.  Using kmalloced buffers avoids this.
> > >    
> > >    Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> > >    Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> > >---
> > > drivers/cdrom/cdrom.c |  274 
> > > ++++++++++++++++++++++++++++++++-----------------
> > > 1 files changed, 181 insertions(+), 93 deletions(-)
> > 
> > Eh... AFAICS this is only really useful in two of the cases converted.
> > 
> > For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> > less code to simply communicate the additional alignment requirements to 
> > the compiler.
> > 
> > What about __attribute__ __aligned__?  Was that tried?
> 
> I used that while narrowing down the bug. But not only the alignment is
> important, but also size needs to be a multiple of the cache line size.
> Which means it needs to be 128 bytes for most SGI machines. That
> and the following in DMA-mapping.txt
> 
> "This rule also means that you may use neither kernel image addresses
> (items in data/text/bss segments), nor module image addresses, nor
> stack addresses for DMA."
> 
> let me choose the kmalloc() solution.

Can we advertise such architecture's dma restrictions? For example, if
we can update dma_pad_mask and dma_alignment in request_queue,
blk_rq_map_kern uses a proper bounce buffer for such
architectures. Then we can avoid putting extra complexity in uppper
drivers such as cdrom.c

Re: cdrom: use kmalloced buffers instead of buffers on stack

[Jens Axboe]

On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> On Tue, 22 Apr 2008 07:48:58 +0200
> tsbogend@alpha.franken.de (Thomas Bogendoerfer) wrote:
> 
> > On Mon, Apr 21, 2008 at 10:01:26PM -0400, Jeff Garzik wrote:
> > > >    cdrom: use kmalloced buffers instead of buffers on stack
> > > >    
> > > >    If cdrom commands are issued to a scsi drive in most cases the buffer 
> > > >    will be
> > > >    filled via dma.  This leads to bad stack corruption on non coherent 
> > > >    platforms,
> > > >    because the buffers are neither cache line aligned nor is the size a 
> > > >    multiple
> > > >    of the cache line size.  Using kmalloced buffers avoids this.
> > > >    
> > > >    Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> > > >    Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> > > >---
> > > > drivers/cdrom/cdrom.c |  274 
> > > > ++++++++++++++++++++++++++++++++-----------------
> > > > 1 files changed, 181 insertions(+), 93 deletions(-)
> > > 
> > > Eh... AFAICS this is only really useful in two of the cases converted.
> > > 
> > > For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> > > less code to simply communicate the additional alignment requirements to 
> > > the compiler.
> > > 
> > > What about __attribute__ __aligned__?  Was that tried?
> > 
> > I used that while narrowing down the bug. But not only the alignment is
> > important, but also size needs to be a multiple of the cache line size.
> > Which means it needs to be 128 bytes for most SGI machines. That
> > and the following in DMA-mapping.txt
> > 
> > "This rule also means that you may use neither kernel image addresses
> > (items in data/text/bss segments), nor module image addresses, nor
> > stack addresses for DMA."
> > 
> > let me choose the kmalloc() solution.
> 
> Can we advertise such architecture's dma restrictions? For example, if
> we can update dma_pad_mask and dma_alignment in request_queue,
> blk_rq_map_kern uses a proper bounce buffer for such
> architectures. Then we can avoid putting extra complexity in uppper
> drivers such as cdrom.c

That would work fine, if cdrom was then also updated to get rid of
->generic_packet() and use the regular queue transport instead. Which
would be a VERY nice thing to do anyway, so I'd welcome the effort :-)

-- 
Jens Axboe

Re: cdrom: use kmalloced buffers instead of buffers on stack

[FUJITA Tomonori]

On Tue, 22 Apr 2008 13:55:19 +0200
Jens Axboe <jens.axboe@oracle.com> wrote:

> On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> > On Tue, 22 Apr 2008 07:48:58 +0200
> > tsbogend@alpha.franken.de (Thomas Bogendoerfer) wrote:
> > 
> > > On Mon, Apr 21, 2008 at 10:01:26PM -0400, Jeff Garzik wrote:
> > > > >    cdrom: use kmalloced buffers instead of buffers on stack
> > > > >    
> > > > >    If cdrom commands are issued to a scsi drive in most cases the buffer 
> > > > >    will be
> > > > >    filled via dma.  This leads to bad stack corruption on non coherent 
> > > > >    platforms,
> > > > >    because the buffers are neither cache line aligned nor is the size a 
> > > > >    multiple
> > > > >    of the cache line size.  Using kmalloced buffers avoids this.
> > > > >    
> > > > >    Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> > > > >    Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> > > > >---
> > > > > drivers/cdrom/cdrom.c |  274 
> > > > > ++++++++++++++++++++++++++++++++-----------------
> > > > > 1 files changed, 181 insertions(+), 93 deletions(-)
> > > > 
> > > > Eh... AFAICS this is only really useful in two of the cases converted.
> > > > 
> > > > For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> > > > less code to simply communicate the additional alignment requirements to 
> > > > the compiler.
> > > > 
> > > > What about __attribute__ __aligned__?  Was that tried?
> > > 
> > > I used that while narrowing down the bug. But not only the alignment is
> > > important, but also size needs to be a multiple of the cache line size.
> > > Which means it needs to be 128 bytes for most SGI machines. That
> > > and the following in DMA-mapping.txt
> > > 
> > > "This rule also means that you may use neither kernel image addresses
> > > (items in data/text/bss segments), nor module image addresses, nor
> > > stack addresses for DMA."
> > > 
> > > let me choose the kmalloc() solution.
> > 
> > Can we advertise such architecture's dma restrictions? For example, if
> > we can update dma_pad_mask and dma_alignment in request_queue,
> > blk_rq_map_kern uses a proper bounce buffer for such
> > architectures. Then we can avoid putting extra complexity in uppper
> > drivers such as cdrom.c
> 
> That would work fine, if cdrom was then also updated to get rid of
> ->generic_packet() and use the regular queue transport instead. Which
> would be a VERY nice thing to do anyway, so I'd welcome the effort :-)

sr_packet calls ioctl so it works for SCSI at least?

Re: cdrom: use kmalloced buffers instead of buffers on stack

[Jens Axboe]

On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> On Tue, 22 Apr 2008 13:55:19 +0200
> Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> > > On Tue, 22 Apr 2008 07:48:58 +0200
> > > tsbogend@alpha.franken.de (Thomas Bogendoerfer) wrote:
> > > 
> > > > On Mon, Apr 21, 2008 at 10:01:26PM -0400, Jeff Garzik wrote:
> > > > > >    cdrom: use kmalloced buffers instead of buffers on stack
> > > > > >    
> > > > > >    If cdrom commands are issued to a scsi drive in most cases the buffer 
> > > > > >    will be
> > > > > >    filled via dma.  This leads to bad stack corruption on non coherent 
> > > > > >    platforms,
> > > > > >    because the buffers are neither cache line aligned nor is the size a 
> > > > > >    multiple
> > > > > >    of the cache line size.  Using kmalloced buffers avoids this.
> > > > > >    
> > > > > >    Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> > > > > >    Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> > > > > >---
> > > > > > drivers/cdrom/cdrom.c |  274 
> > > > > > ++++++++++++++++++++++++++++++++-----------------
> > > > > > 1 files changed, 181 insertions(+), 93 deletions(-)
> > > > > 
> > > > > Eh... AFAICS this is only really useful in two of the cases converted.
> > > > > 
> > > > > For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> > > > > less code to simply communicate the additional alignment requirements to 
> > > > > the compiler.
> > > > > 
> > > > > What about __attribute__ __aligned__?  Was that tried?
> > > > 
> > > > I used that while narrowing down the bug. But not only the alignment is
> > > > important, but also size needs to be a multiple of the cache line size.
> > > > Which means it needs to be 128 bytes for most SGI machines. That
> > > > and the following in DMA-mapping.txt
> > > > 
> > > > "This rule also means that you may use neither kernel image addresses
> > > > (items in data/text/bss segments), nor module image addresses, nor
> > > > stack addresses for DMA."
> > > > 
> > > > let me choose the kmalloc() solution.
> > > 
> > > Can we advertise such architecture's dma restrictions? For example, if
> > > we can update dma_pad_mask and dma_alignment in request_queue,
> > > blk_rq_map_kern uses a proper bounce buffer for such
> > > architectures. Then we can avoid putting extra complexity in uppper
> > > drivers such as cdrom.c
> > 
> > That would work fine, if cdrom was then also updated to get rid of
> > ->generic_packet() and use the regular queue transport instead. Which
> > would be a VERY nice thing to do anyway, so I'd welcome the effort :-)
> 
> sr_packet calls ioctl so it works for SCSI at least?

Yup, it'll work fine for sr currently, but not ide-cd. But we should
just get rid of ->generic_packet() and prepare a request in cdrom.c
instead.

-- 
Jens Axboe

Re: cdrom: use kmalloced buffers instead of buffers on stack

[FUJITA Tomonori]

On Tue, 22 Apr 2008 14:04:38 +0200
Jens Axboe <jens.axboe@oracle.com> wrote:

> On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> > On Tue, 22 Apr 2008 13:55:19 +0200
> > Jens Axboe <jens.axboe@oracle.com> wrote:
> > 
> > > On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> > > > On Tue, 22 Apr 2008 07:48:58 +0200
> > > > tsbogend@alpha.franken.de (Thomas Bogendoerfer) wrote:
> > > > 
> > > > > On Mon, Apr 21, 2008 at 10:01:26PM -0400, Jeff Garzik wrote:
> > > > > > >    cdrom: use kmalloced buffers instead of buffers on stack
> > > > > > >    
> > > > > > >    If cdrom commands are issued to a scsi drive in most cases the buffer 
> > > > > > >    will be
> > > > > > >    filled via dma.  This leads to bad stack corruption on non coherent 
> > > > > > >    platforms,
> > > > > > >    because the buffers are neither cache line aligned nor is the size a 
> > > > > > >    multiple
> > > > > > >    of the cache line size.  Using kmalloced buffers avoids this.
> > > > > > >    
> > > > > > >    Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> > > > > > >    Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> > > > > > >---
> > > > > > > drivers/cdrom/cdrom.c |  274 
> > > > > > > ++++++++++++++++++++++++++++++++-----------------
> > > > > > > 1 files changed, 181 insertions(+), 93 deletions(-)
> > > > > > 
> > > > > > Eh... AFAICS this is only really useful in two of the cases converted.
> > > > > > 
> > > > > > For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> > > > > > less code to simply communicate the additional alignment requirements to 
> > > > > > the compiler.
> > > > > > 
> > > > > > What about __attribute__ __aligned__?  Was that tried?
> > > > > 
> > > > > I used that while narrowing down the bug. But not only the alignment is
> > > > > important, but also size needs to be a multiple of the cache line size.
> > > > > Which means it needs to be 128 bytes for most SGI machines. That
> > > > > and the following in DMA-mapping.txt
> > > > > 
> > > > > "This rule also means that you may use neither kernel image addresses
> > > > > (items in data/text/bss segments), nor module image addresses, nor
> > > > > stack addresses for DMA."
> > > > > 
> > > > > let me choose the kmalloc() solution.
> > > > 
> > > > Can we advertise such architecture's dma restrictions? For example, if
> > > > we can update dma_pad_mask and dma_alignment in request_queue,
> > > > blk_rq_map_kern uses a proper bounce buffer for such
> > > > architectures. Then we can avoid putting extra complexity in uppper
> > > > drivers such as cdrom.c
> > > 
> > > That would work fine, if cdrom was then also updated to get rid of
> > > ->generic_packet() and use the regular queue transport instead. Which
> > > would be a VERY nice thing to do anyway, so I'd welcome the effort :-)
> > 
> > sr_packet calls ioctl so it works for SCSI at least?
> 
> Yup, it'll work fine for sr currently, but not ide-cd. But we should
> just get rid of ->generic_packet() and prepare a request in cdrom.c
> instead.

Ok, I'll see how things work after finishing the work to remove the
request structure on the stack.

So are you ok with a patch to make blk_rq_map_kern handle dma padding
and alignment propely?

http://marc.info/?l=linux-scsi&m=120860454709078&w=2

Re: cdrom: use kmalloced buffers instead of buffers on stack

[Jens Axboe]

On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> On Tue, 22 Apr 2008 14:04:38 +0200
> Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> > > On Tue, 22 Apr 2008 13:55:19 +0200
> > > Jens Axboe <jens.axboe@oracle.com> wrote:
> > > 
> > > > On Tue, Apr 22 2008, FUJITA Tomonori wrote:
> > > > > On Tue, 22 Apr 2008 07:48:58 +0200
> > > > > tsbogend@alpha.franken.de (Thomas Bogendoerfer) wrote:
> > > > > 
> > > > > > On Mon, Apr 21, 2008 at 10:01:26PM -0400, Jeff Garzik wrote:
> > > > > > > >    cdrom: use kmalloced buffers instead of buffers on stack
> > > > > > > >    
> > > > > > > >    If cdrom commands are issued to a scsi drive in most cases the buffer 
> > > > > > > >    will be
> > > > > > > >    filled via dma.  This leads to bad stack corruption on non coherent 
> > > > > > > >    platforms,
> > > > > > > >    because the buffers are neither cache line aligned nor is the size a 
> > > > > > > >    multiple
> > > > > > > >    of the cache line size.  Using kmalloced buffers avoids this.
> > > > > > > >    
> > > > > > > >    Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
> > > > > > > >    Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> > > > > > > >---
> > > > > > > > drivers/cdrom/cdrom.c |  274 
> > > > > > > > ++++++++++++++++++++++++++++++++-----------------
> > > > > > > > 1 files changed, 181 insertions(+), 93 deletions(-)
> > > > > > > 
> > > > > > > Eh... AFAICS this is only really useful in two of the cases converted.
> > > > > > > 
> > > > > > > For all the other cases (<= 32 bytes), it is _far_ less complex, far 
> > > > > > > less code to simply communicate the additional alignment requirements to 
> > > > > > > the compiler.
> > > > > > > 
> > > > > > > What about __attribute__ __aligned__?  Was that tried?
> > > > > > 
> > > > > > I used that while narrowing down the bug. But not only the alignment is
> > > > > > important, but also size needs to be a multiple of the cache line size.
> > > > > > Which means it needs to be 128 bytes for most SGI machines. That
> > > > > > and the following in DMA-mapping.txt
> > > > > > 
> > > > > > "This rule also means that you may use neither kernel image addresses
> > > > > > (items in data/text/bss segments), nor module image addresses, nor
> > > > > > stack addresses for DMA."
> > > > > > 
> > > > > > let me choose the kmalloc() solution.
> > > > > 
> > > > > Can we advertise such architecture's dma restrictions? For example, if
> > > > > we can update dma_pad_mask and dma_alignment in request_queue,
> > > > > blk_rq_map_kern uses a proper bounce buffer for such
> > > > > architectures. Then we can avoid putting extra complexity in uppper
> > > > > drivers such as cdrom.c
> > > > 
> > > > That would work fine, if cdrom was then also updated to get rid of
> > > > ->generic_packet() and use the regular queue transport instead. Which
> > > > would be a VERY nice thing to do anyway, so I'd welcome the effort :-)
> > > 
> > > sr_packet calls ioctl so it works for SCSI at least?
> > 
> > Yup, it'll work fine for sr currently, but not ide-cd. But we should
> > just get rid of ->generic_packet() and prepare a request in cdrom.c
> > instead.
> 
> Ok, I'll see how things work after finishing the work to remove the
> request structure on the stack.
> 
> So are you ok with a patch to make blk_rq_map_kern handle dma padding
> and alignment propely?
> 
> http://marc.info/?l=linux-scsi&m=120860454709078&w=2

I'm getting to that patch, from a cursory look it seems fine.

-- 
Jens Axboe