Direct transfer between SCSI and PCI

Direct transfer between SCSI and PCI

[Steve Rottinger]

Hi All,

I am working with version 2.6.23 of the linux kernel on a custom PowerPC 
board.  As part
of my project, I need to be able to directly transfer data from a large
buffer that resides on
a custom PCI communications device to a SCSI disk without going through
the CPU's main
memory.  Initially, it seemed like opening up the raw SCSI device in
O_DIRECT mode would
be the perfect solution.  However, through some experimentation, and
reading some posts from
a few years back, it appears that O_DIRECT only works on buffer's that
reside in processor's
main memory.   Does anyone have any suggestions as to how to get this to
work?  My environment
is pretty well controlled, so I can patch/hack the kernel without any
problem.

Thanks in advance,

-Steve

Direct transfer between SCSI and PCI

[Steve Rottinger]

Hi All,

I am working with version 2.6.23 of the linux kernel on a custom PowerPC
board.  As part
of my project, I need to be able to directly transfer data from a large
buffer that resides on
a custom PCI communications device to a SCSI disk without going through
the CPU's main
memory.  Initially, it seemed like opening up the raw SCSI device in
O_DIRECT mode would
be the perfect solution.  However, through some experimentation, and
reading some posts from
a few years back, it appears that O_DIRECT only works on buffer's that
reside in processor's
main memory.   Does anyone have any suggestions as to how to get this to
work?  My environment
is pretty well controlled, so I can patch/hack the kernel without any
problem.

Thanks in advance,

-Steve

Re: Direct transfer between SCSI and PCI

[Jiri Slaby]

On 9.3.2009 14:29, Steve Rottinger wrote:
>As part
> of my project, I need to be able to directly transfer data from a large
> buffer that resides on
> a custom PCI communications device to a SCSI disk without going through
> the CPU's main
> memory.  Initially, it seemed like opening up the raw SCSI device in
> O_DIRECT mode would
> be the perfect solution.  However, through some experimentation, and
> reading some posts from
> a few years back, it appears that O_DIRECT only works on buffer's that
> reside in processor's
> main memory.   Does anyone have any suggestions as to how to get this to
> work?

How would you want your scsi device to get the data? I suppose they are 
stored somewhere on the PCI device memory connected to its local bus or 
alike. At best it can be visible as a PCI bar but I doubt it is. As you 
described it here, there is no way, how would the SCSI device see the data.

Or do you want to move the data from the PCI device via DMA implemented 
in it directly to the SCSI device? How would you do this? I guess SCSI 
devices do standard S/G DMA so that they read data from host memory by 
themselves.

Maybe if you elaborate on what exactly do you mean by the idea?

Re: Direct transfer between SCSI and PCI

[Steve Rottinger]

The SCSI device would use it's DMA controller to copy the data, just as it
would if the buffer was in the processor's RAM.  If I understand
correctly, if I open up the SCSI device using the O_DIRECT option, the
driver will DMA directly to/from the user buffer specified by the
write()/read() functions. Instead, of that buffer being an address
pointing to the processors RAM, I would like that buffer to point to the
PCI address of the buffer on my PCI device. Currently, when I try this on
an mmapped buffer to my PCI device, the write() routine returns a "Bad
address" error.

-Steve


> On 9.3.2009 14:29, Steve Rottinger wrote:
>>As part
>> of my project, I need to be able to directly transfer data from a large
>> buffer that resides on
>> a custom PCI communications device to a SCSI disk without going through
>> the CPU's main
>> memory.  Initially, it seemed like opening up the raw SCSI device in
>> O_DIRECT mode would
>> be the perfect solution.  However, through some experimentation, and
>> reading some posts from
>> a few years back, it appears that O_DIRECT only works on buffer's that
>> reside in processor's
>> main memory.   Does anyone have any suggestions as to how to get this to
>> work?
>
> How would you want your scsi device to get the data? I suppose they are
> stored somewhere on the PCI device memory connected to its local bus or
> alike. At best it can be visible as a PCI bar but I doubt it is. As you
> described it here, there is no way, how would the SCSI device see the
> data.
>
> Or do you want to move the data from the PCI device via DMA implemented
> in it directly to the SCSI device? How would you do this? I guess SCSI
> devices do standard S/G DMA so that they read data from host memory by
> themselves.
>
> Maybe if you elaborate on what exactly do you mean by the idea?
>