How to directly pass a dma physical address and length from SCSI upper layer to LLD without mapping the dma memory to kernel pages

How to directly pass a dma physical address and length from SCSI upper layer to LLD without mapping the dma memory to kernel pages

[Pan, Weimin]

SCSI upper layer can pass a scatterlist to middle layer and LLD. Normally the page_link, length, and offset is set in the scatterlist. LLD will convert to dma_address from pages by itself. That means the dma physical memory has to map to kernel memory space before it can be passed to LLD for data transfer. If I have a large dma memroy and it doesn't need to be touched by kernel or user mode Apps, it is a performance penalty to force to do that.

Is there a way to directly pass a dma physical address from upper layer to LLD (like use dma_address in a scatterlist)?
I looked at a couple of LLD drivers and none of them handle this kind of situation.

Re: How to directly pass a dma physical address and length from SCSI upper layer to LLD without mapping the dma memory to kernel pages

[James Bottomley]

On Mon, 2008-06-09 at 20:52 +0000, Pan, Weimin wrote:
> SCSI upper layer can pass a scatterlist to middle layer and LLD.
> Normally the page_link, length, and offset is set in the scatterlist.
> LLD will convert to dma_address from pages by itself. That means the
> dma physical memory has to map to kernel memory space before it can be
> passed to LLD for data transfer.

Actually, it's for exactly the reason that we *don't* want to map the
data into kernel space that we use scatterlists.  A scatterlist has the
struct page (what you call the page link) which can be used to convert
to dma_address independently of any virtual mapping.

If you're saying that the submission path from user space has to have a
mapping, then yes ... for the reason that the user presumably had
something to do with the data they're requesting ...

>  If I have a large dma memroy and it doesn't need to be touched by
> kernel or user mode Apps, it is a performance penalty to force to do
> that.

I don't quite see how a user application needs to do disk I/O on data it
doesn't touch.  If you're writing out, what are you writing out and if
you're reading in, you're not likely to throw the data away are you?
Somewhere, somehow, the data has to be made visible to the user
application ... and via memory mappings are pretty standard ways of
doing it.

> Is there a way to directly pass a dma physical address from upper
> layer to LLD (like use dma_address in a scatterlist)?
> I looked at a couple of LLD drivers and none of them handle this kind
> of situation.

Tell us where the data comes from if it's not in the application address
space and we might be better placed to answer a question like this ...
is this something to do with I2O?

James

RE: How to directly pass a dma physical address and length from SCSI upper layer to LLD without mapping the dma memory to kernel pages

[James Bottomley]

[please don't drop the mailing list from the cc because others might be
interested]

On Mon, 2008-06-09 at 21:19 +0000, Pan, Weimin wrote:
> One simple example is a router. If I want to router data from one
> interface (front-end) to another (back-end), I only need to know where
> the data is (physical memory address DMAed in). I can pass this
> address to another interface driver to move the data out (DMAed out).

There's not really such thing as a "router" use for the SCSI mid-layer.
The closest we come is the target infrastructure.  For user space
implementations like STGT there's no choice, the vma is what you have to
hang the implementation on.  For kernel implementations like SCST you
can get away without having any mappings.

> But I like to put the command process in user space. So I only need to
> know where the data is (from ioctl) and pass this to sg driver (for
> example). Right now, I can pass a S/G list to sg but sg driver assumes
> it is a virtual memory address list. I don't mind to write a new upper
> layer driver to handle physical address S/G instead of user space
> virtual address S/G. But I don't want to change middle layer and LLD
> interface. I saw middler layer passes scatterlist to LLD. But LLD
> always assumes the scatterlist's page_link is set and dma_address is
> not set. Is it possible that I set up dma_address=phy memory address
> and page_link=0? I just don't understand why SCSI subsystem doesn't
> clearly define this usage model.

Then you want to look at how STGT does it (in fact, you probably want to
be using STGT because that's the way it was designed).  However, there's
no getting away from the need for a user space mapping for the data in
that case.  On the flip side, there's not a huge performance overhead
having the mapping set up.  If you don't have a mapping somewhere, the
pages aren't pinned in the page cache and may be reused.

James

RE: How to directly pass a dma physical address and length from SCSI upper layer to LLD without mapping the dma memory to kernel pages

[Pan, Weimin]

>> One simple example is a router. If I want to router data from one
>> interface (front-end) to another (back-end), I only need to know where
>> the data is (physical memory address DMAed in). I can pass this
>> address to another interface driver to move the data out (DMAed out).

>There's not really such thing as a "router" use for the SCSI mid-layer.
>The closest we come is the target infrastructure.  For user space implementations like STGT there's no choice, the vma is what >you have to hang the implementation on.  For kernel implementations like SCST you can get away without having any mappings.

How can you get away without having any mappings in SCST? Do you mean that we can pass physical S/G list without mapping these physical memory into kernel space? Since we have large physical memory (>1GB), there is a performance hit by mapping it into kernel space using HI_MEM.


>> But I like to put the command process in user space. So I only need to
>>know where the data is (from ioctl) and pass this to sg driver (for
>> example). Right now, I can pass a S/G list to sg but sg driver assumes
>> it is a virtual memory address list. I don't mind to write a new upper
>> layer driver to handle physical address S/G instead of user space
>> virtual address S/G. But I don't want to change middle layer and LLD
>> interface. I saw middler layer passes scatterlist to LLD. But LLD
>> always assumes the scatterlist's page_link is set and dma_address is
>> not set. Is it possible that I set up dma_address=phy memory address
>> and page_link=0? I just don't understand why SCSI subsystem doesn't
>> clearly define this usage model.

>Then you want to look at how STGT does it (in fact, you probably want to be using STGT because that's the way it was
>designed).  However, there's no getting away from the need for a user space mapping for the data in that case.  On the flip
>side, there's not a huge performance overhead having the mapping set up.  If you don't have a mapping somewhere, the pages
>aren't pinned in the page cache and may be reused.

I thought that SCST has a virtual user mode handler which can do the cmd process in user space (the same as STGT). Is that a true statement?

Weimin

-----Original Message-----
From: James Bottomley [mailto:James.Bottomley@HansenPartnership.com]
Sent: Monday, June 09, 2008 4:44 PM
To: Pan, Weimin
Cc: linux-scsi
Subject: RE: How to directly pass a dma physical address and length from SCSI upper layer to LLD without mapping the dma memory to kernel pages

[please don't drop the mailing list from the cc because others might be interested]

On Mon, 2008-06-09 at 21:19 +0000, Pan, Weimin wrote:
> One simple example is a router. If I want to router data from one
> interface (front-end) to another (back-end), I only need to know where
> the data is (physical memory address DMAed in). I can pass this
> address to another interface driver to move the data out (DMAed out).

There's not really such thing as a "router" use for the SCSI mid-layer.
The closest we come is the target infrastructure.  For user space implementations like STGT there's no choice, the vma is what you have to hang the implementation on.  For kernel implementations like SCST you can get away without having any mappings.

> But I like to put the command process in user space. So I only need to
> know where the data is (from ioctl) and pass this to sg driver (for
> example). Right now, I can pass a S/G list to sg but sg driver assumes
> it is a virtual memory address list. I don't mind to write a new upper
> layer driver to handle physical address S/G instead of user space
> virtual address S/G. But I don't want to change middle layer and LLD
> interface. I saw middler layer passes scatterlist to LLD. But LLD
> always assumes the scatterlist's page_link is set and dma_address is
> not set. Is it possible that I set up dma_address=phy memory address
> and page_link=0? I just don't understand why SCSI subsystem doesn't
> clearly define this usage model.

Then you want to look at how STGT does it (in fact, you probably want to be using STGT because that's the way it was designed).  However, there's no getting away from the need for a user space mapping for the data in that case.  On the flip side, there's not a huge performance overhead having the mapping set up.  If you don't have a mapping somewhere, the pages aren't pinned in the page cache and may be reused.

James