Re: 2.6.9-rc2 bio sickness with large writes

Jeff V. Merkey
2004-09-20T19:55:38Z

Jens Axboe wrote:

>>page and offset sematics in the interface are also somewhat burdensome.
>>Wouldn't a more reasonable
>>interface for async IO be:
>>
>>address
>>length
>>address
>>length
>>
>>rather than
>>
>>page structure
>>offset in page structure
>>page structure
>>offset in page structure
>>
>>
>
>No, because { address, length } cannot fully describe all memory in any
>given machine.
>
>
>
This response I don't understand. How memory is described in a machine
for DMA addressibility
is pretty standard (with the exception of memory on intel machine in 32
bit systems above 4GBthat need page tables) --
a physical numerical address. But who is going to DMA into memory not in
the address space.

>Any chunk of memory has a page associated with it, but it may not have a
>kernel address mapping associated with it. So some identifier was needed
>other than a virtual address, a page is as good as any so making one up
>would be silly.
>
>Once you understand this, it doesn't seem so odd. You need to pass in a
>single page or sg table to map for dma anyways, the sg table holds page
>pointers as well.
>
>
>
>>I can assume from the interface as designed that if you pass an offset
>>for a page that is not page aligned,
>>and ask for a 4K write, then you will end up dropping the data on the
>>floor than spans beyond the end of the page.
>>
>>
>
>What kind of bogus example is that? Asking for a 4K write from a 4K page
>but asking to start 1K in that page is just stupid and not even remotely
>valid.
>
>
>
Hardware doesn't care about page boundries. It sees hardware addresses
and lengths, at
least most SG hardware I've worked with does. For ease of submission, an
interface that
takes <address,length> would suffice. Why on earth would someone need a
context
pointer into the kernel's page tables to submit an SG into a device,
apart from performing
virtual-to-physical translation?

>It's not difficult at all. Apparently you don't understand it so you
>think it's difficult, that's only natural. But you have access to the
>page mapping of any given piece of data always, or if you have the
>virtual address only it's trivial to go to the { page, offset } mapping.
>
>
No, I do understand, and using page/offset at a low level SG interface
IS burdensome.
I mean, if this is what I have to support I guess I have to use it, but
it will be just another
section of code where I have another **FAT** layer to waste more CPU
cycles calculating
offset/page (oh yeah I have to lookup the struct page * structure also)
when it would be much
simpler to just submit address/len in i386 systems. With this type of
interface, If I have for instance
an on-disk structure that starts in the middle of a 4K page due to other
headers, etc. than spans
a page, I cannot just submit the address and length, I have to break it
into two bio requests instead
of one with a for () loop from hell and calculate the offsets and rumage
around in memory looking
up struct page * addresses.

>I can only imagine that you are used to a very different interface on
>some other OS so you think it's difficult to use. Most of your
>complaints seem to be based on false assumptions or because you don't
>understand why certain design decisions were made.
>
>
>

No. I am used to programming to hardware with SG devices that all OS
use. Is there somewhere a page based
SG device (other than SCI) for disk drive?. I don't think so, I think
they operate address/len, address/len, etc.

:-)

Jeff