Re: Integration of SCST in the mainstream Linux kernel

[Vladislav Bolkhovitin <vst@vlnb.net>]

James Bottomley wrote:
> On Mon, 2008-02-04 at 20:56 +0300, Vladislav Bolkhovitin wrote:
> 
>>James Bottomley wrote:
>>
>>>On Mon, 2008-02-04 at 20:16 +0300, Vladislav Bolkhovitin wrote:
>>>
>>>
>>>>James Bottomley wrote:
>>>>
>>>>
>>>>>>>>So, James, what is your opinion on the above? Or the overall SCSI target 
>>>>>>>>project simplicity doesn't matter much for you and you think it's fine 
>>>>>>>>to duplicate Linux page cache in the user space to keep the in-kernel 
>>>>>>>>part of the project as small as possible?
>>>>>>>
>>>>>>>
>>>>>>>The answers were pretty much contained here
>>>>>>>
>>>>>>>http://marc.info/?l=linux-scsi&m=120164008302435
>>>>>>>
>>>>>>>and here:
>>>>>>>
>>>>>>>http://marc.info/?l=linux-scsi&m=120171067107293
>>>>>>>
>>>>>>>Weren't they?
>>>>>>
>>>>>>No, sorry, it doesn't look so for me. They are about performance, but 
>>>>>>I'm asking about the overall project's architecture, namely about one 
>>>>>>part of it: simplicity. Particularly, what do you think about 
>>>>>>duplicating Linux page cache in the user space to have zero-copy cached 
>>>>>>I/O? Or can you suggest another architectural solution for that problem 
>>>>>>in the STGT's approach?
>>>>>
>>>>>
>>>>>Isn't that an advantage of a user space solution?  It simply uses the
>>>>>backing store of whatever device supplies the data.  That means it takes
>>>>>advantage of the existing mechanisms for caching.
>>>>
>>>>No, please reread this thread, especially this message: 
>>>>http://marc.info/?l=linux-kernel&m=120169189504361&w=2. This is one of 
>>>>the advantages of the kernel space implementation. The user space 
>>>>implementation has to have data copied between the cache and user space 
>>>>buffer, but the kernel space one can use pages in the cache directly, 
>>>>without extra copy.
>>>
>>>
>>>Well, you've said it thrice (the bellman cried) but that doesn't make it
>>>true.
>>>
>>>The way a user space solution should work is to schedule mmapped I/O
>>>from the backing store and then send this mmapped region off for target
>>>I/O.  For reads, the page gather will ensure that the pages are up to
>>>date from the backing store to the cache before sending the I/O out.
>>>For writes, You actually have to do a msync on the region to get the
>>>data secured to the backing store. 
>>
>>James, have you checked how fast is mmaped I/O if work size > size of 
>>RAM? It's several times slower comparing to buffered I/O. It was many 
>>times discussed in LKML and, seems, VM people consider it unavoidable. 
> 
> 
> Erm, but if you're using the case of work size > size of RAM, you'll
> find buffered I/O won't help because you don't have the memory for
> buffers either.

James, just check and you will see, buffered I/O is a lot faster.

>>So, using mmaped IO isn't an option for high performance. Plus, mmaped 
>>IO isn't an option for high reliability requirements, since it doesn't 
>>provide a practical way to handle I/O errors.
> 
> I think you'll find it does ... the page gather returns -EFAULT if
> there's an I/O error in the gathered region. 

Err, to whom return? If you try to read from a mmaped page, which can't 
be populated due to I/O error, you will get SIGBUS or SIGSEGV, I don't 
remember exactly. It's quite tricky to get back to the faulted command 
from the signal handler.

Or do you mean mmap(MAP_POPULATE)/munmap() for each command? Do you 
think that such mapping/unmapping is good for performance?

> msync does something
> similar if there's a write failure.
> 
>>>You also have to pull tricks with
>>>the mmap region in the case of writes to prevent useless data being read
>>>in from the backing store.
>>
>>Can you be more exact and specify what kind of tricks should be done for 
>>that?
> 
> Actually, just avoid touching it seems to do the trick with a recent
> kernel.

Hmm, how can one write to an mmaped page and don't touch it?

> James
> 
> 
>