Re: Fatal crash with NFS, AIO & tcp retransmit

[Alex Bligh <alex@alex.org.uk>]

--On 23 January 2013 18:13:34 +0000 "Myklebust, Trond" 
<Trond.Myklebust@netapp.com> wrote:

>> They can't disappear until they have been successfully transmitted and a
>> response received.  The problem here is that there were two requests
>> sent or being sent and the page(s) can't be released until everyone,
>> including TCP and such, are done with them.
>>
>> 		ps
>
> Right. The O_DIRECT write() system call will not return until it gets a
> reply. Similarly, we don't mark an aio/dio request as complete until it
> too gets a reply. So the data for those requests that need
> retransmission is still available to be resent through the socket.

I apologise for my stupidity here as I think I must be missing something.

I thought we'd established that Xen's grant system doesn't release the page
until QEMU says the block I/O is complete. QEMU only states that the block
I/O is complete when AIO says it is. What's happening (as far as I can tell
from the oops) is that the grant system is releasing the page AFTER the aio
request is complete (and dio may the same), but at that stage the page is
still referenced by the tcp stack. That contradicts what you say about not
marking the aio/dio request as complete until it gets a reply, unless it's
the case that you can get a reply to a request when there is still data
that the TCP stack can ask to retransmit (I suppose that's conceivable
if the reply gets sent before the ACK of the data received).

My understanding (which may well be completely wrong) is that the problem
was that xen was unmapping the page even though it still had kernel
references to it. This is why the problem does not happen in kvm (which
does not as I understand it do a similar map/unmap operation). From Ian C I
understand that just looking at the number of kernel references is not
sufficient.

-- 
Alex Bligh