Re: [Qemu-devel] [PATCH v11 14/15] rdma: introduce MIG_STATE_NONE and change MIG_STATE_SETUP state transition

["Michael R. Hines" <mrhines@linux.vnet.ibm.com>]

On 06/26/2013 02:37 AM, Paolo Bonzini wrote:
> Il 26/06/2013 02:31, Michael R. Hines ha scritto:
>> On 06/25/2013 05:06 PM, Paolo Bonzini wrote:
>>> Il 25/06/2013 22:56, Michael R. Hines ha scritto:
>>>> I was wrong - this does require a protocol extension.
>>>>
>>>> This is because the RDMA transfers are asynchronous, and thus
>>>> we cannot know in advance that it is safe to unregister the memory
>>>> associated with each individual transfer before the transfer actually
>>>> completes.
>>>>
>>>> While the destination currently uses the protocol to participate in
>>>> *registering* the page, the destination does not participate in the
>>>> RDMA transfers themselves, only the source does, and thus would
>>>> require a new exchange of messages to block and instruct the
>>>> destination to unpin the memory.
>>> Yes, that's what I recalled too (really what mst told me :)).  Does it
>>> need to be blocking though?  As long as the pinning is blocking, and
>>> messages are processed in order, the source can proceed immediately
>>> after sending an unpin message.  This assumes of course that the chunk
>>> is not being transmitted, and I am not sure how easy the source can
>>> determine that.
>> No, they're not processed in order. In fact, not only does the device
>> write out of order, but also the PCI bus writes out of order.
>> This was such a problem in fact, that I fixed several bugs as a result
>> a few weeks ago (v7 of the patch with an in-depth description).
>>
>> The destination simply cannot assume whatsoever what the ordering
>> of the writes are - that's really the whole point of using RDMA in the
>> first place so that the software can get out of the way of the transfer
>> process to lower the latency of each transfer.
> The memory is processed out of order, but what about the messages?
> Those must be in order.
>
> Note that I wrote above "This assumes of course that the chunk is not
> being transmitted".  Can the source know when an asynchronous transfer
> finished, and delay the unpinning until that time?
>
> Paolo

Yes, the source does know. There's no problem unpinning on the source.

But both sides must do the unpinning, not just the source.

Did I misunderstand you? Are you suggesting *only* unpinning on the source?

- Michael