Re: [PATCH v5 4/5] Inter-VM shared memory PCI device

[Anthony Liguori <anthony@codemonkey.ws>]

On 05/10/2010 11:20 AM, Cam Macdonell wrote:
> On Mon, May 10, 2010 at 9:38 AM, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>    
>> On 05/10/2010 10:28 AM, Avi Kivity wrote:
>>      
>>> On 05/10/2010 06:22 PM, Cam Macdonell wrote:
>>>        
>>>>          
>>>>>            
>>>>>> +
>>>>>> +    /* if the position is -1, then it's shared memory region fd */
>>>>>> +    if (incoming_posn == -1) {
>>>>>> +
>>>>>> +        s->num_eventfds = 0;
>>>>>> +
>>>>>> +        if (check_shm_size(s, incoming_fd) == -1) {
>>>>>> +            exit(-1);
>>>>>> +        }
>>>>>> +
>>>>>> +        /* creating a BAR in qemu_chr callback may be crazy */
>>>>>> +        create_shared_memory_BAR(s, incoming_fd);
>>>>>>
>>>>>>              
>>>>> It probably is... why can't you create it during initialization?
>>>>>            
>>>> This is for the shared memory server implementation, so the fd for the
>>>> shared memory has to be received (over the qemu char device) from the
>>>> server before the BAR can be created via qemu_ram_mmap() which adds
>>>> the necessary memory
>>>>
>>>>          
>>>
>>> We could do the handshake during initialization.  I'm worried that the
>>> device will appear without the BAR, and strange things will happen.  But the
>>> chardev API is probably not geared for passing data during init.
>>>
>>> Anthony, any ideas?
>>>        
>> Why can't we create the BAR with just normal RAM and then change it to a
>> mmap()'d fd after initialization?  This will be behavior would be important
>> for live migration as it would let you quickly migrate preserving the memory
>> contents without waiting for an external program to reconnect.
>>
>> Regards,
>>
>> Anthony Lioguori
>>
>>      
>>>> Otherwise, if the BAR is allocated during initialization, I would have
>>>> to use MAP_FIXED to mmap the memory.  This is what I did before the
>>>> qemu_ram_mmap() function was added.
>>>>          
>>> What would happen to any data written to the BAR before the the handshake
>>> completed?  I think it would disappear.
>>>        
>> You don't have to do MAP_FIXED.  You can allocate a ram area and map that in
>> when disconnected.  When you connect, you create another ram area and
>> memcpy() the previous ram area to the new one.  You then map the second ram
>> area in.
>>      
> the memcpy() would overwrite the contents of the shared memory each
> time a guest joins which would be dangerous.
>    

I think those are reasonable semantics and is really the only way to get 
guest-transparent reconnect.  The later is pretty critical for guest 
transparent live migration.

>>  From the guest's perspective, it's totally transparent.  For the backend,
>> I'd suggest having an explicit "initialized" ack or something so that it
>> knows that the data is now mapped to the guest.
>>      
> Yes, I think the ack is the way to go, so the guest has to be aware of
> it.  Would setting a flag in the driver-specific config space be an
> acceptable ack that the shared region is now mapped?
>    

You know it's mapped because it's mapped when the pci map function 
returns.  You don't need the guest to explicitly tell you.

Regards,

Anthony Liguori

> Cam
>