Re: [Intel-wired-lan] [PATCH iwl-next v4 08/12] ice: Save and load RX Queue head

[Jacob Keller <jacob.e.keller@intel.com>]

On 12/7/2023 6:53 PM, Tian, Kevin wrote:
>> From: Jason Gunthorpe <jgg@nvidia.com>
>> Sent: Thursday, December 7, 2023 10:46 PM
>>
>> On Thu, Dec 07, 2023 at 07:55:17AM +0000, Tian, Kevin wrote:
>>>> From: Cao, Yahui <yahui.cao@intel.com>
>>>> Sent: Tuesday, November 21, 2023 10:51 AM
>>>>
>>>> +
>>>> +		/* Once RX Queue is enabled, network traffic may come in at
>>>> any
>>>> +		 * time. As a result, RX Queue head needs to be loaded
>>>> before
>>>> +		 * RX Queue is enabled.
>>>> +		 * For simplicity and integration, overwrite RX head just after
>>>> +		 * RX ring context is configured.
>>>> +		 */
>>>> +		if (msg_slot->opcode == VIRTCHNL_OP_CONFIG_VSI_QUEUES)
>>>> {
>>>> +			ret = ice_migration_load_rx_head(vf, devstate);
>>>> +			if (ret) {
>>>> +				dev_err(dev, "VF %d failed to load rx head\n",
>>>> +					vf->vf_id);
>>>> +				goto out_clear_replay;
>>>> +			}
>>>> +		}
>>>> +
>>>
>>> Don't we have the same problem here as for TX head restore that the
>>> vfio migration protocol doesn't carry a way to tell whether the IOAS
>>> associated with the device has been restored then allowing RX DMA
>>> at this point might cause device error?
>>
>> Does this trigger a DMA?
> 
> looks yes from the comment
> 
>>
>>> @Jason, is it a common gap applying to all devices which include a
>>> receiving path from link? How is it handled in mlx migration
>>> driver?
>>
>> There should be no DMA until the device is placed in RUNNING. All
>> devices may instantly trigger DMA once placed in RUNNING.
>>
>> The VMM must ensure the entire environment is ready to go before
>> putting anything in RUNNING, including having setup the IOMMU.
>>
> 
> ah, yes. that is the right behavior.
> 
> so if there is no other way to block DMA before RUNNING is reached,
> here the RX queue should be left disabled until when transitioning 
> to RUNNING.
> 
> Yahui, can you double check?

I think this will require us to ensure that the Rx queue restoration
happens later. I'm looking at refactoring the approach to use an
internal representation of VF state instead of a series of virtchnl
messages, and this should allow us to avoid enabling the Rx queue until
after we're ready for DMA.