[virtio-dev] Re: [virtio-comment] Re: [PATCH 09/11] transport-pci: Describe PCI MMR dev config registers

[Parav Pandit <parav@nvidia.com>]

On 4/13/2023 1:14 AM, Jason Wang wrote:

>> For LM BAR length and number should be same between two PCI VFs. But its
>> orthogonal to this point. Such checks will be done anyway.
> 
> Quoted the wrong sections, I think it should be:
> 
> "
> length MAY include padding, or fields unused by the driver, or future
> extensions. Note: For example, a future device might present a large
> structure size of several MBytes. As current devices never utilize
> structures larger than 4KBytes in size, driver MAY limit the mapped
> structure size to e.g. 4KBytes (thus ignoring parts of structure after
> the first 4KBytes) to allow forward compatibility with such devices
> without loss of functionality and without wasting resources.
> "
yes. This is the one.

> If it's a transitional device but not placed at BAR0, it might have
> side effects for Linux drivers which assumes BAR0 for legacy.
> 
True. Transitional can be at BAR0.

> I don't see how easy it could be a non transitional device:
> 
> "
> Devices or drivers with no legacy compatibility are referred to as
> non-transitional devices and drivers, respectively.
> "
Michael has suggested rewording of the text.
It is anyway new text so lets park it aside for now.
It is mostly tweaking the text.

> 
>> device can expose this optional capability and its attached MMIO region.
>>
>> Spec changes are similar to #2.
>>> - non trivial spec changes which ends up of the tricky cases that
>>> tries to workaround legacy to fit for a hardware implementation
>>> - work only for the case of virtualization with the help of
>>> meditation, can't work for bare metal
>> For bare-metal PFs usually thin hypervisors are used that does very
>> minimal setup. But I agree that bare-metal is relatively less important.
> 
> This is not what I understand. I know several vendors that are using
> virtio devices for bare metal.
> 
I was saying the case for legacy bare metal is less of a problem because 
PCIe does not limit functionality, perf is still limited due to IOBAR.

>>
>>> - only work for some specific archs without SVQ
>>>
>> That is the legacy limitation that we don't worry about.
>>
>>> 2) allow BAR0 to be MMIO for transitional device
>>>
>>> Pros:
>>> - very minor change for the spec
>> Spec changes wise they are similar to #1.
> 
> This is different since the changes for this are trivial.
> 
>>> - work for virtualization (and it work even without dedicated
>>> mediation for some setups)
>> I am not aware where can it work without mediation. Do you know any
>> specific kernel version where it actually works?
> 
> E.g current Linux driver did:
> 
> rc = pci_request_region(pci_dev, 0, "virtio-pci-legacy");
> 
> It doesn't differ from I/O with memory. It means if you had a
> "transitional" device with legacy MMIO BAR0, it just works.
> 

Thanks to the abstract PCI API in Linux.

>>> - work for bare metal for some setups (without mediation)
>>> Cons:
>>> - only work for some specific archs without SVQ
>>> - BAR0 is required
>>>
>> Both are not limitation as they are mainly coming from the legacy side
>> of things.
>>
>>> 3) modern device mediation for legacy
>>>
>>> Pros:
>>> - no changes in the spec
>>> Cons:
>>> - require mediation layer in order to work in bare metal
>>> - require datapath mediation like SVQ to work for virtualization
>>>
>> Spec change is still require for net and blk because modern device do
>> not understand legacy, even with mediation layer.
> 
> That's fine and easy since we work on top of modern devices.
> 
>> FEATURE_1, RW cap via CVQ which is not really owned by the hypervisor.
> 
> Hypervisors can trap if they wish.
> 
Trapping non legacy accessing for 1.x doesn't make sense.

>> A guest may be legacy or non legacy, so mediation shouldn't be always done.
> 
> Yes, so mediation can work only if we found it's a legacy driver.
> 
Mediation will be done only for legacy accesses without cvq, rest will 
go as-is without any cvq and other mediation.

>>
>>> Compared to method 2) the only advantages of method 1) is the
>>> flexibility of BAR0 but it has too many disadvantages. If we only care
>>> about virtualization, modern devices are sufficient. Then why bother
>>> for that?
>>
>> So that a single stack which doesn't always have the knowledge of which
>> driver version is running is guest can utilize it. Otherwise 1.x also
>> end up doing mediation when guest driver = 1.x and device = transitional
>> PCI VF.
> 
> I don't see how this can be solved in your proposal.
> 
This proposal only traps the legacy accesses and doesnt require other 
giant framework.

I think we can make the BAR0 work for transitional with spec change and 
with optional notification region.
I am evaluating further.

>>
>> so (1) and (2) both are equivalent, one is more flexible, if you know
>> more valid cases where BAR0 as MMIO can work as_is, such option is open.
> 
> As said in previous threads, this has been used by several vendors for years.
> 
> E.g I have a handy transitional hardware virtio device that has:
> 
>          Region 0: Memory at f5ff0000 (64-bit, prefetchable) [size=8K]
>          Region 2: Memory at f5fe0000 (64-bit, prefetchable) [size=4K]
>          Region 4: Memory at f5800000 (64-bit, prefetchable) [size=4M]
> 
> And:
> 
>          Capabilities: [64] Vendor Specific Information: VirtIO: CommonCfg
>                  BAR=0 offset=00000888 size=00000078
>          Capabilities: [74] Vendor Specific Information: VirtIO: Notify
>                  BAR=0 offset=00001800 size=00000020 multiplier=00000000
>          Capabilities: [88] Vendor Specific Information: VirtIO: ISR
>                  BAR=0 offset=00000820 size=00000020
>          Capabilities: [98] Vendor Specific Information: VirtIO: DeviceCfg
>                  BAR=0 offset=00000840 size=00000020
> 
>>
>> We can draft the spec that MMIO BAR SHOULD be exposes in BAR0.
>>
Yes, above one.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org