[virtio-dev] Re: [PATCH v2 0/5] virtio mmio specification enhancement

[virtio-dev] Re: [PATCH v2 0/5] virtio mmio specification enhancement

[Alex Bennée]

Pincus, Josh <Josh.Pincus@windriver.com> writes:

> Hi,
>
>  
>
> We were looking into a similar enhancement for the Virt I/O MMIO transport and came across this project.
>
> This enhancement would be perfect for us.

So there is certainly an interest in optimising MMIO based virtio and
the current read/ack cycle adds additional round trip time for any trap
and emulate hypervisor. However I think there is some resistance to
making MMIO a re-implementation of what PCI already gives us for "free".

I believe the current questions that need to be addressed are:

  - Clear definitions in the spec on doorbells/notifications

    The current virtio spec uses different terms in some places so it
    would be nice to clarify the language and formalise what the
    standard expects from transports w.r.t the capabilities of
    notifications and doorbells.

  - Quantifying the memory foot-print difference between PCI/MMIO

    PCI gives a lot for free including a discovery and IRQ model already
    designed to handle MSI/MSI-X. There is a claim that this brings in a
    lot of bloat but I think there was some debate around the numbers.
    My rough initial experiment with a PCI and non-PCI build with
    otherwise identical VIRTIO configs results in the following:

    16:40:15 c.282% [alex@zen:~/l/l/builds] review/rpmb|… + ls -l arm64/vmlinux arm64.nopci/vmlinux
    -rwxr-xr-x 1 alex alex 83914728 Jul 31 16:39 arm64.nopci/vmlinux*
    -rwxr-xr-x 1 alex alex 86368080 Jul 31 16:33 arm64/vmlinux*

    which certainly implies there could be a fair amount of headroom for
    an MMIO version to implement some features. However I don't know if
    it's fully apples to apples as there maybe unneeded PCI bloat that a
    virtio-only kernel doesn't need.


What are the features you are most interested in?

> Has there been any progress since Feb, 2020?  It looks like the effort
> might have stalled?

I can't speak to the OP's but there is certainly interest from others
that are not the original posters.


-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v2 0/5] virtio mmio specification enhancement

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> Pincus, Josh <Josh.Pincus@windriver.com> writes:
>
>> Hi,
>>
>>  
>>
>> We were looking into a similar enhancement for the Virt I/O MMIO transport and came across this project.
>>
>> This enhancement would be perfect for us.
>
> So there is certainly an interest in optimising MMIO based virtio and
> the current read/ack cycle adds additional round trip time for any trap
> and emulate hypervisor. However I think there is some resistance to
> making MMIO a re-implementation of what PCI already gives us for "free".
<snip>
>
>   - Quantifying the memory foot-print difference between PCI/MMIO
>
>     PCI gives a lot for free including a discovery and IRQ model already
>     designed to handle MSI/MSI-X. There is a claim that this brings in a
>     lot of bloat but I think there was some debate around the numbers.
>     My rough initial experiment with a PCI and non-PCI build with
>     otherwise identical VIRTIO configs results in the following:
>
>     16:40:15 c.282% [alex@zen:~/l/l/builds] review/rpmb|… + ls -l arm64/vmlinux arm64.nopci/vmlinux
>     -rwxr-xr-x 1 alex alex 83914728 Jul 31 16:39 arm64.nopci/vmlinux*
>     -rwxr-xr-x 1 alex alex 86368080 Jul 31 16:33 arm64/vmlinux*
>
>     which certainly implies there could be a fair amount of headroom for
>     an MMIO version to implement some features. However I don't know if
>     it's fully apples to apples as there maybe unneeded PCI bloat that a
>     virtio-only kernel doesn't need.

Just following up after cutting the Xgene and ThunderX PCI bloat from
the kernel the margin is a little smaller:

  -rwxr-xr-x 1 alex alex 83914728 Jul 31 16:39 arm64.nopci/vmlinux*
  -rwxr-xr-x 1 alex alex 85639808 Aug  3 17:12 arm64/vmlinux*

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v2 0/5] virtio mmio specification enhancement

[Michael S. Tsirkin]

On Mon, Feb 10, 2020 at 05:05:16PM +0800, Zha Bin wrote:
> We have compared the number of files and the lines of code between
> virtio-mmio and virio-pci.
> 
> 				Virtio-PCI	    Virtio-MMIO	
> 	number of files(Linux)	    161			1
> 	lines of code(Linux)	    78237		538



Something's very wrong here. virtio PCI is 161 files?
Are you counting the whole PCI subsystem?
Sure enough:

$ find drivers/pci -name '*c' |wc -l
150

That's not reasonable, this includes a bunch of drivers that
never run on a typical hypervisor.

MMIO is also not as small as you are trying to show:

$ cloc drivers/virtio/virtio_mmio.c include/uapi/linux/virtio_mmio.h
       2 text files.
       2 unique files.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.82  T=0.01 s (230.7 files/s, 106126.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                                1            144            100            535
C/C++ Header                     1             39             66             36
-------------------------------------------------------------------------------
SUM:                             2            183            166            571
-------------------------------------------------------------------------------


I don't doubt MMIO is smaller than PCI. Of course that's because it has
no features to speak of - just this patch already doubles it's size. If
we keep doing that because we want the features then they will reach
the same size in about 4 iterations.


-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v2 0/5] virtio mmio specification enhancement

[Michael S. Tsirkin]

On Tue, Feb 11, 2020 at 04:05:41PM +0000, Chao Peng wrote:
> On Mon, Feb 10, 2020 at 06:44:50AM -0500, Michael S. Tsirkin wrote:
> > On Mon, Feb 10, 2020 at 05:05:16PM +0800, Zha Bin wrote:
> > > We have compared the number of files and the lines of code between
> > > virtio-mmio and virio-pci.
> > > 
> > > 				Virtio-PCI	    Virtio-MMIO	
> > > 	number of files(Linux)	    161			1
> > > 	lines of code(Linux)	    78237		538
> > 
> > 
> > 
> > Something's very wrong here. virtio PCI is 161 files?
> > Are you counting the whole PCI subsystem?
> 
> Right, that is just a rough statistics.

Please try not to make them look so wrong then.
E.g. you don't include drivers/base/platform-msi.c for
mmio do you? Your patch brings a bunch of code in there.

> Surely enough, some drivers will
> never get enabled in a typcial config.
> 
> > Sure enough:
> > 
> > $ find drivers/pci -name '*c' |wc -l
> > 150
> 
> and plus:
> $ find arch/x86/pci/ -name '*c' |wc -l
> 22

But what's the point? This is code that is maintained by PCI core
people anyway.

> > 
> > That's not reasonable, this includes a bunch of drivers that
> > never run on a typical hypervisor.
> > 
> > MMIO is also not as small as you are trying to show:
> > 
> > $ cloc drivers/virtio/virtio_mmio.c include/uapi/linux/virtio_mmio.h
> >        2 text files.
> >        2 unique files.                              
> >        0 files ignored.
> > 
> > github.com/AlDanial/cloc v 1.82  T=0.01 s (230.7 files/s, 106126.5 lines/s)
> > -------------------------------------------------------------------------------
> > Language                     files          blank        comment           code
> > -------------------------------------------------------------------------------
> > C                                1            144            100            535
> > C/C++ Header                     1             39             66             36
> > -------------------------------------------------------------------------------
> > SUM:                             2            183            166            571
> > -------------------------------------------------------------------------------
> > 
> > 
> > I don't doubt MMIO is smaller than PCI. Of course that's because it has
> > no features to speak of - just this patch already doubles it's size. If
> > we keep doing that because we want the features then they will reach
> > the same size in about 4 iterations.
> 
> Since current virtio-mmio size is small enough, so adding any notable
> feature would easily double it.

But really unlike PCI this is just PV stuff that is not reused by
anyone. We end up maintaining all this by ourselves.

> I have no objection that it may one day
> reach the same level of PCI, but in this patch some are actually
> generic changes and for MSI specific code we provide the option to
> confige away.
> 
> Thanks,
> Chao

The option will make it fall down at runtime but
it does not actually seem to remove all of the overhead.



> > 
> > 
> > -- 
> > MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org