[PATCH 0/3] virtio: Clean up scatterlists and use the DMA API

[PATCH 0/3] virtio: Clean up scatterlists and use the DMA API

[Christopher Covington]

On 08/27/2014 11:50 AM, Andy Lutomirski wrote:
> On Wed, Aug 27, 2014 at 8:45 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> On Wed, Aug 27, 2014 at 08:11:15AM -0700, Andy Lutomirski wrote:
>>> On Aug 26, 2014 11:46 PM, "Stefan Hajnoczi" <stefanha@gmail.com> wrote:
>>>>
>>>> On Tue, Aug 26, 2014 at 10:16 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>>>> There are two outstanding issues.  virtio_net warns if DMA debugging
>>>>> is on because it does DMA from the stack.  (The warning is correct.)
>>>>> This also is likely to do something unpleasant to s390.
>>>>> (Maintainers are cc'd -- I don't know what to do about it.)
>>>>
>>>> This changes the semantics of vring and breaks existing guests when
>>>> bus address != physical address.
>>>>
>>>> Can you use a transport feature bit to indicate that bus addresses are
>>>> used?  That way both approaches can be supported.
>>>
>>> I can try to support both styles of addressing, but I don't think that
>>> this can be negotiated between the device (i.e. host or physical
>>> virtio-speaking device) and the guest.  In the Xen case that I care
>>> about (Linux on Xen on KVM), the host doesn't know about the
>>> translation at all -- Xen is an intermediate layer that only the guest
>>> is aware of.  In this case, there are effectively two layers of
>>> virtualization, and only the inner one (Xen) knows about the
>>> translation despite the fact that the the outer layer is the one
>>> providing the virtio device.
>>>
>>> I could change virtio_ring to use the DMA API only if requested by the
>>> lower driver (virtio_pci, etc) and to have only virtio_pci enable that
>>> feature.  Will that work for all cases?
>>>
>>> On s390, this shouldn't work just like the current code.  On x86, I
>>> think that if QEMU ever starts exposing an IOMMU attached to a
>>> virtio-pci device, then QEMU should expect that IOMMU to be used.  If
>>> QEMU expects to see physical addresses, then it shouldn't advertise an
>>> IOMMU.  Since QEMU doesn't currently support guest IOMMUs, this should
>>> be fine for everything that uses QEMU.
>>>
>>> At least x86's implementation of the DMA ops for devices that aren't
>>> behind an IOMMU should be very fast.
>>>
>>> Are there any other weird cases for which this might be a problem?
>>>
>>>>
>>>> Please also update the virtio specification:
>>>> https://tools.oasis-open.org/version-control/browse/wsvn/virtio/
>>>>
>>>
>>> I'm not sure it will need an update.  Perhaps a note in the PCI
>>> section indicating that, if the host expects the guest to program an
>>> IOMMU, then it should use the appropriate platform-specific mechanism
>>> to expose that IOMMU.
>>>
>>> --Andy
>>
>> If there's no virtio mechanism to negotate enabling/disabling
>> translations, then specification does not need an extension.
> 
> It wouldn't shock me if someone wants to negotiate this for
> virtio_mmio some day, but that might be more of a device tree thing.
> I have no idea how that works, but I think it can wait until someone
> wants it.

At one point I wanted VirtIO-MMIO to not fail miserably on ARM LPAE systems.

http://lists.infradead.org/pipermail/linux-arm-kernel/2014-March/241613.html

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[PATCH 0/3] virtio: Clean up scatterlists and use the DMA API

[Andy Lutomirski]

On Wed, Aug 27, 2014 at 9:13 AM, Christopher Covington
<cov@codeaurora.org> wrote:
> On 08/27/2014 11:50 AM, Andy Lutomirski wrote:
>> On Wed, Aug 27, 2014 at 8:45 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
>>> On Wed, Aug 27, 2014 at 08:11:15AM -0700, Andy Lutomirski wrote:
>>>> On Aug 26, 2014 11:46 PM, "Stefan Hajnoczi" <stefanha@gmail.com> wrote:
>>>>>
>>>>> On Tue, Aug 26, 2014 at 10:16 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>>>>> There are two outstanding issues.  virtio_net warns if DMA debugging
>>>>>> is on because it does DMA from the stack.  (The warning is correct.)
>>>>>> This also is likely to do something unpleasant to s390.
>>>>>> (Maintainers are cc'd -- I don't know what to do about it.)
>>>>>
>>>>> This changes the semantics of vring and breaks existing guests when
>>>>> bus address != physical address.
>>>>>
>>>>> Can you use a transport feature bit to indicate that bus addresses are
>>>>> used?  That way both approaches can be supported.
>>>>
>>>> I can try to support both styles of addressing, but I don't think that
>>>> this can be negotiated between the device (i.e. host or physical
>>>> virtio-speaking device) and the guest.  In the Xen case that I care
>>>> about (Linux on Xen on KVM), the host doesn't know about the
>>>> translation at all -- Xen is an intermediate layer that only the guest
>>>> is aware of.  In this case, there are effectively two layers of
>>>> virtualization, and only the inner one (Xen) knows about the
>>>> translation despite the fact that the the outer layer is the one
>>>> providing the virtio device.
>>>>
>>>> I could change virtio_ring to use the DMA API only if requested by the
>>>> lower driver (virtio_pci, etc) and to have only virtio_pci enable that
>>>> feature.  Will that work for all cases?
>>>>
>>>> On s390, this shouldn't work just like the current code.  On x86, I
>>>> think that if QEMU ever starts exposing an IOMMU attached to a
>>>> virtio-pci device, then QEMU should expect that IOMMU to be used.  If
>>>> QEMU expects to see physical addresses, then it shouldn't advertise an
>>>> IOMMU.  Since QEMU doesn't currently support guest IOMMUs, this should
>>>> be fine for everything that uses QEMU.
>>>>
>>>> At least x86's implementation of the DMA ops for devices that aren't
>>>> behind an IOMMU should be very fast.
>>>>
>>>> Are there any other weird cases for which this might be a problem?
>>>>
>>>>>
>>>>> Please also update the virtio specification:
>>>>> https://tools.oasis-open.org/version-control/browse/wsvn/virtio/
>>>>>
>>>>
>>>> I'm not sure it will need an update.  Perhaps a note in the PCI
>>>> section indicating that, if the host expects the guest to program an
>>>> IOMMU, then it should use the appropriate platform-specific mechanism
>>>> to expose that IOMMU.
>>>>
>>>> --Andy
>>>
>>> If there's no virtio mechanism to negotate enabling/disabling
>>> translations, then specification does not need an extension.
>>
>> It wouldn't shock me if someone wants to negotiate this for
>> virtio_mmio some day, but that might be more of a device tree thing.
>> I have no idea how that works, but I think it can wait until someone
>> wants it.
>
> At one point I wanted VirtIO-MMIO to not fail miserably on ARM LPAE systems.
>
> http://lists.infradead.org/pipermail/linux-arm-kernel/2014-March/241613.html
>

Since I know nothing about LPAE, I'll leave this to you :)  But I'll
cc you on patch v2 soon, and please tell me whether my code makes
sense on ARM.

(My attempt to boot-test s390 failed because I have no clue what
command-line options to pass to QEMU.  If anyone wants to give me some
pointers to get a working configuration with -kernel and some kind of
console, I can add support to virtme.  Alas, I think that no one ever
bothered to implement 9p over virtio-ccw in QEMU.  Why exactly does
the virtio stuff in QEMU require that you instantiate virtio-9p-pci
instead of just asking for an appropriate virtio dievice?)

--Andy

[PATCH 0/3] virtio: Clean up scatterlists and use the DMA API

[Christopher Covington]

On 08/27/2014 12:19 PM, Andy Lutomirski wrote:
> On Wed, Aug 27, 2014 at 9:13 AM, Christopher Covington
> <cov@codeaurora.org> wrote:
>> On 08/27/2014 11:50 AM, Andy Lutomirski wrote:
>>> On Wed, Aug 27, 2014 at 8:45 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
>>>> On Wed, Aug 27, 2014 at 08:11:15AM -0700, Andy Lutomirski wrote:
>>>>> On Aug 26, 2014 11:46 PM, "Stefan Hajnoczi" <stefanha@gmail.com> wrote:
>>>>>>
>>>>>> On Tue, Aug 26, 2014 at 10:16 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>>>>>> There are two outstanding issues.  virtio_net warns if DMA debugging
>>>>>>> is on because it does DMA from the stack.  (The warning is correct.)
>>>>>>> This also is likely to do something unpleasant to s390.
>>>>>>> (Maintainers are cc'd -- I don't know what to do about it.)
>>>>>>
>>>>>> This changes the semantics of vring and breaks existing guests when
>>>>>> bus address != physical address.
>>>>>>
>>>>>> Can you use a transport feature bit to indicate that bus addresses are
>>>>>> used?  That way both approaches can be supported.
>>>>>
>>>>> I can try to support both styles of addressing, but I don't think that
>>>>> this can be negotiated between the device (i.e. host or physical
>>>>> virtio-speaking device) and the guest.  In the Xen case that I care
>>>>> about (Linux on Xen on KVM), the host doesn't know about the
>>>>> translation at all -- Xen is an intermediate layer that only the guest
>>>>> is aware of.  In this case, there are effectively two layers of
>>>>> virtualization, and only the inner one (Xen) knows about the
>>>>> translation despite the fact that the the outer layer is the one
>>>>> providing the virtio device.
>>>>>
>>>>> I could change virtio_ring to use the DMA API only if requested by the
>>>>> lower driver (virtio_pci, etc) and to have only virtio_pci enable that
>>>>> feature.  Will that work for all cases?
>>>>>
>>>>> On s390, this shouldn't work just like the current code.  On x86, I
>>>>> think that if QEMU ever starts exposing an IOMMU attached to a
>>>>> virtio-pci device, then QEMU should expect that IOMMU to be used.  If
>>>>> QEMU expects to see physical addresses, then it shouldn't advertise an
>>>>> IOMMU.  Since QEMU doesn't currently support guest IOMMUs, this should
>>>>> be fine for everything that uses QEMU.
>>>>>
>>>>> At least x86's implementation of the DMA ops for devices that aren't
>>>>> behind an IOMMU should be very fast.
>>>>>
>>>>> Are there any other weird cases for which this might be a problem?
>>>>>
>>>>>>
>>>>>> Please also update the virtio specification:
>>>>>> https://tools.oasis-open.org/version-control/browse/wsvn/virtio/
>>>>>>
>>>>>
>>>>> I'm not sure it will need an update.  Perhaps a note in the PCI
>>>>> section indicating that, if the host expects the guest to program an
>>>>> IOMMU, then it should use the appropriate platform-specific mechanism
>>>>> to expose that IOMMU.
>>>>>
>>>>> --Andy
>>>>
>>>> If there's no virtio mechanism to negotate enabling/disabling
>>>> translations, then specification does not need an extension.
>>>
>>> It wouldn't shock me if someone wants to negotiate this for
>>> virtio_mmio some day, but that might be more of a device tree thing.
>>> I have no idea how that works, but I think it can wait until someone
>>> wants it.
>>
>> At one point I wanted VirtIO-MMIO to not fail miserably on ARM LPAE systems.
>>
>> http://lists.infradead.org/pipermail/linux-arm-kernel/2014-March/241613.html
>>
> 
> Since I know nothing about LPAE, I'll leave this to you :)  But I'll
> cc you on patch v2 soon, and please tell me whether my code makes
> sense on ARM.
>
> (My attempt to boot-test s390 failed because I have no clue what
> command-line options to pass to QEMU.  If anyone wants to give me some
> pointers to get a working configuration with -kernel and some kind of
> console, I can add support to virtme.  Alas, I think that no one ever
> bothered to implement 9p over virtio-ccw in QEMU.  Why exactly does
> the virtio stuff in QEMU require that you instantiate virtio-9p-pci
> instead of just asking for an appropriate virtio dievice?)

Virtme looks interesting. If it's any use, here is my modest QEMU command line
collection.

$dir/bin/x86_64-linux-gnu/qemu-system-x86_64 \
  -m 1024 \
  -kernel $dir/boot/x86_64-linux-gnu/bzImage-x86_64 \
  -append 'root=/dev/root init=/sbin/x86_64-linux-gnu/init rootfstype=9p
rootflags=trans=virtio,version=9p2000.L ro console=ttyS0' \
  -serial stdio \
  -fsdev local,id=root,path=$dir,security_model=none \
  -device virtio-9p-pci,fsdev=root,mount_tag=/dev/root \
  -monitor none

$dir/bin/x86_64-linux-gnu/qemu-system-arm \
  -M virt \
  -cpu cortex-a15 \
  -m 1024 \
  -kernel $dir/boot/arm-linux-gnueabihf/Image \
  -append 'root=/dev/root init=/sbin/arm-linux-gnueabihf/init rootfstype=9p
rootflags=trans=virtio,version=9p2000.L ro console=ttyAMA0' \
  -serial stdio \
  -fsdev local,id=root,path=$dir,security_model=none \
  -device virtio-9p-device,fsdev=root,mount_tag=/dev/root \
  -monitor none

$dir/bin/x86_64-linux-gnu/qemu-system-aarch64 \
  -M virt \
  -cpu cortex-a57 \
  -m 1024 \
  -kernel $dir/boot/aarch64-linux-gnu/Image \
  -append 'root=/dev/root init=/sbin/aarch64-linux-gnu/init rootfstype=9p
rootflags=trans=virtio,version=9p2000.L ro console=ttyAMA0' \
  -serial stdio \
  -fsdev local,id=root,path=$dir,security_model=none \
  -device virtio-9p-device,fsdev=root,mount_tag=/dev/root -monitor none

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

[PATCH 0/3] virtio: Clean up scatterlists and use the DMA API

[Andy Lutomirski]

On Wed, Aug 27, 2014 at 10:34 AM, Christopher Covington
<cov@codeaurora.org> wrote:
> On 08/27/2014 12:19 PM, Andy Lutomirski wrote:
>> On Wed, Aug 27, 2014 at 9:13 AM, Christopher Covington
>> <cov@codeaurora.org> wrote:
>
> Virtme looks interesting. If it's any use, here is my modest QEMU command line
> collection.
>
> $dir/bin/x86_64-linux-gnu/qemu-system-x86_64 \
>   -m 1024 \
>   -kernel $dir/boot/x86_64-linux-gnu/bzImage-x86_64 \
>   -append 'root=/dev/root init=/sbin/x86_64-linux-gnu/init rootfstype=9p
> rootflags=trans=virtio,version=9p2000.L ro console=ttyS0' \
>   -serial stdio \
>   -fsdev local,id=root,path=$dir,security_model=none \
>   -device virtio-9p-pci,fsdev=root,mount_tag=/dev/root \
>   -monitor none

This is similar to what virtme does, except that virtme will give you
a much more functional system at the cost of a small fraction of a
second of additional boot time.  And virtme sometimes supports modules
:)

>
> $dir/bin/x86_64-linux-gnu/qemu-system-arm \
>   -M virt \
>   -cpu cortex-a15 \
>   -m 1024 \
>   -kernel $dir/boot/arm-linux-gnueabihf/Image \
>   -append 'root=/dev/root init=/sbin/arm-linux-gnueabihf/init rootfstype=9p
> rootflags=trans=virtio,version=9p2000.L ro console=ttyAMA0' \
>   -serial stdio \
>   -fsdev local,id=root,path=$dir,security_model=none \
>   -device virtio-9p-device,fsdev=root,mount_tag=/dev/root \
>   -monitor none

Can you give me some hints as to how to configure a kernel for this?
I tried to get anything other than versatilepb working and gave up.

>
> $dir/bin/x86_64-linux-gnu/qemu-system-aarch64 \
>   -M virt \
>   -cpu cortex-a57 \
>   -m 1024 \
>   -kernel $dir/boot/aarch64-linux-gnu/Image \
>   -append 'root=/dev/root init=/sbin/aarch64-linux-gnu/init rootfstype=9p
> rootflags=trans=virtio,version=9p2000.L ro console=ttyAMA0' \
>   -serial stdio \
>   -fsdev local,id=root,path=$dir,security_model=none \
>   -device virtio-9p-device,fsdev=root,mount_tag=/dev/root -monitor none
>

Ooh, thanks!  I'll add support for this to virtme.

--Andy

[PATCH 0/3] virtio: Clean up scatterlists and use the DMA API

[Christopher Covington]

On 08/27/2014 01:37 PM, Andy Lutomirski wrote:
> On Wed, Aug 27, 2014 at 10:34 AM, Christopher Covington
> <cov@codeaurora.org> wrote:
>> On 08/27/2014 12:19 PM, Andy Lutomirski wrote:
>>> On Wed, Aug 27, 2014 at 9:13 AM, Christopher Covington
>>> <cov@codeaurora.org> wrote:
>>
>> Virtme looks interesting. If it's any use, here is my modest QEMU command line
>> collection.
>>
>> $dir/bin/x86_64-linux-gnu/qemu-system-x86_64 \
>>   -m 1024 \
>>   -kernel $dir/boot/x86_64-linux-gnu/bzImage-x86_64 \
>>   -append 'root=/dev/root init=/sbin/x86_64-linux-gnu/init rootfstype=9p
>> rootflags=trans=virtio,version=9p2000.L ro console=ttyS0' \
>>   -serial stdio \
>>   -fsdev local,id=root,path=$dir,security_model=none \
>>   -device virtio-9p-pci,fsdev=root,mount_tag=/dev/root \
>>   -monitor none
> 
> This is similar to what virtme does, except that virtme will give you
> a much more functional system at the cost of a small fraction of a
> second of additional boot time.  And virtme sometimes supports modules
> :)
> 
>>
>> $dir/bin/x86_64-linux-gnu/qemu-system-arm \
>>   -M virt \
>>   -cpu cortex-a15 \
>>   -m 1024 \
>>   -kernel $dir/boot/arm-linux-gnueabihf/Image \
>>   -append 'root=/dev/root init=/sbin/arm-linux-gnueabihf/init rootfstype=9p
>> rootflags=trans=virtio,version=9p2000.L ro console=ttyAMA0' \
>>   -serial stdio \
>>   -fsdev local,id=root,path=$dir,security_model=none \
>>   -device virtio-9p-device,fsdev=root,mount_tag=/dev/root \
>>   -monitor none
> 
> Can you give me some hints as to how to configure a kernel for this?
> I tried to get anything other than versatilepb working and gave up.

I think you can just use vexpress_defconfig. If you want to from there you can
unselect CONFIG_ARCH_VEXPRESS and set CONFIG_ARCH_VIRT=y, but I don't think
that's strictly necessary.

>> $dir/bin/x86_64-linux-gnu/qemu-system-aarch64 \
>>   -M virt \
>>   -cpu cortex-a57 \
>>   -m 1024 \
>>   -kernel $dir/boot/aarch64-linux-gnu/Image \
>>   -append 'root=/dev/root init=/sbin/aarch64-linux-gnu/init rootfstype=9p
>> rootflags=trans=virtio,version=9p2000.L ro console=ttyAMA0' \
>>   -serial stdio \
>>   -fsdev local,id=root,path=$dir,security_model=none \
>>   -device virtio-9p-device,fsdev=root,mount_tag=/dev/root -monitor none
>>
> 
> Ooh, thanks!  I'll add support for this to virtme.

Cheers,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.