[Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Xie, Huawei]

I am developing virtio user space poll mode network driver. We allocate vring physical memory from huge page. On VMs with less than 4GB  memory, it works well.  But on VMs with like 8GB memory, huge page are all allocated from high end memory.
So would virtio support 64bit address for vring virtqueue?

Thanks

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Stefan Hajnoczi]

On Wed, Aug 28, 2013 at 03:18:39AM +0000, Xie, Huawei wrote:
> I am developing virtio user space poll mode network driver. We allocate vring physical memory from huge page. On VMs with less than 4GB  memory, it works well.  But on VMs with like 8GB memory, huge page are all allocated from high end memory.
> So would virtio support 64bit address for vring virtqueue?

The vring takes guest physical addresses and the C type is __u64 (see
/usr/include/linux/virtio_ring.h).  64-bit addresses are fine.

Stefan

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Xie, Huawei]

Hi Stefan:
I think you mention the descriptor address? I mean the vring PFN register.
/* A 32-bit r/w PFN for the currently selected queue */
#define VIRTIO_PCI_QUEUE_PFN		8

And the linux driver sample code:
iowrite32(virt_to_phys(info->queue) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT,
		  vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);

-----Original Message-----
From: Stefan Hajnoczi [mailto:stefanha@gmail.com] 
Sent: Wednesday, August 28, 2013 4:07 PM
To: Xie, Huawei
Cc: qemu-devel@nongnu.org; rusty@rustcorp.com.au; Stefan Hajnoczi
Subject: Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

On Wed, Aug 28, 2013 at 03:18:39AM +0000, Xie, Huawei wrote:
> I am developing virtio user space poll mode network driver. We allocate vring physical memory from huge page. On VMs with less than 4GB  memory, it works well.  But on VMs with like 8GB memory, huge page are all allocated from high end memory.
> So would virtio support 64bit address for vring virtqueue?

The vring takes guest physical addresses and the C type is __u64 (see /usr/include/linux/virtio_ring.h).  64-bit addresses are fine.

Stefan

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Laszlo Ersek]

On 08/28/13 11:22, Xie, Huawei wrote:
> Hi Stefan:
> I think you mention the descriptor address? I mean the vring PFN register.
> /* A 32-bit r/w PFN for the currently selected queue */
> #define VIRTIO_PCI_QUEUE_PFN		8
> 
> And the linux driver sample code:
> iowrite32(virt_to_phys(info->queue) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT,
> 		  vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
> 
> -----Original Message-----
> From: Stefan Hajnoczi [mailto:stefanha@gmail.com] 
> Sent: Wednesday, August 28, 2013 4:07 PM
> To: Xie, Huawei
> Cc: qemu-devel@nongnu.org; rusty@rustcorp.com.au; Stefan Hajnoczi
> Subject: Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?
> 
> On Wed, Aug 28, 2013 at 03:18:39AM +0000, Xie, Huawei wrote:
>> I am developing virtio user space poll mode network driver. We allocate vring physical memory from huge page. On VMs with less than 4GB  memory, it works well.  But on VMs with like 8GB memory, huge page are all allocated from high end memory.
>> So would virtio support 64bit address for vring virtqueue?
> 
> The vring takes guest physical addresses and the C type is __u64 (see /usr/include/linux/virtio_ring.h).  64-bit addresses are fine.

If you mean Queue Address field in the Virtio Header (section 2.2.2 in
the virtio-0.9.5 specification), then please see 2.3 Virtqueue
Configuration, step 3. You have to divide the guest-phys address by 4096
and store the quotient. The Queue Address field takes a page frame
number, not a page frame address.

This allows it to address up to 2^32 * 4096 == 16T bytes of RAM.

Laszlo

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Christian Borntraeger]

On 28/08/13 13:45, Laszlo Ersek wrote:
> On 08/28/13 11:22, Xie, Huawei wrote:
>> Hi Stefan:
>> I think you mention the descriptor address? I mean the vring PFN register.
>> /* A 32-bit r/w PFN for the currently selected queue */
>> #define VIRTIO_PCI_QUEUE_PFN		8
>>
>> And the linux driver sample code:
>> iowrite32(virt_to_phys(info->queue) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT,
>> 		  vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
>>
>> -----Original Message-----
>> From: Stefan Hajnoczi [mailto:stefanha@gmail.com] 
>> Sent: Wednesday, August 28, 2013 4:07 PM
>> To: Xie, Huawei
>> Cc: qemu-devel@nongnu.org; rusty@rustcorp.com.au; Stefan Hajnoczi
>> Subject: Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?
>>
>> On Wed, Aug 28, 2013 at 03:18:39AM +0000, Xie, Huawei wrote:
>>> I am developing virtio user space poll mode network driver. We allocate vring physical memory from huge page. On VMs with less than 4GB  memory, it works well.  But on VMs with like 8GB memory, huge page are all allocated from high end memory.
>>> So would virtio support 64bit address for vring virtqueue?
>>
>> The vring takes guest physical addresses and the C type is __u64 (see /usr/include/linux/virtio_ring.h).  64-bit addresses are fine.
> 
> If you mean Queue Address field in the Virtio Header (section 2.2.2 in
> the virtio-0.9.5 specification), then please see 2.3 Virtqueue
> Configuration, step 3. You have to divide the guest-phys address by 4096
> and store the quotient. The Queue Address field takes a page frame
> number, not a page frame address.
> 
> This allows it to address up to 2^32 * 4096 == 16T bytes of RAM.

To be even more precise. This is only true for virtio-pci. virtio-ccw on s390x can define the
queue address as 64bit.

Christian

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Xie, Huawei]

I know it is PFN, that is why I want to use the high 12bit for combined write in another reply, to fully address 64 bit address.
Is the 16TB address space what it is originally designed for? If it is, that will basically solve the problem. I will try later.
Thanks for reminder

-----Original Message-----
From: Laszlo Ersek [mailto:lersek@redhat.com] 
Sent: Wednesday, August 28, 2013 7:46 PM
To: Xie, Huawei
Cc: Stefan Hajnoczi; rusty@rustcorp.com.au; qemu-devel@nongnu.org; Stefan Hajnoczi
Subject: Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

On 08/28/13 11:22, Xie, Huawei wrote:
> Hi Stefan:
> I think you mention the descriptor address? I mean the vring PFN register.
> /* A 32-bit r/w PFN for the currently selected queue */
> #define VIRTIO_PCI_QUEUE_PFN		8
> 
> And the linux driver sample code:
> iowrite32(virt_to_phys(info->queue) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT,
> 		  vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
> 
> -----Original Message-----
> From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
> Sent: Wednesday, August 28, 2013 4:07 PM
> To: Xie, Huawei
> Cc: qemu-devel@nongnu.org; rusty@rustcorp.com.au; Stefan Hajnoczi
> Subject: Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?
> 
> On Wed, Aug 28, 2013 at 03:18:39AM +0000, Xie, Huawei wrote:
>> I am developing virtio user space poll mode network driver. We allocate vring physical memory from huge page. On VMs with less than 4GB  memory, it works well.  But on VMs with like 8GB memory, huge page are all allocated from high end memory.
>> So would virtio support 64bit address for vring virtqueue?
> 
> The vring takes guest physical addresses and the C type is __u64 (see /usr/include/linux/virtio_ring.h).  64-bit addresses are fine.

If you mean Queue Address field in the Virtio Header (section 2.2.2 in the virtio-0.9.5 specification), then please see 2.3 Virtqueue Configuration, step 3. You have to divide the guest-phys address by 4096 and store the quotient. The Queue Address field takes a page frame number, not a page frame address.

This allows it to address up to 2^32 * 4096 == 16T bytes of RAM.

Laszlo

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Anthony Liguori]

Yes, it was originally designed with the 16TB limit in mind.

PCI doesn't support 64-bit PIO operations so it would have required a
high/low register and additional magic.

When/if we switch to an MMIO region, we can switch to using a proper
64-bit register for setting up queues.

Regards,

Anthony Liguori



On Wed, Aug 28, 2013 at 10:30 AM, Xie, Huawei <huawei.xie@intel.com> wrote:
> I know it is PFN, that is why I want to use the high 12bit for combined write in another reply, to fully address 64 bit address.
> Is the 16TB address space what it is originally designed for? If it is, that will basically solve the problem. I will try later.
> Thanks for reminder
>
> -----Original Message-----
> From: Laszlo Ersek [mailto:lersek@redhat.com]
> Sent: Wednesday, August 28, 2013 7:46 PM
> To: Xie, Huawei
> Cc: Stefan Hajnoczi; rusty@rustcorp.com.au; qemu-devel@nongnu.org; Stefan Hajnoczi
> Subject: Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?
>
> On 08/28/13 11:22, Xie, Huawei wrote:
>> Hi Stefan:
>> I think you mention the descriptor address? I mean the vring PFN register.
>> /* A 32-bit r/w PFN for the currently selected queue */
>> #define VIRTIO_PCI_QUEUE_PFN          8
>>
>> And the linux driver sample code:
>> iowrite32(virt_to_phys(info->queue) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT,
>>                 vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
>>
>> -----Original Message-----
>> From: Stefan Hajnoczi [mailto:stefanha@gmail.com]
>> Sent: Wednesday, August 28, 2013 4:07 PM
>> To: Xie, Huawei
>> Cc: qemu-devel@nongnu.org; rusty@rustcorp.com.au; Stefan Hajnoczi
>> Subject: Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?
>>
>> On Wed, Aug 28, 2013 at 03:18:39AM +0000, Xie, Huawei wrote:
>>> I am developing virtio user space poll mode network driver. We allocate vring physical memory from huge page. On VMs with less than 4GB  memory, it works well.  But on VMs with like 8GB memory, huge page are all allocated from high end memory.
>>> So would virtio support 64bit address for vring virtqueue?
>>
>> The vring takes guest physical addresses and the C type is __u64 (see /usr/include/linux/virtio_ring.h).  64-bit addresses are fine.
>
> If you mean Queue Address field in the Virtio Header (section 2.2.2 in the virtio-0.9.5 specification), then please see 2.3 Virtqueue Configuration, step 3. You have to divide the guest-phys address by 4096 and store the quotient. The Queue Address field takes a page frame number, not a page frame address.
>
> This allows it to address up to 2^32 * 4096 == 16T bytes of RAM.
>
> Laszlo
>
>

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[H. Peter Anvin]

On 08/28/2013 10:35 AM, Anthony Liguori wrote:
> Yes, it was originally designed with the 16TB limit in mind.
> 
> PCI doesn't support 64-bit PIO operations so it would have required a
> high/low register and additional magic.
> 

s/PCI/x86/

"Additional magic" is needed only if atomic changes are necessary, but
in this case this is stuff that is set up during early configuration and
so isn't an issue.  The "additional magic" needed in that case is latch
logic - the write to the low part doesn't actually take effect until the
high part is written.

	-hpa

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Anthony Liguori]

On Wed, Aug 28, 2013 at 4:02 PM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 08/28/2013 10:35 AM, Anthony Liguori wrote:
>> Yes, it was originally designed with the 16TB limit in mind.
>>
>> PCI doesn't support 64-bit PIO operations so it would have required a
>> high/low register and additional magic.
>>
>
> s/PCI/x86/

Ack.  Although I wonder if there is a device out there that actually
expects 64-bit writes to a register in an PIO region...

> "Additional magic" is needed only if atomic changes are necessary, but
> in this case this is stuff that is set up during early configuration and
> so isn't an issue.  The "additional magic" needed in that case is latch
> logic - the write to the low part doesn't actually take effect until the
> high part is written.

Well in this case, writing to the PFN is what causes the queue to be
initialized, there isn't a separate status register for this purpose.

Since it's a PIO region, we tried hard to keep the common case within
a 32-byte region and since config space is after the virtio-pci
registers, we tried to keep it as small as possible to maximize
available config space.

Regards,

Anthony Liguori

>
>         -hpa
>

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Richard Henderson]

On 08/28/2013 02:28 PM, Anthony Liguori wrote:
> Ack.  Although I wonder if there is a device out there that actually
> expects 64-bit writes to a register in an PIO region...

If IBM makes any custom hardware that sits off a PCI bus, that's where I'd
expect to find such these days.  I know DEC had 64-bit pio for some of its
custom video and scsi controllers.

Not that I expect we'll ever bother emulating any of that hw in qemu...


r~

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Xie, Huawei]

If this is the case, one possible fix would be:
	Write two continuous 32bit DWORD to combine a 64bit address
	Use the upper 12 bits of PFN val to indicate if it is combined write
In this way, we wouldn't break other virtio driver, register layout and only need a few lines of modification.

-----Original Message-----
From: Xie, Huawei 
Sent: Wednesday, August 28, 2013 5:23 PM
To: 'Stefan Hajnoczi'
Cc: qemu-devel@nongnu.org; rusty@rustcorp.com.au; Stefan Hajnoczi
Subject: RE: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

Hi Stefan:
I think you mention the descriptor address? I mean the vring PFN register.
/* A 32-bit r/w PFN for the currently selected queue */
#define VIRTIO_PCI_QUEUE_PFN		8

And the linux driver sample code:
iowrite32(virt_to_phys(info->queue) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT,
		  vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);

-----Original Message-----
From: Stefan Hajnoczi [mailto:stefanha@gmail.com] 
Sent: Wednesday, August 28, 2013 4:07 PM
To: Xie, Huawei
Cc: qemu-devel@nongnu.org; rusty@rustcorp.com.au; Stefan Hajnoczi
Subject: Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

On Wed, Aug 28, 2013 at 03:18:39AM +0000, Xie, Huawei wrote:
> I am developing virtio user space poll mode network driver. We allocate vring physical memory from huge page. On VMs with less than 4GB  memory, it works well.  But on VMs with like 8GB memory, huge page are all allocated from high end memory.
> So would virtio support 64bit address for vring virtqueue?

The vring takes guest physical addresses and the C type is __u64 (see /usr/include/linux/virtio_ring.h).  64-bit addresses are fine.

Stefan

Re: [Qemu-devel] Would virtio support 64 bit address for vring virtqueue?

[Rusty Russell]

"Xie, Huawei" <huawei.xie@intel.com> writes:
> If this is the case, one possible fix would be:
> 	Write two continuous 32bit DWORD to combine a 64bit address
> 	Use the upper 12 bits of PFN val to indicate if it is combined write
> In this way, we wouldn't break other virtio driver, register layout
> and only need a few lines of modification.

Note that (with OASIS) we are currently working on a v1.0 of the virtio
spec.  I recommend the OASIS virtio-dev mailing list
( https://www.oasis-open.org/mlmanage/ ) and in particular the thread on
Michael Tsirkin's proposal on using PCI capabilities:

https://lists.oasis-open.org/archives/virtio-comment/201308/msg00087.html

Cheers,
Rusty.