Re: large page size virtio issues

Re: large page size virtio issues

[Hollis Blanchard]

On Wed, 2008-11-05 at 08:06 -0600, Anthony Liguori wrote:
> Rusty Russell wrote:
> > On Wednesday 05 November 2008 09:14:20 Hollis Blanchard wrote:
> >   
> >> Hi Rusty, I'm using a patch that changes the Linux base page size to
> >> 64K. (This is actually pretty common in ppc64 world, but I happen to be
> >> trying it on ppc32.)
> >>
> >> I'm seeing a problem with virtio. I think at least part of it can be
> >> explained by qemu's TARGET_PAGE_BITS==12, and the guest's
> >> PAGE_SHIFT==16. The guest allocates the queue, then passes the pfn (pa
> >> >> PAGE_SHIFT) to the virtio backend (vp_find_vq()). The backend then
> >> calculates the pa as pfn << TARGET_PAGE_BITS.
> >>
> >> I have to run right now, but quickly changing qemu TARGET_PAGE_BITS to
> >> 16 got me a little further but still didn't work. Any thoughts?
> >>     
> >
> > I see Anthony hardwired page size into the queue activation ABI for 
> > virtio_pci.
> 
> So did you FWIW, virtio-balloon passes PFNs which are computed based on 
> PAGE_SHIFT.
> 
> >   I think that this should be an actual 4096 (or 12) rather than 
> > depending on guest page size:

I agree: it's simply a question of both sides of the interface agreeing
on the units used in the interface. As the interface is defined today,
both qemu and the guest virtio should use a fixed constant which is
neither (Linux) PAGE_SHIFT nor (qemu) TARGET_PAGE_BITS.

> So is the issue that PPC can support 4k or 16k pages, and the guest 
> happens to always use 16k pages?  Does the guest set any global flag 
> indicating it is using 16k pages?  Is this anyway we could detect this 
> in QEMU?

To elaborate a little, I'm using a patch to PowerPC 440 Linux that
allows you to configure the base page size at build time; choices are
4K, 16K, and 64K. (The hardware supports more sizes, and with other
patches or other operating systems qemu would need to worry about 256K,
1M, 16M, and 256M pages.)

The page size is set per MMU mapping, and of course it's ridiculous to
walk the TLB to see if all the page sizes are the same to "detect" the
condition. In fact, regardless of the base page size, the (Linux) kernel
is always mapped with 256M pages, and if you consider hugetlbfs the
situation is even more fluid.

> I don't much like the idea of globally hard coding it to 4k. I'd rather 
> make it architecture specific.

Making the units architecture-specific doesn't solve the problem at all
AFAICS. It doesn't even solve my original problem on PowerPC 440 since
the guest page size can vary.

AFAIK the only reason to use a PFN in this interface in the first place
is to allow for physical addresses >32 bits. A hardcoded shift of 12
gives you 44 bits of physical address space (16 TB). This actually isn't
very big today, so using an architecture-specific hardcoded 4K size will
become an issue anyways, *even on x86*.

Brainstorming backwards-compatible interface expansion possibilities:
     1. Rename the current interface to "4K_PFN", and add another, let's
        say "64K_PFN". Of course, if a guest with smaller pages uses the
        new interface, it must properly align its queue allocation.
     2. Rename the current interface to "4K_PFN". Use 64-bit writes to
        set VIRTIO_PCI_QUEUE_PFN. 32-bit architectures couldn't use
        this, which might be OK since practically speaking, I think
        32-bit architectures can address at most 36 bits of physical
        space. I also don't know what the semantics are of 64-bit PCI
        writes (if it's not allowed on physical hardware) -- it looks
        like Linux doesn't have an iowrite64, for example.
     3. Rename the current interface to "4K_PFN". Use multiple writes
        (high/low) to set VIRTIO_PCI_QUEUE_PFN. Not atomic. To simplify
        backend implementation, you could require that PFN_HIGH writes
        come before PFN_LOW.
     4. Use multiple writes (set page size, set PFN). SET_PAGE_SIZE must
        precede SET_PFN. Not atomic.
     5. Create a variable-sized interface (still 32-bit write), where
        the shift value is encoded in the value itself (I guess this is
        the FP mantissa+exponent approach). For example, the low 8 bits
        are the shift beyond 12, so a write of 0x10000004 would mean
        physical address 1<<(12+4).

These solutions would solve both problems: a) making "guest page size"
explicit, and b) addressing more than 16TB of physical memory in the
future. I think I like #3 or #4 the best.

Hardcoding the current interface to mean "4K pages" (and updating qemu
and Linux to match) would solve my problem, and the 16TB limit could be
addressed in the future as needed.

-- 
Hollis Blanchard
IBM Linux Technology Center

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: large page size virtio issues

[Anthony Liguori]

Hollis Blanchard wrote:
> On Wed, 2008-11-05 at 08:06 -0600, Anthony Liguori wrote:
>   
>
>> I don't much like the idea of globally hard coding it to 4k. I'd rather 
>> make it architecture specific.
>>     
>
> Making the units architecture-specific doesn't solve the problem at all
> AFAICS. It doesn't even solve my original problem on PowerPC 440 since
> the guest page size can vary.
>
> AFAIK the only reason to use a PFN in this interface in the first place
> is to allow for physical addresses >32 bits. A hardcoded shift of 12
> gives you 44 bits of physical address space (16 TB). This actually isn't
> very big today, so using an architecture-specific hardcoded 4K size will
> become an issue anyways, *even on x86*.
>   

It's PIO on x86 so there's no way to do a 64-bit operation.

> Brainstorming backwards-compatible interface expansion possibilities:
>      1. Rename the current interface to "4K_PFN", and add another, let's
>         say "64K_PFN". Of course, if a guest with smaller pages uses the
>         new interface, it must properly align its queue allocation.
>      2. Rename the current interface to "4K_PFN". Use 64-bit writes to
>         set VIRTIO_PCI_QUEUE_PFN. 32-bit architectures couldn't use
>         this, which might be OK since practically speaking, I think
>         32-bit architectures can address at most 36 bits of physical
>         space. I also don't know what the semantics are of 64-bit PCI
>         writes (if it's not allowed on physical hardware) -- it looks
>         like Linux doesn't have an iowrite64, for example.
>      3. Rename the current interface to "4K_PFN". Use multiple writes
>         (high/low) to set VIRTIO_PCI_QUEUE_PFN. Not atomic. To simplify
>         backend implementation, you could require that PFN_HIGH writes
>         come before PFN_LOW.
>      4. Use multiple writes (set page size, set PFN). SET_PAGE_SIZE must
>         precede SET_PFN. Not atomic.
>      5. Create a variable-sized interface (still 32-bit write), where
>         the shift value is encoded in the value itself (I guess this is
>         the FP mantissa+exponent approach). For example, the low 8 bits
>         are the shift beyond 12, so a write of 0x10000004 would mean
>         physical address 1<<(12+4).
>   

I think we should just have a VIRTIO_PFN_SHIFT define that is 
architecture specific.

On x86, it'll be 4k.  You can make it whatever you want for PPC.  Fixing 
all architectures to be 4k is going to suck for architectures with < 4k 
pages.

Regards,

Anthony Liguori

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: large page size virtio issues

[Hollis Blanchard]

On Wed, 2008-11-05 at 11:25 -0600, Anthony Liguori wrote:
> 
> I think we should just have a VIRTIO_PFN_SHIFT define that is 
> architecture specific.
> 
> On x86, it'll be 4k.  You can make it whatever you want for PPC.
> Fixing all architectures to be 4k is going to suck for architectures
> with < 4k pages.

So we're now in agreement that it should be hardcoded. I don't see much
value in making the actual value arch-specific, but either way it will
solve (at least part of, and maybe all of) my problem. Rusty?

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: large page size virtio issues

[Anthony Liguori]

Hollis Blanchard wrote:
> On Wed, 2008-11-05 at 11:25 -0600, Anthony Liguori wrote:
>   
>> I think we should just have a VIRTIO_PFN_SHIFT define that is 
>> architecture specific.
>>
>> On x86, it'll be 4k.  You can make it whatever you want for PPC.
>> Fixing all architectures to be 4k is going to suck for architectures
>> with < 4k pages.
>>     
>
> So we're now in agreement that it should be hardcoded. I don't see much
> value in making the actual value arch-specific, but either way it will
> solve (at least part of, and maybe all of) my problem. Rusty?
>   

When you balloon pages, you need to pass something that points to the 
beginning of a page boundary.  If your page size is 2k, then if you are 
only able to pass 4k addresses, you can't below any page that happens to 
not be 4k aligned.

If you were going to make it common across all architectures, you would 
have to make it the smallest possible page size and that would break x86 
backwards compatibility.

Regards,

Anthony Liguori


--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: large page size virtio issues

[Hollis Blanchard]

On Wed, 2008-11-05 at 11:33 -0600, Anthony Liguori wrote:
> Hollis Blanchard wrote:
> > On Wed, 2008-11-05 at 11:25 -0600, Anthony Liguori wrote:
> >   
> >> I think we should just have a VIRTIO_PFN_SHIFT define that is 
> >> architecture specific.
> >>
> >> On x86, it'll be 4k.  You can make it whatever you want for PPC.
> >> Fixing all architectures to be 4k is going to suck for architectures
> >> with < 4k pages.
> >>     
> >
> > So we're now in agreement that it should be hardcoded. I don't see much
> > value in making the actual value arch-specific, but either way it will
> > solve (at least part of, and maybe all of) my problem. Rusty?
> >   
> 
> When you balloon pages, you need to pass something that points to the 
> beginning of a page boundary.  If your page size is 2k, then if you are 
> only able to pass 4k addresses, you can't below any page that happens to 
> not be 4k aligned.
> 
> If you were going to make it common across all architectures, you would 
> have to make it the smallest possible page size and that would break x86 
> backwards compatibility.

OK, makes sense to me.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: large page size virtio issues

[Hollis Blanchard]

On Wed, 2008-11-05 at 23:16 +1100, Rusty Russell wrote:
> On Wednesday 05 November 2008 09:14:20 Hollis Blanchard wrote:
> > Hi Rusty, I'm using a patch that changes the Linux base page size to
> > 64K. (This is actually pretty common in ppc64 world, but I happen to be
> > trying it on ppc32.)
> >
> > I'm seeing a problem with virtio. I think at least part of it can be
> > explained by qemu's TARGET_PAGE_BITS==12, and the guest's
> > PAGE_SHIFT==16. The guest allocates the queue, then passes the pfn (pa
> >
> > >> PAGE_SHIFT) to the virtio backend (vp_find_vq()). The backend then
> >
> > calculates the pa as pfn << TARGET_PAGE_BITS.
> >
> > I have to run right now, but quickly changing qemu TARGET_PAGE_BITS to
> > 16 got me a little further but still didn't work. Any thoughts?
> 
> I see Anthony hardwired page size into the queue activation ABI for 
> virtio_pci.  I think that this should be an actual 4096 (or 12) rather than 
> depending on guest page size:
> 
> virtio_pci.c:247:
> 	iowrite32(virt_to_phys(info->queue) >> PAGE_SHIFT,
> 		  vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);

Actually there's an additional complication: PAGE_SIZE is entrenched in
the layout of the ring structure itself (large "struct vring" comment in
include/linux/virtio_ring.h). Callers of vring_size() demonstrate this
problem.

I believe the vring is split across two pages so that it could be
directly mapped by two untrusted guests: the producer would have RW
access to the descriptors and the "available" fields, while the consumer
would have only RO access. I don't think this is yet implemented; is it
still planned?

It looks like vring_size() et al were carefully written to allow
arbitrary page sizes, so for now I'll assume that a struct vring that is
contained completely within a single guest mapping is OK and work up a
patch.

It's worth noting that the PFN_SHIFT question (in the other thread) is a
separate issue. I could have PFN_SHIFT=10 but define VRING_PAGE_SIZE=4K
for internal alignment. I'm a little nervous about making PPC the only
arch with VRING_PAGE_SIZE=1K, since that might affect my vring layout in
"unusual" ways if it expands beyond 1K but stays inside 4K.

-- 
Hollis Blanchard
IBM Linux Technology Center

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: large page size virtio issues

[Anthony Liguori]

Hollis Blanchard wrote:
> Actually there's an additional complication: PAGE_SIZE is entrenched in
> the layout of the ring structure itself (large "struct vring" comment in
> include/linux/virtio_ring.h). Callers of vring_size() demonstrate this
> problem.
>
> I believe the vring is split across two pages so that it could be
> directly mapped by two untrusted guests: the producer would have RW
> access to the descriptors and the "available" fields, while the consumer
> would have only RO access. I don't think this is yet implemented; is it
> still planned?
>   

Rusty experimented with it but I don't think anyone has done anything 
seriously with it.

> It looks like vring_size() et al were carefully written to allow
> arbitrary page sizes, so for now I'll assume that a struct vring that is
> contained completely within a single guest mapping is OK and work up a
> patch.
>   

It was written that way so that vring_size() could be used in userspace 
where there isn't a PAGE_SIZE defined.  Rusty just passes in 
getpagesize() in the lguest userspace.

Pure evil if you ask me :-)

> It's worth noting that the PFN_SHIFT question (in the other thread) is a
> separate issue. I could have PFN_SHIFT=10 but define VRING_PAGE_SIZE=4K
> for internal alignment. I'm a little nervous about making PPC the only
> arch with VRING_PAGE_SIZE=1K, since that might affect my vring layout in
> "unusual" ways if it expands beyond 1K but stays inside 4K.
>   

This is starting to get a little hairy.  Maybe 64k guest pages is asking 
for trouble?  Are you sure other things aren't breaking all over the 
place with these patches in place?

Regards,

Anthony Liguori

Re: large page size virtio issues

[Hollis Blanchard]

On Wed, 2008-11-05 at 14:08 -0600, Anthony Liguori wrote:
> 
> > It's worth noting that the PFN_SHIFT question (in the other thread) is a
> > separate issue. I could have PFN_SHIFT=10 but define VRING_PAGE_SIZE=4K
> > for internal alignment. I'm a little nervous about making PPC the only
> > arch with VRING_PAGE_SIZE=1K, since that might affect my vring layout in
> > "unusual" ways if it expands beyond 1K but stays inside 4K.
> >   
> 
> This is starting to get a little hairy.

Yes it is.

> Maybe 64k guest pages is asking 
> for trouble?  Are you sure other things aren't breaking all over the 
> place with these patches in place?

Define "other things"? Here's my current boot log from 64K/64K
guest/host configuration:

bash-3.00# ./qemu-system-ppcemb -m 128 -M bamboo -nographic -kernel ~/uImage.64k -L ../pc-bios/ -append "root=/dev/vda rw debug" -net nic -net tap -drive file=/mnt/images/ramdisk_image,if=virtio,boot=on
bamboo_init: START
Ram size passed is: 128 MB
Calling function ppc440_init
setup mmio
setup universal controller
trying to setup sdram controller
sdram_unmap_bcr: Unmap RAM area 0000000000000000 00400000
sdram_set_bcr: Map RAM area 0000000000000000 08000000
Initializing first serial port
ppc405_serial_init: offset 0000000000000300
Done calling ppc440_init
bamboo_init: load kernel
kernel is at guest address: 0x0
bamboo_init: load device tree file
device tree address is at guest address: 0x381488
bamboo_init: DONE
Using PowerPC 44x Platform machine description
Linux version 2.6.28-rc2-dirty (hollisb-EGjIuKC2qUdB0N6nvOmcJFaTQe2KTcn/@public.gmane.org) (gcc version 4.2.3) #101 Tue Nov 4 14:19:54 CST 2008
Found legacy serial port 0 for /plb/opb/serial@ef600300
  mem=ef600300, taddr=ef600300, irq=0, clk=11059200, speed=115200
Found legacy serial port 1 for /plb/opb/serial@ef600400
  mem=ef600400, taddr=ef600400, irq=0, clk=11059200, speed=0
console [udbg0] enabled
setup_arch: bootmem
arch: exit
Top of RAM: 0x8000000, Total RAM: 0x8000000
Memory hole size: 0MB
Zone PFN ranges:
  DMA      0x00000000 -> 0x00000800
  Normal   0x00000800 -> 0x00000800
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00000800
On node 0 totalpages: 2048
free_area_init_node: node 0, pgdat c034baa8, node_mem_map c0440000
  DMA zone: 1 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 2047 pages, LIFO batch:0
  Normal zone: 0 pages used for memmap
  Movable zone: 0 pages used for memmap
Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 2047
Kernel command line: root=/dev/vda rw debug
UIC0 (32 IRQ sources) at DCR 0xc0
PID hash table entries: 512 (order: 9, 2048 bytes)
time_init: decrementer frequency = 666.666660 MHz
time_init: processor frequency   = 666.666660 MHz
clocksource: timebase mult[600000] shift[22] registered
clockevent: decrementer mult[aaaa] shift[16] cpu[0]
Dentry cache hash table entries: 16384 (order: 0, 65536 bytes)
Inode-cache hash table entries: 16384 (order: 0, 65536 bytes)
Memory: 126336k/131072k available (3136k kernel code, 4736k reserved, 256k data, 655k bss, 320k init)
SLUB: Genslabs=14, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop... 2555.90 BogoMIPS (lpj=5111808)
Mount-cache hash table entries: 8192
net_namespace: 288 bytes
program_check_exception: nip c02cad1c reason 2800000
------------[ cut here ]------------
Badness at c02cad1c [verbose debug info unavailable]
NIP: c02cad1c LR: c00013e4 CTR: c02cacf0
REGS: c7e11e00 TRAP: 0700   Not tainted  (2.6.28-rc2-dirty)
MSR: 00029002 <EE,ME>  CR: 84000042  XER: 20000000
TASK = c7e00000[1] 'swapper' THREAD: c7e10000
GPR00: c7fe0000 c7e11eb0 c7e00000 c02cacf0 00000000 00000000 c04a04bc 0000076d 
GPR08: 00000036 00000001 c7e11e38 c7fdffff 44000042 00000000 00000000 00000000 
GPR16: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
GPR24: 00000000 00000000 00000f60 00000000 c0390360 c02cacf0 c7e10000 c032001c 
NIP [c02cad1c] dma_alloc_init+0x2c/0xcc
LR [c00013e4] do_one_initcall+0x38/0x19c
Call Trace:
[c7e11eb0] [c02d78d0] net_ns_init+0x0/0x1e4 (unreliable)
[c7e11ec0] [c00013e4] do_one_initcall+0x38/0x19c
[c7e11fd0] [c02c0168] kernel_init+0x88/0xf4
[c7e11ff0] [c000d5f4] kernel_thread+0x50/0x6c
Instruction dump:
4e800020 9421fff0 7c0802a6 3d20c033 93e1000c 90010014 81290334 37e9001c 
41820094 8009001c 3160ffff 7d2b0110 <0f090000> 7403ffff 4182005c 3c030001 
NET: Registered protocol family 16
             
PCI host bridge /plb/pci@ec000000 (primary) ranges:
 MEM 0x00000000a0000000..0x00000000bfffffff -> 0x00000000a0000000 
  IO 0x00000000e8000000..0x00000000e800ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCI: Probing PCI hardware
PCI: Hiding 4xx host bridge resources 0000:00:00.0
pci 0000:00:01.0: reg 10 io port: [0x00-0x3f]
pci 0000:00:02.0: reg 10 io port: [0x00-0x1f]
bus: 00 index 0 io port: [0x00-0xffff]
bus: 00 index 1 mmio: [0xa0000000-0xbfffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 16384 (order: 0, 65536 bytes)
TCP established hash table entries: 8192 (order: 0, 65536 bytes)
TCP bind hash table entries: 8192 (order: -1, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
NET: Registered protocol family 1
msgmni has been set to 244
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
Serial: 8250/16550 driver4 ports, IRQ sharing enabled
serial8250.0: ttyS0 at MMIO 0xef600300 (irq = 16) is a 16550A
console handover: boot [udbg0] -> real [ttyS0]
ef600300.serial: ttyS0 at MMIO 0xef600300 (irq = 16) is a 16550A
brd: module loaded
Intel(R) PRO/1000 Network Driver - version 7.3.20-k3-NAPI
Copyright (c) 1999-2006 Intel Corporation.
PPC 4xx OCP EMAC driver, version 3.54
ZMII /plb/opb/emac-zmii@ef600d00 initialized
pcnet32.c:v1.35 21.Apr.2008 tsbogend-I1c7kopa9pxLokYuJOExCg@public.gmane.org
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk-zC7DfRvBq/JWk0Htik3J/w@public.gmane.org>
virtio-pci 0000:00:01.0: enabling device (0000 -> 0001)
 vda:qemu-system-ppcemb: Guest moved used index from 0 to 38277

(This is with my in-progress patches applied. Before virtio would just
hang, since it was miscommunicating the vring's physical address.)

-- 
Hollis Blanchard
IBM Linux Technology Center

--
To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html