[Qemu-devel] virtio-blk issues

[Qemu-devel] virtio-blk issues

[Christoph Hellwig]

I've been trying to move setup from kvm 72 in debian testing to the current
qemu CVS.  The setup is using 7 virtio_blk disks and used to work on the
old setup, but with current qemu the kernel (current Linus' HEAD from a few
days ago) hangs when trying to setup the irq routing for the virtio_pci
devices:

virtio pci 0000:00:06.0: can't derive routing for PCI INT A
virtio pci 0000:00:06.0: PCI INT A: no GSI - using IRQ 9

Below is the config file I'm using.  When commenting out the drive lines
so that only the first two are left the system boots, although rather
slowly compared to kvm (this is using the minimal kvm support in qemu
upstream, without it it's of course even slower).


#!/bin/sh

/opt/qemu/bin/qemu -m 1000 -enable-kvm -kernel $1 \
	-drive file=/dev/vg00/qemu-root,if=virtio,media=disk \
	-drive file=/dev/vg00/qemu-test,if=virtio,media=disk \
	-drive file=/dev/vg00/qemu-test-log,if=virtio,media=disk \
	-drive file=/dev/vg00/qemu-test-rt,if=virtio,media=disk \
	-drive file=/dev/vg00/qemu-scratch,if=virtio,media=disk \
	-drive file=/dev/vg00/qemu-scratch-log,if=virtio,media=disk \
	-drive file=/dev/vg00/qemu-scratch-rt,if=virtio,media=disk \
	-append "root=/dev/vda console=tty0 console=ttyS0,38400n8"

Re: [Qemu-devel] virtio-blk issues

[Paul Brook]

On Tuesday 03 March 2009, Christoph Hellwig wrote:
> I've been trying to move setup from kvm 72 in debian testing to the current
> qemu CVS.  The setup is using 7 virtio_blk disks and used to work on the
> old setup, but with current qemu the kernel (current Linus' HEAD from a few
> days ago) hangs when trying to setup the irq routing for the virtio_pci
> devices:
>
> virtio pci 0000:00:06.0: can't derive routing for PCI INT A
> virtio pci 0000:00:06.0: PCI INT A: no GSI - using IRQ 9

IIRC this is partly a bios issue. Real PIIX3 based systems never have more 
than ~5 PCI slots, so it doesn't have interrupt routing for more devices.

There are two solutions: Fix the bios to handle machines with large numbers of 
PCI slots, or have create multifunction PCI devices so lots of virtio 
instances don't use so many virtual PCI slots.

Paul

Re: [Qemu-devel] virtio-blk issues

[Christoph Hellwig]

On Tue, Mar 03, 2009 at 02:22:43PM +0000, Paul Brook wrote:
> On Tuesday 03 March 2009, Christoph Hellwig wrote:
> > I've been trying to move setup from kvm 72 in debian testing to the current
> > qemu CVS.  The setup is using 7 virtio_blk disks and used to work on the
> > old setup, but with current qemu the kernel (current Linus' HEAD from a few
> > days ago) hangs when trying to setup the irq routing for the virtio_pci
> > devices:
> >
> > virtio pci 0000:00:06.0: can't derive routing for PCI INT A
> > virtio pci 0000:00:06.0: PCI INT A: no GSI - using IRQ 9
> 
> IIRC this is partly a bios issue. Real PIIX3 based systems never have more 
> than ~5 PCI slots, so it doesn't have interrupt routing for more devices.
>
> There are two solutions: Fix the bios to handle machines with large numbers of 
> PCI slots, or have create multifunction PCI devices so lots of virtio 
> instances don't use so many virtual PCI slots.
> 

Or we could try to create a pc+PIIX style VM.  Surely having more than
two disks should not be that uncommon..

Re: [Qemu-devel] virtio-blk issues

[Anthony Liguori]

Paul Brook wrote:
> On Tuesday 03 March 2009, Christoph Hellwig wrote:
>   
>> I've been trying to move setup from kvm 72 in debian testing to the current
>> qemu CVS.  The setup is using 7 virtio_blk disks and used to work on the
>> old setup, but with current qemu the kernel (current Linus' HEAD from a few
>> days ago) hangs when trying to setup the irq routing for the virtio_pci
>> devices:
>>
>> virtio pci 0000:00:06.0: can't derive routing for PCI INT A
>> virtio pci 0000:00:06.0: PCI INT A: no GSI - using IRQ 9
>>     
>
> IIRC this is partly a bios issue. Real PIIX3 based systems never have more 
> than ~5 PCI slots, so it doesn't have interrupt routing for more devices.
>
> There are two solutions: Fix the bios to handle machines with large numbers of 
> PCI slots, or have create multifunction PCI devices so lots of virtio 
> instances don't use so many virtual PCI slots.
>   

You then those the ability to hotplug individual block devices (not that 
that's a terribly useful feature anyway).

Regards,

Anthony Liguori

> Paul
>
>
>   

Re: [Qemu-devel] virtio-blk issues

[Avi Kivity]

Paul Brook wrote:
> On Tuesday 03 March 2009, Christoph Hellwig wrote:
>   
>> I've been trying to move setup from kvm 72 in debian testing to the current
>> qemu CVS.  The setup is using 7 virtio_blk disks and used to work on the
>> old setup, but with current qemu the kernel (current Linus' HEAD from a few
>> days ago) hangs when trying to setup the irq routing for the virtio_pci
>> devices:
>>
>> virtio pci 0000:00:06.0: can't derive routing for PCI INT A
>> virtio pci 0000:00:06.0: PCI INT A: no GSI - using IRQ 9
>>     
>
> IIRC this is partly a bios issue. Real PIIX3 based systems never have more 
> than ~5 PCI slots, so it doesn't have interrupt routing for more devices.
>
> There are two solutions: Fix the bios to handle machines with large numbers of 
> PCI slots, or have create multifunction PCI devices so lots of virtio 
> instances don't use so many virtual PCI slots.
>
>   

The kvm bios supports 32 slots; I don't know if upstream bochs picked up 
these changes.

In any case I'd like to see a virtio-blk controller which can support an 
arbitrary number of disks, much like a SCSI controller.  A pci slot or 
function per disk is wasteful.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] virtio-blk issues

[Christoph Hellwig]

On Mon, Mar 16, 2009 at 03:22:24PM +0200, Avi Kivity wrote:
> The kvm bios supports 32 slots; I don't know if upstream bochs picked up 
> these changes.

I tried using the kvm pc-bios with upstream qemu, but it fails pretty
early on when the kernel tries to initialize the timer interrupt.

What's the protocol for syncing all those bios changes?

> In any case I'd like to see a virtio-blk controller which can support an 
> arbitrary number of disks, much like a SCSI controller.  A pci slot or 
> function per disk is wasteful.

I'll see what I can do.

Re: [Qemu-devel] virtio-blk issues

[Avi Kivity]

Christoph Hellwig wrote:
> On Mon, Mar 16, 2009 at 03:22:24PM +0200, Avi Kivity wrote:
>   
>> The kvm bios supports 32 slots; I don't know if upstream bochs picked up 
>> these changes.
>>     
>
> I tried using the kvm pc-bios with upstream qemu, but it fails pretty
> early on when the kernel tries to initialize the timer interrupt.
>
>   

Interrupt routing is different between kvm and qemu due to HPET sending 
interrupts to PIC IRQ0 and IOAPIC INTI2; the kvm up to 2.6.29 only 
supports 1:1 mapping between PIC and IOAPIC pins 0-15.

> What's the protocol for syncing all those bios changes?
>   

I merge them occasionally, but in this case, syncing the changes would 
cause one or the other to break.

What is needed is to sync the qemu changes, and additionally add code to 
detect the non-existence of non-1:1 mapping and revert back to the old 
behavior in that case (and disable HPET as well).

>> In any case I'd like to see a virtio-blk controller which can support an 
>> arbitrary number of disks, much like a SCSI controller.  A pci slot or 
>> function per disk is wasteful.
>>     
>
> I'll see what I can do.
>   

Did I mention my wishlist includes indirect virtio requests, where a 
virtio ring entry points to an sglist instead of the buffer?  Right now 
a 1MB request will consume 256 ring entries, which kills concurrency 
with many disks on a single ring.

-- 
error compiling committee.c: too many arguments to function