[Qemu-devel] dynamic scsi disk attach seems to be broken in qemu(-kvm)-0.11, libvirt-0.7.1

[Qemu-devel] dynamic scsi disk attach seems to be broken in qemu(-kvm)-0.11, libvirt-0.7.1

[Dustin Kirkland]

We're experiencing what we believe to be a regression in qemu-kvm-0.11
and libvirt-0.7.0, from previous versions of kvm-84 and libvirt-0.6.1:
Dynamically attaching sdb (a second scsi disk) fails.  (Note that I
have tested this against libvirt-0.7.1 built from source as well--same
results.)

Here are some instructions to reproduce this issue:

Create some small auxiliary storage to dynamically add to a running VM.
  (host) $ dd if=/dev/zero of=/tmp/foo bs=1M count=64

Boot a vm in under qemu-kvm-0.11.0 like this:
  (host) $ kvm -drive file=server.img,if=scsi,boot=on -boot c

To reproduce the problem, you must use if=scsi. This problem manifests
itself when subsequent scsi storage devices are added. If the device
is the first scsi device on the system, then it would succeed. It's
the 2nd scsi device that causes the problem.

Once the system is booted, login and in the vm, load this module:
  (vm) $ sudo modprobe acpiphp

Check that the acpiphp slots are loaded in dmesg. And note that there
is only one /dev/sd[a-z] device.

Now, drop to the qemu console with ctrl-alt-2, and add the storage:
  (qemu) pci_add auto storage file=/tmp/foo,if=scsi
  OK domain 0, bus 0, slot 6, function 0

Switch back to the vm linux shell with ctrl-alt-1, and look at the dmesg output.
  (vm) $ dmesg | tail -n 12
[ 44.033397] pci 0000:00:06.0: reg 10 io port: [0x00-0xff]
[ 44.033443] pci 0000:00:06.0: reg 14 32bit mmio: [0x000000-0x0003ff]
[ 44.033486] pci 0000:00:06.0: reg 18 32bit mmio: [0x000000-0x001fff]
[ 44.033899] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2
[ 44.033975] decode_hpp: Could not get hotplug parameters. Use defaults
[ 44.042277] sym53c8xx 0000:00:06.0: enabling device (0000 -> 0003)
[ 44.043230] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[ 44.043247] sym53c8xx 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 11
(level, high) -> IRQ 11
[ 44.045237] sym1: <895a> rev 0x0 at pci 0000:00:06.0 irq 11
[ 44.047586] sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking
[ 44.055399] sym1: SCSI BUS has been reset.
[ 44.063329] scsi3 : sym-2.2.3

More importantly, note that no /dev/sd[b-z] device shows up.

If you now drop to the qemu console with ctrl-alt-2, and do this:
  (qemu) pci_add auto storage file=/tmp/foo,if=virtio
  OK domain 0, bus 0, slot 7, function 0

Going back to the vm with ctrl-alt-1, you can now see a new /dev/vda
device registered.

Now, all of that said, it is actually possible to hot-add a second
scsi device. However, as far as I can tell, this method is not yet
supported by libvirt.  It looks to me that with modern qemu, you have
to do it this way:

Drop to a qemu console with ctrl-alt-2. Get the address of the current scsi bus:
  (qemu) info pci
  Look for "SCSI Controller". In my case, it's on Bus 0, device 4, function 0

Now instead of pci_add, use drive_add
  (qemu) drive_add 0:4 file=/tmp/foo,if=scsi
  OK bus 0, unit 1

This is not ideal, however, as I tried re-scan the scsi bus with
rescan-scsi-bus.sh (from scsitools) without luck. It did not pick up
the new sdb device. However, I did reboot the vm, and voila, I now
have /dev/sdb.

I would like some advice on how to proceed with this bug, and where
the solution lies...in qemu or in libvirt.  Ultimately, I would like
the behavior we had in our previous release with kvm-84 and
libvirt-0.6.1, where we could dynamically add scsi devices without a
problem, using:
  pci_add 1 storage file=/tmp/foo,if=scsi

Can anyone else reproduce this?  Is this considered a regression by
anyone else?  Where should I look to solve this, in libvirt, or in
qemu?

-- 
:-Dustin

[Qemu-devel] Re: [libvirt] dynamic scsi disk attach seems to be broken in qemu(-kvm)-0.11, libvirt-0.7.1

[Daniel P. Berrange]

On Thu, Oct 08, 2009 at 10:57:55PM -0500, Dustin Kirkland wrote:
> Now, all of that said, it is actually possible to hot-add a second
> scsi device. However, as far as I can tell, this method is not yet
> supported by libvirt.  It looks to me that with modern qemu, you have
> to do it this way:
> 
> Drop to a qemu console with ctrl-alt-2. Get the address of the current scsi bus:
>   (qemu) info pci
>   Look for "SCSI Controller". In my case, it's on Bus 0, device 4, function 0
> 
> Now instead of pci_add, use drive_add
>   (qemu) drive_add 0:4 file=/tmp/foo,if=scsi
>   OK bus 0, unit 1

That is correct - the SCSI driver hotplug in libvirt is not implemented
in the right way. If you specify multiple SCSI devices at boot, they
all get on one controller, if you hotplug multiple SCSI devices, we're
hotplugging a new SCSI controller per disk. This is clearly not good,
because when you then reboot, all those controllers are merged back into
one.

There is a guy who is working on implementing the correct SCSI hotplug
approach for libvirt, that is still work in progress though. The most
recent patches were here:

http://www.redhat.com/archives/libvir-list/2009-September/msg00551.html

We will ultimately support hotplug of both drives, and drive controllers
independantly, giving apps/users the flexibility they need.

> I would like some advice on how to proceed with this bug, and where
> the solution lies...in qemu or in libvirt.  Ultimately, I would like
> the behavior we had in our previous release with kvm-84 and
> libvirt-0.6.1, where we could dynamically add scsi devices without a
> problem, using:
>   pci_add 1 storage file=/tmp/foo,if=scsi
> 
> Can anyone else reproduce this?  Is this considered a regression by
> anyone else?  Where should I look to solve this, in libvirt, or in
> qemu?

Independently of what I said about libvirt not implementing SCSI hotplug
with the right apporoach, the pci-add stuff should definitely work,  so 
if it doesn't then this is a regression that needs to be fixed

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] dynamic scsi disk attach seems to be broken in qemu(-kvm)-0.11, libvirt-0.7.1

[Anthony Liguori]

>> I would like some advice on how to proceed with this bug, and where
>> the solution lies...in qemu or in libvirt.  Ultimately, I would like
>> the behavior we had in our previous release with kvm-84 and
>> libvirt-0.6.1, where we could dynamically add scsi devices without a
>> problem, using:
>>   pci_add 1 storage file=/tmp/foo,if=scsi
>>
>> Can anyone else reproduce this?  Is this considered a regression by
>> anyone else?  Where should I look to solve this, in libvirt, or in
>> qemu?
>>     
>
> Independently of what I said about libvirt not implementing SCSI hotplug
> with the right apporoach, the pci-add stuff should definitely work,  so 
> if it doesn't then this is a regression that needs to be fixed
>   

There's definitely a bug somewhere.  When you add a second PCI 
controller, the controller shows up in the guest but the disk attached 
to the controller isn't visible.  Rescanning the scsi bus does not make 
the disk appear either.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] dynamic scsi disk attach seems to be broken in qemu(-kvm)-0.11, libvirt-0.7.1

[Dustin Kirkland]

On Fri, Oct 9, 2009 at 9:38 AM, Anthony Liguori <anthony@codemonkey.ws> wrote:
>>> I would like some advice on how to proceed with this bug, and where
>>> the solution lies...in qemu or in libvirt.  Ultimately, I would like
>>> the behavior we had in our previous release with kvm-84 and
>>> libvirt-0.6.1, where we could dynamically add scsi devices without a
>>> problem, using:
>>>  pci_add 1 storage file=/tmp/foo,if=scsi
>>>
>>> Can anyone else reproduce this?  Is this considered a regression by
>>> anyone else?  Where should I look to solve this, in libvirt, or in
>>> qemu?
>>>
>>
>> Independently of what I said about libvirt not implementing SCSI hotplug
>> with the right apporoach, the pci-add stuff should definitely work,  so if
>> it doesn't then this is a regression that needs to be fixed
>>
>
> There's definitely a bug somewhere.  When you add a second PCI controller,
> the controller shows up in the guest but the disk attached to the controller
> isn't visible.  Rescanning the scsi bus does not make the disk appear
> either.

Okay, thanks for confirming this, Anthony.

I have added a task against upstream QEMU at:
 * https://bugs.edge.launchpad.net/ubuntu/+source/qemu-kvm/+bug/432154

:-Dustin

[Qemu-devel] Re: [libvirt] dynamic scsi disk attach seems to be broken in qemu(-kvm)-0.11, libvirt-0.7.1

[Dustin Kirkland]

On Fri, Oct 9, 2009 at 3:18 AM, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Thu, Oct 08, 2009 at 10:57:55PM -0500, Dustin Kirkland wrote:
>> Now, all of that said, it is actually possible to hot-add a second
>> scsi device. However, as far as I can tell, this method is not yet
>> supported by libvirt.  It looks to me that with modern qemu, you have
>> to do it this way:
>>
>> Drop to a qemu console with ctrl-alt-2. Get the address of the current scsi bus:
>>   (qemu) info pci
>>   Look for "SCSI Controller". In my case, it's on Bus 0, device 4, function 0
>>
>> Now instead of pci_add, use drive_add
>>   (qemu) drive_add 0:4 file=/tmp/foo,if=scsi
>>   OK bus 0, unit 1
>
> That is correct - the SCSI driver hotplug in libvirt is not implemented
> in the right way. If you specify multiple SCSI devices at boot, they
> all get on one controller, if you hotplug multiple SCSI devices, we're
> hotplugging a new SCSI controller per disk. This is clearly not good,
> because when you then reboot, all those controllers are merged back into
> one.

Okay, in our previous usage of this, rebooting didn't matter very much.

> There is a guy who is working on implementing the correct SCSI hotplug
> approach for libvirt, that is still work in progress though. The most
> recent patches were here:
>
> http://www.redhat.com/archives/libvir-list/2009-September/msg00551.html
>
> We will ultimately support hotplug of both drives, and drive controllers
> independantly, giving apps/users the flexibility they need.

Thanks.  I'll track those patches with interest.

:-Dustin