[Qemu-devel] libguestfs now uses virtio-scsi, supports large numbers of disks

[Qemu-devel] libguestfs now uses virtio-scsi, supports large numbers of disks

[Richard W.M. Jones]

I switched libguestfs over to using virtio-scsi.  One immediate
benefit is support for large numbers of disks: up to 255 because we're
using 1 target / disk and we reserve one disk for the appliance, in
theory more could be supported if we used LUNs.

This email just contains some notes that may be useful for others
trying to use large numbers of disks, and/or virtio-scsi, and/or qemu,
and/or Linux guests, all together.

* The command line syntax is convoluted.  You need to first of all
  define the virtio-scsi bus once:

    -device virtio-scsi-pci,id=scsi

  and then, for each disk, you need these two parameters:

    -drive file=disk.img,cache=off,format=raw,id=hd0,if=none
    -device scsi-hd,drive=hd0

  (adjust 'id' and 'drive' to be a unique name for each disk)

* Linux probes the SCSI bus asynchronously.

  This was a problem for us because our initramfs doesn't contain
  udev; it originally insmod'd the virtio-scsi.ko module and expected
  the disks to be present (note that this works for virtio-blk).

  Possible solutions include: wait for the disks to become present, or
  use udev, or do something with devtmpfs, or read out the events from
  the kernel like udev does.  We modified our initramfs to wait for
  the disk to be seen under /sys/block, that being the easiest change
  to make at the moment.

  Also scsi_wait_scan is apparently deprecated.

* This describes the naming scheme for > 26 disks under Linux:
  https://rwmj.wordpress.com/2011/01/09/how-are-linux-drives-named-beyond-drive-26-devsdz/

* Devices are named /dev/sdX instead of /dev/vdX (largely an
  improvement).

* With 256 disks, on fast hardware, with KVM, it takes a Linux guest
  about 3 seconds to probe the disks, and udev approx another 3
  seconds to process its event queue.

* With 256 disks, on fast hardware, but using TCG, it takes a Linux
  guest over 7 minutes to probe the disks.  This causes udev to
  timeout while processing the coldstart events, unless you increase
  the `udevadm settle --timeout' in the startup scripts.

  I don't understand why TCG is so much slower, since TCG itself isn't
  100 times slower normally.

* The overall libguestfs startup time (ie. with two disks) is down
  fractionally to 3.6 seconds, which is nice.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Re: [Qemu-devel] libguestfs now uses virtio-scsi, supports large numbers of disks

[Stefan Hajnoczi]

On Thu, Jun 14, 2012 at 2:49 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
> I switched libguestfs over to using virtio-scsi.  One immediate
> benefit is support for large numbers of disks: up to 255 because we're
> using 1 target / disk and we reserve one disk for the appliance, in
> theory more could be supported if we used LUNs.

Cool, thanks for sharing!  Does libguestfs hotplug LUNS/targets?

Cong and Wan Seng are implementing guest driver and QEMU-side changes
so the guest receives notifications of hotplugged LUNs.  It's nicer
than manually rescanning the bus from inside the guest.

Stefan

Re: [Qemu-devel] libguestfs now uses virtio-scsi, supports large numbers of disks

[Richard W.M. Jones]

On Fri, Jun 15, 2012 at 11:44:42AM +0100, Stefan Hajnoczi wrote:
> On Thu, Jun 14, 2012 at 2:49 PM, Richard W.M. Jones <rjones@redhat.com> wrote:
> > I switched libguestfs over to using virtio-scsi.  One immediate
> > benefit is support for large numbers of disks: up to 255 because we're
> > using 1 target / disk and we reserve one disk for the appliance, in
> > theory more could be supported if we used LUNs.
> 
> Cool, thanks for sharing!  Does libguestfs hotplug LUNS/targets?

Not quite, but in the near future it will.  It's something I'm aiming
to have working for RHEL 7.

> Cong and Wan Seng are implementing guest driver and QEMU-side changes
> so the guest receives notifications of hotplugged LUNs.  It's nicer
> than manually rescanning the bus from inside the guest.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora