Oracle RAC in libvirt+KVM environment

Oracle RAC in libvirt+KVM environment

[Timon Wang]

We wan't to setup two Oracle instance and make RAC work on them.
Both VM are setup based on libvirt + KVM, we use a lvm lun which
formated in qcow2 format and set the shareable properties in the disk
driver like this:

    <disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source dev='/dev/81035c32-d2e4-4aaf-82fa-3e76ae587586/ca18f6a5-7c98-46ea-b562-9424e68a52f3'/>
      <target dev='hda' bus='ide'/>
      <shareable/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

But when we use it as shared data disk like we use iscsi/FC disk for
shared data disk, the disk can't be recognized as a share disk, and
RAC can't be setup in the environment.

Any body can give me some guild on this?

I am wondering shareable in the libvirt config really makes the disk shareable?

How can I setup the RAC environment in this situable?

I found that vmware and vbox can do this according to a blog post, I
am wondering if kvm has this ability to make a disk really shareable.

-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Anybody have idea on it?

I tried to set the disk as raw format, and retried the setup process,
but still can't get through.

On Fri, Aug 2, 2013 at 1:58 PM, Timon Wang <timonwst@gmail.com> wrote:
> We wan't to setup two Oracle instance and make RAC work on them.
> Both VM are setup based on libvirt + KVM, we use a lvm lun which
> formated in qcow2 format and set the shareable properties in the disk
> driver like this:
>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='qcow2' cache='none'/>
>       <source dev='/dev/81035c32-d2e4-4aaf-82fa-3e76ae587586/ca18f6a5-7c98-46ea-b562-9424e68a52f3'/>
>       <target dev='hda' bus='ide'/>
>       <shareable/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
>
> But when we use it as shared data disk like we use iscsi/FC disk for
> shared data disk, the disk can't be recognized as a share disk, and
> RAC can't be setup in the environment.
>
> Any body can give me some guild on this?
>
> I am wondering shareable in the libvirt config really makes the disk shareable?
>
> How can I setup the RAC environment in this situable?
>
> I found that vmware and vbox can do this according to a blog post, I
> am wondering if kvm has this ability to make a disk really shareable.
>
> --
> Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
> Blog: http://www.nohouse.net



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Don Koch]

On 08/08/2013 03:54 AM, Timon Wang wrote:
> Anybody have idea on it?
> 
> I tried to set the disk as raw format, and retried the setup process,
> but still can't get through.

Caveat: I know nothing in particular about Oracle RAC, but...

Assuming that RAC uses something like SCSI reservations in order to
share the disk, I would guess it doesn't like the disk being on the IDE
bus.

> 
> On Fri, Aug 2, 2013 at 1:58 PM, Timon Wang <timonwst-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> We wan't to setup two Oracle instance and make RAC work on them.
>> Both VM are setup based on libvirt + KVM, we use a lvm lun which
>> formated in qcow2 format and set the shareable properties in the disk
>> driver like this:
>>
>>     <disk type='block' device='disk'>
>>       <driver name='qemu' type='qcow2' cache='none'/>
>>       <source dev='/dev/81035c32-d2e4-4aaf-82fa-3e76ae587586/ca18f6a5-7c98-46ea-b562-9424e68a52f3'/>
>>       <target dev='hda' bus='ide'/>
>>       <shareable/>
>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>     </disk>
>>
>> But when we use it as shared data disk like we use iscsi/FC disk for
>> shared data disk, the disk can't be recognized as a share disk, and
>> RAC can't be setup in the environment.
>>
>> Any body can give me some guild on this?
>>
>> I am wondering shareable in the libvirt config really makes the disk shareable?
>>
>> How can I setup the RAC environment in this situable?
>>
>> I found that vmware and vbox can do this according to a blog post, I
>> am wondering if kvm has this ability to make a disk really shareable.
>>
>> --
>> Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
>> Blog: http://www.nohouse.net
> 
> 
> 

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

I have tryied change the disk bus to SCSI, add a SCSI controller whose
model is virtio-scsi, still can't setup the RAC instance.

I tried to use windows 2008 Failover Cluster feature to setup a a
Failover Cluster instead, and I can't find any cluster disk to share
between two nodes. So when Failover Cluster is setup, I can't add any
Cluster disk to the Failover Cluster.

Have I missed some thing?

On Thu, Aug 8, 2013 at 10:04 PM, Don Koch <dkoch@verizon.com> wrote:
> On 08/08/2013 03:54 AM, Timon Wang wrote:
>> Anybody have idea on it?
>>
>> I tried to set the disk as raw format, and retried the setup process,
>> but still can't get through.
>
> Caveat: I know nothing in particular about Oracle RAC, but...
>
> Assuming that RAC uses something like SCSI reservations in order to
> share the disk, I would guess it doesn't like the disk being on the IDE
> bus.
>
>>
>> On Fri, Aug 2, 2013 at 1:58 PM, Timon Wang <timonwst@gmail.com> wrote:
>>> We wan't to setup two Oracle instance and make RAC work on them.
>>> Both VM are setup based on libvirt + KVM, we use a lvm lun which
>>> formated in qcow2 format and set the shareable properties in the disk
>>> driver like this:
>>>
>>>     <disk type='block' device='disk'>
>>>       <driver name='qemu' type='qcow2' cache='none'/>
>>>       <source dev='/dev/81035c32-d2e4-4aaf-82fa-3e76ae587586/ca18f6a5-7c98-46ea-b562-9424e68a52f3'/>
>>>       <target dev='hda' bus='ide'/>
>>>       <shareable/>
>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>     </disk>
>>>
>>> But when we use it as shared data disk like we use iscsi/FC disk for
>>> shared data disk, the disk can't be recognized as a share disk, and
>>> RAC can't be setup in the environment.
>>>
>>> Any body can give me some guild on this?
>>>
>>> I am wondering shareable in the libvirt config really makes the disk shareable?
>>>
>>> How can I setup the RAC environment in this situable?
>>>
>>> I found that vmware and vbox can do this according to a blog post, I
>>> am wondering if kvm has this ability to make a disk really shareable.
>>>
>>> --
>>> Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
>>> Blog: http://www.nohouse.net
>>
>>
>>
>



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Stefan Hajnoczi]

On Fri, Aug 02, 2013 at 01:58:24PM +0800, Timon Wang wrote:
> We wan't to setup two Oracle instance and make RAC work on them.
> Both VM are setup based on libvirt + KVM, we use a lvm lun which
> formated in qcow2 format and set the shareable properties in the disk
> driver like this:
> 
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='qcow2' cache='none'/>

qcow2 is not cluster-aware, it cannot be opened by multiple VMs at the
same time.

You must use raw.

Stefan

Re: Oracle RAC in libvirt+KVM environment

[Stefan Hajnoczi]

On Sat, Aug 10, 2013 at 11:14:39AM +0800, Timon Wang wrote:
> I have tryied change the disk bus to SCSI, add a SCSI controller whose
> model is virtio-scsi, still can't setup the RAC instance.
> 
> I tried to use windows 2008 Failover Cluster feature to setup a a
> Failover Cluster instead, and I can't find any cluster disk to share
> between two nodes. So when Failover Cluster is setup, I can't add any
> Cluster disk to the Failover Cluster.
> 
> Have I missed some thing?

I'm not sure what SCSI-level requirements RAC or Failover Cluster have.

If anyone knows which features are needed it would be possible to
confirm whether they are supported under KVM.

I expect this can only work if you are passing through a shared LUN.
Can you describe your configuration?

Stefan

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Yes, SCSI bus likes pass through a shared LUN to the vm, and I am
using a shared LUN for 'share' purpose.

I found a post that vmware use lsilogic bus for the shared disk, but
my qemu/kvm version can't support lsilogic bus.

I'm tring to update qemu/kvm version for lsilogic bus support.


On 8/12/13, Stefan Hajnoczi <stefanha-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Sat, Aug 10, 2013 at 11:14:39AM +0800, Timon Wang wrote:
>> I have tryied change the disk bus to SCSI, add a SCSI controller whose
>> model is virtio-scsi, still can't setup the RAC instance.
>>
>> I tried to use windows 2008 Failover Cluster feature to setup a a
>> Failover Cluster instead, and I can't find any cluster disk to share
>> between two nodes. So when Failover Cluster is setup, I can't add any
>> Cluster disk to the Failover Cluster.
>>
>> Have I missed some thing?
>
> I'm not sure what SCSI-level requirements RAC or Failover Cluster have.
>
> If anyone knows which features are needed it would be possible to
> confirm whether they are supported under KVM.
>
> I expect this can only work if you are passing through a shared LUN.
> Can you describe your configuration?
>
> Stefan
>


-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Stefan Hajnoczi]

On Mon, Aug 12, 2013 at 06:17:51PM +0800, Timon Wang wrote:
> Yes, SCSI bus likes pass through a shared LUN to the vm, and I am
> using a shared LUN for 'share' purpose.
> 
> I found a post that vmware use lsilogic bus for the shared disk, but
> my qemu/kvm version can't support lsilogic bus.
> 
> I'm tring to update qemu/kvm version for lsilogic bus support.

Use virtio-scsi.  The emulated LSI SCSI controller has known bugs and is
not actively developed - don't be surprised if you hit issues with it.

The question is still what commands RAC or Failover Clustering use.  If
you find that the software refuses to run, it could be because
additional work is required to make it work on KVM.

Stefan

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Thanks, i have tried virtio-scsi and LSI SCSI controller, by using a
local file or a LUN as disk. Both raw and qcow2 disk format are be
verified, but no way to make windows Failover Cluster and Oracle RAC
work correctly.

Actually, Oracle RAC and Windows Failover Cluster needs "share"
storage like iscsi disk and fc disk to work correctly.

In iscsi situation, we can use ip network for Failover Cluster "share"
storage, but when we use fc as a high performance solution, we can't
use fc "share" storage direct in VM(Not LUN passthrough).

As I google the windows Failover Cluster requirements, I found that
this issue may be that windows checks only iscsi and fc drivers, the
type of the disk may be check by the driver type. In KVM situation, we
can't emulate a iscsi/fc disk for VM to use, so we can't setup a
environment like this.

I'm wondering, if we want't to make sure the applications and service
in the VM to always work, it's there another way to do this in KVM
environment.

On Tue, Aug 13, 2013 at 5:18 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Mon, Aug 12, 2013 at 06:17:51PM +0800, Timon Wang wrote:
>> Yes, SCSI bus likes pass through a shared LUN to the vm, and I am
>> using a shared LUN for 'share' purpose.
>>
>> I found a post that vmware use lsilogic bus for the shared disk, but
>> my qemu/kvm version can't support lsilogic bus.
>>
>> I'm tring to update qemu/kvm version for lsilogic bus support.
>
> Use virtio-scsi.  The emulated LSI SCSI controller has known bugs and is
> not actively developed - don't be surprised if you hit issues with it.
>
> The question is still what commands RAC or Failover Clustering use.  If
> you find that the software refuses to run, it could be because
> additional work is required to make it work on KVM.
>
> Stefan



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Hi all,

I found a article about Hyper-V virtual Fiber Channel, I think this
will make Failover Cluster work if KVM has the same feature.
http://technet.microsoft.com/en-us/library/hh831413.aspx

Hyper-V uses NPIV for virtual Fiber Channel, I have read some article
about KVM NPIV, but how can I config it with libvirt? Any body can
show me some example?


On Wed, Aug 14, 2013 at 2:33 PM, Timon Wang <timonwst@gmail.com> wrote:
> Thanks, i have tried virtio-scsi and LSI SCSI controller, by using a
> local file or a LUN as disk. Both raw and qcow2 disk format are be
> verified, but no way to make windows Failover Cluster and Oracle RAC
> work correctly.
>
> Actually, Oracle RAC and Windows Failover Cluster needs "share"
> storage like iscsi disk and fc disk to work correctly.
>
> In iscsi situation, we can use ip network for Failover Cluster "share"
> storage, but when we use fc as a high performance solution, we can't
> use fc "share" storage direct in VM(Not LUN passthrough).
>
> As I google the windows Failover Cluster requirements, I found that
> this issue may be that windows checks only iscsi and fc drivers, the
> type of the disk may be check by the driver type. In KVM situation, we
> can't emulate a iscsi/fc disk for VM to use, so we can't setup a
> environment like this.
>
> I'm wondering, if we want't to make sure the applications and service
> in the VM to always work, it's there another way to do this in KVM
> environment.
>
> On Tue, Aug 13, 2013 at 5:18 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Mon, Aug 12, 2013 at 06:17:51PM +0800, Timon Wang wrote:
>>> Yes, SCSI bus likes pass through a shared LUN to the vm, and I am
>>> using a shared LUN for 'share' purpose.
>>>
>>> I found a post that vmware use lsilogic bus for the shared disk, but
>>> my qemu/kvm version can't support lsilogic bus.
>>>
>>> I'm tring to update qemu/kvm version for lsilogic bus support.
>>
>> Use virtio-scsi.  The emulated LSI SCSI controller has known bugs and is
>> not actively developed - don't be surprised if you hit issues with it.
>>
>> The question is still what commands RAC or Failover Clustering use.  If
>> you find that the software refuses to run, it could be because
>> additional work is required to make it work on KVM.
>>
>> Stefan
>
>
>
> --
> Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
> Blog: http://www.nohouse.net



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Stefan Hajnoczi]

On Wed, Aug 14, 2013 at 04:40:44PM +0800, Timon Wang wrote:
> I found a article about Hyper-V virtual Fiber Channel, I think this
> will make Failover Cluster work if KVM has the same feature.
> http://technet.microsoft.com/en-us/library/hh831413.aspx
> 
> Hyper-V uses NPIV for virtual Fiber Channel, I have read some article
> about KVM NPIV, but how can I config it with libvirt? Any body can
> show me some example?

A web search turns up this:

https://docs.fedoraproject.org/en-US/Fedora/18/html/Virtualization_Administration_Guide/sect-Technical_Papers-Identifying_HBAs_in_a_Host_System-Confirming_That_IO_Traffic_is_Going_through_an_NPIV_HBA.html

You can use this if the host has a supported Fibre Channel HBA and your
image is on a SAN LUN.

>From my limited knowledge about this, NPIV itself won't make clustering
possible.  RAC or Failure Cluster probably still require specific SCSI
commands in order to work (like persistent reservations) and that's what
needs to be investigated in order to figure out a solution.

Stefan

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Thanks.

I have read the link you provide, there is another link which tells me
to pass a NPIV discovery lun as a disk, this is seen as a local direct
access disk in windows. RAC and Failure Cluster both consider this
pass through disk as local disk, not a share disk, and the setup
process failed.

Hyper-v provides a virtual Fiber Channel implementation, so I
wondering if kvm has the same solution like it.


On Wed, Aug 14, 2013 at 10:02 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Wed, Aug 14, 2013 at 04:40:44PM +0800, Timon Wang wrote:
>> I found a article about Hyper-V virtual Fiber Channel, I think this
>> will make Failover Cluster work if KVM has the same feature.
>> http://technet.microsoft.com/en-us/library/hh831413.aspx
>>
>> Hyper-V uses NPIV for virtual Fiber Channel, I have read some article
>> about KVM NPIV, but how can I config it with libvirt? Any body can
>> show me some example?
>
> A web search turns up this:
>
> https://docs.fedoraproject.org/en-US/Fedora/18/html/Virtualization_Administration_Guide/sect-Technical_Papers-Identifying_HBAs_in_a_Host_System-Confirming_That_IO_Traffic_is_Going_through_an_NPIV_HBA.html
>
> You can use this if the host has a supported Fibre Channel HBA and your
> image is on a SAN LUN.
>
> From my limited knowledge about this, NPIV itself won't make clustering
> possible.  RAC or Failure Cluster probably still require specific SCSI
> commands in order to work (like persistent reservations) and that's what
> needs to be investigated in order to figure out a solution.
>
> Stefan



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Paolo Bonzini]

Il 15/08/2013 12:01, Timon Wang ha scritto:
> Thanks.
> 
> I have read the link you provide, there is another link which tells me
> to pass a NPIV discovery lun as a disk, this is seen as a local direct
> access disk in windows. RAC and Failure Cluster both consider this
> pass through disk as local disk, not a share disk, and the setup
> process failed.
> 
> Hyper-v provides a virtual Fiber Channel implementation, so I
> wondering if kvm has the same solution like it.

Can you include the XML file you are using for the domain?

Paolo

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

My domain xml is like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>2008-2</name>
  <uuid>6325d8a5-468d-42e9-b5cb-9a04f5f34e80</uuid>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='floppy'>
      <driver name='qemu' type='raw' cache='none'/>
      <target dev='fda' bus='fdc'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/images/win2008_2_sys'/>
      <target dev='hda' bus='ide'/>
      <boot order='3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/isos/windows2008_64r2.iso'/>
      <target dev='sdc' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/fedora/q_disk'/>
      <target dev='sda' bus='virtio'/>
      <shareable/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
function='0x0'/>
    </disk>
    <controller type='fdc' index='0'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </controller>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:71:20:ae'/>
      <source bridge='br0'/>
      <target dev='vport2'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:12:a0:fd'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='spice' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <sound model='ac97'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='32768' heads='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-rtc-td-hack'/>
  </qemu:commandline>
</domain>



On 8/19/13, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 15/08/2013 12:01, Timon Wang ha scritto:
>> Thanks.
>>
>> I have read the link you provide, there is another link which tells me
>> to pass a NPIV discovery lun as a disk, this is seen as a local direct
>> access disk in windows. RAC and Failure Cluster both consider this
>> pass through disk as local disk, not a share disk, and the setup
>> process failed.
>>
>> Hyper-v provides a virtual Fiber Channel implementation, so I
>> wondering if kvm has the same solution like it.
>
> Can you include the XML file you are using for the domain?
>
> Paolo
>
>


-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Right now, I found that Windows Failover Cluster needs SCSI-3
Persistent Reservation, I don't know where if virtio-scsi supports
this, according to http://www.ovirt.org/Features/Virtio-SCSI I found
this:

limited flexibility: virtio-blk does not support all possible storage
scenarios. For example, it does not allow SCSI passthrough or
persistent reservations. In principle, virtio-scsi provides anything
that the underlying SCSI target (be it physical storage, iSCSI or the
in-kernel target) supports.

virtio-blk does not support persistent reservations, but virtio-scsi
may surpport.

Another web page which archive mail lists
http://www.spinics.net/lists/target-devel/msg01813.html, which request
virtio-scsi to implement SPC-3 (persistent reservation) feature, but I
can't found any more information about this.

On Tue, Aug 20, 2013 at 2:00 PM, Timon Wang <timonwst@gmail.com> wrote:
> My domain xml is like this:
>
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>   <name>2008-2</name>
>   <uuid>6325d8a5-468d-42e9-b5cb-9a04f5f34e80</uuid>
>   <memory unit='KiB'>524288</memory>
>   <currentMemory unit='KiB'>524288</currentMemory>
>   <vcpu placement='static'>2</vcpu>
>   <os>
>     <type arch='x86_64' machine='pc-i440fx-1.4'>hvm</type>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <pae/>
>   </features>
>   <clock offset='localtime'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <devices>
>     <emulator>/usr/bin/qemu-kvm</emulator>
>     <disk type='file' device='floppy'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <target dev='fda' bus='fdc'/>
>       <readonly/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <source file='/home/images/win2008_2_sys'/>
>       <target dev='hda' bus='ide'/>
>       <boot order='3'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
>     <disk type='file' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <source file='/home/isos/windows2008_64r2.iso'/>
>       <target dev='sdc' bus='ide'/>
>       <readonly/>
>       <boot order='1'/>
>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>     </disk>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw'/>
>       <source dev='/dev/fedora/q_disk'/>
>       <target dev='sda' bus='virtio'/>
>       <shareable/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
> function='0x0'/>
>     </disk>
>     <controller type='fdc' index='0'/>
>     <controller type='ide' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x1'/>
>     </controller>
>     <controller type='virtio-serial' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
>     </controller>
>     <controller type='usb' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
>     </controller>
>     <controller type='pci' index='0' model='pci-root'/>
>     <controller type='scsi' index='0' model='virtio-scsi'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </controller>
>     <interface type='bridge'>
>       <mac address='52:54:00:71:20:ae'/>
>       <source bridge='br0'/>
>       <target dev='vport2'/>
>       <model type='rtl8139'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>     <interface type='network'>
>       <mac address='52:54:00:12:a0:fd'/>
>       <source network='default'/>
>       <model type='rtl8139'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <target port='0'/>
>     </serial>
>     <console type='pty'>
>       <target type='serial' port='0'/>
>     </console>
>     <input type='tablet' bus='usb'/>
>     <input type='mouse' bus='ps2'/>
>     <graphics type='spice' autoport='yes' listen='0.0.0.0'>
>       <listen type='address' address='0.0.0.0'/>
>     </graphics>
>     <sound model='ac97'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
>     </sound>
>     <video>
>       <model type='qxl' ram='65536' vram='32768' heads='2'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </video>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
>     </memballoon>
>   </devices>
>   <qemu:commandline>
>     <qemu:arg value='-rtc-td-hack'/>
>   </qemu:commandline>
> </domain>
>
>
>
> On 8/19/13, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 15/08/2013 12:01, Timon Wang ha scritto:
>>> Thanks.
>>>
>>> I have read the link you provide, there is another link which tells me
>>> to pass a NPIV discovery lun as a disk, this is seen as a local direct
>>> access disk in windows. RAC and Failure Cluster both consider this
>>> pass through disk as local disk, not a share disk, and the setup
>>> process failed.
>>>
>>> Hyper-v provides a virtual Fiber Channel implementation, so I
>>> wondering if kvm has the same solution like it.
>>
>> Can you include the XML file you are using for the domain?
>>
>> Paolo
>>
>>
>
>
> --
> Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
> Blog: http://www.nohouse.net



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Paolo Bonzini]

Il 20/08/2013 08:00, Timon Wang ha scritto:
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <source file='/home/images/win2008_2_sys'/>
>       <target dev='hda' bus='ide'/>
>       <boot order='3'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
>     <disk type='file' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <source file='/home/isos/windows2008_64r2.iso'/>
>       <target dev='sdc' bus='ide'/>
>       <readonly/>
>       <boot order='1'/>
>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>     </disk>
>     <disk type='block' device='disk'>

I'm not sure this will be enough, but if you want passthrough to the
host device you should use device='lun' here.  However, you still would
not be able to issue SCSI reservations unless you run QEMU with the
CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").

Most important, it still would be unsafe to do this if the same device
is passed to multiple virtual machines on the same host.  You need to
have NPIV and create separate virtual HBAs.  Then each virtual machine
should get a separate virtual HBA.  Otherwise, persistent reservations
are not attached to a particular virtual machine, but generically to the
host.

>       <driver name='qemu' type='raw'/>
>       <source dev='/dev/fedora/q_disk'/>
>       <target dev='sda' bus='virtio'/>

You are not exposing a virtio-scsi disk here.  You are exposing a
virtio-blk disk.  You can see this from the type='pci' address that
libvirt gave to the disk.

If you use bus='scsi', you will see that libvirt will use type='drive'
for the address.

>     <controller type='scsi' index='0' model='virtio-scsi'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </controller>

This is okay.

>   <qemu:commandline>
>     <qemu:arg value='-rtc-td-hack'/>
>   </qemu:commandline>

FWIW, this can be replaced with

  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
  </clock>

(you already have the <clock> element, but no <timer> element inside).

Paolo

> </domain>
> 
> 
> 
> On 8/19/13, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 15/08/2013 12:01, Timon Wang ha scritto:
>>> Thanks.
>>>
>>> I have read the link you provide, there is another link which tells me
>>> to pass a NPIV discovery lun as a disk, this is seen as a local direct
>>> access disk in windows. RAC and Failure Cluster both consider this
>>> pass through disk as local disk, not a share disk, and the setup
>>> process failed.
>>>
>>> Hyper-v provides a virtual Fiber Channel implementation, so I
>>> wondering if kvm has the same solution like it.
>>
>> Can you include the XML file you are using for the domain?
>>
>> Paolo
>>
>>
> 
> 

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 20/08/2013 08:00, Timon Wang ha scritto:
>>     <disk type='file' device='disk'>
>>       <driver name='qemu' type='raw' cache='none'/>
>>       <source file='/home/images/win2008_2_sys'/>
>>       <target dev='hda' bus='ide'/>
>>       <boot order='3'/>
>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>     </disk>
>>     <disk type='file' device='cdrom'>
>>       <driver name='qemu' type='raw'/>
>>       <source file='/home/isos/windows2008_64r2.iso'/>
>>       <target dev='sdc' bus='ide'/>
>>       <readonly/>
>>       <boot order='1'/>
>>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>     </disk>
>>     <disk type='block' device='disk'>
>
> I'm not sure this will be enough, but if you want passthrough to the
> host device you should use device='lun' here.  However, you still would
> not be able to issue SCSI reservations unless you run QEMU with the
> CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
>

After change the libvirt xml like this:
<disk type='block' device='lun' rawio='yes'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/>
      <target dev='sda' bus='scsi'/>
      <shareable/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
I got these errors:
char device redirected to /dev/pts/1 (label charserial0)
qemu-system-x86_64: -device
scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
scsi-block: INQUIRY failed
qemu-system-x86_64: -device
scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
Device 'scsi-block' could not be initialized


> Most important, it still would be unsafe to do this if the same device
> is passed to multiple virtual machines on the same host.  You need to
> have NPIV and create separate virtual HBAs.  Then each virtual machine
> should get a separate virtual HBA.  Otherwise, persistent reservations
> are not attached to a particular virtual machine, but generically to the
> host.

How to use NPIV virtual HBAs with libvirt xml configurations? I can
define nodedev, but have no idea about how to pass the nodedev to the
vm.

>
>>       <driver name='qemu' type='raw'/>
>>       <source dev='/dev/fedora/q_disk'/>
>>       <target dev='sda' bus='virtio'/>
>
> You are not exposing a virtio-scsi disk here.  You are exposing a
> virtio-blk disk.  You can see this from the type='pci' address that
> libvirt gave to the disk.
>
> If you use bus='scsi', you will see that libvirt will use type='drive'
> for the address.
>
>>     <controller type='scsi' index='0' model='virtio-scsi'>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
>> function='0x0'/>
>>     </controller>
>
> This is okay.
>
>>   <qemu:commandline>
>>     <qemu:arg value='-rtc-td-hack'/>
>>   </qemu:commandline>
>
> FWIW, this can be replaced with
>
>   <clock offset='localtime'>
>     <timer name='rtc' tickpolicy='catchup'/>
>   </clock>
>
> (you already have the <clock> element, but no <timer> element inside).

Thanks for this tip.

>
> Paolo
>
>> </domain>
>>
>>
>>
>> On 8/19/13, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>> Il 15/08/2013 12:01, Timon Wang ha scritto:
>>>> Thanks.
>>>>
>>>> I have read the link you provide, there is another link which tells me
>>>> to pass a NPIV discovery lun as a disk, this is seen as a local direct
>>>> access disk in windows. RAC and Failure Cluster both consider this
>>>> pass through disk as local disk, not a share disk, and the setup
>>>> process failed.
>>>>
>>>> Hyper-v provides a virtual Fiber Channel implementation, so I
>>>> wondering if kvm has the same solution like it.
>>>
>>> Can you include the XML file you are using for the domain?
>>>
>>> Paolo
>>>
>>>
>>
>>
>

-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Paolo Bonzini]

Il 20/08/2013 11:59, Timon Wang ha scritto:
> On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 20/08/2013 08:00, Timon Wang ha scritto:
>>>     <disk type='file' device='disk'>
>>>       <driver name='qemu' type='raw' cache='none'/>
>>>       <source file='/home/images/win2008_2_sys'/>
>>>       <target dev='hda' bus='ide'/>
>>>       <boot order='3'/>
>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>     </disk>
>>>     <disk type='file' device='cdrom'>
>>>       <driver name='qemu' type='raw'/>
>>>       <source file='/home/isos/windows2008_64r2.iso'/>
>>>       <target dev='sdc' bus='ide'/>
>>>       <readonly/>
>>>       <boot order='1'/>
>>>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>>     </disk>
>>>     <disk type='block' device='disk'>
>>
>> I'm not sure this will be enough, but if you want passthrough to the
>> host device you should use device='lun' here.  However, you still would
>> not be able to issue SCSI reservations unless you run QEMU with the
>> CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
>>
> 
> After change the libvirt xml like this:
> <disk type='block' device='lun' rawio='yes'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/>
>       <target dev='sda' bus='scsi'/>
>       <shareable/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
> I got these errors:
> char device redirected to /dev/pts/1 (label charserial0)
> qemu-system-x86_64: -device
> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
> scsi-block: INQUIRY failed
> qemu-system-x86_64: -device
> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
> Device 'scsi-block' could not be initialized

Can you do

# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk

?

Paolo

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

[root@localhost /]# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
lrwxrwxrwx. 1 root root 8 8月  20 17:38
/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk -> ../dm-13
[root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
  SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
  EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
    length=36 (0x24)   Peripheral device type: disk
 Vendor identification: MacroSAN
 Product identification: LU
 Product revision level: 1.0
 Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300

This lun is from a vg build based on iscsi target.

[root@localhost /]# libvirtd --version
libvirtd (libvirt) 1.0.5
[root@localhost /]# qemu-kvm --version
QEMU emulator version 1.4.1, Copyright (c) 2003-2008 Fabrice Bellard
[root@localhost /]# uname -a
Linux localhost.localdomain 3.9.2-301.fc19.x86_64 #1 SMP Mon May 13
12:36:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux


On Tue, Aug 20, 2013 at 6:16 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 20/08/2013 11:59, Timon Wang ha scritto:
>> On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>> Il 20/08/2013 08:00, Timon Wang ha scritto:
>>>>     <disk type='file' device='disk'>
>>>>       <driver name='qemu' type='raw' cache='none'/>
>>>>       <source file='/home/images/win2008_2_sys'/>
>>>>       <target dev='hda' bus='ide'/>
>>>>       <boot order='3'/>
>>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>     </disk>
>>>>     <disk type='file' device='cdrom'>
>>>>       <driver name='qemu' type='raw'/>
>>>>       <source file='/home/isos/windows2008_64r2.iso'/>
>>>>       <target dev='sdc' bus='ide'/>
>>>>       <readonly/>
>>>>       <boot order='1'/>
>>>>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>>>     </disk>
>>>>     <disk type='block' device='disk'>
>>>
>>> I'm not sure this will be enough, but if you want passthrough to the
>>> host device you should use device='lun' here.  However, you still would
>>> not be able to issue SCSI reservations unless you run QEMU with the
>>> CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
>>>
>>
>> After change the libvirt xml like this:
>> <disk type='block' device='lun' rawio='yes'>
>>       <driver name='qemu' type='raw' cache='none'/>
>>       <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/>
>>       <target dev='sda' bus='scsi'/>
>>       <shareable/>
>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>     </disk>
>> I got these errors:
>> char device redirected to /dev/pts/1 (label charserial0)
>> qemu-system-x86_64: -device
>> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
>> scsi-block: INQUIRY failed
>> qemu-system-x86_64: -device
>> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
>> Device 'scsi-block' could not be initialized
>
> Can you do
>
> # ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
> # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>
> ?
>
> Paolo
>



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Paolo Bonzini]

Il 20/08/2013 12:42, Timon Wang ha scritto:
> [root@localhost /]# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
> lrwxrwxrwx. 1 root root 8 8月  20 17:38
> /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk -> ../dm-13
> [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
> standard INQUIRY:
>   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>     length=36 (0x24)   Peripheral device type: disk
>  Vendor identification: MacroSAN
>  Product identification: LU
>  Product revision level: 1.0
>  Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
> 
> This lun is from a vg build based on iscsi target.

If it is a logical volume, you cannot pass it as a LUN to the guest.
Only the whole iSCSI LUN can be passed as a LUN.

Paolo

> [root@localhost /]# libvirtd --version
> libvirtd (libvirt) 1.0.5
> [root@localhost /]# qemu-kvm --version
> QEMU emulator version 1.4.1, Copyright (c) 2003-2008 Fabrice Bellard
> [root@localhost /]# uname -a
> Linux localhost.localdomain 3.9.2-301.fc19.x86_64 #1 SMP Mon May 13
> 12:36:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> On Tue, Aug 20, 2013 at 6:16 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 20/08/2013 11:59, Timon Wang ha scritto:
>>> On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>> Il 20/08/2013 08:00, Timon Wang ha scritto:
>>>>>     <disk type='file' device='disk'>
>>>>>       <driver name='qemu' type='raw' cache='none'/>
>>>>>       <source file='/home/images/win2008_2_sys'/>
>>>>>       <target dev='hda' bus='ide'/>
>>>>>       <boot order='3'/>
>>>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>>     </disk>
>>>>>     <disk type='file' device='cdrom'>
>>>>>       <driver name='qemu' type='raw'/>
>>>>>       <source file='/home/isos/windows2008_64r2.iso'/>
>>>>>       <target dev='sdc' bus='ide'/>
>>>>>       <readonly/>
>>>>>       <boot order='1'/>
>>>>>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>>>>     </disk>
>>>>>     <disk type='block' device='disk'>
>>>>
>>>> I'm not sure this will be enough, but if you want passthrough to the
>>>> host device you should use device='lun' here.  However, you still would
>>>> not be able to issue SCSI reservations unless you run QEMU with the
>>>> CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
>>>>
>>>
>>> After change the libvirt xml like this:
>>> <disk type='block' device='lun' rawio='yes'>
>>>       <driver name='qemu' type='raw' cache='none'/>
>>>       <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/>
>>>       <target dev='sda' bus='scsi'/>
>>>       <shareable/>
>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>     </disk>
>>> I got these errors:
>>> char device redirected to /dev/pts/1 (label charserial0)
>>> qemu-system-x86_64: -device
>>> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
>>> scsi-block: INQUIRY failed
>>> qemu-system-x86_64: -device
>>> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
>>> Device 'scsi-block' could not be initialized
>>
>> Can you do
>>
>> # ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>> # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>
>> ?
>>
>> Paolo
>>
> 
> 
> 

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Thanks, the whole iSCSI LUN have been passed to the VM.

But I test it with scsicmd, and found that the driver may be not
support SPC-3, but if i use this by microsoft iscsi initiator, I can
pass all the scsi3_test tests.

Tool can be found here:
http://www.symantec.com/business/support/index?page=content&id=TECH72086

It's this means that the scsi passthrough windows driver does not
support SPC-3 feature, I have read a post about this, it says if
support this we should change both the implementation and the
documents in virtio spec.

I am new to this list, so I don't know what is the situation right now?

Would somebody please give me some advise on it?


On Tue, Aug 20, 2013 at 6:49 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 20/08/2013 12:42, Timon Wang ha scritto:
>> [root@localhost /]# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>> lrwxrwxrwx. 1 root root 8 8月  20 17:38
>> /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk -> ../dm-13
>> [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>> standard INQUIRY:
>>   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>>   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>>   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>>   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>>     length=36 (0x24)   Peripheral device type: disk
>>  Vendor identification: MacroSAN
>>  Product identification: LU
>>  Product revision level: 1.0
>>  Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
>>
>> This lun is from a vg build based on iscsi target.
>
> If it is a logical volume, you cannot pass it as a LUN to the guest.
> Only the whole iSCSI LUN can be passed as a LUN.
>
> Paolo
>
>> [root@localhost /]# libvirtd --version
>> libvirtd (libvirt) 1.0.5
>> [root@localhost /]# qemu-kvm --version
>> QEMU emulator version 1.4.1, Copyright (c) 2003-2008 Fabrice Bellard
>> [root@localhost /]# uname -a
>> Linux localhost.localdomain 3.9.2-301.fc19.x86_64 #1 SMP Mon May 13
>> 12:36:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>>
>>
>> On Tue, Aug 20, 2013 at 6:16 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>> Il 20/08/2013 11:59, Timon Wang ha scritto:
>>>> On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>>> Il 20/08/2013 08:00, Timon Wang ha scritto:
>>>>>>     <disk type='file' device='disk'>
>>>>>>       <driver name='qemu' type='raw' cache='none'/>
>>>>>>       <source file='/home/images/win2008_2_sys'/>
>>>>>>       <target dev='hda' bus='ide'/>
>>>>>>       <boot order='3'/>
>>>>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>>>     </disk>
>>>>>>     <disk type='file' device='cdrom'>
>>>>>>       <driver name='qemu' type='raw'/>
>>>>>>       <source file='/home/isos/windows2008_64r2.iso'/>
>>>>>>       <target dev='sdc' bus='ide'/>
>>>>>>       <readonly/>
>>>>>>       <boot order='1'/>
>>>>>>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>>>>>     </disk>
>>>>>>     <disk type='block' device='disk'>
>>>>>
>>>>> I'm not sure this will be enough, but if you want passthrough to the
>>>>> host device you should use device='lun' here.  However, you still would
>>>>> not be able to issue SCSI reservations unless you run QEMU with the
>>>>> CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
>>>>>
>>>>
>>>> After change the libvirt xml like this:
>>>> <disk type='block' device='lun' rawio='yes'>
>>>>       <driver name='qemu' type='raw' cache='none'/>
>>>>       <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/>
>>>>       <target dev='sda' bus='scsi'/>
>>>>       <shareable/>
>>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>     </disk>
>>>> I got these errors:
>>>> char device redirected to /dev/pts/1 (label charserial0)
>>>> qemu-system-x86_64: -device
>>>> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
>>>> scsi-block: INQUIRY failed
>>>> qemu-system-x86_64: -device
>>>> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
>>>> Device 'scsi-block' could not be initialized
>>>
>>> Can you do
>>>
>>> # ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>> # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>>
>>> ?
>>>
>>> Paolo
>>>
>>
>>
>>
>



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

I found when I use "scsicmd -d1 -s13" test command to test the
"controller bus reset" request, there will be a blue screen on windows
2008 r2.

The error code is :
BugCheck D1, {4, a, 0, fffff8800154dd06}

1: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000004, memory referenced
Arg2: 000000000000000a, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff8800154dd06, address which referenced memory

Debugging Details:
------------------

Page 17c41 not present in the dump file. Type ".hh dbgerr004" for details

READ_ADDRESS:  0000000000000004

CURRENT_IRQL:  a

FAULTING_IP:
vioscsi+1d06
fffff880`0154dd06 458b4804        mov     r9d,dword ptr [r8+4]

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0xD1

PROCESS_NAME:  scsicmd.exe

TRAP_FRAME:  fffff880009f7670 -- (.trap 0xfffff880009f7670)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000002 rbx=0000000000000000 rcx=fffffa800065e738
rdx=fffffa800065e8f8 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8800154dd06 rsp=fffff880009f7800 rbp=fffffa800065e8f8
 r8=0000000000000000  r9=0000000000000000 r10=fffffa80009155b0
r11=fffff880009f7848 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
vioscsi+0x1d06:
fffff880`0154dd06 458b4804        mov     r9d,dword ptr [r8+4]
ds:e630:0004=????????
Resetting default scope

LAST_CONTROL_TRANSFER:  from fffff800016ca469 to fffff800016caf00

STACK_TEXT:
fffff880`009f7528 fffff800`016ca469 : 00000000`0000000a
00000000`00000004 00000000`0000000a 00000000`00000000 :
nt!KeBugCheckEx
fffff880`009f7530 fffff800`016c90e0 : 00000000`00000000
fffffa80`009151b0 fffffa80`0155a290 fffff880`01339110 :
nt!KiBugCheckDispatch+0x69
fffff880`009f7670 fffff880`0154dd06 : 00000000`00000001
fffff880`0154dcec fffffa80`009151b0 fffff880`01323934 :
nt!KiPageFault+0x260
fffff880`009f7800 fffff880`0132abcf : fffffa80`009151b0
fffffa80`0065e8f8 fffffa80`0065e738 00000000`00000001 : vioscsi+0x1d06
fffff880`009f7850 fffff880`0154d971 : 00000000`00000001
00000000`00000001 00000000`002d5000 fffffa80`00925000 :
storport!StorPortSynchronizeAccess+0x4f
fffff880`009f7890 fffff880`01323a0c : fffffa80`00000fb1
fffffa80`0155a200 00000000`002d5000 fffffa80`01576010 : vioscsi+0x1971
fffff880`009f78d0 fffff880`01333adf : fffffa80`006eeb30
fffffa80`006e2070 00000000`00000000 00000000`00000801 :
storport!RaCallMiniportResetBus+0x1c
fffff880`009f7900 fffff880`01333b68 : fffffa80`0155a290
fffffa80`006b39f0 00000040`00000000 00000000`00000000 :
storport!RaidAdapterResetBus+0x2f
fffff880`009f7950 fffff880`0136de0b : 00000000`20206f49
00000000`00000001 00000000`00000001 00000000`20206f49 :
storport!RaidAdapterStorageResetBusIoctl+0x28
fffff880`009f7980 fffff880`0136d1d0 : fffff880`01339110
fffffa80`00915060 00000000`00000000 fffffa80`006e2070 : storport! ??
::NNGAKEGL::`string'+0x3c8
fffff880`009f79d0 fffff800`019e33a7 : fffffa80`006e2070
fffff880`009f7ca0 fffffa80`006e2070 fffffa80`0155a290 :
storport!RaDriverDeviceControlIrp+0x90
fffff880`009f7a10 fffff800`019e3c06 : 00000000`00000000
00000000`00000000 00000000`00000000 00000000`00000000 :
nt!IopXxxControlFile+0x607
fffff880`009f7b40 fffff800`016ca153 : 00000000`001aeb01
00000000`00000001 00000000`001aeba0 fffff800`019db152 :
nt!NtDeviceIoControlFile+0x56
fffff880`009f7bb0 00000000`77a2ff2a : 00000000`00000000
00000000`00000000 00000000`00000000 00000000`00000000 :
nt!KiSystemServiceCopyEnd+0x13
00000000`001af1d8 00000000`00000000 : 00000000`00000000
00000000`00000000 00000000`00000000 00000000`00000000 : 0x77a2ff2a


STACK_COMMAND:  kb

FOLLOWUP_IP:
vioscsi+1d06
fffff880`0154dd06 458b4804        mov     r9d,dword ptr [r8+4]

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  vioscsi+1d06

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: vioscsi

IMAGE_NAME:  vioscsi.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  5200724f

FAILURE_BUCKET_ID:  X64_0xD1_vioscsi+1d06

BUCKET_ID:  X64_0xD1_vioscsi+1d06

Followup: MachineOwner
---------

On Tue, Aug 20, 2013 at 7:43 PM, Timon Wang <timonwst@gmail.com> wrote:
> Thanks, the whole iSCSI LUN have been passed to the VM.
>
> But I test it with scsicmd, and found that the driver may be not
> support SPC-3, but if i use this by microsoft iscsi initiator, I can
> pass all the scsi3_test tests.
>
> Tool can be found here:
> http://www.symantec.com/business/support/index?page=content&id=TECH72086
>
> It's this means that the scsi passthrough windows driver does not
> support SPC-3 feature, I have read a post about this, it says if
> support this we should change both the implementation and the
> documents in virtio spec.
>
> I am new to this list, so I don't know what is the situation right now?
>
> Would somebody please give me some advise on it?
>
>
> On Tue, Aug 20, 2013 at 6:49 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> Il 20/08/2013 12:42, Timon Wang ha scritto:
>>> [root@localhost /]# ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>> lrwxrwxrwx. 1 root root 8 8月  20 17:38
>>> /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk -> ../dm-13
>>> [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>> standard INQUIRY:
>>>   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>>>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>>>   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>>>   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>>>   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>>>     length=36 (0x24)   Peripheral device type: disk
>>>  Vendor identification: MacroSAN
>>>  Product identification: LU
>>>  Product revision level: 1.0
>>>  Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
>>>
>>> This lun is from a vg build based on iscsi target.
>>
>> If it is a logical volume, you cannot pass it as a LUN to the guest.
>> Only the whole iSCSI LUN can be passed as a LUN.
>>
>> Paolo
>>
>>> [root@localhost /]# libvirtd --version
>>> libvirtd (libvirt) 1.0.5
>>> [root@localhost /]# qemu-kvm --version
>>> QEMU emulator version 1.4.1, Copyright (c) 2003-2008 Fabrice Bellard
>>> [root@localhost /]# uname -a
>>> Linux localhost.localdomain 3.9.2-301.fc19.x86_64 #1 SMP Mon May 13
>>> 12:36:24 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>>
>>> On Tue, Aug 20, 2013 at 6:16 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>> Il 20/08/2013 11:59, Timon Wang ha scritto:
>>>>> On Tue, Aug 20, 2013 at 4:33 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>>>> Il 20/08/2013 08:00, Timon Wang ha scritto:
>>>>>>>     <disk type='file' device='disk'>
>>>>>>>       <driver name='qemu' type='raw' cache='none'/>
>>>>>>>       <source file='/home/images/win2008_2_sys'/>
>>>>>>>       <target dev='hda' bus='ide'/>
>>>>>>>       <boot order='3'/>
>>>>>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>>>>     </disk>
>>>>>>>     <disk type='file' device='cdrom'>
>>>>>>>       <driver name='qemu' type='raw'/>
>>>>>>>       <source file='/home/isos/windows2008_64r2.iso'/>
>>>>>>>       <target dev='sdc' bus='ide'/>
>>>>>>>       <readonly/>
>>>>>>>       <boot order='1'/>
>>>>>>>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>>>>>>     </disk>
>>>>>>>     <disk type='block' device='disk'>
>>>>>>
>>>>>> I'm not sure this will be enough, but if you want passthrough to the
>>>>>> host device you should use device='lun' here.  However, you still would
>>>>>> not be able to issue SCSI reservations unless you run QEMU with the
>>>>>> CAP_SYS_RAWIO capability (using "<disk ... rawio='yes'>").
>>>>>>
>>>>>
>>>>> After change the libvirt xml like this:
>>>>> <disk type='block' device='lun' rawio='yes'>
>>>>>       <driver name='qemu' type='raw' cache='none'/>
>>>>>       <source dev='/dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk'/>
>>>>>       <target dev='sda' bus='scsi'/>
>>>>>       <shareable/>
>>>>>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>>>>>     </disk>
>>>>> I got these errors:
>>>>> char device redirected to /dev/pts/1 (label charserial0)
>>>>> qemu-system-x86_64: -device
>>>>> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
>>>>> scsi-block: INQUIRY failed
>>>>> qemu-system-x86_64: -device
>>>>> scsi-block,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0:
>>>>> Device 'scsi-block' could not be initialized
>>>>
>>>> Can you do
>>>>
>>>> # ls -l /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>>> # sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>>>
>>>> ?
>>>>
>>>> Paolo
>>>>
>>>
>>>
>>>
>>
>
>
>
> --
> Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
> Blog: http://www.nohouse.net



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Paolo Bonzini]

Il 20/08/2013 13:43, Timon Wang ha scritto:
> Thanks, the whole iSCSI LUN have been passed to the VM.
> 
> But I test it with scsicmd, and found that the driver may be not
> support SPC-3, but if i use this by microsoft iscsi initiator, I can
> pass all the scsi3_test tests.

If you are passing the LUN to the VM with device='lun', the driver and
VMM do not interpret any SCSI command.  You should see exactly the same
data as in the host, which includes support for SPC-3:

>>> [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>> standard INQUIRY:
>>>   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>>>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>>>   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>>>   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>>>   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>>>     length=36 (0x24)   Peripheral device type: disk
>>>  Vendor identification: MacroSAN
>>>  Product identification: LU
>>>  Product revision level: 1.0
>>>  Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300

Can you try using a Linux VM and executing sg_inq in the VM?

Paolo

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

>From the fedora 19 host:
[root@fedora ~]# sg_inq /dev/sdc
standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
  SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
  EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
    length=36 (0x24)   Peripheral device type: disk
 Vendor identification: MacroSAN
 Product identification: LU
 Product revision level: 1.0
 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300

>From the fedora 19 vm:
[root@fedoravm ~]# sg_inq /dev/sdb
standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
  SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
  EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
    length=36 (0x24)   Peripheral device type: disk
 Vendor identification: MacroSAN
 Product identification: LU
 Product revision level: 1.0
 Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300

The result from fedora 19 host and fedora 19 vm are the same. It's
that means I got a wrong windows vm scsi pass-through driver?
Or is there any tool like sg_inq in windows 2008?


On Tue, Aug 20, 2013 at 8:09 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 20/08/2013 13:43, Timon Wang ha scritto:
>> Thanks, the whole iSCSI LUN have been passed to the VM.
>>
>> But I test it with scsicmd, and found that the driver may be not
>> support SPC-3, but if i use this by microsoft iscsi initiator, I can
>> pass all the scsi3_test tests.
>
> If you are passing the LUN to the VM with device='lun', the driver and
> VMM do not interpret any SCSI command.  You should see exactly the same
> data as in the host, which includes support for SPC-3:
>
>>>> [root@localhost /]# sg_inq /dev/VM-IMAGES-BACKUP-DO-NOT-REMOVE/q_disk
>>>> standard INQUIRY:
>>>>   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>>>>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>>>>   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>>>>   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>>>>   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>>>>     length=36 (0x24)   Peripheral device type: disk
>>>>  Vendor identification: MacroSAN
>>>>  Product identification: LU
>>>>  Product revision level: 1.0
>>>>  Unit serial number: 0d9281ae-aea4-6da0-0000-02180142b300
>
> Can you try using a Linux VM and executing sg_inq in the VM?
>
> Paolo
>



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Paolo Bonzini]

Il 21/08/2013 04:11, Timon Wang ha scritto:
> From the fedora 19 host:
> [root@fedora ~]# sg_inq /dev/sdc
> standard INQUIRY:
>   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>     length=36 (0x24)   Peripheral device type: disk
>  Vendor identification: MacroSAN
>  Product identification: LU
>  Product revision level: 1.0
>  Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
> 
> From the fedora 19 vm:
> [root@fedoravm ~]# sg_inq /dev/sdb
> standard INQUIRY:
>   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>     length=36 (0x24)   Peripheral device type: disk
>  Vendor identification: MacroSAN
>  Product identification: LU
>  Product revision level: 1.0
>  Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
> 
> The result from fedora 19 host and fedora 19 vm are the same. It's
> that means I got a wrong windows vm scsi pass-through driver?
> Or is there any tool like sg_inq in windows 2008?

Yeah, there's something weird in the Windows VM.

sg_inq should be available for Windows too, but I don't know where to
get a precompiled binary from.

Paolo

Re: Oracle RAC in libvirt+KVM environment

[Nicholas A. Bellinger]

On Wed, 2013-08-21 at 11:09 +0200, Paolo Bonzini wrote:
> Il 21/08/2013 04:11, Timon Wang ha scritto:
> > From the fedora 19 host:
> > [root@fedora ~]# sg_inq /dev/sdc
> > standard INQUIRY:
> >   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
> >   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
> >   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
> >   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
> >   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
> >     length=36 (0x24)   Peripheral device type: disk
> >  Vendor identification: MacroSAN
> >  Product identification: LU
> >  Product revision level: 1.0
> >  Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
> > 
> > From the fedora 19 vm:
> > [root@fedoravm ~]# sg_inq /dev/sdb
> > standard INQUIRY:
> >   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
> >   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
> >   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
> >   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
> >   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
> >     length=36 (0x24)   Peripheral device type: disk
> >  Vendor identification: MacroSAN
> >  Product identification: LU
> >  Product revision level: 1.0
> >  Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
> > 
> > The result from fedora 19 host and fedora 19 vm are the same. It's
> > that means I got a wrong windows vm scsi pass-through driver?
> > Or is there any tool like sg_inq in windows 2008?
> 
> Yeah, there's something weird in the Windows VM.
> 
> sg_inq should be available for Windows too, but I don't know where to
> get a precompiled binary from.
> 

AFAIK, the latest sg3-utils build for MSFT is here:

http://sg.danny.cz/sg/p/sg3_utils-1.36exe.zip

--nab

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Thanks Nicholas.

I found that scsicmd can't pass all the scsi3_test but the result of
sg_inq is the same as it in the host.

I am absolutely confused about this situation. Am I missed some
information about it?

On Wed, Aug 21, 2013 at 5:40 PM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
> On Wed, 2013-08-21 at 11:09 +0200, Paolo Bonzini wrote:
>> Il 21/08/2013 04:11, Timon Wang ha scritto:
>> > From the fedora 19 host:
>> > [root@fedora ~]# sg_inq /dev/sdc
>> > standard INQUIRY:
>> >   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>> >   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>> >   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>> >   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>> >   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>> >     length=36 (0x24)   Peripheral device type: disk
>> >  Vendor identification: MacroSAN
>> >  Product identification: LU
>> >  Product revision level: 1.0
>> >  Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
>> >
>> > From the fedora 19 vm:
>> > [root@fedoravm ~]# sg_inq /dev/sdb
>> > standard INQUIRY:
>> >   PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
>> >   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=0
>> >   SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  [BQue=0]
>> >   EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>> >   [RelAdr=0]  WBus16=1  Sync=1  Linked=0  [TranDis=0]  CmdQue=1
>> >     length=36 (0x24)   Peripheral device type: disk
>> >  Vendor identification: MacroSAN
>> >  Product identification: LU
>> >  Product revision level: 1.0
>> >  Unit serial number: fd01ece6-8540-f4c7-0000-fe170142b300
>> >
>> > The result from fedora 19 host and fedora 19 vm are the same. It's
>> > that means I got a wrong windows vm scsi pass-through driver?
>> > Or is there any tool like sg_inq in windows 2008?
>>
>> Yeah, there's something weird in the Windows VM.
>>
>> sg_inq should be available for Windows too, but I don't know where to
>> get a precompiled binary from.
>>
>
> AFAIK, the latest sg3-utils build for MSFT is here:
>
> http://sg.danny.cz/sg/p/sg3_utils-1.36exe.zip
>
> --nab
>
>



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Oracle RAC in libvirt+KVM environment

[Paolo Bonzini]

Il 22/08/2013 09:46, Timon Wang ha scritto:
> Thanks Nicholas.
> 
> I found that scsicmd can't pass all the scsi3_test but the result of
> sg_inq is the same as it in the host.
> 
> I am absolutely confused about this situation. Am I missed some
> information about it?

I am also confused.  You need to understand the limitations that the
clustering software is putting.

Can you also try using the megaraid sas controller ("lsisas1078" in
libvirt) instead of virtio-scsi?

Paolo

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

I don't know if lsisas1078 controller in qemu is not work property or
I did not configure it correct.
Sometimes I can see the lun which passed to the vm, sometimes not.

I have tried that controller before, and sometimes the vm will crash
because qemu process quit, but I did not get the detail logs about the
process quit.
I am wondering if lsisas1078 is still not stable or I did not config it right.

On Thu, Aug 22, 2013 at 5:02 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 22/08/2013 09:46, Timon Wang ha scritto:
>> Thanks Nicholas.
>>
>> I found that scsicmd can't pass all the scsi3_test but the result of
>> sg_inq is the same as it in the host.
>>
>> I am absolutely confused about this situation. Am I missed some
>> information about it?
>
> I am also confused.  You need to understand the limitations that the
> clustering software is putting.
>
> Can you also try using the megaraid sas controller ("lsisas1078" in
> libvirt) instead of virtio-scsi?
>
> Paolo



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net

Re: Re: Oracle RAC in libvirt+KVM environment

[Masaki Kimura]

>> I found that scsicmd can't pass all the scsi3_test but the result of
>> sg_inq is the same as it in the host.
>>
>> I am absolutely confused about this situation. Am I missed some
>> information about it?

I guess this is caused by the lack of capability.
Please check if enough capability was added to kvm process by the following 
steps. 

  1. Check the pid of kvm process.
     # ps -C qemu-system-x86_64 -o pid=
      5177
  2. Check the capability for the process.
     # getpcaps 5177
     Capabilities for `5177': = cap_sys_rawio+i

In my fedora19 environment, as seen in above, only cap_sys_rawio+i was added
with rawio='yes'. Even though, cap_sys_rawio+ep is required to pass-through 
SCSI Reservation from the guest. 

Note that I succeeded to pass-through SCSI Reservation with the following steps
in my environment, not a Windows guest though.

  1. Stop the guest.
  2. Add CAP_SYS_RAWIO(effective, permitted) to qemu-kvm.
     # setcap cap_sys_rawio=ep /usr/bin/qemu-system-x86_64
  3. Start the guest.

However, I don't think this is the right way to workaround it, because it 
gives cap_sys_rawio+ep to all the kvm processes executed from this binary.

I believe following patches, which are not merged yet, are trying to solve 
this problem in a different approach.
     - [PATCH v3 part2] Add per-device sysfs knob to enable unrestricted, 
      unprivileged SG_IO
      https://lkml.org/lkml/2013/5/23/294
    - [RFC PATCH 0/4] SG_IO filtering via sysfs and minimal whitelist
      https://lkml.org/lkml/2013/5/27/230


Any comments on this?


Masaki

Re: Oracle RAC in libvirt+KVM environment

[Paolo Bonzini]

Il 27/08/2013 12:29, Masaki Kimura ha scritto:
> I believe following patches, which are not merged yet, are trying to solve 
> this problem in a different approach.
>      - [PATCH v3 part2] Add per-device sysfs knob to enable unrestricted, 
>       unprivileged SG_IO
>       https://lkml.org/lkml/2013/5/23/294
>     - [RFC PATCH 0/4] SG_IO filtering via sysfs and minimal whitelist
>       https://lkml.org/lkml/2013/5/27/230
> 
> 
> Any comments on this?

Yes, that's correct.

Paolo

Re: Oracle RAC in libvirt+KVM environment

[Timon Wang]

Thanks, all the test in SCSICMD.exe can pass now, when I set the ep
caps on qemu process.

We found that sg_persist --read-keys can't get the same result in VM
and host, and now we get the same result in both VM and host mode.

But Windows 2008 Failover Cluster Still can't get the cluster disk
configured, with some error said that the driver can't support the
cluster.

Thanks, I think I have dig into the windows Failover Cluster to find
why it can't support cluster, I will ask somebody in MSDN.



On Tue, Aug 27, 2013 at 7:07 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 27/08/2013 12:29, Masaki Kimura ha scritto:
>> I believe following patches, which are not merged yet, are trying to solve
>> this problem in a different approach.
>>      - [PATCH v3 part2] Add per-device sysfs knob to enable unrestricted,
>>       unprivileged SG_IO
>>       https://lkml.org/lkml/2013/5/23/294
>>     - [RFC PATCH 0/4] SG_IO filtering via sysfs and minimal whitelist
>>       https://lkml.org/lkml/2013/5/27/230
>>
>>
>> Any comments on this?
>
> Yes, that's correct.
>
> Paolo



-- 
Focus on: Server Vitualization, Network security,Scanner,NodeJS,JAVA,WWW
Blog: http://www.nohouse.net