KVM, SCSI & OpenSolaris

KVM, SCSI & OpenSolaris

[james]

The Use Case I am trying to achieve:
1. I have an AMD X4 with 4GB ram that is the central server. Given the  
level of horse power it has the server does multiple roles, email, web  
server, mythbuntu backend and file server.
The base/host OS is Ubuntu 8.04 using standard packages so I am using  
KVM-62. While I am prepared to uninstall this and move to a self  
compiled KVM-70 I want to determine if that move is worth it based on  
my target outcome, or if I need to move to an alternative VM solution.

2. The desire is to get the file server role moved from running LVM to  
ZFS raidz to allow "easy" upgrades of disk size on a disk by disk  
basis (not available as an option under normal raid 5). e.g. pull out  
a single 320GB disk and put in a 500GB or 750GB disk into the raidz  
and it all "just works" still with the extra storage being available.

3. Looking to achieve this move by using a VM running either  
OpenSolaris or Nexenta. The idea being to have the VM as a NAS setup  
to be using the disks directly as block devices. So the setup is to  
have a a boot img (can be IDE) and 4 other direct access block devices  
(need to be SCSI as there are not enough IDE devices available). Not  
these are all 64 bit installs based on the advice that ZFS needs a 64  
bit OS to behave well.

4. Options tried:
a] I have tried using FreeBSD 7 using ZFS under this VM model. However  
when put it under load I get scsi errors an the VM segment faults/core- 
dumps. This is
b] I have been trying to get OpenSolaris and Nexenta (basically the  
same at the core) working but neither recognise the KVM scsi  
controler. It seems to coming through as id PCI1000,12 which is a  
LSI53C895A PCI to Ultra2 SCSI Controller which is supposed to use the  
symhisl driver. Now from good old Google I have found that there are  
supposed to be problems with this driver and it will not be ported to  
64 bit. Indeed looking at the OpenSolaris /etc/driver_aliases this  
driver to PCI mapping has been dropped.

5. So the questions are:
a]  How stable/robust is the scsi implementation under KVM? i.e. are  
there known weaknesses here that moving to higher KVM versions will  
address such that using FreeBSD 7 will be a viable option.

b] It would appear that KVM has a general exposure in the SCSI space  
for OpenSolaris and its variants. Due to dropped driver support the  
current SCSI implementation on KVM will no longer work with  
OpenSolaris (at least in its 64bit variant). Or is this resolved in  
later KVM versions (after KVM-62)?

All help and suggestions gratefully received.

Re: KVM, SCSI & OpenSolaris

[Anthony Liguori]

james wrote:
> The Use Case I am trying to achieve:
> 1. I have an AMD X4 with 4GB ram that is the central server. Given the 
> level of horse power it has the server does multiple roles, email, web 
> server, mythbuntu backend and file server.
> The base/host OS is Ubuntu 8.04 using standard packages so I am using 
> KVM-62. While I am prepared to uninstall this and move to a self 
> compiled KVM-70 I want to determine if that move is worth it based on 
> my target outcome, or if I need to move to an alternative VM solution.
>
> 2. The desire is to get the file server role moved from running LVM to 
> ZFS raidz to allow "easy" upgrades of disk size on a disk by disk 
> basis (not available as an option under normal raid 5). e.g. pull out 
> a single 320GB disk and put in a 500GB or 750GB disk into the raidz 
> and it all "just works" still with the extra storage being available.
>
> 3. Looking to achieve this move by using a VM running either 
> OpenSolaris or Nexenta. The idea being to have the VM as a NAS setup 
> to be using the disks directly as block devices. So the setup is to 
> have a a boot img (can be IDE) and 4 other direct access block devices 
> (need to be SCSI as there are not enough IDE devices available). Not 
> these are all 64 bit installs based on the advice that ZFS needs a 64 
> bit OS to behave well.
>
> 4. Options tried:
> a] I have tried using FreeBSD 7 using ZFS under this VM model. However 
> when put it under load I get scsi errors an the VM segment 
> faults/core-dumps. This is
> b] I have been trying to get OpenSolaris and Nexenta (basically the 
> same at the core) working but neither recognise the KVM scsi 
> controler. It seems to coming through as id PCI1000,12 which is a 
> LSI53C895A PCI to Ultra2 SCSI Controller which is supposed to use the 
> symhisl driver. Now from good old Google I have found that there are 
> supposed to be problems with this driver and it will not be ported to 
> 64 bit. Indeed looking at the OpenSolaris /etc/driver_aliases this 
> driver to PCI mapping has been dropped.
>
> 5. So the questions are:
> a]  How stable/robust is the scsi implementation under KVM? i.e. are 
> there known weaknesses here that moving to higher KVM versions will 
> address such that using FreeBSD 7 will be a viable option.

It hasn't been tested much (if at all) with the FreeBSD 7 drivers.  In 
fact, I don't think there's been much testing at all of FreeBSD as a 
guest in KVM.

> b] It would appear that KVM has a general exposure in the SCSI space 
> for OpenSolaris and its variants. Due to dropped driver support the 
> current SCSI implementation on KVM will no longer work with 
> OpenSolaris (at least in its 64bit variant). Or is this resolved in 
> later KVM versions (after KVM-62)?

I don't quite following what you are saying.  Are you saying that 
OpenSolaris no longer supports the SCSI card we emulate?  That seems 
unfortunate on their part.  I would also be surprised by that since the 
same SCSI implementation is used by Xen and Sun is heavily invested in 
OpenSolaris for Xen at this point.

Regards,

Anthony Liguori

> All help and suggestions gratefully received.
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM, SCSI & OpenSolaris

[jameswalker]

> james wrote:
>> The Use Case I am trying to achieve:
>> 1. I have an AMD X4 with 4GB ram that is the central server. Given the
>> level of horse power it has the server does multiple roles, email, web
>> server, mythbuntu backend and file server.
>> The base/host OS is Ubuntu 8.04 using standard packages so I am using
>> KVM-62. While I am prepared to uninstall this and move to a self
>> compiled KVM-70 I want to determine if that move is worth it based on
>> my target outcome, or if I need to move to an alternative VM solution.
>>
>> 2. The desire is to get the file server role moved from running LVM to
>> ZFS raidz to allow "easy" upgrades of disk size on a disk by disk
>> basis (not available as an option under normal raid 5). e.g. pull out
>> a single 320GB disk and put in a 500GB or 750GB disk into the raidz
>> and it all "just works" still with the extra storage being available.
>>
>> 3. Looking to achieve this move by using a VM running either
>> OpenSolaris or Nexenta. The idea being to have the VM as a NAS setup
>> to be using the disks directly as block devices. So the setup is to
>> have a a boot img (can be IDE) and 4 other direct access block devices
>> (need to be SCSI as there are not enough IDE devices available). Not
>> these are all 64 bit installs based on the advice that ZFS needs a 64
>> bit OS to behave well.
>>
>> 4. Options tried:
>> a] I have tried using FreeBSD 7 using ZFS under this VM model. However
>> when put it under load I get scsi errors an the VM segment
>> faults/core-dumps. This is
>> b] I have been trying to get OpenSolaris and Nexenta (basically the
>> same at the core) working but neither recognise the KVM scsi
>> controler. It seems to coming through as id PCI1000,12 which is a
>> LSI53C895A PCI to Ultra2 SCSI Controller which is supposed to use the
>> symhisl driver. Now from good old Google I have found that there are
>> supposed to be problems with this driver and it will not be ported to
>> 64 bit. Indeed looking at the OpenSolaris /etc/driver_aliases this
>> driver to PCI mapping has been dropped.
>>
>> 5. So the questions are:
>> a]  How stable/robust is the scsi implementation under KVM? i.e. are
>> there known weaknesses here that moving to higher KVM versions will
>> address such that using FreeBSD 7 will be a viable option.
>
> It hasn't been tested much (if at all) with the FreeBSD 7 drivers.  In
> fact, I don't think there's been much testing at all of FreeBSD as a
> guest in KVM.
>

Ok - that is good to know. If I can get my head around it enough I might
be able to report more specifics on where/what the failure is under this
OS.

>> b] It would appear that KVM has a general exposure in the SCSI space
>> for OpenSolaris and its variants. Due to dropped driver support the
>> current SCSI implementation on KVM will no longer work with
>> OpenSolaris (at least in its 64bit variant). Or is this resolved in
>> later KVM versions (after KVM-62)?
>
> I don't quite following what you are saying.  Are you saying that
> OpenSolaris no longer supports the SCSI card we emulate?  That seems
> unfortunate on their part.  I would also be surprised by that since the
> same SCSI implementation is used by Xen and Sun is heavily invested in
> OpenSolaris for Xen at this point.
>

Yes - I am saying that the scsi card KVM emulates does not seem to have
support in current and from the sounds of it future versions of
OpenSolaris 64bit. I can not comment re 32 bit as I am not interested in
or tesing for that.  From what I can see the SCSI emulation for the
LSI53C895A PCI to Ultra2 SCSI Controller which is what OpenSolaris reports
the KVM scsi emulation as being is missing from current 64 bit OpenSolaris
encarnations.

> Regards,
>
> Anthony Liguori
>
>> All help and suggestions gratefully received.
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: KVM, SCSI & OpenSolaris

[Lynn Kerby]

On Jul 10, 2008, at 1:50 PM, Anthony Liguori wrote:

> james wrote:
>>
>> 5. So the questions are:
>> a]  How stable/robust is the scsi implementation under KVM? i.e.  
>> are there known weaknesses here that moving to higher KVM versions  
>> will address such that using FreeBSD 7 will be a viable option.
>
> It hasn't been tested much (if at all) with the FreeBSD 7 drivers.   
> In fact, I don't think there's been much testing at all of FreeBSD  
> as a guest in KVM.
>

I've been running 2-3 production FreeBSD guests under KVM for about a  
year now.  I haven't yet upgraded to FreeBSD 7 though, so they are  
running 6.2 and 6.3 STABLE.  I'm also running FreeNAS which is  
FreeBSD based in a guest.

It all works quite well.

Lynn Kerby

Re: KVM, SCSI & OpenSolaris

[jameswalker]

>
> On Jul 10, 2008, at 1:50 PM, Anthony Liguori wrote:
>
>> james wrote:
>>>
>>> 5. So the questions are:
>>> a]  How stable/robust is the scsi implementation under KVM? i.e.
>>> are there known weaknesses here that moving to higher KVM versions
>>> will address such that using FreeBSD 7 will be a viable option.
>>
>> It hasn't been tested much (if at all) with the FreeBSD 7 drivers.
>> In fact, I don't think there's been much testing at all of FreeBSD
>> as a guest in KVM.
>>
>
> I've been running 2-3 production FreeBSD guests under KVM for about a
> year now.  I haven't yet upgraded to FreeBSD 7 though, so they are
> running 6.2 and 6.3 STABLE.  I'm also running FreeNAS which is
> FreeBSD based in a guest.
>

Now THAT sounds promising. I have tinkered with the FreeNAS stuff. In a VM
how many disk devices (i.e. real HDs not image files) are you allocating
to the VM? Are you getting past the 4 IDE limit by using SCSI?
I am watching the new version of FreeNAS under FreeBSD 7 with the ZFS
addition very closely. It seems to be my ideal solution.

Cheers,
James.

Re: KVM, SCSI & OpenSolaris

[Lynn Kerby]

On Jul 10, 2008, at 3:54 PM, jameswalker@clear.net.nz wrote:

>>
>> On Jul 10, 2008, at 1:50 PM, Anthony Liguori wrote:
>>
>>> james wrote:
>>>>
>>>> 5. So the questions are:
>>>> a]  How stable/robust is the scsi implementation under KVM? i.e.
>>>> are there known weaknesses here that moving to higher KVM versions
>>>> will address such that using FreeBSD 7 will be a viable option.
>>>
>>> It hasn't been tested much (if at all) with the FreeBSD 7 drivers.
>>> In fact, I don't think there's been much testing at all of FreeBSD
>>> as a guest in KVM.
>>>
>>
>> I've been running 2-3 production FreeBSD guests under KVM for about a
>> year now.  I haven't yet upgraded to FreeBSD 7 though, so they are
>> running 6.2 and 6.3 STABLE.  I'm also running FreeNAS which is
>> FreeBSD based in a guest.
>>
>
> Now THAT sounds promising. I have tinkered with the FreeNAS stuff.  
> In a VM
> how many disk devices (i.e. real HDs not image files) are you  
> allocating
> to the VM? Are you getting past the 4 IDE limit by using SCSI?
> I am watching the new version of FreeNAS under FreeBSD 7 with the ZFS
> addition very closely. It seems to be my ideal solution.
>
> Cheers,
> James.

Sorry, at the moment I'm only attaching a couple of large lvm disks  
(on the host) using the IDE emulation.  I just started playing with  
SCSI support recently and was a little surprised to find that I  
couldn't get KVM/qemu to boot a SCSI cdrom (attached to a local ISO  
image on the host).  I was trying to install an ISO image for a  
custom linux based applicance that only had SCSI support in its  
installer and kernel.

Lynn Kerby

Re: KVM, SCSI & OpenSolaris

[Thomas Mueller]

On Thu, 10 Jul 2008 15:50:41 -0500, Anthony Liguori wrote:


> 
> I don't quite following what you are saying.  Are you saying that
> OpenSolaris no longer supports the SCSI card we emulate?  That seems
> unfortunate on their part.  I would also be surprised by that since the
> same SCSI implementation is used by Xen and Sun is heavily invested in
> OpenSolaris for Xen at this point.
> 

i can also confirm that OpenSolaris 2008.05 AMD64 says "Driver missing!" 
for the  Qemu (LSI Logic 53c895a) SCSI card. 

- Thomas

Re: KVM, SCSI & OpenSolaris

[james]

On 12/07/2008, at 12:15 AM, Thomas Mueller wrote:

> On Thu, 10 Jul 2008 15:50:41 -0500, Anthony Liguori wrote:
>
>
>>
>> I don't quite following what you are saying.  Are you saying that
>> OpenSolaris no longer supports the SCSI card we emulate?  That seems
>> unfortunate on their part.  I would also be surprised by that since  
>> the
>> same SCSI implementation is used by Xen and Sun is heavily invested  
>> in
>> OpenSolaris for Xen at this point.
>>
>
> i can also confirm that OpenSolaris 2008.05 AMD64 says "Driver  
> missing!"
> for the  Qemu (LSI Logic 53c895a) SCSI card.
>
> - Thomas

I did some more digging.  The symhisl driver definitely seems to have  
been dropped from OpenSolaris 64bit versions.  See this post: http://osdir.com/ml/os.solaris.solarisx86/2004-12/msg00179.html
While this isn't "official" looking it fits with the evidence i.e. the  
symhisl driver seems to be no more in the 64bit world.

Anyway the point being this effects the following scsi chipsets:
LSI Logic SYM53C895A, SYM53C1010-33, and SYM53C1010-66 SCSI I/O  
processors
where QEMU/KVM emulates the SYM53C895A (as I understand it)

The glm driver is still distributed. This driver supports the  
following scsi chipsets:
LSI 53c810, LSI 53c875, LSI 53c876, LSI 53c896 and LSI 53c1010 SCSI I/ 
O processors
VMWare supports the 53c1010 so it is fine with scsi images/drives  
under OpenSolaris

Conclusion:
Since this was all driven by my interest in ZFS my wrap up comes from  
that angle: As it stands if you want to experiment with ZFS using  
OpenSolaris under KVM you need to use IDE which means you can only  
play with up to 4 disks. Enough for PoC work.

If you want to use KVM to run an OpenSolaris VM that directly manages  
4+ physical disks plus a boot img then you are probably stumped as  
KVM's scsi isn't supported under OpenSolaris or its derivatives like  
Nexus. Plus 4 IDE devices will not give you the 5+ devices you need.

I have tried the FreeBSD7 path and while I can get scsi working, the  
ZFS implementation in that OS is so new it seems to be pretty fragile.  
I can consistently get it to die (i.e. KVM seg faults) under either  
scsi or ide disks, and I have tuned ZFS a number of ways per  
instructions.

On the plus side I have been running 6 Ubuntu JEOS VMs under KVM for a  
wee while now and it is working VERY VERY well. I continue to resist  
moving to VMWare. :-)

As always if some kind soul can show I have missed a trick please let  
me know. I am still rather keen on getting that ZFS NAS working. I  
just can't justify buying new/more h/w to do it.

Cheers,
James.