Xen PVSCSI: status, issues and some tests

Xen PVSCSI: status, issues and some tests

[Thomas Weyergraf]

Hi, I am working as a system administrator at an internet platform 
service provider, and I am currently seeking to re-new our Xen 
virtualization infrastructure for which I am mostly responsible for.
Currently, we run Xen 3.4.2/3.4.3 on RHEL/CentOS 5.x (5.7) as Dom0 with 
CentOS 5.x pv-guests.

Based on my experiments, I am currently looking into Xen 4.1.2 on 
RHEL/CentOS 6.x (6.2), with a vanilla 3.2 (3.2.0/3.2.6) kernel as Dom0 
and stock RHEL/CentOS 6.x guests as HVM/pv-ops DomUs.

Our DomU's have their disk-devices on iscsi-storage backends (EMC, 
netapp, Linux IET), we attach the disks to Dom0 via open-iscsi initiator 
and pass them as 'phy'-type blockdevices, using blkfront/back.

As this is the time for a major overhaul anyway, I looked at 'pvscsi', 
which is very attractive to me for a bunch of (administrative) reasons. 
I have created a working pvscsi-setup with the following steps:
1. Use CentOS 6.2 as base system
2. Add Xen hypervisor from Michael Young's repository [1]. This is stock 
4.1.2 with a patch from 4.1-testing pulled-in and various patches fixing 
compile-time issues, toolchain problems and system service management on 
RHEL-alike systems. (Nice work, btw!)
3. Use a vanilla 3.2.0 kernel from kernel.org and a .config based on a 
Fedora 16 stock-kernel config, tweaked to use pvscsi. As you know, 
Fedora 16 supports Dom0 operation and the only changes to the config are 
non-Xen related or add the Xen pvscsi config. I refrained from attaching 
the config to avoid mailinglist-clutter, but can do so on demand.
4. I pulled and applied the pvscsi-patch found in this post [2] from 
Konrad Rzeszutek Wilk.

Putting it all together yielded a "working setup" in which I am able to 
pass an Dom0-attached iscsi target to a DomU as a vscsi-device. In DomU 
context, I could partition the device, create a filesystem on it and 
copy/read/verify some 10gig of data to it. Very basic testing until now, 
essentially. However, the whole issue raised some questions:

- I pvscsi actively maintained? Is someone working on this or even 
prepare it for upstream inclusion in the pv-ops framework of the 3.x series?
- Has anybody started on adding the vscsi-semantics to the xl toolstack?
- Using 'xm', I was only able to add a device via its SCSI tupel (HCTL, 
e.g: vscsi = [ '7:0:0:0, 0:0:0:1' ]) but not by any other device-node, 
such as '/dev/sde' or the '/dev/disk/by-path/<iscsi-iqn>' link. For 
these invocations, an "Error: Cannot find device <device>" message is 
given. I have not yet looked at the issue in detail (in 
/usr/lib64/python2.6/site-packages/xen/util/vscsi_util.py). Is this a 
known issue?

Is there any developer(s) working on getting pvscsi ready for kernel.org 
3.x upstream inclusion? Should i better retool my efforts using pv-ops 
xen/stable 2.6.32 series?

If applicable, I'd be more than happy to provide assistance to move 
forward pvscsi. Maybe, as a start, provide a short, practical howto 
about what I did above to get pvscsi working and reference it by the Xen 
PVSCSI wiki page [3]?

Regards,
Thomas Weyergraf

[1] http://xenbits.xen.org/people/mayoung/EL6.xen/
[2] http://lists.xen.org/archives/html/xen-devel/2011-11/msg02268.html
[3] http://wiki.xen.org/xenwiki/XenPVSCSI

Re: Xen PVSCSI: status, issues and some tests

[Konrad Rzeszutek Wilk]

On Sun, Feb 19, 2012 at 03:22:08AM +0100, Thomas Weyergraf wrote:
> Hi, I am working as a system administrator at an internet platform
> service provider, and I am currently seeking to re-new our Xen
> virtualization infrastructure for which I am mostly responsible for.
> Currently, we run Xen 3.4.2/3.4.3 on RHEL/CentOS 5.x (5.7) as Dom0
> with CentOS 5.x pv-guests.
> 
> Based on my experiments, I am currently looking into Xen 4.1.2 on
> RHEL/CentOS 6.x (6.2), with a vanilla 3.2 (3.2.0/3.2.6) kernel as
> Dom0 and stock RHEL/CentOS 6.x guests as HVM/pv-ops DomUs.

Sweet.
> 
> Our DomU's have their disk-devices on iscsi-storage backends (EMC,
> netapp, Linux IET), we attach the disks to Dom0 via open-iscsi
> initiator and pass them as 'phy'-type blockdevices, using
> blkfront/back.

OK.
> 
> As this is the time for a major overhaul anyway, I looked at
> 'pvscsi', which is very attractive to me for a bunch of
> (administrative) reasons. I have created a working pvscsi-setup with
> the following steps:
> 1. Use CentOS 6.2 as base system
> 2. Add Xen hypervisor from Michael Young's repository [1]. This is
> stock 4.1.2 with a patch from 4.1-testing pulled-in and various
> patches fixing compile-time issues, toolchain problems and system
> service management on RHEL-alike systems. (Nice work, btw!)
> 3. Use a vanilla 3.2.0 kernel from kernel.org and a .config based on
> a Fedora 16 stock-kernel config, tweaked to use pvscsi. As you know,
> Fedora 16 supports Dom0 operation and the only changes to the config
> are non-Xen related or add the Xen pvscsi config. I refrained from
> attaching the config to avoid mailinglist-clutter, but can do so on
> demand.

Awesome.
> 4. I pulled and applied the pvscsi-patch found in this post [2] from
> Konrad Rzeszutek Wilk.
> 
> Putting it all together yielded a "working setup" in which I am able
> to pass an Dom0-attached iscsi target to a DomU as a vscsi-device.
> In DomU context, I could partition the device, create a filesystem
> on it and copy/read/verify some 10gig of data to it. Very basic
> testing until now, essentially. However, the whole issue raised some
> questions:
> 
> - I pvscsi actively maintained? Is someone working on this or even
> prepare it for upstream inclusion in the pv-ops framework of the 3.x
> series?

So nobody has stepped up to do the upstream task and it is on my todo list
but mostly at the end.

> - Has anybody started on adding the vscsi-semantics to the xl toolstack?

Not that I know of.

> - Using 'xm', I was only able to add a device via its SCSI tupel
> (HCTL, e.g: vscsi = [ '7:0:0:0, 0:0:0:1' ]) but not by any other
> device-node, such as '/dev/sde' or the
> '/dev/disk/by-path/<iscsi-iqn>' link. For these invocations, an
> "Error: Cannot find device <device>" message is given. I have not
> yet looked at the issue in detail (in
> /usr/lib64/python2.6/site-packages/xen/util/vscsi_util.py). Is this
> a known issue?

First time I hear of it. But then I didn't even think to use /dev/sde
to pass in a device since SCSI devices aren't neccessarily block ones.
They can be scanners, medical devices, tape drivers, etc - which
might not have a /dev/sdX (thought they will have an /dev/sgX).

> 
> Is there any developer(s) working on getting pvscsi ready for
> kernel.org 3.x upstream inclusion? Should i better retool my efforts
> using pv-ops xen/stable 2.6.32 series?

For 3.x upstream inclusion - not that I know off.
Why would you want to retool your effort in using the 2.6.32 series
as opposed to 3.x? What other items is the 3.x missing compared to 2.6.32?

Now the good news is that I am happy to keep this going (pv-scsi)
and fixing it, and having incremental updates to make it better.

> 
> If applicable, I'd be more than happy to provide assistance to move
> forward pvscsi. Maybe, as a start, provide a short, practical howto
> about what I did above to get pvscsi working and reference it by the
> Xen PVSCSI wiki page [3]?

Sure. Also, did you use the git tree to pull the patches or just
the big patch file ? The git tree has fixes to make it work under 3.2 as well.

> 
> Regards,
> Thomas Weyergraf
> 
> [1] http://xenbits.xen.org/people/mayoung/EL6.xen/
> [2] http://lists.xen.org/archives/html/xen-devel/2011-11/msg02268.html
> [3] http://wiki.xen.org/xenwiki/XenPVSCSI
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Xen PVSCSI: status, issues and some tests

[jacek burghardt]

[-- Attachment #1.1: Type: text/plain, Size: 5190 bytes --]

I had just compiled pvscsi into 3.3.0-rc1 based on Konrad Rzeszutek Wilk
git. I had extracted patch from pvscssi-01 and applied it back to
3.3.0-rc1. I am able to load both xen-scsiback and xen-scsifront modules
but I don't see anything happening.
does xl supports pvscsi or it needs to be patched. I am running 4.2
unstable.


On Tue, Feb 21, 2012 at 7:55 AM, Konrad Rzeszutek Wilk <
konrad.wilk@oracle.com> wrote:

> On Sun, Feb 19, 2012 at 03:22:08AM +0100, Thomas Weyergraf wrote:
> > Hi, I am working as a system administrator at an internet platform
> > service provider, and I am currently seeking to re-new our Xen
> > virtualization infrastructure for which I am mostly responsible for.
> > Currently, we run Xen 3.4.2/3.4.3 on RHEL/CentOS 5.x (5.7) as Dom0
> > with CentOS 5.x pv-guests.
> >
> > Based on my experiments, I am currently looking into Xen 4.1.2 on
> > RHEL/CentOS 6.x (6.2), with a vanilla 3.2 (3.2.0/3.2.6) kernel as
> > Dom0 and stock RHEL/CentOS 6.x guests as HVM/pv-ops DomUs.
>
> Sweet.
> >
> > Our DomU's have their disk-devices on iscsi-storage backends (EMC,
> > netapp, Linux IET), we attach the disks to Dom0 via open-iscsi
> > initiator and pass them as 'phy'-type blockdevices, using
> > blkfront/back.
>
> OK.
> >
> > As this is the time for a major overhaul anyway, I looked at
> > 'pvscsi', which is very attractive to me for a bunch of
> > (administrative) reasons. I have created a working pvscsi-setup with
> > the following steps:
> > 1. Use CentOS 6.2 as base system
> > 2. Add Xen hypervisor from Michael Young's repository [1]. This is
> > stock 4.1.2 with a patch from 4.1-testing pulled-in and various
> > patches fixing compile-time issues, toolchain problems and system
> > service management on RHEL-alike systems. (Nice work, btw!)
> > 3. Use a vanilla 3.2.0 kernel from kernel.org and a .config based on
> > a Fedora 16 stock-kernel config, tweaked to use pvscsi. As you know,
> > Fedora 16 supports Dom0 operation and the only changes to the config
> > are non-Xen related or add the Xen pvscsi config. I refrained from
> > attaching the config to avoid mailinglist-clutter, but can do so on
> > demand.
>
> Awesome.
> > 4. I pulled and applied the pvscsi-patch found in this post [2] from
> > Konrad Rzeszutek Wilk.
> >
> > Putting it all together yielded a "working setup" in which I am able
> > to pass an Dom0-attached iscsi target to a DomU as a vscsi-device.
> > In DomU context, I could partition the device, create a filesystem
> > on it and copy/read/verify some 10gig of data to it. Very basic
> > testing until now, essentially. However, the whole issue raised some
> > questions:
> >
> > - I pvscsi actively maintained? Is someone working on this or even
> > prepare it for upstream inclusion in the pv-ops framework of the 3.x
> > series?
>
> So nobody has stepped up to do the upstream task and it is on my todo list
> but mostly at the end.
>
> > - Has anybody started on adding the vscsi-semantics to the xl toolstack?
>
> Not that I know of.
>
> > - Using 'xm', I was only able to add a device via its SCSI tupel
> > (HCTL, e.g: vscsi = [ '7:0:0:0, 0:0:0:1' ]) but not by any other
> > device-node, such as '/dev/sde' or the
> > '/dev/disk/by-path/<iscsi-iqn>' link. For these invocations, an
> > "Error: Cannot find device <device>" message is given. I have not
> > yet looked at the issue in detail (in
> > /usr/lib64/python2.6/site-packages/xen/util/vscsi_util.py). Is this
> > a known issue?
>
> First time I hear of it. But then I didn't even think to use /dev/sde
> to pass in a device since SCSI devices aren't neccessarily block ones.
> They can be scanners, medical devices, tape drivers, etc - which
> might not have a /dev/sdX (thought they will have an /dev/sgX).
>
> >
> > Is there any developer(s) working on getting pvscsi ready for
> > kernel.org 3.x upstream inclusion? Should i better retool my efforts
> > using pv-ops xen/stable 2.6.32 series?
>
> For 3.x upstream inclusion - not that I know off.
> Why would you want to retool your effort in using the 2.6.32 series
> as opposed to 3.x? What other items is the 3.x missing compared to 2.6.32?
>
> Now the good news is that I am happy to keep this going (pv-scsi)
> and fixing it, and having incremental updates to make it better.
>
> >
> > If applicable, I'd be more than happy to provide assistance to move
> > forward pvscsi. Maybe, as a start, provide a short, practical howto
> > about what I did above to get pvscsi working and reference it by the
> > Xen PVSCSI wiki page [3]?
>
> Sure. Also, did you use the git tree to pull the patches or just
> the big patch file ? The git tree has fixes to make it work under 3.2 as
> well.
>
> >
> > Regards,
> > Thomas Weyergraf
> >
> > [1] http://xenbits.xen.org/people/mayoung/EL6.xen/
> > [2] http://lists.xen.org/archives/html/xen-devel/2011-11/msg02268.html
> > [3] http://wiki.xen.org/xenwiki/XenPVSCSI
> >
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 6885 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen PVSCSI: status, issues and some tests

[Konrad Rzeszutek Wilk]

On Thu, Feb 23, 2012 at 05:19:00PM -0700, jacek burghardt wrote:
> I had just compiled pvscsi into 3.3.0-rc1 based on Konrad Rzeszutek Wilk
> git. I had extracted patch from pvscssi-01 and applied it back to
> 3.3.0-rc1. I am able to load both xen-scsiback and xen-scsifront modules
> but I don't see anything happening.
> does xl supports pvscsi or it needs to be patched. I am running 4.2
> unstable.


Did you try to use xm?

Re: Xen PVSCSI: status, issues and some tests

[Pasi Kärkkäinen]

On Thu, Feb 23, 2012 at 05:19:00PM -0700, jacek burghardt wrote:
>    I had just compiled pvscsi into 3.3.0-rc1 based on Konrad Rzeszutek Wilk
>    git. I had extracted patch from pvscssi-01 and applied it back to
>    3.3.0-rc1. I am able to load both xen-scsiback and xen-scsifront modules
>    but I don't see anything happening.
>    does xl supports pvscsi or it needs to be patched. I am running 4.2
>    unstable.

I don't think there is currently any PVSCSI support in xl/libxl.

-- Pasi

> 
>    On Tue, Feb 21, 2012 at 7:55 AM, Konrad Rzeszutek Wilk
>    <[1]konrad.wilk@oracle.com> wrote:
> 
>      On Sun, Feb 19, 2012 at 03:22:08AM +0100, Thomas Weyergraf wrote:
>      > Hi, I am working as a system administrator at an internet platform
>      > service provider, and I am currently seeking to re-new our Xen
>      > virtualization infrastructure for which I am mostly responsible for.
>      > Currently, we run Xen 3.4.2/3.4.3 on RHEL/CentOS 5.x (5.7) as Dom0
>      > with CentOS 5.x pv-guests.
>      >
>      > Based on my experiments, I am currently looking into Xen 4.1.2 on
>      > RHEL/CentOS 6.x (6.2), with a vanilla 3.2 (3.2.0/3.2.6) kernel as
>      > Dom0 and stock RHEL/CentOS 6.x guests as HVM/pv-ops DomUs.
> 
>      Sweet.
>      >
>      > Our DomU's have their disk-devices on iscsi-storage backends (EMC,
>      > netapp, Linux IET), we attach the disks to Dom0 via open-iscsi
>      > initiator and pass them as 'phy'-type blockdevices, using
>      > blkfront/back.
> 
>      OK.
>      >
>      > As this is the time for a major overhaul anyway, I looked at
>      > 'pvscsi', which is very attractive to me for a bunch of
>      > (administrative) reasons. I have created a working pvscsi-setup with
>      > the following steps:
>      > 1. Use CentOS 6.2 as base system
>      > 2. Add Xen hypervisor from Michael Young's repository [1]. This is
>      > stock 4.1.2 with a patch from 4.1-testing pulled-in and various
>      > patches fixing compile-time issues, toolchain problems and system
>      > service management on RHEL-alike systems. (Nice work, btw!)
>      > 3. Use a vanilla 3.2.0 kernel from [2]kernel.org and a .config based
>      on
>      > a Fedora 16 stock-kernel config, tweaked to use pvscsi. As you know,
>      > Fedora 16 supports Dom0 operation and the only changes to the config
>      > are non-Xen related or add the Xen pvscsi config. I refrained from
>      > attaching the config to avoid mailinglist-clutter, but can do so on
>      > demand.
> 
>      Awesome.
>      > 4. I pulled and applied the pvscsi-patch found in this post [2] from
>      > Konrad Rzeszutek Wilk.
>      >
>      > Putting it all together yielded a "working setup" in which I am able
>      > to pass an Dom0-attached iscsi target to a DomU as a vscsi-device.
>      > In DomU context, I could partition the device, create a filesystem
>      > on it and copy/read/verify some 10gig of data to it. Very basic
>      > testing until now, essentially. However, the whole issue raised some
>      > questions:
>      >
>      > - I pvscsi actively maintained? Is someone working on this or even
>      > prepare it for upstream inclusion in the pv-ops framework of the 3.x
>      > series?
> 
>      So nobody has stepped up to do the upstream task and it is on my todo
>      list
>      but mostly at the end.
>      > - Has anybody started on adding the vscsi-semantics to the xl
>      toolstack?
> 
>      Not that I know of.
>      > - Using 'xm', I was only able to add a device via its SCSI tupel
>      > (HCTL, e.g: vscsi = [ '7:0:0:0, 0:0:0:1' ]) but not by any other
>      > device-node, such as '/dev/sde' or the
>      > '/dev/disk/by-path/<iscsi-iqn>' link. For these invocations, an
>      > "Error: Cannot find device <device>" message is given. I have not
>      > yet looked at the issue in detail (in
>      > /usr/lib64/python2.6/site-packages/xen/util/vscsi_util.py). Is this
>      > a known issue?
> 
>      First time I hear of it. But then I didn't even think to use /dev/sde
>      to pass in a device since SCSI devices aren't neccessarily block ones.
>      They can be scanners, medical devices, tape drivers, etc - which
>      might not have a /dev/sdX (thought they will have an /dev/sgX).
>      >
>      > Is there any developer(s) working on getting pvscsi ready for
>      > [3]kernel.org 3.x upstream inclusion? Should i better retool my
>      efforts
>      > using pv-ops xen/stable 2.6.32 series?
> 
>      For 3.x upstream inclusion - not that I know off.
>      Why would you want to retool your effort in using the 2.6.32 series
>      as opposed to 3.x? What other items is the 3.x missing compared to
>      2.6.32?
> 
>      Now the good news is that I am happy to keep this going (pv-scsi)
>      and fixing it, and having incremental updates to make it better.
>      >
>      > If applicable, I'd be more than happy to provide assistance to move
>      > forward pvscsi. Maybe, as a start, provide a short, practical howto
>      > about what I did above to get pvscsi working and reference it by the
>      > Xen PVSCSI wiki page [3]?
> 
>      Sure. Also, did you use the git tree to pull the patches or just
>      the big patch file ? The git tree has fixes to make it work under 3.2 as
>      well.
>      >
>      > Regards,
>      > Thomas Weyergraf
>      >
>      > [1] [4]http://xenbits.xen.org/people/mayoung/EL6.xen/
>      > [2]
>      [5]http://lists.xen.org/archives/html/xen-devel/2011-11/msg02268.html
>      > [3] [6]http://wiki.xen.org/xenwiki/XenPVSCSI
>      >
>      >
>      >
>      > _______________________________________________
>      > Xen-devel mailing list
>      > [7]Xen-devel@lists.xen.org
>      > [8]http://lists.xen.org/xen-devel
> 
>      _______________________________________________
>      Xen-devel mailing list
>      [9]Xen-devel@lists.xen.org
>      [10]http://lists.xen.org/xen-devel
> 
> References
> 
>    Visible links
>    1. mailto:konrad.wilk@oracle.com
>    2. http://kernel.org/
>    3. http://kernel.org/
>    4. http://xenbits.xen.org/people/mayoung/EL6.xen/
>    5. http://lists.xen.org/archives/html/xen-devel/2011-11/msg02268.html
>    6. http://wiki.xen.org/xenwiki/XenPVSCSI
>    7. mailto:Xen-devel@lists.xen.org
>    8. http://lists.xen.org/xen-devel
>    9. mailto:Xen-devel@lists.xen.org
>   10. http://lists.xen.org/xen-devel

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel