Writing a tool for Shared Persistent Windows Boot Image

Writing a tool for Shared Persistent Windows Boot Image

[Jim Burnes]

Before, in my "Hard Problem" email I was trying to communicate a  
design issue were trying to solve with Xen.

This is what we need to do:

1) Deploy 24 Windows XP VMs in parallel.
2) Boot them from a shared Windows XP C: drive.
3) Since this is a read-only shared image we obviously can't have  
multiple VM's writing to it.
4) All writes to the boot image for logging, registry and other  
purposes should be diverted to an auxiliary shadow drive specific to  
each VM.
5) After we shut down the VM we need to mount and examine the  
contents of the shadow drive
6) When we are done examining the contents of the shadow drive, we  
need to fast format it for the next VM to use.

Is this supported natively in Xen?  What does everyone else who needs  
to run a lot of Windows VMs do?  There must be a way to support  
shared images.

The reason I posted this to xen-devel is that I could probably  
implement a UnionFS for Windows by writing a kernel hook and  
intercepting all reads and writes to the C: drive, but I don't have  
enough time to do that right now.  Because of schedule constraints,  
if we don't find a way to do this in Xen/XenSource we'll have to drop  
Xen and move on to VMWare ESX at considerable cost to our project.

Are there any senior Xen software engineers out there who've done  
this or who might know how?

Thanks,

Jim Burnes

Re: Writing a tool for Shared Persistent Windows Boot Image

[Anthony Liguori]

Jim Burnes wrote:
> Before, in my "Hard Problem" email I was trying to communicate a design 
> issue were trying to solve with Xen.
> 
> This is what we need to do:
> 
> 1) Deploy 24 Windows XP VMs in parallel.

I have strong doubts that this would be kosher from a licensing 
perspective, however...

> 2) Boot them from a shared Windows XP C: drive.
> 3) Since this is a read-only shared image we obviously can't have 
> multiple VM's writing to it.
> 4) All writes to the boot image for logging, registry and other purposes 
> should be diverted to an auxiliary shadow drive specific to each VM.

If you start with a single image, and then create "COW" files using the 
qcow format, then you can have a shared base image.

> 5) After we shut down the VM we need to mount and examine the contents 
> of the shadow drive

Mounting is tricky.  If you look on qemu-devel, you'll find a couple 
references to tool that allow you to mount a qcow file (usually with nbd).

> 6) When we are done examining the contents of the shadow drive, we need 
> to fast format it for the next VM to use.

You can just delete the cow file and create a new one.

Regards,

Anthony Liguori

> Is this supported natively in Xen?  What does everyone else who needs to 
> run a lot of Windows VMs do?  There must be a way to support shared images.
> 
> The reason I posted this to xen-devel is that I could probably implement 
> a UnionFS for Windows by writing a kernel hook and intercepting all 
> reads and writes to the C: drive, but I don't have enough time to do 
> that right now.  Because of schedule constraints, if we don't find a way 
> to do this in Xen/XenSource we'll have to drop Xen and move on to VMWare 
> ESX at considerable cost to our project.
> 
> Are there any senior Xen software engineers out there who've done this 
> or who might know how?
> 
> Thanks,
> 
> Jim Burnes

Re: Writing a tool for Shared Persistent Windows Boot Image

[Anthony Liguori]

Jim Burnes wrote:
>> If you start with a single image, and then create "COW" files using 
>> the qcow format, then you can have a shared base image.
>>
>
> Let me make sure I understand your answer.   COW files are sparse 
> storage for the QEMU environment (which tools you use for Xen).
>
> We want a static filesystem image that represents a snapshot of a 
> Windows XP system right after boot.  When we activate that image we 
> want to perform a few housekeeping issues (like set the MAC and 
> re-DHCP etc), but we also want to make sure that any writes to the 
> image are redirected to an overlay writable file system.  In other 
> words we want a single shared image of the OS itself with all writes 
> going to the COW / shadow image of that specific VM.

COW == Copy On Write.  It's a separate file that only stores the data 
that has been written since the cow was created.

> Is the COW file you speak of overlayed on top of a single static 
> Windows image or does the COW file contain the entire Windows XP boot 
> image plus any writes.

It just contains the writes after the COW was created.

> I know this depends on your previous answer, but if we delete the COW 
> doesn't that delete the XP boot image also?

If you delete the COW, the underlying base image is unaffected.

> If the COW file is used as an extent-space to the static Windows XP 
> image, then that should work.
>
> Otherwise I think it would require a recreation of the full Windows XP 
> image.  Ideally that image can be kept static as the continual 
> deletion and re-creation of relatively large COW files would be 
> time-consuming and would tend to fragment hard disk space.

You really shouldn't worry about disk fragmentation but that's a whole 
other thread :-)

Regards,

Anthony Liguori

> Other than that, I'm very grateful for your kind assistance.
>
> Thanks again,
>
> Jim Burnes
> Boulder, CO
>
>

Re: Writing a tool for Shared Persistent Windows Boot Image

[Alan Cox]

> Is this supported natively in Xen?  What does everyone else who needs  
> to run a lot of Windows VMs do?  There must be a way to support  
> shared images.

The Linux dom0 kernel has some snapshotting support which seems to be
what you are asking for ?

Re: Writing a tool for Shared Persistent Windows Boot Image

[Jim Burnes]

[-- Attachment #1.1: Type: text/plain, Size: 1368 bytes --]

Alan, thanks for your comment.   Sorry the reply too so long.   I believe
that primitive support for dom0 is available.

So far I've been able to implement persistent copy-on-write snapshots with
LVM snapshotting, however that method seems to consume way to too much main
memory.  We're limited to about 7 VMs of 256M of memory each.  I'm trying to
fit about 12 VMs on our 4 GB test box.

To achieve that we'll probably have to move over to qcow copy-on-write
files.  They seem to be the ideal way to implement our solution, however
I've had the devil's own time getting them to work.  I created the required
qcow image files and updated my configuration, but 'xm create <vmname>' just
runs and the VM immediately quits with no debugging information.

I'm wondering whether (1) I've screwed up the qcow images or (2) I'm using a
set of RPMs that don't support tap:qcow (using 3.1 RPMS from XenSource) or
(3) The default RHEL 5 Xen kernel doesn't include TAP support.

Jim Burnes
Boulder, CO


I'd like to use qcow files, since they seem to do

On 6/21/07, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>
> > Is this supported natively in Xen?  What does everyone else who needs
> > to run a lot of Windows VMs do?  There must be a way to support
> > shared images.
>
> The Linux dom0 kernel has some snapshotting support which seems to be
> what you are asking for ?
>
>

[-- Attachment #1.2: Type: text/html, Size: 1817 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Writing a tool for Shared Persistent Windows Boot Image

[Jim Burnes]

[-- Attachment #1.1: Type: text/plain, Size: 4651 bytes --]

Anthony (and anyone else listening),

I've installed Centos 5, Xen 3.1 (from the Xensource RPMs) and installed a
base Windows image to an LVM to test.

The installation wen't fairly well and I resolved various issues like
getting the VMs to render to VNC displays properly.

I even used LVM snapshotting to do some quick copy-on-write tests.  In that
configuration I've had up to 7 Windows VMs running at the same time, on
separate IPs, writing to LVM copy-on-write snapshots.  The only problem is
that LVM cow snapshots consume too much main memory.

So I proceeded to my final test which was to replace the LVM storage with
QCOW storage and see how many Windows VMs I could run concurrently.  (I'm
trying for 12).

The only cow tool I had was qcow-create, so I downloaded the qemu RPMs and
installed them.

I used img2qcow to convert my installed Windows LVM image to the qcow base
image.
I used qcow-img to create a copy-on-write snapshot of the base image.
I tried to boot with this as the new storage type by specifiying:

disk = [ 'tap:qcow:/var/lib/xen/image/winflp.img,hda1,w',
'file:/var/lib/xen/install/winflp.iso,hdc:cdrom,r']

(and other variations, by using 'hda' instead of 'hda1', using 'tap:aio'
instead of 'tap:qcow')

But everytime I try to start the vm by 'xm create' it seems to create the VM
and the VM immediately exits, leaving hardly any trace of even booting.  I
tried booting directly to hard drive C: and I tried booting to the CDROM.
Booting to the CDROM worked fine but reported no attached hard drive.

The only thing I see is a trace in the xend.log file which indicates that
xend (or the VM) is attempting to generate some sort of hotplug event for
tap.  Then the whole VM shuts down.

Is there something I need to do before tap:qcow works?

I suspect, but am having a hard time proving that either:

(1) My version of Xen 3.1 doesn't support tap:qcow or
(2) My XenSource kernel doesn't contain tap support or
(3 I'm loosing my mind

Any ideas that anyone has would be greatly appreciated.

Regards,

Jim Burnes



On 6/21/07, Anthony Liguori <anthony@codemonkey.ws> wrote:
>
> Jim Burnes wrote:
> >
> > On Jun 21, 2007, at 2:39 PM, Anthony Liguori wrote:
> >
> >> Jim Burnes wrote:
> >>>> If you start with a single image, and then create "COW" files using
> >>>> the qcow format, then you can have a shared base image.
> >>>>
> >>>
> >>> Let me make sure I understand your answer.   COW files are sparse
> >>> storage for the QEMU environment (which tools you use for Xen).
> >>>
> >>> We want a static filesystem image that represents a snapshot of a
> >>> Windows XP system right after boot.  When we activate that image we
> >>> want to perform a few housekeeping issues (like set the MAC and
> >>> re-DHCP etc), but we also want to make sure that any writes to the
> >>> image are redirected to an overlay writable file system.  In other
> >>> words we want a single shared image of the OS itself with all writes
> >>> going to the COW / shadow image of that specific VM.
> >>
> >> COW == Copy On Write.  It's a separate file that only stores the data
> >> that has been written since the cow was created.
> >>
> >
> > Excellent.  This is exactly what we need.  I know this is tedious, but
> > just to make absolutely certain (and I'll check the documentation),
> > you're saying that:
> >
> > 1. I can create a COW file and associate it with my C: volume.
>
> Not quite.  You can install Windows to a disk image (drive letters like
> C: are individual partitions within a drive).  You then create a cow
> file that uses that original disk image as a base.
> > 2. All subsequent writes to C: after the COW creation go to the COW file
>
> All subsequent writes to the drive will go to the COW file after COW
> creation.
>
> > 3. All updates stored in the COW will be read back exactly as if they
> > were on the original C: volume
>
> Yes.
>
> > 4. After VM shutdown, we can attach to the COW image using NBD
> > (network block device) for R/O access
>
> With the appropriate tools, yes.
>
> > 5. We can reset the shared volume back to zero by simply truncating
> > the associated COW file.
>
> Delete and create or:
>
> rm winimg001-cow.img
> qemu-img create -f qcow -b win-base.img winimg001-cow.img
>
> That all happens very quickly.
>
> Regards,
>
> Anthony Liguori
>
> > If this works, it will be great.
> >
> >> You really shouldn't worry about disk fragmentation but that's a
> >> whole other thread :-)
> >>
> >
> > Yeah, I'm always trying to pre-optimize everything 8-)
> >
> > In any case, that can be fixed by using the correct file system.
> >
> > Thanks, Anthony...
> >
> > Jim Burnes
> >
> >
> >
>
>

[-- Attachment #1.2: Type: text/html, Size: 5651 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Daniel P. Berrange]

On Thu, Jun 28, 2007 at 12:18:47PM -0600, Jim Burnes wrote:
> Anthony (and anyone else listening),
> 
> I've installed Centos 5, Xen 3.1 (from the Xensource RPMs) and installed a
> base Windows image to an LVM to test.
> 
> The installation wen't fairly well and I resolved various issues like
> getting the VMs to render to VNC displays properly.
> 
> I even used LVM snapshotting to do some quick copy-on-write tests.  In that
> configuration I've had up to 7 Windows VMs running at the same time, on
> separate IPs, writing to LVM copy-on-write snapshots.  The only problem is
> that LVM cow snapshots consume too much main memory.
> 
> So I proceeded to my final test which was to replace the LVM storage with
> QCOW storage and see how many Windows VMs I could run concurrently.  (I'm
> trying for 12).
> 
> The only cow tool I had was qcow-create, so I downloaded the qemu RPMs and
> installed them.
> 
> I used img2qcow to convert my installed Windows LVM image to the qcow base
> image.
> I used qcow-img to create a copy-on-write snapshot of the base image.
> I tried to boot with this as the new storage type by specifiying:
> 
> disk = [ 'tap:qcow:/var/lib/xen/image/winflp.img,hda1,w',
> 'file:/var/lib/xen/install/winflp.iso,hdc:cdrom,r']
> 
> (and other variations, by using 'hda' instead of 'hda1', using 'tap:aio'
> instead of 'tap:qcow')
> 
> But everytime I try to start the vm by 'xm create' it seems to create the VM
> and the VM immediately exits, leaving hardly any trace of even booting.  I
> tried booting directly to hard drive C: and I tried booting to the CDROM.
> Booting to the CDROM worked fine but reported no attached hard drive.
> 
> The only thing I see is a trace in the xend.log file which indicates that
> xend (or the VM) is attempting to generate some sort of hotplug event for
> tap.  Then the whole VM shuts down.
> 
> Is there something I need to do before tap:qcow works?
> 
> I suspect, but am having a hard time proving that either:
> 
> (1) My version of Xen 3.1 doesn't support tap:qcow or

It is not possible to use  blktap with HVM guests - there was code which tried
to make it work, but its utterly broken[1]. In my spare time I'm trying to fix
it, but no ETA.

Dan.
[1] it is calling APIS in XenD which no longer exist & has a try..except
    block which silently catches & ignores the errors, so you never notice
    that it failed.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Re: Writing a tool for Shared Persistent Windows Boot Image

[Jim Burnes]

[-- Attachment #1.1: Type: text/plain, Size: 3231 bytes --]

Dan,

Since blktap looks broken on HVM for the forseeable future (a real shame if
you ask me), what's the next best thing? LVM snapshots?  How can I get LVM
snapshots to use use less main memory?  Will it help if I decrease the max
size of the copy-on-write snapshot?  Usually our copy-on-write area is no
larger than 20MB.

Jim Burnes


On 6/28/07, Daniel P. Berrange <berrange@redhat.com> wrote:
>
> On Thu, Jun 28, 2007 at 12:18:47PM -0600, Jim Burnes wrote:
> > Anthony (and anyone else listening),
> >
> > I've installed Centos 5, Xen 3.1 (from the Xensource RPMs) and installed
> a
> > base Windows image to an LVM to test.
> >
> > The installation wen't fairly well and I resolved various issues like
> > getting the VMs to render to VNC displays properly.
> >
> > I even used LVM snapshotting to do some quick copy-on-write tests.  In
> that
> > configuration I've had up to 7 Windows VMs running at the same time, on
> > separate IPs, writing to LVM copy-on-write snapshots.  The only problem
> is
> > that LVM cow snapshots consume too much main memory.
> >
> > So I proceeded to my final test which was to replace the LVM storage
> with
> > QCOW storage and see how many Windows VMs I could run
> concurrently.  (I'm
> > trying for 12).
> >
> > The only cow tool I had was qcow-create, so I downloaded the qemu RPMs
> and
> > installed them.
> >
> > I used img2qcow to convert my installed Windows LVM image to the qcow
> base
> > image.
> > I used qcow-img to create a copy-on-write snapshot of the base image.
> > I tried to boot with this as the new storage type by specifiying:
> >
> > disk = [ 'tap:qcow:/var/lib/xen/image/winflp.img,hda1,w',
> > 'file:/var/lib/xen/install/winflp.iso,hdc:cdrom,r']
> >
> > (and other variations, by using 'hda' instead of 'hda1', using 'tap:aio'
> > instead of 'tap:qcow')
> >
> > But everytime I try to start the vm by 'xm create' it seems to create
> the VM
> > and the VM immediately exits, leaving hardly any trace of even
> booting.  I
> > tried booting directly to hard drive C: and I tried booting to the
> CDROM.
> > Booting to the CDROM worked fine but reported no attached hard drive.
> >
> > The only thing I see is a trace in the xend.log file which indicates
> that
> > xend (or the VM) is attempting to generate some sort of hotplug event
> for
> > tap.  Then the whole VM shuts down.
> >
> > Is there something I need to do before tap:qcow works?
> >
> > I suspect, but am having a hard time proving that either:
> >
> > (1) My version of Xen 3.1 doesn't support tap:qcow or
>
> It is not possible to use  blktap with HVM guests - there was code which
> tried
> to make it work, but its utterly broken[1]. In my spare time I'm trying to
> fix
> it, but no ETA.
>
> Dan.
> [1] it is calling APIS in XenD which no longer exist & has a try..except
>     block which silently catches & ignores the errors, so you never notice
>     that it failed.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
> -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
>

[-- Attachment #1.2: Type: text/html, Size: 4306 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Andrew Warfield]

This should have been fixed a few months ago:

# HG changeset patch
# User wim@xen-wim.site
# Date 1170895981 28800
# Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
# Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
[blktap] Allow HVM booting from blktap device(s)

What problems are you seeing?

a.

On 6/28/07, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Thu, Jun 28, 2007 at 12:18:47PM -0600, Jim Burnes wrote:
> > Anthony (and anyone else listening),
> >
> > I've installed Centos 5, Xen 3.1 (from the Xensource RPMs) and installed a
> > base Windows image to an LVM to test.
> >
> > The installation wen't fairly well and I resolved various issues like
> > getting the VMs to render to VNC displays properly.
> >
> > I even used LVM snapshotting to do some quick copy-on-write tests.  In that
> > configuration I've had up to 7 Windows VMs running at the same time, on
> > separate IPs, writing to LVM copy-on-write snapshots.  The only problem is
> > that LVM cow snapshots consume too much main memory.
> >
> > So I proceeded to my final test which was to replace the LVM storage with
> > QCOW storage and see how many Windows VMs I could run concurrently.  (I'm
> > trying for 12).
> >
> > The only cow tool I had was qcow-create, so I downloaded the qemu RPMs and
> > installed them.
> >
> > I used img2qcow to convert my installed Windows LVM image to the qcow base
> > image.
> > I used qcow-img to create a copy-on-write snapshot of the base image.
> > I tried to boot with this as the new storage type by specifiying:
> >
> > disk = [ 'tap:qcow:/var/lib/xen/image/winflp.img,hda1,w',
> > 'file:/var/lib/xen/install/winflp.iso,hdc:cdrom,r']
> >
> > (and other variations, by using 'hda' instead of 'hda1', using 'tap:aio'
> > instead of 'tap:qcow')
> >
> > But everytime I try to start the vm by 'xm create' it seems to create the VM
> > and the VM immediately exits, leaving hardly any trace of even booting.  I
> > tried booting directly to hard drive C: and I tried booting to the CDROM.
> > Booting to the CDROM worked fine but reported no attached hard drive.
> >
> > The only thing I see is a trace in the xend.log file which indicates that
> > xend (or the VM) is attempting to generate some sort of hotplug event for
> > tap.  Then the whole VM shuts down.
> >
> > Is there something I need to do before tap:qcow works?
> >
> > I suspect, but am having a hard time proving that either:
> >
> > (1) My version of Xen 3.1 doesn't support tap:qcow or
>
> It is not possible to use  blktap with HVM guests - there was code which tried
> to make it work, but its utterly broken[1]. In my spare time I'm trying to fix
> it, but no ETA.
>
> Dan.
> [1] it is calling APIS in XenD which no longer exist & has a try..except
>     block which silently catches & ignores the errors, so you never notice
>     that it failed.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/              -=|
> |=-               Projects: http://freshmeat.net/~danielpb/               -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

Re: Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Daniel P. Berrange]

On Fri, Jun 29, 2007 at 07:38:47AM -0700, Andrew Warfield wrote:
> This should have been fixed a few months ago:
> 
> # HG changeset patch
> # User wim@xen-wim.site
> # Date 1170895981 28800
> # Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
> # Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
> [blktap] Allow HVM booting from blktap device(s)
> 
> What problems are you seeing?

Yeah I saw that and am still trying to track down why it isn't doing what it
ought to. Basically I see the guest start & immediately quit/crash

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: [Xen-users] Re: Re: Writing a tool for Shared Persistent Windows Boot Image

[Andrew Warfield]

If you can send some more details on the crash we should be able to
sort this out -- it's certainly something that has worked in the past.

a.

On 6/29/07, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Fri, Jun 29, 2007 at 07:38:47AM -0700, Andrew Warfield wrote:
> > This should have been fixed a few months ago:
> >
> > # HG changeset patch
> > # User wim@xen-wim.site
> > # Date 1170895981 28800
> > # Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
> > # Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
> > [blktap] Allow HVM booting from blktap device(s)
> >
> > What problems are you seeing?
>
> Yeah I saw that and am still trying to track down why it isn't doing what it
> ought to. Basically I see the guest start & immediately quit/crash
>
> Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/              -=|
> |=-               Projects: http://freshmeat.net/~danielpb/               -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|
>

Re: Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Jim Burnes]

[-- Attachment #1.1: Type: text/plain, Size: 1364 bytes --]

Just as a side note, I'm running from the RHEL 5 3.1.0 RPMs distributed by
XenSource.  I don't know if they include this fix.

Regards,

Jim Burnes


On 6/29/07, Andrew Warfield <andrew.warfield@cl.cam.ac.uk> wrote:
>
> If you can send some more details on the crash we should be able to
> sort this out -- it's certainly something that has worked in the past.
>
> a.
>
> On 6/29/07, Daniel P. Berrange <berrange@redhat.com> wrote:
> > On Fri, Jun 29, 2007 at 07:38:47AM -0700, Andrew Warfield wrote:
> > > This should have been fixed a few months ago:
> > >
> > > # HG changeset patch
> > > # User wim@xen-wim.site
> > > # Date 1170895981 28800
> > > # Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
> > > # Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
> > > [blktap] Allow HVM booting from blktap device(s)
> > >
> > > What problems are you seeing?
> >
> > Yeah I saw that and am still trying to track down why it isn't doing
> what it
> > ought to. Basically I see the guest start & immediately quit/crash
> >
> > Dan.
> > --
> > |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392
> 2496 -=|
> > |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> > |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> > |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
> >
>

[-- Attachment #1.2: Type: text/html, Size: 2309 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: [Xen-users] Re: Re: Writing a tool for Shared Persistent Windows Boot Image

[Jim Burnes]

[-- Attachment #1.1: Type: text/plain, Size: 2400 bytes --]

Speaking of the XenSource 3.1.0 RPMs, I'd like to make a suggestion.  When
you build the xen-kernel for those 3.1.0 RPMs, can you make sure that the
loop device is built as a module?  By default the loop device is limited to
8 instances and when you build it into the kernel it makes it very difficult
to change max_loop.

I tried adding it as a xen-kernel boot option in grub.conf, but as far as I
can tell it's ignored when the loop device is compiled in the kernel.

For example, even though I added max_loop=16 to the boot options and
modified the udev configuration to build 16 loop devices, every time xen
does an 'losetup' on any device greater than loop7 it fails because the
system doesn't think the device exists.

Just FYI.  This means I'll have to raise the limit by recompiling the whole
kernel.

Jim Burnes


On 6/29/07, Jim Burnes <jvburnes@gmail.com> wrote:
>
> Just as a side note, I'm running from the RHEL 5 3.1.0 RPMs distributed by
> XenSource.  I don't know if they include this fix.
>
> Regards,
>
> Jim Burnes
>
>
> On 6/29/07, Andrew Warfield <andrew.warfield@cl.cam.ac.uk> wrote:
> >
> > If you can send some more details on the crash we should be able to
> > sort this out -- it's certainly something that has worked in the past.
> >
> > a.
> >
> > On 6/29/07, Daniel P. Berrange < berrange@redhat.com> wrote:
> > > On Fri, Jun 29, 2007 at 07:38:47AM -0700, Andrew Warfield wrote:
> > > > This should have been fixed a few months ago:
> > > >
> > > > # HG changeset patch
> > > > # User wim@xen-wim.site
> > > > # Date 1170895981 28800
> > > > # Node ID 6524e02edbeb8aebe65aa5400af4b09dfccb8729
> > > > # Parent  780f097b54c5f9161f7c6cf3f86b2bb72cc43587
> > > > [blktap] Allow HVM booting from blktap device(s)
> > > >
> > > > What problems are you seeing?
> > >
> > > Yeah I saw that and am still trying to track down why it isn't doing
> > what it
> > > ought to. Basically I see the guest start & immediately quit/crash
> > >
> > > Dan.
> > > --
> > > |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392
> > 2496 -=|
> > > |=-           Perl modules: http://search.cpan.org/~danberr/
> > <http://search.cpan.org/%7Edanberr/>              -=|
> > > |=-               Projects: http://freshmeat.net/~danielpb/<http://freshmeat.net/%7Edanielpb/>
> > -=|
> > > |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> > 9505  -=|
> > >
> >
>
>

[-- Attachment #1.2: Type: text/html, Size: 3976 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Daniel P. Berrange]

[-- Attachment #1: Type: text/plain, Size: 3325 bytes --]

On Fri, Jun 29, 2007 at 09:18:50AM -0700, Andrew Warfield wrote:
> If you can send some more details on the crash we should be able to
> sort this out -- it's certainly something that has worked in the past.

Ok, so here's the scenario. Traditionally with HVM I have a disk

   file:/xen/rhel4i386.img,hda,w

Having seen the changeset 13827:6524e02edbeb I tried

  tap:aio:/xen/rhel4i386.img,hda,w

And

  tap:aio:/xen/rhel4i386.img,xvda,w

The latter is the preferred, since paravirt drivers should not be hijacking
the IDE devices inside the guest. However, the changeset 13827 doesn't seem
to support xvd* since QEMU filters out any devices with such a name.

With vanilla  Xen 3.1.0 qemu goes defunct when starting the guest
logging

 qemu: could not open hard disk image 'aio:/xen/rhel4i386.img'

After a little investigation I found that in BlktapController

       try:
           imagetype = self.vm.info['image']['type']
       except:
           imagetype = ""

Has long ago been broken and should instead be

        try:
            imagetype = self.vm.info.image_type()
        except:
            imagetype = ""


Once I made that change I can see a phantom device being created, but QEMU
still crashes & burns with  qemu-dm-XXXX.log showing

  qemu: could not open hard disk image '/dev/xvdc1'

I started to debug this, but looking at changeset 13827:6524e02edbeb I rapidly
came to the conclusion that the whole idea of phantom devices is complete
overkill & the entire problem could be addressed with a couple of lines in
ioemu/xenstore.c.  QEMU already knows how to handle all the different 
types of file format blktap does, so there's no need to setup extra phantom
devices. All thats needed is for QEMU to a) strip the aio: (or equivalent)
prefix and b) convert xvdN -> hdN if required.

So I'm attaching two patches. The first reverts 13827:6524e02edbeb
and the second tweaks ioemu/xenstore.c so it can handle blktap devices
directly. With these applied ontop of Xen 3.1.0 I can successfully
start HVM guests using  the two example tap:aio lines I show earlier
in this mail. The patch also adds a couple of logging lines which 
end up in qemu-dm-XXX.log as they'll be useful if ever debugging QEMU
boot issues - it is far too silent when things go wrong which makes
diagnosis hard.

    Signed-off-by: Daniel Berrange <berrange@redhat.com>


The patch being reverted:
 
 $ diffstat xen-revert-phantom.patch 
 ioemu/xenstore.c                           |   46 ---------------------
 python/xen/xend/XendConfig.py              |   41 -------------------
 python/xen/xend/XendDomainInfo.py          |   58 ---------------------------
 python/xen/xend/server/BlktapController.py |   62 -----------------------------
 python/xen/xend/server/DevController.py    |   13 ------
 5 files changed, 3 insertions(+), 217 deletions(-)

The new patch:

 $ diffstat xen-qemu-blktap.patch 
 xenstore.c |   28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

[-- Attachment #2: xen-revert-phantom.patch --]
[-- Type: text/plain, Size: 12359 bytes --]

diff -r c0b0974fb055 tools/ioemu/xenstore.c
--- a/tools/ioemu/xenstore.c	Fri May 18 16:59:32 2007 +0100
+++ b/tools/ioemu/xenstore.c	Fri Jun 29 15:48:53 2007 -0400
@@ -10,7 +10,6 @@
 
 #include "vl.h"
 #include "block_int.h"
-#include <unistd.h>
 #include <sys/ipc.h>
 #include <sys/shm.h>
 #include <sys/types.h>
@@ -61,28 +60,11 @@ void xenstore_check_new_media_present(in
     qemu_mod_timer(insert_timer, qemu_get_clock(rt_clock) + timeout);
 }
 
-static void waitForDevice(char *fn)
-{ 
-    struct stat sbuf;
-    int status;
-    int uwait = UWAIT_MAX;
-
-    do {
-        status = stat(fn, &sbuf);
-        if (!status) break;
-        usleep(UWAIT);
-        uwait -= UWAIT;
-    } while (uwait > 0);
-
-    return;
-}
-
 void xenstore_parse_domain_config(int domid)
 {
     char **e = NULL;
     char *buf = NULL, *path;
-    char *fpath = NULL, *bpath = NULL,
-        *dev = NULL, *params = NULL, *type = NULL;
+    char *bpath = NULL, *dev = NULL, *params = NULL, *type = NULL;
     int i, is_scsi;
     unsigned int len, num, hd_index;
 
@@ -140,36 +122,12 @@ void xenstore_parse_domain_config(int do
         params = xs_read(xsh, XBT_NULL, buf, &len);
         if (params == NULL)
             continue;
-        /* 
-         * check if device has a phantom vbd; the phantom is hooked
-         * to the frontend device (for ease of cleanup), so lookup 
-         * the frontend device, and see if there is a phantom_vbd
-         * if there is, we will use resolution as the filename
-         */
-        if (pasprintf(&buf, "%s/device/vbd/%s/phantom_vbd", path, e[i]) == -1)
-            continue;
-        free(fpath);
-        fpath = xs_read(xsh, XBT_NULL, buf, &len);
-        if (fpath) {
-            if (pasprintf(&buf, "%s/dev", fpath) == -1)
-                continue;
-            free(params);
-            params = xs_read(xsh, XBT_NULL, buf , &len);
-            if (params) {
-                /* 
-                 * wait for device, on timeout silently fail because we will 
-                 * fail to open below
-                 */
-                waitForDevice(params);
-            }
-        }
 
         bs_table[hd_index + (is_scsi ? MAX_DISKS : 0)] = bdrv_new(dev);
         /* check if it is a cdrom */
         if (type && !strcmp(type, "cdrom")) {
             bdrv_set_type_hint(bs_table[hd_index], BDRV_TYPE_CDROM);
-            if (pasprintf(&buf, "%s/params", bpath) != -1)
-                xs_watch(xsh, buf, dev);
+	    xs_watch(xsh, buf, dev);
         }
         /* open device now if media present */
         if (params[0]) {
diff -r c0b0974fb055 tools/python/xen/xend/XendConfig.py
--- a/tools/python/xen/xend/XendConfig.py	Fri May 18 16:59:32 2007 +0100
+++ b/tools/python/xen/xend/XendConfig.py	Fri Jun 29 15:37:43 2007 -0400
@@ -1126,47 +1126,6 @@ class XendConfig(dict):
         # no valid device to add
         return ''
 
-    def phantom_device_add(self, dev_type, cfg_xenapi = None,
-                   target = None):
-        """Add a phantom tap device configuration in XenAPI struct format.
-        """
-
-        if target == None:
-            target = self
-        
-        if dev_type not in XendDevices.valid_devices() and \
-           dev_type not in XendDevices.pseudo_devices():        
-            raise XendConfigError("XendConfig: %s not a valid device type" %
-                            dev_type)
-
-        if cfg_xenapi == None:
-            raise XendConfigError("XendConfig: device_add requires some "
-                                  "config.")
-
-        if cfg_xenapi:
-            log.debug("XendConfig.phantom_device_add: %s" % str(cfg_xenapi))
- 
-        if cfg_xenapi:
-            dev_info = {}            
-            if dev_type in ('vbd', 'tap'):
-                if dev_type == 'vbd':
-                    dev_info['uname'] = cfg_xenapi.get('image', '')
-                    dev_info['dev'] = '%s:disk' % cfg_xenapi.get('device')
-                elif dev_type == 'tap':
-                    if cfg_xenapi.get('image').find('tap:') == -1:
-                        dev_info['uname'] = 'tap:qcow:%s' % cfg_xenapi.get('image')
-                    dev_info['dev'] =  '/dev/%s' % cfg_xenapi.get('device')
-                    dev_info['uname'] = cfg_xenapi.get('image')
-                dev_info['mode'] = cfg_xenapi.get('mode')
-                dev_info['backend'] = '0'
-                dev_uuid = cfg_xenapi.get('uuid', uuid.createString())
-                dev_info['uuid'] = dev_uuid
-                self['devices'][dev_uuid] = (dev_type, dev_info)
-                self['vbd_refs'].append(dev_uuid)
-                return dev_uuid
-
-        return ''
-
     def console_add(self, protocol, location, other_config = {}):
         dev_uuid = uuid.createString()
         if protocol == 'vt100':
diff -r c0b0974fb055 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Fri May 18 16:59:32 2007 +0100
+++ b/tools/python/xen/xend/XendDomainInfo.py	Fri Jun 29 15:37:43 2007 -0400
@@ -1623,49 +1623,14 @@ class XendDomainInfo:
     # VM Destroy
     # 
 
-    def _prepare_phantom_paths(self):
-        # get associated devices to destroy
-        # build list of phantom devices to be removed after normal devices
-        plist = []
-        if self.domid is not None:
-            from xen.xend.xenstore.xstransact import xstransact
-            t = xstransact("%s/device/vbd" % GetDomainPath(self.domid))
-            for dev in t.list():
-                backend_phantom_vbd = xstransact.Read("%s/device/vbd/%s/phantom_vbd" \
-                                      % (self.dompath, dev))
-                if backend_phantom_vbd is not None:
-                    frontend_phantom_vbd =  xstransact.Read("%s/frontend" \
-                                      % backend_phantom_vbd)
-                    plist.append(backend_phantom_vbd)
-                    plist.append(frontend_phantom_vbd)
-        return plist
-
-    def _cleanup_phantom_devs(self, plist):
-        # remove phantom devices
-        if not plist == []:
-            time.sleep(2)
-        for paths in plist:
-            if paths.find('backend') != -1:
-                from xen.xend.server import DevController
-                # Modify online status /before/ updating state (latter is watched by
-                # drivers, so this ordering avoids a race).
-                xstransact.Write(paths, 'online', "0")
-                xstransact.Write(paths, 'state', str(DevController.xenbusState['Closing']))
-            # force
-            xstransact.Remove(paths)
-
     def destroy(self):
         """Cleanup VM and destroy domain.  Nothrow guarantee."""
 
         log.debug("XendDomainInfo.destroy: domid=%s", str(self.domid))
-
-        paths = self._prepare_phantom_paths()
 
         self._cleanupVm()
         if self.dompath is not None:
             self.destroyDomain()
-
-        self._cleanup_phantom_devs(paths)
 
         if "transient" in self.info["other_config"] \
            and bool(self.info["other_config"]["transient"]):
@@ -1675,8 +1640,6 @@ class XendDomainInfo:
 
     def destroyDomain(self):
         log.debug("XendDomainInfo.destroyDomain(%s)", str(self.domid))
-
-        paths = self._prepare_phantom_paths()
 
         try:
             if self.domid is not None:
@@ -1692,7 +1655,7 @@ class XendDomainInfo:
         XendDomain.instance().remove_domain(self)
 
         self.cleanupDomain()
-        self._cleanup_phantom_devs(paths)
+
 
 
     def resumeDomain(self):
@@ -2403,25 +2366,6 @@ class XendDomainInfo:
             
         return dev_uuid
 
-    def create_phantom_vbd_with_vdi(self, xenapi_vbd, vdi_image_path):
-        """Create a VBD using a VDI from XendStorageRepository.
-
-        @param xenapi_vbd: vbd struct from the Xen API
-        @param vdi_image_path: VDI UUID
-        @rtype: string
-        @return: uuid of the device
-        """
-        xenapi_vbd['image'] = vdi_image_path
-        dev_uuid = self.info.phantom_device_add('tap', cfg_xenapi = xenapi_vbd)
-        if not dev_uuid:
-            raise XendError('Failed to create device')
-
-        if self._stateGet() == XEN_API_VM_POWER_STATE_RUNNING:
-            _, config = self.info['devices'][dev_uuid]
-            config['devid'] = self.getDeviceController('tap').createDevice(config)
-
-        return config['devid']
-
     def create_vif(self, xenapi_vif):
         """Create VIF device from the passed struct in Xen API format.
 
diff -r c0b0974fb055 tools/python/xen/xend/server/BlktapController.py
--- a/tools/python/xen/xend/server/BlktapController.py	Fri May 18 16:59:32 2007 +0100
+++ b/tools/python/xen/xend/server/BlktapController.py	Fri Jun 29 15:37:43 2007 -0400
@@ -2,10 +2,7 @@
 
 
 from xen.xend.server.blkif import BlkifController
-from xen.xend.XendLogging import log
 
-phantomDev = 0;
-phantomId = 0;
 
 class BlktapController(BlkifController):
     def __init__(self, vm):
@@ -15,62 +12,3 @@ class BlktapController(BlkifController):
         """@see DevController#frontendRoot"""
         
         return "%s/device/vbd" % self.vm.getDomainPath()
-
-    def getDeviceDetails(self, config):
-        (devid, back, front) = BlkifController.getDeviceDetails(self, config)
-
-        phantomDevid = 0
-        wrapped = False
-
-        try:
-            imagetype = self.vm.info['image']['type']
-        except:
-            imagetype = ""
-
-        if imagetype == 'hvm':
-            tdevname = back['dev']
-            index = ['c', 'd', 'e', 'f', 'g', 'h', 'i', \
-                     'j', 'l', 'm', 'n', 'o', 'p']
-            while True:
-                global phantomDev
-                global phantomId
-                import os, stat
-
-                phantomId = phantomId + 1
-                if phantomId == 16:
-                    if index[phantomDev] == index[-1]:
-                        if wrapped:
-                            raise VmError(" No loopback block \
-                                       devices are available. ")
-                        wrapped = True
-                        phantomDev = 0
-                    else:
-                        phantomDev = phantomDev + 1
-                    phantomId = 1
-                devname = 'xvd%s%d' % (index[phantomDev], phantomId)
-                try:
-                    info = os.stat('/dev/%s' % devname)
-                except:
-                    break
-
-            vbd = { 'mode': 'w', 'device': devname }
-            fn = 'tap:%s' % back['params']
-
-            # recurse ... by creating the vbd, then fallthrough
-            # and finish creating the original device
-
-            from xen.xend import XendDomain
-            dom0 = XendDomain.instance().privilegedDomain()
-            phantomDevid = dom0.create_phantom_vbd_with_vdi(vbd, fn)
-            # we need to wait for this device at a higher level
-            # the vbd that gets created will have a link to us
-            # and will let them do it there
-
-        # add a hook to point to the phantom device,
-        # root path is always the same (dom0 tap)
-        if phantomDevid != 0:
-            front['phantom_vbd'] = '/local/domain/0/backend/tap/0/%s' \
-                                   % str(phantomDevid)
-
-        return (devid, back, front)
-
diff -r c0b0974fb055 tools/python/xen/xend/server/DevController.py
--- a/tools/python/xen/xend/server/DevController.py	Fri May 18 16:59:32 2007 +0100
+++ b/tools/python/xen/xend/server/DevController.py	Fri Jun 29 15:37:43 2007 -0400
@@ -474,19 +474,6 @@ class DevController:
     def waitForBackend(self, devid):
 
         frontpath = self.frontendPath(devid)
-        # lookup a phantom 
-        phantomPath = xstransact.Read(frontpath, 'phantom_vbd')
-        if phantomPath is not None:
-            log.debug("Waiting for %s's phantom %s.", devid, phantomPath)
-            statusPath = phantomPath + '/' + HOTPLUG_STATUS_NODE
-            ev = Event()
-            result = { 'status': Timeout }
-            xswatch(statusPath, hotplugStatusCallback, ev, result)
-            ev.wait(DEVICE_CREATE_TIMEOUT)
-            err = xstransact.Read(statusPath, HOTPLUG_ERROR_NODE)
-            if result['status'] != 'Connected':
-                return (result['status'], err)
-            
         backpath = xstransact.Read(frontpath, "backend")
 
 

[-- Attachment #3: xen-qemu-blktap.patch --]
[-- Type: text/plain, Size: 2846 bytes --]

diff -r ff1a3395024c tools/ioemu/xenstore.c
--- a/tools/ioemu/xenstore.c	Fri Jun 29 15:37:48 2007 -0400
+++ b/tools/ioemu/xenstore.c	Fri Jun 29 16:06:13 2007 -0400
@@ -64,7 +64,7 @@ void xenstore_parse_domain_config(int do
 {
     char **e = NULL;
     char *buf = NULL, *path;
-    char *bpath = NULL, *dev = NULL, *params = NULL, *type = NULL;
+    char *bpath = NULL, *dev = NULL, *params = NULL, *type = NULL, *drv = NULL;
     int i, is_scsi;
     unsigned int len, num, hd_index;
 
@@ -98,6 +98,13 @@ void xenstore_parse_domain_config(int do
         bpath = xs_read(xsh, XBT_NULL, buf, &len);
         if (bpath == NULL)
             continue;
+        /* read the driver type of the device */
+        if (pasprintf(&buf, "%s/type", bpath) == -1)
+            continue;
+        free(drv);
+        drv = xs_read(xsh, XBT_NULL, buf, &len);
+        if (drv == NULL)
+            continue;
         /* read the name of the device */
         if (pasprintf(&buf, "%s/dev", bpath) == -1)
             continue;
@@ -105,6 +112,13 @@ void xenstore_parse_domain_config(int do
         dev = xs_read(xsh, XBT_NULL, buf, &len);
         if (dev == NULL)
             continue;
+	/* Force xvdN to look like  hdN */
+	if (!strncmp(dev, "xvd", 3)) {
+	    fprintf(logfile, "Converting device type '%s'\n", dev);
+	    memmove(dev, dev+1, strlen(dev));
+	    dev[0] = 'h';
+	    dev[1] = 'd';
+	}
         is_scsi = !strncmp(dev, "sd", 2);
         if ((strncmp(dev, "hd", 2) && !is_scsi) || strlen(dev) != 3 )
             continue;
@@ -122,7 +136,15 @@ void xenstore_parse_domain_config(int do
         params = xs_read(xsh, XBT_NULL, buf, &len);
         if (params == NULL)
             continue;
-
+	/* Strip off blktap sub-type prefix aio: - QEMU can autodetect this */
+	if (!strcmp(drv, "tap") && params[0]) {
+	    char *offset = strchr(params, ':');
+	    if (!offset)
+	        continue;
+	    fprintf(logfile, "Stripping blktap sub-type prefix from %s\n", params);
+	    memmove(params, offset+1, strlen(offset+1)+1);
+	}
+	fprintf(logfile, "Creating disk '%s' with driver '%s'\n", dev, drv);
         bs_table[hd_index + (is_scsi ? MAX_DISKS : 0)] = bdrv_new(dev);
         /* check if it is a cdrom */
         if (type && !strcmp(type, "cdrom")) {
@@ -131,6 +153,7 @@ void xenstore_parse_domain_config(int do
         }
         /* open device now if media present */
         if (params[0]) {
+            fprintf(logfile, "Initializing disk '%s' with media '%s'\n", dev, params);
             if (bdrv_open(bs_table[hd_index + (is_scsi ? MAX_DISKS : 0)],
                           params, 0 /* snapshot */) < 0)
                 fprintf(stderr, "qemu: could not open hard disk image '%s'\n",
@@ -146,6 +169,7 @@ void xenstore_parse_domain_config(int do
 
 
  out:
+    free(drv);
     free(type);
     free(params);
     free(dev);

[-- Attachment #4: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: [Xen-users] Re: Re: Writing a tool for Shared Persistent Windows Boot Image

[Jim Burnes]

[-- Attachment #1.1: Type: text/plain, Size: 3665 bytes --]

Daniel,

Does this take care of both "tap:aio" and "tap:qcow" disk specifications or
is "tap:qcow" deprecated?

Thanks,

Jim Burnes


On 6/29/07, Daniel P. Berrange <berrange@redhat.com> wrote:
>
> On Fri, Jun 29, 2007 at 09:18:50AM -0700, Andrew Warfield wrote:
> > If you can send some more details on the crash we should be able to
> > sort this out -- it's certainly something that has worked in the past.
>
> Ok, so here's the scenario. Traditionally with HVM I have a disk
>
>    file:/xen/rhel4i386.img,hda,w
>
> Having seen the changeset 13827:6524e02edbeb I tried
>
>   tap:aio:/xen/rhel4i386.img,hda,w
>
> And
>
>   tap:aio:/xen/rhel4i386.img,xvda,w
>
> The latter is the preferred, since paravirt drivers should not be
> hijacking
> the IDE devices inside the guest. However, the changeset 13827 doesn't
> seem
> to support xvd* since QEMU filters out any devices with such a name.
>
> With vanilla  Xen 3.1.0 qemu goes defunct when starting the guest
> logging
>
> qemu: could not open hard disk image 'aio:/xen/rhel4i386.img'
>
> After a little investigation I found that in BlktapController
>
>        try:
>            imagetype = self.vm.info['image']['type']
>        except:
>            imagetype = ""
>
> Has long ago been broken and should instead be
>
>         try:
>             imagetype = self.vm.info.image_type()
>         except:
>             imagetype = ""
>
>
> Once I made that change I can see a phantom device being created, but QEMU
> still crashes & burns with  qemu-dm-XXXX.log showing
>
>   qemu: could not open hard disk image '/dev/xvdc1'
>
> I started to debug this, but looking at changeset 13827:6524e02edbeb I
> rapidly
> came to the conclusion that the whole idea of phantom devices is complete
> overkill & the entire problem could be addressed with a couple of lines in
> ioemu/xenstore.c.  QEMU already knows how to handle all the different
> types of file format blktap does, so there's no need to setup extra
> phantom
> devices. All thats needed is for QEMU to a) strip the aio: (or equivalent)
> prefix and b) convert xvdN -> hdN if required.
>
> So I'm attaching two patches. The first reverts 13827:6524e02edbeb
> and the second tweaks ioemu/xenstore.c so it can handle blktap devices
> directly. With these applied ontop of Xen 3.1.0 I can successfully
> start HVM guests using  the two example tap:aio lines I show earlier
> in this mail. The patch also adds a couple of logging lines which
> end up in qemu-dm-XXX.log as they'll be useful if ever debugging QEMU
> boot issues - it is far too silent when things go wrong which makes
> diagnosis hard.
>
>     Signed-off-by: Daniel Berrange <berrange@redhat.com>
>
>
> The patch being reverted:
>
> $ diffstat xen-revert-phantom.patch
> ioemu/xenstore.c                           |   46 ---------------------
> python/xen/xend/XendConfig.py              |   41 -------------------
> python/xen/xend/XendDomainInfo.py          |   58
> ---------------------------
> python/xen/xend/server/BlktapController.py |   62
> -----------------------------
> python/xen/xend/server/DevController.py    |   13 ------
> 5 files changed, 3 insertions(+), 217 deletions(-)
>
> The new patch:
>
> $ diffstat xen-qemu-blktap.patch
> xenstore.c |   28 ++++++++++++++++++++++++++--
> 1 file changed, 26 insertions(+), 2 deletions(-)
>
> Regards,
> Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
> -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/
>               -=|
> |=-               Projects: http://freshmeat.net/~danielpb/
> -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B
> 9505  -=|
>
>

[-- Attachment #1.2: Type: text/html, Size: 5447 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Daniel P. Berrange]

On Fri, Jun 29, 2007 at 02:32:23PM -0600, Jim Burnes wrote:
> Daniel,
> 
> Does this take care of both "tap:aio" and "tap:qcow" disk specifications or
> is "tap:qcow" deprecated?

I've not explicitly tested anything other than tap:aio, but QEMU supports
basically all the formats that blktap does, so I imagine tap:qcow ought
to work with the patch I posted.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: [Xen-users] Re: Re: Writing a tool for Shared Persistent Windows Boot Image

[Ian Campbell]

On Fri, 2007-06-29 at 13:07 -0600, Jim Burnes wrote:
> Speaking of the XenSource 3.1.0 RPMs, I'd like to make a suggestion.
> When you build the xen-kernel for those 3.1.0 RPMs, can you make sure
> that the loop device is built as a module?  By default the loop device
> is limited to 8 instances and when you build it into the kernel it
> makes it very difficult to change max_loop. 
> 
> I tried adding it as a xen-kernel boot option in grub.conf, but as far
> as I can tell it's ignored when the loop device is compiled in the
> kernel.

When a module is builtin you often need to prefix the option on the
kernel command line with the module name. So adding loop.max_loop=<N>
may work.

Ian.

Re: Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Andrew Warfield]

The problem with this approach is that you end up using two instances
of whatever virtual disk code you want.  In the case of raw writes to
an image file (tap:aio) this is more or less okay, except for the fact
that qemu has a bad habit of buffering writes and so you can get stuck
in a nasty late write race when you switch from emulated writes over
to using pv drivers.

For non-raw writes this is worse, because most of the image file
drivers cache metadata in memory and don't necessarily pick up changes
to the image files when you switch from emulated to pv.  Also, the
disparate implementations make me a bit nervous about headers exactly
matching and so on.

The intention of the phantom device is to try to keep things down to a
single data path, and a single driver implementation.

a.

On 6/29/07, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Fri, Jun 29, 2007 at 09:18:50AM -0700, Andrew Warfield wrote:
> > If you can send some more details on the crash we should be able to
> > sort this out -- it's certainly something that has worked in the past.
>
> Ok, so here's the scenario. Traditionally with HVM I have a disk
>
>    file:/xen/rhel4i386.img,hda,w
>
> Having seen the changeset 13827:6524e02edbeb I tried
>
>   tap:aio:/xen/rhel4i386.img,hda,w
>
> And
>
>   tap:aio:/xen/rhel4i386.img,xvda,w
>
> The latter is the preferred, since paravirt drivers should not be hijacking
> the IDE devices inside the guest. However, the changeset 13827 doesn't seem
> to support xvd* since QEMU filters out any devices with such a name.
>
> With vanilla  Xen 3.1.0 qemu goes defunct when starting the guest
> logging
>
>  qemu: could not open hard disk image 'aio:/xen/rhel4i386.img'
>
> After a little investigation I found that in BlktapController
>
>        try:
>            imagetype = self.vm.info['image']['type']
>        except:
>            imagetype = ""
>
> Has long ago been broken and should instead be
>
>         try:
>             imagetype = self.vm.info.image_type()
>         except:
>             imagetype = ""
>
>
> Once I made that change I can see a phantom device being created, but QEMU
> still crashes & burns with  qemu-dm-XXXX.log showing
>
>   qemu: could not open hard disk image '/dev/xvdc1'
>
> I started to debug this, but looking at changeset 13827:6524e02edbeb I rapidly
> came to the conclusion that the whole idea of phantom devices is complete
> overkill & the entire problem could be addressed with a couple of lines in
> ioemu/xenstore.c.  QEMU already knows how to handle all the different
> types of file format blktap does, so there's no need to setup extra phantom
> devices. All thats needed is for QEMU to a) strip the aio: (or equivalent)
> prefix and b) convert xvdN -> hdN if required.
>
> So I'm attaching two patches. The first reverts 13827:6524e02edbeb
> and the second tweaks ioemu/xenstore.c so it can handle blktap devices
> directly. With these applied ontop of Xen 3.1.0 I can successfully
> start HVM guests using  the two example tap:aio lines I show earlier
> in this mail. The patch also adds a couple of logging lines which
> end up in qemu-dm-XXX.log as they'll be useful if ever debugging QEMU
> boot issues - it is far too silent when things go wrong which makes
> diagnosis hard.
>
>     Signed-off-by: Daniel Berrange <berrange@redhat.com>
>
>
> The patch being reverted:
>
>  $ diffstat xen-revert-phantom.patch
>  ioemu/xenstore.c                           |   46 ---------------------
>  python/xen/xend/XendConfig.py              |   41 -------------------
>  python/xen/xend/XendDomainInfo.py          |   58 ---------------------------
>  python/xen/xend/server/BlktapController.py |   62 -----------------------------
>  python/xen/xend/server/DevController.py    |   13 ------
>  5 files changed, 3 insertions(+), 217 deletions(-)
>
> The new patch:
>
>  $ diffstat xen-qemu-blktap.patch
>  xenstore.c |   28 ++++++++++++++++++++++++++--
>  1 file changed, 26 insertions(+), 2 deletions(-)
>
> Regards,
> Dan.
> --
> |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
> |=-           Perl modules: http://search.cpan.org/~danberr/              -=|
> |=-               Projects: http://freshmeat.net/~danielpb/               -=|
> |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|
>
>

Re: Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Daniel P. Berrange]

On Sun, Jul 01, 2007 at 01:28:56PM -0700, Andrew Warfield wrote:
> The problem with this approach is that you end up using two instances
> of whatever virtual disk code you want.  In the case of raw writes to
> an image file (tap:aio) this is more or less okay, except for the fact
> that qemu has a bad habit of buffering writes and so you can get stuck
> in a nasty late write race when you switch from emulated writes over
> to using pv drivers.

AFAIR, if the guest OS sends a flush request to the IDE device, then
QEMU should immediately be flushing the data to disk in the host - if
it doesn't, then this is already a potential data corrupter if either
the guest or host crashes because journaling fileystems rely on the
fact that when they ask for a journal flush it is not buffered in RAM.

I don't think a guest OS would ever be activating both the IDE and
paravirt drivers for a device though would it ? You either load IDE
drivers, or paravirt at any given time. If you've got a guest using
PV drivers, then the only point where the IDE interface would come
into play is for the initial BIOS boot process & that should be 
read-only access.

> For non-raw writes this is worse, because most of the image file
> drivers cache metadata in memory and don't necessarily pick up changes
> to the image files when you switch from emulated to pv.  Also, the
> disparate implementations make me a bit nervous about headers exactly
> matching and so on.

Are you refering to the qcow file format headers here ? If blktap isn't
in compliance with the QEMU QCOW spec then that is a bug in itself which
needs fixing, because it is not good for portability of FS images.

> The intention of the phantom device is to try to keep things down to a
> single data path, and a single driver implementation.

Can you explain just how the phantom device is supposed to work, because
its not working in current code, even with the fix I mentioned blow & it
is not immediately clear to me what else needs fixing ...

> >On Fri, Jun 29, 2007 at 09:18:50AM -0700, Andrew Warfield wrote:
> >> If you can send some more details on the crash we should be able to
> >> sort this out -- it's certainly something that has worked in the past.
> >
> >Ok, so here's the scenario. Traditionally with HVM I have a disk
> >
> >   file:/xen/rhel4i386.img,hda,w
> >
> >Having seen the changeset 13827:6524e02edbeb I tried
> >
> >  tap:aio:/xen/rhel4i386.img,hda,w
> >
> >And
> >
> >  tap:aio:/xen/rhel4i386.img,xvda,w
> >
> >The latter is the preferred, since paravirt drivers should not be hijacking
> >the IDE devices inside the guest. However, the changeset 13827 doesn't seem
> >to support xvd* since QEMU filters out any devices with such a name.
> >
> >With vanilla  Xen 3.1.0 qemu goes defunct when starting the guest
> >logging
> >
> > qemu: could not open hard disk image 'aio:/xen/rhel4i386.img'
> >
> >After a little investigation I found that in BlktapController
> >
> >       try:
> >           imagetype = self.vm.info['image']['type']
> >       except:
> >           imagetype = ""
> >
> >Has long ago been broken and should instead be
> >
> >        try:
> >            imagetype = self.vm.info.image_type()
> >        except:
> >            imagetype = ""
> >
> >
> >Once I made that change I can see a phantom device being created, but QEMU
> >still crashes & burns with  qemu-dm-XXXX.log showing
> >
> >  qemu: could not open hard disk image '/dev/xvdc1'
> >
> >I started to debug this, but looking at changeset 13827:6524e02edbeb I 
> >rapidly
> >came to the conclusion that the whole idea of phantom devices is complete
> >overkill & the entire problem could be addressed with a couple of lines in
> >ioemu/xenstore.c.  QEMU already knows how to handle all the different
> >types of file format blktap does, so there's no need to setup extra phantom
> >devices. All thats needed is for QEMU to a) strip the aio: (or equivalent)
> >prefix and b) convert xvdN -> hdN if required.
> >
> >So I'm attaching two patches. The first reverts 13827:6524e02edbeb
> >and the second tweaks ioemu/xenstore.c so it can handle blktap devices
> >directly. With these applied ontop of Xen 3.1.0 I can successfully
> >start HVM guests using  the two example tap:aio lines I show earlier
> >in this mail. The patch also adds a couple of logging lines which
> >end up in qemu-dm-XXX.log as they'll be useful if ever debugging QEMU
> >boot issues - it is far too silent when things go wrong which makes
> >diagnosis hard.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: Re: [Xen-devel] Re: Writing a tool for Shared Persistent Windows Boot Image

[Daniel P. Berrange]

On Sun, Jul 01, 2007 at 10:41:37PM +0100, Daniel P. Berrange wrote:
> On Sun, Jul 01, 2007 at 01:28:56PM -0700, Andrew Warfield wrote:
> > The problem with this approach is that you end up using two instances
> > of whatever virtual disk code you want.  In the case of raw writes to
> > an image file (tap:aio) this is more or less okay, except for the fact
> > that qemu has a bad habit of buffering writes and so you can get stuck
> > in a nasty late write race when you switch from emulated writes over
> > to using pv drivers.
> 
> AFAIR, if the guest OS sends a flush request to the IDE device, then
> QEMU should immediately be flushing the data to disk in the host - if
> it doesn't, then this is already a potential data corrupter if either
> the guest or host crashes because journaling fileystems rely on the
> fact that when they ask for a journal flush it is not buffered in RAM.
> 
> I don't think a guest OS would ever be activating both the IDE and
> paravirt drivers for a device though would it ? You either load IDE
> drivers, or paravirt at any given time. If you've got a guest using
> PV drivers, then the only point where the IDE interface would come
> into play is for the initial BIOS boot process & that should be 
> read-only access.

Thinking about it from the safety POV, the QEMU process could register
a xenstore watch to be notified when the paravirt frontend driver
connected to the backend. At this time it could forceably disable the
IDE device associated with the backend, thus ensuring you never have 
two concurrently active data paths to the same underlying disk.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|