[PATCH] Xen Guest Kexec

[PATCH] Xen Guest Kexec

[Mark Williamson]

All,

I'm posting to the Xen-devel list and the OSDL fastboot list.  There are a 
number of Xen folks who'll want to look at the code.  For the fastboot folks, 
this is mostly intended as an announcement of the port.  Please limit your 
replies to the list(s) that they applies to, thanks :-)

A number of people have expressed interest in kexec support for Xen guests.  
In particular, it's mainly useful for purposes of implementing an in-guest 
bootloader app and avoiding the need for dom0 to access the guest filesystem.

It's a largish patch, so I stuck it online: 
http://www.cl.cam.ac.uk/~maw48/xenguest_kexec.patch
Tested on i386.  I suspect it'll have build issues on x86_64 but those should 
be easy to fix - actually the code is probably generic enough to work on 
both.

You'll also need a modified version of the kexec tools, which I'll post 
details of later.

Notes on the implementation:
Kexec in a Xen guest doesn't work like on a real machine.  The guest relies on 
outside assistance to complete the job.  The guest communicates details of 
the kexec to Xend.  Xend extracts the data from the guest's memory image and 
uses it to rebuild the domain with a new kernel, etc.  This approach 
simplifies the changes needed to support kexec and minimises code duplication 
between Linux and the domain builder itself.

Kexec-ing the whole host is a separate issue, I think it's best addressed with 
a port of kexec to Xen itself.

Cheers,
Mark

Re: [Fastboot] [PATCH] Xen Guest Kexec

[Eric W. Biederman]

Mark Williamson <mark.williamson@cl.cam.ac.uk> writes:

> All,
>
> I'm posting to the Xen-devel list and the OSDL fastboot list.  There are a 
> number of Xen folks who'll want to look at the code.  For the fastboot folks, 
> this is mostly intended as an announcement of the port.  Please limit your 
> replies to the list(s) that they applies to, thanks :-)
>
> A number of people have expressed interest in kexec support for Xen guests.  
> In particular, it's mainly useful for purposes of implementing an in-guest 
> bootloader app and avoiding the need for dom0 to access the guest filesystem.

Is kexec at all useful for generating the initial image as well?
I believe all it would require would be defining an extra kexec type,
to load an image into the new domain.

> It's a largish patch, so I stuck it online: 
> http://www.cl.cam.ac.uk/~maw48/xenguest_kexec.patch
> Tested on i386.  I suspect it'll have build issues on x86_64 but those should 
> be easy to fix - actually the code is probably generic enough to work on 
> both.

Part of that patch is that you have another patch file showing up
in your diff.

> You'll also need a modified version of the kexec tools, which I'll post 
> details of later.
>
> Notes on the implementation:
> Kexec in a Xen guest doesn't work like on a real machine.  The guest relies on 
> outside assistance to complete the job.  The guest communicates details of 
> the kexec to Xend.  Xend extracts the data from the guest's memory image and 
> uses it to rebuild the domain with a new kernel, etc.  This approach 
> simplifies the changes needed to support kexec and minimises code duplication 
> between Linux and the domain builder itself.

How useful will you your kexec implementation handle the kexec on panic
case?  In that case we kexec a kernel at an alternative address and
then read from the memory what the previous kernel had left in it's
physical memory, and generate a crash dump of it.

> Kexec-ing the whole host is a separate issue, I think it's best addressed with 
> a port of kexec to Xen itself.

Sounds fun. :)

Eric

Re: [Fastboot] [PATCH] Xen Guest Kexec

[Mark Williamson]

> > A number of people have expressed interest in kexec support for Xen
> > guests. In particular, it's mainly useful for purposes of implementing an
> > in-guest bootloader app and avoiding the need for dom0 to access the
> > guest filesystem.
>
> Is kexec at all useful for generating the initial image as well?
> I believe all it would require would be defining an extra kexec type,
> to load an image into the new domain.

Actually, the mechanism is already practically the same: we already have a 
domain builder that could stick the kernel into memory and build a set of 
bootstrap page tables before kicking the domain into life.  Kexec support 
hooks into that by causing the domain builder to be invoked with the new 
kernel, etc.

> > It's a largish patch, so I stuck it online:
> > http://www.cl.cam.ac.uk/~maw48/xenguest_kexec.patch
> > Tested on i386.  I suspect it'll have build issues on x86_64 but those
> > should be easy to fix - actually the code is probably generic enough to
> > work on both.
>
> Part of that patch is that you have another patch file showing up
> in your diff.

Yep.  We've got a "patches" directory for things we're waiting on being pushed 
upstream.  I added the Linux generic / i386 kexec support into there for the 
time-being.  Possibly I should omit this from a "lite" version of the patch 
for easy reading, since it's a bit confusing if you're just reading...

> How useful will you your kexec implementation handle the kexec on panic
> case?  In that case we kexec a kernel at an alternative address and
> then read from the memory what the previous kernel had left in it's
> physical memory, and generate a crash dump of it.

I've thought a bit about this but there are various restrictions that stop Xen 
from building a new image into an existing domain.  Because of this, my patch 
destroys the domain and builds a completely new one with the new kernel.  
Obviously this precludes kdump for now...

The limitations in Xen could be fixed, however we already have a mechanism for 
core-dumping a domain from the "outside", so the motivation doesn't look that 
strong for now.  KDump might still be useful to people who want their 
standard distribution support for dumping to "just work", so it might still 
be worth looking into.

> > Kexec-ing the whole host is a separate issue, I think it's best addressed
> > with a port of kexec to Xen itself.
>
> Sounds fun. :)

Indeed :-)  kdump could be more useful here for debugging whole-host issues.  
The more sensible way to do this would probably be to jump into a minimal 
(Xen-free) Linux, just as normal kdump does, in the event that either Xen or 
the dom0 Linux crashes.

kexec support here would be quite interesting: machine_kexec.c and 
relocate_kernel.S will have to be in Xen itself.  Code for shutting down PCI 
devices will have to be in dom0!  :-)

A neat trick would be avoid implementing the actual "loading" process in Xen - 
just use a kernel that's been loaded by dom0's kernel.  Then Xen only has to 
contain the machine-specific code.  I've designed the mechanism so that it 
can be used for this in future.

Cheers,
Mark

Re: [PATCH] Xen Guest Kexec

[Mark Williamson]

As promised, I'm following up with further patches:

This patch:
* http://www.cl.cam.ac.uk/~maw48/kexec/xenguest_kexec_tools.patch
Adds Xen guest kexec support to the kexec tools.  You'll need this to actually 
use the kexec support.

This patch:
* http://www.cl.cam.ac.uk/~maw48/kexec/xenguest_kexec.patch
Is the patch to the Xen distribution, including all my changes *and* the 
"core" kexec changes.  Applying this to a xen-unstable repository allows you 
to build kexec-enabled guest kernels.

This patch:
* http://www.cl.cam.ac.uk/~maw48/kexec/xenguest_kexec_litepatch.patch
Is a reduced version of the above patch, omitting the "core" kexec changes, so 
that it's easier to see what I've added.

My intention is that the changes to the Xen distribution should get merged 
once we've replaced the control message interface in the control tools.  It 
should be there for the 3.0 release.

Eric: do you have any objections to my approach?  I'd ideally like to aim for 
a merge of the tools support at the same time (or before) Xen support arrives 
in mainline Linux.

Cheers,
Mark

On Thursday 07 July 2005 18:16, Mark Williamson wrote:
> All,
>
> I'm posting to the Xen-devel list and the OSDL fastboot list.  There are a
> number of Xen folks who'll want to look at the code.  For the fastboot
> folks, this is mostly intended as an announcement of the port.  Please
> limit your replies to the list(s) that they applies to, thanks :-)
>
> A number of people have expressed interest in kexec support for Xen guests.
> In particular, it's mainly useful for purposes of implementing an in-guest
> bootloader app and avoiding the need for dom0 to access the guest
> filesystem.
>
> It's a largish patch, so I stuck it online:
> http://www.cl.cam.ac.uk/~maw48/xenguest_kexec.patch
> Tested on i386.  I suspect it'll have build issues on x86_64 but those
> should be easy to fix - actually the code is probably generic enough to
> work on both.
>
> You'll also need a modified version of the kexec tools, which I'll post
> details of later.
>
> Notes on the implementation:
> Kexec in a Xen guest doesn't work like on a real machine.  The guest relies
> on outside assistance to complete the job.  The guest communicates details
> of the kexec to Xend.  Xend extracts the data from the guest's memory image
> and uses it to rebuild the domain with a new kernel, etc.  This approach
> simplifies the changes needed to support kexec and minimises code
> duplication between Linux and the domain builder itself.
>
> Kexec-ing the whole host is a separate issue, I think it's best addressed
> with a port of kexec to Xen itself.
>
> Cheers,
> Mark
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH] Xen Guest Kexec

[Horms]

Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> All,
> 
> I'm posting to the Xen-devel list and the OSDL fastboot list.  There are a 
> number of Xen folks who'll want to look at the code.  For the fastboot folks, 
> this is mostly intended as an announcement of the port.  Please limit your 
> replies to the list(s) that they applies to, thanks :-)
> 
> A number of people have expressed interest in kexec support for Xen guests.  
> In particular, it's mainly useful for purposes of implementing an in-guest 
> bootloader app and avoiding the need for dom0 to access the guest filesystem.
> 
> It's a largish patch, so I stuck it online: 
> http://www.cl.cam.ac.uk/~maw48/xenguest_kexec.patch
> Tested on i386.  I suspect it'll have build issues on x86_64 but those should 
> be easy to fix - actually the code is probably generic enough to work on 
> both.
> 
> You'll also need a modified version of the kexec tools, which I'll post 
> details of later.
> 
> Notes on the implementation:
> Kexec in a Xen guest doesn't work like on a real machine.  The guest relies on 
> outside assistance to complete the job.  The guest communicates details of 
> the kexec to Xend.  Xend extracts the data from the guest's memory image and 
> uses it to rebuild the domain with a new kernel, etc.  This approach 
> simplifies the changes needed to support kexec and minimises code duplication 
> between Linux and the domain builder itself.
> 
> Kexec-ing the whole host is a separate issue, I think it's best addressed with 
> a port of kexec to Xen itself.

Hi,

I have been looking into various crash-dump options, and with that in
mind, exploring kexec for Dom0. I'm wondering if the patch above is
still the latest, and if this is still the prefered approach
for kexec in Dom0.

-- 
Horms

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

> I have been looking into various crash-dump options, and with that in
> mind, exploring kexec for Dom0. I'm wondering if the patch above is
> still the latest, and if this is still the prefered approach
> for kexec in Dom0.

For crash-dumping you'll can simply ask xend to write out an dump on
domain crash, then inspect it using gdbserver-xen ;)

For kexec I'm looking into getting kexec work right now.  For domU's it
shouldn't be that hard I think.  dom0 likely is much harder given how
xen and dom0 work hand-in-hand to drive the hardware ...

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Mark Williamson]

> For crash-dumping you'll can simply ask xend to write out an dump on
> domain crash, then inspect it using gdbserver-xen ;)
>
> For kexec I'm looking into getting kexec work right now.  For domU's it
> shouldn't be that hard I think.  dom0 likely is much harder given how
> xen and dom0 work hand-in-hand to drive the hardware ...

How are you planning to do domU kexec?

For domUs the major problem I found was:
a) the existing kexec code doesn't understand pseudophysical memory
b) we had no way (at the time) of resetting virtual devices - disconnecting 
cleanly and then reconnecting again
c) creating a start-of-day environment for the reboot kernel duplicates a load 
of code that's already in the domain builder

Because of this, I used a technique I called "assisted kexec" where the guest 
would write out a descriptor which could be used by Xend to (safely, without 
trusting the guest) extract the reboot kernel from guest memory, and rebuild 
the domain with that kernel, using the standard domain builder.  This worked 
quite well actually resulted in less code overall.

For dom0 kexec, I thought the best approach would be to port the existing 
Linux kexec machinery to Xen (should be quite straightforward - just the part 
which copies the kernel image to the correct place, switches off paging, 
jumps into the new kernel).  Then the dom0 kexec code would make a hypercall 
after closing its devices, rather than trying to execute that code itself.

What do you think?

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

Mark Williamson wrote:
> How are you planning to do domU kexec?

As close as possible to normal i386 kexec ...

> For domUs the major problem I found was:
> a) the existing kexec code doesn't understand pseudophysical memory
> b) we had no way (at the time) of resetting virtual devices - disconnecting 
> cleanly and then reconnecting again
> c) creating a start-of-day environment for the reboot kernel duplicates a load 
> of code that's already in the domain builder

Dis- and reconnecting should be ok by now I guess.  I expect the paging
setup being the most tricky part:  First because the pseudophysical
memory (probably not a major issue though).  Second because unlike i386
kexec we'll have to run with paging enabled all the time ...

My plan is to first make xenlinux kernels work with correct ELF headers
(patches on the list last days ;), then make kexec userspace work
(unmodified if possible), last step the in-kernel stuff which performs
the actual kexec ...

> For dom0 kexec, I thought the best approach would be to port the existing 
> Linux kexec machinery to Xen (should be quite straightforward - just the part 
> which copies the kernel image to the correct place, switches off paging, 
> jumps into the new kernel).  Then the dom0 kexec code would make a hypercall 
> after closing its devices, rather than trying to execute that code itself.

Right now linux kexec depends on the new kernel having a different
physical (and virtual) start address, so taking the very same approach
likely doesn't work.

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Mark Williamson]

> As close as possible to normal i386 kexec ...
>
> Dis- and reconnecting should be ok by now I guess.  I expect the paging
> setup being the most tricky part:  First because the pseudophysical
> memory (probably not a major issue though).  Second because unlike i386
> kexec we'll have to run with paging enabled all the time ...

I doubt having paging enabled would be too painful.  i386 kexec disables 
paging right at the end of the process so that the new kernel will have a 
sensible start-of-day.  We'd just need start-of-day to contain bootstrap 
pagetables, same as for normal Linux.  Ideally you'd need to find a slot in 
the bootstrap tables for the trampoline code to live, if you take that 
approach.

You've got a load of other things to worry about in this approach, like 
un-type-pinning all pages you own, etc.

The generic kexec code doesn't understand phys vs machine memory, IIRC, so you 
may need to worry about it mis-allocating your trampoline page (this is an 
issue because you need to identity map the trampoline page later on in the 
process).

> My plan is to first make xenlinux kernels work with correct ELF headers
> (patches on the list last days ;), then make kexec userspace work
> (unmodified if possible), last step the in-kernel stuff which performs
> the actual kexec ...

I patched kexec userspace, but partly that was so that it wouldn't complain 
about the Xen-format image files.  I also seem to remember I built a 
descriptor that described the size of the kernel, ramdisk, etc for the 
benefit of the kexec code but this probably wasn't strictly necessary.  I 
might even have removed that in the end...  Can't remember :-)

> > For dom0 kexec, I thought the best approach would be to port the existing
> > Linux kexec machinery to Xen (should be quite straightforward - just the
> > part which copies the kernel image to the correct place, switches off
> > paging, jumps into the new kernel).  Then the dom0 kexec code would make
> > a hypercall after closing its devices, rather than trying to execute that
> > code itself.
>
> Right now linux kexec depends on the new kernel having a different
> physical (and virtual) start address, so taking the very same approach
> likely doesn't work.

I'm not convinced: the reboot kernel doesn't need to be any different from the 
standard kernel *unless* you're running kdump (when the kernel will need to 
live in a different place so that it doesn't stomp on the main kernel - not a 
limitation of kexec).  Or am I misunderstanding what you meant?

Anyhow, it'd be nice to have kexec working.  But I'd still suggest that you 
take a quick look at my patch and consider implementing it as a platform 
service, rather than implementing all the low-level gunk.  It really is much 
less code that way.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

Mark Williamson wrote:
>> As close as possible to normal i386 kexec ...
>>
>> Dis- and reconnecting should be ok by now I guess.  I expect the paging
>> setup being the most tricky part:  First because the pseudophysical
>> memory (probably not a major issue though).  Second because unlike i386
>> kexec we'll have to run with paging enabled all the time ...
> 
> I doubt having paging enabled would be too painful.  i386 kexec disables 
> paging right at the end of the process so that the new kernel will have a 
> sensible start-of-day.  We'd just need start-of-day to contain bootstrap 
> pagetables, same as for normal Linux.  Ideally you'd need to find a slot in 
> the bootstrap tables for the trampoline code to live, if you take that 
> approach.

x86-64 has paging enabled while trampoline is running too, so I can get
some ideas there ;)

> You've got a load of other things to worry about in this approach, like 
> un-type-pinning all pages you own, etc.

Yep, that is a problem.  What pages are pinned (other than pgd)?  I've
seen plenty of pages with PG_pinned set, but can't figure easily what
pages that are ...

Also switching page tables seems to be not so easy.  Is it possible to
switch atomically to a new, completely independant page table tree?
i.e. old tree is valid (of cource), new tree is too, but the pages of
the old tree are not mapped read-only in the new tree (and visa versa).

> The generic kexec code doesn't understand phys vs machine memory, IIRC, so you 
> may need to worry about it mis-allocating your trampoline page (this is an 
> issue because you need to identity map the trampoline page later on in the 
> process).

Not a big issue if paging is enabled anyway, we can use a identity map
then (virtual == physical, not virtual == machine), so kexec doesn't
even notice outside the arch-specific code.

>> Right now linux kexec depends on the new kernel having a different
>> physical (and virtual) start address, so taking the very same approach
>> likely doesn't work.
> 
> I'm not convinced: the reboot kernel doesn't need to be any different from the 
> standard kernel *unless* you're running kdump (when the kernel will need to 
> live in a different place so that it doesn't stomp on the main kernel - not a 
> limitation of kexec).  Or am I misunderstanding what you meant?

I think it's needed in both cases, at least I had problems making normal
kexec work (without xen) when both kernels had the same physical
address.  Might have been a simple bug though.

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Mark Williamson]

> > You've got a load of other things to worry about in this approach, like
> > un-type-pinning all pages you own, etc.
>
> Yep, that is a problem.  What pages are pinned (other than pgd)?  I've
> seen plenty of pages with PG_pinned set, but can't figure easily what
> pages that are ...

When I was looking at this approach I was intending to retreive the pageframe 
info from Xen and just run unpin on anything pinned, according to its type.

> > The generic kexec code doesn't understand phys vs machine memory, IIRC,
> > so you may need to worry about it mis-allocating your trampoline page
> > (this is an issue because you need to identity map the trampoline page
> > later on in the process).
>
> Not a big issue if paging is enabled anyway, we can use a identity map
> then (virtual == physical, not virtual == machine), so kexec doesn't
> even notice outside the arch-specific code.

Actually I was talking rubbish there anyhow; of course if you don't disable 
paging the identity mapping isn't a problem.

> >> Right now linux kexec depends on the new kernel having a different
> >> physical (and virtual) start address, so taking the very same approach
> >> likely doesn't work.
<snip>
> I think it's needed in both cases, at least I had problems making normal
> kexec work (without xen) when both kernels had the same physical
> address.  Might have been a simple bug though.

When did you try that?  I've not tried it outside Xen, but it didn't look like 
a requirement from my reading of the code.

A really big chunk of the generic kexec code is there to make sure the new 
kernel is not loaded in an area that conflicts with its destination, much of 
the trampoline is there to enable the new kernel to be copied on top of the 
old one.  Address conflicts should only be an issue if you want the second 
kernel to be a dump kernel, in which case it needs to run in-place after 
loading.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

> When I was looking at this approach I was intending to retreive the pageframe 
> info from Xen and just run unpin on anything pinned, according to its type.

Is that possible?  As far I know the frametable where that info is
stored isn't readable by domains ...

> A really big chunk of the generic kexec code is there to make sure the new 
> kernel is not loaded in an area that conflicts with its destination, much of 
> the trampoline is there to enable the new kernel to be copied on top of the 
> old one.  Address conflicts should only be an issue if you want the second 
> kernel to be a dump kernel, in which case it needs to run in-place after 
> loading.

Yep, sure, all the trampoline stuff is effectively for that.
Retested, works.  Not sure what I did wrong last time ...

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Mark Williamson]

> I have been looking into various crash-dump options, and with that in
> mind, exploring kexec for Dom0. I'm wondering if the patch above is
> still the latest, and if this is still the prefered approach
> for kexec in Dom0.

My patch was out-of-tree so it's rotted.  I've been waiting for various other 
things to get sorted before it's worth updating it again (updating to a 
kernel that actually has the kexec system call, one or two other patches).

The patch wouldn't allow kexec for dom0 anyhow.  Enabling kdump for dom0 would 
be useful.  I think there was somebody considering looking at this in the 
future...

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Re: [PATCH] Xen Guest Kexec

[Horms]

On Thu, Feb 23, 2006 at 11:36:36AM +0000, Mark Williamson wrote:
> > I have been looking into various crash-dump options, and with that in
> > mind, exploring kexec for Dom0. I'm wondering if the patch above is
> > still the latest, and if this is still the prefered approach
> > for kexec in Dom0.
> 
> My patch was out-of-tree so it's rotted.  I've been waiting for various other 
> things to get sorted before it's worth updating it again (updating to a 
> kernel that actually has the kexec system call, one or two other patches).

Could you be a little more specific about that. I am pretty interested
in getting this working. As far as I understand Xen is now running
on 2.6.16-rc4 which does have kexec. I'm happy to put some cycles into
updating the patch, though I must confess I am quite interested in
Gerd's approach which is closer to what I was originally thinking.

> The patch wouldn't allow kexec for dom0 anyhow.  Enabling kdump for dom0 would 
> be useful.  I think there was somebody considering looking at this in the 
> future...

Understood. I was only talking of DomU with regards to the assisted
kexec patch that you sent earlier. I am also intersted in getting
kexec working in Dom0, however I agree with Gerd's statements elsewhere
in this thread that it can be acchived by porting kexec to Xen (or as
I understand it given the current state of kexec, teaching kexec a few
things about Xen in the prevailing architecture).

I'm actually quite interested in getting both kexec and kdump working
on both DomU and Dom0. I understand that for the most port there are
really four different problems there. I'd really like to help out
with existing efforts to help get all or any of them working.

For now I'm going to poke around with Gerd's idea of asking xend to
do system dumps for crash analysis of DomU, as that seems to be an area
of little activity at this time. 

-- 
Horms

Re: Re: [PATCH] Xen Guest Kexec

[Mark Williamson]

> > My patch was out-of-tree so it's rotted.  I've been waiting for various
> > other things to get sorted before it's worth updating it again (updating
> > to a kernel that actually has the kexec system call, one or two other
> > patches).
>
> Could you be a little more specific about that. I am pretty interested
> in getting this working. As far as I understand Xen is now running
> on 2.6.16-rc4 which does have kexec. I'm happy to put some cycles into
> updating the patch, though I must confess I am quite interested in
> Gerd's approach which is closer to what I was originally thinking.

My original intention was also like Gerd's approach, I just decided it was 
simpler overall to make it into a platform service - we depend on daemons in 
other domains *anyhow* in order to talk to devices, make reboot work, etc.

I was waiting for the build-from-memory patches to maybe be accepted (which 
would avoid updating that part of my patch) but I haven't had a chance to 
look over them yet.

> Understood. I was only talking of DomU with regards to the assisted
> kexec patch that you sent earlier. I am also intersted in getting
> kexec working in Dom0, however I agree with Gerd's statements elsewhere
> in this thread that it can be acchived by porting kexec to Xen (or as
> I understand it given the current state of kexec, teaching kexec a few
> things about Xen in the prevailing architecture).

Yes.  This shouldn't be so hard, except possible from the PoV of actually 
allocating the memory correctly.

> I'm actually quite interested in getting both kexec and kdump working
> on both DomU and Dom0. I understand that for the most port there are
> really four different problems there. I'd really like to help out
> with existing efforts to help get all or any of them working.
>
> For now I'm going to poke around with Gerd's idea of asking xend to
> do system dumps for crash analysis of DomU, as that seems to be an area
> of little activity at this time.

That should be pretty much as good as kdump, except that the image obviously 
won't appear within the guests filesystem, and it won't integrate with domU 
distros that want to use kdump.

Cheers,
Mark


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

RE: Re: [PATCH] Xen Guest Kexec

[Ian Pratt]

> Also switching page tables seems to be not so easy.  Is it 
> possible to switch atomically to a new, completely 
> independant page table tree?
> i.e. old tree is valid (of cource), new tree is too, but the 
> pages of the old tree are not mapped read-only in the new 
> tree (and visa versa).

If the pagetable you're running on isn't pinned the refcounts will drop
to zero when you switch away from it, hence its only necessary for the
destination pagetable to be internally consistent.

Ian

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 26 Feb 2006, at 22:49, Ian Pratt wrote:

>> Also switching page tables seems to be not so easy.  Is it
>> possible to switch atomically to a new, completely
>> independant page table tree?
>> i.e. old tree is valid (of cource), new tree is too, but the
>> pages of the old tree are not mapped read-only in the new
>> tree (and visa versa).
>
> If the pagetable you're running on isn't pinned the refcounts will drop
> to zero when you switch away from it, hence its only necessary for the
> destination pagetable to be internally consistent.

We run on the old pagetables while we validate the new ones, so 
currently they need to map each other read-only. We could have a slow 
path in the pagetable switch code that switches to the idle pagetable, 
drops the old pagetable, checks the new pagetable then switches again. 
On failure it would have to crash the domain dealing with the fact that 
this VCPU no longer has a valid 'cr3'.

It's a bit of a pain, but it might simplify mm/init.c. I'm not sure by 
how much though.

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

  Hi,

> It's a bit of a pain, but it might simplify mm/init.c. I'm not sure by
> how much though.

The kexec trampoline would be *much* easier then, every bit of state
information I have to carry around is a pain.  mm/init.c hopefully
doesn't need any tweaks for kexec.

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 27 Feb 2006, at 15:34, Gerd Hoffmann wrote:

>> It's a bit of a pain, but it might simplify mm/init.c. I'm not sure by
>> how much though.
>
> The kexec trampoline would be *much* easier then, every bit of state
> information I have to carry around is a pain.  mm/init.c hopefully
> doesn't need any tweaks for kexec.

Okay, done. Changeset 8999 (5adaa69...).

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 321 bytes --]

Keir Fraser wrote:
> 
> Okay, done. Changeset 8999 (5adaa69...).

Thanks.  Unfortunaly I can take down Xen now by trying to use that code
path, see the log below.

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

[-- Attachment #2: oops --]
[-- Type: text/plain, Size: 56110 bytes --]

(XEN) DOM2: (file=mm.c, line=1510) Bad type (saw 33020001 != exp e0000000) for mfn 1853e (pfn 0)
(XEN) DOM2: (file=mm.c, line=478) Error getting mfn 1853e (pfn 0) from L1 entry 1853e063 for dom2
(XEN) DOM2: (file=mm.c, line=694) Failure in alloc_l1_table: entry 0
(XEN) DOM2: (file=mm.c, line=1551) Error while validating mfn 19ee6 (pfn 3332) for type 20000000: caf=80000003 taf=20000001
(XEN) DOM2: (file=mm.c, line=402) Attempt to create linear p.t. with write perms
(XEN) DOM2: (file=mm.c, line=855) Failure in alloc_l2_table: entry 0
(XEN) DOM2: (file=mm.c, line=1551) Error while validating mfn 19bb9 (pfn 365f) for type 40000000: caf=80000003 taf=40000001
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff13383c>] free_l2_table+0x53/0x78
(XEN) EFLAGS: 00010286   CONTEXT: hypervisor
(XEN) eax: 0001867e   ebx: 00019bb9   ecx: 0000000f   edx: fed2f000
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7c88   esp: ffbf7c70
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7c70:
(XEN)    0001867e 0001867e ffbf7ca8 0001867e fed2f000 00000000 ffbf7cb8 ff1346ab 
(XEN)    fc649bd0 00000000 ffbf7cb4 ff1be736 00000000 40000000 ffbf7d58 ff13502e 
(XEN)    ffbe6080 00000002 ffbf7d78 ff13493d fc649bd0 50000001 80000003 40000001 
(XEN)    ff132bbd 00000004 00000000 f0000000 00000004 50000001 32ff0001 00000000 
(XEN)    50000001 00000000 50000000 50000001 50000001 50000001 80000002 f0000000 
(XEN)    80000002 ffbf7fb4 ffbf7d88 00000282 00019bb9 00000003 00000004 40000001 
(XEN)    50000001 fc649be0 50000001 00000203 00000203 00000204 00000282 00000203 
(XEN)    fc669960 80000003 ffbf7d88 ff132c7c fc669958 40000000 ffbe6594 80000003 
(XEN)    80000004 00000004 ffbf7d88 ff13201c fc649bd0 00000203 ffbf7dc8 ff135311 
(XEN)    fc649bd0 40000000 ffbe6080 ffbf7dbc 00000002 fed2d000 0001aee6 ffbf7ddc 
(XEN)    0001aee6 00000000 ffbb3080 ffbe6080 00000000 0001867e ffbf7f58 ff1425df 
(XEN)    00019bb9 ffbf7fb4 00000000 12ca4ab7 0000010d ffbe6080 00000000 ffbf7eac 
(XEN)    ff13493d fc686590 ffbf7fb4 ffbf7e6c 0000001c 0e532f05 0000010d ffbb3080 
(XEN)    00000004 000001f8 0000000f 00000000 000001f8 000001f8 000001f9 00000282 
(XEN)    000001f8 47ff0001 57ff0001 0000000f 12ca4ab7 0000010d ffbf7ec4 0e5ad025 
(XEN)    00000001 00000000 ffffffff ffbf7e6c ff13c951 00000003 00019bb9 ffbf7e7c 
(XEN)    ff13c592 ffbb3080 ffbf7fbc c010eca8 00000286 000203da 000008fc 00000000 
(XEN)    00000004 000008fc 00000286 ffbf7eac c010ec0f 00000000 c010eca5 ffbf7ecc 
(XEN)    ff13cc31 00000001 000000fc ffbf7ecc ff1075a7 00000002 ffbb3080 ffbf7eec 
(XEN)    ff107589 00000001 ff1fc1e8 00079a6e 00000000 00000001 00000001 ffbf7f0c 
(XEN) Xen call trace:
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed2f000:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed30fec   ebx: ffbb3080   ecx: 00000010   edx: ffbb74c0
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7c08   esp: ffbf7be0
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7be0:
(XEN)    000001b4 fed2f000 e0107c08 e010e010 00000000 8005003b 000001b4 fed30000 
(XEN)    00000003 c0731600 ffbf7c38 ff140fea fed2f000 ffbf7c44 0000012f 0000000a 
(XEN)    00000053 fed2f000 00000000 00000000 00000370 ffbb3080 004083a7 ff147d12 
(XEN)    ffbf7c44 00019bb9 0000000f fed2f000 00000003 c0731600 ffbf7c88 0001867e 
(XEN)    000e0000 ff13383c 0000e008 00010286 0001867e 0001867e ffbf7ca8 0001867e 
(XEN)    fed2f000 00000000 ffbf7cb8 ff1346ab fc649bd0 00000000 ffbf7cb4 ff1be736 
(XEN)    00000000 40000000 ffbf7d58 ff13502e ffbe6080 00000002 ffbf7d78 ff13493d 
(XEN)    fc649bd0 50000001 80000003 40000001 ff132bbd 00000004 00000000 f0000000 
(XEN)    00000004 50000001 32ff0001 00000000 50000001 00000000 50000000 50000001 
(XEN)    50000001 50000001 80000002 f0000000 80000002 ffbf7fb4 ffbf7d88 00000282 
(XEN)    00019bb9 00000003 00000004 40000001 50000001 fc649be0 50000001 00000203 
(XEN)    00000203 00000204 00000282 00000203 fc669960 80000003 ffbf7d88 ff132c7c 
(XEN)    fc669958 40000000 ffbe6594 80000003 80000004 00000004 ffbf7d88 ff13201c 
(XEN)    fc649bd0 00000203 ffbf7dc8 ff135311 fc649bd0 40000000 ffbe6080 ffbf7dbc 
(XEN)    00000002 fed2d000 0001aee6 ffbf7ddc 0001aee6 00000000 ffbb3080 ffbe6080 
(XEN)    00000000 0001867e ffbf7f58 ff1425df 00019bb9 ffbf7fb4 00000000 12ca4ab7 
(XEN)    0000010d ffbe6080 00000000 ffbf7eac ff13493d fc686590 ffbf7fb4 ffbf7e6c 
(XEN)    0000001c 0e532f05 0000010d ffbb3080 00000004 000001f8 0000000f 00000000 
(XEN)    000001f8 000001f8 000001f9 00000282 000001f8 47ff0001 57ff0001 0000000f 
(XEN)    12ca4ab7 0000010d ffbf7ec4 0e5ad025 00000001 00000000 ffffffff ffbf7e6c 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed30fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed31fec   ebx: ffbb3080   ecx: 00000011   edx: ffbb74c4
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7b78   esp: ffbf7b50
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7b50:
(XEN)    000001b4 fed30fec e0107b78 e010e010 00000000 8005003b 000001b4 fed31000 
(XEN)    00000003 c0731600 ffbf7ba8 ff140fea fed30fec ffbf7bb4 00000130 ff11f59a 
(XEN)    0000000a fed30fec 00000000 00000000 00000400 ffbb3080 00408437 ff147d12 
(XEN)    ffbf7bb4 ffbb3080 00000010 ffbb74c0 00000003 c0731600 ffbf7c08 fed30fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed2f000 e0107c08 e010e010 
(XEN)    00000000 8005003b 000001b4 fed30000 00000003 c0731600 ffbf7c38 ff140fea 
(XEN)    fed2f000 ffbf7c44 0000012f 0000000a 00000053 fed2f000 00000000 00000000 
(XEN)    00000370 ffbb3080 004083a7 ff147d12 ffbf7c44 00019bb9 0000000f fed2f000 
(XEN)    00000003 c0731600 ffbf7c88 0001867e 000e0000 ff13383c 0000e008 00010286 
(XEN)    0001867e 0001867e ffbf7ca8 0001867e fed2f000 00000000 ffbf7cb8 ff1346ab 
(XEN)    fc649bd0 00000000 ffbf7cb4 ff1be736 00000000 40000000 ffbf7d58 ff13502e 
(XEN)    ffbe6080 00000002 ffbf7d78 ff13493d fc649bd0 50000001 80000003 40000001 
(XEN)    ff132bbd 00000004 00000000 f0000000 00000004 50000001 32ff0001 00000000 
(XEN)    50000001 00000000 50000000 50000001 50000001 50000001 80000002 f0000000 
(XEN)    80000002 ffbf7fb4 ffbf7d88 00000282 00019bb9 00000003 00000004 40000001 
(XEN)    50000001 fc649be0 50000001 00000203 00000203 00000204 00000282 00000203 
(XEN)    fc669960 80000003 ffbf7d88 ff132c7c fc669958 40000000 ffbe6594 80000003 
(XEN)    80000004 00000004 ffbf7d88 ff13201c fc649bd0 00000203 ffbf7dc8 ff135311 
(XEN)    fc649bd0 40000000 ffbe6080 ffbf7dbc 00000002 fed2d000 0001aee6 ffbf7ddc 
(XEN)    0001aee6 00000000 ffbb3080 ffbe6080 00000000 0001867e ffbf7f58 ff1425df 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed31fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed32fec   ebx: ffbb3080   ecx: 00000012   edx: ffbb74c8
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7ae8   esp: ffbf7ac0
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7ac0:
(XEN)    000001b4 fed31fec e0107ae8 e010e010 00000000 8005003b 000001b4 fed32000 
(XEN)    00000003 c0731600 ffbf7b18 ff140fea fed31fec ffbf7b24 00000131 ff11f59a 
(XEN)    0000000a fed31fec 00000000 00000000 00000490 ffbb3080 004084c7 ff147d12 
(XEN)    ffbf7b24 ffbb3080 00000011 ffbb74c4 00000003 c0731600 ffbf7b78 fed31fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed30fec e0107b78 e010e010 
(XEN)    00000000 8005003b 000001b4 fed31000 00000003 c0731600 ffbf7ba8 ff140fea 
(XEN)    fed30fec ffbf7bb4 00000130 ff11f59a 0000000a fed30fec 00000000 00000000 
(XEN)    00000400 ffbb3080 00408437 ff147d12 ffbf7bb4 ffbb3080 00000010 ffbb74c0 
(XEN)    00000003 c0731600 ffbf7c08 fed30fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed2f000 e0107c08 e010e010 00000000 8005003b 000001b4 fed30000 
(XEN)    00000003 c0731600 ffbf7c38 ff140fea fed2f000 ffbf7c44 0000012f 0000000a 
(XEN)    00000053 fed2f000 00000000 00000000 00000370 ffbb3080 004083a7 ff147d12 
(XEN)    ffbf7c44 00019bb9 0000000f fed2f000 00000003 c0731600 ffbf7c88 0001867e 
(XEN)    000e0000 ff13383c 0000e008 00010286 0001867e 0001867e ffbf7ca8 0001867e 
(XEN)    fed2f000 00000000 ffbf7cb8 ff1346ab fc649bd0 00000000 ffbf7cb4 ff1be736 
(XEN)    00000000 40000000 ffbf7d58 ff13502e ffbe6080 00000002 ffbf7d78 ff13493d 
(XEN)    fc649bd0 50000001 80000003 40000001 ff132bbd 00000004 00000000 f0000000 
(XEN)    00000004 50000001 32ff0001 00000000 50000001 00000000 50000000 50000001 
(XEN)    50000001 50000001 80000002 f0000000 80000002 ffbf7fb4 ffbf7d88 00000282 
(XEN)    00019bb9 00000003 00000004 40000001 50000001 fc649be0 50000001 00000203 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed32fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed33fec   ebx: ffbb3080   ecx: 00000013   edx: ffbb74cc
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7a58   esp: ffbf7a30
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7a30:
(XEN)    000001b4 fed32fec e0107a58 e010e010 00000000 8005003b 000001b4 fed33000 
(XEN)    00000003 c0731600 ffbf7a88 ff140fea fed32fec ffbf7a94 00000132 ff11f59a 
(XEN)    0000000a fed32fec 00000000 00000000 00000520 ffbb3080 00408557 ff147d12 
(XEN)    ffbf7a94 ffbb3080 00000012 ffbb74c8 00000003 c0731600 ffbf7ae8 fed32fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed31fec e0107ae8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed32000 00000003 c0731600 ffbf7b18 ff140fea 
(XEN)    fed31fec ffbf7b24 00000131 ff11f59a 0000000a fed31fec 00000000 00000000 
(XEN)    00000490 ffbb3080 004084c7 ff147d12 ffbf7b24 ffbb3080 00000011 ffbb74c4 
(XEN)    00000003 c0731600 ffbf7b78 fed31fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed30fec e0107b78 e010e010 00000000 8005003b 000001b4 fed31000 
(XEN)    00000003 c0731600 ffbf7ba8 ff140fea fed30fec ffbf7bb4 00000130 ff11f59a 
(XEN)    0000000a fed30fec 00000000 00000000 00000400 ffbb3080 00408437 ff147d12 
(XEN)    ffbf7bb4 ffbb3080 00000010 ffbb74c0 00000003 c0731600 ffbf7c08 fed30fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed2f000 e0107c08 e010e010 
(XEN)    00000000 8005003b 000001b4 fed30000 00000003 c0731600 ffbf7c38 ff140fea 
(XEN)    fed2f000 ffbf7c44 0000012f 0000000a 00000053 fed2f000 00000000 00000000 
(XEN)    00000370 ffbb3080 004083a7 ff147d12 ffbf7c44 00019bb9 0000000f fed2f000 
(XEN)    00000003 c0731600 ffbf7c88 0001867e 000e0000 ff13383c 0000e008 00010286 
(XEN)    0001867e 0001867e ffbf7ca8 0001867e fed2f000 00000000 ffbf7cb8 ff1346ab 
(XEN)    fc649bd0 00000000 ffbf7cb4 ff1be736 00000000 40000000 ffbf7d58 ff13502e 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed33fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed34fec   ebx: ffbb3080   ecx: 00000014   edx: ffbb74d0
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf79c8   esp: ffbf79a0
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf79a0:
(XEN)    000001b4 fed33fec e01079c8 e010e010 00000000 8005003b 000001b4 fed34000 
(XEN)    00000003 c0731600 ffbf79f8 ff140fea fed33fec ffbf7a04 00000133 ff11f59a 
(XEN)    0000000a fed33fec 00000000 00000000 000005b0 ffbb3080 004085e7 ff147d12 
(XEN)    ffbf7a04 ffbb3080 00000013 ffbb74cc 00000003 c0731600 ffbf7a58 fed33fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed32fec e0107a58 e010e010 
(XEN)    00000000 8005003b 000001b4 fed33000 00000003 c0731600 ffbf7a88 ff140fea 
(XEN)    fed32fec ffbf7a94 00000132 ff11f59a 0000000a fed32fec 00000000 00000000 
(XEN)    00000520 ffbb3080 00408557 ff147d12 ffbf7a94 ffbb3080 00000012 ffbb74c8 
(XEN)    00000003 c0731600 ffbf7ae8 fed32fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed31fec e0107ae8 e010e010 00000000 8005003b 000001b4 fed32000 
(XEN)    00000003 c0731600 ffbf7b18 ff140fea fed31fec ffbf7b24 00000131 ff11f59a 
(XEN)    0000000a fed31fec 00000000 00000000 00000490 ffbb3080 004084c7 ff147d12 
(XEN)    ffbf7b24 ffbb3080 00000011 ffbb74c4 00000003 c0731600 ffbf7b78 fed31fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed30fec e0107b78 e010e010 
(XEN)    00000000 8005003b 000001b4 fed31000 00000003 c0731600 ffbf7ba8 ff140fea 
(XEN)    fed30fec ffbf7bb4 00000130 ff11f59a 0000000a fed30fec 00000000 00000000 
(XEN)    00000400 ffbb3080 00408437 ff147d12 ffbf7bb4 ffbb3080 00000010 ffbb74c0 
(XEN)    00000003 c0731600 ffbf7c08 fed30fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed2f000 e0107c08 e010e010 00000000 8005003b 000001b4 fed30000 
(XEN)    00000003 c0731600 ffbf7c38 ff140fea fed2f000 ffbf7c44 0000012f 0000000a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed34fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed35fec   ebx: ffbb3080   ecx: 00000015   edx: ffbb74d4
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7938   esp: ffbf7910
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7910:
(XEN)    000001b4 fed34fec e0107938 e010e010 00000000 8005003b 000001b4 fed35000 
(XEN)    00000003 c0731600 ffbf7968 ff140fea fed34fec ffbf7974 00000134 ff11f59a 
(XEN)    0000000a fed34fec 00000000 00000000 00000640 ffbb3080 00408677 ff147d12 
(XEN)    ffbf7974 ffbb3080 00000014 ffbb74d0 00000003 c0731600 ffbf79c8 fed34fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed33fec e01079c8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed34000 00000003 c0731600 ffbf79f8 ff140fea 
(XEN)    fed33fec ffbf7a04 00000133 ff11f59a 0000000a fed33fec 00000000 00000000 
(XEN)    000005b0 ffbb3080 004085e7 ff147d12 ffbf7a04 ffbb3080 00000013 ffbb74cc 
(XEN)    00000003 c0731600 ffbf7a58 fed33fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed32fec e0107a58 e010e010 00000000 8005003b 000001b4 fed33000 
(XEN)    00000003 c0731600 ffbf7a88 ff140fea fed32fec ffbf7a94 00000132 ff11f59a 
(XEN)    0000000a fed32fec 00000000 00000000 00000520 ffbb3080 00408557 ff147d12 
(XEN)    ffbf7a94 ffbb3080 00000012 ffbb74c8 00000003 c0731600 ffbf7ae8 fed32fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed31fec e0107ae8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed32000 00000003 c0731600 ffbf7b18 ff140fea 
(XEN)    fed31fec ffbf7b24 00000131 ff11f59a 0000000a fed31fec 00000000 00000000 
(XEN)    00000490 ffbb3080 004084c7 ff147d12 ffbf7b24 ffbb3080 00000011 ffbb74c4 
(XEN)    00000003 c0731600 ffbf7b78 fed31fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed30fec e0107b78 e010e010 00000000 8005003b 000001b4 fed31000 
(XEN)    00000003 c0731600 ffbf7ba8 ff140fea fed30fec ffbf7bb4 00000130 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed35fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed36fec   ebx: ffbb3080   ecx: 00000016   edx: ffbb74d8
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf78a8   esp: ffbf7880
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7880:
(XEN)    000001b4 fed35fec e01078a8 e010e010 00000000 8005003b 000001b4 fed36000 
(XEN)    00000003 c0731600 ffbf78d8 ff140fea fed35fec ffbf78e4 00000135 ff11f59a 
(XEN)    0000000a fed35fec 00000000 00000000 000006d0 ffbb3080 00408707 ff147d12 
(XEN)    ffbf78e4 ffbb3080 00000015 ffbb74d4 00000003 c0731600 ffbf7938 fed35fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed34fec e0107938 e010e010 
(XEN)    00000000 8005003b 000001b4 fed35000 00000003 c0731600 ffbf7968 ff140fea 
(XEN)    fed34fec ffbf7974 00000134 ff11f59a 0000000a fed34fec 00000000 00000000 
(XEN)    00000640 ffbb3080 00408677 ff147d12 ffbf7974 ffbb3080 00000014 ffbb74d0 
(XEN)    00000003 c0731600 ffbf79c8 fed34fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed33fec e01079c8 e010e010 00000000 8005003b 000001b4 fed34000 
(XEN)    00000003 c0731600 ffbf79f8 ff140fea fed33fec ffbf7a04 00000133 ff11f59a 
(XEN)    0000000a fed33fec 00000000 00000000 000005b0 ffbb3080 004085e7 ff147d12 
(XEN)    ffbf7a04 ffbb3080 00000013 ffbb74cc 00000003 c0731600 ffbf7a58 fed33fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed32fec e0107a58 e010e010 
(XEN)    00000000 8005003b 000001b4 fed33000 00000003 c0731600 ffbf7a88 ff140fea 
(XEN)    fed32fec ffbf7a94 00000132 ff11f59a 0000000a fed32fec 00000000 00000000 
(XEN)    00000520 ffbb3080 00408557 ff147d12 ffbf7a94 ffbb3080 00000012 ffbb74c8 
(XEN)    00000003 c0731600 ffbf7ae8 fed32fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed31fec e0107ae8 e010e010 00000000 8005003b 000001b4 fed32000 
(XEN)    00000003 c0731600 ffbf7b18 ff140fea fed31fec ffbf7b24 00000131 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed36fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed37fec   ebx: ffbb3080   ecx: 00000017   edx: ffbb74dc
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7818   esp: ffbf77f0
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf77f0:
(XEN)    000001b4 fed36fec e0107818 e010e010 00000000 8005003b 000001b4 fed37000 
(XEN)    00000003 c0731600 ffbf7848 ff140fea fed36fec ffbf7854 00000136 ff11f59a 
(XEN)    0000000a fed36fec 00000000 00000000 00000760 ffbb3080 00408797 ff147d12 
(XEN)    ffbf7854 ffbb3080 00000016 ffbb74d8 00000003 c0731600 ffbf78a8 fed36fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed35fec e01078a8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed36000 00000003 c0731600 ffbf78d8 ff140fea 
(XEN)    fed35fec ffbf78e4 00000135 ff11f59a 0000000a fed35fec 00000000 00000000 
(XEN)    000006d0 ffbb3080 00408707 ff147d12 ffbf78e4 ffbb3080 00000015 ffbb74d4 
(XEN)    00000003 c0731600 ffbf7938 fed35fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed34fec e0107938 e010e010 00000000 8005003b 000001b4 fed35000 
(XEN)    00000003 c0731600 ffbf7968 ff140fea fed34fec ffbf7974 00000134 ff11f59a 
(XEN)    0000000a fed34fec 00000000 00000000 00000640 ffbb3080 00408677 ff147d12 
(XEN)    ffbf7974 ffbb3080 00000014 ffbb74d0 00000003 c0731600 ffbf79c8 fed34fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed33fec e01079c8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed34000 00000003 c0731600 ffbf79f8 ff140fea 
(XEN)    fed33fec ffbf7a04 00000133 ff11f59a 0000000a fed33fec 00000000 00000000 
(XEN)    000005b0 ffbb3080 004085e7 ff147d12 ffbf7a04 ffbb3080 00000013 ffbb74cc 
(XEN)    00000003 c0731600 ffbf7a58 fed33fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed32fec e0107a58 e010e010 00000000 8005003b 000001b4 fed33000 
(XEN)    00000003 c0731600 ffbf7a88 ff140fea fed32fec ffbf7a94 00000132 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed37fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed38fec   ebx: ffbb3080   ecx: 00000018   edx: ffbb74e0
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7788   esp: ffbf7760
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7760:
(XEN)    000001b4 fed37fec e0107788 e010e010 00000000 8005003b 000001b4 fed38000 
(XEN)    00000003 c0731600 ffbf77b8 ff140fea fed37fec ffbf77c4 00000137 ff11f59a 
(XEN)    0000000a fed37fec 00000000 00000000 000007f0 ffbb3080 00408827 ff147d12 
(XEN)    ffbf77c4 ffbb3080 00000017 ffbb74dc 00000003 c0731600 ffbf7818 fed37fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed36fec e0107818 e010e010 
(XEN)    00000000 8005003b 000001b4 fed37000 00000003 c0731600 ffbf7848 ff140fea 
(XEN)    fed36fec ffbf7854 00000136 ff11f59a 0000000a fed36fec 00000000 00000000 
(XEN)    00000760 ffbb3080 00408797 ff147d12 ffbf7854 ffbb3080 00000016 ffbb74d8 
(XEN)    00000003 c0731600 ffbf78a8 fed36fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed35fec e01078a8 e010e010 00000000 8005003b 000001b4 fed36000 
(XEN)    00000003 c0731600 ffbf78d8 ff140fea fed35fec ffbf78e4 00000135 ff11f59a 
(XEN)    0000000a fed35fec 00000000 00000000 000006d0 ffbb3080 00408707 ff147d12 
(XEN)    ffbf78e4 ffbb3080 00000015 ffbb74d4 00000003 c0731600 ffbf7938 fed35fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed34fec e0107938 e010e010 
(XEN)    00000000 8005003b 000001b4 fed35000 00000003 c0731600 ffbf7968 ff140fea 
(XEN)    fed34fec ffbf7974 00000134 ff11f59a 0000000a fed34fec 00000000 00000000 
(XEN)    00000640 ffbb3080 00408677 ff147d12 ffbf7974 ffbb3080 00000014 ffbb74d0 
(XEN)    00000003 c0731600 ffbf79c8 fed34fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed33fec e01079c8 e010e010 00000000 8005003b 000001b4 fed34000 
(XEN)    00000003 c0731600 ffbf79f8 ff140fea fed33fec ffbf7a04 00000133 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed38fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed39fec   ebx: ffbb3080   ecx: 00000019   edx: ffbb74e4
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf76f8   esp: ffbf76d0
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf76d0:
(XEN)    000001b4 fed38fec e01076f8 e010e010 00000000 8005003b 000001b4 fed39000 
(XEN)    00000003 c0731600 ffbf7728 ff140fea fed38fec ffbf7734 00000138 ff11f59a 
(XEN)    0000000a fed38fec 00000000 00000000 00000880 ffbb3080 004088b7 ff147d12 
(XEN)    ffbf7734 ffbb3080 00000018 ffbb74e0 00000003 c0731600 ffbf7788 fed38fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed37fec e0107788 e010e010 
(XEN)    00000000 8005003b 000001b4 fed38000 00000003 c0731600 ffbf77b8 ff140fea 
(XEN)    fed37fec ffbf77c4 00000137 ff11f59a 0000000a fed37fec 00000000 00000000 
(XEN)    000007f0 ffbb3080 00408827 ff147d12 ffbf77c4 ffbb3080 00000017 ffbb74dc 
(XEN)    00000003 c0731600 ffbf7818 fed37fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed36fec e0107818 e010e010 00000000 8005003b 000001b4 fed37000 
(XEN)    00000003 c0731600 ffbf7848 ff140fea fed36fec ffbf7854 00000136 ff11f59a 
(XEN)    0000000a fed36fec 00000000 00000000 00000760 ffbb3080 00408797 ff147d12 
(XEN)    ffbf7854 ffbb3080 00000016 ffbb74d8 00000003 c0731600 ffbf78a8 fed36fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed35fec e01078a8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed36000 00000003 c0731600 ffbf78d8 ff140fea 
(XEN)    fed35fec ffbf78e4 00000135 ff11f59a 0000000a fed35fec 00000000 00000000 
(XEN)    000006d0 ffbb3080 00408707 ff147d12 ffbf78e4 ffbb3080 00000015 ffbb74d4 
(XEN)    00000003 c0731600 ffbf7938 fed35fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed34fec e0107938 e010e010 00000000 8005003b 000001b4 fed35000 
(XEN)    00000003 c0731600 ffbf7968 ff140fea fed34fec ffbf7974 00000134 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed39fec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed3afec   ebx: ffbb3080   ecx: 0000001a   edx: ffbb74e8
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7668   esp: ffbf7640
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7640:
(XEN)    000001b4 fed39fec e0107668 e010e010 00000000 8005003b 000001b4 fed3a000 
(XEN)    00000003 c0731600 ffbf7698 ff140fea fed39fec ffbf76a4 00000139 ff11f59a 
(XEN)    0000000a fed39fec 00000000 00000000 00000910 ffbb3080 00408947 ff147d12 
(XEN)    ffbf76a4 ffbb3080 00000019 ffbb74e4 00000003 c0731600 ffbf76f8 fed39fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed38fec e01076f8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed39000 00000003 c0731600 ffbf7728 ff140fea 
(XEN)    fed38fec ffbf7734 00000138 ff11f59a 0000000a fed38fec 00000000 00000000 
(XEN)    00000880 ffbb3080 004088b7 ff147d12 ffbf7734 ffbb3080 00000018 ffbb74e0 
(XEN)    00000003 c0731600 ffbf7788 fed38fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed37fec e0107788 e010e010 00000000 8005003b 000001b4 fed38000 
(XEN)    00000003 c0731600 ffbf77b8 ff140fea fed37fec ffbf77c4 00000137 ff11f59a 
(XEN)    0000000a fed37fec 00000000 00000000 000007f0 ffbb3080 00408827 ff147d12 
(XEN)    ffbf77c4 ffbb3080 00000017 ffbb74dc 00000003 c0731600 ffbf7818 fed37fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed36fec e0107818 e010e010 
(XEN)    00000000 8005003b 000001b4 fed37000 00000003 c0731600 ffbf7848 ff140fea 
(XEN)    fed36fec ffbf7854 00000136 ff11f59a 0000000a fed36fec 00000000 00000000 
(XEN)    00000760 ffbb3080 00408797 ff147d12 ffbf7854 ffbb3080 00000016 ffbb74d8 
(XEN)    00000003 c0731600 ffbf78a8 fed36fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed35fec e01078a8 e010e010 00000000 8005003b 000001b4 fed36000 
(XEN)    00000003 c0731600 ffbf78d8 ff140fea fed35fec ffbf78e4 00000135 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed3afec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed3bfec   ebx: ffbb3080   ecx: 0000001b   edx: ffbb74ec
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf75d8   esp: ffbf75b0
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf75b0:
(XEN)    000001b4 fed3afec e01075d8 e010e010 00000000 8005003b 000001b4 fed3b000 
(XEN)    00000003 c0731600 ffbf7608 ff140fea fed3afec ffbf7614 0000013a ff11f59a 
(XEN)    0000000a fed3afec 00000000 00000000 000009a0 ffbb3080 004089d7 ff147d12 
(XEN)    ffbf7614 ffbb3080 0000001a ffbb74e8 00000003 c0731600 ffbf7668 fed3afec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed39fec e0107668 e010e010 
(XEN)    00000000 8005003b 000001b4 fed3a000 00000003 c0731600 ffbf7698 ff140fea 
(XEN)    fed39fec ffbf76a4 00000139 ff11f59a 0000000a fed39fec 00000000 00000000 
(XEN)    00000910 ffbb3080 00408947 ff147d12 ffbf76a4 ffbb3080 00000019 ffbb74e4 
(XEN)    00000003 c0731600 ffbf76f8 fed39fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed38fec e01076f8 e010e010 00000000 8005003b 000001b4 fed39000 
(XEN)    00000003 c0731600 ffbf7728 ff140fea fed38fec ffbf7734 00000138 ff11f59a 
(XEN)    0000000a fed38fec 00000000 00000000 00000880 ffbb3080 004088b7 ff147d12 
(XEN)    ffbf7734 ffbb3080 00000018 ffbb74e0 00000003 c0731600 ffbf7788 fed38fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed37fec e0107788 e010e010 
(XEN)    00000000 8005003b 000001b4 fed38000 00000003 c0731600 ffbf77b8 ff140fea 
(XEN)    fed37fec ffbf77c4 00000137 ff11f59a 0000000a fed37fec 00000000 00000000 
(XEN)    000007f0 ffbb3080 00408827 ff147d12 ffbf77c4 ffbb3080 00000017 ffbb74dc 
(XEN)    00000003 c0731600 ffbf7818 fed37fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed36fec e0107818 e010e010 00000000 8005003b 000001b4 fed37000 
(XEN)    00000003 c0731600 ffbf7848 ff140fea fed36fec ffbf7854 00000136 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed3bfec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed3cfec   ebx: ffbb3080   ecx: 0000001c   edx: ffbb74f0
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7548   esp: ffbf7520
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7520:
(XEN)    000001b4 fed3bfec e0107548 e010e010 00000000 8005003b 000001b4 fed3c000 
(XEN)    00000003 c0731600 ffbf7578 ff140fea fed3bfec ffbf7584 0000013b ff11f59a 
(XEN)    0000000a fed3bfec 00000000 00000000 00000a30 ffbb3080 00408a67 ff147d12 
(XEN)    ffbf7584 ffbb3080 0000001b ffbb74ec 00000003 c0731600 ffbf75d8 fed3bfec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed3afec e01075d8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed3b000 00000003 c0731600 ffbf7608 ff140fea 
(XEN)    fed3afec ffbf7614 0000013a ff11f59a 0000000a fed3afec 00000000 00000000 
(XEN)    000009a0 ffbb3080 004089d7 ff147d12 ffbf7614 ffbb3080 0000001a ffbb74e8 
(XEN)    00000003 c0731600 ffbf7668 fed3afec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed39fec e0107668 e010e010 00000000 8005003b 000001b4 fed3a000 
(XEN)    00000003 c0731600 ffbf7698 ff140fea fed39fec ffbf76a4 00000139 ff11f59a 
(XEN)    0000000a fed39fec 00000000 00000000 00000910 ffbb3080 00408947 ff147d12 
(XEN)    ffbf76a4 ffbb3080 00000019 ffbb74e4 00000003 c0731600 ffbf76f8 fed39fec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed38fec e01076f8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed39000 00000003 c0731600 ffbf7728 ff140fea 
(XEN)    fed38fec ffbf7734 00000138 ff11f59a 0000000a fed38fec 00000000 00000000 
(XEN)    00000880 ffbb3080 004088b7 ff147d12 ffbf7734 ffbb3080 00000018 ffbb74e0 
(XEN)    00000003 c0731600 ffbf7788 fed38fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed37fec e0107788 e010e010 00000000 8005003b 000001b4 fed38000 
(XEN)    00000003 c0731600 ffbf77b8 ff140fea fed37fec ffbf77c4 00000137 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed3cfec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed3dfec   ebx: ffbb3080   ecx: 0000001d   edx: ffbb74f4
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf74b8   esp: ffbf7490
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7490:
(XEN)    000001b4 fed3cfec e01074b8 e010e010 00000000 8005003b 000001b4 fed3d000 
(XEN)    00000003 c0731600 ffbf74e8 ff140fea fed3cfec ffbf74f4 0000013c ff11f59a 
(XEN)    0000000a fed3cfec 00000000 00000000 00000ac0 ffbb3080 00408af7 ff147d12 
(XEN)    ffbf74f4 ffbb3080 0000001c ffbb74f0 00000003 c0731600 ffbf7548 fed3cfec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed3bfec e0107548 e010e010 
(XEN)    00000000 8005003b 000001b4 fed3c000 00000003 c0731600 ffbf7578 ff140fea 
(XEN)    fed3bfec ffbf7584 0000013b ff11f59a 0000000a fed3bfec 00000000 00000000 
(XEN)    00000a30 ffbb3080 00408a67 ff147d12 ffbf7584 ffbb3080 0000001b ffbb74ec 
(XEN)    00000003 c0731600 ffbf75d8 fed3bfec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed3afec e01075d8 e010e010 00000000 8005003b 000001b4 fed3b000 
(XEN)    00000003 c0731600 ffbf7608 ff140fea fed3afec ffbf7614 0000013a ff11f59a 
(XEN)    0000000a fed3afec 00000000 00000000 000009a0 ffbb3080 004089d7 ff147d12 
(XEN)    ffbf7614 ffbb3080 0000001a ffbb74e8 00000003 c0731600 ffbf7668 fed3afec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed39fec e0107668 e010e010 
(XEN)    00000000 8005003b 000001b4 fed3a000 00000003 c0731600 ffbf7698 ff140fea 
(XEN)    fed39fec ffbf76a4 00000139 ff11f59a 0000000a fed39fec 00000000 00000000 
(XEN)    00000910 ffbb3080 00408947 ff147d12 ffbf76a4 ffbb3080 00000019 ffbb74e4 
(XEN)    00000003 c0731600 ffbf76f8 fed39fec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed38fec e01076f8 e010e010 00000000 8005003b 000001b4 fed39000 
(XEN)    00000003 c0731600 ffbf7728 ff140fea fed38fec ffbf7734 00000138 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff14a29f>] show_page_walk+0x43/0x127
(XEN)    [<ff140fea>] do_page_fault+0x20b/0x25d
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    [<ff13383c>] free_l2_table+0x53/0x78
(XEN)    [<ff1346ab>] free_page_type+0x19d/0x1fa
(XEN)    [<ff13493d>] put_page_type+0x235/0x382
(XEN)    [<ff13201c>] put_page_and_type+0x11/0x1e
(XEN)    [<ff135311>] new_guest_cr3+0x11d/0x320
(XEN)    [<ff1425df>] emulate_privileged_op+0x152a/0x1805
(XEN)    [<ff142d01>] do_general_protection+0x260/0x3d8
(XEN)    [<ff147d12>] error_code+0x62/0x7d
(XEN)    
(XEN) Pagetable walk from fed3dfec:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff14a29f>] show_page_walk+0x43/0x127
(XEN) EFLAGS: 00010282   CONTEXT: hypervisor
(XEN) eax: fed3efec   ebx: ffbb3080   ecx: 0000001e   edx: ffbb74f8
(XEN) esi: 00000003   edi: c0731600   ebp: ffbf7428   esp: ffbf7400
(XEN) cr0: 8005003b   cr3: 001b4000
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf7400:
(XEN)    000001b4 fed3dfec e0107428 e010e010 00000000 8005003b 000001b4 fed3e000 
(XEN)    00000003 c0731600 ffbf7458 ff140fea fed3dfec ffbf7464 0000013d ff11f59a 
(XEN)    0000000a fed3dfec 00000000 00000000 00000b50 ffbb3080 00408b87 ff147d12 
(XEN)    ffbf7464 ffbb3080 0000001d ffbb74f4 00000003 c0731600 ffbf74b8 fed3dfec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed3cfec e01074b8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed3d000 00000003 c0731600 ffbf74e8 ff140fea 
(XEN)    fed3cfec ffbf74f4 0000013c ff11f59a 0000000a fed3cfec 00000000 00000000 
(XEN)    00000ac0 ffbb3080 00408af7 ff147d12 ffbf74f4 ffbb3080 0000001c ffbb74f0 
(XEN)    00000003 c0731600 ffbf7548 fed3cfec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed3bfec e0107548 e010e010 00000000 8005003b 000001b4 fed3c000 
(XEN)    00000003 c0731600 ffbf7578 ff140fea fed3bfec ffbf7584 0000013b ff11f59a 
(XEN)    0000000a fed3bfec 00000000 00000000 00000a30 ffbb3080 00408a67 ff147d12 
(XEN)    ffbf7584 ffbb3080 0000001b ffbb74ec 00000003 c0731600 ffbf75d8 fed3bfec 
(XEN)    000e0000 ff14a29f 0000e008 00010282 000001b4 fed3afec e01075d8 e010e010 
(XEN)    00000000 8005003b 000001b4 fed3b000 00000003 c0731600 ffbf7608 ff140fea 
(XEN)    fed3afec ffbf7614 0000013a ff11f59a 0000000a fed3afec 00000000 00000000 
(XEN)    000009a0 ffbb3080 004089d7 ff147d12 ffbf7614 ffbb3080 0000001a ffbb74e8 
(XEN)    00000003 c0731600 ffbf7668 fed3afec 000e0000 ff14a29f 0000e008 00010282 
(XEN)    000001b4 fed39fec e0107668 e010e010 00000000 8005003b 000001b4 fed3a000 
(XEN)    00000003 c0731600 ffbf7698 ff140fea fed39fec ffbf76a4 00000139 ff11f59a 
(XEN) Xen call trace:
(XEN)    [<ff14a29f>]CPU:    1
(XEN) EIP:    e008:[<ff11d4ce>] number+0x9/0x329
(XEN) EFLAGS: 00010287
(XEN) CR3:    00000000
(XEN) eax: 00000043   ebx: 00000000   ecx: 00000053   edx: 00000000
(XEN) esi: 0000e010   edi: 0000e008   ebp: ffbf7058   esp: ffbf6fd8
(XEN) ds: e010   es: e010   fs: 0000   gs: 0000   ss: e010
(XEN) ************************************
(XEN) CPU1 DOUBLE FAULT -- system shutdown
(XEN) System needs manual reset.
(XEN) ************************************

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

> Thanks.  Unfortunaly I can take down Xen now by trying to use that code
> path, see the log below.

> (XEN) ----[ Xen-3.0.0    Not tainted ]----
> (XEN) CPU:    1
> (XEN) EIP:    e008:[<ff13383c>] free_l2_table+0x53/0x78

xen tries to dereference a pointer it got from map_domain_page() here.
Guess it points into nowhere because we are running the idle page
tables, not the domain page tables at that point.

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 1 Mar 2006, at 14:10, Gerd Hoffmann wrote:

>> Thanks.  Unfortunaly I can take down Xen now by trying to use that 
>> code
>> path, see the log below.
>
>> (XEN) ----[ Xen-3.0.0    Not tainted ]----
>> (XEN) CPU:    1
>> (XEN) EIP:    e008:[<ff13383c>] free_l2_table+0x53/0x78
>
> xen tries to dereference a pointer it got from map_domain_page() here.
> Guess it points into nowhere because we are running the idle page
> tables, not the domain page tables at that point.

Yep, that's the problem. I have an idea how to fix this. I'll get back 
to you.

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 1 Mar 2006, at 15:03, Keir Fraser wrote:

>> xen tries to dereference a pointer it got from map_domain_page() here.
>> Guess it points into nowhere because we are running the idle page
>> tables, not the domain page tables at that point.
>
> Yep, that's the problem. I have an idea how to fix this. I'll get back 
> to you.

Okay, try changeset 9025:e0f66dbe4b13 which (hopefully) fixes 
map_domain_page() to work properly when the guest is running on the 
host idle page tables.

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

> Okay, try changeset 9025:e0f66dbe4b13 which (hopefully) fixes
> map_domain_page() to work properly when the guest is running on the host
> idle page tables.

Thanks, it at least doesn't crash any more.  It doesn't work yet either,
but that most likely is a bug in setting up my page tables ...

btw: What happens if I try to install a gdt with zero entries?  Can I do
that?  Are there some default code/data/stack segments (full 4G minus
xen hole) available I can use then?  IIRC the domain builders don't
setup a gdt, so there must be some which are used at boot time until the
 OS kernel installes its own ...

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 2 Mar 2006, at 09:16, Gerd Hoffmann wrote:

> Thanks, it at least doesn't crash any more.  It doesn't work yet 
> either,
> but that most likely is a bug in setting up my page tables ...
>
> btw: What happens if I try to install a gdt with zero entries?  Can I 
> do
> that?  Are there some default code/data/stack segments (full 4G minus
> xen hole) available I can use then?  IIRC the domain builders don't
> setup a gdt, so there must be some which are used at boot time until 
> the
>  OS kernel installes its own ...

The default segment descriptors are *always* available. A guest can 
only install up to a 14-page GDT: the 15th and 16th pages are reserved 
by Xen and always present.

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

> The default segment descriptors are *always* available. A guest can only
> install up to a 14-page GDT: the 15th and 16th pages are reserved by Xen
> and always present.

Ok, managed to fixup that, switching to the default descriptors and
killing the GDT works, so the gdt doesn't stop validating any more.

But I ran into the next bug now.  I think we have a funny
chicken-and-egg problem when validating a self-consistent but new page
table set (which also maps itself) from scratch, see messages attached
below.

cheers,
  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

[-- Attachment #2: oops --]
[-- Type: text/plain, Size: 2035 bytes --]

(XEN) DOM1: (file=mm.c, line=1501) Bad type (saw 33020001 != exp e0000000) for mfn 1c711 (pfn 0)
(XEN) DOM1: (file=mm.c, line=479) Error getting mfn 1c711 (pfn 0) from L1 entry 1c711063 for dom1
(XEN) DOM1: (file=mm.c, line=685) Failure in alloc_l1_table: entry 0
(XEN) DOM1: (file=mm.c, line=1542) Error while validating mfn 18c92 (pfn 35a8) for type 20000000: caf=80000003 taf=20000001
(XEN) DOM1: (file=mm.c, line=402) Attempt to create linear p.t. with write perms
(XEN) DOM1: (file=mm.c, line=846) Failure in alloc_l2_table: entry 0
(XEN) DOM1: (file=mm.c, line=1542) Error while validating mfn 18c95 (pfn 35a5) for type 40000000: caf=80000003 taf=40000001
(XEN) DOM1: (file=mm.c, line=1579) trying slow path via idle pagetables ...
(XEN) DOM1: (file=mm.c, line=1501) Bad type (saw 40000001 != exp e0000000) for mfn 18c95 (pfn 35a5)
(XEN) DOM1: (file=mm.c, line=479) Error getting mfn 18c95 (pfn 35a5) from L1 entry 18c95063 for dom1
(XEN) DOM1: (file=mm.c, line=685) Failure in alloc_l1_table: entry 421
(XEN) DOM1: (file=mm.c, line=1542) Error while validating mfn 18c85 (pfn 35b5) for type 200d0000: caf=80000002 taf=200d0001
(XEN) DOM1: (file=mm.c, line=402) Attempt to create linear p.t. with write perms
(XEN) DOM1: (file=mm.c, line=846) Failure in alloc_l2_table: entry 13
(XEN) DOM1: (file=mm.c, line=1542) Error while validating mfn 18c95 (pfn 35a5) for type 40000000: caf=80000002 taf=40000001
(XEN) DOM1: (file=mm.c, line=1593) Fatal error while installing new baseptr 18c95
(XEN) domain_crash called from mm.c:1594
(XEN) Domain 1 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e019:[<c010ed3d>]
(XEN) EFLAGS: 00010206   CONTEXT: guest
(XEN) eax: c0012000   ebx: 00000000   ecx: fbf8d000   edx: 18c95000
(XEN) esi: 035a4000   edi: c0732400   ebp: c3e57e88   esp: c3e57e60
(XEN) cr0: 8005003b   cr3: 001b6000
(XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
(XEN) Guest stack trace from esp=c3e57e60:
(XEN)    Fault while accessing guest memory.

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 2 Mar 2006, at 11:34, Gerd Hoffmann wrote:

> Ok, managed to fixup that, switching to the default descriptors and
> killing the GDT works, so the gdt doesn't stop validating any more.
>
> But I ran into the next bug now.  I think we have a funny
> chicken-and-egg problem when validating a self-consistent but new page
> table set (which also maps itself) from scratch, see messages attached
> below.

Looks like your new pagetables have a writable mapping of MFN 18c95, 
which is the new page directory. i.e., the new p.t. set is not self 
consistent?

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

> Looks like your new pagetables have a writable mapping of MFN 18c95,
> which is the new page directory. i.e., the new p.t. set is not self
> consistent?

They where indeed.  Wrong usage of pte_wrprotect() :-/
Next question: Is there some way to _clear_ the trap table?

cheers,
  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 2 Mar 2006, at 16:36, Gerd Hoffmann wrote:

>> Looks like your new pagetables have a writable mapping of MFN 18c95,
>> which is the new page directory. i.e., the new p.t. set is not self
>> consistent?
>
> They where indeed.  Wrong usage of pte_wrprotect() :-/
> Next question: Is there some way to _clear_ the trap table?

Hehe. The interface wasn't so much designed with this in mind, but it 
can be done. Pass in an array of 256 trap_info structs, where the i'th 
struct contains:
   vector=i, flags=0, cs=0, addr=1

Note the address must be non-zero, as that otherwise marks the end of 
the list! You can put whatever non-zero value you like in that field -- 
it really doesn't matter. In fact you need to pass a list of *257* 
trap_info structs, the last of which has addr==0. :-)

The critical thing that disables the trap is flags==0: that sets the 
'dpl' for the 'trap gate' to zero so it effectively is unusable.

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

> The critical thing that disables the trap is flags==0: that sets the
> 'dpl' for the 'trap gate' to zero so it effectively is unusable.

Well, xen still tries to forward the trap to that vector then, but I
want to avoid exactly that.

The problem I have is that my kernel faults somewhere, but I'm already
that far in doing kexec that the usual linux kernel trap handling isn't
going to work any more.  What I'd like to see is xen print out a
register dump, with EIP being the faulting instruction, not the
(non-working) fault handler entry point.

Guess I have to hack xen a bit for that ...

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 3 Mar 2006, at 08:39, Gerd Hoffmann wrote:

>> The critical thing that disables the trap is flags==0: that sets the
>> 'dpl' for the 'trap gate' to zero so it effectively is unusable.
>
> Well, xen still tries to forward the trap to that vector then, but I
> want to avoid exactly that.
>
> The problem I have is that my kernel faults somewhere, but I'm already
> that far in doing kexec that the usual linux kernel trap handling isn't
> going to work any more.  What I'd like to see is xen print out a
> register dump, with EIP being the faulting instruction, not the
> (non-working) fault handler entry point.
>
> Guess I have to hack xen a bit for that ...

Well, the trick I described is good for at least 'non-exceptions' 
(i.e., it'll disable 'int x' software interrupts). For exceptions, I 
guess you should simply avoid causing them? :-)

In fact, why not install a dummy handler for all vectors (0-255). That 
is, a valid cs:eip that vectors to a small kexec function to print the 
state you desire? Setting flags==0 will ensure that softints don't 
work, but you can print out a useful message if you take a fault. You 
could try and put in a safe-ish place so it continues to work even for 
the early stages of the new kernel boot.

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 3 Mar 2006, at 09:14, Keir Fraser wrote:

> Well, the trick I described is good for at least 'non-exceptions' 
> (i.e., it'll disable 'int x' software interrupts). For exceptions, I 
> guess you should simply avoid causing them? :-)
>
> In fact, why not install a dummy handler for all vectors (0-255). That 
> is, a valid cs:eip that vectors to a small kexec function to print the 
> state you desire? Setting flags==0 will ensure that softints don't 
> work, but you can print out a useful message if you take a fault. You 
> could try and put in a safe-ish place so it continues to work even for 
> the early stages of the new kernel boot.

Actually a clear-trap-table call is quite easy to add: I checked in a 
patch that will clear the entire trap table if you pass a NULL pointer 
to the set_trap_table hypercall. With that patch in place, if you take 
a fault you will get a register dump in Xen as desired.

Changeset 9109:29f8c87...

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

> Actually a clear-trap-table call is quite easy to add: I checked in a
> patch that will clear the entire trap table if you pass a NULL pointer
> to the set_trap_table hypercall. With that patch in place, if you take a
> fault you will get a register dump in Xen as desired.

Thanks.

For now I have a different problem: get the userspace bits right.  Not
sure whenever it is easier to teach kexec-tools what xen is or teach
libxc what kexec is.  On a quick check the later looks easier.

stay tuned ;)

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

  Hi,

Ok, one of the more intresting issues is p2m map and ballooning, I'm not
sure yet how to address that best.

One problem are the "holes" in guest physical memory created by
ballooning.  The other one is the time gap between loading and booting
the kexec kernel (and the p2m table which may change inbetween).

My first attempt to address that issue by avoiding it
(features="auto_translated_physmap" ;) resulted in this:
kernel BUG at drivers/xen/balloon/balloon.c:216!

For now I have a few questions:

 * Which events can change the p2m map?  I think for domU that is only
   ballooning, right?  For dom0 additionally the backend drivers (when
   mapping foreign pages).  Anything else?
 * Is there some way to rebuild the p2m map from scratch using
   hypercalls?
 * Is there some easy way to "compress" the memory, i.e. move all pages
   to the start of (guest physical) memory?

 * Are unpriviledged domains allowed to do Dom0 ops for DOM_SELF?
   getdomaininfo for example?  Or hypercall_init?

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Michael Vrable]

On Mon, Mar 06, 2006 at 03:15:00PM +0100, Gerd Hoffmann wrote:
>  * Which events can change the p2m map?  I think for domU that is only
>    ballooning, right?  For dom0 additionally the backend drivers (when
>    mapping foreign pages).  Anything else?

Networking as well, since network packet reception involves flipping
pages between domains.

>  * Is there some way to rebuild the p2m map from scratch using
>    hypercalls?

As far as I know not completely reliably, since the P2M map is
maintained by the guest and Xen is almost entirely uninvolved.

You could probably get most of the way there by retrieving a list of
pages belonging to the guest (not sure if this can be done from within
an unprivileged guest though, or only in Domain-0), then building a
reverse map of entries in the M2P table.  This will only work if all
pages have accurate M2P entries (certainly won't work for domain-0,
since it maps foreign pages, but might work for unprivileged domains).

--Michael Vrable

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

> Actually a clear-trap-table call is quite easy to add: I checked in a
> patch that will clear the entire trap table if you pass a NULL pointer
> to the set_trap_table hypercall. With that patch in place, if you take a
> fault you will get a register dump in Xen as desired.

Well, I've tried that now, and got this:

(XEN) domain_crash_sync called from entry.S (ff147f35)
(XEN) Domain 1 (vcpu#0) crashed on cpu#1:
(XEN) Assertion '(diff != 0) || VM86_MODE(regs) || !RING_0(regs) ||
HVM_DOMAIN(current)' failed, line 35, file x86_32/traps.c
(XEN) BUG at x86_32/traps.c:35

(changeset 9137)

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: Re: [PATCH] Xen Guest Kexec

[Keir Fraser]

On 7 Mar 2006, at 13:06, Gerd Hoffmann wrote:

>> Actually a clear-trap-table call is quite easy to add: I checked in a
>> patch that will clear the entire trap table if you pass a NULL pointer
>> to the set_trap_table hypercall. With that patch in place, if you 
>> take a
>> fault you will get a register dump in Xen as desired.
>
> Well, I've tried that now, and got this:
>
> (XEN) domain_crash_sync called from entry.S (ff147f35)
> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> (XEN) Assertion '(diff != 0) || VM86_MODE(regs) || !RING_0(regs) ||
> HVM_DOMAIN(current)' failed, line 35, file x86_32/traps.c
> (XEN) BUG at x86_32/traps.c:35

The assertion is spurious. I've tightened up the assertion condition in 
changeset 9157:51c59d5d7... "Tighten up the assertion conditions in the 
GUEST_MODE() macro.".

  -- Keir

Re: Re: [PATCH] Xen Guest Kexec

[Gerd Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

> The assertion is spurious. I've tightened up the assertion condition in
> changeset 9157:51c59d5d7... "Tighten up the assertion conditions in the
> GUEST_MODE() macro.".

Looks like I'm triggering lots of funny bugs, see log below ...

Current plan is to get kexec work first with
features=auto_translated_physmap due to some non-trivial issues with
handling the p2m map.  That attempt killed Xen too ;)

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

[-- Attachment #2: oops --]
[-- Type: text/plain, Size: 2762 bytes --]

(XEN) get_mfn_from_gpfn_foreign(d->id=3, gpfn=65786) => 0 l2e=00000000
(XEN) get_mfn_from_gpfn_foreign(d->id=3, gpfn=30312) => 0 l2e=00000000
(XEN) ----[ Xen-3.0.0    Not tainted ]----
(XEN) CPU:    1
(XEN) EIP:    e008:[<ff17ad4f>] __shadow_get_l2e+0x77/0x80
(XEN) EFLAGS: 00010246   CONTEXT: hypervisor
(XEN) eax: fe7f9000   ebx: ffbf5ed0   ecx: ffbb1080   edx: fe7f9000
(XEN) esi: 033b6fd4   edi: 00000000   ebp: ffbf5df0   esp: ffbf5dd8
(XEN) cr0: 8005003b   cr3: 001b6000
(XEN) ds: e010   es: e010   fs: e010   gs: e021   ss: e010   cs: e008
(XEN) Xen stack trace from esp=ffbf5dd8:
(XEN)    00000000 ffbb0104 00000000 19a16063 00000000 00000001 ffbf5eb0 ff17ce17 
(XEN)    ffbb1080 0000b000 ffbf5e2c 000003ff e0000029 e0000029 ffbe4080 20000000 
(XEN)    e0000028 ffbf5fb4 ffbf5e40 00000286 fc67f270 ffbe4080 00000092 ffbaf0f0 
(XEN)    00000400 00019047 fec56000 fec58000 fec57000 00000001 00000000 00000000 
(XEN)    00000400 00000000 00000000 00000000 000003ff 00000000 000003ff ff13cafd 
(XEN)    fec56000 fec58000 fec57000 00000001 00000001 00000020 fc67f270 ffbe4080 
(XEN)    00000001 00000055 ffbf5eb0 ff17d798 00000001 ffbf5ed0 ffbf5ef0 ff17d6e7 
(XEN)    ffbe4080 20000000 000191de 00000002 ffbf5f08 00000020 00000000 1aa1a061 
(XEN)    1aa1a063 00000000 00000001 fec55008 ff1bd080 ff1179ef ffbf5f20 ff135bea 
(XEN)    ffbe4080 ffbb1108 ffbf5f20 ff128bd5 00000000 ffffffff ffbf5f60 00000001 
(XEN)    00000000 00000001 ffbf5f30 ff13a915 ffbe4080 ffbe431c ffbf5f60 ff105732 
(XEN)    ffbe4080 ffbe4318 00000000 00000001 ff1be05c ffbf2900 ffbe4080 00000000 
(XEN)    00000001 00000001 ffbf5f90 ff11b5fd 00000006 ff1be480 00000001 ff1be480 
(XEN)    ffbf5f98 ff105545 ffbe4080 00000006 00000040 00000001 ffbf5fb0 ff12681a 
(XEN)    ffbf5fa8 ff105583 ffbe4080 00000000 00000000 00000001 ffbf5ea8 00000000 
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 000d0000 0000420d 
(XEN)    0000e019 00000246 033b6fd4 0000e021 0000e021 0000e021 0000e021 0000e021 
(XEN)    00000001 ffbf6080 
(XEN) Xen call trace:
(XEN)    [<ff17ad4f>] __shadow_get_l2e+0x77/0x80
(XEN)    [<ff17ce17>] resync_all+0x3bd/0x852
(XEN)    [<ff17d6e7>] __shadow_sync_all+0x1c9/0x251
(XEN)    [<ff135bea>] shadow_sync_all+0x90/0x195
(XEN)    [<ff13a915>] cleanup_writable_pagetable+0x42/0x86
(XEN)    [<ff105732>] domain_shutdown_finalise+0x1ad/0x277
(XEN)    [<ff11b5fd>] do_softirq+0xa1/0xb8
(XEN)    [<ff12681a>] idle_loop+0x5b/0x5d
(XEN)    
(XEN) Pagetable walk from fe7f9000:
(XEN)   L2 = 00000000 55555555 
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) CPU1 FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: fe7f9000
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel