Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"

Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"

[tinnycloud]

Hi Steven:

	I was trying to make use of balloon driver on my win2003 HVM Guests,
but confront a Guest crash problem with blue screen.
	The serial output is much like 
	(XEN)grant_table.c:555:d0 Iomem mapping not permitted
ffffffffffffffff

	My pvdriver changeset is 823. From
http://xenbits.xensource.com/ext/win-pvdrivers.hg
	Xen is 4.0.0

	I've done quite a lot experiments to locate the crash, and finally
make it easy to reproduce.
	Here is the situation:

	1) Start several Win2003 Guest with 2G memory to occupy host memory
till from xm info | grep free to see xen has only almost 3G memory
	2) Start another Guest with 2G memory, say domain X, and set its
target to be 512M, after balloon, dom X will has only 512M, and xen has 2.5G
memory
	3) Start another Guest with 2G memory, then xen will have only 0.5G
memory left
	4) At this time, I set the domain X target to 2G (use xenstore-write
/local/domain/x/memory/target 2097152)
    5) domain X got blue screen at once. With code says 0x0000007f
(0x0000000d, 0x00000000, 0x00000000, 0x00000000)"

	So, in short, when a domain wants more memory, it deflates its
balloon, but if Xen doesn't enough memory, the domain will crash.

	I've go through
http://lists.xensource.com/archives/html/xen-devel/2010-01/msg01230.html
	
http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00074.html

	it looks like the crash is because windows crash dumper go through
those balloon out pages
	and from the discuss it would be harmless, but why I got this crash?
	
	Could you kindly offer me some help, thanks.

	

RE: Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"

[Paul Durrant]

If the frontend GPL PV storport driver is not ignoring block write failures in crashdump mode then you are unlikely the get a good dump from the ballooned down VM.
As for the original crash, I guess the balloon driver is just not handling the populate hypercall failure gracefully.

  Paul 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of tinnycloud
> Sent: 17 December 2010 03:24
> To: Steven Smith; xen-devel@lists.xensource.com
> Cc: james.harper@bendigoit.com.au; Keir Fraser; JBeulich@novell.com
> Subject: [Xen-devel] Win2003 got blue Screen when ballooning with
> serial out put: "Iomem mapping not permitted"
> 
> Hi Steven:
> 
> 	I was trying to make use of balloon driver on my win2003 HVM
> Guests,
> but confront a Guest crash problem with blue screen.
> 	The serial output is much like
> 	(XEN)grant_table.c:555:d0 Iomem mapping not permitted
> ffffffffffffffff
> 
> 	My pvdriver changeset is 823. From
> http://xenbits.xensource.com/ext/win-pvdrivers.hg
> 	Xen is 4.0.0
> 
> 	I've done quite a lot experiments to locate the crash, and
> finally
> make it easy to reproduce.
> 	Here is the situation:
> 
> 	1) Start several Win2003 Guest with 2G memory to occupy host
> memory
> till from xm info | grep free to see xen has only almost 3G memory
> 	2) Start another Guest with 2G memory, say domain X, and set
> its
> target to be 512M, after balloon, dom X will has only 512M, and xen
> has 2.5G
> memory
> 	3) Start another Guest with 2G memory, then xen will have
> only 0.5G
> memory left
> 	4) At this time, I set the domain X target to 2G (use
> xenstore-write
> /local/domain/x/memory/target 2097152)
>     5) domain X got blue screen at once. With code says 0x0000007f
> (0x0000000d, 0x00000000, 0x00000000, 0x00000000)"
> 
> 	So, in short, when a domain wants more memory, it deflates
> its
> balloon, but if Xen doesn't enough memory, the domain will crash.
> 
> 	I've go through
> http://lists.xensource.com/archives/html/xen-devel/2010-
> 01/msg01230.html
> 
> http://lists.xensource.com/archives/html/xen-devel/2010-
> 02/msg00074.html
> 
> 	it looks like the crash is because windows crash dumper go
> through
> those balloon out pages
> 	and from the discuss it would be harmless, but why I got this
> crash?
> 
> 	Could you kindly offer me some help, thanks.
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"

[James Harper]

> 
> If the frontend GPL PV storport driver is not ignoring block write
failures in
> crashdump mode then you are unlikely the get a good dump from the
ballooned
> down VM.

It does, because I had to balloon down a few pages for migration to work
(probably an old bug by now). I did put a limit as to how many failures
it was prepared to put up with though.

> As for the original crash, I guess the balloon driver is just not
handling the
> populate hypercall failure gracefully.
> 

Quite likely.

James

re: Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"

[tinnycloud]

Hi Paul:

	You are right. Thanks. 

	From the code PV driver, the exception handle is not finished by
James.

        KdPrint((__DRIVER_NAME "     Calling
HYPERVISOR_memory_op(XENMEM_populate_physmap) - pfn_count = %d\n",
pfn_count));
        ret = HYPERVISOR_memory_op(xpdd, XENMEM_populate_physmap,
&reservation);
        ExFreePoolWithTag(pfns, XENPCI_POOL_TAG);
        KdPrint((__DRIVER_NAME "     populated %d pages\n", ret));
        /* TODO: what do we do if less than the required number of pages
were populated??? can this happen??? *


	I will first try to implement myself.
	Also, that would be great appreciation if someone could offer this
patch. 


-----mail-----
: Paul Durrant [mailto:Paul.Durrant@citrix.com] 
sent: 2010年12月17日 16:53
to: tinnycloud; Steven Smith; xen-devel@lists.xensource.com
cc: james.harper@bendigoit.com.au; Keir Fraser; JBeulich@novell.com
Subject: RE: [Xen-devel] Win2003 got blue Screen when ballooning with serial
out put: "Iomem mapping not permitted"

If the frontend GPL PV storport driver is not ignoring block write failures
in crashdump mode then you are unlikely the get a good dump from the
ballooned down VM.
As for the original crash, I guess the balloon driver is just not handling
the populate hypercall failure gracefully.

  Paul 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of tinnycloud
> Sent: 17 December 2010 03:24
> To: Steven Smith; xen-devel@lists.xensource.com
> Cc: james.harper@bendigoit.com.au; Keir Fraser; JBeulich@novell.com
> Subject: [Xen-devel] Win2003 got blue Screen when ballooning with
> serial out put: "Iomem mapping not permitted"
> 
> Hi Steven:
> 
> 	I was trying to make use of balloon driver on my win2003 HVM
> Guests,
> but confront a Guest crash problem with blue screen.
> 	The serial output is much like
> 	(XEN)grant_table.c:555:d0 Iomem mapping not permitted
> ffffffffffffffff
> 
> 	My pvdriver changeset is 823. From
> http://xenbits.xensource.com/ext/win-pvdrivers.hg
> 	Xen is 4.0.0
> 
> 	I've done quite a lot experiments to locate the crash, and
> finally
> make it easy to reproduce.
> 	Here is the situation:
> 
> 	1) Start several Win2003 Guest with 2G memory to occupy host
> memory
> till from xm info | grep free to see xen has only almost 3G memory
> 	2) Start another Guest with 2G memory, say domain X, and set
> its
> target to be 512M, after balloon, dom X will has only 512M, and xen
> has 2.5G
> memory
> 	3) Start another Guest with 2G memory, then xen will have
> only 0.5G
> memory left
> 	4) At this time, I set the domain X target to 2G (use
> xenstore-write
> /local/domain/x/memory/target 2097152)
>     5) domain X got blue screen at once. With code says 0x0000007f
> (0x0000000d, 0x00000000, 0x00000000, 0x00000000)"
> 
> 	So, in short, when a domain wants more memory, it deflates
> its
> balloon, but if Xen doesn't enough memory, the domain will crash.
> 
> 	I've go through
> http://lists.xensource.com/archives/html/xen-devel/2010-
> 01/msg01230.html
> 
> http://lists.xensource.com/archives/html/xen-devel/2010-
> 02/msg00074.html
> 
> 	it looks like the crash is because windows crash dumper go
> through
> those balloon out pages
> 	and from the discuss it would be harmless, but why I got this
> crash?
> 
> 	Could you kindly offer me some help, thanks.
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

re: Win2003 got blue Screen when ballooning with serial out put: "Iomem mapping not permitted"

[tinnycloud]

Hi James:

	Thanks for your help.
	Beside not handling populating page failure, it seems like PV driver
hasn't support POD yet.
 	Another question is, in Linux Balloon driver, every time balloon
2MB, while in windows PV driver, the balloon unit is 1MB,
	Is there anything make this difference?

	Thanks.
	

----- -----
From: James Harper [mailto:james.harper@bendigoit.com.au] 
date: 20101217 17:20
tp: Paul Durrant; tinnycloud; Steven Smith; xen-devel@lists.xensource.com
cc: Keir Fraser; JBeulich@novell.com
subject: RE: [Xen-devel] Win2003 got blue Screen when ballooning with serial
out put: "Iomem mapping not permitted"

> 
> If the frontend GPL PV storport driver is not ignoring block write
failures in
> crashdump mode then you are unlikely the get a good dump from the
ballooned
> down VM.

It does, because I had to balloon down a few pages for migration to work
(probably an old bug by now). I did put a limit as to how many failures
it was prepared to put up with though.

> As for the original crash, I guess the balloon driver is just not
handling the
> populate hypercall failure gracefully.
> 

Quite likely.

James

Could the page sharing inside domain? (memory_sharing on xen)

[xiaoyun.maoxy]

Hi Keir:

	One of the features listed as having been added to Xen 4 is page
sharing for HVM domUs. 
Pity is it has been disabled since buggy. Well I tried to make it work these
days.

	After go through the code and get thoroughly concept of page
sharing, I've
been able to setup test environment, but confronted domU crash.

	One abnormal output from serial is 

(XEN) mm.c:859:d0 Error getting mfn 2df15f (pfn fffffffffffffffe) from L1
entry 80000002df15f627 for l1e_owner=0, pg_owner=4
(XEN) mm.c:859:d0 Error getting mfn 2df160 (pfn fffffffffffffffe) from L1
entry 80000002df160627 for l1e_owner=0, pg_owner=4

	Later, I found this is related to page sharing inside domain itself.

	That is when a domain X first read IO sectors Y, it will nominate
the gref into xen(that is build a hash entry in xen)
	, after IO completion, it will build a hash entry in tapdisk2 hash
table.
	
	But later domain X read same sectors again, it can find entry in
both hash tables, and page sharing works, one page freed.
 Then I can see above serial output.

	So my question is: Is page sharing legal inside domain?

	Thanks. 

	

TODO in windows balloon driver

[tinnycloud]

HI James:

	I've noticed the TODO tag in xenpci/xenpci_fdo.c, line 294

   290         KdPrint((__DRIVER_NAME "     Calling
HYPERVISOR_memory_op(XENMEM_populate_physmap) - pfn_count = %d\n",
pfn_count));
   291         ret = HYPERVISOR_memory_op(xpdd, XENMEM_populate_physmap,
&reservation);
   292         ExFreePoolWithTag(pfns, XENPCI_POOL_TAG);
   293         KdPrint((__DRIVER_NAME "     populated %d pages\n", ret));
   294         /* TODO: what do we do if less than the required number of
pages were populated??? can this happen??? */


	Could we fix it in linux balloon driver way? 
	When no enough pfn_count got, give memory back to xen.
	What do you think?
	Thanks.

       /* TODO: what do we do if less than the required number of pages were
populated??? can this happen??? */
        /* fixed! */
        if(ret < (ULONG)pfn_count){
            if(ret > 0){
                /* We hit the Xen hard limit: reprobe. */
                reservation.nr_extents = ret;
                ret = HYPERVISOR_memory_op(xpdd,
XENMEM_decrease_reservation, &reservation);
                KdPrint((__DRIVER_NAME "     decreased %d pages\n", ret));
            }
            mdl->Next = head;
            head = mdl;
            ExFreePoolWithTag(pfns, XENPCI_POOL_TAG);
            break;
        }

blue screen in windows balloon driver

[xiaoyun.maoxy]

Hi:

	We've confronted the windows 2003 Blue screen when using balloon
driver.
	
	The blue screen has the information of "NO_PAGES_AVAILABLE", and 
	Technical information:
  	***STOP: 0x0000004D (0x0002A8FB,0x0002A8FB,0x00000000,0x00000000)

	In fact we are doing the stress test on 24 windows2003 HVM on a 24G,
16core host.
	In dom0, there is a daemon process (balloond) will give memory back
to the VM who acclaim memory.
	The balloond will ensure every VM at least has 512M memory.
	
	Each VM is started with memory=512 maxmem=2048
	Inside each of the VM, there is two processes eat memory, each of
which will eat 1G memory.
	And the page file size configured on C:\ is init size 1536M, max
size 3072M, which I think even
	If balloond process not give back the memory back to VM, the VM can
has 512M+3072M = 3.5G virtual
	memory.

	Am I right?, if so , since our eat memory process only consume 2G
memory, h
	how could "NO_PAGES_AVAILABLE" happen?

	Also, since we enable VM writes its memory status into xenstore, and
when blue screen, it has only has 972KB memory.

	Any comments? Many thanks.




	

 
	
	
	

RE: blue screen in windows balloon driver

[James Harper]

> Hi:
> 
> 	We've confronted the windows 2003 Blue screen when using balloon
> driver.
> 
> 	The blue screen has the information of "NO_PAGES_AVAILABLE", and
> 	Technical information:
>   	***STOP: 0x0000004D
(0x0002A8FB,0x0002A8FB,0x00000000,0x00000000)
> 
> 	In fact we are doing the stress test on 24 windows2003 HVM on a
24G,
> 16core host.
> 	In dom0, there is a daemon process (balloond) will give memory
back
> to the VM who acclaim memory.
> 	The balloond will ensure every VM at least has 512M memory.
> 
> 	Each VM is started with memory=512 maxmem=2048
> 	Inside each of the VM, there is two processes eat memory, each
of
> which will eat 1G memory.
> 	And the page file size configured on C:\ is init size 1536M, max
> size 3072M, which I think even
> 	If balloond process not give back the memory back to VM, the VM
can
> has 512M+3072M = 3.5G virtual
> 	memory.
> 
> 	Am I right?, if so , since our eat memory process only consume
2G
> memory, h
> 	how could "NO_PAGES_AVAILABLE" happen?
> 
> 	Also, since we enable VM writes its memory status into xenstore,
and
> when blue screen, it has only has 972KB memory.
> 
> 	Any comments? Many thanks.
> 

You've put myself (I wrote GPLPV) and George Dunlap (from Citrix) in the
CC list but you don't say if you are using GPLPV drivers or Citrix PV
drivers. If you are using GPLPV drivers let me know and I'll get some
more information about the problem from you. If you aren't using GPLPV
drivers then I don't think I can be of much assistance...

Thanks

James

RE: TODO in windows balloon driver

[James Harper]

> 
> HI James:
> 
> 	I've noticed the TODO tag in xenpci/xenpci_fdo.c, line 294
> 
>    290         KdPrint((__DRIVER_NAME "     Calling
> HYPERVISOR_memory_op(XENMEM_populate_physmap) - pfn_count = %d\n",
> pfn_count));
>    291         ret = HYPERVISOR_memory_op(xpdd,
XENMEM_populate_physmap,
> &reservation);
>    292         ExFreePoolWithTag(pfns, XENPCI_POOL_TAG);
>    293         KdPrint((__DRIVER_NAME "     populated %d pages\n",
ret));
>    294         /* TODO: what do we do if less than the required number
of
> pages were populated??? can this happen??? */
> 
> 
> 	Could we fix it in linux balloon driver way?
> 	When no enough pfn_count got, give memory back to xen.
> 	What do you think?
> 	Thanks.

I guess that would work, but then it would keep trying to decrease again
every second. Maybe that's what we want but I'd rather put in some sort
of backoff in there eg retry in one second then two seconds then 4
seconds etc up to a maximum of 1 minute or so, to reduce the noise.

Are you seeing this happen? I'm guessing it's because xen doesn't have
enough pages for you or because you've ballooned up above the maxmem
limit or something...

James

RE: TODO in windows balloon driver

[MaoXiaoyun]

[-- Attachment #1.1: Type: text/plain, Size: 1767 bytes --]


Ye, You are right, it happens when xen doesn't have enough pages for the VM. 
And windows will crash now.
 
In memory overcommit, If there are many VMs on a physical host, and in dom0 use a daemon for 
memory managment between VMS, it is possible that xen has no pages for the VM needs memory.
 
So I think , it is better to fix it.
 
Thanks.
 
> Subject: RE: TODO in windows balloon driver
> Date: Sat, 26 Feb 2011 23:17:54 +1100
> From: james.harper@bendigoit.com.au
> To: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> 
> > 
> > HI James:
> > 
> > I've noticed the TODO tag in xenpci/xenpci_fdo.c, line 294
> > 
> > 290 KdPrint((__DRIVER_NAME " Calling
> > HYPERVISOR_memory_op(XENMEM_populate_physmap) - pfn_count = %d\n",
> > pfn_count));
> > 291 ret = HYPERVISOR_memory_op(xpdd,
> XENMEM_populate_physmap,
> > &reservation);
> > 292 ExFreePoolWithTag(pfns, XENPCI_POOL_TAG);
> > 293 KdPrint((__DRIVER_NAME " populated %d pages\n",
> ret));
> > 294 /* TODO: what do we do if less than the required number
> of
> > pages were populated??? can this happen??? */
> > 
> > 
> > Could we fix it in linux balloon driver way?
> > When no enough pfn_count got, give memory back to xen.
> > What do you think?
> > Thanks.
> 
> I guess that would work, but then it would keep trying to decrease again
> every second. Maybe that's what we want but I'd rather put in some sort
> of backoff in there eg retry in one second then two seconds then 4
> seconds etc up to a maximum of 1 minute or so, to reduce the noise.
> 
> Are you seeing this happen? I'm guessing it's because xen doesn't have
> enough pages for you or because you've ballooned up above the maxmem
> limit or something...
> 
> James
> 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 2336 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: TODO in windows balloon driver

[James Harper]

> 
> Ye, You are right, it happens when xen doesn't have enough pages for
the VM.
> And windows will crash now.
> 
> In memory overcommit, If there are many VMs on a physical host, and in
dom0
> use a daemon for
> memory managment between VMS, it is possible that xen has no pages for
the VM
> needs memory.
> 
> So I think , it is better to fix it.
> 

I have put your fix in as well as the backoff delay I mentioned earlier,
but I'm reviewing the ballooning stuff now. I can see one bug where I
don't do the initial balloon down in the case of PoD. I've fixed that,
but I'm not doing it early enough, so for Windows 2008 32 bit with
memory=512 and maxmem=1024, Windows still crashes and 'xm dmesg' shows
"Out of populate-on-demand memory!" just after my driver starts to load
(in DriverEntry). It seems to work fine when memory=768 and maxmem=1024
but I still think it could crash on startup if you are unlucky.

The only possible workaround is to do it in DriverEntry of xenpci but
that will require a bit of a refactoring. I think it's probably worth
doing though.

James

RE: RE: TODO in windows balloon driver

[James Harper]

> 
> >
> > Ye, You are right, it happens when xen doesn't have enough pages for
> the VM.
> > And windows will crash now.
> >
> > In memory overcommit, If there are many VMs on a physical host, and
in
> dom0
> > use a daemon for
> > memory managment between VMS, it is possible that xen has no pages
for
> the VM
> > needs memory.
> >
> > So I think , it is better to fix it.
> >
> 
> I have put your fix in as well as the backoff delay I mentioned
earlier,
> but I'm reviewing the ballooning stuff now. I can see one bug where I
> don't do the initial balloon down in the case of PoD. I've fixed that,
> but I'm not doing it early enough, so for Windows 2008 32 bit with
> memory=512 and maxmem=1024, Windows still crashes and 'xm dmesg' shows
> "Out of populate-on-demand memory!" just after my driver starts to
load
> (in DriverEntry). It seems to work fine when memory=768 and
maxmem=1024
> but I still think it could crash on startup if you are unlucky.
> 
> The only possible workaround is to do it in DriverEntry of xenpci but
> that will require a bit of a refactoring. I think it's probably worth
> doing though.
> 

I've just pushed a partial fix for this. The initial balloon-down is now
done during DriverEntry and I'm able to boot with memory=256 and
maxmem=1024 without any problems (other than that Windows doesn't run
particularly well with 256MB of memory :)

I haven't worked out a way to pass the memory reserved in DriverEntry
back to the xenpci FDO yet though so it's not particularly useful at
this time as you can't balloon up again. I'll fix that soon now that I
have the DriverEntry thing working.

James