More driver domain issues - rebooting backend hangs

More driver domain issues - rebooting backend hangs

[B.G. Bruce]

When rebooting the backend driver domain, if it has been utilized by a
normal domU, it will close down properly and then hang (not reboot). I'm
left rebooting the whole box.  Is this to be expected?  This even
happens if the normal domU has been powered off.

Second issue.  About half the time that I reboot the driver domain (no
connections from any outside domain), I'm left xm consoling to the
domain to hit the enter key on a simple line of:

echo "0" > /sys/bus/pci/slots:0000\:01:\01.1/power

I'm using the fake pci hotplug module so that I can reboot the driver
domain and still have the network card come back up after the reboot.

Any ideas?

B. 


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

Re: More driver domain issues - rebooting backend hangs

[B.G. Bruce]

On Fri, 2005-02-04 at 11:37, B.G. Bruce wrote:
> When rebooting the backend driver domain, if it has been utilized by a
> normal domU, it will close down properly and then hang (not reboot). I'm
> left rebooting the whole box.  Is this to be expected?  This even
> happens if the normal domU has been powered off.

I would still be really interested to know if anyone else using driver
domains experiences this.  If this is "normal/expected" behavior, I will
stop bug hunting and continue on with life.  If not, then I may
reconsider using driver domains at this time.

> 
> Second issue.  About half the time that I reboot the driver domain (no
> connections from any outside domain), I'm left xm consoling to the
> domain to hit the enter key on a simple line of:
> 
> echo "0" > /sys/bus/pci/slots:0000\:01:\01.1/power
> 
> I'm using the fake pci hotplug module so that I can reboot the driver
> domain and still have the network card come back up after the reboot.
> 
> Any ideas?
This one has been solved ... it was a hotplug issue.



> B. 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: More driver domain issues - rebooting backend hangs

[Ian Pratt]

> When rebooting the backend driver domain, if it has been utilized by a
> normal domU, it will close down properly and then hang (not 
> reboot). I'm
> left rebooting the whole box.  Is this to be expected?  This even
> happens if the normal domU has been powered off.

It certainly never used to be a problem with the old xend, but running
separate driver domains has not been widely tested with the current one.

If you're using a debug build of Xen with a serial console, it might be
worth hitting 'o' to find out what resources the domain is still
holding.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: More driver domain issues - rebooting backend hangs

[B.G. Bruce]

Thanks for the response Ian.

I've tried this with xen-2.0.3, 2.0.4 and xen-2.0-testing.bk (the latest
from yesterday at 16:00 AST -4:00).  Looking into the
xend.log/xend-debug.log files, what it appears I should see is:

[2005-02-06 21:13:28 xend] INFO (XendRoot:91) EVENT>
xend.domain.shutdown ['be03-3c59x', '3', 'poweroff']
[2005-02-06 21:13:38 xend] INFO (XendDomain:564) Destroying domain:
name=be03-3c59x
[2005-02-06 21:13:38 xend] INFO (XendRoot:91) EVENT> xend.virq 4
[2005-02-06 21:13:38 xend] INFO (XendRoot:91) EVENT> xend.domain.exit
['be03-3c59x', '3', 'poweroff']
[2005-02-06 21:13:38 xend] INFO (XendRoot:91) EVENT> xend.domain.destroy
['be03-3c59x', '3']
[2005-02-06 21:13:39 xend] INFO (XendRoot:91) EVENT> xend.domain.died
['be03-3c59x', '3']

but instead get (looping):

[2005-02-06 20:30:22 xend] INFO (XendRoot:91) EVENT>
xend.domain.shutdown ['be03-3c59x', '3', 'poweroff']
[2005-02-06 20:30:34 xend] INFO (XendDomain:564) Destroying domain:
name=be03-3c59x
[2005-02-06 20:30:34 xend] INFO (XendRoot:91) EVENT> xend.virq 4
[2005-02-06 20:30:34 xend] INFO (XendRoot:91) EVENT> xend.domain.exit
['be03-3c59x', '3', 'poweroff']
[2005-02-06 20:30:34 xend] INFO (XendRoot:91) EVENT> xend.domain.destroy
['be03-3c59x', '3']
[2005-02-06 20:32:58 xend] INFO (XendDomain:564) Destroying domain:
name=be03-3c59x
[2005-02-06 20:32:58 xend] INFO (XendRoot:91) EVENT> xend.domain.exit
['be03-3c59x', '3', 'poweroff']
[2005-02-06 20:32:58 xend] INFO (XendRoot:91) EVENT> xend.domain.destroy
['be03-3c59x', '3']
[2005-02-06 20:33:01 xend] INFO (XendDomain:564) Destroying domain:
name=be03-3c59x
[2005-02-06 20:33:01 xend] INFO (XendRoot:91) EVENT> xend.domain.exit
['be03-3c59x', '3', 'poweroff']
[2005-02-06 20:33:01 xend] INFO (XendRoot:91) EVENT> xend.domain.destroy
['be03-3c59x', '3']
[2005-02-06 20:33:03 xend] INFO (XendDomain:564) Destroying domain:
name=be03-3c59x
[2005-02-06 20:33:03 xend] INFO (XendRoot:91) EVENT> xend.domain.exit
['be03-3c59x', '3', 'poweroff']
[2005-02-06 20:33:03 xend] INFO (XendRoot:91) EVENT> xend.domain.destroy
['be03-3c59x', '3']
[2005-02-06 20:33:05 xend] INFO (XendDomain:564) Destroying domain:
name=be03-3c59x
[2005-02-06 20:33:05 xend] INFO (XendRoot:91) EVENT> xend.domain.exit
['be03-3c59x', '3', 'poweroff']
[2005-02-06 20:33:05 xend] INFO (XendRoot:91) EVENT> xend.domain.destroy
['be03-3c59x', '3']


On Sun, 2005-02-06 at 21:51, Ian Pratt wrote:
> > When rebooting the backend driver domain, if it has been utilized by a
> > normal domU, it will close down properly and then hang (not 
> > reboot). I'm
> > left rebooting the whole box.  Is this to be expected?  This even
> > happens if the normal domU has been powered off.
> 
> It certainly never used to be a problem with the old xend, but running
> separate driver domains has not been widely tested with the current one.
> 
> If you're using a debug build of Xen with a serial console, it might be
> worth hitting 'o' to find out what resources the domain is still
> holding.

I will try this and email you back

> Ian
> 

If there is any other information that I can/should provide, please let
me know.

B.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl

RE: More driver domain issues - rebooting backend hangs

[B.G. Bruce]

Ian,

Here is what "o" gave (I hope it helps):


xen0 root # (XEN) *** Serial input -> Xen (type 'CTRL-a' three times to
switch input to DOM0).
(XEN) pt base=235b sh_info=1bc
(XEN) Audit 0: [2a21,2ec] Skip foreign page dom=fc5b5a00 pfn=1a1
c=00000001 t=f0000002 m2p=ffffffff
(XEN) Audit 0: [2a21,2ed] Skip foreign page dom=fc5b4d20 pfn=1a9
c=00000001 t=f0000002 m2p=ffffffff
(XEN) Audit 0: [2a21,34f] Skip foreign page dom=fc5b4040 pfn=1b2
c=00000001 t=f0000002 m2p=ffffffff
(XEN) Audit 0: Found 256 lowmem mappings and 33025 io mappings
(XEN) Audit 0: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=1103
(XEN)      found dom=0 i=103 pfn=1346 t=36000000 c=80000001
(XEN) *pde = 00000000
(XEN)  -- POSSIBLY AN ACCESS TO FREED MEMORY? --
(XEN) CPU:    0
(XEN) EIP:    0808:[<fc520890>]
(XEN) EFLAGS: 00211206
(XEN) eax: f0300073   ebx: fea84800   ecx: fd400000   edx: 000f0300
(XEN) esi: 00000000   edi: ffa4a000   ebp: 002d0900   esp: fc503eec
(XEN) ds: 0810   es: 0810   fs: 0810   gs: 0810   ss: 0810
(XEN) Stack trace from ESP=fc503eec:
(XEN) 01041000 00000000 00000103 00001346 36000000 80000001 00000001
f0000002
(XEN)        ffffffff fc54f190 00000000 fc5b5a18 53458220 00000002
00001041 00006be1
(XEN)        00000000 0000006f fc503f80 00000001 00006be1 fd418618
00000100 00008101
(XEN)        fc503f6c 00003701 00000000 fc59e6c0 00000000 fc59e6c0
fc53a580 [fc5217d8]
(XEN)        fc59e6c0 00000000 00000008 [fc507ecd] 0000006f 00000000
fc53a580 [fc510cd2]
(XEN)        00000000 fc59e6c0 fc53a580 00000000 00040000 00000000
00000000 [fc519421]
(XEN)        00000000 fc59e6c0 00000198 00000001 53913e80 50964e00
00000001 c0306200
(XEN)        00000000 00000006 00070000 c0106c05 00000061 00201246
c02e1fe4 00000069
(XEN)        0000007b 0000007b 00000000 00000000 fc53a580
(XEN) Call Trace from ESP=fc503eec: [<fc5217d8>] [<fc507ecd>]
[<fc510cd2>] [<fc519421>]
                                                                                                                                                                                                     
****************************************
CPU0 FATAL PAGE FAULT
[error_code=0000]
Faulting linear address might be fea8480c
Aieee! CPU0 is toast...
****************************************
                                                                                                                                                                                                     
Reboot in five seconds...




On Sun, 2005-02-06 at 21:51, Ian Pratt wrote:
> > When rebooting the backend driver domain, if it has been utilized by a
> > normal domU, it will close down properly and then hang (not 
> > reboot). I'm
> > left rebooting the whole box.  Is this to be expected?  This even
> > happens if the normal domU has been powered off.
> 
> It certainly never used to be a problem with the old xend, but running
> separate driver domains has not been widely tested with the current one.
> 
> If you're using a debug build of Xen with a serial console, it might be
> worth hitting 'o' to find out what resources the domain is still
> holding.
> 
> Ian
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: More driver domain issues - rebooting backend hangs

[B.G. Bruce]

With the latest pull for testing, I now get more info in xend-debug.log
that may help to diagnose as well.  This is what is reported as I
shutdown the driver domains AFTER the front end domain has properly
terminated.

{}
{}
{}
{}
VirqClient.virqReceived> 4
vif-bridge down vif=vif5.0 domain=be01-vsw mac=aa:00:04:fa:00:02
bridge=mgmt
VirqClient.virqReceived> 4
vif-bridge down vif=vif7.0 domain=be03-3c59x mac=aa:00:04:fa:00:04
bridge=mgmt
VirqClient.virqReceived> 4
vif-bridge down vif=vif8.0 domain=fwmgmt mac=aa:00:02:fa:00:05
bridge=e1000
bridge e1000 doesn't exist!
vif-bridge down vif=vif8.1 domain=fwmgmt mac=aa:00:03:fa:00:05
bridge=3c59x
bridge 3c59x doesn't exist!
CtrlMsgRcvr>writeRequest> no channel!
<xen.xend.server.netif.NetifBackendController instance at 0xb7671e2c>
vif-bridge down vif=vif8.2 domain=fwmgmt mac=aa:00:01:fa:00:05
bridge=vsw
bridge vsw doesn't exist!
CtrlMsgRcvr>writeRequest> no channel!
<xen.xend.server.netif.NetifBackendController instance at 0xb7671f6c>
vif-bridge down vif=vif8.3 domain=fwmgmt mac=aa:00:04:fa:00:05
bridge=mgmt
VirqClient.virqReceived> 4
vif-bridge down vif=vif6.0 domain=be02-e1000 mac=aa:00:04:fa:00:03
bridge=mgmt




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: More driver domain issues - rebooting backend hangs

[B.G. Bruce]

Well, that idea was wrong.  I've got my front end now using a
vif-be-bridge script that is basically the vif-bridge script with the
brctl/ifconfig/iptables commands removed.  I get nothing reported in
xend-debug now, but the backends still hang on a poweroff or reboot.

B.

On Tue, 2005-02-08 at 07:58, B.G. Bruce wrote:
> With the latest pull for testing, I now get more info in xend-debug.log
> that may help to diagnose as well.  This is what is reported as I
> shutdown the driver domains AFTER the front end domain has properly
> terminated.
> 
> {}
> {}
> {}
> {}
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif5.0 domain=be01-vsw mac=aa:00:04:fa:00:02
> bridge=mgmt
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif7.0 domain=be03-3c59x mac=aa:00:04:fa:00:04
> bridge=mgmt
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif8.0 domain=fwmgmt mac=aa:00:02:fa:00:05
> bridge=e1000
> bridge e1000 doesn't exist!
> vif-bridge down vif=vif8.1 domain=fwmgmt mac=aa:00:03:fa:00:05
> bridge=3c59x
> bridge 3c59x doesn't exist!
> CtrlMsgRcvr>writeRequest> no channel!
> <xen.xend.server.netif.NetifBackendController instance at 0xb7671e2c>
> vif-bridge down vif=vif8.2 domain=fwmgmt mac=aa:00:01:fa:00:05
> bridge=vsw
> bridge vsw doesn't exist!
> CtrlMsgRcvr>writeRequest> no channel!
> <xen.xend.server.netif.NetifBackendController instance at 0xb7671f6c>
> vif-bridge down vif=vif8.3 domain=fwmgmt mac=aa:00:04:fa:00:05
> bridge=mgmt
> VirqClient.virqReceived> 4
> vif-bridge down vif=vif6.0 domain=be02-e1000 mac=aa:00:04:fa:00:03
> bridge=mgmt
> 
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

RE: More driver domain issues - rebooting backend hangs

[Ian Pratt]

If you build a kernel from the latest repo, I'd be interested to see
what ouput you get from hitting 'o' after starting the driver domain,
and then again after doing an 'xm destroy' on it.

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_ide95&alloc_id\x14396&op=click

RE: More driver domain issues - rebooting backend hangs

[B.G. Bruce]

[-- Attachment #1: Type: text/plain, Size: 9339 bytes --]

On Tue, 2005-02-08 at 18:45, Ian Pratt wrote:
> If you build a kernel from the latest repo, I'd be interested to see
> what ouput you get from hitting 'o' after starting the driver domain,
> and then again after doing an 'xm destroy' on it.
> 
> Ian
> 

Ian, I sure hope this helps ... 
B.

START:  One driver domain (be02-e1000 - no front end domains):

(XEN) physdev_pci_access_modify: 01:01:00
(XEN)   add RW 01:01:00
(XEN) Giving domain 2 IO resources (bc00 - bc1f) for device 01:01.0
(XEN) pt base=1c52 sh_info=1bc
(XEN) Audit 0: [1070,148] Skip foreign page dom=fc5b4040 pfn=1ec59
c=80000003 t=f0000002 m2p=10b7
(XEN) Audit 0: [1070,178] Skip foreign page dom=fc5b4040 pfn=1ec54
c=80000003 t=f0000002 m2p=10bc
(XEN) Audit 0: [1070,179] Skip foreign page dom=fc5b4040 pfn=1ec53
c=80000003 t=f0000002 m2p=10bd
(XEN) Audit 0: [2fce,254] Skip foreign page dom=fc5b4040 pfn=1ad
c=80000003 t=f0000003 m2p=ffffffff
(XEN) Audit 0: Found 256 lowmem mappings and 33025 io mappings
(XEN) Audit 0: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=1103
(XEN)      found dom=0 i=103 pfn=1346 t=36000000 c=80000001
(XEN) Audit 0: Done. pages=2137 l1=132 l2=20 ctot=22050 ttot=1174
(XEN) pt base=73e5 sh_info=1ad
(XEN) Audit 2: Found 0 lowmem mappings and 32 io mappings
(XEN) Audit 2: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=6d5d
(XEN)      found dom=2 i=103 pfn=64ea t=36000000 c=80000001
(XEN) Audit 2: gen count!=1 (c=80000002) t=f0000002 ot=f0000002
pfn=1ec59
(XEN)      found dom=0 i=148 pfn=1070 t=36640014 c=80000016
(XEN)      found dom=2 i=b7 pfn=6478 t=36080000 c=80000001
(XEN) Audit 2: gen count!=1 (c=80000002) t=f0000002 ot=f0000002
pfn=1ec54
(XEN)      found dom=0 i=178 pfn=1070 t=36640014 c=80000016
(XEN)      found dom=2 i=bc pfn=6478 t=36080000 c=80000001
(XEN) Audit 2: gen count!=1 (c=80000002) t=f0000002 ot=f0000002
pfn=1ec53
(XEN)      found dom=0 i=179 pfn=1070 t=36640014 c=80000016
(XEN)      found dom=2 i=bd pfn=6478 t=36080000 c=80000001
(XEN) Audit 2: Done. pages=1744 l1=34 l2=6 ctot=9627 ttot=111



DESTROY:  One driver domain (be02-e1000 - no front end domain):
(XEN) pt base=1c52 sh_info=1bc
(XEN) Audit 0: Found 256 lowmem mappings and 33025 io mappings
(XEN) Audit 0: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=1103
(XEN)      found dom=0 i=103 pfn=1346 t=36000000 c=80000001
(XEN) Audit 0: Done. pages=1025 l1=137 l2=23 ctot=22211 ttot=1335


START:  One driver domain (be02-e1000) & 1 front end domain
(fe99-fwmgmt):
(XEN) physdev_pci_access_modify: 01:01:00
(XEN)   add RW 01:01:00
(XEN) Giving domain 3 IO resources (bc00 - bc1f) for device 01:01.0
(XEN) pt base=1c52 sh_info=1bc
(XEN) Audit 0: [1070,148] Skip foreign page dom=fc5b4040 pfn=6665
c=80000003 t=f0000002 m2p=10b5
(XEN) Audit 0: [1070,176] Skip foreign page dom=fc5b4d20 pfn=854b
c=80000003 t=f0000002 m2p=10e0
(XEN) Audit 0: [1070,178] Skip foreign page dom=fc5b4040 pfn=6660
c=80000003 t=f0000002 m2p=10ba
(XEN) Audit 0: [1070,179] Skip foreign page dom=fc5b4040 pfn=665f
c=80000003 t=f0000002 m2p=10bb
(XEN) Audit 0: [1070,17c] Skip foreign page dom=fc5b4d20 pfn=8541
c=80000003 t=f0000002 m2p=10ea
(XEN) Audit 0: [1070,17d] Skip foreign page dom=fc5b4d20 pfn=8540
c=80000003 t=f0000002 m2p=10eb
(XEN) Audit 0: [2fce,252] Skip foreign page dom=fc5b4d20 pfn=1a4
c=80000003 t=f0000003 m2p=ffffffff
(XEN) Audit 0: [2fce,254] Skip foreign page dom=fc5b4040 pfn=1ba
c=80000003 t=f0000003 m2p=ffffffff
(XEN) Audit 0: Found 256 lowmem mappings and 33025 io mappings
(XEN) Audit 0: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=1103
(XEN)      found dom=0 i=103 pfn=1346 t=36000000 c=80000001
(XEN) Audit 0: Done. pages=2230 l1=132 l2=16 ctot=21878 ttot=966
(XEN) pt base=6431 sh_info=1ba
(XEN) Audit 3: [6669,f0] Skip foreign page dom=fc5b4d20 pfn=8545
c=80000003 t=f0000002 m2p=10e6
(XEN) Audit 3: [6669,f1] Skip foreign page dom=fc5b4d20 pfn=8544
c=80000003 t=f0000002 m2p=10e7
(XEN) Audit 3: Found 0 lowmem mappings and 32 io mappings
(XEN) Audit 3: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=e17
(XEN)      found dom=3 i=103 pfn=6bb4 t=36000000 c=80000001
(XEN) Audit 3: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=6665
(XEN)      found dom=0 i=148 pfn=1070 t=36640010 c=80000012
(XEN)      found dom=3 i=b5 pfn=4744 t=36080000 c=80000001
(XEN) Audit 3: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=6660
(XEN)      found dom=0 i=178 pfn=1070 t=36640010 c=80000012
(XEN)      found dom=3 i=ba pfn=4744 t=36080000 c=80000001
(XEN) Audit 3: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=665f
(XEN)      found dom=0 i=179 pfn=1070 t=36640010 c=80000012
(XEN)      found dom=3 i=bb pfn=4744 t=36080000 c=80000001
(XEN) Audit 3: Done. pages=1774 l1=34 l2=7 ctot=9636 ttot=122
(XEN) pt base=7b49 sh_info=1a4
(XEN) Audit 4: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=69a8
(XEN)      found dom=4 i=103 pfn=7ad1 t=36000000 c=80000001
(XEN) Audit 4: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=854b
(XEN)      found dom=0 i=176 pfn=1070 t=36640010 c=80000012
(XEN)      found dom=4 i=e0 pfn=d67 t=36080000 c=80000001
(XEN) Audit 4: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=8545
(XEN)      found dom=3 i=f0 pfn=6669 t=36140007 c=80000009
(XEN)      found dom=4 i=e6 pfn=d67 t=36080000 c=80000001
(XEN) Audit 4: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=8544
(XEN)      found dom=3 i=f1 pfn=6669 t=36140007 c=80000009
(XEN)      found dom=4 i=e7 pfn=d67 t=36080000 c=80000001
(XEN) Audit 4: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=8541
(XEN)      found dom=0 i=17c pfn=1070 t=36640010 c=80000012
(XEN)      found dom=4 i=ea pfn=d67 t=36080000 c=80000001
(XEN) Audit 4: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=8540
(XEN)      found dom=0 i=17d pfn=1070 t=36640010 c=80000012
(XEN)      found dom=4 i=eb pfn=d67 t=36080000 c=80000001
(XEN) Audit 4: Done. pages=1309 l1=70 l2=14 ctot=18820 ttot=371


SHUTDOWN:  front end domain (fe99-fwmgmt):
(XEN) pt base=1c52 sh_info=1bc
(XEN) Audit 0: [1070,148] Skip foreign page dom=fc5b4040 pfn=6665
c=80000003 t=f0000002 m2p=10b5
(XEN) Audit 0: [1070,178] Skip foreign page dom=fc5b4040 pfn=6660
c=80000003 t=f0000002 m2p=10ba
(XEN) Audit 0: [1070,179] Skip foreign page dom=fc5b4040 pfn=665f
c=80000003 t=f0000002 m2p=10bb
(XEN) Audit 0: [2fce,254] Skip foreign page dom=fc5b4040 pfn=1ba
c=80000003 t=f0000003 m2p=ffffffff
(XEN) Audit 0: Found 256 lowmem mappings and 33025 io mappings
(XEN) Audit 0: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=1103
(XEN)      found dom=0 i=103 pfn=1346 t=36000000 c=80000001
(XEN) Audit 0: Done. pages=2212 l1=131 l2=16 ctot=21877 ttot=965
(XEN) pt base=6431 sh_info=1ba
(XEN) Audit 3: Found 0 lowmem mappings and 32 io mappings
(XEN) Audit 3: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=e17
(XEN)      found dom=3 i=103 pfn=6bb4 t=36000000 c=80000001
(XEN) Audit 3: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=6665
(XEN)      found dom=0 i=148 pfn=1070 t=36640010 c=80000012
(XEN)      found dom=3 i=b5 pfn=4744 t=36080000 c=80000001
(XEN) Audit 3: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=6660
(XEN)      found dom=0 i=178 pfn=1070 t=36640010 c=80000012
(XEN)      found dom=3 i=ba pfn=4744 t=36080000 c=80000001
(XEN) Audit 3: gen count!=1 (c=80000002) t=f0000002 ot=f0000002 pfn=665f
(XEN)      found dom=0 i=179 pfn=1070 t=36640010 c=80000012
(XEN)      found dom=3 i=bb pfn=4744 t=36080000 c=80000001
(XEN) Audit 3: Done. pages=1910 l1=30 l2=6 ctot=9625 ttot=107

DESTROY:  One driver domain (be02-e1000) after front end ShUTDOWN:
1)	Gave "Error: Internal Server Error"
2)	Could not "xm console be02-e1000" - 111 - connection refused, however
xend was still running .....
3)	xm shutdown be02-e1000 still worked

DESTROY:  One driver domain (be02-e1000) after front end ShUTDOWN:
(XEN) pt base=426c sh_info=1bc
(XEN) Audit 0: [2fce,254] Skip foreign page dom=fc5b4040 pfn=1ba
c=80000003 t=f0000003 m2p=ffffffff
(XEN) Audit 0: Found 256 lowmem mappings and 33025 io mappings
(XEN) Audit 0: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=1103
(XEN)      found dom=0 i=103 pfn=1346 t=36000000 c=80000001
(XEN) Audit 0: Done. pages=2330 l1=122 l2=17 ctot=21934 ttot=1008
(XEN) pt base=6088 sh_info=1ba
(XEN) Audit 3: Found 0 lowmem mappings and 32 io mappings
(XEN) Audit 3: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=e17
(XEN)      found dom=3 i=103 pfn=6bb4 t=36000000 c=80000001
(XEN) Audit 3: Done. pages=1910 l1=31 l2=6 ctot=9626 ttot=108

SHUTDOWN: One driver domain (be02-e1000) - after DESTROY failed:
1)	xm list shows domain to be in state "s" but does not disappear.

(XEN) pt base=1c52 sh_info=1bc
(XEN) Audit 0: [2fce,254] Skip foreign page dom=fc5b4040 pfn=1ba
c=80000003 t=f0000003 m2p=ffffffff
(XEN) Audit 0: Found 256 lowmem mappings and 33025 io mappings
(XEN) Audit 0: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=1103
(XEN)      found dom=0 i=103 pfn=1346 t=36000000 c=80000001
(XEN) Audit 0: Done. pages=3387 l1=112 l2=16 ctot=21872 ttot=946
(XEN) pt base=6431 sh_info=1ba
(XEN) Audit 3: Found 0 lowmem mappings and 32 io mappings
(XEN) Audit 3: gen count!=1 (c=80000002) t=b0000001 ot=b0000001 pfn=e17
(XEN)      found dom=3 i=103 pfn=6bb4 t=36000000 c=80000001
(XEN) Audit 3: Done. pages=1910 l1=30 l2=6 ctot=9625 ttot=107

SXP configurations attached ....


[-- Attachment #2: e1000.sxp --]
[-- Type: text/plain, Size: 676 bytes --]

(vm
    (name be02-e1000)
    (memory 32)
    (restart onreboot)
    (backend(netif))
    (image
        (linux
            (kernel /boot/vmlinuz-2.6.10-xen-be)
            (root '/dev/hda1 ro')
            (args 'panic=1 elevator=deadline e1000.RxDescriptors=1000 e1000.TxDescriptors=1000 e1000.InterruptThrottleRate=1')
        )
    )
    (device (vbd (uname phy:raid1/e1000) (dev /dev/hda1) (mode w)))
    (device (vbd (uname phy:raid1/portage) (dev /dev/hda2) (mode r)))
    (device (vbd (uname phy:raid0/swap_e1000) (dev /dev/hda3) (mode w)))
    (device (pci (bus 0x1) (dev 0x1) (func 0x0)))
    (device (vif (mac aa:00:04:fa:00:03) (bridge mgmt)))
)

[-- Attachment #3: fwmgmt.sxp --]
[-- Type: text/plain, Size: 611 bytes --]

(vm
    (name fe99-fwmgmt)
    (memory 64)
    (restart onreboot)
    (image
        (linux
            (kernel /boot/vmlinuz-2.6.10-xen-fe)
            (root '/dev/hda1 ro')
            (args 'panic=1')
        )
    )
    (device (vbd (uname phy:raid1/fwmgmt) (dev /dev/hda1) (mode w)))
    (device (vbd (uname phy:raid1/portage) (dev /dev/hda2) (mode r)))
    (device (vbd (uname phy:raid0/swap_fwmgmt) (dev /dev/hda3) (mode w)))
    (device (vif (mac aa:00:02:fa:00:05) (bridge e1000) (backend be02-e1000) (script vif-be-bridge)))
    (device (vif (mac aa:00:04:fa:00:05) (bridge mgmt)))
)