From: Kip Macy <kmacy@fsmware.com>
To: Keir Fraser <Keir.Fraser@cl.cam.ac.uk>
Cc: Ian Pratt <Ian.Pratt@cl.cam.ac.uk>, xen-devel@lists.sourceforge.net
Subject: Re: refcount errors then crash on XenoLinux with the latest source
Date: Mon, 23 Feb 2004 19:44:32 -0800 (PST) [thread overview]
Message-ID: <20040223194202.P71501@demos.bsdclusters.com> (raw)
In-Reply-To: <20040223165800.D71501@demos.bsdclusters.com>
It happened again. Is it possible that Xen isn't disabling network
interrupts while it is "auditing all pages"?
-Kip
Killing domain 1
Releasing task 1
DOM0: INIT: Id "x1" respawning too fast: disabled for 5 minutes
DOM0: INIT: Id "x1" respawning too fast: disabled for 5 minutes
DOM0: INIT: Id "x1" respawning too fast: disabled for 5 minutes
audit_all_pages
refcount error: pfn=000247 cf=00000001 refcount=0
audit page: pfn=247 info: cf=1 tf=f0000001 ts=0 dom=fc648040
refcount error: pfn=00024d cf=00000001 refcount=0
audit page: pfn=24d info: cf=1 tf=f0000001 ts=0 dom=fc648040
refcount error: pfn=00036f cf=40000002 refcount=1
audit page: pfn=36f info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
pte_pfn=002207 cf=10000023 tf=30000021 dom=fc648be0
pte_idx=3f9 *pte_idx=0036f063
refcount error: pfn=000371 cf=40000002 refcount=1
audit page: pfn=371 info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
pte_pfn=002207 cf=10000023 tf=30000021 dom=fc648be0
pte_idx=3fe *pte_idx=00371063
refcount error: pfn=000372 cf=40000002 refcount=1
audit page: pfn=372 info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
pte_pfn=002207 cf=10000023 tf=30000021 dom=fc648be0
pte_idx=3fd *pte_idx=00372063
CPU: 1
EIP: 0808:[<fc532ddf>]
EFLAGS: 00010206
eax: 06081012 ebx: 00000010 ecx: fc657560 edx: fc650da0
esi: fc650da0 edi: fc657540 ebp: 00000000 esp: fc64fd70
ds: 0810 es: 0810 fs: 0810 gs: 0810 ss: 0810
Stack trace from ESP=fc64fd70:
ff803012 00000008 [fc51302a] fc76bb40 0000003c fc657540 fc657540
[fc5328c2]
fc657540 fc657400 00000017 ffffffff 00000017 fc648040 00000040
0000003e
fc657400 fc76bb40 00000040 fc76a740 04000001 fc657540 00005048
[fc5323a8]
fc657540 fc5ebc80 fc5d1f1c 00000001 00000046 00000004 [fc509639]
fc76a740
04000001 fc64fe60 00000010 [fc5b1f2d] 00000010 fc657400 fc64fe60
3d6e6670
33303030 63203237 00000001 fc76a740 fc600bc0 00000010 fc64fe60
[fc5b20f3]
00000010 fc64fe60 fc76a740 00002207 00372063 34429e8f 00000001
0007fff0
0007fff0 00000002 00000004 [fc5af970] 0007fff0 fd800000 00000001
0007fff0
00000002 00000004 00000040 00010810 00000810 00000810 fc500810
ffffff10
[fc50d0e5] 00000808 00000202 0000004d fc64ff6c fc64ff6c
[fc509f22] 0000004d
00000000 fc64ff6c [fc5162b6] 00000003 00000040 fc5ebc80
[fc517c4e] 0000004d
fc64ff6c fc64ff6c [fc512529] 00000003 fc6501e0 02000001
[fc517ca4] fc5ebc80
fc64ff6c 0092578a 00000000 fc651200 00000006 00000006 [fc5b1f2d]
00000004
fc5ebc80 fc64ff6c [fc5b151c] 00000004 00000001 00000001 fc6501e0
fc6008c0
00000004 fc64ff6c [fc5b20f3] 00000004 fc64ff6c fc6501e0
[fc511e2e] fc624494
431ea128 00000001 00000040 fc648040 00000040 fc649780 [fc5af970]
00000040
00000001 00000040 fc648040 00000040 fc649780 00000040 fc640810
fc640810
00000810 fc640810 ffffff04 [fc5b5e04] 00000808 00000246
[fc5b5e40] fc648040
004c4b40 ffffffff 655f6464 7972746e 5f636100 656d6974 61007372
69745f63
5f72656d 74666f73 5f717269 69746361 64006e6f 5f706d75 656d6974
62007172
On Mon, 23 Feb 2004, Kip Macy wrote:
>
> >
> > This is a Xen crash dump. ksymoops won't help -- you'll need to map
> > the crash dump to Xen code by hand. It doesn't take long. The
> > addresses in the stack trace that are enclosed in square brackets are
> > likely to be return addresses in the function-call trace.
>
> This is sufficiently tedious that if this happens again I'm going to
> either run screaming or write a ksymoops for xen.
>
> >
> > 'objdump -d xen >xen.s'. Then you can search in xen.s with a text
> > editor to find the call-trace addresses.
>
> I did this and got what you see below. It looks like to backtraces
> interleaved. All of the values in brackets are legitimate return
> addresses (they immediately follow a call instruction). "function addr"
> is the address of the function itself and "ret addr" is the address
> taken from the oops.
>
> function function addr ret addr
> ================================================
> putchar fc5095be fc5095ef
> e100_rx_srv fc532048 fc53240a
> printf fc5095f7 fc509664
> putchar_serial fc50927c fc509299
> e100intr fc531d8f fc531ef0
> handle_IRQ_event fc5b1a25 fc5b1a7d
> do_IRQ fc5b1bbb fc5b1c43
> call_do_IRQ fc5af4bb fc5af4c0
> serial_rx_int fc51801d fc518078
> serial_rx_int fc51801d fc518046
> handle_IRQ_event fc5b1a25 fc5b1a7d
> reprogram_ac_timer fc5af087 fc5af0aa
> do_IRQ fc5b1bbb fc5b1c43
> ac_timer_softirq_action fc50455c fc50465b
> call_do_IRQ fc5af4bb fc5af4c0
> default_idle fc5b585c fc5b582e
> continue_cpu_idle_loop fc5b585f fc5b5898
>
>
> The fault instruction is this:
> fc532927: 66 83 38 00 cmpw $0x0,(%eax)
> It is in e100_start_ru. Obviously eax is pointing at some piece of
> unmapped memory. I'm not sufficiently versed in assembler, particularly
> optimized, to tell where in we are going wrong:
>
>
> list_for_each(entry_ptr, &(bdp->active_rx_list)) {
> rx_struct =
> list_entry(entry_ptr, struct rx_list_elem, list_elem);
> pci_dma_sync_single(bdp->pdev, rx_struct->dma_addr,
> bdp->rfd_size, PCI_DMA_FROMDEVICE);
> if (!((SKB_RFD_STATUS(rx_struct->skb, bdp) &
> __constant_cpu_to_le16(RFD_STATUS_COMPLETE)))) {
> buffer_found = 1;
> break;
> }
> }
>
> Could the list have been corrupted?
>
>
> -Kip
>
>
> >
> > -- Keir
> >
> > > After a few more minutes the following popped out on the console:
> > >
> > > CPU: 1
> > > EIP: 0808:[<fc532927>]
> > > EFLAGS: 00010206
> > > eax: 0a725012 ebx: 00000010 ecx: fc657560 edx: fc76a460
> > > esi: fc76a460 edi: fc657540 ebp: 00000000 esp: fc64fda0
> > > ds: 0810 es: 0810 fs: 0810 gs: 0810 ss: 0810
> > > Stack trace from ESP=fc64fda0:
> > > ff865012 0000000a [fc5095ef] fc780140 0000003c fc657540 fc657540 [fc53240a]
> > > fc657540 fc657400 [fc509664] 0000000a [fc509299] fc648040 00000040 0000003e
> > > fc657400 fc780140 00000040 fc76a740 04000001 fc657540 00005048 [fc531ef0]
> > > fc657540 00000046 [fc509664] 0000000a 30303030 74203130 00000046 fc76a740
> > > 04000001 fc64fe90 00000010 [fc5b1a7d] 00000010 fc657400 fc64fe90 3d6e6670
> > > 33303030 63203462 00000001 fc76a740 fc600200 00000010 fc64fe90 [fc5b1c43]
> > > 00000010 fc64fe90 fc76a740 0007fff0 000003b4 25c4fe2d 00000001 0007fff0
> > > 0007fff0 00000000 00000000 [fc5af4c0] 0007fff0 fd800000 00000001 0007fff0
> > > 00000000 00000000 00000040 00010810 00000810 00000810 fc500810 ffffff10
> > > [fc50cff5] 00000808 00000202 fc654d4d 0000004d fc64ff6c [fc518078] 0000004d
> > > 00000000 fc64ff6c [fc518046] 0036bfec 00000000 00000292 fc654d00 02000001
> > > fc64ff6c 00000004 [fc5b1a7d] 00000004 00000000 fc64ff6c [fc5af0aa] fc650200
> > > 00000086 00000001 fc654d00 fc5fff00 00000004 fc64ff6c [fc5b1c43] 00000004
> > > fc64ff6c fc654d00 [fc50465b] 35c9c161 50d04d38 00000001 00000040 fc648040
> > > 00000040 fc7b8080 [fc5af4c0] 00000040 00000028 00000040 fc648040 00000040
> > > fc7b8080 00000040 fc640810 fc640810 00000810 fc7b0810 ffffff04 [fc5b585c]
> > > 00000808 00000246 [fc5b5898] fc648040 004c4b40 ffffffff 61007372 69745f63
> > > 5f72656d 74666f73 5f717269 69746361 64006e6f 5f706d75 656d6974 62007172
> > > 636f6c72 00632e6b 736e6f63 2e656c6f 65640063 2e677562 65640063 fc648040
> > >
> > > ****************************************
> > > CPU1 FATAL PAGE FAULT
> > > [error_code=00000000]
> > > Faulting linear address might be 0a725012
> > > Aieee! CPU1 is toast...
> > > ****************************************
> > >
> > > Is this oops from Xen or from XenoLinux? I downloaded the latest
> > > ksymoops and did the following:
> > > kmacy@xentap ./ksymoops -v ../xenolinux-2.4.25/vmlinux -m ../xenolinux-2.4.25/System.map < ../xeno-unstable.bk.home/tools/xc/lib/crash1.txt
> > > ksymoops 2.4.9 on i686 2.4.25-xeno. Options used
> > > -v ../xenolinux-2.4.25/vmlinux (specified)
> > > -k /proc/ksyms (default)
> > > -l /proc/modules (default)
> > > -o /lib/modules/2.4.25-xeno/ (default)
> > > -m ../xenolinux-2.4.25/System.map (specified)
> > >
> > > No modules in ksyms, skipping objects
> > > Warning (read_lsmod): no symbols in lsmod, is /proc/modules a valid
> > > lsmod file?
> > > Warning (compare_maps): mismatch on symbol state d, System.map says
> > > c0175ca8, vmlinux says 0. Ignoring System.map entry
> > > Warning (compare_maps): mismatch on symbol state a, vmlinux says 0,
> > > System.map says c0175ca8. Ignoring System.map entry
> > > CPU: 1
> > > EIP: 0808:[<fc532927>]
> > > Using defaults from ksymoopsSegmentation fault
> > >
> > >
> > > -Kip
> > >
> > > On Mon, 23 Feb 2004, Kip Macy wrote:
> > >
> > > > I had just tested my domain builder for the nth time on xeno-unstable
> > > > (very latest source), when I saw the messages below on the console.
> > > > DOM0 no longer responds to ping - I'm hoping that it will recover,
> > > > however, in all likelihood I will be hitting the rpb in a few minutes.
> > > >
> > > > audit_all_pages
> > > > zombie: pfn=00000000 cf=fffffffd tf=fffffffd dom=00000000
> > > > refcount error: pfn=000000 cf=fffffffd refcount=1
> > > > audit page: pfn=0 info: cf=fffffffd tf=fffffffd ts=0 dom=0
> > > >
> > > > refcount error: pfn=000247 cf=00000001 refcount=0
> > > > audit page: pfn=247 info: cf=1 tf=f0000001 ts=0 dom=fc648040
> > > >
> > > > refcount error: pfn=00024d cf=00000001 refcount=0
> > > > audit page: pfn=24d info: cf=1 tf=f0000001 ts=0 dom=fc648040
> > > >
> > > > refcount error: pfn=00036f cf=40000002 refcount=1
> > > > audit page: pfn=36f info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
> > > > pte_pfn=002207 cf=10000026 tf=30000024 dom=fc648be0
> > > > pte_idx=3f9 *pte_idx=0036f063
> > > >
> > > > refcount error: pfn=000371 cf=40000002 refcount=1
> > > > audit page: pfn=371 info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
> > > > pte_pfn=002207 cf=10000026 tf=30000024 dom=fc648be0
> > > > pte_idx=3fe *pte_idx=00371063
> > > >
> > > > refcount error: pfn=000372 cf=40000002 refcount=1
> > > > audit page: pfn=372 info: cf=40000002 tf=f0000002 ts=0 dom=fc648be0
> > > > pte_pfn=002207 cf=10000026 tf=30000024 dom=fc648be0
> > > > pte_idx=3fd *pte_idx=00372063
> > > >
> > > > refcount error: pfn=000390 cf=00000001 refcount=0
> > > > audit page: pfn=390 info: cf=1 tf=f0000001 ts=4ddca dom=fc649780
> > > >
> > > > refcount error: pfn=000392 cf=00000001 refcount=0
> > > > audit page: pfn=392 info: cf=1 tf=f0000001 ts=4ddca dom=fc649780
> > > >
> > > > refcount error: pfn=000393 cf=00000001 refcount=0
> > > > audit page: pfn=393 info: cf=1 tf=f0000001 ts=4ae4c dom=fc64a320
> > > >
> > > > refcount error: pfn=000395 cf=00000001 refcount=0
> > > > audit page: pfn=395 info: cf=1 tf=f0000001 ts=0 dom=fc64a320
> > > >
> > > > refcount error: pfn=00039f cf=00000001 refcount=0
> > > > audit page: pfn=39f info: cf=1 tf=f0000001 ts=0 dom=fc64aec0
> > > >
> > > > refcount error: pfn=0003a1 cf=00000001 refcount=0
> > > > audit page: pfn=3a1 info: cf=1 tf=f0000001 ts=0 dom=fc64aec0
> > > >
> > > > refcount error: pfn=0003a2 cf=00000001 refcount=0
> > > > audit page: pfn=3a2 info: cf=1 tf=f0000001 ts=0 dom=fc7a4060
> > > >
> > > > refcount error: pfn=0003a8 cf=00000001 refcount=0
> > > > audit page: pfn=3a8 info: cf=1 tf=f0000001 ts=0 dom=fc7a4060
> > > >
> > > > refcount error: pfn=0003a9 cf=00000001 refcount=0
> > > > audit page: pfn=3a9 info: cf=1 tf=f0000001 ts=0 dom=fc7a4c00
> > > >
> > > > refcount error: pfn=0003ab cf=00000001 refcount=0
> > > > audit page: pfn=3ab info: cf=1 tf=f0000001 ts=0 dom=fc7a4c00
> > > >
> > > > refcount error: pfn=0003ac cf=00000001 refcount=0
> > > > audit page: pfn=3ac info: cf=1 tf=f0000001 ts=191ab3 dom=fc7a57a0
> > > >
> > > > refcount error: pfn=0003ae cf=00000001 refcount=0
> > > > audit page: pfn=3ae info: cf=1 tf=f0000001 ts=191ab3 dom=fc7a57a0
> > > >
> > > > refcount error: pfn=0003af cf=00000001 refcount=0
> > > > audit page: pfn=3af info: cf=1 tf=f0000001 ts=191ab2 dom=fc7a6340
> > > >
> > > > refcount error: pfn=0003b1 cf=00000001 refcount=0
> > > > audit page: pfn=3b1 info: cf=1 tf=f0000001 ts=0 dom=fc7a6340
> > > >
> > > > refcount error: pfn=0003b2 cf=00000001 refcount=0
> > > > audit page: pfn=3b2 info: cf=1 tf=f0000001 ts=0 dom=fc7a6ee0
> > > >
> > > > refcount error: pfn=0003b4 cf=00000001 refcount=0
> > > > audit page: pfn=3b4 info: cf=1 tf=f0000001 ts=0 dom=fc7a6ee0
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -------------------------------------------------------
> > > > SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> > > > Build and deploy apps & Web services for Linux with
> > > > a free DVD software kit from IBM. Click Now!
> > > > http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/xen-devel
> > > >
> > >
> > >
> > > -------------------------------------------------------
> > > SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> > > Build and deploy apps & Web services for Linux with
> > > a free DVD software kit from IBM. Click Now!
> > > http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
> >
> >
> > -------------------------------------------------------
> > SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> > Build and deploy apps & Web services for Linux with
> > a free DVD software kit from IBM. Click Now!
> > http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
>
>
> -------------------------------------------------------
> SF.Net is sponsored by: Speed Start Your Linux Apps Now.
> Build and deploy apps & Web services for Linux with
> a free DVD software kit from IBM. Click Now!
> http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>
-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
next prev parent reply other threads:[~2004-02-24 3:44 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2004-02-22 20:46 dumping a domain's core Kip Macy
2004-02-22 21:02 ` Ian Pratt
2004-02-22 21:17 ` Kip Macy
2004-02-22 21:38 ` Ian Pratt
2004-02-22 22:02 ` DEBUG output starving network requests Kip Macy
2004-02-22 22:23 ` Keir Fraser
2004-02-22 23:35 ` problems with recursively mapping page directory as a page table Kip Macy
2004-02-23 13:15 ` Christian Limpach
2004-02-23 21:02 ` refcount errors then crash on XenoLinux with the latest source Kip Macy
2004-02-23 21:36 ` Kip Macy
2004-02-23 23:35 ` Keir Fraser
2004-02-24 1:11 ` Kip Macy
2004-02-24 3:44 ` Kip Macy [this message]
2004-02-24 8:15 ` Ian Pratt
2004-02-24 8:35 ` Keir Fraser
2004-02-24 17:21 ` Kip Macy
2004-02-24 17:45 ` Ian Pratt
2004-02-24 8:40 ` Keir Fraser
-- strict thread matches above, loose matches on Subject: below --
2004-02-23 23:52 Neugebauer, Rolf
2004-02-24 0:21 ` Kip Macy
2004-02-24 18:42 ` Rolf Neugebauer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20040223194202.P71501@demos.bsdclusters.com \
--to=kmacy@fsmware.com \
--cc=Ian.Pratt@cl.cam.ac.uk \
--cc=Keir.Fraser@cl.cam.ac.uk \
--cc=xen-devel@lists.sourceforge.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.