From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: George Dunlap <George.Dunlap@eu.citrix.com>
Cc: Jan Beulich <JBeulich@suse.com>,
"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Subject: Re: Xen 4.3 + tmem = Xen BUG at domain_page.c:143
Date: Wed, 12 Jun 2013 10:13:26 -0400 [thread overview]
Message-ID: <20130612141326.GA7675@phenom.dumpdata.com> (raw)
In-Reply-To: <CAFLBxZb0F8SGc2fGEQQSG_AofvjdO-oEXHH4G+L4ADwvof0k4g@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 7132 bytes --]
On Wed, Jun 12, 2013 at 12:37:53PM +0100, George Dunlap wrote:
> On Wed, Jun 12, 2013 at 12:00 PM, George Dunlap
> <George.Dunlap@eu.citrix.com> wrote:
> > This is probably not something we need to fix for 4.3, but we should
> > put it on our to-do list.
>
> Konrad,
>
> Could you try the attached patch with your debug patch, to see if it
> successfully falls back to the "replace a map hash" patch?
It does, but I believe it falls in the BUG_ON scenario where
the idx = 32 and the BUG_ON gets hit. Please see the serial log:
(attached is the debug patch + your patch)
(XEN) domain_page.c:137:d1 mfn: 1eb533, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 1eb864, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 1eb862, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 1eb531, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 1ebd14, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 18e8bb, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 2062e2, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 1eb525, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 20653e, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 1ed3a6, [0]: ff000000->ff000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
(XEN) domain_page.c:137:d1 mfn: 1ed3a6, [0]: ff000000->7f000000, garbage: 80000000 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffff7fffffff
(XEN) domain_page.c:137:d1 mfn: 1eb525, [0]: ff000000->df000000, garbage: 20000000 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffdfffffff
(XEN) domain_page.c:137:d1 mfn: 1ed3a4, [0]: df000000->df000000, garbage: 0 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffffffffff
BusyBox v1.14.3 (2013-06-10 16:30:12 EDT) built-in shell (ash)
Enter 'help' for a list of built-in commands.
#
(XEN) domain_page.c:137:d1 mfn: 18e8c8, [0]: ff000000->7f000000, garbage: 80000000 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffff7fffffff
(XEN) domain_page.c:137:d1 mfn: 1eb531, [0]: ff000000->ef000000, garbage: 10000000 -> ~0 (mask: 0), idx: 0, ~garbage: ffffffffeb531) -> 1 idx: 32(i:1,j:8), branch:a accum: 0x0
(XEN) domain_page.c:222:d1 [0] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [1] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [2] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [3] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [4] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [5] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [6] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [7] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [8] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [9] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [10] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [11] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [12] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [13] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [14] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [15] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [16] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [17] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [18] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [19] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [20] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [21] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [22] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [23] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [24] mfn=1eb531, [EIP=0]
(XEN) domain_page.c:222:d1 [25] mfn=2062e2, [EIP=0]
(XEN) domain_page.c:222:d1 [26] mfn=18e8bb, [EIP=0]
(XEN) domain_page.c:222:d1 [27] mfn=18e8c8, [EIP=0]
(XEN) domain_page.c:222:d1 [28] mfn=0, [EIP=0]
(XEN) domain_page.c:222:d1 [29] mfn=1ed3a5, [EIP=0]
(XEN) domain_page.c:222:d1 [30] mfn=1ed3a6, [EIP=0]
(XEN) domain_page.c:222:d1 [31] mfn=1ebc14, [EIP=0]
(XEN) Xen BUG at domain_page.c:226
(XEN) ----[ Xen-4.3-unstable x86_64 debug=y Tainted: C ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff82c4c01609bf>] map_domain_page+0x920/0xa03
(XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff8300c68f8000 rcx: 0000000000000000
(XEN) rdx: ffff82c4c03191a0 rsi: 000000000000000a rdi: ffff82c4c027a6e8
(XEN) rbp: ffff82c4c02c7e58 rsp: ffff82c4c02c7d78 r8: 0000000000000004
(XEN) r9: 0000000000000004 r10: 0000000000000004 r11: 0000000000000001
(XEN) r12: ffff83022d7b7000 r13: 00000000001eb531 r14: 0000000000000020
(XEN) r15: 0000000000000020 cr0: 000000008005003b cr4: 00000000000426f0
(XEN) cr3: 000000020653e000 cr2: 00007f808ece33e0
(XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008
(XEN) Xen stack trace from rsp=ffff82c4c02c7d78:
(XEN) ffff82c4c02f0de0 0000000000000001 0000000000000008 000000000000000a
(XEN) 0000000000000000 ffff830000000000 ffffffffefffffff 0000000000000000
(XEN) 0000000000000000 0000002000000000 00000000ff000000 00000000001eb531
(XEN) ffff82c4c02c7e20 0000000000000020 ffff82c4c02c7e18 ffff8300c68f83c8
(XEN) 0000000ac02c7f08 ffff82c400000000 ffff83022d7b72d8 0000000000000286
(XEN) ffff82c4c03191c0 0000000000000002 0000000900000000 ffff83022d7b7000
(XEN) ffff83022d7b7000 ffff8300c68f8000 0000000000000000 ffff88003f904000
(XEN) ffff82c4c02c7ee8 ffff82c4c017a242 ffffffff81039c19 00000000000426f0
(XEN) 80100001ebc14065 ffff82c4c02c0000 ffffffffffffffff ffff82c4c02c7ed0
(XEN) ffff82c4c0126e77 ffff82c4c02e0000 ffff82c4c02c00e7 00000000001eb531
(XEN) ffffffffffffffff ffff8300c68f8000 0000000000000000 ffff88003e29fe40
(XEN) ffff88003e29fee0 0000000000000080 ffff82c4c02c7ef8 ffff82c4c017a50f
(XEN) 00007d3b3fd380c7 ffff82c4c0223c4b ffffffff810011ca 000000000000000e
(XEN) ffff88003e27f800 ffff88003f90c740 0000000000000001 ffff88003f911640
(XEN) ffff88003e29fe30 ffff88003f904000 0000000000000286 0000000000000000
(XEN) ffff880000000000 00003ffffffff000 000000000000000e ffffffff810011ca
(XEN) 0000000000000000 80100001ebc14065 ffff88003f904000 0001010000000000
(XEN) ffffffff810011ca 000000000000e033 0000000000000286 ffff88003e29fdf8
(XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 ffff8300c68f8000 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82c4c01609bf>] map_domain_page+0x920/0xa03
(XEN) [<ffff82c4c017a242>] __do_update_va_mapping+0xd6/0x316
(XEN) [<ffff82c4c017a50f>] do_update_va_mapping+0x1e/0x22
(XEN) [<ffff82c4c0223c4b>] syscall_enter+0xeb/0x145
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at domain_page.c:226
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)
[-- Attachment #2: xen-domain_page-v5.patch --]
[-- Type: text/plain, Size: 7487 bytes --]
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index efda6af..d6eb783 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -14,6 +14,7 @@
#include <asm/current.h>
#include <asm/flushtlb.h>
#include <asm/hardirq.h>
+#include <xen/symbols.h>
static struct vcpu *__read_mostly override;
@@ -59,12 +60,12 @@ void __init mapcache_override_current(struct vcpu *v)
void *map_domain_page(unsigned long mfn)
{
unsigned long flags;
- unsigned int idx, i;
+ unsigned int idx, i, j = 0;
struct vcpu *v;
struct mapcache_domain *dcache;
struct mapcache_vcpu *vcache;
struct vcpu_maphash_entry *hashent;
-
+ int branch = 0;
#ifdef NDEBUG
if ( mfn <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
return mfn_to_virt(mfn);
@@ -110,36 +111,118 @@ void *map_domain_page(unsigned long mfn)
idx = find_next_zero_bit(dcache->inuse, dcache->entries, dcache->cursor);
if ( unlikely(idx >= dcache->entries) )
{
- unsigned long accum = 0;
+ unsigned long accum = 0, mask;
+ int entries_rem;
+ entries_rem = dcache->entries;
/* /First/, clean the garbage map and update the inuse list. */
for ( i = 0; i < BITS_TO_LONGS(dcache->entries); i++ )
{
+ unsigned long garbage = dcache->garbage[i];
+ unsigned long _inuse = dcache->inuse[i];
+
+ ASSERT(entries_rem > 0);
+
+ barrier();
+
dcache->inuse[i] &= ~xchg(&dcache->garbage[i], 0);
- accum |= ~dcache->inuse[i];
+ mask = (entries_rem >= 64) ? ~0UL : ((1 << entries_rem) - 1);
+ accum |= (~dcache->inuse[i]) & mask;
+ entries_rem -= BITS_PER_LONG;
+
+ if (v->domain->domain_id) {
+ if (~dcache->inuse[i]) {
+ gdprintk(XENLOG_INFO, "mfn: %lx, [%d]: %lx->%lx, garbage: %lx -> ~%lx (mask: %lx), idx: %d, ~garbage: %lx \n",
+ mfn, i, _inuse, dcache->inuse[i], garbage, accum, mask,
+ find_first_zero_bit(dcache->inuse, dcache->entries),~garbage);
+ branch |= 8;
+ }
+ }
}
- if ( accum )
+ if ( accum ) {
idx = find_first_zero_bit(dcache->inuse, dcache->entries);
+ branch |= 1;
+ }
else
{
+ branch |= 2;
/* Replace a hash entry instead. */
i = MAPHASH_HASHFN(mfn);
do {
hashent = &vcache->hash[i];
if ( hashent->idx != MAPHASHENT_NOTINUSE && !hashent->refcnt )
{
+ branch |= 4;
idx = hashent->idx;
ASSERT(l1e_get_pfn(MAPCACHE_L1ENT(idx)) == hashent->mfn);
l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
hashent->idx = MAPHASHENT_NOTINUSE;
hashent->mfn = ~0UL;
+ if (idx >= dcache->entries) {
+ branch |= 8;
+ gdprintk(XENLOG_INFO, "mfn (%lx) -> %ld idx (iter:%d)\n", mfn, MAPHASH_HASHFN(mfn), j);
+
+ for (i = 0; i < MAPHASH_ENTRIES;i++) {
+ hashent = &vcache->hash[i];
+
+ gdprintk(XENLOG_INFO, "[%d] idx=%d, mfn=0x%lx, refcnt: %d\n",
+ i, hashent->idx, hashent->mfn, hashent->refcnt);
+ }
+ }
break;
}
if ( ++i == MAPHASH_ENTRIES )
i = 0;
+ j++;
} while ( i != MAPHASH_HASHFN(mfn) );
}
+ if (idx >= dcache->entries) {
+ unsigned long _mfn;
+ const char *name;
+ unsigned long offset, size;
+
+ static char namebuf[KSYM_NAME_LEN+1];
+ #define BUFFER_SIZE sizeof("%s+%#lx/%#lx [%s]") + KSYM_NAME_LEN + \
+ 2*(BITS_PER_LONG*3/10) + 1
+ static char buffer[BUFFER_SIZE];
+
+
+
+ gdprintk(XENLOG_INFO, "mfn (%lx) -> %ld idx: %d(i:%d,j:%d), branch:%x accum: 0x%lx\n",
+ mfn, MAPHASH_HASHFN(mfn), idx, i, j, branch, accum);
+
+ for (i = 0; i < dcache->entries; i++) {
+ int in_mapcache = 0;
+ _mfn = l1e_get_pfn(MAPCACHE_L1ENT(i));
+ hashent = NULL;
+
+ for (j = 0; j < MAPHASH_ENTRIES;j++) {
+ hashent = &vcache->hash[j];
+
+ if (hashent->mfn == _mfn) {
+ in_mapcache = 1;
+ break;
+ }
+ }
+ if (i <= 32 && dcache->ips[i]) {
+ name = symbols_lookup(dcache->ips[i], &size, &offset, namebuf);
+
+ if (!name)
+ snprintf(buffer, BUFFER_SIZE, "0x%lx", dcache->ips[i]);
+ else
+ snprintf(buffer, BUFFER_SIZE, "%s+%#lx/%#lx", name, offset, size);
+ } else
+ snprintf(buffer, BUFFER_SIZE, "EIP=0");
+
+ if (in_mapcache)
+ gdprintk(XENLOG_INFO, "[%d] mfn=0x%lx idx=%d, mfn=0x%lx, refcnt: %d [%s]\n",
+ i, _mfn, hashent->idx, hashent->mfn, hashent->refcnt, buffer);
+ else
+ gdprintk(XENLOG_INFO, "[%d] mfn=%lx, [%s]\n", i, _mfn, buffer);
+ }
+
+ }
BUG_ON(idx >= dcache->entries);
/* /Second/, flush TLBs. */
@@ -152,6 +235,9 @@ void *map_domain_page(unsigned long mfn)
set_bit(idx, dcache->inuse);
dcache->cursor = idx + 1;
+ if (v->domain->domain_id && idx <= 32)
+ dcache->ips[idx] = (unsigned long)__builtin_return_address(0);
+
spin_unlock(&dcache->lock);
l1e_write(&MAPCACHE_L1ENT(idx), l1e_from_pfn(mfn, __PAGE_HYPERVISOR));
@@ -215,7 +301,8 @@ void unmap_domain_page(const void *ptr)
/* /Second/, mark as garbage. */
set_bit(idx, dcache->garbage);
}
-
+ if (v->domain->domain_id && idx <= 32)
+ dcache->ips[idx] = 0;
local_irq_restore(flags);
}
@@ -254,6 +341,7 @@ int mapcache_domain_init(struct domain *d)
2 * PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long))) >
MAPCACHE_VIRT_START + (PERDOMAIN_SLOT_MBYTES << 20));
bitmap_pages = PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long));
+ gdprintk(XENLOG_INFO, "domain bitmap pages: %d\n", bitmap_pages);
dcache->inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
dcache->garbage = dcache->inuse +
(bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
@@ -276,6 +364,7 @@ int mapcache_vcpu_init(struct vcpu *v)
if ( is_hvm_vcpu(v) || !dcache->inuse )
return 0;
+ gdprintk(XENLOG_INFO, "ents: %d, entries: %d\n", ents, dcache->entries);
if ( ents > dcache->entries )
{
/* Populate page tables. */
diff --git a/xen/include/asm-x86/domain.h b/xen/include/asm-x86/domain.h
index d79464d..5389ea9 100644
--- a/xen/include/asm-x86/domain.h
+++ b/xen/include/asm-x86/domain.h
@@ -67,6 +67,7 @@ struct mapcache_domain {
/* Which mappings are in use, and which are garbage to reap next epoch? */
unsigned long *inuse;
unsigned long *garbage;
+ unsigned long ips[33];
};
int mapcache_domain_init(struct domain *);
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2013-06-12 14:13 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-06-11 13:45 Xen 4.3 + tmem = Xen BUG at domain_page.c:143 konrad wilk
2013-06-11 14:46 ` Jan Beulich
2013-06-11 15:30 ` konrad wilk
2013-06-11 15:56 ` George Dunlap
2013-06-11 16:38 ` Jan Beulich
2013-06-11 17:30 ` konrad wilk
2013-06-11 18:52 ` konrad wilk
2013-06-11 21:06 ` konrad wilk
2013-06-12 6:38 ` Jan Beulich
2013-06-12 11:00 ` George Dunlap
2013-06-12 11:15 ` Processed: " xen
2013-06-12 11:37 ` George Dunlap
2013-06-12 12:46 ` Jan Beulich
2013-06-12 14:13 ` Konrad Rzeszutek Wilk [this message]
2013-06-12 12:12 ` Jan Beulich
2013-06-12 13:16 ` George Dunlap
2013-06-12 13:27 ` Jan Beulich
2013-06-12 15:11 ` Keir Fraser
2013-06-12 15:27 ` Keir Fraser
2013-06-12 15:54 ` Jan Beulich
2013-06-12 15:48 ` Jan Beulich
2013-06-12 17:26 ` Keir Fraser
2013-07-05 16:56 ` George Dunlap
2013-07-08 8:58 ` Jan Beulich
2013-07-08 9:07 ` George Dunlap
2013-07-08 9:15 ` Processed: " xen
2013-07-08 9:25 ` George Dunlap
2013-07-08 9:30 ` Processed: " xen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130612141326.GA7675@phenom.dumpdata.com \
--to=konrad.wilk@oracle.com \
--cc=George.Dunlap@eu.citrix.com \
--cc=JBeulich@suse.com \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).