[PATCH 0 of 5] update xenctx to dump pagetables

[PATCH 0 of 5] update xenctx to dump pagetables

[Olaf Hering]

The following series adds an option -P to xenctx to dump the guests pagetable.
Another option to work an all vcpus is also added.

The reason for such a change is that on some systems a live migration from a
system with a small amount of memory to a system with a huge amount of memory
(>100GB) fails. DomU crashes on the target.
To get some debug data I'm trying to dump the pagetable, just to verify that
nothing bad to them happend during transit.

The code to walk the pagetables is/was based on
xc_translate_foreign_address(), but I still think I have some major bugs in
there (the last patch). I cant figure out why some l3/l2/l1 entries can not be mapped.
Any help with getting that output fixed for at least a a 64bit PV guest is much
appreciated.

Olaf


part of xenctx output:
...
  0 l4e  ff 00000000a1240067:  x   0     a1240000 0 0 1 A c t u W P
  0 l3e 1ff 00000000bc133067:  x   0     bc133000 0 0 s 1 A c t U W P
  0 l2e  64 00000000a1471067:  x   0     a1471000 0 0 s 1 A c t U W P
  0 l1e  46 80000000bd56f167:  X   0     bd56f000 0 G p s D A c t U W P     7fffcc846000
  0 l1e  47 80000000bd57c167:  X   0     bd57c000 0 G p s D A c t U W P     7fffcc847000
  0 l1e  48 80000000bd5af165:  X   0     bd5af000 0 G p s D A c t U w P     7fffcc848000
  0 l1e  db 00000000a1972125:  x   0     a1972000 0 G p s d A c t U w P     7fffcc8db000
  0 l4e 100 000000013fff8067:  x   0    13fff8000 0 0 1 A c t u W P
xc_map_foreign_range l3: Invalid argument
  0 l4e 101 000000013fff0063:  x   0    13fff0000 0 0 1 A c t u W P
xc_map_foreign_range l3: Invalid argument
  0 l4e 102 00000000bc5b0063:  x   0     bc5b0000 0 0 1 A c t u W P
  0 l3e  fe 00000000a7ed2067:  x   0     a7ed2000 0 0 s 1 A c t U W P
  0 l2e 163 00000000a154a067:  x   0     a154a000 0 0 s 1 A c t U W P
  0 l1e 1eb 00000000bc1a7067:  x   0     bc1a7000 0 g p s D A c t U W P     813fac7eb000
  0 l1e 1ec 00000000a154f067:  x   0     a154f000 0 g p s D A c t U W P     813fac7ec000
...

in dmesg:
...
(XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=19
(XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=19
...

Olaf

-- 

[PATCH 1/5] xenctx: recognize also -S option for --stack-trace
[PATCH 2/5] xenctx: move all globals into struct xenctx
[PATCH 3/5] xenctx: move xc_* access out of dump_ctx
[PATCH 4/5] xenctx: add option -C to dump context for all vcpus
[PATCH 5/5] xenctx: dump pagetable

 tools/xentrace/xenctx.c |  405 ++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 329 insertions(+), 76 deletions(-)

[PATCH 1 of 5] xenctx: recognize also -S option for --stack-trace

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1308312572 -7200
# Node ID 434e772168bc2208e1b9219d690b5e4909b552a7
# Parent  eca057e4475ca455ec36f962b9179fd2c9674196
xenctx: recognize also -S option for --stack-trace

Update help text.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r eca057e4475c -r 434e772168bc tools/xentrace/xenctx.c
--- a/tools/xentrace/xenctx.c	Fri Jun 17 08:08:13 2011 +0100
+++ b/tools/xentrace/xenctx.c	Fri Jun 17 14:09:32 2011 +0200
@@ -924,7 +924,7 @@ static void usage(void)
     printf("                    frame pointers.\n");
     printf("  -s SYMTAB, --symbol-table=SYMTAB\n");
     printf("                    read symbol table from SYMTAB.\n");
-    printf("  --stack-trace     print a complete stack trace.\n");
+    printf("  -S --stack-trace  print a complete stack trace.\n");
     printf("  -k, --kernel-start\n");
     printf("                    set user/kernel split. (default 0xc0000000)\n");
 #ifdef __ia64__
@@ -938,7 +938,7 @@ static void usage(void)
 int main(int argc, char **argv)
 {
     int ch;
-    static const char *sopts = "fs:hak:"
+    static const char *sopts = "fs:hak:S"
 #ifdef __ia64__
         "r:"
 #endif

[PATCH 2 of 5] xenctx: move all globals into struct xenctx

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1308312618 -7200
# Node ID 132480af62e4cb35d24350cf6ea1c2769f1cd1b5
# Parent  434e772168bc2208e1b9219d690b5e4909b552a7
xenctx: move all globals into struct xenctx

Move all globals used for options and libxc data into a new struct xenctx.
This is used in subsequent changes.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 434e772168bc -r 132480af62e4 tools/xentrace/xenctx.c
--- a/tools/xentrace/xenctx.c	Fri Jun 17 14:09:32 2011 +0200
+++ b/tools/xentrace/xenctx.c	Fri Jun 17 14:10:18 2011 +0200
@@ -29,11 +29,14 @@
 #include <xen/foreign/x86_64.h>
 #include <xen/hvm/save.h>
 
-xc_interface *xc_handle = 0;
-int domid = 0;
-int frame_ptrs = 0;
-int stack_trace = 0;
-int disp_all = 0;
+static struct xenctx {
+    xc_interface *xc_handle;
+    int domid;
+    int frame_ptrs;
+    int stack_trace;
+    int disp_all;
+    xc_dominfo_t dominfo;
+} xenctx;
 
 #if defined (__i386__) || defined (__x86_64__)
 typedef unsigned long long guest_word_t;
@@ -300,7 +303,7 @@ static void print_ctx_32(vcpu_guest_cont
     printf(" fs:     %04x\t", regs->fs);
     printf(" gs:     %04x\n", regs->gs);
 
-    if (disp_all) {
+    if (xenctx.disp_all) {
         print_special(ctx->ctrlreg, "cr", 0x1d, 4);
         print_special(ctx->debugreg, "dr", 0xcf, 4);
     }
@@ -329,7 +332,7 @@ static void print_ctx_32on64(vcpu_guest_
     printf(" fs:     %04x\t", regs->fs);
     printf(" gs:     %04x\n", regs->gs);
 
-    if (disp_all) {
+    if (xenctx.disp_all) {
         print_special(ctx->ctrlreg, "cr", 0x1d, 4);
         print_special(ctx->debugreg, "dr", 0xcf, 4);
     }
@@ -373,7 +376,7 @@ static void print_ctx_64(vcpu_guest_cont
     printf(" gs: %04x @ %016"PRIx64"/%016"PRIx64"\n", regs->gs,
            ctx->gs_base_kernel, ctx->gs_base_user);
 
-    if (disp_all) {
+    if (xenctx.disp_all) {
         print_special(ctx->ctrlreg, "cr", 0x1d, 8);
         print_special(ctx->debugreg, "dr", 0xcf, 8);
     }
@@ -681,7 +684,7 @@ static void *map_page(vcpu_guest_context
     static unsigned long previous_mfn = 0;
     static void *mapped = NULL;
 
-    unsigned long mfn = xc_translate_foreign_address(xc_handle, domid, vcpu, virt);
+    unsigned long mfn = xc_translate_foreign_address(xenctx.xc_handle, xenctx.domid, vcpu, virt);
     unsigned long offset = virt & ~XC_PAGE_MASK;
 
     if (mapped && mfn == previous_mfn)
@@ -692,7 +695,7 @@ static void *map_page(vcpu_guest_context
 
     previous_mfn = mfn;
 
-    mapped = xc_map_foreign_range(xc_handle, domid, XC_PAGE_SIZE, PROT_READ, mfn);
+    mapped = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, XC_PAGE_SIZE, PROT_READ, mfn);
 
     if (mapped == NULL) {
         fprintf(stderr, "failed to map page.\n");
@@ -764,21 +767,21 @@ static void print_stack(vcpu_guest_conte
     }
     printf("\n");
 
-    if(stack_trace)
+    if(xenctx.stack_trace)
         printf("Stack Trace:\n");
     else
         printf("Call Trace:\n");
-    printf("%c [<", stack_trace ? '*' : ' ');
+    printf("%c [<", xenctx.stack_trace ? '*' : ' ');
     print_stack_word(instr_pointer(ctx), width);
     printf(">] ");
 
     print_symbol(instr_pointer(ctx));
     printf(" <--\n");
-    if (frame_ptrs) {
+    if (xenctx.frame_ptrs) {
         stack = stack_pointer(ctx);
         frame = frame_pointer(ctx);
         while(frame && stack < stack_limit) {
-            if (stack_trace) {
+            if (xenctx.stack_trace) {
                 while (stack < frame) {
                     p = map_page(ctx, vcpu, stack);
                     printf("|   ");
@@ -792,7 +795,7 @@ static void print_stack(vcpu_guest_conte
 
             p = map_page(ctx, vcpu, stack);
             frame = read_stack_word(p, width);
-            if (stack_trace) {
+            if (xenctx.stack_trace) {
                 printf("|-- ");
                 print_stack_word(read_stack_word(p, width), width);
                 printf("\n");
@@ -802,7 +805,7 @@ static void print_stack(vcpu_guest_conte
             if (frame) {
                 p = map_page(ctx, vcpu, stack);
                 word = read_stack_word(p, width);
-                printf("%c [<", stack_trace ? '|' : ' ');
+                printf("%c [<", xenctx.stack_trace ? '|' : ' ');
                 print_stack_word(word, width);
                 printf(">] ");
                 print_symbol(word);
@@ -821,7 +824,7 @@ static void print_stack(vcpu_guest_conte
                 printf(">] ");
                 print_symbol(word);
                 printf("\n");
-            } else if (stack_trace) {
+            } else if (xenctx.stack_trace) {
                 printf("    ");
                 print_stack_word(word, width);
                 printf("\n");
@@ -836,37 +839,36 @@ static void dump_ctx(int vcpu)
 {
     int ret;
     vcpu_guest_context_any_t ctx;
-    xc_dominfo_t dominfo;
 
-    xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
+    xenctx.xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
 
-    ret = xc_domain_getinfo(xc_handle, domid, 1, &dominfo);
+    ret = xc_domain_getinfo(xenctx.xc_handle, xenctx.domid, 1, &xenctx.dominfo);
     if (ret < 0) {
         perror("xc_domain_getinfo");
         exit(-1);
     }
     
-    ret = xc_domain_pause(xc_handle, domid);
+    ret = xc_domain_pause(xenctx.xc_handle, xenctx.domid);
     if (ret < 0) {
         perror("xc_domain_pause");
         exit(-1);
     }
 
-    ret = xc_vcpu_getcontext(xc_handle, domid, vcpu, &ctx);
+    ret = xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &ctx);
     if (ret < 0) {
-        if (!dominfo.paused)
-            xc_domain_unpause(xc_handle, domid);
+        if (!xenctx.dominfo.paused)
+            xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
         perror("xc_vcpu_getcontext");
         exit(-1);
     }
 
 #if defined(__i386__) || defined(__x86_64__)
     {
-        if (dominfo.hvm) {
+        if (xenctx.dominfo.hvm) {
             struct hvm_hw_cpu cpuctx;
             xen_capabilities_info_t xen_caps = "";
             if (xc_domain_hvm_getcontext_partial(
-                    xc_handle, domid, HVM_SAVE_CODE(CPU), 
+                    xenctx.xc_handle, xenctx.domid, HVM_SAVE_CODE(CPU),
                     vcpu, &cpuctx, sizeof cpuctx) != 0) {
                 perror("xc_domain_hvm_getcontext_partial");
                 exit(-1);
@@ -874,7 +876,7 @@ static void dump_ctx(int vcpu)
             guest_word_size = (cpuctx.msr_efer & 0x400) ? 8 : 4;
             guest_protected_mode = (cpuctx.cr0 & CR0_PE);
             /* HVM guest context records are always host-sized */
-            if (xc_version(xc_handle, XENVER_capabilities, &xen_caps) != 0) {
+            if (xc_version(xenctx.xc_handle, XENVER_capabilities, &xen_caps) != 0) {
                 perror("xc_version");
                 exit(-1);
             }
@@ -882,9 +884,9 @@ static void dump_ctx(int vcpu)
         } else {
             struct xen_domctl domctl;
             memset(&domctl, 0, sizeof domctl);
-            domctl.domain = domid;
+            domctl.domain = xenctx.domid;
             domctl.cmd = XEN_DOMCTL_get_address_size;
-            if (xc_domctl(xc_handle, &domctl) == 0)
+            if (xc_domctl(xenctx.xc_handle, &domctl) == 0)
                 ctxt_word_size = guest_word_size = domctl.u.address_size.size / 8;
         }
     }
@@ -897,15 +899,15 @@ static void dump_ctx(int vcpu)
         print_stack(&ctx, vcpu, guest_word_size);
 #endif
 
-    if (!dominfo.paused) {
-        ret = xc_domain_unpause(xc_handle, domid);
+    if (!xenctx.dominfo.paused) {
+        ret = xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
         if (ret < 0) {
             perror("xc_domain_unpause");
             exit(-1);
         }
     }
 
-    ret = xc_interface_close(xc_handle);
+    ret = xc_interface_close(xenctx.xc_handle);
     if (ret < 0) {
         perror("xc_interface_close");
         exit(-1);
@@ -962,13 +964,13 @@ int main(int argc, char **argv)
     while ((ch = getopt_long(argc, argv, sopts, lopts, NULL)) != -1) {
         switch(ch) {
         case 'f':
-            frame_ptrs = 1;
+            xenctx.frame_ptrs = 1;
             break;
         case 's':
             symbol_table = optarg;
             break;
         case 'S':
-            stack_trace = 1;
+            xenctx.stack_trace = 1;
             break;
 #ifdef __ia64__
         case 'r':
@@ -1004,7 +1006,7 @@ int main(int argc, char **argv)
             break;
 #else
         case 'a':
-            disp_all = 1;
+            xenctx.disp_all = 1;
             break;
 #endif
         case 'k':
@@ -1026,8 +1028,8 @@ int main(int argc, char **argv)
         exit(-1);
     }
 
-    domid = atoi(argv[0]);
-    if (domid==0) {
+    xenctx.domid = atoi(argv[0]);
+    if (xenctx.domid==0) {
             fprintf(stderr, "cannot trace dom0\n");
             exit(-1);
     }

[PATCH 3 of 5] xenctx: move xc_* access out of dump_ctx

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1308312712 -7200
# Node ID 0e0c9fbedc2e29e2d507e2e1dce9536d06af442d
# Parent  132480af62e4cb35d24350cf6ea1c2769f1cd1b5
xenctx: move xc_* access out of dump_ctx

move xc_* access out of dump_ctx.
Update code paths to return an error instead of calling exit().
On error unpause the guest if it was paused by xenctx.

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 132480af62e4 -r 0e0c9fbedc2e tools/xentrace/xenctx.c
--- a/tools/xentrace/xenctx.c	Fri Jun 17 14:10:18 2011 +0200
+++ b/tools/xentrace/xenctx.c	Fri Jun 17 14:11:52 2011 +0200
@@ -35,6 +35,7 @@ static struct xenctx {
     int frame_ptrs;
     int stack_trace;
     int disp_all;
+    int self_paused;
     xc_dominfo_t dominfo;
 } xenctx;
 
@@ -699,7 +700,7 @@ static void *map_page(vcpu_guest_context
 
     if (mapped == NULL) {
         fprintf(stderr, "failed to map page.\n");
-        exit(-1);
+        return NULL;
     }
 
  out:
@@ -722,7 +723,7 @@ static void print_stack_word(guest_word_
         printf(FMT_64B_WORD, word);
 }
 
-static void print_code(vcpu_guest_context_any_t *ctx, int vcpu)
+static int print_code(vcpu_guest_context_any_t *ctx, int vcpu)
 {
     guest_word_t instr;
     int i;
@@ -732,6 +733,8 @@ static void print_code(vcpu_guest_contex
     instr -= 21;
     for(i=0; i<32; i++) {
         unsigned char *c = map_page(ctx, vcpu, instr+i);
+        if (!c)
+            return -1;
         if (instr+i == instr_pointer(ctx))
             printf("<%02x> ", *c);
         else
@@ -740,9 +743,10 @@ static void print_code(vcpu_guest_contex
     printf("\n");
 
     printf("\n");
+    return 0;
 }
 
-static void print_stack(vcpu_guest_context_any_t *ctx, int vcpu, int width)
+static int print_stack(vcpu_guest_context_any_t *ctx, int vcpu, int width)
 {
     guest_word_t stack = stack_pointer(ctx);
     guest_word_t stack_limit;
@@ -758,6 +762,8 @@ static void print_stack(vcpu_guest_conte
     for (i=1; i<5 && stack < stack_limit; i++) {
         while(stack < stack_limit && stack < stack_pointer(ctx) + i*32) {
             p = map_page(ctx, vcpu, stack);
+            if (!p)
+                return -1;
             word = read_stack_word(p, width);
             printf(" ");
             print_stack_word(word, width);
@@ -784,6 +790,8 @@ static void print_stack(vcpu_guest_conte
             if (xenctx.stack_trace) {
                 while (stack < frame) {
                     p = map_page(ctx, vcpu, stack);
+                    if (!p)
+                        return -1;
                     printf("|   ");
                     print_stack_word(read_stack_word(p, width), width);
                     printf("   \n");
@@ -794,6 +802,8 @@ static void print_stack(vcpu_guest_conte
             }
 
             p = map_page(ctx, vcpu, stack);
+            if (!p)
+                return -1;
             frame = read_stack_word(p, width);
             if (xenctx.stack_trace) {
                 printf("|-- ");
@@ -804,6 +814,8 @@ static void print_stack(vcpu_guest_conte
 
             if (frame) {
                 p = map_page(ctx, vcpu, stack);
+                if (!p)
+                    return -1;
                 word = read_stack_word(p, width);
                 printf("%c [<", xenctx.stack_trace ? '|' : ' ');
                 print_stack_word(word, width);
@@ -817,6 +829,8 @@ static void print_stack(vcpu_guest_conte
         stack = stack_pointer(ctx);
         while(stack < stack_limit) {
             p = map_page(ctx, vcpu, stack);
+            if (!p)
+                return -1;
             word = read_stack_word(p, width);
             if (is_kernel_text(word)) {
                 printf("  [<");
@@ -832,34 +846,17 @@ static void print_stack(vcpu_guest_conte
             stack += width;
         }
     }
+    return 0;
 }
 #endif
 
 static void dump_ctx(int vcpu)
 {
-    int ret;
     vcpu_guest_context_any_t ctx;
 
-    xenctx.xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
-
-    ret = xc_domain_getinfo(xenctx.xc_handle, xenctx.domid, 1, &xenctx.dominfo);
-    if (ret < 0) {
-        perror("xc_domain_getinfo");
-        exit(-1);
-    }
-    
-    ret = xc_domain_pause(xenctx.xc_handle, xenctx.domid);
-    if (ret < 0) {
-        perror("xc_domain_pause");
-        exit(-1);
-    }
-
-    ret = xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &ctx);
-    if (ret < 0) {
-        if (!xenctx.dominfo.paused)
-            xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
+    if (xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &ctx) < 0) {
         perror("xc_vcpu_getcontext");
-        exit(-1);
+        return;
     }
 
 #if defined(__i386__) || defined(__x86_64__)
@@ -871,14 +868,14 @@ static void dump_ctx(int vcpu)
                     xenctx.xc_handle, xenctx.domid, HVM_SAVE_CODE(CPU),
                     vcpu, &cpuctx, sizeof cpuctx) != 0) {
                 perror("xc_domain_hvm_getcontext_partial");
-                exit(-1);
+                return;
             }
             guest_word_size = (cpuctx.msr_efer & 0x400) ? 8 : 4;
             guest_protected_mode = (cpuctx.cr0 & CR0_PE);
             /* HVM guest context records are always host-sized */
             if (xc_version(xenctx.xc_handle, XENVER_capabilities, &xen_caps) != 0) {
                 perror("xc_version");
-                exit(-1);
+                return;
             }
             ctxt_word_size = (strstr(xen_caps, "xen-3.0-x86_64")) ? 8 : 4;
         } else {
@@ -894,24 +891,12 @@ static void dump_ctx(int vcpu)
 
     print_ctx(&ctx);
 #ifndef NO_TRANSLATION
-    print_code(&ctx, vcpu);
+    if (print_code(&ctx, vcpu))
+        return;
     if (is_kernel_text(instr_pointer(&ctx)))
-        print_stack(&ctx, vcpu, guest_word_size);
+        if (print_stack(&ctx, vcpu, guest_word_size))
+            return;
 #endif
-
-    if (!xenctx.dominfo.paused) {
-        ret = xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
-        if (ret < 0) {
-            perror("xc_domain_unpause");
-            exit(-1);
-        }
-    }
-
-    ret = xc_interface_close(xenctx.xc_handle);
-    if (ret < 0) {
-        perror("xc_interface_close");
-        exit(-1);
-    }
 }
 
 static void usage(void)
@@ -940,6 +925,7 @@ static void usage(void)
 int main(int argc, char **argv)
 {
     int ch;
+    int ret;
     static const char *sopts = "fs:hak:S"
 #ifdef __ia64__
         "r:"
@@ -1040,8 +1026,43 @@ int main(int argc, char **argv)
     if (symbol_table)
         read_symbol_table(symbol_table);
 
+    xenctx.xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
+    if (xenctx.xc_handle < 0) {
+        perror("xc_interface_open");
+        exit(-1);
+    }
+
+    ret = xc_domain_getinfo(xenctx.xc_handle, xenctx.domid, 1, &xenctx.dominfo);
+    if (ret < 0) {
+        perror("xc_domain_getinfo");
+        exit(-1);
+    }
+
+    if (!xenctx.dominfo.paused) {
+        ret = xc_domain_pause(xenctx.xc_handle, xenctx.domid);
+        if (ret < 0) {
+            perror("xc_domain_pause");
+            exit(-1);
+        }
+        xenctx.self_paused = 1;
+    }
+
     dump_ctx(vcpu);
 
+    if (xenctx.self_paused) {
+        ret = xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
+        if (ret < 0) {
+            perror("xc_domain_unpause");
+            exit(-1);
+        }
+    }
+
+    ret = xc_interface_close(xenctx.xc_handle);
+    if (ret < 0) {
+        perror("xc_interface_close");
+        exit(-1);
+    }
+
     return 0;
 }

[PATCH 4 of 5] xenctx: add option -C to dump context for all vcpus

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1308313376 -7200
# Node ID ddf16ea954876d8af371f9751fc06eb4c9e78b36
# Parent  0e0c9fbedc2e29e2d507e2e1dce9536d06af442d
xenctx: add option -C to dump context for all vcpus

Signed-off-by: Olaf Hering <olaf@aepfle.de>

diff -r 0e0c9fbedc2e -r ddf16ea95487 tools/xentrace/xenctx.c
--- a/tools/xentrace/xenctx.c	Fri Jun 17 14:11:52 2011 +0200
+++ b/tools/xentrace/xenctx.c	Fri Jun 17 14:22:56 2011 +0200
@@ -35,6 +35,7 @@ static struct xenctx {
     int frame_ptrs;
     int stack_trace;
     int disp_all;
+    int all_vcpus;
     int self_paused;
     xc_dominfo_t dominfo;
 } xenctx;
@@ -899,6 +900,19 @@ static void dump_ctx(int vcpu)
 #endif
 }
 
+static void dump_all_vcpus(void)
+{
+    xc_vcpuinfo_t vinfo;
+    int vcpu;
+    for (vcpu = 0; vcpu <= xenctx.dominfo.max_vcpu_id; vcpu++)
+    {
+        if ( xc_vcpu_getinfo(xenctx.xc_handle, xenctx.domid, vcpu, &vinfo) )
+            continue;
+        if ( vinfo.online )
+            dump_ctx(vcpu);
+    }
+}
+
 static void usage(void)
 {
     printf("usage:\n\n");
@@ -920,13 +934,14 @@ static void usage(void)
 #else
     printf("  -a --all          display more registers\n");
 #endif
+    printf("  -C --all-vcpus    print info for all vcpus\n");
 }
 
 int main(int argc, char **argv)
 {
     int ch;
     int ret;
-    static const char *sopts = "fs:hak:S"
+    static const char *sopts = "fs:hak:SC"
 #ifdef __ia64__
         "r:"
 #endif
@@ -940,6 +955,7 @@ int main(int argc, char **argv)
         {"regs", 1, NULL, 'r'},
 #endif
         {"all", 0, NULL, 'a'},
+        {"all-vcpus", 0, NULL, 'C'},
         {"help", 0, NULL, 'h'},
         {0, 0, 0, 0}
     };
@@ -995,6 +1011,9 @@ int main(int argc, char **argv)
             xenctx.disp_all = 1;
             break;
 #endif
+        case 'C':
+            xenctx.all_vcpus = 1;
+            break;
         case 'k':
             kernel_start = strtoull(optarg, NULL, 0);
             break;
@@ -1047,7 +1066,10 @@ int main(int argc, char **argv)
         xenctx.self_paused = 1;
     }
 
-    dump_ctx(vcpu);
+    if (xenctx.all_vcpus)
+        dump_all_vcpus();
+    else
+        dump_ctx(vcpu);
 
     if (xenctx.self_paused) {
         ret = xc_domain_unpause(xenctx.xc_handle, xenctx.domid);

[PATCH 5 of 5] xenctx: dump pagetable

[Olaf Hering]

# HG changeset patch
# User Olaf Hering <olaf@aepfle.de>
# Date 1308582353 -7200
# Node ID 0443de5faea9b904b92219b3b804dec147697366
# Parent  ddf16ea954876d8af371f9751fc06eb4c9e78b36
xenctx: dump pagetable

This change is buggy...

diff -r ddf16ea95487 -r 0443de5faea9 tools/xentrace/xenctx.c
--- a/tools/xentrace/xenctx.c	Fri Jun 17 14:22:56 2011 +0200
+++ b/tools/xentrace/xenctx.c	Mon Jun 20 17:05:53 2011 +0200
@@ -25,21 +25,34 @@
 #include <getopt.h>
 
 #include "xenctrl.h"
+#include "xg_private.h"
+#include "xc_private.h"
 #include <xen/foreign/x86_32.h>
 #include <xen/foreign/x86_64.h>
 #include <xen/hvm/save.h>
 
+#define CR4_PAE 0x20
+#define EFER_LMA 0x400
+
 static struct xenctx {
     xc_interface *xc_handle;
     int domid;
     int frame_ptrs;
     int stack_trace;
+    int dump_pagetable;
     int disp_all;
     int all_vcpus;
     int self_paused;
     xc_dominfo_t dominfo;
 } xenctx;
 
+struct cpuctx {
+    vcpu_guest_context_any_t any;
+#if defined(__i386__) || defined(__x86_64__)
+    struct hvm_hw_cpu hvm;
+#endif
+} cpuctx;
+
 #if defined (__i386__) || defined (__x86_64__)
 typedef unsigned long long guest_word_t;
 #define FMT_32B_WORD "%08llx"
@@ -436,6 +449,184 @@ static guest_word_t frame_pointer(vcpu_g
         return ctx->x64.user_regs.rbp;
 }
 
+static void walk_l1(int vcpu, uint64_t l1, uint64_t virt)
+{
+    uint64_t *map;
+    unsigned long long e, l1e, phys_mask = ~(-1ull << 52) & (-1ull << 12);
+    char buf[123];
+    fflush(stdout);
+    map = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, PAGE_SIZE, PROT_READ, l1 >> PAGE_SHIFT);
+    if (!map) {
+        perror("xc_map_foreign_range l1");
+        return;
+    }
+    for (e = 0; e < L1_PAGETABLE_ENTRIES_X86_64; e++) {
+        l1e = map[e];
+        if (!l1e)
+            continue;
+        buf[0] = '\0';
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & ( 1ull << 63) ? 'X' : 'x'); /* NX */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %3llx", (l1e >> 52) & ~( 1ull << 11)); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %12llx", l1e & phys_mask); /* phys base adress */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l1e >> 9) & 0x7); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_GLOBAL ? 'G' : 'g');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PAT ? 'P' : 'p');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PSE ? 'S' : 's');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_DIRTY ? 'D' : 'd');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_ACCESSED ? 'A' : 'a');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PCD ? 'C' : 'c');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PWT ? 'T' : 't');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_USER ? 'U' : 'u');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_RW ? 'W' : 'w');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PRESENT ? 'P' : 'p');
+        printf("%3d l1e %3llx %016llx: %s %16llx\n", vcpu, e, l1e, buf, (unsigned long long)virt | (e << L1_PAGETABLE_SHIFT_X86_64));
+    }
+    munmap(map, PAGE_SIZE);
+}
+
+static void walk_l2(int vcpu, uint64_t l2, uint64_t virt)
+{
+    uint64_t *map;
+    unsigned long long e, l2e, l1_mask = ~(-1ull << 52) & (-1ull << 12);
+    char buf[123];
+    fflush(stdout);
+    map = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, PAGE_SIZE, PROT_READ, l2 >> PAGE_SHIFT);
+    if (!map) {
+        perror("xc_map_foreign_range l2");
+        return;
+    }
+    for (e = 0; e < L2_PAGETABLE_ENTRIES_X86_64; e++) {
+        l2e = map[e];
+        if (!l2e)
+            continue;
+        buf[0] = '\0';
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & ( 1ull << 63) ? 'X' : 'x'); /* NX */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %3llx", (l2e >> 52) & ~( 1ull << 11)); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %12llx", l2e & l1_mask); /* l1 base adress */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l2e >> 9) & 0x7); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l2e >> 8) & 0x1); /* Ignored */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", (l2e >> 7) & 0x1 ? 'S' : 's'); /* SP */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l2e >> 6) & 0x1); /* Ignored */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_ACCESSED ? 'A' : 'a');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_PCD ? 'C' : 'c');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_PWT ? 'T' : 't');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_USER ? 'U' : 'u');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_RW ? 'W' : 'w');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_PRESENT ? 'P' : 'p');
+        printf("%3d l2e %3llx %016llx: %s\n", vcpu, e, l2e, buf);
+        if (l2e & _PAGE_PRESENT)
+            walk_l1(vcpu, l2e & l1_mask, virt | (e << L2_PAGETABLE_SHIFT_X86_64));
+    }
+    munmap(map, PAGE_SIZE);
+}
+
+static void walk_l3(int vcpu, uint64_t l3, uint64_t virt)
+{
+    uint64_t *map;
+    unsigned long long e, l3e, l2_mask = ~(-1ull << 52) & (-1ull << 12);
+    char buf[123];
+    fflush(stdout);
+    map = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, PAGE_SIZE, PROT_READ, l3 >> PAGE_SHIFT);
+    if (!map) {
+        perror("xc_map_foreign_range l3");
+        return;
+    }
+    for (e = 0; e < L3_PAGETABLE_ENTRIES_X86_64; e++) {
+        l3e = map[e];
+        if (!l3e)
+            continue;
+        buf[0] = '\0';
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & ( 1ull << 63) ? 'X' : 'x'); /* NX */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %3llx", (l3e >> 52) & ~( 1ull << 11)); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %12llx", l3e & l2_mask); /* l2 base adress */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l3e >> 9) & 0x7); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l3e >> 8) & 0x1); /* MBZ */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", (l3e >> 7) & 0x1 ? 'S' : 's'); /* SP */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l3e >> 6) & 0x1); /* Ignored */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_ACCESSED ? 'A' : 'a');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_PCD ? 'C' : 'c');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_PWT ? 'T' : 't');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_USER ? 'U' : 'u');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_RW ? 'W' : 'w');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_PRESENT ? 'P' : 'p');
+        printf("%3d l3e %3llx %016llx: %s\n", vcpu, e, l3e, buf);
+        if (l3e & _PAGE_PRESENT)
+            walk_l2(vcpu, l3e & l2_mask, virt | (e << L3_PAGETABLE_SHIFT_X86_64));
+    }
+    munmap(map, PAGE_SIZE);
+}
+
+static void walk_l4(int vcpu, uint64_t l4)
+{
+    uint64_t *map;
+    unsigned long long e, l4e, l3_mask = ~(-1ull << 52) & (-1ull << 12), virt = 0;
+    char buf[123];
+    fflush(stdout);
+    map = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, PAGE_SIZE, PROT_READ, l4 >> PAGE_SHIFT);
+    if (!map) {
+        perror("xc_map_foreign_range l4");
+        return;
+    }
+    for (e = 0; e < L4_PAGETABLE_ENTRIES_X86_64; e++) {
+        l4e = map[e];
+        if (!l4e)
+            continue;
+        buf[0] = '\0';
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & ( 1ull << 63) ? 'X' : 'x'); /* NX */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %3llx", (l4e >> 52) & ~( 1ull << 11)); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %12llx", l4e & l3_mask); /* l3 base adress */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l4e >> 9) & 0x7); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l4e >> 7) & 0x3); /* MBZ */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l4e >> 6) & 0x1); /* Ignored */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_ACCESSED ? 'A' : 'a');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_PCD ? 'C' : 'c');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_PWT ? 'T' : 't');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_USER ? 'u' : 'u');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_RW ? 'W' : 'w');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_PRESENT ? 'P' : 'p');
+        printf("%3d l4e %3llx %016llx: %s\n", vcpu, e, l4e, buf);
+        if (l4e & _PAGE_PRESENT)
+            walk_l3(vcpu, l4e & l3_mask, virt | (e << L4_PAGETABLE_SHIFT_X86_64));
+    }
+    munmap(map, PAGE_SIZE);
+}
+
+static void dump_pagetable(struct cpuctx *cpuctx, int vcpu)
+{
+    uint64_t paddr;
+    int pt_levels, size;
+
+    printf("\npagetable for vcpu %d\n", vcpu);
+    /* What kind of paging are we dealing with? */
+    if (xenctx.dominfo.hvm) {
+        if (!guest_protected_mode) {
+            printf("protected mode disabled\n");
+            return;
+        }
+        pt_levels = (cpuctx->hvm.msr_efer & EFER_LMA) ? 4 : (cpuctx->hvm.cr4 & CR4_PAE) ? 3 : 2;
+        paddr = cpuctx->hvm.cr3 & ((pt_levels == 3) ? ~0x1full : ~0xfffull);
+    } else {
+        if (guest_word_size == 8) {
+            pt_levels = 4;
+            paddr = cpuctx->any.x64.ctrlreg[3] & ~0xfffull;
+        } else {
+            pt_levels = 3;
+            paddr = ((uint64_t) xen_cr3_to_pfn(cpuctx->any.x32.ctrlreg[3])) << PAGE_SHIFT;
+        }
+    }
+
+    size = (pt_levels == 2 ? 4 : 8);
+    if (pt_levels == 4) {
+        walk_l4(vcpu, paddr);
+    } else if (pt_levels == 3) {
+        walk_l3(vcpu, paddr, 0);
+    } else {
+        walk_l2(vcpu, paddr, 0);
+    }
+    printf("\n");
+    fflush(stdout);
+}
+
 #elif defined(__ia64__)
 
 #define PTE_ED_SHIFT              52
@@ -853,9 +1044,11 @@ static int print_stack(vcpu_guest_contex
 
 static void dump_ctx(int vcpu)
 {
-    vcpu_guest_context_any_t ctx;
+    struct cpuctx cpuctx;
 
-    if (xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &ctx) < 0) {
+    memset(&cpuctx, 0, sizeof(cpuctx));
+
+    if (xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &cpuctx.any) < 0) {
         perror("xc_vcpu_getcontext");
         return;
     }
@@ -863,16 +1056,15 @@ static void dump_ctx(int vcpu)
 #if defined(__i386__) || defined(__x86_64__)
     {
         if (xenctx.dominfo.hvm) {
-            struct hvm_hw_cpu cpuctx;
             xen_capabilities_info_t xen_caps = "";
             if (xc_domain_hvm_getcontext_partial(
                     xenctx.xc_handle, xenctx.domid, HVM_SAVE_CODE(CPU),
-                    vcpu, &cpuctx, sizeof cpuctx) != 0) {
+                    vcpu, &cpuctx.hvm, sizeof (cpuctx.hvm)) != 0) {
                 perror("xc_domain_hvm_getcontext_partial");
                 return;
             }
-            guest_word_size = (cpuctx.msr_efer & 0x400) ? 8 : 4;
-            guest_protected_mode = (cpuctx.cr0 & CR0_PE);
+            guest_word_size = (cpuctx.hvm.msr_efer & EFER_LMA) ? 8 : 4;
+            guest_protected_mode = (cpuctx.hvm.cr0 & CR0_PE);
             /* HVM guest context records are always host-sized */
             if (xc_version(xenctx.xc_handle, XENVER_capabilities, &xen_caps) != 0) {
                 perror("xc_version");
@@ -890,14 +1082,18 @@ static void dump_ctx(int vcpu)
     }
 #endif
 
-    print_ctx(&ctx);
+    print_ctx(&cpuctx.any);
 #ifndef NO_TRANSLATION
-    if (print_code(&ctx, vcpu))
+    if (print_code(&cpuctx.any, vcpu))
         return;
-    if (is_kernel_text(instr_pointer(&ctx)))
-        if (print_stack(&ctx, vcpu, guest_word_size))
+    if (is_kernel_text(instr_pointer(&cpuctx.any)))
+        if (print_stack(&cpuctx.any, vcpu, guest_word_size))
             return;
 #endif
+#if defined(__i386__) || defined(__x86_64__)
+    if (xenctx.dump_pagetable)
+        dump_pagetable(&cpuctx, vcpu);
+#endif
 }
 
 static void dump_all_vcpus(void)
@@ -926,6 +1122,10 @@ static void usage(void)
     printf("  -s SYMTAB, --symbol-table=SYMTAB\n");
     printf("                    read symbol table from SYMTAB.\n");
     printf("  -S --stack-trace  print a complete stack trace.\n");
+#if defined(__i386__) || defined(__x86_64__)
+    printf("  -P --dump-pagetable\n");
+    printf("                    print a complete pagetable for the given vcpu.\n");
+#endif
     printf("  -k, --kernel-start\n");
     printf("                    set user/kernel split. (default 0xc0000000)\n");
 #ifdef __ia64__
@@ -941,7 +1141,7 @@ int main(int argc, char **argv)
 {
     int ch;
     int ret;
-    static const char *sopts = "fs:hak:SC"
+    static const char *sopts = "fs:hak:SPC"
 #ifdef __ia64__
         "r:"
 #endif
@@ -951,6 +1151,9 @@ int main(int argc, char **argv)
         {"symbol-table", 1, NULL, 's'},
         {"frame-pointers", 0, NULL, 'f'},
         {"kernel-start", 1, NULL, 'k'},
+#if defined(__i386__) || defined(__x86_64__)
+        {"dump-pagetable", 0, NULL, 'P'},
+#endif
 #ifdef __ia64__
         {"regs", 1, NULL, 'r'},
 #endif
@@ -974,6 +1177,11 @@ int main(int argc, char **argv)
         case 'S':
             xenctx.stack_trace = 1;
             break;
+#if defined(__i386__) || defined(__x86_64__)
+        case 'P':
+            xenctx.dump_pagetable = 1;
+            break;
+#endif
 #ifdef __ia64__
         case 'r':
             {

Re: [PATCH 0 of 5] update xenctx to dump pagetables

[Tim Deegan]

Hi, 

At 17:26 +0200 on 20 Jun (1308590816), Olaf Hering wrote:
> The code to walk the pagetables is/was based on
> xc_translate_foreign_address(), but I still think I have some major bugs in
> there (the last patch). I cant figure out why some l3/l2/l1 entries can not be mapped.
> Any help with getting that output fixed for at least a a 64bit PV guest is much
> appreciated.

I didn't spot anything broken in the walker (though it will need a bunch
of cleaning up) and the output looks very plausible.  I take it this is
a PV guest?  Is it a well-behaved one or a broken post-migrate one?

One thing that might be causing trouble is if the domain's not paused
while you do the walk, then you might see inconsistent tables, though
I'd expect them to be garbage rather than looking like this.  Your patch
3/5 does seem to make the pausing conditional where before it always
happened. 

Tim.

> part of xenctx output:
> ...
>   0 l4e  ff 00000000a1240067:  x   0     a1240000 0 0 1 A c t u W P
>   0 l3e 1ff 00000000bc133067:  x   0     bc133000 0 0 s 1 A c t U W P
>   0 l2e  64 00000000a1471067:  x   0     a1471000 0 0 s 1 A c t U W P
>   0 l1e  46 80000000bd56f167:  X   0     bd56f000 0 G p s D A c t U W P     7fffcc846000
>   0 l1e  47 80000000bd57c167:  X   0     bd57c000 0 G p s D A c t U W P     7fffcc847000
>   0 l1e  48 80000000bd5af165:  X   0     bd5af000 0 G p s D A c t U w P     7fffcc848000
>   0 l1e  db 00000000a1972125:  x   0     a1972000 0 G p s d A c t U w P     7fffcc8db000
>   0 l4e 100 000000013fff8067:  x   0    13fff8000 0 0 1 A c t u W P
> xc_map_foreign_range l3: Invalid argument
>   0 l4e 101 000000013fff0063:  x   0    13fff0000 0 0 1 A c t u W P
> xc_map_foreign_range l3: Invalid argument
>   0 l4e 102 00000000bc5b0063:  x   0     bc5b0000 0 0 1 A c t u W P
>   0 l3e  fe 00000000a7ed2067:  x   0     a7ed2000 0 0 s 1 A c t U W P
>   0 l2e 163 00000000a154a067:  x   0     a154a000 0 0 s 1 A c t U W P
>   0 l1e 1eb 00000000bc1a7067:  x   0     bc1a7000 0 g p s D A c t U W P     813fac7eb000
>   0 l1e 1ec 00000000a154f067:  x   0     a154f000 0 g p s D A c t U W P     813fac7ec000
> ...
> 
> in dmesg:
> ...
> (XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=19
> (XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=19


-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: [PATCH 0 of 5] update xenctx to dump pagetables

[Olaf Hering]

On Mon, Jun 20, Tim Deegan wrote:

> Hi, 
> 
> At 17:26 +0200 on 20 Jun (1308590816), Olaf Hering wrote:
> > The code to walk the pagetables is/was based on
> > xc_translate_foreign_address(), but I still think I have some major bugs in
> > there (the last patch). I cant figure out why some l3/l2/l1 entries can not be mapped.
> > Any help with getting that output fixed for at least a a 64bit PV guest is much
> > appreciated.
> 
> I didn't spot anything broken in the walker (though it will need a bunch
> of cleaning up) and the output looks very plausible.  I take it this is
> a PV guest?  Is it a well-behaved one or a broken post-migrate one?

Yes, its a PV guest. I cant reproduce the migrate crashes, it happens
only on very few systems or on a certain configuration. The logs I
posted are from my test system.
Any ideas why some mfns are not accessible?

Are there any other paging states maintained outside of the guests
memory?

> One thing that might be causing trouble is if the domain's not paused
> while you do the walk, then you might see inconsistent tables, though
> I'd expect them to be garbage rather than looking like this.  Your patch
> 3/5 does seem to make the pausing conditional where before it always
> happened. 

It tries it preserve the state, if the guest was paused it probably
should not be unpaused. Is the dominfo.paused flag somehow unreliable, I
thought it comes right from struct domain?

Re: [PATCH 0 of 5] update xenctx to dump pagetables

[Konrad Rzeszutek Wilk]

On Mon, Jun 20, 2011 at 07:51:58PM +0200, Olaf Hering wrote:
> On Mon, Jun 20, Tim Deegan wrote:
> 
> > Hi, 
> > 
> > At 17:26 +0200 on 20 Jun (1308590816), Olaf Hering wrote:
> > > The code to walk the pagetables is/was based on
> > > xc_translate_foreign_address(), but I still think I have some major bugs in
> > > there (the last patch). I cant figure out why some l3/l2/l1 entries can not be mapped.
> > > Any help with getting that output fixed for at least a a 64bit PV guest is much
> > > appreciated.
> > 
> > I didn't spot anything broken in the walker (though it will need a bunch
> > of cleaning up) and the output looks very plausible.  I take it this is
> > a PV guest?  Is it a well-behaved one or a broken post-migrate one?
> 
> Yes, its a PV guest. I cant reproduce the migrate crashes, it happens
> only on very few systems or on a certain configuration. The logs I
> posted are from my test system.
> Any ideas why some mfns are not accessible?

They look to be the special I/O PFNs. The ones that cover ACPI, framebuffer,
PCI IO bars, MP tale.
> 
> Are there any other paging states maintained outside of the guests
> memory?

They look to be I/O pages.

But not sure why they are mapped to your guest?
> 
> > One thing that might be causing trouble is if the domain's not paused
> > while you do the walk, then you might see inconsistent tables, though
> > I'd expect them to be garbage rather than looking like this.  Your patch
> > 3/5 does seem to make the pausing conditional where before it always
> > happened. 
> 
> It tries it preserve the state, if the guest was paused it probably
> should not be unpaused. Is the dominfo.paused flag somehow unreliable, I
> thought it comes right from struct domain?
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH 0 of 5] update xenctx to dump pagetables

[Tim Deegan]

Hi, 

At 17:56 -0400 on 20 Jun (1308592583), Konrad Rzeszutek Wilk wrote:
> > Any ideas why some mfns are not accessible?
> 
> They look to be the special I/O PFNs. The ones that cover ACPI, framebuffer,
> PCI IO bars, MP tale.

I think they're too high for that, but if you can post the e820 map of
the system this happened on then we'll know. 

> > Are there any other paging states maintained outside of the guests
> > memory?
> 
> They look to be I/O pages.
> 
> But not sure why they are mapped to your guest?

But they're not mapped into the guest - from the look of them they're
not mapped anywhere.  You could add some extra printouts around that
warning in mm.c to show whether the MFNs are valid and if so which
domain owns them.

Also, interesting that it's the addresses just above 0xffff800000000000
that are different - what lives at that address in the PV kernel you're
running?

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Re: [PATCH 0 of 5] update xenctx to dump pagetables

[Jan Beulich]

>>> On 21.06.11 at 11:59, Tim Deegan <Tim.Deegan@citrix.com> wrote:
> Hi, 
> 
> At 17:56 -0400 on 20 Jun (1308592583), Konrad Rzeszutek Wilk wrote:
>> > Any ideas why some mfns are not accessible?
>> 
>> They look to be the special I/O PFNs. The ones that cover ACPI, framebuffer,
>> PCI IO bars, MP tale.
> 
> I think they're too high for that, but if you can post the e820 map of
> the system this happened on then we'll know. 
> 
>> > Are there any other paging states maintained outside of the guests
>> > memory?
>> 
>> They look to be I/O pages.
>> 
>> But not sure why they are mapped to your guest?
> 
> But they're not mapped into the guest - from the look of them they're
> not mapped anywhere.  You could add some extra printouts around that
> warning in mm.c to show whether the MFNs are valid and if so which
> domain owns them.
> 
> Also, interesting that it's the addresses just above 0xffff800000000000
> that are different - what lives at that address in the PV kernel you're
> running?

That's Xen's space, isn't it. Clearly any non-hypervisor based page
table walking code has to ignore this range for PV guests' page
tables.

Jan

Re: [PATCH 5 of 5] xenctx: dump pagetable

[Ian Jackson]

Olaf Hering writes ("[Xen-devel] [PATCH 5 of 5] xenctx: dump pagetable"):
> # HG changeset patch
> # User Olaf Hering <olaf@aepfle.de>
> # Date 1308582353 -7200
> # Node ID 0443de5faea9b904b92219b3b804dec147697366
> # Parent  ddf16ea954876d8af371f9751fc06eb4c9e78b36
> xenctx: dump pagetable
> 
> This change is buggy...

Uh... ?  Did this slip through by mistake somehow ?

Ian.

Re: [PATCH 2 of 5] xenctx: move all globals into struct xenctx

[Ian Jackson]

Olaf Hering writes ("[Xen-devel] [PATCH 2 of 5] xenctx: move all globals into struct xenctx"):
> xenctx: move all globals into struct xenctx

Having read 1-4 I think they look reasonable and I'm happy to apply
them when you give the word you think they're ready.  It wasn't
entirely clear from your messages whether you want me to do so.

Thanks,
Ian.

Re: [PATCH 5 of 5] xenctx: dump pagetable

[Olaf Hering]

On Tue, Jun 21, Ian Jackson wrote:

> Olaf Hering writes ("[Xen-devel] [PATCH 5 of 5] xenctx: dump pagetable"):
> > # HG changeset patch
> > # User Olaf Hering <olaf@aepfle.de>
> > # Date 1308582353 -7200
> > # Node ID 0443de5faea9b904b92219b3b804dec147697366
> > # Parent  ddf16ea954876d8af371f9751fc06eb4c9e78b36
> > xenctx: dump pagetable
> > 
> > This change is buggy...
> 
> Uh... ?  Did this slip through by mistake somehow ?

Its here for review and discussion, I forgot to add a reasonable description.
The change itself needs PAE and i386 support at least.
I will work on a better version.

Olaf

Re: [PATCH 2 of 5] xenctx: move all globals into struct xenctx

[Olaf Hering]

On Tue, Jun 21, Ian Jackson wrote:

> Olaf Hering writes ("[Xen-devel] [PATCH 2 of 5] xenctx: move all globals into struct xenctx"):
> > xenctx: move all globals into struct xenctx
> 
> Having read 1-4 I think they look reasonable and I'm happy to apply
> them when you give the word you think they're ready.  It wasn't
> entirely clear from your messages whether you want me to do so.

1-4 is ready, just last one (the pagetable change) needs more work.
If the concept is ok, please apply these changes.

Olaf

Re: [PATCH 2 of 5] xenctx: move all globals into struct xenctx

[Ian Jackson]

Olaf Hering writes ("Re: [Xen-devel] [PATCH 2 of 5] xenctx: move all globals into struct xenctx"):
> 1-4 is ready, just last one (the pagetable change) needs more work.
> If the concept is ok, please apply these changes.

Done, thanks.

Ian.

Re: [PATCH 0 of 5] update xenctx to dump pagetables

[Olaf Hering]

On Tue, Jun 21, Jan Beulich wrote:

> >>> On 21.06.11 at 11:59, Tim Deegan <Tim.Deegan@citrix.com> wrote:
> > Hi, 
> > 
> > At 17:56 -0400 on 20 Jun (1308592583), Konrad Rzeszutek Wilk wrote:
> >> > Any ideas why some mfns are not accessible?
> >> 
> >> They look to be the special I/O PFNs. The ones that cover ACPI, framebuffer,
> >> PCI IO bars, MP tale.
> > 
> > I think they're too high for that, but if you can post the e820 map of
> > the system this happened on then we'll know. 
> > 
> >> > Are there any other paging states maintained outside of the guests
> >> > memory?
> >> 
> >> They look to be I/O pages.
> >> 
> >> But not sure why they are mapped to your guest?
> > 
> > But they're not mapped into the guest - from the look of them they're
> > not mapped anywhere.  You could add some extra printouts around that
> > warning in mm.c to show whether the MFNs are valid and if so which
> > domain owns them.
> > 
> > Also, interesting that it's the addresses just above 0xffff800000000000
> > that are different - what lives at that address in the PV kernel you're
> > running?
> 
> That's Xen's space, isn't it. Clearly any non-hypervisor based page
> table walking code has to ignore this range for PV guests' page
> tables.

Is there a way to detect that? I cant seem to match any of these ranges to
something in guests dmesg or /proc.

After adding some debug to my xenctx from sles11 4.0, I get this on vcpu 0:

xc_map_foreign_range: walk_l3: 0 virt 0000800000000000 mfn 13fff8: Invalid argument
xc_map_foreign_range: walk_l3: 0 virt 0000808000000000 mfn 13fff0: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814000000000 mfn 13fff8: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814040000000 mfn 13fff0: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0000000 mfn 13fff8: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0200000 mfn 13fff0: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0800000 mfn 137ff8: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0a00000 mfn bf49a: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0c00000 mfn bf495: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814100000000 mfn 137ff8: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814140000000 mfn bf49a: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814180000000 mfn bf495: Invalid argument
xc_map_foreign_range: walk_l3: 0 virt 0000820000000000 mfn 137ff8: Invalid argument
xc_map_foreign_range: walk_l3: 0 virt 0000828000000000 mfn bf49a: Invalid argument
xc_map_foreign_range: walk_l3: 0 virt 0000830000000000 mfn bf495: Invalid argument


And dmesg has:

(XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn 137ff8 (pfn f180d) from L1 entry 8000000137ff8625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn bf49a (pfn 5555555555555555) from L1 entry 80000000bf49a625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn bf495 (pfn 5555555555555555) from L1 entry 80000000bf495625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn 137ff8 (pfn f180d) from L1 entry 8000000137ff8625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn bf49a (pfn 5555555555555555) from L1 entry 80000000bf49a625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn bf495 (pfn 5555555555555555) from L1 entry 80000000bf495625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn 137ff8 (pfn f180d) from L1 entry 8000000137ff8625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn bf49a (pfn 5555555555555555) from L1 entry 80000000bf49a625 for l1e_owner=0, pg_owner=1
(XEN) mm.c:880:d0 Error getting mfn bf495 (pfn 5555555555555555) from L1 entry 80000000bf495625 for l1e_owner=0, pg_owner=1

Olaf

Re: [PATCH 0 of 5] update xenctx to dump pagetables

[Jan Beulich]

>>> On 22.06.11 at 15:50, Olaf Hering <olaf@aepfle.de> wrote:
> On Tue, Jun 21, Jan Beulich wrote:
> 
>> >>> On 21.06.11 at 11:59, Tim Deegan <Tim.Deegan@citrix.com> wrote:
>> > Hi, 
>> > 
>> > At 17:56 -0400 on 20 Jun (1308592583), Konrad Rzeszutek Wilk wrote:
>> >> > Any ideas why some mfns are not accessible?
>> >> 
>> >> They look to be the special I/O PFNs. The ones that cover ACPI, 
> framebuffer,
>> >> PCI IO bars, MP tale.
>> > 
>> > I think they're too high for that, but if you can post the e820 map of
>> > the system this happened on then we'll know. 
>> > 
>> >> > Are there any other paging states maintained outside of the guests
>> >> > memory?
>> >> 
>> >> They look to be I/O pages.
>> >> 
>> >> But not sure why they are mapped to your guest?
>> > 
>> > But they're not mapped into the guest - from the look of them they're
>> > not mapped anywhere.  You could add some extra printouts around that
>> > warning in mm.c to show whether the MFNs are valid and if so which
>> > domain owns them.
>> > 
>> > Also, interesting that it's the addresses just above 0xffff800000000000
>> > that are different - what lives at that address in the PV kernel you're
>> > running?
>> 
>> That's Xen's space, isn't it. Clearly any non-hypervisor based page
>> table walking code has to ignore this range for PV guests' page
>> tables.
> 
> Is there a way to detect that? I cant seem to match any of these ranges to
> something in guests dmesg or /proc.

No need for detection - the hole is part of the ABI (minus its dynamic
size in 32-bit pv guests running on 64-bit hypervisor - that may require
use of heuristics).

Jan