* Faulting linear address?? @ 2017-09-08 5:33 Minjun Hong 2017-09-08 7:40 ` Dario Faggioli 0 siblings, 1 reply; 4+ messages in thread From: Minjun Hong @ 2017-09-08 5:33 UTC (permalink / raw) To: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 4443 bytes --] Hello~ I think this might be duplicate issue but, I have not resolved for long time. So, please understand this post generously. First, I should explain my history. 1) I worked on the scheduler(credit scheduler) and I had a kernel panic by my modification. 2) I tried to get any information for debugging so that, I used serial console and could gain the serial logs like following: (XEN) ----[ Xen-4.5.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 2 (XEN) RIP: e008:[<ffff82d080120973>] csched_schedule+0x373/0x1180 (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor (XEN) rax: 00000000ffffffff rbx: ffff830087ffa000 rcx: ffff830461d20000 (XEN) rdx: ffff830088002c98 rsi: ffff830461d20000 rdi: 0000000000000000 (XEN) rbp: ffff830461ce2ae0 rsp: ffff830461d27d10 r8: 0000001e582339ec (XEN) r9: 0000000000000004 r10: 000000000000003c r11: 0000000000000004 (XEN) r12: 0000000000000001 r13: ffff82d0803f26a0 r14: ffff830461c53000 (XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000003526f0 (XEN) cr3: 0000000086077000 cr2: ffff830088002c98 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff830461d27d10: (XEN) ffff830461d03950 ffff82d0804081e0 ffff830461c74068 ffff830461d27de0 (XEN) ffff830461c24c30 ffff830461cec800 ffff82d0804081e0 0000000600000002 (XEN) ffff830461ce29d0 ffff830461d20000 ffff82d0804081e0 ffff830461d3a720 (XEN) 0000000000000002 ffff830461d3a700 00ffffc000000000 ffff830461d27dd0 (XEN) ffff830461d27e68 ffff82d080408180 0000001e5c106499 0000000001c9c380 (XEN) 0000000000000000 0000000000000000 ffff8300864e3000 ffff8302e1596fb0 (XEN) ffff830461d27dd0 ffff830461d27dd0 000000000000004b 0000000000000000 (XEN) 0000000000000000 0000000000000000 ffff830461d3a738 ffff8300864e3000 (XEN) ffff82d0804081e0 ffff830461d2e068 0000001e5c106499 ffff830461d2e060 (XEN) ffff82d0803f26a0 ffff82d080128cb3 0000001e00000000 ffff830461d2e080 (XEN) ffff82d080279944 ffff82d08015f295 0000001e5c0504ce ffff830461d3ad80 (XEN) 0000001e5c1054ba ffff82d08012f64e ffff82d0803f26a0 00000000ffffffff (XEN) ffff82d0803df880 0000000000000001 ffff82d0803df780 ffffffffffffffff (XEN) ffff830461d20000 ffff82d08012c03c ffffffffffffffff 00000000ffffffff (XEN) ffff830461d20000 ffff830461d2e068 0000001e5b762541 ffff830461d2e060 (XEN) ffff82d0803f26a0 ffff82d080162e3a 0000000000000000 ffff8300864e3000 (XEN) ffff8300864e3000 ffff8800f8bbbfd8 0000000000000000 ffff8800f8bbbfd8 (XEN) 0000000000000003 ffff8800f8bbbec0 0000000000000000 0000000000000246 (XEN) 0000000000007ff0 0000000000000000 0000000000000000 0000000000000000 (XEN) ffffffff810013aa 0000000000000001 0000000000000000 0000000000000001 (XEN) Xen call trace: (XEN) [<ffff82d080120973>] csched_schedule+0x373/0x1180 (XEN) [<ffff82d080128cb3>] schedule+0xf3/0x590 (XEN) [<ffff82d08015f295>] reprogram_timer+0x75/0xe0 (XEN) [<ffff82d08012f64e>] timer_softirq_action+0x13e/0x210 (XEN) [<ffff82d08012c03c>] __do_softirq+0x7c/0xd0 (XEN) [<ffff82d080162e3a>] idle_loop+0x3a/0x70 (XEN) (XEN) Pagetable walk from ffff830088002c98: (XEN) L4[0x106] = 0000000086075063 ffffffffffffffff (XEN) L3[0x002] = 0000000086071063 ffffffffffffffff (XEN) L2[0x040] = 0000000000000000 ffffffffffffffff (XEN) (XEN) **************************************** (XEN) Panic on CPU 2: (XEN) FATAL PAGE FAULT (XEN) [error_code=0000] (XEN) Faulting linear address: ffff830088002c98 (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... (XEN) Resetting with ACPI MEMORY or I/O RESET_REG. I want to know where I should start debugging from. However, although I'm using serial console, I could get not enough clues only from the kernel log: 1) I could figure out what line and file caused the panic by its call trace, but it is too rough so it does not help me. 2) What linear address brings about this situation; 'Faulting linear address', but it is just an address and not recognizable something that human cannot read. I think, literally, the 'Faulting linear address' is key point because I heard that it represents bad address that I should never access. With the pointer(bad address), is there any way to figure out its real data or more accurate line in C source code? If you have any other approach that can be used in some cases like this, could you please give me the guide? I hope your help. Thank you! [-- Attachment #1.2: Type: text/html, Size: 6089 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Faulting linear address?? 2017-09-08 5:33 Faulting linear address?? Minjun Hong @ 2017-09-08 7:40 ` Dario Faggioli 2017-09-10 17:00 ` Minjun Hong 0 siblings, 1 reply; 4+ messages in thread From: Dario Faggioli @ 2017-09-08 7:40 UTC (permalink / raw) To: Minjun Hong, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 3645 bytes --] On Fri, 2017-09-08 at 14:33 +0900, Minjun Hong wrote: > 1) I worked on the scheduler(credit scheduler) and I had a kernel > panic by my modification. > To do what? Can you show here what you are changing, e.g., by putting together a quick patch? It does not have to be properly formatted and follow all the rules of a proper patch submission... it can just be a diff against original code, to understand what you are changing. > 2) I tried to get any information for debugging so that, I used > serial console and could gain the serial logs like following: > > (XEN) ----[ Xen-4.5.0 x86_64 debug=n Not tainted ]---- > First of all, when debugging, you should use a debug hypervisor, i.e., a build of Xen, done with 'debug=y', or in general, with debug enabled. Also, Xen-4.5.0. Can you move to a more recent version? > (XEN) CPU: 2 > (XEN) RIP: e008:[<ffff82d080120973>] csched_schedule+0x373/0x1180 > (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor > [..] > (XEN) Xen call trace: > (XEN) [<ffff82d080120973>] > csched_schedule+0x373/0x1180 > (XEN) [<ffff82d080128cb3>] schedule+0xf3/0x590 > (XEN) [<ffff82d08015f295>] reprogram_timer+0x75/0xe0 > (XEN) [<ffff82d08012f64e>] timer_softirq_action+0x13e/0x210 > (XEN) [<ffff82d08012c03c>] __do_softirq+0x7c/0xd0 > (XEN) [<ffff82d080162e3a>] idle_loop+0x3a/0x70 > Again, use a debug hypervisor, compiled with frame pointers. > (XEN) Pagetable walk from ffff830088002c98: > (XEN) L4[0x106] = 0000000086075063 ffffffffffffffff > (XEN) L3[0x002] = 0000000086071063 ffffffffffffffff > (XEN) L2[0x040] = 0000000000000000 ffffffffffffffff > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 2: > (XEN) FATAL PAGE FAULT > (XEN) [error_code=0000] > (XEN) Faulting linear address: ffff830088002c98 > (XEN) **************************************** > I want to know where I should start debugging from. > However, although I'm using serial console, I could get not enough > clues only from the kernel log: > 1) I could figure out what line and file caused the panic by its call > trace, but it is too rough so it does not help me. > That's exactly from where you usually start: looking at what's at the instruction that cause the system to explode, in your case, at address 0xffff82d080120973. You can figure that out by disassembling the Xen hypervisor binary, with `objdump', and looking up that address. Or you can use addr2line, to have an indication of the same thing, but in the C sources. I'm not sure what you mean with "it is too rough". At least the address of the instruction that caused the system to fail in the Xen binary, is usually pretty accurate (then, of course, you have to look at surrounding instructions, check how you got there, etc.). Again, make sure you use a debug hypervisor. > 2) What linear address brings about this situation; 'Faulting linear > address', but it is just an address and not recognizable something > that human cannot read. > > I think, literally, the 'Faulting linear address' is key point > because I heard that it represents bad address that I should never > access. > It is, but the only way of understanding why you hit such an access violation, is understand what the code is doing when it happens. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 819 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Faulting linear address?? 2017-09-08 7:40 ` Dario Faggioli @ 2017-09-10 17:00 ` Minjun Hong 2017-09-11 13:49 ` Dario Faggioli 0 siblings, 1 reply; 4+ messages in thread From: Minjun Hong @ 2017-09-10 17:00 UTC (permalink / raw) To: Dario Faggioli; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 7582 bytes --] This is late reply but, thank you for your kind reply, Dario Faggioli. I made the new Xen4.5 binary with 'debug=y' option that I modified and install it. Then, there was a kernel panic caused by the debugging code triggered by 'debug=y' during booting process(of dom0): (XEN) ----[ Xen-4.5.0 x86_64 debug=y Not tainted ]---- (XEN) CPU: 7 (XEN) RIP: e008:[<ffff82d08012ba6c>] vcpu_migrate+0x1bd/0x374 (XEN) RFLAGS: 0000000000010096 CONTEXT: hypervisor (XEN) rax: ffff82d080413320 rbx: ffff830461c3e068 rcx: ffff82d080428e60 (XEN) rdx: 0000000000201da0 rsi: ffff830086d2f000 rdi: ffff82d080280bc0 (XEN) rbp: ffff83045e77fe48 rsp: ffff83045e77fdd8 r8: 0000000000000007 (XEN) r9: 00000000deadbeef r10: ffff82d08024d870 r11: 0000000000000246 (XEN) r12: ffff830461c3e068 r13: 0000000000000007 r14: ffff82d080428e60 (XEN) r15: ffff830461c3e068 cr0: 0000000080050033 cr4: 00000000003526f0 (XEN) cr3: 0000000459c0c000 cr2: ffff82d081422020 (XEN) ds: 002b es: 002b fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff83045e77fdd8: (XEN) ffff82d080428e60 ffff82d080428e60 ffff83045e77fdf8 ffff82d080428e60 (XEN) ffff83045e77fe00 ffff830086d2f000 00201da000000086 0000000000000246 (XEN) ffff82d08012e0c6 ffff830086d2f000 ffff830461c3e068 ffff82d080428e60 (XEN) ffff82d080413320 ffff83045e78a000 ffff83045e77fe78 ffff82d08012be23 (XEN) 0000000000000007 ffff830086d2f000 0000000000000000 0000000000000000 (XEN) ffff83045e77fef8 ffff82d080107052 ffffffff00000000 0000000000000008 (XEN) 0000000000a0fb00 0000000000000000 ffffffffffffffff 0000000001000c02 (XEN) ffff83045e77fe32 ffff82d080196da9 0f00000000000001 ffff830086d2f000 (XEN) ffff880456bcc4c0 0000000000000007 0000000000000000 0000000000000000 (XEN) 00007cfba18800c7 ffff82d080234d9b ffffffff8100130a 0000000000000018 (XEN) 0000000000000000 0000000000000002 0000000001000c02 ffff880450b43e3c (XEN) ffff880450b43e88 ffff880456bcc4c0 0000000000000246 0000000000000004 (XEN) 0000000000000000 0000000000000004 0000000000000018 ffffffff8100130a (XEN) 0000000000000000 0000000000000007 0000000000000007 0001010000000000 (XEN) ffffffff8100130a 000000000000e033 0000000000000246 ffff880450b43e70 (XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000007 ffff830086d2f000 00000033e1815200 (XEN) 0000000000000000 (XEN) Xen call trace: (XEN) [<ffff82d08012ba6c>] vcpu_migrate+0x1bd/0x374 (XEN) [<ffff82d08012be23>] vcpu_force_reschedule+0x9e/0xa7 (XEN) [<ffff82d080107052>] do_vcpu_op+0x2e7/0x69d (XEN) [<ffff82d080234d9b>] syscall_enter+0xeb/0x145 (XEN) (XEN) Pagetable walk from ffff82d081422020: (XEN) L4[0x105] = 0000000086092063 ffffffffffffffff (XEN) L3[0x142] = 000000008608f063 ffffffffffffffff (XEN) L2[0x00a] = 0000000000000000 ffffffffffffffff (XEN) (XEN) **************************************** (XEN) Panic on CPU 7: (XEN) FATAL PAGE FAULT (XEN) [error_code=0000] (XEN) Faulting linear address: ffff82d081422020 (XEN) **************************************** Because I received a solution from my professor, I think it is a hard work to change Xen version. Anyway, even if I turned on the 'debug=y' option, I could not get accurate information like with 'debug=n'; I get only linear address(ffff82d081422020). So, I want to use a dis-assembly utility like 'addr2line' or 'objdump', which binaries can I use as input to the utility? I'm using Ubuntu and previously I used '/boot/xen-syms-4.5.0' as input to the utilities. But I could get wrong information, which told me a code line that is never related this problem. I know that a beginner in the Xen developer community like me might be annoying you, but I ask you one more help. Thanks for your help again. On Fri, Sep 8, 2017 at 4:40 PM, Dario Faggioli <dario.faggioli@citrix.com> wrote: > On Fri, 2017-09-08 at 14:33 +0900, Minjun Hong wrote: > > 1) I worked on the scheduler(credit scheduler) and I had a kernel > > panic by my modification. > > > To do what? > > Can you show here what you are changing, e.g., by putting together a > quick patch? > > It does not have to be properly formatted and follow all the rules of a > proper patch submission... it can just be a diff against original code, > to understand what you are changing. > > > 2) I tried to get any information for debugging so that, I used > > serial console and could gain the serial logs like following: > > > > (XEN) ----[ Xen-4.5.0 x86_64 debug=n Not tainted ]---- > > > First of all, when debugging, you should use a debug hypervisor, i.e., > a build of Xen, done with 'debug=y', or in general, with debug enabled. > > Also, Xen-4.5.0. Can you move to a more recent version? > > > (XEN) CPU: 2 > > (XEN) RIP: e008:[<ffff82d080120973>] csched_schedule+0x373/0x1180 > > (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor > > [..] > > (XEN) Xen call trace: > > (XEN) [<ffff82d080120973>] > > csched_schedule+0x373/0x1180 > > (XEN) [<ffff82d080128cb3>] schedule+0xf3/0x590 > > (XEN) [<ffff82d08015f295>] reprogram_timer+0x75/0xe0 > > (XEN) [<ffff82d08012f64e>] timer_softirq_action+0x13e/0x210 > > (XEN) [<ffff82d08012c03c>] __do_softirq+0x7c/0xd0 > > (XEN) [<ffff82d080162e3a>] idle_loop+0x3a/0x70 > > > Again, use a debug hypervisor, compiled with frame pointers. > > > (XEN) Pagetable walk from ffff830088002c98: > > (XEN) L4[0x106] = 0000000086075063 ffffffffffffffff > > (XEN) L3[0x002] = 0000000086071063 ffffffffffffffff > > (XEN) L2[0x040] = 0000000000000000 ffffffffffffffff > > (XEN) > > (XEN) **************************************** > > (XEN) Panic on CPU 2: > > (XEN) FATAL PAGE FAULT > > (XEN) [error_code=0000] > > (XEN) Faulting linear address: ffff830088002c98 > > (XEN) **************************************** > > > I want to know where I should start debugging from. > > However, although I'm using serial console, I could get not enough > > clues only from the kernel log: > > 1) I could figure out what line and file caused the panic by its call > > trace, but it is too rough so it does not help me. > > > That's exactly from where you usually start: looking at what's at the > instruction that cause the system to explode, in your case, at address > 0xffff82d080120973. > > You can figure that out by disassembling the Xen hypervisor binary, > with `objdump', and looking up that address. Or you can use addr2line, > to have an indication of the same thing, but in the C sources. > > I'm not sure what you mean with "it is too rough". At least the address > of the instruction that caused the system to fail in the Xen binary, is > usually pretty accurate (then, of course, you have to look at > surrounding instructions, check how you got there, etc.). > > Again, make sure you use a debug hypervisor. > > > 2) What linear address brings about this situation; 'Faulting linear > > address', but it is just an address and not recognizable something > > that human cannot read. > > > > I think, literally, the 'Faulting linear address' is key point > > because I heard that it represents bad address that I should never > > access. > > > It is, but the only way of understanding why you hit such an access > violation, is understand what the code is doing when it happens. > > Regards, > Dario > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: Type: text/html, Size: 9644 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Faulting linear address?? 2017-09-10 17:00 ` Minjun Hong @ 2017-09-11 13:49 ` Dario Faggioli 0 siblings, 0 replies; 4+ messages in thread From: Dario Faggioli @ 2017-09-11 13:49 UTC (permalink / raw) To: Minjun Hong; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 3445 bytes --] On Mon, 2017-09-11 at 02:00 +0900, Minjun Hong wrote: > I made the new Xen4.5 binary with 'debug=y' option that I modified > and install it. > Then, there was a kernel panic caused by the debugging code triggered > by 'debug=y' during booting process(of dom0): > Once again, can you show us here what you are changing? > (XEN) ----[ Xen-4.5.0 x86_64 debug=y Not tainted ]---- > (XEN) CPU: 7 > (XEN) RIP: e008:[<ffff82d08012ba6c>] vcpu_migrate+0x1bd/0x374 > (XEN) RFLAGS: 0000000000010096 CONTEXT: hypervisor > [...] > (XEN) Xen call trace: > (XEN) [<ffff82d08012ba6c>] vcpu_migrate+0x1bd/0x374 > (XEN) [<ffff82d08012be23>] vcpu_force_reschedule+0x9e/0xa7 > (XEN) [<ffff82d080107052>] do_vcpu_op+0x2e7/0x69d > (XEN) [<ffff82d080234d9b>] syscall_enter+0xeb/0x145 > (XEN) > (XEN) Pagetable walk from ffff82d081422020: > (XEN) L4[0x105] = 0000000086092063 ffffffffffffffff > (XEN) L3[0x142] = 000000008608f063 ffffffffffffffff > (XEN) L2[0x00a] = 0000000000000000 ffffffffffffffff > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 7: > (XEN) FATAL PAGE FAULT > (XEN) [error_code=0000] > (XEN) Faulting linear address: ffff82d081422020 > (XEN) **************************************** > > Because I received a solution from my professor, I think it is a hard > work to change Xen version. > This makes it a bit harder for us to give effective advices, but if you really can't move forward, then fine, we still can at least try. > Anyway, even if I turned on the 'debug=y' option, I could not get > accurate information like with 'debug=n'; I get only linear > address(ffff82d081422020). > Well, the difference is that now, if Xen is compiled with frame pointers, we are (much more) sure that the stack trace is accurate, i.e., about where the problem is actually happening, and how you got there. > So, I want to use a dis-assembly utility like 'addr2line' or > 'objdump', which binaries can I use as input to the utility? > I'm using Ubuntu and previously I used '/boot/xen-syms-4.5.0' as > input to the utilities. > Yes, if that is the binary of the hypervisor you compiled (with debug=y), that's what you should use. You should also have it, in the source tree, as xen/xen-syms. > But I could get wrong information, which told me a code line that is > never related this problem. > The address you pass to addr2line is not the 'Faulting linear address'. It must be the address of the instruction that was being executing when the exception occurred. IOW, you shall use ffff82d08012ba6c, from here: (XEN) [<ffff82d08012ba6c>] vcpu_migrate+0x1bd/0x374 Which, in fact, is the address present in the program counter register (RIP): (XEN) RIP: e008:[<ffff82d08012ba6c>] vcpu_migrate+0x1bd/0x374 > I know that a beginner in the Xen developer community like me might > be annoying you, but I ask you one more help. > You being a beginner is not a problem. :-) The biggest problem I have right now, while trying to help you, is that I can't see what you are doing, and how you're changing the code. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 819 bytes --] [-- Attachment #2: Type: text/plain, Size: 127 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-09-11 13:49 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-09-08 5:33 Faulting linear address?? Minjun Hong 2017-09-08 7:40 ` Dario Faggioli 2017-09-10 17:00 ` Minjun Hong 2017-09-11 13:49 ` Dario Faggioli
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).