crash read error: kernel virtual address / vmcore address mismatch ?

crash read error: kernel virtual address / vmcore address mismatch ?

[Anand Raj Manickam]

Hi ,
Sorry about re posting this as i did not find solution ...

I m facing a issue where on

#crash /data/linux-2.6.30.8/vmlinux /proc/vmcore

crash 6.1.6
Copyright (C) 2002-2013  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

crash: read error: kernel virtual address: c127a9c8  type: "cpu_possible_mask"


I did follow few threads around , I have a case similar to
https://www.redhat.com/archives/crash-utility/2010-August/msg00029.html

Both the System Kernel and the Debug Kernel are the same.

My current config for

CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x400000

The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel.
On the System Kernel it starts from 0x400000 .On the Debug kernel it
starts from 0x1000000 .

if  i change the CONFIG_PHYSICAL_START=0x400000 . It does NOT core dump.

my crashkernel=128M is the setting on grub.
Let me know if you need more info



Anand

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [Crash-utility] crash read error: kernel virtual address / vmcore address mismatch ?

[Dave Anderson]

----- Original Message -----
> Hi ,
> Sorry about re posting this as i did not find solution ...
> 
> I m facing a issue where on
> 
> #crash /data/linux-2.6.30.8/vmlinux /proc/vmcore

Do yourself a favor and copy /proc/vmcore to somewhere on disk.  Then
reboot the system into the original kernel, and run crash on the saved
vmcore.  I've never seen crash run on a /proc/vmcore file from the
secondary kernel.  

Also, after rebooting the the original kernel, please confirm that
crash runs OK on the live system.
 
> crash 6.1.6
> Copyright (C) 2002-2013  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for details.
> 
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu"...
> 
> crash: read error: kernel virtual address: c127a9c8  type: "cpu_possible_mask"

Please show the full output of "crash -d8 vmlinux vmcore".
 
> 
> I did follow few threads around , I have a case similar to
> https://www.redhat.com/archives/crash-utility/2010-August/msg00029.html
> 
> Both the System Kernel and the Debug Kernel are the same.

I don't know what you mean by "System Kernel" and "Debug Kernel". 

With respect to kdump and the crash utility, the only kernel that is
of interest is the vmlinux that was running when the primary system
crashed.

> 
> My current config for
> 
> CONFIG_PHYSICAL_START=0x1000000
> CONFIG_RELOCATABLE=y
> CONFIG_PHYSICAL_ALIGN=0x400000
> 
> The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel.
> On the System Kernel it starts from 0x400000 .On the Debug kernel it
> starts from 0x1000000.

Again, the "Debug" kernel is irrelevant -- presuming by "Debug" you mean
the relocated "secondary" kernel that was kexec'd after the primary kernel
crashed.

> 
> if  i change the CONFIG_PHYSICAL_START=0x400000 . It does NOT core dump.
> 
> my crashkernel=128M is the setting on grub.
> Let me know if you need more info
> 

What "core dumps", the crash utility?  Again, trying to run the crash utility
while running on a 128MB secondary kernel is definitely not advisable.  

Dave


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [Crash-utility] crash read error: kernel virtual address / vmcore address mismatch ?

[Dave Anderson]

----- Original Message -----
> On Wed, May 8, 2013 at 11:23 AM, Anand Raj Manickam <anandrm@gmail.com>
> wrote:
> >> Hi ,
> >> Sorry about re posting this as i did not find solution ...
> >>
> >> I m facing a issue where on
> >>
> >> #crash /data/linux-2.6.30.8/vmlinux /proc/vmcore
> >
> > Do yourself a favor and copy /proc/vmcore to somewhere on disk.  Then
> > reboot the system into the original kernel, and run crash on the saved
> > vmcore.  I've never seen crash run on a /proc/vmcore file from the
> > secondary kernel.
> >
> > Also, after rebooting the the original kernel, please confirm that
> > crash runs OK on the live system.
> >
> >
> >> crash 6.1.6
> >> Copyright (C) 2002-2013  Red Hat, Inc.
> >> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> >> Copyright (C) 1999-2006  Hewlett-Packard Co
> >> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> >> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> >> Copyright (C) 2005, 2011  NEC Corporation
> >> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> >> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> >> This program is free software, covered by the GNU General Public License,
> >> and you are welcome to change it and/or distribute copies of it under
> >> certain conditions.  Enter "help copying" to see the conditions.
> >> This program has absolutely no warranty.  Enter "help warranty" for
> >> details.
> >>
> >> GNU gdb (GDB) 7.3.1
> >> Copyright (C) 2011 Free Software Foundation, Inc.
> >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> >> and "show warranty" for details.
> >> This GDB was configured as "i686-pc-linux-gnu"...
> >>
> >> crash: read error: kernel virtual address: c127a9c8  type: "cpu_possible_mask"
> >
> > Please show the full output of "crash -d8 vmlinux vmcore".
> 
> After rebooting to the primary / live kernel
> 
>    I have attached the output for "crash -d8 vmlinux /data/vmcore" in d8.txt

Looking at the ELF header data in the debug output, your configuration 
does require special handling.  X86 kernels are typically not configured
with CONFIG_PHYSICAL_START larger than CONFIG_PHYSICAL_ALIGN, because
when that is done, the vmlinux unity-mapped symbol values do not match
their relocated values when the kernel is loaded.  This entry in the 
crash.changelog describes the situation:

http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5

4.0-4.5  - Addresses FC7/upstream x86 kernels that have been configured such
           that the vmlinux symbol values do not match their relocated values
           when loaded.  If CONFIG_PHYSICAL_START contains a value that is
           greater then CONFIG_PHYSICAL_ALIGN, then this mismatch occurs.
           Since the crash utility and its embedded gdb have always expected
           that the compiled-in kernel symbol addresses are "real", the virtual
           to physical translation fails, leading to an initialization-time
           failure with the message: "crash: vmlinux and /dev/crash do not
           match!" (/dev/mem or the dumpfile name may replace /dev/crash).
           To deal with this issue, there are several alternatives:

            1) Configure the kernel with CONFIG_PHYSICAL_START less than
               or equal to CONFIG_PHYSICAL_ALIGN.  Having done that, there
               is no problem; the resultant vmlinux file will be loaded at
               the address for which it was compiled, which has always
               been the case.
            2) Since /proc/kallsyms uses the same format as a System.map file,
               and since it reflects the relocated symbol addresses, it
               can be placed on the crash command line as if it were
               a System.map file.  (Note that the System.map file created
               by these relocated kernels contains the same "wrong" symbol
               values as the vmlinux file from which it was created.)
            3) On a live system that has /proc/kallsyms (i.e., the kernel was
               configured with CONFIG_KALLSYMS), this version of the crash
               utility will replace/patch the vmlinux symbol values with those
               seen in /proc/kallsyms.  The relocation value will be displayed
               as a WARNING message during initialization.
            4) On a dumpfile, the relocation will not be performed automatically
               as on a live system.  It will require the addition of the 
               /proc/kallsyms on the command line, or if run on a different
               host, a copy of the crashed system's /proc/kallsyms may be
               used.
            5) Alternatively on a dumpfile, a new command line option has been
               created to specify the relocation amount.  For example, if a  
               kernel was configured with a CONFIG_PHYSICAL_START value of 16MB
               and a CONFIG_PHYSICAL_ALIGN of 4MB, that results in a relocation
               of 12MB.  To specify that, enter "crash --reloc=12m ..." on the
               command line.  (Recall that if crash is run on the live system,
               a WARNING message will specify the relocation amount.)

           Using /proc/kallsyms or a --reloc=[size] as a command line argument
           is similar to using a System.map file, in that it results in the loss
           of the use of line number debug data.  (anderson@redhat.com)

           ...

>> My current config for
>>
>> CONFIG_PHYSICAL_START=0x1000000
>> CONFIG_RELOCATABLE=y
>> CONFIG_PHYSICAL_ALIGN=0x400000
>>
>> The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel.
>> On the System Kernel it starts from 0x400000 .On the Debug kernel it
>> starts from 0x1000000.

So your configuration exactly matches paragraph 5) above, you should
be able to come up OK by entering either:

 $ crash vmlinux vmcore --reloc=12m

or

 $ crash vmlinux vmcore /proc/kallsyms

where the /proc/kallsyms is from the *primary* kernel, so option #2
will *not* work when running from the secondary kernel.

You didn't show any output of a live crash session (while running
on the primary kernel), but I believe it should work without using 
the /proc/kallsyms as an argument or using --reloc=12m option.  There
was a further fix for these configurations for running on live systems:

http://people.redhat.com/anderson/crash.changelog.html#5_1_4

5.1.4    - Fix for RT kernels in which the schedule() function has become a 
           ... [ cut] ...

         - Fix for running against live x86 kernels that were configured with
           CONFIG_PHYSICAL_START containing a value that is greater than its
           CONFIG_PHYSICAL_ALIGN value, and where the first symbol listed by
           /proc/kallsyms is not "_text".  Without the patch, the crash session
           fails during invocation with the error message "crash: vmlinux and
           /dev/mem do not match!" (or "/dev/crash" if applicable).  As a work-
           around, "/proc/kallsyms" can be entered on the command line, or the
           "--reloc=<size>" option could be used, but the fix obviates that
           requirement for live systems.  It should be noted that dumpfiles of
           kernels configured that way still do require that "/proc/kallsyms",
           or a copy of it, or alternatively the "--reloc=<size>" option, to
           be entered on the command line, as detailed in this changelog entry:
           http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
           (anderson@redhat.com)

So on the live system on the primary kernel, I believe this should just work:

 $ crash vmlinux

Normally you don't even need to put the vmlinux file on the command line for
running on the live system because it will be "found" automatically unless
you've put it in some non-standard location.

Dave


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [Crash-utility] crash read error: kernel virtual address / vmcore address mismatch ?

[Anand Raj Manickam]

On Wed, May 8, 2013 at 6:59 PM, Dave Anderson <anderson@redhat.com> wrote:
>
>
>
> ----- Original Message -----
>> On Wed, May 8, 2013 at 11:23 AM, Anand Raj Manickam <anandrm@gmail.com>
>> wrote:
>> >> Hi ,
>> >> Sorry about re posting this as i did not find solution ...
>> >>
>> >> I m facing a issue where on
>> >>
>> >> #crash /data/linux-2.6.30.8/vmlinux /proc/vmcore
>> >
>> > Do yourself a favor and copy /proc/vmcore to somewhere on disk.  Then
>> > reboot the system into the original kernel, and run crash on the saved
>> > vmcore.  I've never seen crash run on a /proc/vmcore file from the
>> > secondary kernel.
>> >
>> > Also, after rebooting the the original kernel, please confirm that
>> > crash runs OK on the live system.
>> >
>> >
>> >> crash 6.1.6
>> >> Copyright (C) 2002-2013  Red Hat, Inc.
>> >> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>> >> Copyright (C) 1999-2006  Hewlett-Packard Co
>> >> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>> >> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>> >> Copyright (C) 2005, 2011  NEC Corporation
>> >> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>> >> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>> >> This program is free software, covered by the GNU General Public License,
>> >> and you are welcome to change it and/or distribute copies of it under
>> >> certain conditions.  Enter "help copying" to see the conditions.
>> >> This program has absolutely no warranty.  Enter "help warranty" for
>> >> details.
>> >>
>> >> GNU gdb (GDB) 7.3.1
>> >> Copyright (C) 2011 Free Software Foundation, Inc.
>> >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> >> This is free software: you are free to change and redistribute it.
>> >> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> >> and "show warranty" for details.
>> >> This GDB was configured as "i686-pc-linux-gnu"...
>> >>
>> >> crash: read error: kernel virtual address: c127a9c8  type: "cpu_possible_mask"
>> >
>> > Please show the full output of "crash -d8 vmlinux vmcore".
>>
>> After rebooting to the primary / live kernel
>>
>>    I have attached the output for "crash -d8 vmlinux /data/vmcore" in d8.txt
>
> Looking at the ELF header data in the debug output, your configuration
> does require special handling.  X86 kernels are typically not configured
> with CONFIG_PHYSICAL_START larger than CONFIG_PHYSICAL_ALIGN, because
> when that is done, the vmlinux unity-mapped symbol values do not match
> their relocated values when the kernel is loaded.  This entry in the
> crash.changelog describes the situation:
>
> http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
>
> 4.0-4.5  - Addresses FC7/upstream x86 kernels that have been configured such
>            that the vmlinux symbol values do not match their relocated values
>            when loaded.  If CONFIG_PHYSICAL_START contains a value that is
>            greater then CONFIG_PHYSICAL_ALIGN, then this mismatch occurs.
>            Since the crash utility and its embedded gdb have always expected
>            that the compiled-in kernel symbol addresses are "real", the virtual
>            to physical translation fails, leading to an initialization-time
>            failure with the message: "crash: vmlinux and /dev/crash do not
>            match!" (/dev/mem or the dumpfile name may replace /dev/crash).
>            To deal with this issue, there are several alternatives:
>
>             1) Configure the kernel with CONFIG_PHYSICAL_START less than
>                or equal to CONFIG_PHYSICAL_ALIGN.  Having done that, there
>                is no problem; the resultant vmlinux file will be loaded at
>                the address for which it was compiled, which has always
>                been the case.
>             2) Since /proc/kallsyms uses the same format as a System.map file,
>                and since it reflects the relocated symbol addresses, it
>                can be placed on the crash command line as if it were
>                a System.map file.  (Note that the System.map file created
>                by these relocated kernels contains the same "wrong" symbol
>                values as the vmlinux file from which it was created.)
>             3) On a live system that has /proc/kallsyms (i.e., the kernel was
>                configured with CONFIG_KALLSYMS), this version of the crash
>                utility will replace/patch the vmlinux symbol values with those
>                seen in /proc/kallsyms.  The relocation value will be displayed
>                as a WARNING message during initialization.
>             4) On a dumpfile, the relocation will not be performed automatically
>                as on a live system.  It will require the addition of the
>                /proc/kallsyms on the command line, or if run on a different
>                host, a copy of the crashed system's /proc/kallsyms may be
>                used.
>             5) Alternatively on a dumpfile, a new command line option has been
>                created to specify the relocation amount.  For example, if a
>                kernel was configured with a CONFIG_PHYSICAL_START value of 16MB
>                and a CONFIG_PHYSICAL_ALIGN of 4MB, that results in a relocation
>                of 12MB.  To specify that, enter "crash --reloc=12m ..." on the
>                command line.  (Recall that if crash is run on the live system,
>                a WARNING message will specify the relocation amount.)
>
>            Using /proc/kallsyms or a --reloc=[size] as a command line argument
>            is similar to using a System.map file, in that it results in the loss
>            of the use of line number debug data.  (anderson@redhat.com)
>
>            ...
>
>>> My current config for
>>>
>>> CONFIG_PHYSICAL_START=0x1000000
>>> CONFIG_RELOCATABLE=y
>>> CONFIG_PHYSICAL_ALIGN=0x400000
>>>
>>> The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel.
>>> On the System Kernel it starts from 0x400000 .On the Debug kernel it
>>> starts from 0x1000000.
>
> So your configuration exactly matches paragraph 5) above, you should
> be able to come up OK by entering either:
>

>  $ crash vmlinux vmcore --reloc=12m

This did the trick :-)

Great !! and Thanks Dave.

>
> or
>
>  $ crash vmlinux vmcore /proc/kallsyms
>
> where the /proc/kallsyms is from the *primary* kernel, so option #2
> will *not* work when running from the secondary kernel.
>
> You didn't show any output of a live crash session (while running
> on the primary kernel), but I believe it should work without using
> the /proc/kallsyms as an argument or using --reloc=12m option.  There
> was a further fix for these configurations for running on live systems:
>
> http://people.redhat.com/anderson/crash.changelog.html#5_1_4
>
> 5.1.4    - Fix for RT kernels in which the schedule() function has become a
>            ... [ cut] ...
>
>          - Fix for running against live x86 kernels that were configured with
>            CONFIG_PHYSICAL_START containing a value that is greater than its
>            CONFIG_PHYSICAL_ALIGN value, and where the first symbol listed by
>            /proc/kallsyms is not "_text".  Without the patch, the crash session
>            fails during invocation with the error message "crash: vmlinux and
>            /dev/mem do not match!" (or "/dev/crash" if applicable).  As a work-
>            around, "/proc/kallsyms" can be entered on the command line, or the
>            "--reloc=<size>" option could be used, but the fix obviates that
>            requirement for live systems.  It should be noted that dumpfiles of
>            kernels configured that way still do require that "/proc/kallsyms",
>            or a copy of it, or alternatively the "--reloc=<size>" option, to
>            be entered on the command line, as detailed in this changelog entry:
>            http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
>            (anderson@redhat.com)
>
> So on the live system on the primary kernel, I believe this should just work:
>
>  $ crash vmlinux
>
> Normally you don't even need to put the vmlinux file on the command line for
> running on the live system because it will be "found" automatically unless
> you've put it in some non-standard location.
>
> Dave
>

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec