2.6.19-rc1: kexec broken on x86_64

2.6.19-rc1: kexec broken on x86_64

[Magnus Damm]

Kexec is broken on x86_64 under 2.6.19-rc1.

Or rather - kexec works ok under 2.6.19-rc1, but something related to
the vmlinux format has probably changed and kexec-tools fails to load
a vmlinux from 2.6.19-rc1.

Loading bzImage works as usual, but vmlinux does not load properly.

The kexec binary fails with the following message:

Overlapping memory segments at 0x351000
sort_segments failed
/ #

Again, running kexec under 2.6.19-rc1 works ok. But rebooting into a
2.6.19-rc1 vmlinux does not work correctly. This is regardless of
which kexec-tools tree used.

A coarse-grained bisect says that the problem was introduced in 2.6.18-git7.

But maybe this is a kexec-tools issue?

I've used Horms testing tree as usual, tested the latest version
5aa1e11a27f1dc1ce96f850966e94f68c9cd8bce to be exact. Horms testing
tree that can be found here:

http://www.kernel.org/git/?p=linux/kernel/git/horms/kexec-tools-testing.git;a=summary

I've also tried version 71d2424c8ac4f93a60c3eee5c95df269f584a9da of
Eric's main tree:

http://www.kernel.org/git/?p=linux/kernel/git/ebiederm/kexec-tools.git;a=summary

That tree is unfortunately broken on x86_64. You need to apply the
follwing patch on x86_64 to be able to compile:

http://www.kernel.org/git/?p=linux/kernel/git/horms/kexec-tools-testing.git;a=commit;h=6492b850281a106c042221b836a693141fd9b49b

None of the kexec-tools trees can load a 2.6.19-rc1 vmlinux under
x86_64. i386 works fine.

/ magnus

Re: 2.6.19-rc1: kexec broken on x86_64

[Vivek Goyal]

On Thu, Oct 05, 2006 at 07:28:35PM +0900, Magnus Damm wrote:
> Kexec is broken on x86_64 under 2.6.19-rc1.
> 
> Or rather - kexec works ok under 2.6.19-rc1, but something related to
> the vmlinux format has probably changed and kexec-tools fails to load
> a vmlinux from 2.6.19-rc1.
> 
> Loading bzImage works as usual, but vmlinux does not load properly.
> 
> The kexec binary fails with the following message:
> 
> Overlapping memory segments at 0x351000
> sort_segments failed
> / #
> 

Hi Magnus,

Can you please post the readelf -l output of the vmlinux you are trying
to load. That's will give some indication if the segments are really
overlapping in vmlinux or is it some processing bug at kexec-tools part.

Thanks
Vivek

Re: 2.6.19-rc1: kexec broken on x86_64

[Magnus Damm]

Hi Vivek,

On 10/5/06, Vivek Goyal <vgoyal@in.ibm.com> wrote:
> On Thu, Oct 05, 2006 at 07:28:35PM +0900, Magnus Damm wrote:
> > Kexec is broken on x86_64 under 2.6.19-rc1.
> >
> > Or rather - kexec works ok under 2.6.19-rc1, but something related to
> > the vmlinux format has probably changed and kexec-tools fails to load
> > a vmlinux from 2.6.19-rc1.
> >
> > Loading bzImage works as usual, but vmlinux does not load properly.
> >
> > The kexec binary fails with the following message:
> >
> > Overlapping memory segments at 0x351000
> > sort_segments failed
> > / #
> >
>
> Hi Magnus,
>
> Can you please post the readelf -l output of the vmlinux you are trying
> to load. That's will give some indication if the segments are really
> overlapping in vmlinux or is it some processing bug at kexec-tools part.

Elf file type is EXEC (Executable file)
Entry point 0x100100
There are 4 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000100000 0xffffffff80100000 0x0000000000100000
                 0x00000000001a4888 0x00000000001a4888  R E    100000
  LOAD           0x00000000002a5000 0xffffffff802a5000 0x00000000002a5000
                 0x000000000008e086 0x00000000000c1504  RWE    100000
  LOAD           0x0000000000400000 0xffffffffff600000 0x00000000002fd000
                 0x0000000000000c08 0x0000000000000c08  RWE    100000
  NOTE           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  R      8

 Section to Segment mapping:
  Segment Sections...
   00     .text __ex_table .rodata .pci_fixup __ksymtab __ksymtab_gpl
__ksymtab_unused __ksymtab_strings __param
   01     .data .data.cacheline_aligned .data.read_mostly
.data.init_task .data.page_aligned .init.text .init.data .init.setup
.initcall.init .con_initcall.init .altinstructions
.altinstr_replacement .exit.text .init.ramfs .bss
   02     .vsyscall_0 .xtime_lock .vxtime .vgetcpu_mode .sys_tz
.sysctl_vsyscall .xtime .jiffies .vsyscall_1 .vsyscall_2 .vsyscall_3
   03

Thanks,

/ magnus

Re: 2.6.19-rc1: kexec broken on x86_64

[Vivek Goyal]

On Thu, Oct 05, 2006 at 10:56:51PM +0900, Magnus Damm wrote:
> Hi Vivek,
> 
> On 10/5/06, Vivek Goyal <vgoyal@in.ibm.com> wrote:
> >On Thu, Oct 05, 2006 at 07:28:35PM +0900, Magnus Damm wrote:
> >> Kexec is broken on x86_64 under 2.6.19-rc1.
> >>
> >> Or rather - kexec works ok under 2.6.19-rc1, but something related to
> >> the vmlinux format has probably changed and kexec-tools fails to load
> >> a vmlinux from 2.6.19-rc1.
> >>
> >> Loading bzImage works as usual, but vmlinux does not load properly.
> >>
> >> The kexec binary fails with the following message:
> >>
> >> Overlapping memory segments at 0x351000
> >> sort_segments failed
> >> / #
> >>
> >
> >Hi Magnus,
> >
> >Can you please post the readelf -l output of the vmlinux you are trying
> >to load. That's will give some indication if the segments are really
> >overlapping in vmlinux or is it some processing bug at kexec-tools part.
> 
> Elf file type is EXEC (Executable file)
> Entry point 0x100100
> There are 4 program headers, starting at offset 64
> 
> Program Headers:
>  Type           Offset             VirtAddr           PhysAddr
>                 FileSiz            MemSiz              Flags  Align
>  LOAD           0x0000000000100000 0xffffffff80100000 0x0000000000100000
>                 0x00000000001a4888 0x00000000001a4888  R E    100000
>  LOAD           0x00000000002a5000 0xffffffff802a5000 0x00000000002a5000
>                 0x000000000008e086 0x00000000000c1504  RWE    100000
>  LOAD           0x0000000000400000 0xffffffffff600000 0x00000000002fd000
>                 0x0000000000000c08 0x0000000000000c08  RWE    100000
>  NOTE           0x0000000000000000 0x0000000000000000 0x0000000000000000
>                 0x0000000000000000 0x0000000000000000  R      8
> 
> Section to Segment mapping:
>  Segment Sections...
>   00     .text __ex_table .rodata .pci_fixup __ksymtab __ksymtab_gpl
> __ksymtab_unused __ksymtab_strings __param
>   01     .data .data.cacheline_aligned .data.read_mostly
> .data.init_task .data.page_aligned .init.text .init.data .init.setup
> .initcall.init .con_initcall.init .altinstructions
> .altinstr_replacement .exit.text .init.ramfs .bss
>   02     .vsyscall_0 .xtime_lock .vxtime .vgetcpu_mode .sys_tz
> .sysctl_vsyscall .xtime .jiffies .vsyscall_1 .vsyscall_2 .vsyscall_3
>   03
> 

Ok. So second and third program header are overlapping in physical
address space and that's why kexec-tools is cribbing.

Looking at these headers, it looks like program header 02 which contains
will overwrite some of the data of program header 01 and I think that's
wrong. Will look more into it.

Thanks
Vivek

Re: 2.6.19-rc1: kexec broken on x86_64

[Eric W. Biederman]

"Magnus Damm" <magnus.damm@gmail.com> writes:

> Hi Vivek,
>
> On 10/5/06, Vivek Goyal <vgoyal@in.ibm.com> wrote:
>> On Thu, Oct 05, 2006 at 07:28:35PM +0900, Magnus Damm wrote:
>> > Kexec is broken on x86_64 under 2.6.19-rc1.
>> >
>> > Or rather - kexec works ok under 2.6.19-rc1, but something related to
>> > the vmlinux format has probably changed and kexec-tools fails to load
>> > a vmlinux from 2.6.19-rc1.
>> >
>> > Loading bzImage works as usual, but vmlinux does not load properly.
>> >
>> > The kexec binary fails with the following message:
>> >
>> > Overlapping memory segments at 0x351000
>> > sort_segments failed
>> > / #
>> >
>>
>> Hi Magnus,
>>
>> Can you please post the readelf -l output of the vmlinux you are trying
>> to load. That's will give some indication if the segments are really
>> overlapping in vmlinux or is it some processing bug at kexec-tools part.
>
> Elf file type is EXEC (Executable file)
> Entry point 0x100100
> There are 4 program headers, starting at offset 64
>
> Program Headers:
>  Type           Offset             VirtAddr           PhysAddr
>                 FileSiz            MemSiz              Flags  Align
>  LOAD           0x0000000000100000 0xffffffff80100000 0x0000000000100000
>                 0x00000000001a4888 0x00000000001a4888  R E    100000
>  LOAD           0x00000000002a5000 0xffffffff802a5000 0x00000000002a5000
>                 0x000000000008e086 0x00000000000c1504  RWE    100000
>  LOAD           0x0000000000400000 0xffffffffff600000 0x00000000002fd000
>                 0x0000000000000c08 0x0000000000000c08  RWE    100000
>  NOTE           0x0000000000000000 0x0000000000000000 0x0000000000000000
>                 0x0000000000000000 0x0000000000000000  R      8
>
> Section to Segment mapping:
>  Segment Sections...
>   00     .text __ex_table .rodata .pci_fixup __ksymtab __ksymtab_gpl
> __ksymtab_unused __ksymtab_strings __param
>   01     .data .data.cacheline_aligned .data.read_mostly
> .data.init_task .data.page_aligned .init.text .init.data .init.setup
> .initcall.init .con_initcall.init .altinstructions
> .altinstr_replacement .exit.text .init.ramfs .bss
>   02     .vsyscall_0 .xtime_lock .vxtime .vgetcpu_mode .sys_tz
> .sysctl_vsyscall .xtime .jiffies .vsyscall_1 .vsyscall_2 .vsyscall_3
>   03
>
> Thanks,

Ok.  There does not appear anything here that is not page aligned.  So
it looks like something is triggering a kexec-tools bug.

Eric

Re: 2.6.19-rc1: kexec broken on x86_64

[Eric W. Biederman]

Vivek Goyal <vgoyal@in.ibm.com> writes:

> On Thu, Oct 05, 2006 at 10:56:51PM +0900, Magnus Damm wrote:
>> Hi Vivek,
>> 
>> On 10/5/06, Vivek Goyal <vgoyal@in.ibm.com> wrote:
>> >On Thu, Oct 05, 2006 at 07:28:35PM +0900, Magnus Damm wrote:
>> >> Kexec is broken on x86_64 under 2.6.19-rc1.
>> >>
>> >> Or rather - kexec works ok under 2.6.19-rc1, but something related to
>> >> the vmlinux format has probably changed and kexec-tools fails to load
>> >> a vmlinux from 2.6.19-rc1.
>> >>
>> >> Loading bzImage works as usual, but vmlinux does not load properly.
>> >>
>> >> The kexec binary fails with the following message:
>> >>
>> >> Overlapping memory segments at 0x351000
>> >> sort_segments failed
>> >> / #
>> >>
>> >
>> >Hi Magnus,
>> >
>> >Can you please post the readelf -l output of the vmlinux you are trying
>> >to load. That's will give some indication if the segments are really
>> >overlapping in vmlinux or is it some processing bug at kexec-tools part.
>> 
>> Elf file type is EXEC (Executable file)
>> Entry point 0x100100
>> There are 4 program headers, starting at offset 64
>> 
>> Program Headers:
>>  Type           Offset             VirtAddr           PhysAddr
>>                 FileSiz            MemSiz              Flags  Align
>>  LOAD           0x0000000000100000 0xffffffff80100000 0x0000000000100000
>>                 0x00000000001a4888 0x00000000001a4888  R E    100000
>>  LOAD           0x00000000002a5000 0xffffffff802a5000 0x00000000002a5000
>>                 0x000000000008e086 0x00000000000c1504  RWE    100000
>>  LOAD           0x0000000000400000 0xffffffffff600000 0x00000000002fd000
>>                 0x0000000000000c08 0x0000000000000c08  RWE    100000
>>  NOTE           0x0000000000000000 0x0000000000000000 0x0000000000000000
>>                 0x0000000000000000 0x0000000000000000  R      8
>> 
>> Section to Segment mapping:
>>  Segment Sections...
>>   00     .text __ex_table .rodata .pci_fixup __ksymtab __ksymtab_gpl
>> __ksymtab_unused __ksymtab_strings __param
>>   01     .data .data.cacheline_aligned .data.read_mostly
>> .data.init_task .data.page_aligned .init.text .init.data .init.setup
>> .initcall.init .con_initcall.init .altinstructions
>> .altinstr_replacement .exit.text .init.ramfs .bss
>>   02     .vsyscall_0 .xtime_lock .vxtime .vgetcpu_mode .sys_tz
>> .sysctl_vsyscall .xtime .jiffies .vsyscall_1 .vsyscall_2 .vsyscall_3
>>   03
>> 
>
> Ok. So second and third program header are overlapping in physical
> address space and that's why kexec-tools is cribbing.
>
> Looking at these headers, it looks like program header 02 which contains
> will overwrite some of the data of program header 01 and I think that's
> wrong. Will look more into it.

Ack.  I forgot to check the lengths...
That last segment (which I presume is the vdso) is quite short.

Eric

Re: 2.6.19-rc1: kexec broken on x86_64

[Vivek Goyal]

On Thu, Oct 05, 2006 at 10:56:51PM +0900, Magnus Damm wrote:
> Hi Vivek,
> 
> On 10/5/06, Vivek Goyal <vgoyal@in.ibm.com> wrote:
> >On Thu, Oct 05, 2006 at 07:28:35PM +0900, Magnus Damm wrote:
> >> Kexec is broken on x86_64 under 2.6.19-rc1.
> >>
> >> Or rather - kexec works ok under 2.6.19-rc1, but something related to
> >> the vmlinux format has probably changed and kexec-tools fails to load
> >> a vmlinux from 2.6.19-rc1.
> >>
> >> Loading bzImage works as usual, but vmlinux does not load properly.
> >>
> >> The kexec binary fails with the following message:
> >>
> >> Overlapping memory segments at 0x351000
> >> sort_segments failed
> >> / #
> >>
> >
> >Hi Magnus,
> >
> >Can you please post the readelf -l output of the vmlinux you are trying
> >to load. That's will give some indication if the segments are really
> >overlapping in vmlinux or is it some processing bug at kexec-tools part.
> 
> Elf file type is EXEC (Executable file)
> Entry point 0x100100
> There are 4 program headers, starting at offset 64
> 
> Program Headers:
>  Type           Offset             VirtAddr           PhysAddr
>                 FileSiz            MemSiz              Flags  Align
>  LOAD           0x0000000000100000 0xffffffff80100000 0x0000000000100000
>                 0x00000000001a4888 0x00000000001a4888  R E    100000
>  LOAD           0x00000000002a5000 0xffffffff802a5000 0x00000000002a5000
>                 0x000000000008e086 0x00000000000c1504  RWE    100000
>  LOAD           0x0000000000400000 0xffffffffff600000 0x00000000002fd000
>                 0x0000000000000c08 0x0000000000000c08  RWE    100000
>  NOTE           0x0000000000000000 0x0000000000000000 0x0000000000000000
>                 0x0000000000000000 0x0000000000000000  R      8
> 
> Section to Segment mapping:
>  Segment Sections...
>   00     .text __ex_table .rodata .pci_fixup __ksymtab __ksymtab_gpl
> __ksymtab_unused __ksymtab_strings __param
>   01     .data .data.cacheline_aligned .data.read_mostly
> .data.init_task .data.page_aligned .init.text .init.data .init.setup
> .initcall.init .con_initcall.init .altinstructions
> .altinstr_replacement .exit.text .init.ramfs .bss
>   02     .vsyscall_0 .xtime_lock .vxtime .vgetcpu_mode .sys_tz
> .sysctl_vsyscall .xtime .jiffies .vsyscall_1 .vsyscall_2 .vsyscall_3
>   03
>

Hi Magnus,

I think this got introduced because of Ian Cambell's patch for creating
PT_NOTE headers. Can you please try attached patch. I think it should
fix the issue.

Thanks
Vivek
 


o A recent change to vmlinux.ld.S file broke kexec as now resulting vmlinux
  program headers are overlapping in physical address space.

o Now all the vsyscall related sections are placed after data and after
  that mostly init data sections are placed. To avoid physical overlap
  among phdrs, there are three possible solutions.	
	- Place vsyscall sections also in data phdrs instead of user
	- move vsyscal sections after init data in bss.
	- create another phdrs say data.init and move all the sections
	  after vsyscall into this new phdr.

o This patch implements the third solution. 

Signed-off-by: Vivek Goyal <vgoyal@in.ibm.com>
---

 arch/x86_64/kernel/vmlinux.lds.S |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff -puN arch/x86_64/kernel/vmlinux.lds.S~x86_64-physical-addr-space-overlap-in-phdrs-fix arch/x86_64/kernel/vmlinux.lds.S
--- linux-2.6.19-rc1-1M/arch/x86_64/kernel/vmlinux.lds.S~x86_64-physical-addr-space-overlap-in-phdrs-fix	2006-10-05 12:15:00.000000000 -0400
+++ linux-2.6.19-rc1-1M-root/arch/x86_64/kernel/vmlinux.lds.S	2006-10-05 12:15:00.000000000 -0400
@@ -17,6 +17,7 @@ PHDRS {
 	text PT_LOAD FLAGS(5);	/* R_E */
 	data PT_LOAD FLAGS(7);	/* RWE */
 	user PT_LOAD FLAGS(7);	/* RWE */
+	data.init PT_LOAD FLAGS(7);	/* RWE */
 	note PT_NOTE FLAGS(4);	/* R__ */
 }
 SECTIONS
@@ -131,7 +132,7 @@ SECTIONS
   . = ALIGN(8192);		/* init_task */
   .data.init_task : AT(ADDR(.data.init_task) - LOAD_OFFSET) {
 	*(.data.init_task)
-  } :data
+  }:data.init
 
   . = ALIGN(4096);
   .data.page_aligned : AT(ADDR(.data.page_aligned) - LOAD_OFFSET) {
_

Re: 2.6.19-rc1: kexec broken on x86_64

[Magnus Damm]

On Thu, 2006-10-05 at 12:33 -0400, Vivek Goyal wrote:
> On Thu, Oct 05, 2006 at 10:56:51PM +0900, Magnus Damm wrote:
> > Hi Vivek,
> > 
> > On 10/5/06, Vivek Goyal <vgoyal@in.ibm.com> wrote:
> > >On Thu, Oct 05, 2006 at 07:28:35PM +0900, Magnus Damm wrote:
> > >> Kexec is broken on x86_64 under 2.6.19-rc1.
> > >>
> > >> Or rather - kexec works ok under 2.6.19-rc1, but something related to
> > >> the vmlinux format has probably changed and kexec-tools fails to load
> > >> a vmlinux from 2.6.19-rc1.
> > >>
> > >> Loading bzImage works as usual, but vmlinux does not load properly.
> > >>
> > >> The kexec binary fails with the following message:
> > >>
> > >> Overlapping memory segments at 0x351000
> > >> sort_segments failed
> > >> / #
> > >>
> > >
> > >Hi Magnus,
> > >
> > >Can you please post the readelf -l output of the vmlinux you are trying
> > >to load. That's will give some indication if the segments are really
> > >overlapping in vmlinux or is it some processing bug at kexec-tools part.
> > 
> > Elf file type is EXEC (Executable file)
> > Entry point 0x100100
> > There are 4 program headers, starting at offset 64
> > 
> > Program Headers:
> >  Type           Offset             VirtAddr           PhysAddr
> >                 FileSiz            MemSiz              Flags  Align
> >  LOAD           0x0000000000100000 0xffffffff80100000 0x0000000000100000
> >                 0x00000000001a4888 0x00000000001a4888  R E    100000
> >  LOAD           0x00000000002a5000 0xffffffff802a5000 0x00000000002a5000
> >                 0x000000000008e086 0x00000000000c1504  RWE    100000
> >  LOAD           0x0000000000400000 0xffffffffff600000 0x00000000002fd000
> >                 0x0000000000000c08 0x0000000000000c08  RWE    100000
> >  NOTE           0x0000000000000000 0x0000000000000000 0x0000000000000000
> >                 0x0000000000000000 0x0000000000000000  R      8
> > 
> > Section to Segment mapping:
> >  Segment Sections...
> >   00     .text __ex_table .rodata .pci_fixup __ksymtab __ksymtab_gpl
> > __ksymtab_unused __ksymtab_strings __param
> >   01     .data .data.cacheline_aligned .data.read_mostly
> > .data.init_task .data.page_aligned .init.text .init.data .init.setup
> > .initcall.init .con_initcall.init .altinstructions
> > .altinstr_replacement .exit.text .init.ramfs .bss
> >   02     .vsyscall_0 .xtime_lock .vxtime .vgetcpu_mode .sys_tz
> > .sysctl_vsyscall .xtime .jiffies .vsyscall_1 .vsyscall_2 .vsyscall_3
> >   03
> >
> 
> Hi Magnus,
> 
> I think this got introduced because of Ian Cambell's patch for creating
> PT_NOTE headers. Can you please try attached patch. I think it should
> fix the issue.

You are right, the patch solves the problem. It is now possible to use
kexec on x86_64 to reboot into a vmlinux built from 2.6.19-rc1 + your
patch.

Nice work, thanks!

/ magnus