ftrace 'failed to modify' bug when loading reiserfs.ko

ftrace 'failed to modify' bug when loading reiserfs.ko

[Dave Jones]

For whatever dumb reason, when running 'make install' on a Fedora system,
os-prober tries to figure out what filesystems are needed by loading filesystems,
and seeing what sticks..  Today it blew up spectacularly when it got to
loading reiserfs..  System wedged entirely afterwards.

	Dave

------------[ cut here ]------------
WARNING: CPU: 2 PID: 30566 at kernel/trace/ftrace.c:1694 ftrace_bug+0x25d/0x270()
Modules linked in: reiserfs(+) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc xfs snd_timer libcrc32c snd e1000e ptp usb_debug pps_core pcspkr soundcore
CPU: 2 PID: 30566 Comm: modprobe Not tainted 3.11.0+ #57 
 ffffffff81a2809d ffff88008de19c30 ffffffff817171e9 0000000000000000
 ffff88008de19c68 ffffffff81053dad 0000000000000010 ffffffffa02738b0
 ffff8802419e3518 0000000000000000 ffff8801ab16e100 ffff88008de19c78
Call Trace:
 [<ffffffff817171e9>] dump_stack+0x54/0x74
 [<ffffffff81053dad>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff81053e8a>] warn_slowpath_null+0x1a/0x20
 [<ffffffff8111924d>] ftrace_bug+0x25d/0x270
 [<ffffffff81119568>] ftrace_process_locs+0x308/0x630
 [<ffffffff811198cc>] ftrace_module_notify_enter+0x3c/0x40
 [<ffffffff817257c6>] notifier_call_chain+0x66/0x150
 [<ffffffff81088d97>] __blocking_notifier_call_chain+0x67/0xc0
 [<ffffffff81088e06>] blocking_notifier_call_chain+0x16/0x20
 [<ffffffff810d23cd>] load_module+0x1f7d/0x2680
 [<ffffffff810cd6f0>] ? store_uevent+0x40/0x40
 [<ffffffffa0240000>] ? reiserfs_xattr_register_handlers+0xf9f/0xf9f [reiserfs]
 [<ffffffffa0240000>] ? reiserfs_xattr_register_handlers+0xf9f/0xf9f [reiserfs]
 [<ffffffff810d2c66>] SyS_finit_module+0x86/0xb0
 [<ffffffff8172aa14>] tracesys+0xdd/0xe2
---[ end trace 956db59f53237fe4 ]---
ftrace failed to modify [<ffffffffa02738b0>] reiserfs_init_bitmap_cache+0x0/0xffffffffffff5750 [reiserfs]
 actual: 14:00:00:00:00
------------[ cut here ]------------
WARNING: CPU: 2 PID: 30566 at arch/x86/mm/pageattr.c:677 __cpa_process_fault+0x91/0xa0()
CPA: called for zero pte. vaddr = ffffffffa0249000 cpa->vaddr = ffffffffa0249000
Modules linked in: reiserfs(+) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc xfs snd_timer libcrc32c snd e1000e ptp usb_debug pps_core pcspkr soundcore
CPU: 2 PID: 30566 Comm: modprobe Tainted: G        W    3.11.0+ #57 
 ffffffff81a0ba44 ffff88008de19b40 ffffffff817171e9 ffff88008de19b88
 ffff88008de19b78 ffffffff81053dad ffff88008de19d08 00000000fffffff2
 ffffffffa0249000 ffff880238646248 ffff88008de19d08 ffff88008de19bd8
Call Trace:
 [<ffffffff817171e9>] dump_stack+0x54/0x74
 [<ffffffff81053dad>] warn_slowpath_common+0x7d/0xa0
 [<ffffffffa0249000>] ? reiserfs_xattr_register_handlers+0x9f9f/0x29f9f [reiserfs]
 [<ffffffff81053e1c>] warn_slowpath_fmt+0x4c/0x50
 [<ffffffffa0248000>] ? reiserfs_xattr_register_handlers+0x8f9f/0xf9f [reiserfs]
 [<ffffffffa0249000>] ? reiserfs_xattr_register_handlers+0x9f9f/0x29f9f [reiserfs]
 [<ffffffffa0249000>] ? reiserfs_xattr_register_handlers+0x9f9f/0x29f9f [reiserfs]
 [<ffffffff8103b421>] __cpa_process_fault+0x91/0xa0
 [<ffffffff8103b852>] __change_page_attr_set_clr+0x392/0xab0
 [<ffffffffa023f000>] ? 0xffffffffa023efff
 [<ffffffff8103c093>] change_page_attr_set_clr+0x123/0x460
 [<ffffffffa023f000>] ? 0xffffffffa023efff
 [<ffffffff8103c86f>] set_memory_ro+0x2f/0x40
 [<ffffffffa0249000>] ? reiserfs_xattr_register_handlers+0x9f9f/0x29f9f [reiserfs]
 [<ffffffff81713e0d>] set_section_ro_nx+0x3a/0x71
 [<ffffffff810d23ee>] load_module+0x1f9e/0x2680
 [<ffffffff810cd6f0>] ? store_uevent+0x40/0x40
 [<ffffffffa0240000>] ? reiserfs_xattr_register_handlers+0xf9f/0xf9f [reiserfs]
 [<ffffffffa0240000>] ? reiserfs_xattr_register_handlers+0xf9f/0xf9f [reiserfs]
 [<ffffffff810d2c66>] SyS_finit_module+0x86/0xb0
 [<ffffffff8172aa14>] tracesys+0xdd/0xe2
---[ end trace 956db59f53237fe5 ]---
Oops: 0003 [#1] SMP 
Modules linked in: reiserfs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc xfs snd_timer libcrc32c snd e1000e ptp usb_debug pps_core pcspkr soundcore
CPU: 1 PID: 30571 Comm: modprobe Tainted: G        W    3.11.0+ #57 
task: ffff8801238a0000 ti: ffff8801ab314000 task.ti: ffff8801ab314000
RIP: 0010:[<ffffffff810d1a6b>]  [<ffffffff810d1a6b>] load_module+0x161b/0x2680
RSP: 0018:ffff8801ab315dc0  EFLAGS: 00010202
RAX: ffffffffa009c000 RBX: ffff8801ab315ef8 RCX: ffffffffa00c2000
RDX: ffffffffa00c2000 RSI: 0000005500000000 RDI: ffffffffa00c3f98
RBP: ffff8801ab315ee8 R08: ffffffffa009fa68 R09: ffffffffa009c000
R10: ffffffffa00c3f98 R11: 0000000000000002 R12: ffffffffa02d2838
R13: 0000000000000001 R14: 0000000000000000 R15: ffffffffa02d2820
FS:  00007f6f48b51740(0000) GS:ffff880245800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffa00c2000 CR3: 00000002211e9000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
 00000000003fa26b ffff8801238a0000 ffff8801ab315e48 ffff8801238a0000
 ffffffffa009c000 ffffffffa02d2a58 ffffffffa02d2838 0000000000003a80
 ffffffffa009c000 ffffffffa00c2000 0000003a94a10969 ffffffffa00c3f98
Call Trace:
 [<ffffffffa00c2000>] ? xfs_setattr_nonsize+0x240/0x5d0 [xfs]
 [<ffffffffa00c3f98>] ? xfs_inumbers+0x248/0x420 [xfs]
 [<ffffffff810cdeba>] ? copy_module_from_fd.isra.48+0x12a/0x190
 [<ffffffff810d2c66>] SyS_finit_module+0x86/0xb0
 [<ffffffff8172aa14>] tracesys+0xdd/0xe2
Code: 48 83 7a 38 00 78 6a 48 8b 30 44 89 ea 4c 89 d7 48 8d 14 52 4c 89 4c 24 40 41 83 c5 01 48 8d 14 d1 48 89 4c 24 48 4c 89 54 24 58 <48> 89 32 48 8b 70 08 48 89 72 08 48 8b 70 10 48 89 72 10 4c 89 
RIP  [<ffffffff810d1a6b>] load_module+0x161b/0x2680
 RSP <ffff8801ab315dc0>
CR2: ffffffffa00c2000
---[ end trace 956db59f53237fe6 ]---
Oops: 0003 [#2] SMP 
Modules linked in: reiserfs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc xfs snd_timer libcrc32c snd e1000e ptp usb_debug pps_core pcspkr soundcore
CPU: 3 PID: 30573 Comm: modprobe Tainted: G      D W    3.11.0+ #57 
task: ffff8801238a2a60 ti: ffff8800939ec000 task.ti: ffff8800939ec000
RIP: 0010:[<ffffffff810d1a6b>]  [<ffffffff810d1a6b>] load_module+0x161b/0x2680
RSP: 0018:ffff8800939eddc0  EFLAGS: 00010202
RAX: ffffffffa01d9000 RBX: ffff8800939edef8 RCX: ffffffffa01e6035
RDX: ffffffffa01e6035 RSI: 0000005500000000 RDI: ffffffffa01e71ed
RBP: ffff8800939edee8 R08: ffffffffa01db250 R09: ffffffffa01d9000
R10: ffffffffa01e71ed R11: 0000000000000002 R12: ffffffffa0257138
R13: 0000000000000001 R14: 0000000000000000 R15: ffffffffa0257120
FS:  00007f8207d62740(0000) GS:ffff880245c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffa01e6035 CR3: 000000009f46b000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
 000000000016abca ffff8801238a2a60 ffff8800939ede48 ffff8801238a2a60
 ffffffffa01d9000 ffffffffa0257358 ffffffffa0257138 0000000000002268
 ffffffffa01d9000 ffffffffa01e6035 0000003a94a10969 ffffffffa01e71ed
Call Trace:
 [<ffffffffa0257358>] ? 0xffffffffa0257357
 [<ffffffffa0257138>] ? 0xffffffffa0257137
 [<ffffffffa01e6035>] ? snd_pcm_xrun_debug_write+0x5/0x70 [snd_pcm]
 [<ffffffffa01e71ed>] ? snd_pcm_control_ioctl+0xad/0x260 [snd_pcm]
 [<ffffffff810cdeba>] ? copy_module_from_fd.isra.48+0x12a/0x190
 [<ffffffff810d2c66>] SyS_finit_module+0x86/0xb0
 [<ffffffff8172aa14>] tracesys+0xdd/0xe2
Code: 48 83 7a 38 00 78 6a 48 8b 30 44 89 ea 4c 89 d7 48 8d 14 52 4c 89 4c 24 40 41 83 c5 01 48 8d 14 d1 48 89 4c 24 48 4c 89 54 24 58 <48> 89 32 48 8b 70 08 48 89 72 08 48 8b 70 10 48 89 72 10 4c 89 
RIP  [<ffffffff810d1a6b>] load_module+0x161b/0x2680
 RSP <ffff8800939eddc0>
CR2: ffffffffa01e6035
---[ end trace 956db59f53237fe7 ]---

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Steven Rostedt]

On Thu, 5 Sep 2013 21:19:24 -0400
Dave Jones <davej@redhat.com> wrote:

> For whatever dumb reason, when running 'make install' on a Fedora system,
> os-prober tries to figure out what filesystems are needed by loading filesystems,
> and seeing what sticks..  Today it blew up spectacularly when it got to
> loading reiserfs..  System wedged entirely afterwards.

Could it be that the reiserfs module was compiled differently than the
running kernel?

> 
> 	Dave
> 
> ------------[ cut here ]------------
> WARNING: CPU: 2 PID: 30566 at kernel/trace/ftrace.c:1694 ftrace_bug+0x25d/0x270()
> Modules linked in: reiserfs(+) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc xfs snd_timer libcrc32c snd e1000e ptp usb_debug pps_core pcspkr soundcore
> CPU: 2 PID: 30566 Comm: modprobe Not tainted 3.11.0+ #57 
>  ffffffff81a2809d ffff88008de19c30 ffffffff817171e9 0000000000000000
>  ffff88008de19c68 ffffffff81053dad 0000000000000010 ffffffffa02738b0
>  ffff8802419e3518 0000000000000000 ffff8801ab16e100 ffff88008de19c78
> Call Trace:
>  [<ffffffff817171e9>] dump_stack+0x54/0x74
>  [<ffffffff81053dad>] warn_slowpath_common+0x7d/0xa0
>  [<ffffffff81053e8a>] warn_slowpath_null+0x1a/0x20
>  [<ffffffff8111924d>] ftrace_bug+0x25d/0x270
>  [<ffffffff81119568>] ftrace_process_locs+0x308/0x630
>  [<ffffffff811198cc>] ftrace_module_notify_enter+0x3c/0x40
>  [<ffffffff817257c6>] notifier_call_chain+0x66/0x150
>  [<ffffffff81088d97>] __blocking_notifier_call_chain+0x67/0xc0
>  [<ffffffff81088e06>] blocking_notifier_call_chain+0x16/0x20
>  [<ffffffff810d23cd>] load_module+0x1f7d/0x2680
>  [<ffffffff810cd6f0>] ? store_uevent+0x40/0x40
>  [<ffffffffa0240000>] ? reiserfs_xattr_register_handlers+0xf9f/0xf9f [reiserfs]
>  [<ffffffffa0240000>] ? reiserfs_xattr_register_handlers+0xf9f/0xf9f [reiserfs]
>  [<ffffffff810d2c66>] SyS_finit_module+0x86/0xb0
>  [<ffffffff8172aa14>] tracesys+0xdd/0xe2
> ---[ end trace 956db59f53237fe4 ]---
> ftrace failed to modify [<ffffffffa02738b0>] reiserfs_init_bitmap_cache+0x0/0xffffffffffff5750 [reiserfs]
>  actual: 14:00:00:00:00

Hmm, where it expected to see a call to mcount, instead is sees the
instruction:

 0x14 00 00 00 00


Can you do an objdump of that same binary, and show me what's located
at: reiserfs_init_bitmap_cache+0x0

-- Steve

> ------------[ cut here ]------------
> WARNING: CPU: 2 PID: 30566 at arch/x86/mm/pageattr.c:677 __cpa_process_fault+0x91/0xa0()
> CPA: called for zero pte. vaddr = ffffffffa0249000 cpa->vaddr = ffffffffa0249000
> Modules linked in: reiserfs(+) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc xfs snd_timer libcrc32c snd e1000e ptp usb_debug pps_core pcspkr soundcore
> CPU: 2 PID: 30566 Comm: modprobe Tainted: G        W    3.11.0+ #57 
>  ffffffff81a0ba44 ffff88008de19b40 ffffffff817171e9 ffff88008de19b88
>  ffff88008de19b78 ffffffff81053dad ffff88008de19d08 00000000fffffff2
>  ffffffffa0249000 ffff880238646248 ffff88008de19d08 ffff88008de19bd8
> Call Trace:
>  [<ffffffff817171e9>] dump_stack+0x54/0x74
>  [<ffffffff81053dad>] warn_slowpath_common+0x7d/0xa0
>  [<ffffffffa0249000>] ? reiserfs_xattr_register_handlers+0x9f9f/0x29f9f [reiserfs]
>  [<ffffffff81053e1c>] warn_slowpath_fmt+0x4c/0x50
>  [<ffffffffa0248000>] ? reiserfs_xattr_register_handlers+0x8f9f/0xf9f [reiserfs]
>  [<ffffffffa0249000>] ? reiserfs_xattr_register_handlers+0x9f9f/0x29f9f [reiserfs]
>  [<ffffffffa0249000>] ? reiserfs_xattr_register_handlers+0x9f9f/0x29f9f [reiserfs]
>  [<ffffffff8103b421>] __cpa_process_fault+0x91/0xa0
>  [<ffffffff8103b852>] __change_page_attr_set_clr+0x392/0xab0
>  [<ffffffffa023f000>] ? 0xffffffffa023efff
>  [<ffffffff8103c093>] change_page_attr_set_clr+0x123/0x460
>  [<ffffffffa023f000>] ? 0xffffffffa023efff
>  [<ffffffff8103c86f>] set_memory_ro+0x2f/0x40
>  [<ffffffffa0249000>] ? reiserfs_xattr_register_handlers+0x9f9f/0x29f9f [reiserfs]
>  [<ffffffff81713e0d>] set_section_ro_nx+0x3a/0x71
>  [<ffffffff810d23ee>] load_module+0x1f9e/0x2680
>  [<ffffffff810cd6f0>] ? store_uevent+0x40/0x40
>  [<ffffffffa0240000>] ? reiserfs_xattr_register_handlers+0xf9f/0xf9f [reiserfs]
>  [<ffffffffa0240000>] ? reiserfs_xattr_register_handlers+0xf9f/0xf9f [reiserfs]
>  [<ffffffff810d2c66>] SyS_finit_module+0x86/0xb0
>  [<ffffffff8172aa14>] tracesys+0xdd/0xe2
> ---[ end trace 956db59f53237fe5 ]---
> Oops: 0003 [#1] SMP 
> Modules linked in: reiserfs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc xfs snd_timer libcrc32c snd e1000e ptp usb_debug pps_core pcspkr soundcore
> CPU: 1 PID: 30571 Comm: modprobe Tainted: G        W    3.11.0+ #57 
> task: ffff8801238a0000 ti: ffff8801ab314000 task.ti: ffff8801ab314000
> RIP: 0010:[<ffffffff810d1a6b>]  [<ffffffff810d1a6b>] load_module+0x161b/0x2680
> RSP: 0018:ffff8801ab315dc0  EFLAGS: 00010202
> RAX: ffffffffa009c000 RBX: ffff8801ab315ef8 RCX: ffffffffa00c2000
> RDX: ffffffffa00c2000 RSI: 0000005500000000 RDI: ffffffffa00c3f98
> RBP: ffff8801ab315ee8 R08: ffffffffa009fa68 R09: ffffffffa009c000
> R10: ffffffffa00c3f98 R11: 0000000000000002 R12: ffffffffa02d2838
> R13: 0000000000000001 R14: 0000000000000000 R15: ffffffffa02d2820
> FS:  00007f6f48b51740(0000) GS:ffff880245800000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffffffffa00c2000 CR3: 00000002211e9000 CR4: 00000000001407e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Stack:
>  00000000003fa26b ffff8801238a0000 ffff8801ab315e48 ffff8801238a0000
>  ffffffffa009c000 ffffffffa02d2a58 ffffffffa02d2838 0000000000003a80
>  ffffffffa009c000 ffffffffa00c2000 0000003a94a10969 ffffffffa00c3f98
> Call Trace:
>  [<ffffffffa00c2000>] ? xfs_setattr_nonsize+0x240/0x5d0 [xfs]
>  [<ffffffffa00c3f98>] ? xfs_inumbers+0x248/0x420 [xfs]
>  [<ffffffff810cdeba>] ? copy_module_from_fd.isra.48+0x12a/0x190
>  [<ffffffff810d2c66>] SyS_finit_module+0x86/0xb0
>  [<ffffffff8172aa14>] tracesys+0xdd/0xe2
> Code: 48 83 7a 38 00 78 6a 48 8b 30 44 89 ea 4c 89 d7 48 8d 14 52 4c 89 4c 24 40 41 83 c5 01 48 8d 14 d1 48 89 4c 24 48 4c 89 54 24 58 <48> 89 32 48 8b 70 08 48 89 72 08 48 8b 70 10 48 89 72 10 4c 89 
> RIP  [<ffffffff810d1a6b>] load_module+0x161b/0x2680
>  RSP <ffff8801ab315dc0>
> CR2: ffffffffa00c2000
> ---[ end trace 956db59f53237fe6 ]---
> Oops: 0003 [#2] SMP 
> Modules linked in: reiserfs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc xfs snd_timer libcrc32c snd e1000e ptp usb_debug pps_core pcspkr soundcore
> CPU: 3 PID: 30573 Comm: modprobe Tainted: G      D W    3.11.0+ #57 
> task: ffff8801238a2a60 ti: ffff8800939ec000 task.ti: ffff8800939ec000
> RIP: 0010:[<ffffffff810d1a6b>]  [<ffffffff810d1a6b>] load_module+0x161b/0x2680
> RSP: 0018:ffff8800939eddc0  EFLAGS: 00010202
> RAX: ffffffffa01d9000 RBX: ffff8800939edef8 RCX: ffffffffa01e6035
> RDX: ffffffffa01e6035 RSI: 0000005500000000 RDI: ffffffffa01e71ed
> RBP: ffff8800939edee8 R08: ffffffffa01db250 R09: ffffffffa01d9000
> R10: ffffffffa01e71ed R11: 0000000000000002 R12: ffffffffa0257138
> R13: 0000000000000001 R14: 0000000000000000 R15: ffffffffa0257120
> FS:  00007f8207d62740(0000) GS:ffff880245c00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffffffffa01e6035 CR3: 000000009f46b000 CR4: 00000000001407e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Stack:
>  000000000016abca ffff8801238a2a60 ffff8800939ede48 ffff8801238a2a60
>  ffffffffa01d9000 ffffffffa0257358 ffffffffa0257138 0000000000002268
>  ffffffffa01d9000 ffffffffa01e6035 0000003a94a10969 ffffffffa01e71ed
> Call Trace:
>  [<ffffffffa0257358>] ? 0xffffffffa0257357
>  [<ffffffffa0257138>] ? 0xffffffffa0257137
>  [<ffffffffa01e6035>] ? snd_pcm_xrun_debug_write+0x5/0x70 [snd_pcm]
>  [<ffffffffa01e71ed>] ? snd_pcm_control_ioctl+0xad/0x260 [snd_pcm]
>  [<ffffffff810cdeba>] ? copy_module_from_fd.isra.48+0x12a/0x190
>  [<ffffffff810d2c66>] SyS_finit_module+0x86/0xb0
>  [<ffffffff8172aa14>] tracesys+0xdd/0xe2
> Code: 48 83 7a 38 00 78 6a 48 8b 30 44 89 ea 4c 89 d7 48 8d 14 52 4c 89 4c 24 40 41 83 c5 01 48 8d 14 d1 48 89 4c 24 48 4c 89 54 24 58 <48> 89 32 48 8b 70 08 48 89 72 08 48 8b 70 10 48 89 72 10 4c 89 
> RIP  [<ffffffff810d1a6b>] load_module+0x161b/0x2680
>  RSP <ffff8800939eddc0>
> CR2: ffffffffa01e6035
> ---[ end trace 956db59f53237fe7 ]---

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Dave Jones]

On Thu, Sep 05, 2013 at 09:28:34PM -0400, Steven Rostedt wrote:
 > On Thu, 5 Sep 2013 21:19:24 -0400
 > Dave Jones <davej@redhat.com> wrote:
 > 
 > > For whatever dumb reason, when running 'make install' on a Fedora system,
 > > os-prober tries to figure out what filesystems are needed by loading filesystems,
 > > and seeing what sticks..  Today it blew up spectacularly when it got to
 > > loading reiserfs..  System wedged entirely afterwards.
 > 
 > Could it be that the reiserfs module was compiled differently than the
 > running kernel?
 
ohhhh... it was probably installing the just-built version over the same '3.11+'
modules tree that was running.  This has never been a problem before though..

	Dave
 

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Steven Rostedt]

On Thu, 5 Sep 2013 21:34:55 -0400
Dave Jones <davej@redhat.com> wrote:

> On Thu, Sep 05, 2013 at 09:28:34PM -0400, Steven Rostedt wrote:
>  > On Thu, 5 Sep 2013 21:19:24 -0400
>  > Dave Jones <davej@redhat.com> wrote:
>  > 
>  > > For whatever dumb reason, when running 'make install' on a Fedora system,
>  > > os-prober tries to figure out what filesystems are needed by loading filesystems,
>  > > and seeing what sticks..  Today it blew up spectacularly when it got to
>  > > loading reiserfs..  System wedged entirely afterwards.
>  > 
>  > Could it be that the reiserfs module was compiled differently than the
>  > running kernel?
>  
> ohhhh... it was probably installing the just-built version over the same '3.11+'
> modules tree that was running.  This has never been a problem before though..
> 

Did you change a config option, or update your gcc?

Although, it doesn't really explain why the location would have
something that it doesn't expect. As the mcount/fentry table is created
in the module itself.

 -- Steve

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Dave Jones]

On Thu, Sep 05, 2013 at 09:44:55PM -0400, Steven Rostedt wrote:
 > On Thu, 5 Sep 2013 21:34:55 -0400
 > Dave Jones <davej@redhat.com> wrote:
 > 
 > > On Thu, Sep 05, 2013 at 09:28:34PM -0400, Steven Rostedt wrote:
 > >  > On Thu, 5 Sep 2013 21:19:24 -0400
 > >  > Dave Jones <davej@redhat.com> wrote:
 > >  > 
 > >  > > For whatever dumb reason, when running 'make install' on a Fedora system,
 > >  > > os-prober tries to figure out what filesystems are needed by loading filesystems,
 > >  > > and seeing what sticks..  Today it blew up spectacularly when it got to
 > >  > > loading reiserfs..  System wedged entirely afterwards.
 > >  > 
 > >  > Could it be that the reiserfs module was compiled differently than the
 > >  > running kernel?
 > >  
 > > ohhhh... it was probably installing the just-built version over the same '3.11+'
 > > modules tree that was running.  This has never been a problem before though..
 > > 
 > 
 > Did you change a config option, or update your gcc?

Yeah, changed CONFIG_DEBUG_KOBJECT, which rebuilt the world.

	Dave

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Steven Rostedt]

On Thu, 5 Sep 2013 21:48:59 -0400
Dave Jones <davej@redhat.com> wrote:

> On Thu, Sep 05, 2013 at 09:44:55PM -0400, Steven Rostedt wrote:
>  > On Thu, 5 Sep 2013 21:34:55 -0400
>  > Dave Jones <davej@redhat.com> wrote:
>  > 
>  > > On Thu, Sep 05, 2013 at 09:28:34PM -0400, Steven Rostedt wrote:

>  > Did you change a config option, or update your gcc?
> 
> Yeah, changed CONFIG_DEBUG_KOBJECT, which rebuilt the world.

Still doesn't explain why it gave you that splat there.

Do you still have that binary module, and can you show me what's at
reiserfs_init_bitmap_cache+0x0 with objdump?

-- Steve

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Dave Jones]

On Thu, Sep 05, 2013 at 09:51:54PM -0400, Steven Rostedt wrote:
 > On Thu, 5 Sep 2013 21:48:59 -0400
 > Dave Jones <davej@redhat.com> wrote:
 > 
 > > On Thu, Sep 05, 2013 at 09:44:55PM -0400, Steven Rostedt wrote:
 > >  > On Thu, 5 Sep 2013 21:34:55 -0400
 > >  > Dave Jones <davej@redhat.com> wrote:
 > >  > 
 > >  > > On Thu, Sep 05, 2013 at 09:28:34PM -0400, Steven Rostedt wrote:
 > 
 > >  > Did you change a config option, or update your gcc?
 > > 
 > > Yeah, changed CONFIG_DEBUG_KOBJECT, which rebuilt the world.
 > 
 > Still doesn't explain why it gave you that splat there.
 > 
 > Do you still have that binary module, and can you show me what's at
 > reiserfs_init_bitmap_cache+0x0 with objdump?

I didn't, but it turns out I can recreate this. A little convoluted but..

disable DEBUG_KOBJECT_RELEASE
build, install and boot into kernel

enable DEBUG_KOBJECT_RELEASE
build kernel
install -> boom


00000000000028b0 <reiserfs_init_bitmap_cache>:

        return bh;
}

int reiserfs_init_bitmap_cache(struct super_block *sb)
{
    28b0:       e8 00 00 00 00          callq  28b5 <reiserfs_init_bitmap_cache+0x5>
    28b5:       55                      push   %rbp

/* Don't trust REISERFS_SB(sb)->s_bmap_nr, it's a u16
 * which overflows on large file systems. */
static inline __u32 reiserfs_bmap_count(struct super_block *sb)
{
        return (SB_BLOCK_COUNT(sb) - 1) / (sb->s_blocksize * 8) + 1;
    28b6:       31 d2                   xor    %edx,%edx
    28b8:       48 89 e5                mov    %rsp,%rbp
    28bb:       41 54                   push   %r12
    28bd:       53                      push   %rbx
    28be:       48 89 fb                mov    %rdi,%rbx
    28c1:       48 8b 87 50 07 00 00    mov    0x750(%rdi),%rax
    28c8:       48 8b 77 18             mov    0x18(%rdi),%rsi
    28cc:       48 8b 40 08             mov    0x8(%rax),%rax
    28d0:       48 8d 0c f5 00 00 00    lea    0x0(,%rsi,8),%rcx
    28d7:       00 
    28d8:       8b 00                   mov    (%rax),%eax
    28da:       83 e8 01                sub    $0x1,%eax
    28dd:       48 f7 f1                div    %rcx

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Steven Rostedt]

On Thu, 5 Sep 2013 23:58:27 -0400
Dave Jones <davej@redhat.com> wrote:

> On Thu, Sep 05, 2013 at 09:51:54PM -0400, Steven Rostedt wrote:
>  > On Thu, 5 Sep 2013 21:48:59 -0400
>  > Dave Jones <davej@redhat.com> wrote:
>  > 
>  > > On Thu, Sep 05, 2013 at 09:44:55PM -0400, Steven Rostedt wrote:
>  > >  > On Thu, 5 Sep 2013 21:34:55 -0400
>  > >  > Dave Jones <davej@redhat.com> wrote:
>  > >  > 
>  > >  > > On Thu, Sep 05, 2013 at 09:28:34PM -0400, Steven Rostedt wrote:
>  > 
>  > >  > Did you change a config option, or update your gcc?
>  > > 
>  > > Yeah, changed CONFIG_DEBUG_KOBJECT, which rebuilt the world.
>  > 
>  > Still doesn't explain why it gave you that splat there.
>  > 
>  > Do you still have that binary module, and can you show me what's at
>  > reiserfs_init_bitmap_cache+0x0 with objdump?
> 
> I didn't, but it turns out I can recreate this. A little convoluted but..
> 
> disable DEBUG_KOBJECT_RELEASE
> build, install and boot into kernel
> 
> enable DEBUG_KOBJECT_RELEASE
> build kernel
> install -> boom

Did it report failing on the same function as before?

> 
> 
> 00000000000028b0 <reiserfs_init_bitmap_cache>:
> 
>         return bh;
> }
> 
> int reiserfs_init_bitmap_cache(struct super_block *sb)
> {
>     28b0:       e8 00 00 00 00          callq  28b5 <reiserfs_init_bitmap_cache+0x5>

That looks to be the call to fentry, but without being resolved. What
we saw in the previous report wasn't 0xe8 but 0x14, and it was
unresolved after loading!

I'm surprised that the module built with a different
DEBUG_KOBJECT_RELEASE config would load. Does it not affect module
versions?

-- Steve

>     28b5:       55                      push   %rbp
> 
> /* Don't trust REISERFS_SB(sb)->s_bmap_nr, it's a u16
>  * which overflows on large file systems. */
> static inline __u32 reiserfs_bmap_count(struct super_block *sb)
> {
>         return (SB_BLOCK_COUNT(sb) - 1) / (sb->s_blocksize * 8) + 1;
>     28b6:       31 d2                   xor    %edx,%edx
>     28b8:       48 89 e5                mov    %rsp,%rbp
>     28bb:       41 54                   push   %r12
>     28bd:       53                      push   %rbx
>     28be:       48 89 fb                mov    %rdi,%rbx
>     28c1:       48 8b 87 50 07 00 00    mov    0x750(%rdi),%rax
>     28c8:       48 8b 77 18             mov    0x18(%rdi),%rsi
>     28cc:       48 8b 40 08             mov    0x8(%rax),%rax
>     28d0:       48 8d 0c f5 00 00 00    lea    0x0(,%rsi,8),%rcx
>     28d7:       00 
>     28d8:       8b 00                   mov    (%rax),%eax
>     28da:       83 e8 01                sub    $0x1,%eax
>     28dd:       48 f7 f1                div    %rcx

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Dave Jones]

On Fri, Sep 06, 2013 at 09:32:52AM -0400, Steven Rostedt wrote:
 > On Thu, 5 Sep 2013 23:58:27 -0400
 > Dave Jones <davej@redhat.com> wrote:
 > 
 > > On Thu, Sep 05, 2013 at 09:51:54PM -0400, Steven Rostedt wrote:
 > >  > On Thu, 5 Sep 2013 21:48:59 -0400
 > >  > Dave Jones <davej@redhat.com> wrote:
 > >  > 
 > >  > > On Thu, Sep 05, 2013 at 09:44:55PM -0400, Steven Rostedt wrote:
 > >  > >  > On Thu, 5 Sep 2013 21:34:55 -0400
 > >  > >  > Dave Jones <davej@redhat.com> wrote:
 > >  > >  > 
 > >  > >  > > On Thu, Sep 05, 2013 at 09:28:34PM -0400, Steven Rostedt wrote:
 > >  > 
 > >  > >  > Did you change a config option, or update your gcc?
 > >  > > 
 > >  > > Yeah, changed CONFIG_DEBUG_KOBJECT, which rebuilt the world.
 > >  > 
 > >  > Still doesn't explain why it gave you that splat there.
 > >  > 
 > >  > Do you still have that binary module, and can you show me what's at
 > >  > reiserfs_init_bitmap_cache+0x0 with objdump?
 > > 
 > > I didn't, but it turns out I can recreate this. A little convoluted but..
 > > 
 > > disable DEBUG_KOBJECT_RELEASE
 > > build, install and boot into kernel
 > > 
 > > enable DEBUG_KOBJECT_RELEASE
 > > build kernel
 > > install -> boom
 > 
 > Did it report failing on the same function as before?
 
Yes.

 > > 00000000000028b0 <reiserfs_init_bitmap_cache>:
 > > 
 > >         return bh;
 > > }
 > > 
 > > int reiserfs_init_bitmap_cache(struct super_block *sb)
 > > {
 > >     28b0:       e8 00 00 00 00          callq  28b5 <reiserfs_init_bitmap_cache+0x5>
 > 
 > That looks to be the call to fentry, but without being resolved. What
 > we saw in the previous report wasn't 0xe8 but 0x14, and it was
 > unresolved after loading!
 > 
 > I'm surprised that the module built with a different
 > DEBUG_KOBJECT_RELEASE config would load. Does it not affect module
 > versions?
 
# CONFIG_MODVERSIONS is not set


	Dave

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Steven Rostedt]

On Fri, 6 Sep 2013 10:18:52 -0400
Dave Jones <davej@redhat.com> wrote:


>  > > 00000000000028b0 <reiserfs_init_bitmap_cache>:
>  > > 
>  > >         return bh;
>  > > }
>  > > 
>  > > int reiserfs_init_bitmap_cache(struct super_block *sb)
>  > > {
>  > >     28b0:       e8 00 00 00 00          callq  28b5 <reiserfs_init_bitmap_cache+0x5>
>  > 
>  > That looks to be the call to fentry, but without being resolved. What
>  > we saw in the previous report wasn't 0xe8 but 0x14, and it was
>  > unresolved after loading!
>  > 
>  > I'm surprised that the module built with a different
>  > DEBUG_KOBJECT_RELEASE config would load. Does it not affect module
>  > versions?
>  
> # CONFIG_MODVERSIONS is not set
> 

Is it really safe to load modules that were compiled with different
options? Sounds like a failure in the install scripts to me.

Anyway, if you can send me the binary file, I can take a deeper look at
it.

-- Steve

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Kyle McMartin]

On Fri, Sep 06, 2013 at 10:24:13AM -0400, Steven Rostedt wrote:
> 
> Is it really safe to load modules that were compiled with different
> options?

Certainly not, offsets can change based on config options.

> Sounds like a failure in the install scripts to me.
> 

Definitely a bug in those scripts, os-prober would need to run before
modules_install, since if $INSTALL_MOD_PATH is the same between config
changes, there's no guarantee the modules are loadable on the running
kernel without a reboot.

--Kyle

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Dave Jones]

On Fri, Sep 06, 2013 at 10:41:13AM -0400, Kyle McMartin wrote:
 > On Fri, Sep 06, 2013 at 10:24:13AM -0400, Steven Rostedt wrote:
 > > 
 > > Is it really safe to load modules that were compiled with different
 > > options?
 > 
 > Certainly not, offsets can change based on config options.
 > 
 > > Sounds like a failure in the install scripts to me.
 > > 
 > 
 > Definitely a bug in those scripts, os-prober would need to run before
 > modules_install, since if $INSTALL_MOD_PATH is the same between config
 > changes, there's no guarantee the modules are loadable on the running
 > kernel without a reboot.

Ugh, it's actually because my local install kernel script calls
grub2-mkconfig after installing a kernel to clean out old ones and not
clutter up the boot menu. (Which in turn scans every disk using os-prober[*])

Don't waste time on this Steve, I'll just work around it somehow.

	Dave

[*] Worst thing.

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Steven Rostedt]

On Fri, 6 Sep 2013 11:22:21 -0400
Dave Jones <davej@redhat.com> wrote:

> Ugh, it's actually because my local install kernel script calls
> grub2-mkconfig after installing a kernel to clean out old ones and not
> clutter up the boot menu. (Which in turn scans every disk using os-prober[*])

Fedora should really look into defaulting to syslinux.


> 
> Don't waste time on this Steve, I'll just work around it somehow.

I wont.

Thanks,

-- Steve

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Josh Boyer]

On Fri, Sep 6, 2013 at 11:34 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, 6 Sep 2013 11:22:21 -0400
> Dave Jones <davej@redhat.com> wrote:
>
>> Ugh, it's actually because my local install kernel script calls
>> grub2-mkconfig after installing a kernel to clean out old ones and not
>> clutter up the boot menu. (Which in turn scans every disk using os-prober[*])
>
> Fedora should really look into defaulting to syslinux.

Doesn't support all the usecases we have, and it's not multi-arch.
Grub2 has warts, but it's all around more flexible.

josh

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Steven Rostedt]

On Fri, 6 Sep 2013 11:46:13 -0400
Josh Boyer <jwboyer@fedoraproject.org> wrote:


> > Fedora should really look into defaulting to syslinux.
> 
> Doesn't support all the usecases we have, and it's not multi-arch.
> Grub2 has warts, but it's all around more flexible.

We should be working on making syslinux work for all cases then.

Grub2 has more than warts, it's a total disaster. I have not found a
single user that likes grub2 (outside of the grub2 developers
themselves).

The "oneshot" booting is totally broken, I could never get it to work
consistently, thus I had to abandon it for ktest. Grub2 may be
flexible, but it is overly complicated. It's horrible to do anything
but the "default" settings.

-- Steve

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Josh Boyer]

On Fri, Sep 6, 2013 at 12:07 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, 6 Sep 2013 11:46:13 -0400
> Josh Boyer <jwboyer@fedoraproject.org> wrote:
>
>
>> > Fedora should really look into defaulting to syslinux.
>>
>> Doesn't support all the usecases we have, and it's not multi-arch.
>> Grub2 has warts, but it's all around more flexible.
>
> We should be working on making syslinux work for all cases then.

Is there something stopping you? :)  More specifically, I don't think
syslinux will ever support e.g. powerpc or ARM (Aarch64) so it doesn't
seem like a general purpose solution.  I could be wrong.

> Grub2 has more than warts, it's a total disaster. I have not found a
> single user that likes grub2 (outside of the grub2 developers
> themselves).
>
> The "oneshot" booting is totally broken, I could never get it to work
> consistently, thus I had to abandon it for ktest. Grub2 may be
> flexible, but it is overly complicated. It's horrible to do anything
> but the "default" settings.

We have essentially one person working on bootloaders and that person
has to support them in RHEL too.  I'm sure they'd appreciate help.

josh

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[richard -rw- weinberger]

On Fri, Sep 6, 2013 at 6:12 PM, Josh Boyer <jwboyer@fedoraproject.org> wrote:
> On Fri, Sep 6, 2013 at 12:07 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>> On Fri, 6 Sep 2013 11:46:13 -0400
>> Josh Boyer <jwboyer@fedoraproject.org> wrote:
>>
>>
>>> > Fedora should really look into defaulting to syslinux.
>>>
>>> Doesn't support all the usecases we have, and it's not multi-arch.
>>> Grub2 has warts, but it's all around more flexible.
>>
>> We should be working on making syslinux work for all cases then.
>
> Is there something stopping you? :)  More specifically, I don't think
> syslinux will ever support e.g. powerpc or ARM (Aarch64) so it doesn't
> seem like a general purpose solution.  I could be wrong.

Is there a list of use cases which are missing in syslinux?

On all systems where I have to touch the bootloader
(running custom kernels, using one-shot, ...) I always install
gummiboot or syslinux because
grub2 sucks so much. grub1 worked quite well but grub2 is a PITA.

-- 
Thanks,
//richard

P.s: Sorry for flaming but gub2 disappointed me more than once.

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Josh Boyer]

On Fri, Sep 6, 2013 at 12:40 PM, richard -rw- weinberger
<richard.weinberger@gmail.com> wrote:
> On Fri, Sep 6, 2013 at 6:12 PM, Josh Boyer <jwboyer@fedoraproject.org> wrote:
>> On Fri, Sep 6, 2013 at 12:07 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>>> On Fri, 6 Sep 2013 11:46:13 -0400
>>> Josh Boyer <jwboyer@fedoraproject.org> wrote:
>>>
>>>
>>>> > Fedora should really look into defaulting to syslinux.
>>>>
>>>> Doesn't support all the usecases we have, and it's not multi-arch.
>>>> Grub2 has warts, but it's all around more flexible.
>>>
>>> We should be working on making syslinux work for all cases then.
>>
>> Is there something stopping you? :)  More specifically, I don't think
>> syslinux will ever support e.g. powerpc or ARM (Aarch64) so it doesn't
>> seem like a general purpose solution.  I could be wrong.
>
> Is there a list of use cases which are missing in syslinux?

The aforementioned lack of non-x86 architecture support is probably
the biggest one from a Fedora perspective.  However, I don't really
think that's something that will be _added_ to syslinux, so it kind of
makes discussing the rest irrelevant for us.

My original reply was as to why Fedora doesn't just switch.  If there
are bootloaders that people want to use instead of grub2, then go for
it.  I have no ties to any of them.

josh

Re: ftrace 'failed to modify' bug when loading reiserfs.ko

[Steven Rostedt]

On Fri, 6 Sep 2013 12:12:48 -0400
Josh Boyer <jwboyer@fedoraproject.org> wrote:

> On Fri, Sep 6, 2013 at 12:07 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> > On Fri, 6 Sep 2013 11:46:13 -0400
> > Josh Boyer <jwboyer@fedoraproject.org> wrote:
> >
> >
> >> > Fedora should really look into defaulting to syslinux.
> >>
> >> Doesn't support all the usecases we have, and it's not multi-arch.
> >> Grub2 has warts, but it's all around more flexible.
> >
> > We should be working on making syslinux work for all cases then.
> 
> Is there something stopping you? :)

Yes, it's called a day job that takes up most my time ;-)

I'd love to work more on it. I just don't have the time to do so.

> 
> > Grub2 has more than warts, it's a total disaster. I have not found a
> > single user that likes grub2 (outside of the grub2 developers
> > themselves).
> >
> > The "oneshot" booting is totally broken, I could never get it to work
> > consistently, thus I had to abandon it for ktest. Grub2 may be
> > flexible, but it is overly complicated. It's horrible to do anything
> > but the "default" settings.
> 
> We have essentially one person working on bootloaders and that person
> has to support them in RHEL too.  I'm sure they'd appreciate help.

I'd love to. But there's a bunch of other things I need to do first
before I ever get a chance to work on that.

I wonder how much time John Hawley has?

-- Steve