Re: [PATCH] x86/livepatch: Fix crash with !CONFIG_DEBUG_SET_MODULE_RONX

[Josh Poimboeuf <jpoimboe@redhat.com>]

On Tue, Nov 03, 2015 at 11:22:12AM +0100, Miroslav Benes wrote:
> On Mon, 2 Nov 2015, Josh Poimboeuf wrote:
> 
> > When loading a patch module on a kernel with
> > !CONFIG_DEBUG_SET_MODULE_RONX, the following crash occurs:
> > 
> >   [  205.988776] livepatch: enabling patch 'kpatch_meminfo_string'
> >   [  205.989829] BUG: unable to handle kernel paging request at ffffffffa08d2fc0
> >   [  205.989863] IP: [<ffffffff8154fecb>] do_init_module+0x8c/0x1ba
> >   [  205.989888] PGD 1a10067 PUD 1a11063 PMD 7bcde067 PTE 3740e161
> >   [  205.989915] Oops: 0003 [#1] SMP
> >   [  205.990187] CPU: 2 PID: 14570 Comm: insmod Tainted: G           O  K 4.1.12
> >   [  205.990214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.1-20150318_183358- 04/01/2014
> >   [  205.990249] task: ffff8800374aaa90 ti: ffff8800794b8000 task.ti: ffff8800794b8000
> >   [  205.990276] RIP: 0010:[<ffffffff8154fecb>]  [<ffffffff8154fecb>] do_init_module+0x8c/0x1ba
> >   [  205.990307] RSP: 0018:ffff8800794bbd58  EFLAGS: 00010246
> >   [  205.990327] RAX: 0000000000000000 RBX: ffffffffa08d2fc0 RCX: 0000000000000000
> >   [  205.990356] RDX: 01ffff8000000080 RSI: 0000000000000000 RDI: ffffffff81a54b40
> >   [  205.990382] RBP: ffff88007b4c4d80 R08: 0000000000000007 R09: 0000000000000000
> >   [  205.990408] R10: 0000000000000008 R11: ffffea0001f18840 R12: 0000000000000000
> >   [  205.990433] R13: 0000000000000001 R14: ffffffffa08d2fc0 R15: ffff88007bd0bc40
> >   [  205.990459] FS:  00007f1128fbc700(0000) GS:ffff88007fc80000(0000) knlGS:0000000000000000
> >   [  205.990488] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> >   [  205.990509] CR2: ffffffffa08d2fc0 CR3: 000000002606e000 CR4: 00000000001406e0
> >   [  205.990536] Stack:
> >   [  205.990545]  ffff8800794bbec8 0000000000000001 ffffffffa08d3010 ffffffff810ecea9
> >   [  205.990576]  ffffffff810e8e40 000000000005f360 ffff88007bd0bc50 ffffffffa08d3240
> >   [  205.990608]  ffffffffa08d52c0 ffffffffa08d3210 ffff8800794bbed8 ffff8800794bbf1c
> >   [  205.990639] Call Trace:
> >   [  205.990651]  [<ffffffff810ecea9>] ? load_module+0x1e59/0x23a0
> >   [  205.990672]  [<ffffffff810e8e40>] ? store_uevent+0x40/0x40
> >   [  205.990693]  [<ffffffff810e99b5>] ? copy_module_from_fd.isra.49+0xb5/0x140
> >   [  205.990718]  [<ffffffff810ed5bd>] ? SyS_finit_module+0x7d/0xa0
> >   [  205.990741]  [<ffffffff81556832>] ? system_call_fastpath+0x16/0x75
> >   [  205.990763] Code: f9 00 00 00 74 23 49 c7 c0 92 e1 60 81 48 8d 53 18 89 c1 4c 89 c6 48 c7 c7 f0 85 7d 81 31 c0 e8 71 fa ff ff e8 58 0e 00 00 31 f6 <c7> 03 00 00 00 00 48 89 da 48 c7 c7 20 c7 a5 81 e8 d0 ec b3 ff
> >   [  205.990916] RIP  [<ffffffff8154fecb>] do_init_module+0x8c/0x1ba
> >   [  205.990940]  RSP <ffff8800794bbd58>
> >   [  205.990953] CR2: ffffffffa08d2fc0
> > 
> > With !CONFIG_DEBUG_SET_MODULE_RONX, module text and rodata pages are
> > writable, and the debug_align() macro allows the module struct to share
> > a page with executable text.  When klp_write_module_reloc() calls
> > set_memory_ro() on the page, it effectively turns the module struct into
> > a read-only structure, resulting in a page fault when load_module() does
> > "mod->state = MODULE_STATE_LIVE".
> > 
> > Reported-by: Cyril B. <cbay@alwaysdata.com>
> > Tested-by: Cyril B. <cbay@alwaysdata.com>
> > Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
> > ---
> 
> Oh, this reminds me I had to solve this issue in kgraft while playing with 
> relocations. I wanted to fix it in klp as well and forgot about it so it 
> is great to see it done.
> 
> Anyway...
> 
> >  arch/x86/kernel/livepatch.c | 9 +++++----
> >  1 file changed, 5 insertions(+), 4 deletions(-)
> > 
> > diff --git a/arch/x86/kernel/livepatch.c b/arch/x86/kernel/livepatch.c
> > index ff3c3101d..d1d35cc 100644
> > --- a/arch/x86/kernel/livepatch.c
> > +++ b/arch/x86/kernel/livepatch.c
> > @@ -42,7 +42,6 @@ int klp_write_module_reloc(struct module *mod, unsigned long type,
> >  	bool readonly;
> >  	unsigned long val;
> >  	unsigned long core = (unsigned long)mod->module_core;
> > -	unsigned long core_ro_size = mod->core_ro_size;
> >  	unsigned long core_size = mod->core_size;
> >  
> >  	switch (type) {
> > @@ -70,10 +69,12 @@ int klp_write_module_reloc(struct module *mod, unsigned long type,
> >  		/* loc does not point to any symbol inside the module */
> >  		return -EINVAL;
> >  
> > -	if (loc < core + core_ro_size)
> > +	readonly = false;
> > +
> > +#ifdef CONFIG_DEBUG_SET_MODULE_RONX
> > +	if (loc < core + mod->core_ro_size)
> >  		readonly = true;
> > -	else
> > -		readonly = false;
> > +#endif
> >  
> 
> Wouldn't it be better to use static inline function defined somewhere in 
> our header file which would do the job for CONFIG_DEBUG_SET_MODULE_RONX 
> set and nothing for the opposite? I think it is better than to have ifdef 
> here. Probably just a matter of taste.

I agree that would be better.  Though I think I'd make a static function
in the .c file so it stays internal.

> Secondly, why do we call set_memory_rw/ro here for each relocation? 
> Wouldn't it be possible to do it once for all the relevant pages of the 
> patched module? I mean we could call set_memory_rw in 
> klp_write_object_relocations from kernel/livepatch/core.c just before the 
> for loop and set_memory_ro on the way out. With the static inline function 
> it could look like this
> 
> +#ifdef CONFIG_DEBUG_SET_MODULE_RONX
> +static inline void set_module_text(void *start, void *end, int (*set)(unsigned long start,
> +       int num_pages))
> +{
> +       unsigned long begin_pfn = PFN_DOWN((unsigned long)start);
> +       unsigned long end_pfn = PFN_DOWN((unsigned long)end);
> +
> +       if (end_pfn > begin_pfn)
> +               set(begin_pfn << PAGE_SHIFT, end_pfn - begin_pfn);
> +}
> +#else
> +static inline void set_module_text(void *start, void *end, int (*set)(unsigned long start,
> +       int num_pages)) { }
> +#endif
> 
> somewhere in include/linux/livepatch.h. And in 
> klp_write_object_relocations() there could be 
> 
> +       set_module_text(mod->module_core, mod->module_core +
> +               mod->core_text_size, set_memory_rw);
> 
> just before the for loop which goes through all the relocations
> 
> and 
> 
> +       set_module_text(mod->module_core, mod->module_core +
> +               mod->core_text_size, set_memory_ro);
> 
> on the way out. 
> 
> We would also get rid of readonly variable.
> 
> At least this is how I solved it. Take it as an idea. I don't know if it 
> is even better and I certainly do not insist on it. Your approach is ok.

Yeah, I like this idea.  Though we have to be careful.  Not only can
relocations occur in text, they can also occur in data.  So we'd have to
set read-only data writable as well.  So it should be "mod->module_core
+ mod->core_ro_size".

-- 
Josh