Re: ppp/pppoe, still panic 4.15.3 in ppp_push

[Denys Fedoryshchenko <nuclearcat@nuclearcat.com>]

On 2018-02-14 19:25, Guillaume Nault wrote:
> On Wed, Feb 14, 2018 at 06:49:19PM +0200, Denys Fedoryshchenko wrote:
>> On 2018-02-14 18:47, Guillaume Nault wrote:
>> > On Wed, Feb 14, 2018 at 06:29:34PM +0200, Denys Fedoryshchenko wrote:
>> > > On 2018-02-14 18:07, Guillaume Nault wrote:
>> > > > On Wed, Feb 14, 2018 at 03:17:23PM +0200, Denys Fedoryshchenko wrote:
>> > > > > Hi,
>> > > > >
>> > > > > Upgraded kernel to 4.15.3, still it crashes after while (several
>> > > > > hours,
>> > > > > cannot do bisect, as it is production server).
>> > > > >
>> > > > > dev ppp # gdb ppp_generic.o
>> > > > > GNU gdb (Gentoo 7.12.1 vanilla) 7.12.1
>> > > > > <<skipped>>
>> > > > > Reading symbols from ppp_generic.o...done.
>> > > > > (gdb) list *ppp_push+0x73
>> > > > > 0x681 is in ppp_push (drivers/net/ppp/ppp_generic.c:1663).
>> > > > > 1658			list = list->next;
>> > > > > 1659			pch = list_entry(list, struct channel, clist);
>> > > > > 1660
>> > > > > 1661			spin_lock(&pch->downl);
>> > > > > 1662			if (pch->chan) {
>> > > > > 1663				if (pch->chan->ops->start_xmit(pch->chan, skb))
>> > > > > 1664					ppp->xmit_pending = NULL;
>> > > > > 1665			} else {
>> > > > > 1666				/* channel got unregistered */
>> > > > > 1667				kfree_skb(skb);
>> > > > >
>> > > > >
>> > > > I expect a memory corruption. Do you have the possibility to run with
>> > > > KASAN by any chance?
>> > > I will try to enable it tonight. For now i reverted "drivers, net,
>> > > ppp:
>> > > convert ppp_file.refcnt from atomic_t to refcount_t" for test.
>> > >
>> > This commit looks good to me. Do you have doubts about it because it's
>> > new in 4.15? Does it mean that your last known-good kernel is 4.14?
>> 
>> I am just doing "manual" bisect, checking all possibilities, and 
>> picking
>> patch to revert randomly.
>> 
> Must be a painful process. Are all of your networking modules required?
> With luck, you might be able to isolate a faulty module in fewer steps.
> 
>> Yes, correct, my known-good is 4.14.2.
>> 
> Good to know.
> 
> Let me know if you can get a KASAN trace.
Here we go:

  <srv> [24558.921549] 
==================================================================
  <srv> [24558.922167] BUG: KASAN: use-after-free in 
ppp_ioctl+0xa6a/0x1522 [ppp_generic]
  <srv> [24558.922776] Write of size 8 at addr ffff8803d35bf3f8 by task 
accel-pppd/12622
  <srv> [24558.923113]
  <srv> [24558.923451] CPU: 0 PID: 12622 Comm: accel-pppd Tainted: G      
   W        4.15.3-build-0134 #1
  <srv> [24558.924058] Hardware name: HP ProLiant DL320e Gen8 v2, BIOS 
P80 04/02/2015
  <srv> [24558.924406] Call Trace:
  <srv> [24558.924753]  dump_stack+0x46/0x59
  <srv> [24558.925103]  print_address_description+0x6b/0x23b
  <srv> [24558.925451]  ? ppp_ioctl+0xa6a/0x1522 [ppp_generic]
  <srv> [24558.925797]  kasan_report+0x21b/0x241
  <srv> [24558.926136]  ppp_ioctl+0xa6a/0x1522 [ppp_generic]
  <srv> [24558.926479]  ? ppp_nl_newlink+0x1da/0x1da [ppp_generic]
  <srv> [24558.926829]  ? sock_sendmsg+0x89/0x99
  <srv> [24558.927176]  ? __vfs_write+0xd9/0x4ad
  <srv> [24558.927523]  ? kernel_read+0xed/0xed
  <srv> [24558.927872]  ? SyS_getpeername+0x18c/0x18c
  <srv> [24558.928213]  ? bit_waitqueue+0x2a/0x2a
  <srv> [24558.928561]  ? wake_atomic_t_function+0x115/0x115
  <srv> [24558.928898]  vfs_ioctl+0x6e/0x81
  <srv> [24558.929228]  do_vfs_ioctl+0xa00/0xb10
  <srv> [24558.929571]  ? sigprocmask+0x1a6/0x1d0
  <srv> [24558.929907]  ? sigsuspend+0x13e/0x13e
  <srv> [24558.930239]  ? ioctl_preallocate+0x14e/0x14e
  <srv> [24558.930568]  ? SyS_rt_sigprocmask+0xf1/0x142
  <srv> [24558.930904]  ? sigprocmask+0x1d0/0x1d0
  <srv> [24558.931252]  SyS_ioctl+0x39/0x55
  <srv> [24558.931595]  ? do_vfs_ioctl+0xb10/0xb10
  <srv> [24558.931942]  do_syscall_64+0x1b1/0x31f
  <srv> [24558.932288]  entry_SYSCALL_64_after_hwframe+0x21/0x86
  <srv> [24558.932627] RIP: 0033:0x7f302849d8a7
  <srv> [24558.932965] RSP: 002b:00007f3029a52af8 EFLAGS: 00000206 
ORIG_RAX: 0000000000000010
  <srv> [24558.933578] RAX: ffffffffffffffda RBX: 00007f3027d861e3 RCX: 
00007f302849d8a7
  <srv> [24558.933927] RDX: 00007f3023f49468 RSI: 000000004004743a RDI: 
0000000000003a67
  <srv> [24558.934266] RBP: 00007f3029a52b20 R08: 0000000000000000 R09: 
000055c8308d8e40
  <srv> [24558.934607] R10: 0000000000000008 R11: 0000000000000206 R12: 
00007f3023f49358
  <srv> [24558.934947] R13: 00007ffe86e5723f R14: 0000000000000000 R15: 
00007f3029a53700
  <srv> [24558.935288]
  <srv> [24558.935626] Allocated by task 12622:
  <srv> [24558.935972]  ppp_register_net_channel+0x5f/0x5c6 [ppp_generic]
  <srv> [24558.936306]  pppoe_connect+0xab7/0xc71 [pppoe]
  <srv> [24558.936640]  SyS_connect+0x14b/0x1b7
  <srv> [24558.936975]  do_syscall_64+0x1b1/0x31f
  <srv> [24558.937319]  entry_SYSCALL_64_after_hwframe+0x21/0x86
  <srv> [24558.937655]
  <srv> [24558.937993] Freed by task 12622:
  <srv> [24558.938321]  kfree+0xb0/0x11d
  <srv> [24558.938658]  ppp_release+0x111/0x120 [ppp_generic]
  <srv> [24558.938994]  __fput+0x2ba/0x51a
  <srv> [24558.939332]  task_work_run+0x11c/0x13d
  <srv> [24558.939676]  exit_to_usermode_loop+0x7c/0xaf
  <srv> [24558.940022]  do_syscall_64+0x2ea/0x31f
  <srv> [24558.940368]  entry_SYSCALL_64_after_hwframe+0x21/0x86
  <srv> [24558.947099]
  <srv> [24558.947443] The buggy address belongs to the object at 
ffff8803d35bf340
[24558.947443]  which belongs to the cache kmalloc-256 of size 256
  <srv> [24558.948064] The buggy address is located 184 bytes inside of
[24558.948064]  256-byte region [ffff8803d35bf340, ffff8803d35bf440)
  <srv> [24558.948676] The buggy address belongs to the page:
  <srv> [24558.949019] page:ffffea000f4d6f00 count:1 mapcount:0 mapping:  
         (null) index:0xffff8803d35bfc00 compound_mapcount: 0
  <srv> [24558.949633] flags: 0x17ffe00000008100(slab|head)
  <srv> [24558.949980] raw: 17ffe00000008100 0000000000000000 
ffff8803d35bfc00 000000010033002e
  <srv> [24558.950597] raw: ffffea000d98c020 ffffea000df57d20 
ffff8803f1c0f480 0000000000000000
  <srv> [24558.951209] page dumped because: kasan: bad access detected
  <srv> [24558.951546]
  <srv> [24558.951880] Memory state around the buggy address:
  <srv> [24558.952217]  ffff8803d35bf280: fb fb fb fb fb fb fb fb fb fb 
fb fb fb fb fb fb
  <srv> [24558.952818]  ffff8803d35bf300: fc fc fc fc fc fc fc fc fb fb 
fb fb fb fb fb fb
  <srv> [24558.953415] >ffff8803d35bf380: fb fb fb fb fb fb fb fb fb fb 
fb fb fb fb fb fb
  <srv> [24558.954021]                                                    
              ^
  <srv> [24558.954365]  ffff8803d35bf400: fb fb fb fb fb fb fb fb fc fc 
fc fc fc fc fc fc
  <srv> [24558.954969]  ffff8803d35bf480: 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00
  <srv> [24558.955572] 
==================================================================
  <srv> [24558.956169] Disabling lock debugging due to kernel taint