Re: BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build

public inbox for llvm@lists.linux.dev
 help / color / mirror / Atom feed

* Re: BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb)
       [not found]         ` <CAABZP2yb7-xa4F_2c6tuzkv7x902wU-hqgD_pqRooGC6C7S20A@mail.gmail.com>
@ 2022-02-16 13:19           ` Paul Menzel
  2022-02-17  1:16             ` Nathan Chancellor
  0 siblings, 1 reply; 6+ messages in thread
From: Paul Menzel @ 2022-02-16 13:19 UTC (permalink / raw)
  To: Zhouyi Zhou
  Cc: Paul E. McKenney, Josh Triplett, rcu, LKML, David S. Miller,
	Jakub Kicinski, netdev, Nathan Chancellor, Nick Desaulniers, llvm

[-- Attachment #1: Type: text/plain, Size: 20515 bytes --]

[Cc: +LLVM/clang build support folks]


Dear Zhouyi, dear Nathan, dear Nick,


To recap: On a ppc64le machine, building Linux in Ubuntu 21.10 with 
*llvm* and *clang* 1:13.0-53~exp1

     $ clang --version
     Ubuntu clang version 13.0.0-2
     Target: powerpc64le-unknown-linux-gnu
     Thread model: posix
     InstalledDir: /usr/bin

results in a segmentation fault, while it works when building with GCC.

     $ gcc --version
     gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0


```
[…]
[    T1] ipip: IPv4 and MPLS over IPv4 tunneling driver
[    T1] NET: Registered PF_INET6 protocol family
[    T1] Segment Routing with IPv6
[    T1] In-situ OAM (IOAM) with IPv6
[    T1] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    T1] BUG: Kernel NULL pointer dereference on write at 0x00000000
[    T1] Faulting instruction address: 0xc0000000008e2400
[    T1] Oops: Kernel access of bad area, sig: 11 [#1]
[    T1] LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=16 NUMA PowerNV
[    T1] Modules linked in:
[    T1] CPU: 11 PID: 1 Comm: swapper/0 Not tainted 
5.17.0-rc1-00032-gdd81e1c7d5fb #29
[    T1] NIP:  c0000000008e2400 LR: c000000000d65db0 CTR: c000000000f0bb60
[    T1] REGS: c0000000125033e0 TRAP: 0380   Not tainted 
(5.17.0-rc1-00032-gdd81e1c7d5fb)
[    T1] MSR:  9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 42800c40 
XER: 00000000
[    T1] CFAR: c000000000d65dac IRQMASK: 0
[    T1] GPR00: c000000000d65b40 c000000012503680 c00000000290c600 
0000000000000000
[    T1] GPR04: ffffffffffffffff 00000000ffffffff 0000000000000000 
0000000000000cc0
[    T1] GPR08: 0000000000000000 0000000000000000 ffffffffffffffff 
0000000000000001
[    T1] GPR12: 0000000000000000 c000007fffff6c00 c000000000012478 
0000000000000000
[    T1] GPR16: 0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[    T1] GPR20: 0000000000000000 c000000002810100 0000000000000cc0 
0000000000000000
[    T1] GPR24: 0000000000000010 c00000000294cf50 0000000000000000 
0000000000000000
[    T1] GPR28: 0000000000000000 c00000001ec61000 0000000000000000 
c000000012503680
[    T1] NIP [c0000000008e2400] strlen+0x10/0x30
[    T1] LR [c000000000d65db0] if_nlmsg_size+0x150/0x360
[    T1] Call Trace:
[    T1] [c000000012503680] [c0000000125036c0] 0xc0000000125036c0 
(unreliable)
[    T1] [c0000000125036f0] [c000000000d65b40] 
rtmsg_ifinfo_build_skb+0x80/0x1a0
[    T1] [c0000000125037b0] [c000000000d66be0] rtmsg_ifinfo+0x70/0xd0
[    T1] [c000000012503800] [c000000000d4de50] 
register_netdevice+0x690/0x770
[    T1] [c000000012503890] [c000000000d4e2bc] register_netdev+0x4c/0x80
[    T1] [c0000000125038c0] [c000000000f4784c] sit_init_net+0x10c/0x1d0
[    T1] [c000000012503910] [c000000000d33c0c] ops_init+0x13c/0x1b0
[    T1] [c000000012503970] [c000000000d331bc] 
register_pernet_operations+0xec/0x1e0
[    T1] [c0000000125039d0] [c000000000d33440] 
register_pernet_device+0x60/0xd0
[    T1] [c000000012503a20] [c000000002085478] sit_init+0x54/0x160
[    T1] [c000000012503ab0] [c000000000011ba8] do_one_initcall+0xd8/0x3b0
[    T1] [c000000012503c70] [c000000002006064] do_initcall_level+0xe4/0x1c4
[    T1] [c000000012503cc0] [c000000002005f20] do_initcalls+0x84/0xe4
[    T1] [c000000012503d40] [c000000002005c7c] 
kernel_init_freeable+0x160/0x1ec
[    T1] [c000000012503da0] [c0000000000124ac] kernel_init+0x3c/0x270
[    T1] [c000000012503e10] [c00000000000cd64] 
ret_from_kernel_thread+0x5c/0x64
[    T1] Instruction dump:
[    T1] eb81ffe0 7c0803a6 4e800020 00000000 00000000 00000000 60000000 
60000000
[    T1] 3883ffff 60000000 60000000 60000000 <8ca40001> 28050000 
4082fff8 7c632050
[    T1] ---[ end trace 0000000000000000 ]---
[    T1]
[    T1] Kernel panic - not syncing: Attempted to kill init! 
exitcode=0x0000000b
[…]
```

Am 30.01.22 um 14:24 schrieb Zhouyi Zhou:

> On Sun, Jan 30, 2022 at 4:19 PM Paul Menzel wrote:

>> Am 30.01.22 um 01:21 schrieb Zhouyi Zhou:
>>
>>> Thank you for your instructions, I learned a lot from this process.
>>
>> Same on my end.
>>
>>> On Sun, Jan 30, 2022 at 12:52 AM Paul Menzel <pmenzel@molgen.mpg.de> wrote:
>>
>>>> Am 29.01.22 um 03:23 schrieb Zhouyi Zhou:
>>>>
>>>>> I don't have an IBM machine, but I tried to analyze the problem using
>>>>> my x86_64 kvm virtual machine, I can't reproduce the bug using my
>>>>> x86_64 kvm virtual machine.
>>>>
>>>> No idea, if it’s architecture specific.
>>>>
>>>>> I saw the panic is caused by registration of sit device (A sit device
>>>>> is a type of virtual network device that takes our IPv6 traffic,
>>>>> encapsulates/decapsulates it in IPv4 packets, and sends/receives it
>>>>> over the IPv4 Internet to another host)
>>>>>
>>>>> sit device is registered in function sit_init_net:
>>>>> 1895    static int __net_init sit_init_net(struct net *net)
>>>>> 1896    {
>>>>> 1897        struct sit_net *sitn = net_generic(net, sit_net_id);
>>>>> 1898        struct ip_tunnel *t;
>>>>> 1899        int err;
>>>>> 1900
>>>>> 1901        sitn->tunnels[0] = sitn->tunnels_wc;
>>>>> 1902        sitn->tunnels[1] = sitn->tunnels_l;
>>>>> 1903        sitn->tunnels[2] = sitn->tunnels_r;
>>>>> 1904        sitn->tunnels[3] = sitn->tunnels_r_l;
>>>>> 1905
>>>>> 1906        if (!net_has_fallback_tunnels(net))
>>>>> 1907            return 0;
>>>>> 1908
>>>>> 1909        sitn->fb_tunnel_dev = alloc_netdev(sizeof(struct ip_tunnel), "sit0",
>>>>> 1910                           NET_NAME_UNKNOWN,
>>>>> 1911                           ipip6_tunnel_setup);
>>>>> 1912        if (!sitn->fb_tunnel_dev) {
>>>>> 1913            err = -ENOMEM;
>>>>> 1914            goto err_alloc_dev;
>>>>> 1915        }
>>>>> 1916        dev_net_set(sitn->fb_tunnel_dev, net);
>>>>> 1917        sitn->fb_tunnel_dev->rtnl_link_ops = &sit_link_ops;
>>>>> 1918        /* FB netdevice is special: we have one, and only one per netns.
>>>>> 1919         * Allowing to move it to another netns is clearly unsafe.
>>>>> 1920         */
>>>>> 1921        sitn->fb_tunnel_dev->features |= NETIF_F_NETNS_LOCAL;
>>>>> 1922
>>>>> 1923        err = register_netdev(sitn->fb_tunnel_dev);
>>>>> register_netdev on line 1923 will call if_nlmsg_size indirectly.
>>>>>
>>>>> On the other hand, the function that calls the paniced strlen is if_nlmsg_size:
>>>>> (gdb) disassemble if_nlmsg_size
>>>>> Dump of assembler code for function if_nlmsg_size:
>>>>>       0xffffffff81a0dc20 <+0>:    nopl   0x0(%rax,%rax,1)
>>>>>       0xffffffff81a0dc25 <+5>:    push   %rbp
>>>>>       0xffffffff81a0dc26 <+6>:    push   %r15
>>>>>       0xffffffff81a0dd04 <+228>:    je     0xffffffff81a0de20 <if_nlmsg_size+512>
>>>>>       0xffffffff81a0dd0a <+234>:    mov    0x10(%rbp),%rdi
>>>>>       ...
>>>>>     => 0xffffffff81a0dd0e <+238>:    callq  0xffffffff817532d0 <strlen>
>>>>>       0xffffffff81a0dd13 <+243>:    add    $0x10,%eax
>>>>>       0xffffffff81a0dd16 <+246>:    movslq %eax,%r12
>>>>
>>>> Excuse my ignorance, would that look the same for ppc64le?
>>>> Unfortunately, I didn’t save the problematic `vmlinuz` file, but on a
>>>> current build (without rcutorture) I have the line below, where strlen
>>>> shows up.
>>>>
>>>>        (gdb) disassemble if_nlmsg_size
>>>>        […]
>>>>        0xc000000000f7f82c <+332>: bl      0xc000000000a10e30 <strlen>
>>>>        […]
>>>>
>>>>> and the C code for 0xffffffff81a0dd0e is following (line 524):
>>>>> 515    static size_t rtnl_link_get_size(const struct net_device *dev)
>>>>> 516    {
>>>>> 517        const struct rtnl_link_ops *ops = dev->rtnl_link_ops;
>>>>> 518        size_t size;
>>>>> 519
>>>>> 520        if (!ops)
>>>>> 521            return 0;
>>>>> 522
>>>>> 523        size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
>>>>> 524               nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
>>>>
>>>> How do I connect the disassemby output with the corresponding line?
>>> I use "make  ARCH=powerpc CC=powerpc64le-linux-gnu-gcc-9
>>> CROSS_COMPILE=powerpc64le-linux-gnu- -j 16" to cross compile kernel
>>> for powerpc64le in my Ubuntu 20.04 x86_64.
>>>
>>> gdb-multiarch ./vmlinux
>>> (gdb)disassemble if_nlmsg_size
>>> [...]
>>> 0xc00000000191bf40 <+112>:    bl      0xc000000001c28ad0 <strlen>
>>> [...]
>>> (gdb) break *0xc00000000191bf40
>>> Breakpoint 1 at 0xc00000000191bf40: file ./include/net/netlink.h, line 1112.
>>>
>>> But in include/net/netlink.h:1112, I can't find the call to strlen
>>> 1110static inline int nla_total_size(int payload)
>>> 1111{
>>> 1112        return NLA_ALIGN(nla_attr_size(payload));
>>> 1113}
>>> This may be due to the compiler wrongly encode the debug information, I guess.
>>
>> `rtnl_link_get_size()` contains:
>>
>>               size = nla_total_size(sizeof(struct nlattr)) + /*
>> IFLA_LINKINFO */
>>                      nla_total_size(strlen(ops->kind) + 1);  /*
>> IFLA_INFO_KIND */
>>
>> Is that inlined(?) and the code at fault?
> Yes, that is inlined! because
> (gdb) disassemble if_nlmsg_size
> Dump of assembler code for function if_nlmsg_size:
> [...]
> 0xc00000000191bf38 <+104>:    beq     0xc00000000191c1f0 <if_nlmsg_size+800>
> 0xc00000000191bf3c <+108>:    ld      r3,16(r31)
> 0xc00000000191bf40 <+112>:    bl      0xc000000001c28ad0 <strlen>
> [...]
> (gdb)
> (gdb) break *0xc00000000191bf40
> Breakpoint 1 at 0xc00000000191bf40: file ./include/net/netlink.h, line 1112.
> (gdb) break *0xc00000000191bf38
> Breakpoint 2 at 0xc00000000191bf38: file net/core/rtnetlink.c, line 520.
> 
>>
>>>>> But ops is assigned the value of sit_link_ops in function sit_init_net
>>>>> line 1917, so I guess something must happened between the calls.
>>>>>
>>>>> Do we have KASAN in IBM machine? would KASAN help us find out what
>>>>> happened in between?
>>>>
>>>> Unfortunately, KASAN is not support on Power, I have, as far as I can
>>>> see. From `arch/powerpc/Kconfig`:
>>>>
>>>>            select HAVE_ARCH_KASAN                  if PPC32 && PPC_PAGE_SHIFT <= 14
>>>>            select HAVE_ARCH_KASAN_VMALLOC          if PPC32 && PPC_PAGE_SHIFT <= 14
>>>>
>>> en, agree, I invoke "make  menuconfig  ARCH=powerpc
>>> CC=powerpc64le-linux-gnu-gcc-9 CROSS_COMPILE=powerpc64le-linux-gnu- -j
>>> 16", I can't find KASAN under Memory Debugging, I guess we should find
>>> the bug by bisecting instead.
>>
>> I do not know, if it is a regression, as it was the first time I tried
>> to run a Linux kernel built with rcutorture on real hardware.
> I tried to add some debug statements to the kernel to locate the bug
> more accurately,  you can try it when you're not busy in the future,
> or just ignore it if the following patch looks not very effective ;-)
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 1baab07820f6..969ac7c540cc 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -9707,6 +9707,9 @@ int register_netdevice(struct net_device *dev)
>        *    Prevent userspace races by waiting until the network
>        *    device is fully setup before sending notifications.
>        */
> +    if (dev->rtnl_link_ops)
> +        printk(KERN_INFO "%lx IFLA_INFO_KIND %s %s\n", dev->rtnl_link_ops,
> +               dev->rtnl_link_ops->kind, __FUNCTION__);
>       if (!dev->rtnl_link_ops ||
>           dev->rtnl_link_state == RTNL_LINK_INITIALIZED)
>           rtmsg_ifinfo(RTM_NEWLINK, dev, ~0U, GFP_KERNEL);
> @@ -9788,6 +9791,9 @@ int register_netdev(struct net_device *dev)
> 
>       if (rtnl_lock_killable())
>           return -EINTR;
> +    if (dev->rtnl_link_ops)
> +        printk(KERN_INFO "%lx IFLA_INFO_KIND %s %s\n", dev->rtnl_link_ops,
> +               dev->rtnl_link_ops->kind, __FUNCTION__);
>       err = register_netdevice(dev);
>       rtnl_unlock();
>       return err;
> diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
> index e476403231f0..e08986ae6238 100644
> --- a/net/core/rtnetlink.c
> +++ b/net/core/rtnetlink.c
> @@ -520,6 +520,8 @@ static size_t rtnl_link_get_size(const struct
> net_device *dev)

Google Mail unfortunately wraps lines, so it’s better to attach patches.

>       if (!ops)
>           return 0;
> 
> +    printk(KERN_INFO "%lx IFLA_INFO_KIND %s %s\n", ops,
> +           ops->kind, __FUNCTION__);
>       size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
>              nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
> 
> @@ -1006,6 +1008,9 @@ static size_t rtnl_proto_down_size(const struct
> net_device *dev)
>   static noinline size_t if_nlmsg_size(const struct net_device *dev,
>                        u32 ext_filter_mask)
>   {
> +    if (dev->rtnl_link_ops)
> +        printk(KERN_INFO "%lx IFLA_INFO_KIND  %s %s\n", dev->rtnl_link_ops,
> +               dev->rtnl_link_ops->kind, __FUNCTION__);
>       return NLMSG_ALIGN(sizeof(struct ifinfomsg))
>              + nla_total_size(IFNAMSIZ) /* IFLA_IFNAME */
>              + nla_total_size(IFALIASZ) /* IFLA_IFALIAS */
> @@ -3825,7 +3830,9 @@ struct sk_buff *rtmsg_ifinfo_build_skb(int type,
> struct net_device *dev,
>       struct net *net = dev_net(dev);
>       struct sk_buff *skb;
>       int err = -ENOBUFS;
> -
> +    if (dev->rtnl_link_ops)
> +        printk(KERN_INFO "%lx IFLA_INFO_KIND %s %s\n", dev->rtnl_link_ops,
> +               dev->rtnl_link_ops->kind, __FUNCTION__);
>       skb = nlmsg_new(if_nlmsg_size(dev, 0), flags);
>       if (skb == NULL)
>           goto errout;
> @@ -3861,7 +3868,9 @@ static void rtmsg_ifinfo_event(int type, struct
> net_device *dev,
> 
>       if (dev->reg_state != NETREG_REGISTERED)
>           return;
> -
> +    if (dev->rtnl_link_ops)
> +        printk(KERN_INFO "%lx IFLA_INFO_KIND  %s %s\n", dev->rtnl_link_ops,
> +               dev->rtnl_link_ops->kind, __FUNCTION__);
>       skb = rtmsg_ifinfo_build_skb(type, dev, change, event, flags, new_nsid,
>                        new_ifindex);
>       if (skb)
> @@ -3871,6 +3880,9 @@ static void rtmsg_ifinfo_event(int type, struct
> net_device *dev,
>   void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change,
>             gfp_t flags)
>   {
> +    if (dev->rtnl_link_ops)
> +        printk(KERN_INFO "%lx IFLA_INFO_KIND  %s %s\n", dev->rtnl_link_ops,
> +               dev->rtnl_link_ops->kind, __FUNCTION__);
>       rtmsg_ifinfo_event(type, dev, change, rtnl_get_event(0), flags,
>                  NULL, 0);
>   }
> diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
> index c0b138c20992..fa5b2725811c 100644
> --- a/net/ipv6/sit.c
> +++ b/net/ipv6/sit.c
> @@ -1919,6 +1919,8 @@ static int __net_init sit_init_net(struct net *net)
>        * Allowing to move it to another netns is clearly unsafe.
>        */
>       sitn->fb_tunnel_dev->features |= NETIF_F_NETNS_LOCAL;
> -
> +    printk(KERN_INFO "%lx IFLA_INFO_KIND %s %s\n",
> +           sitn->fb_tunnel_dev->rtnl_link_ops,
> +           sitn->fb_tunnel_dev->rtnl_link_ops->kind, __FUNCTION__);
>       err = register_netdev(sitn->fb_tunnel_dev);
>       if (err)
>           goto err_reg_dev;

Thank you for the diff. I *am* able to reproduce the crash also in a 
QEMU/KVM virtual machine. config and Linux log is attached. Here is the 
excerpt with your added messages:

```
$ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=1 -net 
none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial 
stdio -m 512 -kernel /dev/shm/linux/vmlinux -append 
"debug_boot_weak_hash panic=-1 console=ttyS0 
rcupdate.rcu_cpu_stall_suppress_at_boot=1 torture.disable_onoff_at_boot 
rcupdate.rcu_task_stall_timeout=30000 rcupdate.rcu_self_test=1 
rcutorture.onoff_interval=1000 rcutorture.onoff_holdoff=30 
rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15 
rcutorture.shutdown_secs=420 rcutorture.test_no_idle_hz=1 
rcutorture.verbose=1"
[…]
[    0.445514][    T1] c00000000295c988 IFLA_INFO_KIND ipip 
register_netdevice
[    0.446330][    T1] c00000000295c988 IFLA_INFO_KIND  ipip rtmsg_ifinfo
[    0.447107][    T1] c00000000295c988 IFLA_INFO_KIND  ipip 
rtmsg_ifinfo_event
[    0.447935][    T1] c00000000295c988 IFLA_INFO_KIND ipip 
rtmsg_ifinfo_build_skb
[    0.448789][    T1] c00000000295c988 IFLA_INFO_KIND  ipip if_nlmsg_size
[    0.449563][    T1] c00000000295c988 IFLA_INFO_KIND ipip 
rtnl_link_get_size
[    0.450497][    T1] NET: Registered PF_INET6 protocol family
[    0.451402][    T1] Segment Routing with IPv6
[    0.451922][    T1] In-situ OAM (IOAM) with IPv6
[    0.452480][    T1] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.453259][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) sit_init_net
[    0.454035][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) 
register_netdev
[    0.454939][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) 
register_netdevice
[    0.455780][    T1] c00000000295cfd8 IFLA_INFO_KIND  (null) rtmsg_ifinfo
[    0.456563][    T1] c00000000295cfd8 IFLA_INFO_KIND  (null) 
rtmsg_ifinfo_event
[    0.457409][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) 
rtmsg_ifinfo_build_skb
[    0.458288][    T1] c00000000295cfd8 IFLA_INFO_KIND  (null) if_nlmsg_size
[    0.459085][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) 
rtnl_link_get_size
[    0.459921][    T1] BUG: Kernel NULL pointer dereference on read at 
0x00000000
[    0.460766][    T1] Faulting instruction address: 0xc00000000090b640
[    0.461513][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
[    0.462225][    T1] LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=16 
NUMA pSeries
[    0.463108][    T1] Modules linked in:
[    0.463549][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 
5.17.0-rc4-00219-g43a6dd55dd9d #28
[    0.464584][    T1] NIP:  c00000000090b640 LR: c000000000d9785c CTR: 
0000000000000000
[    0.465499][    T1] REGS: c0000000073e32b0 TRAP: 0380   Not tainted 
(5.17.0-rc4-00219-g43a6dd55dd9d)
[    0.466581][    T1] MSR:  8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE> 
  CR: 22800c47  XER: 00000000
[    0.467642][    T1] CFAR: c000000000d97858 IRQMASK: 0
[    0.467642][    T1] GPR00: c000000000d97850 c0000000073e3550 
c000000002919d00 0000000000000000
[    0.467642][    T1] GPR04: ffffffffffffffff ffffffffff1e7ef8 
ffffffffff1e9984 c00000000267ae88
[    0.467642][    T1] GPR08: 0000000000000003 0000000000000004 
c00000000267ae88 0000000000000000
[    0.467642][    T1] GPR12: 0000000000880000 c000000002ac0000 
c000000000012518 0000000000000000
[    0.467642][    T1] GPR16: 0000000000000000 0000000000000000 
0000000000000000 0000000000000000
[    0.467642][    T1] GPR20: 0000000000000000 c00000000281dc00 
0000000000000000 0000000000000cc0
[    0.467642][    T1] GPR24: 0000000000000000 0000000000000000 
0000000000000000 0000000000000000
[    0.467642][    T1] GPR28: c00000000295cfd8 c0000000079d3000 
0000000000000000 c0000000073e3550
[    0.476429][    T1] NIP [c00000000090b640] strlen+0x10/0x30
[    0.477085][    T1] LR [c000000000d9785c] if_nlmsg_size+0x2dc/0x3b0
[    0.477822][    T1] Call Trace:
[    0.478190][    T1] [c0000000073e3550] [c000000000d97850] 
if_nlmsg_size+0x2d0/0x3b0 (unreliable)
[    0.479226][    T1] [c0000000073e3600] [c000000000d9743c] 
rtmsg_ifinfo_build_skb+0x8c/0x1d0
[    0.480210][    T1] [c0000000073e36c0] [c000000000d98298] 
rtmsg_ifinfo+0x88/0x130
[    0.481086][    T1] [c0000000073e3750] [c000000000d7e118] 
register_netdevice+0x5c8/0x690
[    0.482037][    T1] [c0000000073e37e0] [c000000000d7e578] 
register_netdev+0x58/0xb0
[    0.482946][    T1] [c0000000073e3850] [c000000000f83ad0] 
sit_init_net+0x150/0x1a0
[    0.483838][    T1] [c0000000073e38d0] [c000000000d6469c] 
ops_init+0x13c/0x1b0
[    0.484691][    T1] [c0000000073e3930] [c000000000d63c4c] 
register_pernet_operations+0xec/0x1e0
[    0.485714][    T1] [c0000000073e3990] [c000000000d63ed0] 
register_pernet_device+0x60/0xd0
[    0.486689][    T1] [c0000000073e39e0] [c000000002085228] 
sit_init+0x54/0x160
[    0.487530][    T1] [c0000000073e3a70] [c000000000011c58] 
do_one_initcall+0x108/0x3e0
[    0.488455][    T1] [c0000000073e3c70] [c000000002006190] 
do_initcall_level+0xe4/0x1c4
[    0.489389][    T1] [c0000000073e3cc0] [c00000000200604c] 
do_initcalls+0x84/0xe4
[    0.490260][    T1] [c0000000073e3d40] [c000000002005da8] 
kernel_init_freeable+0x160/0x1ec
[    0.491236][    T1] [c0000000073e3da0] [c00000000001254c] 
kernel_init+0x3c/0x270
[    0.492108][    T1] [c0000000073e3e10] [c00000000000cd64] 
ret_from_kernel_thread+0x5c/0x64
[    0.493078][    T1] Instruction dump:
[    0.493509][    T1] eb81ffe0 7c0803a6 4e800020 00000000 00000000 
00000000 60000000 60000000
[    0.494524][    T1] 3883ffff 60000000 60000000 60000000 <8ca40001> 
28050000 4082fff8 7c632050
[    0.495542][    T1] ---[ end trace 0000000000000000 ]---
```

[…]


Kind regards,

Paul

[-- Attachment #2: linux-5.17-rc4-rcu-dev-messages.txt --]
[-- Type: text/plain, Size: 28451 bytes --]

[    0.000000][    T0] debug_boot_weak_hash enabled
[    0.000000][    T0] hash-mmu: Page sizes from device-tree:
[    0.000000][    T0] hash-mmu: base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
[    0.000000][    T0] hash-mmu: base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
[    0.000000][    T0] Using 1TB segments
[    0.000000][    T0] hash-mmu: Initializing hash mmu with SLB
[    0.000000][    T0] Linux version 5.17.0-rc4-00219-g43a6dd55dd9d (pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (Ubuntu clang version 13.0.0-2, LLD 13.0.0) #28 SMP PREEMPT Wed Feb 16 13:02:59 CET 2022
[    0.000000][    T0] Using pSeries machine description
[    0.000000][    T0] printk: bootconsole [udbg0] enabled
[    0.000000][    T0] Partition configured for 1 cpus.
[    0.000000][    T0] CPU maps initialized for 1 thread per core
[    0.000000][    T0] numa: Partition configured for 1 NUMA nodes.
[    0.000000][    T0] -----------------------------------------------------
[    0.000000][    T0] phys_mem_size     = 0x20000000
[    0.000000][    T0] dcache_bsize      = 0x80
[    0.000000][    T0] icache_bsize      = 0x80
[    0.000000][    T0] cpu_features      = 0x000000eb8f4d9187
[    0.000000][    T0]   possible        = 0x000ffbfbcf5fb187
[    0.000000][    T0]   always          = 0x0000000380008181
[    0.000000][    T0] cpu_user_features = 0xdc0065c2 0xae000000
[    0.000000][    T0] mmu_features      = 0x78006001
[    0.000000][    T0] firmware_features = 0x00000085455a445f
[    0.000000][    T0] vmalloc start     = 0xc008000000000000
[    0.000000][    T0] IO start          = 0xc00a000000000000
[    0.000000][    T0] vmemmap start     = 0xc00c000000000000
[    0.000000][    T0] hash-mmu: ppc64_pft_size    = 0x16
[    0.000000][    T0] hash-mmu: htab_hash_mask    = 0x7fff
[    0.000000][    T0] -----------------------------------------------------
[    0.000000][    T0] numa:   NODE_DATA [mem 0x1ffbb000-0x1ffbffff]
[    0.000000][    T0] rfi-flush: fallback displacement flush available
[    0.000000][    T0] rfi-flush: ori type flush available
[    0.000000][    T0] rfi-flush: mttrig type flush available
[    0.000000][    T0] count-cache-flush: hardware flush enabled.
[    0.000000][    T0] link-stack-flush: software flush enabled.
[    0.000000][    T0] stf-barrier: hwsync barrier available
[    0.000000][    T0] PPC64 nvram contains 65536 bytes
[    0.000000][    T0] PV qspinlock hash table entries: 4096 (order: 0, 65536 bytes, linear)
[    0.000000][    T0] barrier-nospec: using ORI speculation barrier
[    0.000000][    T0] Zone ranges:
[    0.000000][    T0]   Normal   [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000][    T0] Movable zone start for each node
[    0.000000][    T0] Early memory node ranges
[    0.000000][    T0]   node   0: [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000][    T0] Initmem setup node 0 [mem 0x0000000000000000-0x000000001fffffff]
[    0.000000][    T0] percpu: Embedded 10 pages/cpu s585880 r0 d69480 u1048576
[    0.000000][    T0] Fallback order for Node 0: 0
[    0.000000][    T0] Built 1 zonelists, mobility grouping on.  Total pages: 8184
[    0.000000][    T0] Policy zone: Normal
[    0.000000][    T0] Kernel command line: debug_boot_weak_hash panic=-1 console=ttyS0 rcupdate.rcu_cpu_stall_suppress_at_boot=1 torture.disable_onoff_at_boot rcupdate.rcu_task_stall_timeout=30000 rcupdate.rcu_self_test=1 rcutorture.onoff_interval=1000 rcutorture.onoff_holdoff=30 rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15 rcutorture.shutdown_secs=420 rcutorture.test_no_idle_hz=1 rcutorture.verbose=1
[    0.000000][    T0] Dentry cache hash table entries: 65536 (order: 3, 524288 bytes, linear)
[    0.000000][    T0] Inode-cache hash table entries: 32768 (order: 2, 262144 bytes, linear)
[    0.000000][    T0] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000][    T0] Memory: 411136K/524288K available (16448K kernel code, 3712K rwdata, 2688K rodata, 6016K init, 1265K bss, 113152K reserved, 0K cma-reserved)
[    0.000000][    T0] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000][    T0] ftrace: allocating 37449 entries in 14 pages
[    0.000000][    T0] ftrace: allocated 14 pages with 3 groups
[    0.000000][    T0] trace event string verifier disabled
[    0.000000][    T0] rcu: Preemptible hierarchical RCU implementation.
[    0.000000][    T0] rcu:     RCU event tracing is enabled.
[    0.000000][    T0] rcu:     CONFIG_RCU_FANOUT set to non-default value of 2.
[    0.000000][    T0] rcu:     Four(or more)-level hierarchy is enabled.
[    0.000000][    T0] rcu:     Build-time adjustment of leaf fanout to 2.
[    0.000000][    T0] rcu:     RCU restricting CPUs from NR_CPUS=16 to nr_cpu_ids=1.
[    0.000000][    T0] rcu:     RCU priority boosting: priority 1 delay 500 ms.
[    0.000000][    T0]  Tasks-RCU CPU stall warnings timeout set to 30000 (rcu_task_stall_timeout).
[    0.000000][    T0]  Trampoline variant of Tasks RCU enabled.
[    0.000000][    T0]  Rude variant of Tasks RCU enabled.
[    0.000000][    T0]  Tracing variant of Tasks RCU enabled.
[    0.000000][    T0] rcu: sanitize_kthread_prio: Limited prio to 2 from 1
[    0.000000][    T0] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000][    T0] rcu: Adjusting geometry for rcu_fanout_leaf=2, nr_cpu_ids=1
[    0.000000][    T0] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[    0.000000][    T0] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000][    T0] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[    0.001308][    T0] clocksource: timebase mult[1f40000] shift[24] registered
[    0.002178][    T0] Console: colour dummy device 80x25
[    0.002820][    T0] pid_max: default: 32768 minimum: 301
[    0.003483][    T0] Mount-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.004450][    T0] Mountpoint-cache hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.005901][    T1] cblist_init_generic: Setting adjustable number of callback queues.
[    0.006850][    T1] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.007682][    T1] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.008507][    T1] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.009345][    T1] POWER8 performance monitor hardware support registered
[    0.010189][    T1] rcu: Hierarchical SRCU implementation.
[    0.011779][    T1] smp: Bringing up secondary CPUs ...
[    0.012422][    T1] smp: Brought up 1 node, 1 CPU
[    0.013002][    T1] numa: Node 0 CPUs: 0
[    0.013598][    T1] devtmpfs: initialized
[    0.014824][    T1] random: get_random_u32 called from rhashtable_init+0x204/0x390 with crng_init=0
[    0.014935][    T1] PCI host bridge /pci@800000020000000  ranges:
[    0.016742][    T1]   IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[    0.017666][    T1]  MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[    0.018607][    T1]  MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[    0.019587][    T1] PCI: OF: PROBE_ONLY disabled
[    0.020166][    T1] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.021416][    T1] futex hash table entries: 256 (order: -1, 32768 bytes, linear)
[    0.022446][    T1] BUG: using smp_processor_id() in preemptible [00000000] code: swapper/0/1
[    0.023456][    T1] caller is apply_to_pte_range+0x380/0x480
[    0.024124][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00219-g43a6dd55dd9d #28
[    0.025171][    T1] Call Trace:
[    0.025541][    T1] [c0000000073e3530] [c0000000008f78a0] dump_stack_lvl+0x78/0xb8 (unreliable)
[    0.026583][    T1] [c0000000073e3570] [c000000000ffae04] check_preemption_disabled+0x134/0x150
[    0.027620][    T1] [c0000000073e3600] [c000000000424450] apply_to_pte_range+0x380/0x480
[    0.028572][    T1] [c0000000073e3680] [c00000000042401c] apply_to_pmd_range+0x2ac/0x360
[    0.029532][    T1] [c0000000073e3720] [c000000000423cb0] apply_to_pud_range+0x2d0/0x390
[    0.030500][    T1] [c0000000073e37d0] [c00000000041ba94] __apply_to_page_range+0x1d4/0x290
[    0.031491][    T1] [c0000000073e3880] [c00000000008c0a8] change_memory_attr+0x98/0x120
[    0.032443][    T1] [c0000000073e38c0] [c0000000003101ac] bpf_prog_select_runtime+0x9c/0x390
[    0.033442][    T1] [c0000000073e3930] [c000000000da51b4] bpf_prepare_filter+0x574/0x5b0
[    0.034408][    T1] [c0000000073e39a0] [c000000000da4be4] bpf_prog_create+0xb4/0x110
[    0.035329][    T1] [c0000000073e39f0] [c00000000207d064] ptp_classifier_init+0x50/0x78
[    0.036279][    T1] [c0000000073e3a30] [c00000000207c208] sock_init+0x110/0x128
[    0.037147][    T1] [c0000000073e3a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
[    0.038075][    T1] [c0000000073e3c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
[    0.039016][    T1] [c0000000073e3cc0] [c00000000200604c] do_initcalls+0x84/0xe4
[    0.039891][    T1] [c0000000073e3d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
[    0.040875][    T1] [c0000000073e3da0] [c00000000001254c] kernel_init+0x3c/0x270
[    0.041753][    T1] [c0000000073e3e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
[    0.042733][    T1] BUG: using smp_processor_id() in preemptible [00000000] code: swapper/0/1
[    0.043739][    T1] caller is apply_to_pte_range+0x420/0x480
[    0.044410][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00219-g43a6dd55dd9d #28
[    0.045448][    T1] Call Trace:
[    0.045817][    T1] [c0000000073e3530] [c0000000008f78a0] dump_stack_lvl+0x78/0xb8 (unreliable)
[    0.046857][    T1] [c0000000073e3570] [c000000000ffae04] check_preemption_disabled+0x134/0x150
[    0.047894][    T1] [c0000000073e3600] [c0000000004244f0] apply_to_pte_range+0x420/0x480
[    0.048855][    T1] [c0000000073e3680] [c00000000042401c] apply_to_pmd_range+0x2ac/0x360
[    0.049810][    T1] [c0000000073e3720] [c000000000423cb0] apply_to_pud_range+0x2d0/0x390
[    0.050776][    T1] [c0000000073e37d0] [c00000000041ba94] __apply_to_page_range+0x1d4/0x290
[    0.051766][    T1] [c0000000073e3880] [c00000000008c0a8] change_memory_attr+0x98/0x120
[    0.052713][    T1] [c0000000073e38c0] [c0000000003101ac] bpf_prog_select_runtime+0x9c/0x390
[    0.053717][    T1] [c0000000073e3930] [c000000000da51b4] bpf_prepare_filter+0x574/0x5b0
[    0.054680][    T1] [c0000000073e39a0] [c000000000da4be4] bpf_prog_create+0xb4/0x110
[    0.055598][    T1] [c0000000073e39f0] [c00000000207d064] ptp_classifier_init+0x50/0x78
[    0.056551][    T1] [c0000000073e3a30] [c00000000207c208] sock_init+0x110/0x128
[    0.057421][    T1] [c0000000073e3a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
[    0.058349][    T1] [c0000000073e3c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
[    0.059295][    T1] [c0000000073e3cc0] [c00000000200604c] do_initcalls+0x84/0xe4
[    0.060172][    T1] [c0000000073e3d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
[    0.061151][    T1] [c0000000073e3da0] [c00000000001254c] kernel_init+0x3c/0x270
[    0.062028][    T1] [c0000000073e3e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
[    0.063015][    T1] BUG: using smp_processor_id() in preemptible [00000000] code: swapper/0/1
[    0.064024][    T1] caller is __flush_tlb_pending+0x54/0xe0
[    0.064679][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00219-g43a6dd55dd9d #28
[    0.065714][    T1] Call Trace:
[    0.066085][    T1] [c0000000073e34f0] [c0000000008f78a0] dump_stack_lvl+0x78/0xb8 (unreliable)
[    0.067117][    T1] [c0000000073e3530] [c000000000ffae04] check_preemption_disabled+0x134/0x150
[    0.068145][    T1] [c0000000073e35c0] [c0000000000961e4] __flush_tlb_pending+0x54/0xe0
[    0.069095][    T1] [c0000000073e3600] [c000000000424524] apply_to_pte_range+0x454/0x480
[    0.070047][    T1] [c0000000073e3680] [c00000000042401c] apply_to_pmd_range+0x2ac/0x360
[    0.070998][    T1] [c0000000073e3720] [c000000000423cb0] apply_to_pud_range+0x2d0/0x390
[    0.071958][    T1] [c0000000073e37d0] [c00000000041ba94] __apply_to_page_range+0x1d4/0x290
[    0.072943][    T1] [c0000000073e3880] [c00000000008c0a8] change_memory_attr+0x98/0x120
[    0.073888][    T1] [c0000000073e38c0] [c0000000003101ac] bpf_prog_select_runtime+0x9c/0x390
[    0.074885][    T1] [c0000000073e3930] [c000000000da51b4] bpf_prepare_filter+0x574/0x5b0
[    0.075837][    T1] [c0000000073e39a0] [c000000000da4be4] bpf_prog_create+0xb4/0x110
[    0.076748][    T1] [c0000000073e39f0] [c00000000207d064] ptp_classifier_init+0x50/0x78
[    0.077691][    T1] [c0000000073e3a30] [c00000000207c208] sock_init+0x110/0x128
[    0.078557][    T1] [c0000000073e3a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
[    0.079485][    T1] [c0000000073e3c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
[    0.080415][    T1] [c0000000073e3cc0] [c00000000200604c] do_initcalls+0x84/0xe4
[    0.081284][    T1] [c0000000073e3d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
[    0.082260][    T1] [c0000000073e3da0] [c00000000001254c] kernel_init+0x3c/0x270
[    0.083138][    T1] [c0000000073e3e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
[    0.084149][    T1] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.085136][    T1] cpuidle: using governor menu
Linux ppc64le
#28 SMP PREEMPT [    0.086193][    T1] EEH: pSeries platform initialized
[    0.087192][    T1] software IO TLB: tearing down default memory pool
[    0.088124][    T1] PCI: Probing PCI hardware
[    0.088690][    T1] PCI host bridge to bus 0000:00
[    0.089257][    T1] pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    0.090398][    T1] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff])
[    0.091762][    T1] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff 64bit]
[    0.092799][    T1] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.094562][    T1] IOMMU table initialized, virtual merging enabled
[    0.095323][    T1] pci_bus 0000:00: resource 4 [io  0x10000-0x1ffff]
[    0.096083][    T1] pci_bus 0000:00: resource 5 [mem 0x200080000000-0x2000ffffffff]
[    0.096986][    T1] pci_bus 0000:00: resource 6 [mem 0x210000000000-0x21ffffffffff 64bit]
[    0.097946][    T1] EEH: No capable adapters found: recovery disabled.
[    0.101687][    T1] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[    0.103307][    T1] iommu: Default domain type: Translated
[    0.103969][    T1] iommu: DMA domain TLB invalidation policy: strict mode
[    0.104830][    T1] vgaarb: loaded
[    0.105353][    T1] SCSI subsystem initialized
[    0.105953][    T1] usbcore: registered new interface driver usbfs
[    0.106699][    T1] usbcore: registered new interface driver hub
[    0.107417][    T1] usbcore: registered new device driver usb
[    0.108114][    T1] pps_core: LinuxPPS API ver. 1 registered
[    0.108784][    T1] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.109946][    T1] PTP clock support registered
[    0.110521][    T1] EDAC MC: Ver: 3.0.0
[    0.111202][    T1] clocksource: Switched to clocksource timebase
[    0.132640][    T1] hugetlbfs: disabling because there are no supported hugepage sizes
[    0.141692][    T1] NET: Registered PF_INET protocol family
[    0.142391][    T1] IP idents hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.143581][    T1] tcp_listen_portaddr_hash hash table entries: 4096 (order: 0, 65536 bytes, linear)
[    0.144693][    T1] TCP established hash table entries: 8192 (order: 0, 65536 bytes, linear)
[    0.145710][    T1] TCP bind hash table entries: 8192 (order: 1, 131072 bytes, linear)
[    0.146673][    T1] TCP: Hash tables configured (established 8192 bind 8192)
[    0.147527][    T1] UDP hash table entries: 2048 (order: 0, 65536 bytes, linear)
[    0.148416][    T1] UDP-Lite hash table entries: 2048 (order: 0, 65536 bytes, linear)
[    0.149373][    T1] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.161303][    T1] RPC: Registered named UNIX socket transport module.
[    0.162089][    T1] RPC: Registered udp transport module.
[    0.162732][    T1] RPC: Registered tcp transport module.
[    0.163367][    T1] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.164209][    T1] PCI: CLS 0 bytes, default 128
[    0.170221][    T1] vas: API is supported only with radix page tables
[    0.171032][    T1] rcu-torture:--- Start of test: nreaders=1 nfakewriters=4 stat_interval=15 verbose=1 test_no_idle_hz=1 shuffle_interval=3 stutter=5 irqreader=1 fqs_duration=0 fqs_holdoff=0 fqs_stutter=3 test_boost=1/1 test_boost_interval=7 test_boost_duration=4 shutdown_secs=420 stall_cpu=0 stall_cpu_holdoff=10 stall_cpu_irqsoff=0 stall_cpu_block=0 n_barrier_cbs=4 onoff_interval=1000 onoff_holdoff=30 read_exit_delay=13 read_exit_burst=16 nocbs_nthreads=0 nocbs_toggle=1000
[    0.181210][    T1] rcu:  Start-test grace-period state: g-1187 f0x0
[    0.181966][    T1] rcu_torture_write_types: Testing conditional GPs.
[    0.182732][    T1] rcu_torture_write_types: Testing conditional expedited GPs.
[    0.183594][    T1] rcu_torture_write_types: Testing expedited GPs.
[    0.184330][    T1] rcu_torture_write_types: Testing asynchronous GPs.
[    0.185100][    T1] rcu_torture_write_types: Testing polling GPs.
[    0.185827][    T1] rcu_torture_write_types: Testing polling expedited GPs.
[    0.186652][    T1] rcu_torture_write_types: Testing normal GPs.
[    0.187360][    T1] rcu-torture: Creating rcu_torture_writer task
[    0.201208][   T37] rcu-torture: rcu_torture_writer task started
[    0.201922][   T37] rcu-torture: GP expediting controlled from boot/sysfs for rcu.
[    0.202832][    T1] rcu-torture: Creating rcu_torture_fakewriter task
[    0.204203][    T1] rcu-torture: Creating rcu_torture_fakewriter task
[    0.205812][   T38] rcu-torture: rcu_torture_fakewriter task started
[    0.207256][    T1] rcu-torture: Creating rcu_torture_fakewriter task
[    0.208357][   T39] rcu-torture: rcu_torture_fakewriter task started
[    0.209117][    T1] rcu-torture: Creating rcu_torture_fakewriter task
[    0.209902][   T40] rcu-torture: rcu_torture_fakewriter task started
[    0.210661][    T1] rcu-torture: Creating rcu_torture_reader task
[    0.211397][   T41] rcu-torture: rcu_torture_fakewriter task started
[    0.212152][    T1] rcu-torture: Creating rcu_torture_stats task
[    0.212871][   T42] rcu-torture: rcu_torture_reader task started
[    0.221222][    T1] rcu-torture: Creating torture_shuffle task
[    0.223316][   T43] rcu-torture: rcu_torture_stats task started
[    0.224023][    T1] rcu-torture: Creating torture_stutter task
[    0.224725][   T44] rcu-torture: torture_shuffle task started
[    0.225412][   T17] rcu-torture: Creating rcu_torture_boost task
[    0.226129][   T45] rcu-torture: torture_stutter task started
[    0.226846][    T1] rcu-torture: Creating torture_shutdown task
[    0.227553][   T46] rcu-torture: rcu_torture_boost started
[    0.228210][    T1] rcu-torture: Creating torture_onoff task
[    0.228884][   T47] rcu-torture: torture_shutdown task started
[    0.229570][   T47] rcu-torture:torture_shutdown task: 419997 ms remaining
[    0.241213][    T1] rcu-torture: rcu_torture_fwd_prog_init: Limiting fwd_progress to # CPUs.
[    0.241213][    T1]
[    0.242908][    T1] rcu-torture: Creating rcu_torture_fwd_prog task
[    0.243720][   T48] rcu-torture: torture_onoff task started
[    0.244380][   T48] rcu-torture: Only one CPU, so CPU-hotplug testing is disabled
[    0.245262][   T48] rcu-torture: torture_onoff is stopping
[    0.245916][    T1] rcu-torture: Creating rcu_torture_barrier_cbs task
[    0.246700][   T49] rcu-torture: rcu_torture_fwd_progress task started
[    0.247477][    T1] rcu-torture: Creating rcu_torture_barrier_cbs task
[    0.248255][   T50] rcu-torture: rcu_torture_barrier_cbs task started
[    0.249020][    T1] rcu-torture: Creating rcu_torture_barrier_cbs task
[    0.249799][   T51] rcu-torture: rcu_torture_barrier_cbs task started
[    0.250567][    T1] rcu-torture: Creating rcu_torture_barrier_cbs task
[    0.251357][   T52] rcu-torture: rcu_torture_barrier_cbs task started
[    0.252119][    T1] rcu-torture: Creating rcu_torture_barrier task
[    0.252855][   T53] rcu-torture: rcu_torture_barrier_cbs task started
[    0.253630][    T1] rcu-torture: Creating rcu_torture_read_exit task
[    0.254394][   T54] rcu-torture: rcu_torture_barrier task starting
[    0.255306][   T55] rcu-torture: rcu_torture_read_exit: Start of test
[    0.311311][   T56] rcu_torture_rea (56) used greatest stack depth: 14256 bytes left
[    0.362561][    T1] workingset: timestamp_bits=38 max_order=13 bucket_order=0
[    0.365842][    T1] NFS: Registering the id_resolver key type
[    0.366535][    T1] Key type id_resolver registered
[    0.367106][    T1] Key type id_legacy registered
[    0.367668][    T1] SGI XFS with ACLs, security attributes, no debug enabled
[    0.368871][    T1] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
[    0.369822][    T1] io scheduler mq-deadline registered
[    0.370437][    T1] io scheduler kyber registered
[    0.379363][    T1] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    0.380412][    T1] Non-volatile memory driver v1.3
[    0.382786][    T1] brd: module loaded
[    0.384256][    T1] loop: module loaded
[    0.384745][    T1] ipr: IBM Power RAID SCSI Device Driver version: 2.6.4 (March 14, 2017)
[    0.386523][    T1] ibmvscsi 71000002: SRP_VERSION: 16.a
[    0.387285][    T1] ibmvscsi 71000002: Maximum ID: 64 Maximum LUN: 32 Maximum Channel: 3
[    0.388237][    T1] scsi host0: IBM POWER Virtual SCSI Adapter 1.5.9
[    0.389217][    C0] ibmvscsi 71000002: partner initialization complete
[    0.389995][    C0] ibmvscsi 71000002: host srp version: 16.a, host partition qemu (0), OS 2, max io 2097152
[    0.391161][    C0] ibmvscsi 71000002: sent SRP login
[    0.391764][    C0] ibmvscsi 71000002: SRP_LOGIN succeeded
[    0.405762][    C0] random: fast init done
[    0.433462][    T1] e100: Intel(R) PRO/100 Network Driver
[    0.434101][    T1] e100: Copyright(c) 1999-2006 Intel Corporation
[    0.434834][    T1] e1000: Intel(R) PRO/1000 Network Driver
[    0.435487][    T1] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    0.436248][    T1] e1000e: Intel(R) PRO/1000 Network Driver
[    0.436913][    T1] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    0.437706][    T1] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.438561][    T1] ehci-pci: EHCI PCI platform driver
[    0.439179][    T1] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.439995][    T1] ohci-pci: OHCI PCI platform driver
[    0.440653][    T1] i2c_dev: i2c /dev entries driver
[    0.441401][    T1] device-mapper: uevent: version 1.0.3
[    0.442102][    T1] device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
[    0.443348][    T1] usbcore: registered new interface driver usbhid
[    0.444095][    T1] usbhid: USB HID core driver
[    0.444685][    T1] ipip: IPv4 and MPLS over IPv4 tunneling driver
[    0.445514][    T1] c00000000295c988 IFLA_INFO_KIND ipip register_netdevice
[    0.446330][    T1] c00000000295c988 IFLA_INFO_KIND  ipip rtmsg_ifinfo
[    0.447107][    T1] c00000000295c988 IFLA_INFO_KIND  ipip rtmsg_ifinfo_event
[    0.447935][    T1] c00000000295c988 IFLA_INFO_KIND ipip rtmsg_ifinfo_build_skb
[    0.448789][    T1] c00000000295c988 IFLA_INFO_KIND  ipip if_nlmsg_size
[    0.449563][    T1] c00000000295c988 IFLA_INFO_KIND ipip rtnl_link_get_size
[    0.450497][    T1] NET: Registered PF_INET6 protocol family
[    0.451402][    T1] Segment Routing with IPv6
[    0.451922][    T1] In-situ OAM (IOAM) with IPv6
[    0.452480][    T1] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.453259][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) sit_init_net
[    0.454035][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) register_netdev
[    0.454939][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) register_netdevice
[    0.455780][    T1] c00000000295cfd8 IFLA_INFO_KIND  (null) rtmsg_ifinfo
[    0.456563][    T1] c00000000295cfd8 IFLA_INFO_KIND  (null) rtmsg_ifinfo_event
[    0.457409][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) rtmsg_ifinfo_build_skb
[    0.458288][    T1] c00000000295cfd8 IFLA_INFO_KIND  (null) if_nlmsg_size
[    0.459085][    T1] c00000000295cfd8 IFLA_INFO_KIND (null) rtnl_link_get_size
[    0.459921][    T1] BUG: Kernel NULL pointer dereference on read at 0x00000000
[    0.460766][    T1] Faulting instruction address: 0xc00000000090b640
[    0.461513][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
[    0.462225][    T1] LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=16 NUMA pSeries
[    0.463108][    T1] Modules linked in:
[    0.463549][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00219-g43a6dd55dd9d #28
[    0.464584][    T1] NIP:  c00000000090b640 LR: c000000000d9785c CTR: 0000000000000000
[    0.465499][    T1] REGS: c0000000073e32b0 TRAP: 0380   Not tainted  (5.17.0-rc4-00219-g43a6dd55dd9d)
[    0.466581][    T1] MSR:  8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE>  CR: 22800c47  XER: 00000000
[    0.467642][    T1] CFAR: c000000000d97858 IRQMASK: 0
[    0.467642][    T1] GPR00: c000000000d97850 c0000000073e3550 c000000002919d00 0000000000000000
[    0.467642][    T1] GPR04: ffffffffffffffff ffffffffff1e7ef8 ffffffffff1e9984 c00000000267ae88
[    0.467642][    T1] GPR08: 0000000000000003 0000000000000004 c00000000267ae88 0000000000000000
[    0.467642][    T1] GPR12: 0000000000880000 c000000002ac0000 c000000000012518 0000000000000000
[    0.467642][    T1] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.467642][    T1] GPR20: 0000000000000000 c00000000281dc00 0000000000000000 0000000000000cc0
[    0.467642][    T1] GPR24: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    0.467642][    T1] GPR28: c00000000295cfd8 c0000000079d3000 0000000000000000 c0000000073e3550
[    0.476429][    T1] NIP [c00000000090b640] strlen+0x10/0x30
[    0.477085][    T1] LR [c000000000d9785c] if_nlmsg_size+0x2dc/0x3b0
[    0.477822][    T1] Call Trace:
[    0.478190][    T1] [c0000000073e3550] [c000000000d97850] if_nlmsg_size+0x2d0/0x3b0 (unreliable)
[    0.479226][    T1] [c0000000073e3600] [c000000000d9743c] rtmsg_ifinfo_build_skb+0x8c/0x1d0
[    0.480210][    T1] [c0000000073e36c0] [c000000000d98298] rtmsg_ifinfo+0x88/0x130
[    0.481086][    T1] [c0000000073e3750] [c000000000d7e118] register_netdevice+0x5c8/0x690
[    0.482037][    T1] [c0000000073e37e0] [c000000000d7e578] register_netdev+0x58/0xb0
[    0.482946][    T1] [c0000000073e3850] [c000000000f83ad0] sit_init_net+0x150/0x1a0
[    0.483838][    T1] [c0000000073e38d0] [c000000000d6469c] ops_init+0x13c/0x1b0
[    0.484691][    T1] [c0000000073e3930] [c000000000d63c4c] register_pernet_operations+0xec/0x1e0
[    0.485714][    T1] [c0000000073e3990] [c000000000d63ed0] register_pernet_device+0x60/0xd0
[    0.486689][    T1] [c0000000073e39e0] [c000000002085228] sit_init+0x54/0x160
[    0.487530][    T1] [c0000000073e3a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
[    0.488455][    T1] [c0000000073e3c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
[    0.489389][    T1] [c0000000073e3cc0] [c00000000200604c] do_initcalls+0x84/0xe4
[    0.490260][    T1] [c0000000073e3d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
[    0.491236][    T1] [c0000000073e3da0] [c00000000001254c] kernel_init+0x3c/0x270
[    0.492108][    T1] [c0000000073e3e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
[    0.493078][    T1] Instruction dump:
[    0.493509][    T1] eb81ffe0 7c0803a6 4e800020 00000000 00000000 00000000 60000000 60000000
[    0.494524][    T1] 3883ffff 60000000 60000000 60000000 <8ca40001> 28050000 4082fff8 7c632050
[    0.495542][    T1] ---[ end trace 0000000000000000 ]---
[    0.498158][    T1]
[    1.501404][    T1] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

[-- Attachment #3: linux-5.17-rc4-rcu-dev-config.txt --]
[-- Type: text/plain, Size: 112822 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/powerpc 5.17.0-rc4 Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="Ubuntu clang version 13.0.0-2"
CONFIG_GCC_VERSION=0
CONFIG_CC_IS_CLANG=y
CONFIG_CLANG_VERSION=130000
CONFIG_AS_IS_GNU=y
CONFIG_AS_VERSION=23700
CONFIG_LD_VERSION=0
CONFIG_LD_IS_LLD=y
CONFIG_LLD_VERSION=130000
CONFIG_CC_CAN_LINK=y
CONFIG_CC_CAN_LINK_STATIC=y
CONFIG_CC_HAS_ASM_GOTO=y
CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
CONFIG_TOOLS_SUPPORT_RELR=y
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_TABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_COMPILE_TEST is not set
# CONFIG_WERROR is not set
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_BUILD_SALT=""
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_XZ is not set
CONFIG_DEFAULT_INIT=""
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
# CONFIG_WATCH_QUEUE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
# CONFIG_USELIB is not set
# CONFIG_AUDIT is not set
CONFIG_HAVE_ARCH_AUDITSYSCALL=y

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_IRQ_SHOW_LEVEL=y
CONFIG_GENERIC_IRQ_MIGRATION=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_GENERIC_MSI_IRQ=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
# CONFIG_GENERIC_IRQ_DEBUGFS is not set
# end of IRQ subsystem

CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_ARCH_HAS_TICK_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CMOS_UPDATE=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ_FULL is not set
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
# end of Timers subsystem

CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y

#
# BPF subsystem
#
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
# CONFIG_BPF_PRELOAD is not set
# end of BPF subsystem

CONFIG_PREEMPT_BUILD=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
# CONFIG_SCHED_CORE is not set

#
# CPU/Task time and stats accounting
#
CONFIG_VIRT_CPU_ACCOUNTING=y
# CONFIG_TICK_CPU_ACCOUNTING is not set
CONFIG_VIRT_CPU_ACCOUNTING_NATIVE=y
# CONFIG_VIRT_CPU_ACCOUNTING_GEN is not set
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
# CONFIG_TASK_XACCT is not set
# CONFIG_PSI is not set
# end of CPU/Task time and stats accounting

CONFIG_CPU_ISOLATION=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
CONFIG_PREEMPT_RCU=y
CONFIG_RCU_EXPERT=y
CONFIG_SRCU=y
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU_GENERIC=y
CONFIG_TASKS_RCU=y
CONFIG_TASKS_RUDE_RCU=y
CONFIG_TASKS_TRACE_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_NEED_SEGCBLIST=y
CONFIG_RCU_FANOUT=2
CONFIG_RCU_FANOUT_LEAF=2
CONFIG_RCU_BOOST=y
CONFIG_RCU_BOOST_DELAY=500
# CONFIG_RCU_NOCB_CPU is not set
# CONFIG_TASKS_TRACE_RCU_READ_MB is not set
# end of RCU Subsystem

CONFIG_BUILD_BIN2C=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_IKHEADERS is not set
CONFIG_LOG_BUF_SHIFT=18
CONFIG_LOG_CPU_MAX_BUF_SHIFT=13
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
# CONFIG_PRINTK_INDEX is not set

#
# Scheduler features
#
# end of Scheduler features

CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_CC_HAS_INT128=y
CONFIG_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_CGROUPS=y
CONFIG_PAGE_COUNTER=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_KMEM=y
# CONFIG_BLK_CGROUP is not set
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_CFS_BANDWIDTH is not set
# CONFIG_RT_GROUP_SCHED is not set
# CONFIG_CGROUP_PIDS is not set
# CONFIG_CGROUP_RDMA is not set
CONFIG_CGROUP_FREEZER=y
# CONFIG_CGROUP_HUGETLB is not set
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
# CONFIG_CGROUP_MISC is not set
# CONFIG_CGROUP_DEBUG is not set
CONFIG_SOCK_CGROUP_DATA=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_TIME_NS=y
CONFIG_IPC_NS=y
# CONFIG_USER_NS is not set
CONFIG_PID_NS=y
CONFIG_NET_NS=y
# CONFIG_CHECKPOINT_RESTORE is not set
# CONFIG_SCHED_AUTOGROUP is not set
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE="/dev/shm/linux/tools/testing/selftests/rcutorture/initrd"
CONFIG_INITRAMFS_ROOT_UID=0
CONFIG_INITRAMFS_ROOT_GID=0
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
CONFIG_RD_ZSTD=y
CONFIG_INITRAMFS_COMPRESSION_GZIP=y
# CONFIG_INITRAMFS_COMPRESSION_BZIP2 is not set
# CONFIG_INITRAMFS_COMPRESSION_LZMA is not set
# CONFIG_INITRAMFS_COMPRESSION_XZ is not set
# CONFIG_INITRAMFS_COMPRESSION_LZO is not set
# CONFIG_INITRAMFS_COMPRESSION_LZ4 is not set
# CONFIG_INITRAMFS_COMPRESSION_ZSTD is not set
# CONFIG_INITRAMFS_COMPRESSION_NONE is not set
# CONFIG_BOOT_CONFIG is not set
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
CONFIG_LD_ORPHAN_WARN=y
CONFIG_SYSCTL=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
# CONFIG_EXPERT is not set
CONFIG_MULTIUSER=y
CONFIG_SGETMASK_SYSCALL=y
CONFIG_SYSFS_SYSCALL=y
CONFIG_FHANDLE=y
CONFIG_POSIX_TIMERS=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_FUTEX_PI=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_IO_URING=y
CONFIG_ADVISE_SYSCALLS=y
CONFIG_MEMBARRIER=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
CONFIG_KALLSYMS_BASE_RELATIVE=y
# CONFIG_USERFAULTFD is not set
CONFIG_ARCH_HAS_MEMBARRIER_CALLBACKS=y
CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y
CONFIG_RSEQ=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# end of Kernel Performance Events And Counters

CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
CONFIG_SLAB_MERGE_DEFAULT=y
# CONFIG_SLAB_FREELIST_RANDOM is not set
# CONFIG_SLAB_FREELIST_HARDENED is not set
# CONFIG_SHUFFLE_PAGE_ALLOCATOR is not set
CONFIG_SLUB_CPU_PARTIAL=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
# end of General setup

CONFIG_PPC64=y

#
# Processor support
#
CONFIG_PPC_BOOK3S_64=y
# CONFIG_PPC_BOOK3E_64 is not set
CONFIG_GENERIC_CPU=y
# CONFIG_POWER7_CPU is not set
# CONFIG_POWER8_CPU is not set
# CONFIG_POWER9_CPU is not set
CONFIG_PPC_BOOK3S=y
CONFIG_PPC_FPU_REGS=y
CONFIG_PPC_FPU=y
CONFIG_ALTIVEC=y
CONFIG_VSX=y
CONFIG_PPC_64S_HASH_MMU=y
CONFIG_PPC_RADIX_MMU=y
CONFIG_PPC_RADIX_MMU_DEFAULT=y
CONFIG_PPC_KUEP=y
CONFIG_PPC_KUAP=y
# CONFIG_PPC_KUAP_DEBUG is not set
CONFIG_PPC_PKEY=y
CONFIG_PPC_MM_SLICES=y
CONFIG_PPC_HAVE_PMU_SUPPORT=y
# CONFIG_PMU_SYSFS is not set
CONFIG_PPC_PERF_CTRS=y
CONFIG_FORCE_SMP=y
CONFIG_SMP=y
CONFIG_NR_CPUS=16
CONFIG_PPC_DOORBELL=y
# end of Processor support

# CONFIG_CPU_BIG_ENDIAN is not set
CONFIG_CPU_LITTLE_ENDIAN=y
CONFIG_PPC64_BOOT_WRAPPER=y
CONFIG_64BIT=y
CONFIG_MMU=y
CONFIG_ARCH_MMAP_RND_BITS_MAX=29
CONFIG_ARCH_MMAP_RND_BITS_MIN=14
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MAX=13
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=7
CONFIG_NR_IRQS=512
CONFIG_NMI_IPI=y
CONFIG_PPC_WATCHDOG=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_GENERIC_LOCKBREAK=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_PPC=y
CONFIG_PPC_BARRIER_NOSPEC=y
CONFIG_EARLY_PRINTK=y
CONFIG_PANIC_TIMEOUT=180
# CONFIG_COMPAT is not set
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_PPC_UDBG_16550=y
CONFIG_AUDIT_ARCH=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_EPAPR_BOOT=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_SUSPEND_NONZERO_CPU=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_PPC_DAWR=y
CONFIG_PGTABLE_LEVELS=4
CONFIG_PPC_MSI_BITMAP=y
CONFIG_PPC_XICS=y
CONFIG_PPC_ICP_NATIVE=y
CONFIG_PPC_ICP_HV=y
CONFIG_PPC_ICS_RTAS=y
CONFIG_PPC_XIVE=y
CONFIG_PPC_XIVE_NATIVE=y
CONFIG_PPC_XIVE_SPAPR=y

#
# Platform support
#
CONFIG_PPC_POWERNV=y
# CONFIG_OPAL_PRD is not set
# CONFIG_PPC_MEMTRACE is not set
# CONFIG_SCOM_DEBUGFS is not set
CONFIG_PPC_PSERIES=y
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_PPC_SPLPAR=y
CONFIG_DTL=y
CONFIG_PSERIES_ENERGY=y
CONFIG_IO_EVENT_IRQ=y
CONFIG_LPARCFG=y
CONFIG_PPC_SMLPAR=y
CONFIG_CMM=y
CONFIG_HV_PERF_CTRS=y
CONFIG_IBMVIO=y
# CONFIG_PAPR_SCM is not set
CONFIG_PPC_SVM=y
CONFIG_PPC_VAS=y
CONFIG_KVM_GUEST=y
CONFIG_EPAPR_PARAVIRT=y
CONFIG_PPC_HASH_MMU_NATIVE=y
CONFIG_PPC_OF_BOOT_TRAMPOLINE=y
CONFIG_PPC_DT_CPU_FTRS=y
# CONFIG_UDBG_RTAS_CONSOLE is not set
CONFIG_PPC_SMP_MUXED_IPI=y
CONFIG_MPIC=y
# CONFIG_MPIC_MSGR is not set
CONFIG_PPC_I8259=y
CONFIG_PPC_RTAS=y
CONFIG_RTAS_ERROR_LOGGING=y
CONFIG_PPC_RTAS_DAEMON=y
CONFIG_RTAS_PROC=y
CONFIG_RTAS_FLASH=m
CONFIG_EEH=y
CONFIG_PPC_P7_NAP=y
CONFIG_PPC_BOOK3S_IDLE=y
CONFIG_PPC_INDIRECT_PIO=y

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
# CONFIG_CPU_FREQ_STAT is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
# CONFIG_CPU_FREQ_GOV_SCHEDUTIL is not set

#
# CPU frequency scaling drivers
#
CONFIG_POWERNV_CPUFREQ=y
# end of CPU Frequency scaling

#
# CPUIdle driver
#

#
# CPU Idle
#
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_CPU_IDLE_GOV_TEO is not set
# CONFIG_CPU_IDLE_GOV_HALTPOLL is not set

#
# POWERPC CPU Idle Drivers
#
CONFIG_PSERIES_CPUIDLE=y
CONFIG_POWERNV_CPUIDLE=y
# end of POWERPC CPU Idle Drivers
# end of CPU Idle
# end of CPUIdle driver

# CONFIG_GEN_RTC is not set
# end of Platform support

#
# Kernel options
#
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_PPC_TRANSACTIONAL_MEM=y
CONFIG_HOTPLUG_CPU=y
CONFIG_PPC_QUEUED_SPINLOCKS=y
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_PPC64_SUPPORTS_MEMORY_FAILURE=y
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_ARCH_HAS_KEXEC_PURGATORY=y
CONFIG_RELOCATABLE=y
# CONFIG_RELOCATABLE_TEST is not set
CONFIG_CRASH_DUMP=y
CONFIG_FA_DUMP=y
# CONFIG_OPAL_CORE is not set
CONFIG_IRQ_ALL_CPUS=y
CONFIG_NUMA=y
CONFIG_NODES_SHIFT=8
CONFIG_HAVE_MEMORYLESS_NODES=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ILLEGAL_POINTER_VALUE=0x5deadbeef0000000
CONFIG_ARCH_MEMORY_PROBE=y
# CONFIG_PPC_4K_PAGES is not set
CONFIG_PPC_64K_PAGES=y
CONFIG_PPC_PAGE_SHIFT=16
CONFIG_THREAD_SHIFT=14
CONFIG_DATA_SHIFT=24
CONFIG_FORCE_MAX_ZONEORDER=9
# CONFIG_PPC_SUBPAGE_PROT is not set
# CONFIG_PPC_PROT_SAO_LPAR is not set
CONFIG_PPC_COPRO_BASE=y
CONFIG_SCHED_SMT=y
CONFIG_PPC_DENORMALISATION=y
CONFIG_CMDLINE=""
CONFIG_EXTRA_TARGETS=""
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_HIBERNATION is not set
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP_SMP_NONZERO_CPU=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
# CONFIG_ENERGY_MODEL is not set
CONFIG_PPC_MEM_KEYS=y
CONFIG_PPC_RTAS_FILTER=y
# end of Kernel options

CONFIG_ISA_DMA_API=y

#
# Bus options
#
CONFIG_GENERIC_ISA_DMA=y
# CONFIG_FSL_LBC is not set
# end of Bus options

CONFIG_NONSTATIC_KERNEL=y
CONFIG_PAGE_OFFSET=0xc000000000000000
CONFIG_KERNEL_START=0xc000000000000000
CONFIG_PHYSICAL_START=0x00000000
CONFIG_ARCH_RANDOM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_VFIO=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_HAVE_KVM_VCPU_ASYNC_IOCTL=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
CONFIG_KVM_BOOK3S_HANDLER=y
CONFIG_KVM_BOOK3S_64_HANDLER=y
CONFIG_KVM_BOOK3S_HV_POSSIBLE=y
CONFIG_KVM_BOOK3S_64=m
CONFIG_KVM_BOOK3S_64_HV=m
# CONFIG_KVM_BOOK3S_64_PR is not set
# CONFIG_KVM_BOOK3S_HV_EXIT_TIMING is not set
CONFIG_KVM_BOOK3S_HV_NESTED_PMU_WORKAROUND=y
CONFIG_KVM_XICS=y
CONFIG_KVM_XIVE=y

#
# General architecture-dependent options
#
CONFIG_CRASH_CORE=y
CONFIG_KEXEC_CORE=y
CONFIG_KEXEC_ELF=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
# CONFIG_STATIC_KEYS_SELFTEST is not set
CONFIG_OPTPROBES=y
CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y
CONFIG_HAVE_NMI=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_HAS_FORTIFY_SOURCE=y
CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_HAVE_ASM_MODVERSIONS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_RSEQ=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_NMI_WATCHDOG=y
CONFIG_HAVE_HARDLOCKUP_DETECTOR_ARCH=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
CONFIG_MMU_GATHER_TABLE_FREE=y
CONFIG_MMU_GATHER_RCU_TABLE_FREE=y
CONFIG_MMU_GATHER_PAGE_SIZE=y
CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_ARCH_WEAK_RELEASE_ACQUIRE=y
CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y
CONFIG_HAVE_ARCH_SECCOMP=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP=y
CONFIG_SECCOMP_FILTER=y
# CONFIG_SECCOMP_CACHE_DEBUG is not set
CONFIG_LTO_NONE=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING=y
CONFIG_ARCH_HAS_SCALED_CPUTIME=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_MOVE_PUD=y
CONFIG_HAVE_MOVE_PMD=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_HUGE_VMAP=y
CONFIG_HAVE_ARCH_HUGE_VMALLOC=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_ARCH_MMAP_RND_BITS=y
CONFIG_ARCH_MMAP_RND_BITS=14
CONFIG_PAGE_SIZE_LESS_THAN_256KB=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_HAVE_ARCH_NVRAM_OPS=y
CONFIG_CLONE_BACKWARDS=y
CONFIG_OLD_SIGSUSPEND=y
# CONFIG_COMPAT_32BIT_TIME is not set
CONFIG_ARCH_OPTIONAL_KERNEL_RWX=y
CONFIG_ARCH_OPTIONAL_KERNEL_RWX_DEFAULT=y
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y
CONFIG_STRICT_MODULE_RWX=y
CONFIG_ARCH_HAS_PHYS_TO_DMA=y
# CONFIG_LOCK_EVENT_COUNTS is not set
CONFIG_ARCH_HAS_MEM_ENCRYPT=y
CONFIG_ARCH_HAS_CC_PLATFORM=y
CONFIG_ARCH_WANT_LD_ORPHAN_WARN=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# end of GCOV-based kernel profiling
# end of General architecture-dependent options

CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_MODVERSIONS=y
CONFIG_ASM_MODVERSIONS=y
CONFIG_MODULE_REL_CRCS=y
CONFIG_MODULE_SRCVERSION_ALL=y
# CONFIG_MODULE_SIG is not set
CONFIG_MODULE_COMPRESS_NONE=y
# CONFIG_MODULE_COMPRESS_GZIP is not set
# CONFIG_MODULE_COMPRESS_XZ is not set
# CONFIG_MODULE_COMPRESS_ZSTD is not set
# CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS is not set
CONFIG_MODPROBE_PATH="/sbin/modprobe"
CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG_COMMON=y
CONFIG_BLK_DEV_BSGLIB=y
# CONFIG_BLK_DEV_INTEGRITY is not set
# CONFIG_BLK_DEV_ZONED is not set
# CONFIG_BLK_WBT is not set
CONFIG_BLK_DEBUG_FS=y
# CONFIG_BLK_SED_OPAL is not set
# CONFIG_BLK_INLINE_ENCRYPTION is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_AIX_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
# CONFIG_CMDLINE_PARTITION is not set
# end of Partition Types

CONFIG_BLK_MQ_PCI=y
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_BLK_MQ_RDMA=y
CONFIG_BLK_PM=y
CONFIG_BLOCK_HOLDER_DEPRECATED=y

#
# IO Schedulers
#
CONFIG_MQ_IOSCHED_DEADLINE=y
CONFIG_MQ_IOSCHED_KYBER=y
# CONFIG_IOSCHED_BFQ is not set
# end of IO Schedulers

CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_ASN1=m
CONFIG_UNINLINE_SPIN_UNLOCK=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
CONFIG_QUEUED_SPINLOCKS=y
CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
CONFIG_QUEUED_RWLOCKS=y
CONFIG_ARCH_HAS_MMIOWB=y
CONFIG_MMIOWB=y
CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE=y
CONFIG_FREEZER=y

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_ELFCORE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_SCRIPT=y
CONFIG_BINFMT_MISC=m
CONFIG_COREDUMP=y
# end of Executable file formats

#
# Memory Management options
#
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_FAST_GUP=y
CONFIG_ARCH_KEEP_MEMBLOCK=y
CONFIG_NUMA_KEEP_MEMINFO=y
CONFIG_MEMORY_ISOLATION=y
CONFIG_EXCLUSIVE_SYSTEM_RAM=y
CONFIG_HAVE_BOOTMEM_INFO_NODE=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG=y
# CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
CONFIG_MEMORY_BALLOON=y
CONFIG_BALLOON_COMPACTION=y
CONFIG_COMPACTION=y
CONFIG_PAGE_REPORTING=y
CONFIG_MIGRATION=y
CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
CONFIG_ARCH_ENABLE_THP_MIGRATION=y
CONFIG_HUGETLB_PAGE_SIZE_VARIABLE=y
CONFIG_CONTIG_ALLOC=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_MEMORY_FAILURE is not set
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_CMA=y
# CONFIG_CMA_DEBUG is not set
# CONFIG_CMA_DEBUGFS is not set
# CONFIG_CMA_SYSFS is not set
CONFIG_CMA_AREAS=19
# CONFIG_ZSWAP is not set
# CONFIG_ZPOOL is not set
# CONFIG_ZSMALLOC is not set
CONFIG_GENERIC_EARLY_IOREMAP=y
# CONFIG_DEFERRED_STRUCT_PAGE_INIT is not set
# CONFIG_IDLE_PAGE_TRACKING is not set
CONFIG_ARCH_HAS_PTE_DEVMAP=y
# CONFIG_ZONE_DEVICE is not set
CONFIG_HMM_MIRROR=y
CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y
CONFIG_ARCH_HAS_PKEYS=y
# CONFIG_PERCPU_STATS is not set
# CONFIG_GUP_TEST is not set
# CONFIG_READ_ONLY_THP_FOR_FS is not set
CONFIG_ARCH_HAS_PTE_SPECIAL=y
CONFIG_ARCH_HAS_HUGEPD=y
# CONFIG_ANON_VMA_NAME is not set

#
# Data Access Monitoring
#
# CONFIG_DAMON is not set
# end of Data Access Monitoring
# end of Memory Management options

CONFIG_NET=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y
CONFIG_SKB_EXTENSIONS=y

#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_DIAG is not set
CONFIG_UNIX=y
CONFIG_UNIX_SCM=y
CONFIG_AF_UNIX_OOB=y
# CONFIG_UNIX_DIAG is not set
# CONFIG_TLS is not set
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=m
CONFIG_XFRM_USER=m
# CONFIG_XFRM_INTERFACE is not set
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_AH=m
CONFIG_XFRM_ESP=m
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
# CONFIG_NET_KEY_MIGRATE is not set
# CONFIG_SMC is not set
# CONFIG_XDP_SOCKETS is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
# CONFIG_IP_PNP_RARP is not set
CONFIG_NET_IPIP=y
# CONFIG_NET_IPGRE_DEMUX is not set
CONFIG_NET_IP_TUNNEL=y
# CONFIG_IP_MROUTE is not set
CONFIG_SYN_COOKIES=y
# CONFIG_NET_IPVTI is not set
# CONFIG_NET_FOU is not set
# CONFIG_NET_FOU_IP_TUNNELS is not set
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
# CONFIG_INET_ESP_OFFLOAD is not set
# CONFIG_INET_ESPINTCP is not set
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_INET_UDP_DIAG is not set
# CONFIG_INET_RAW_DIAG is not set
# CONFIG_INET_DIAG_DESTROY is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=y
# CONFIG_IPV6_ROUTER_PREF is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
# CONFIG_INET6_AH is not set
# CONFIG_INET6_ESP is not set
# CONFIG_INET6_IPCOMP is not set
# CONFIG_IPV6_MIP6 is not set
# CONFIG_IPV6_ILA is not set
# CONFIG_IPV6_VTI is not set
CONFIG_IPV6_SIT=y
# CONFIG_IPV6_SIT_6RD is not set
CONFIG_IPV6_NDISC_NODETYPE=y
# CONFIG_IPV6_TUNNEL is not set
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
# CONFIG_IPV6_SEG6_LWTUNNEL is not set
# CONFIG_IPV6_SEG6_HMAC is not set
# CONFIG_IPV6_RPL_LWTUNNEL is not set
# CONFIG_IPV6_IOAM6_LWTUNNEL is not set
# CONFIG_MPTCP is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NET_PTP_CLASSIFY=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_ADVANCED is not set

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_INGRESS=y
CONFIG_NETFILTER_EGRESS=y
CONFIG_NETFILTER_SKIP_EGRESS=y
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_LOG_SYSLOG=m
CONFIG_NF_CONNTRACK_PROCFS=y
# CONFIG_NF_CONNTRACK_LABELS is not set
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_IRC=m
# CONFIG_NF_CONNTRACK_NETBIOS_NS is not set
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CT_NETLINK=m
# CONFIG_NETFILTER_NETLINK_GLUE_CT is not set
CONFIG_NF_NAT=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
CONFIG_NF_NAT_MASQUERADE=y
# CONFIG_NF_TABLES is not set
CONFIG_NETFILTER_XTABLES=m

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m

#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_LOG=m
CONFIG_NETFILTER_XT_NAT=m
# CONFIG_NETFILTER_XT_TARGET_NETMAP is not set
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
# CONFIG_NETFILTER_XT_TARGET_REDIRECT is not set
CONFIG_NETFILTER_XT_TARGET_MASQUERADE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
# end of Core Netfilter Configuration

# CONFIG_IP_SET is not set
# CONFIG_IP_VS is not set

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
# CONFIG_NF_SOCKET_IPV4 is not set
# CONFIG_NF_TPROXY_IPV4 is not set
# CONFIG_NF_DUP_IPV4 is not set
CONFIG_NF_LOG_ARP=m
CONFIG_NF_LOG_IPV4=m
CONFIG_NF_REJECT_IPV4=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_NAT=m
# CONFIG_IP_NF_TARGET_MASQUERADE is not set
CONFIG_IP_NF_MANGLE=m
# CONFIG_IP_NF_RAW is not set
# end of IP: Netfilter Configuration

#
# IPv6: Netfilter Configuration
#
# CONFIG_NF_SOCKET_IPV6 is not set
# CONFIG_NF_TPROXY_IPV6 is not set
# CONFIG_NF_DUP_IPV6 is not set
CONFIG_NF_REJECT_IPV6=m
CONFIG_NF_LOG_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
# CONFIG_IP6_NF_RAW is not set
# end of IPv6: Netfilter Configuration

CONFIG_NF_DEFRAG_IPV6=m
# CONFIG_NF_CONNTRACK_BRIDGE is not set
# CONFIG_BRIDGE_NF_EBTABLES is not set
# CONFIG_BPFILTER is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_RDS is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_L2TP is not set
CONFIG_STP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_BRIDGE_MRP is not set
# CONFIG_BRIDGE_CFM is not set
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
CONFIG_LLC=m
# CONFIG_LLC2 is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_6LOWPAN is not set
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
# CONFIG_NET_SCH_CBQ is not set
# CONFIG_NET_SCH_HTB is not set
# CONFIG_NET_SCH_HFSC is not set
# CONFIG_NET_SCH_PRIO is not set
# CONFIG_NET_SCH_MULTIQ is not set
# CONFIG_NET_SCH_RED is not set
# CONFIG_NET_SCH_SFB is not set
# CONFIG_NET_SCH_SFQ is not set
# CONFIG_NET_SCH_TEQL is not set
# CONFIG_NET_SCH_TBF is not set
# CONFIG_NET_SCH_CBS is not set
# CONFIG_NET_SCH_ETF is not set
# CONFIG_NET_SCH_TAPRIO is not set
# CONFIG_NET_SCH_GRED is not set
# CONFIG_NET_SCH_DSMARK is not set
# CONFIG_NET_SCH_NETEM is not set
# CONFIG_NET_SCH_DRR is not set
# CONFIG_NET_SCH_MQPRIO is not set
# CONFIG_NET_SCH_SKBPRIO is not set
# CONFIG_NET_SCH_CHOKE is not set
# CONFIG_NET_SCH_QFQ is not set
# CONFIG_NET_SCH_CODEL is not set
# CONFIG_NET_SCH_FQ_CODEL is not set
# CONFIG_NET_SCH_CAKE is not set
# CONFIG_NET_SCH_FQ is not set
# CONFIG_NET_SCH_HHF is not set
# CONFIG_NET_SCH_PIE is not set
# CONFIG_NET_SCH_INGRESS is not set
# CONFIG_NET_SCH_PLUG is not set
# CONFIG_NET_SCH_ETS is not set
# CONFIG_NET_SCH_DEFAULT is not set

#
# Classification
#
CONFIG_NET_CLS=y
# CONFIG_NET_CLS_BASIC is not set
# CONFIG_NET_CLS_TCINDEX is not set
# CONFIG_NET_CLS_ROUTE4 is not set
# CONFIG_NET_CLS_FW is not set
# CONFIG_NET_CLS_U32 is not set
# CONFIG_NET_CLS_RSVP is not set
# CONFIG_NET_CLS_RSVP6 is not set
# CONFIG_NET_CLS_FLOW is not set
# CONFIG_NET_CLS_CGROUP is not set
CONFIG_NET_CLS_BPF=m
# CONFIG_NET_CLS_FLOWER is not set
# CONFIG_NET_CLS_MATCHALL is not set
# CONFIG_NET_EMATCH is not set
CONFIG_NET_CLS_ACT=y
# CONFIG_NET_ACT_POLICE is not set
# CONFIG_NET_ACT_GACT is not set
# CONFIG_NET_ACT_MIRRED is not set
# CONFIG_NET_ACT_SAMPLE is not set
# CONFIG_NET_ACT_IPT is not set
# CONFIG_NET_ACT_NAT is not set
# CONFIG_NET_ACT_PEDIT is not set
# CONFIG_NET_ACT_SIMP is not set
# CONFIG_NET_ACT_SKBEDIT is not set
# CONFIG_NET_ACT_CSUM is not set
# CONFIG_NET_ACT_MPLS is not set
# CONFIG_NET_ACT_VLAN is not set
CONFIG_NET_ACT_BPF=m
# CONFIG_NET_ACT_SKBMOD is not set
# CONFIG_NET_ACT_IFE is not set
# CONFIG_NET_ACT_TUNNEL_KEY is not set
# CONFIG_NET_ACT_GATE is not set
# CONFIG_NET_TC_SKB_EXT is not set
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set
CONFIG_DNS_RESOLVER=y
# CONFIG_BATMAN_ADV is not set
# CONFIG_OPENVSWITCH is not set
# CONFIG_VSOCKETS is not set
# CONFIG_NETLINK_DIAG is not set
# CONFIG_MPLS is not set
# CONFIG_NET_NSH is not set
# CONFIG_HSR is not set
# CONFIG_NET_SWITCHDEV is not set
# CONFIG_NET_L3_MASTER_DEV is not set
# CONFIG_QRTR is not set
# CONFIG_NET_NCSI is not set
CONFIG_PCPU_DEV_REFCNT=y
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_SOCK_RX_QUEUE_MAPPING=y
CONFIG_XPS=y
# CONFIG_CGROUP_NET_PRIO is not set
# CONFIG_CGROUP_NET_CLASSID is not set
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
# CONFIG_BPF_STREAM_PARSER is not set
CONFIG_NET_FLOW_LIMIT=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NET_DROP_MONITOR is not set
# end of Network testing
# end of Networking options

# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_AF_KCM is not set
# CONFIG_MCTP is not set
CONFIG_WIRELESS=y
# CONFIG_CFG80211 is not set

#
# CFG80211 needs to be enabled for MAC80211
#
CONFIG_MAC80211_STA_HASH_MAX_SIZE=0
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
# CONFIG_NFC is not set
# CONFIG_PSAMPLE is not set
# CONFIG_NET_IFE is not set
# CONFIG_LWTUNNEL is not set
CONFIG_DST_CACHE=y
CONFIG_GRO_CELLS=y
CONFIG_NET_SELFTESTS=y
CONFIG_NET_SOCK_MSG=y
CONFIG_NET_DEVLINK=y
CONFIG_FAILOVER=m
CONFIG_ETHTOOL_NETLINK=y

#
# Device Drivers
#
CONFIG_HAVE_PCI=y
CONFIG_FORCE_PCI=y
CONFIG_PCI=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCI_SYSCALL=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_POWER_SUPERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
# CONFIG_PCIE_PTM is not set
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_IRQ_DOMAIN=y
CONFIG_PCI_MSI_ARCH_FALLBACKS=y
CONFIG_PCI_QUIRKS=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
# CONFIG_PCI_IOV is not set
# CONFIG_PCI_PRI is not set
# CONFIG_PCI_PASID is not set
CONFIG_HOTPLUG_PCI=y
# CONFIG_HOTPLUG_PCI_CPCI is not set
# CONFIG_HOTPLUG_PCI_SHPC is not set
# CONFIG_HOTPLUG_PCI_POWERNV is not set
CONFIG_HOTPLUG_PCI_RPA=m
CONFIG_HOTPLUG_PCI_RPA_DLPAR=m

#
# PCI controller drivers
#
# CONFIG_PCI_FTPCI100 is not set
# CONFIG_PCI_HOST_GENERIC is not set
# CONFIG_PCIE_XILINX is not set
# CONFIG_PCIE_MICROCHIP_HOST is not set

#
# DesignWare PCI Core Support
#
# CONFIG_PCIE_DW_PLAT_HOST is not set
# CONFIG_PCI_MESON is not set
# end of DesignWare PCI Core Support

#
# Mobiveil PCIe Core Support
#
# end of Mobiveil PCIe Core Support

#
# Cadence PCIe controllers support
#
# CONFIG_PCIE_CADENCE_PLAT_HOST is not set
# CONFIG_PCI_J721E_HOST is not set
# end of Cadence PCIe controllers support
# end of PCI controller drivers

#
# PCI Endpoint
#
# CONFIG_PCI_ENDPOINT is not set
# end of PCI Endpoint

#
# PCI switch controller drivers
#
# CONFIG_PCI_SW_SWITCHTEC is not set
# end of PCI switch controller drivers

# CONFIG_CXL_BUS is not set
CONFIG_PCCARD=y
CONFIG_PCMCIA=y
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

#
# PC-card bridges
#
# CONFIG_YENTA is not set
# CONFIG_PD6729 is not set
# CONFIG_I82092 is not set
# CONFIG_RAPIDIO is not set

#
# Generic Driver Options
#
CONFIG_AUXILIARY_BUS=y
# CONFIG_UEVENT_HELPER is not set
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
# CONFIG_DEVTMPFS_SAFE is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y

#
# Firmware loader
#
CONFIG_FW_LOADER=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_FW_LOADER_USER_HELPER is not set
# CONFIG_FW_LOADER_COMPRESS is not set
CONFIG_FW_CACHE=y
# end of Firmware loader

CONFIG_ALLOW_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_DEBUG_TEST_DRIVER_REMOVE is not set
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_GENERIC_CPU_VULNERABILITIES=y
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
CONFIG_DMA_SHARED_BUFFER=y
# CONFIG_DMA_FENCE_TRACE is not set
# end of Generic Driver Options

#
# Bus devices
#
# CONFIG_MHI_BUS is not set
# end of Bus devices

# CONFIG_CONNECTOR is not set

#
# Firmware Drivers
#

#
# ARM System Control and Management Interface Protocol
#
# end of ARM System Control and Management Interface Protocol

# CONFIG_GOOGLE_FIRMWARE is not set

#
# Tegra firmware driver
#
# end of Tegra firmware driver
# end of Firmware Drivers

# CONFIG_GNSS is not set
# CONFIG_MTD is not set
CONFIG_DTC=y
CONFIG_OF=y
# CONFIG_OF_UNITTEST is not set
CONFIG_OF_FLATTREE=y
CONFIG_OF_EARLY_FLATTREE=y
CONFIG_OF_KOBJ=y
CONFIG_OF_DYNAMIC=y
CONFIG_OF_ADDRESS=y
CONFIG_OF_IRQ=y
CONFIG_OF_RESERVED_MEM=y
# CONFIG_OF_OVERLAY is not set
CONFIG_OF_DMA_DEFAULT_COHERENT=y
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
# CONFIG_PARPORT is not set
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_NULL_BLK is not set
CONFIG_BLK_DEV_FD=y
CONFIG_CDROM=y
# CONFIG_BLK_DEV_PCIESSD_MTIP32XX is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
# CONFIG_BLK_DEV_DRBD is not set
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=65536
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_VIRTIO_BLK=m
# CONFIG_BLK_DEV_RBD is not set

#
# NVME Support
#
# CONFIG_BLK_DEV_NVME is not set
# CONFIG_NVME_RDMA is not set
# CONFIG_NVME_FC is not set
# CONFIG_NVME_TCP is not set
# end of NVME Support

#
# Misc devices
#
# CONFIG_AD525X_DPOT is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_IBMVMC is not set
# CONFIG_PHANTOM is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_HP_ILO is not set
# CONFIG_APDS9802ALS is not set
# CONFIG_ISL29003 is not set
# CONFIG_ISL29020 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_SENSORS_BH1770 is not set
# CONFIG_SENSORS_APDS990X is not set
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
# CONFIG_SRAM is not set
# CONFIG_DW_XDATA_PCIE is not set
# CONFIG_PCI_ENDPOINT_TEST is not set
# CONFIG_XILINX_SDFEC is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_LEGACY is not set
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_EEPROM_IDT_89HPESX is not set
# CONFIG_EEPROM_EE1004 is not set
# end of EEPROM support

# CONFIG_CB710_CORE is not set

#
# Texas Instruments shared transport line discipline
#
# end of Texas Instruments shared transport line discipline

# CONFIG_SENSORS_LIS3_I2C is not set
# CONFIG_ALTERA_STAPL is not set
# CONFIG_GENWQE is not set
# CONFIG_ECHO is not set
CONFIG_CXL_BASE=y
CONFIG_CXL=m
# CONFIG_BCM_VK is not set
# CONFIG_MISC_ALCOR_PCI is not set
# CONFIG_MISC_RTSX_PCI is not set
# CONFIG_MISC_RTSX_USB is not set
# CONFIG_HABANA_AI is not set
# CONFIG_UACCE is not set
# CONFIG_PVPANIC is not set
# end of Misc devices

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI_COMMON=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_BLK_DEV_BSG=y
# CONFIG_CHR_DEV_SCH is not set
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
# CONFIG_SCSI_SAS_LIBSAS is not set
CONFIG_SCSI_SRP_ATTRS=y
# end of SCSI Transports

CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
CONFIG_BE2ISCSI=m
CONFIG_CXLFLASH=m
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_HPSA is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_3W_SAS is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_MVUMI is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_SCSI_ESAS2R is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
CONFIG_SCSI_MPT3SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
CONFIG_SCSI_MPT3SAS_MAX_SGE=128
CONFIG_SCSI_MPT2SAS=m
# CONFIG_SCSI_MPI3MR is not set
# CONFIG_SCSI_SMARTPQI is not set
# CONFIG_SCSI_UFSHCD is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_MYRB is not set
# CONFIG_SCSI_MYRS is not set
# CONFIG_LIBFC is not set
# CONFIG_SCSI_SNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_FDOMAIN_PCI is not set
# CONFIG_SCSI_IPS is not set
CONFIG_SCSI_IBMVSCSI=y
CONFIG_SCSI_IBMVFC=m
CONFIG_SCSI_IBMVFC_TRACE=y
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_STEX is not set
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=0
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
CONFIG_SCSI_IPR=y
CONFIG_SCSI_IPR_TRACE=y
CONFIG_SCSI_IPR_DUMP=y
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA_FC=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_WD719X is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_PMCRAID is not set
# CONFIG_SCSI_PM8001 is not set
# CONFIG_SCSI_BFA_FC is not set
CONFIG_SCSI_VIRTIO=m
# CONFIG_SCSI_CHELSIO_FCOE is not set
# CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_RDAC=m
# CONFIG_SCSI_DH_HP_SW is not set
# CONFIG_SCSI_DH_EMC is not set
CONFIG_SCSI_DH_ALUA=m
# end of SCSI device support

CONFIG_ATA=y
CONFIG_SATA_HOST=y
CONFIG_PATA_TIMINGS=y
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_FORCE=y
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=y
CONFIG_SATA_MOBILE_LPM_POLICY=0
# CONFIG_SATA_AHCI_PLATFORM is not set
# CONFIG_AHCI_CEVA is not set
# CONFIG_AHCI_QORIQ is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_SATA_ACARD_AHCI is not set
CONFIG_SATA_SIL24=y
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_SX4 is not set
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#
# CONFIG_ATA_PIIX is not set
CONFIG_SATA_MV=y
# CONFIG_SATA_NV is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
CONFIG_SATA_SVW=y
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set

#
# PATA SFF controllers with BMDMA
#
# CONFIG_PATA_ALI is not set
CONFIG_PATA_AMD=y
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_ATP867X is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RDC is not set
# CONFIG_PATA_SCH is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_TOSHIBA is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set

#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_PCMCIA is not set
# CONFIG_PATA_PLATFORM is not set
# CONFIG_PATA_RZ1000 is not set

#
# Generic fallback / legacy drivers
#
CONFIG_ATA_GENERIC=y
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
# CONFIG_BCACHE is not set
CONFIG_BLK_DEV_DM_BUILTIN=y
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_DEBUG is not set
CONFIG_DM_BUFIO=m
# CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING is not set
# CONFIG_DM_UNSTRIPED is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
# CONFIG_DM_THIN_PROVISIONING is not set
# CONFIG_DM_CACHE is not set
# CONFIG_DM_WRITECACHE is not set
# CONFIG_DM_EBS is not set
# CONFIG_DM_ERA is not set
# CONFIG_DM_CLONE is not set
CONFIG_DM_MIRROR=m
# CONFIG_DM_LOG_USERSPACE is not set
# CONFIG_DM_RAID is not set
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
# CONFIG_DM_MULTIPATH_HST is not set
# CONFIG_DM_MULTIPATH_IOA is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_DUST is not set
# CONFIG_DM_INIT is not set
CONFIG_DM_UEVENT=y
# CONFIG_DM_FLAKEY is not set
# CONFIG_DM_VERITY is not set
# CONFIG_DM_SWITCH is not set
# CONFIG_DM_LOG_WRITES is not set
# CONFIG_DM_INTEGRITY is not set
# CONFIG_TARGET_CORE is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_FIREWIRE_NOSY is not set
# end of IEEE 1394 (FireWire) support

# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_MII=y
CONFIG_NET_CORE=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
# CONFIG_WIREGUARD is not set
# CONFIG_EQUALIZER is not set
# CONFIG_NET_FC is not set
# CONFIG_NET_TEAM is not set
# CONFIG_MACVLAN is not set
# CONFIG_IPVLAN is not set
# CONFIG_VXLAN is not set
# CONFIG_GENEVE is not set
# CONFIG_BAREUDP is not set
# CONFIG_GTP is not set
# CONFIG_AMT is not set
# CONFIG_MACSEC is not set
CONFIG_NETCONSOLE=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=m
# CONFIG_TUN_VNET_CROSS_LE is not set
# CONFIG_VETH is not set
CONFIG_VIRTIO_NET=m
# CONFIG_NLMON is not set
CONFIG_SUNGEM_PHY=y
# CONFIG_ARCNET is not set
CONFIG_ETHERNET=y
CONFIG_MDIO=m
CONFIG_NET_VENDOR_3COM=y
# CONFIG_PCMCIA_3C574 is not set
# CONFIG_PCMCIA_3C589 is not set
CONFIG_VORTEX=m
# CONFIG_TYPHOON is not set
CONFIG_NET_VENDOR_ADAPTEC=y
# CONFIG_ADAPTEC_STARFIRE is not set
CONFIG_NET_VENDOR_AGERE=y
# CONFIG_ET131X is not set
CONFIG_NET_VENDOR_ALACRITECH=y
# CONFIG_SLICOSS is not set
CONFIG_NET_VENDOR_ALTEON=y
CONFIG_ACENIC=m
CONFIG_ACENIC_OMIT_TIGON_I=y
# CONFIG_ALTERA_TSE is not set
CONFIG_NET_VENDOR_AMAZON=y
# CONFIG_ENA_ETHERNET is not set
CONFIG_NET_VENDOR_AMD=y
# CONFIG_AMD8111_ETH is not set
CONFIG_PCNET32=m
# CONFIG_PCMCIA_NMCLAN is not set
CONFIG_NET_VENDOR_AQUANTIA=y
# CONFIG_AQTION is not set
CONFIG_NET_VENDOR_ARC=y
CONFIG_NET_VENDOR_ASIX=y
CONFIG_NET_VENDOR_ATHEROS=y
# CONFIG_ATL2 is not set
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
# CONFIG_ALX is not set
CONFIG_NET_VENDOR_BROADCOM=y
# CONFIG_B44 is not set
# CONFIG_BCMGENET is not set
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_TIGON3=y
CONFIG_TIGON3_HWMON=y
CONFIG_BNX2X=m
# CONFIG_SYSTEMPORT is not set
# CONFIG_BNXT is not set
CONFIG_NET_VENDOR_BROCADE=y
# CONFIG_BNA is not set
CONFIG_NET_VENDOR_CADENCE=y
CONFIG_NET_VENDOR_CAVIUM=y
# CONFIG_THUNDER_NIC_PF is not set
# CONFIG_THUNDER_NIC_VF is not set
# CONFIG_THUNDER_NIC_BGX is not set
# CONFIG_THUNDER_NIC_RGX is not set
# CONFIG_CAVIUM_PTP is not set
# CONFIG_LIQUIDIO is not set
# CONFIG_LIQUIDIO_VF is not set
CONFIG_NET_VENDOR_CHELSIO=y
CONFIG_CHELSIO_T1=m
# CONFIG_CHELSIO_T1_1G is not set
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
# CONFIG_CHELSIO_T4VF is not set
CONFIG_CHELSIO_LIB=m
CONFIG_CHELSIO_INLINE_CRYPTO=y
CONFIG_NET_VENDOR_CISCO=y
# CONFIG_ENIC is not set
CONFIG_NET_VENDOR_CORTINA=y
# CONFIG_GEMINI_ETHERNET is not set
# CONFIG_DNET is not set
CONFIG_NET_VENDOR_DEC=y
# CONFIG_NET_TULIP is not set
CONFIG_NET_VENDOR_DLINK=y
# CONFIG_DL2K is not set
# CONFIG_SUNDANCE is not set
CONFIG_NET_VENDOR_EMULEX=y
CONFIG_BE2NET=m
CONFIG_BE2NET_HWMON=y
CONFIG_BE2NET_BE2=y
CONFIG_BE2NET_BE3=y
CONFIG_BE2NET_LANCER=y
CONFIG_BE2NET_SKYHAWK=y
CONFIG_NET_VENDOR_ENGLEDER=y
# CONFIG_TSNEP is not set
CONFIG_NET_VENDOR_EZCHIP=y
# CONFIG_EZCHIP_NPS_MANAGEMENT_ENET is not set
CONFIG_NET_VENDOR_FUJITSU=y
# CONFIG_PCMCIA_FMVJ18X is not set
CONFIG_NET_VENDOR_GOOGLE=y
# CONFIG_GVE is not set
CONFIG_NET_VENDOR_HUAWEI=y
CONFIG_NET_VENDOR_I825XX=y
CONFIG_NET_VENDOR_IBM=y
CONFIG_IBMVETH=m
CONFIG_IBMVNIC=m
CONFIG_NET_VENDOR_INTEL=y
CONFIG_E100=y
CONFIG_E1000=y
CONFIG_E1000E=y
# CONFIG_IGB is not set
# CONFIG_IGBVF is not set
CONFIG_IXGB=m
CONFIG_IXGBE=m
CONFIG_IXGBE_HWMON=y
# CONFIG_IXGBEVF is not set
CONFIG_I40E=m
# CONFIG_I40EVF is not set
# CONFIG_ICE is not set
# CONFIG_FM10K is not set
# CONFIG_IGC is not set
CONFIG_NET_VENDOR_MICROSOFT=y
# CONFIG_JME is not set
CONFIG_NET_VENDOR_LITEX=y
# CONFIG_LITEX_LITEETH is not set
CONFIG_NET_VENDOR_MARVELL=y
# CONFIG_MVMDIO is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX4_EN=m
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_MLX4_CORE_GEN2=y
# CONFIG_MLX5_CORE is not set
# CONFIG_MLXSW_CORE is not set
# CONFIG_MLXFW is not set
CONFIG_NET_VENDOR_MICREL=y
# CONFIG_KS8851_MLL is not set
# CONFIG_KSZ884X_PCI is not set
CONFIG_NET_VENDOR_MICROCHIP=y
# CONFIG_LAN743X is not set
CONFIG_NET_VENDOR_MICROSEMI=y
CONFIG_NET_VENDOR_MYRI=y
CONFIG_MYRI10GE=m
# CONFIG_FEALNX is not set
CONFIG_NET_VENDOR_NATSEMI=y
# CONFIG_NATSEMI is not set
# CONFIG_NS83820 is not set
CONFIG_NET_VENDOR_NETERION=y
CONFIG_S2IO=m
# CONFIG_VXGE is not set
CONFIG_NET_VENDOR_NETRONOME=y
# CONFIG_NFP is not set
CONFIG_NET_VENDOR_NI=y
# CONFIG_NI_XGE_MANAGEMENT_ENET is not set
CONFIG_NET_VENDOR_8390=y
# CONFIG_PCMCIA_AXNET is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_PCMCIA_PCNET is not set
CONFIG_NET_VENDOR_NVIDIA=y
# CONFIG_FORCEDETH is not set
CONFIG_NET_VENDOR_OKI=y
# CONFIG_ETHOC is not set
CONFIG_NET_VENDOR_PACKET_ENGINES=y
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
CONFIG_NET_VENDOR_PENSANDO=y
# CONFIG_IONIC is not set
CONFIG_NET_VENDOR_QLOGIC=y
# CONFIG_QLA3XXX is not set
# CONFIG_QLCNIC is not set
CONFIG_NETXEN_NIC=m
# CONFIG_QED is not set
CONFIG_NET_VENDOR_QUALCOMM=y
# CONFIG_QCOM_EMAC is not set
# CONFIG_RMNET is not set
CONFIG_NET_VENDOR_RDC=y
# CONFIG_R6040 is not set
CONFIG_NET_VENDOR_REALTEK=y
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_R8169 is not set
CONFIG_NET_VENDOR_RENESAS=y
CONFIG_NET_VENDOR_ROCKER=y
CONFIG_NET_VENDOR_SAMSUNG=y
# CONFIG_SXGBE_ETH is not set
CONFIG_NET_VENDOR_SEEQ=y
CONFIG_NET_VENDOR_SOLARFLARE=y
# CONFIG_SFC is not set
# CONFIG_SFC_FALCON is not set
CONFIG_NET_VENDOR_SILAN=y
# CONFIG_SC92031 is not set
CONFIG_NET_VENDOR_SIS=y
# CONFIG_SIS900 is not set
# CONFIG_SIS190 is not set
CONFIG_NET_VENDOR_SMSC=y
# CONFIG_PCMCIA_SMC91C92 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SMSC911X is not set
# CONFIG_SMSC9420 is not set
CONFIG_NET_VENDOR_SOCIONEXT=y
CONFIG_NET_VENDOR_STMICRO=y
# CONFIG_STMMAC_ETH is not set
CONFIG_NET_VENDOR_SUN=y
# CONFIG_HAPPYMEAL is not set
CONFIG_SUNGEM=y
# CONFIG_CASSINI is not set
# CONFIG_NIU is not set
CONFIG_NET_VENDOR_SYNOPSYS=y
# CONFIG_DWC_XLGMAC is not set
CONFIG_NET_VENDOR_TEHUTI=y
# CONFIG_TEHUTI is not set
CONFIG_NET_VENDOR_TI=y
# CONFIG_TI_CPSW_PHY_SEL is not set
# CONFIG_TLAN is not set
CONFIG_NET_VENDOR_VERTEXCOM=y
CONFIG_NET_VENDOR_VIA=y
# CONFIG_VIA_RHINE is not set
# CONFIG_VIA_VELOCITY is not set
CONFIG_NET_VENDOR_WIZNET=y
# CONFIG_WIZNET_W5100 is not set
# CONFIG_WIZNET_W5300 is not set
CONFIG_NET_VENDOR_XILINX=y
# CONFIG_XILINX_EMACLITE is not set
# CONFIG_XILINX_AXI_EMAC is not set
# CONFIG_XILINX_LL_TEMAC is not set
CONFIG_NET_VENDOR_XIRCOM=y
# CONFIG_PCMCIA_XIRC2PS is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PHYLIB=y
CONFIG_SWPHY=y
CONFIG_FIXED_PHY=y

#
# MII PHY device drivers
#
# CONFIG_AMD_PHY is not set
# CONFIG_ADIN_PHY is not set
# CONFIG_AQUANTIA_PHY is not set
# CONFIG_AX88796B_PHY is not set
CONFIG_BROADCOM_PHY=m
# CONFIG_BCM54140_PHY is not set
# CONFIG_BCM7XXX_PHY is not set
# CONFIG_BCM84881_PHY is not set
# CONFIG_BCM87XX_PHY is not set
CONFIG_BCM_NET_PHYLIB=m
# CONFIG_CICADA_PHY is not set
# CONFIG_CORTINA_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_ICPLUS_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_INTEL_XWAY_PHY is not set
# CONFIG_LSI_ET1011C_PHY is not set
CONFIG_MARVELL_PHY=y
# CONFIG_MARVELL_10G_PHY is not set
# CONFIG_MARVELL_88X2222_PHY is not set
# CONFIG_MAXLINEAR_GPHY is not set
# CONFIG_MEDIATEK_GE_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_MICROCHIP_PHY is not set
# CONFIG_MICROCHIP_T1_PHY is not set
# CONFIG_MICROSEMI_PHY is not set
# CONFIG_MOTORCOMM_PHY is not set
# CONFIG_NATIONAL_PHY is not set
# CONFIG_NXP_C45_TJA11XX_PHY is not set
# CONFIG_NXP_TJA11XX_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_REALTEK_PHY is not set
# CONFIG_RENESAS_PHY is not set
# CONFIG_ROCKCHIP_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_TERANETICS_PHY is not set
# CONFIG_DP83822_PHY is not set
# CONFIG_DP83TC811_PHY is not set
# CONFIG_DP83848_PHY is not set
# CONFIG_DP83867_PHY is not set
# CONFIG_DP83869_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_XILINX_GMII2RGMII is not set
CONFIG_MDIO_DEVICE=y
CONFIG_MDIO_BUS=y
CONFIG_FWNODE_MDIO=y
CONFIG_OF_MDIO=y
CONFIG_MDIO_DEVRES=y
# CONFIG_MDIO_BITBANG is not set
# CONFIG_MDIO_BCM_UNIMAC is not set
# CONFIG_MDIO_HISI_FEMAC is not set
# CONFIG_MDIO_MVUSB is not set
# CONFIG_MDIO_OCTEON is not set
# CONFIG_MDIO_THUNDER is not set

#
# MDIO Multiplexers
#
# CONFIG_MDIO_BUS_MUX_MULTIPLEXER is not set
# CONFIG_MDIO_BUS_MUX_MMIOREG is not set

#
# PCS device drivers
#
# CONFIG_PCS_XPCS is not set
# end of PCS device drivers

CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
# CONFIG_PPP_FILTER is not set
# CONFIG_PPP_MPPE is not set
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPPOE=m
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
# CONFIG_SLIP is not set
CONFIG_SLHC=m
CONFIG_USB_NET_DRIVERS=y
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_RTL8152 is not set
# CONFIG_USB_LAN78XX is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_IPHETH is not set
CONFIG_WLAN=y
CONFIG_WLAN_VENDOR_ADMTEK=y
CONFIG_WLAN_VENDOR_ATH=y
# CONFIG_ATH_DEBUG is not set
# CONFIG_ATH5K_PCI is not set
CONFIG_WLAN_VENDOR_ATMEL=y
CONFIG_WLAN_VENDOR_BROADCOM=y
CONFIG_WLAN_VENDOR_CISCO=y
CONFIG_WLAN_VENDOR_INTEL=y
CONFIG_WLAN_VENDOR_INTERSIL=y
# CONFIG_HOSTAP is not set
CONFIG_WLAN_VENDOR_MARVELL=y
CONFIG_WLAN_VENDOR_MEDIATEK=y
CONFIG_WLAN_VENDOR_MICROCHIP=y
CONFIG_WLAN_VENDOR_RALINK=y
CONFIG_WLAN_VENDOR_REALTEK=y
CONFIG_WLAN_VENDOR_RSI=y
CONFIG_WLAN_VENDOR_ST=y
CONFIG_WLAN_VENDOR_TI=y
CONFIG_WLAN_VENDOR_ZYDAS=y
CONFIG_WLAN_VENDOR_QUANTENNA=y
# CONFIG_PCMCIA_RAYCS is not set
# CONFIG_WAN is not set

#
# Wireless WAN
#
# CONFIG_WWAN is not set
# end of Wireless WAN

# CONFIG_NETDEVSIM is not set
CONFIG_NET_FAILOVER=m
# CONFIG_ISDN is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_LEDS=m
CONFIG_INPUT_FF_MEMLESS=y
# CONFIG_INPUT_SPARSEKMAP is not set
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1050 is not set
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_DLINK_DIR685 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_LM8333 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_OMAP4 is not set
# CONFIG_KEYBOARD_TM2_TOUCHKEY is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_CAP11XX is not set
# CONFIG_KEYBOARD_CYPRESS_SF is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_BYD=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_SYNAPTICS_SMBUS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_PS2_FOCALTECH=y
CONFIG_MOUSE_PS2_SMBUS=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_CYAPA is not set
# CONFIG_MOUSE_ELAN_I2C is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
# CONFIG_MOUSE_SYNAPTICS_USB is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_ATMEL_CAPTOUCH is not set
# CONFIG_INPUT_BMA150 is not set
# CONFIG_INPUT_E3X0_BUTTON is not set
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_MMA8450 is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_KXTJ9 is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_DA7280_HAPTICS is not set
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_IMS_PCU is not set
# CONFIG_INPUT_IQS269A is not set
# CONFIG_INPUT_IQS626A is not set
# CONFIG_INPUT_CMA3000 is not set
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
# CONFIG_INPUT_DRV2665_HAPTICS is not set
# CONFIG_INPUT_DRV2667_HAPTICS is not set
# CONFIG_RMI4_CORE is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_SERIO_XILINX_XPS_PS2 is not set
# CONFIG_SERIO_ALTERA_PS2 is not set
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_SERIO_ARC_PS2 is not set
# CONFIG_SERIO_APBPS2 is not set
# CONFIG_USERIO is not set
# CONFIG_GAMEPORT is not set
# end of Hardware I/O ports
# end of Input device support

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_LDISC_AUTOLOAD=y

#
# Serial drivers
#
CONFIG_SERIAL_EARLYCON=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_DEPRECATED_OPTIONS=y
CONFIG_SERIAL_8250_16550A_VARIANTS=y
# CONFIG_SERIAL_8250_FINTEK is not set
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_EXAR=y
# CONFIG_SERIAL_8250_CS is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set
CONFIG_SERIAL_8250_FSL=y
# CONFIG_SERIAL_8250_DW is not set
# CONFIG_SERIAL_8250_RT288X is not set
CONFIG_SERIAL_8250_PERICOM=y
# CONFIG_SERIAL_OF_PLATFORM is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_ICOM=m
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_SIFIVE is not set
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_SC16IS7XX is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_XILINX_PS_UART is not set
# CONFIG_SERIAL_ARC is not set
# CONFIG_SERIAL_RP2 is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_FSL_LINFLEXUART is not set
# CONFIG_SERIAL_CONEXANT_DIGICOLOR is not set
# end of Serial drivers

# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_PPC_EPAPR_HV_BYTECHAN is not set
# CONFIG_N_GSM is not set
# CONFIG_NOZOMI is not set
# CONFIG_NULL_TTY is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_CONSOLE=y
# CONFIG_HVC_OLD_HVSI is not set
CONFIG_HVC_OPAL=y
CONFIG_HVC_RTAS=y
# CONFIG_HVC_UDBG is not set
CONFIG_HVCS=m
# CONFIG_SERIAL_DEV_BUS is not set
CONFIG_VIRTIO_CONSOLE=y
CONFIG_IBM_BSR=m
CONFIG_POWERNV_OP_PANEL=m
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=m
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
# CONFIG_HW_RANDOM_BA431 is not set
# CONFIG_HW_RANDOM_VIRTIO is not set
CONFIG_HW_RANDOM_PSERIES=m
CONFIG_HW_RANDOM_POWERNV=m
# CONFIG_HW_RANDOM_CCTRNG is not set
# CONFIG_HW_RANDOM_XIPHERA is not set
# CONFIG_APPLICOM is not set

#
# PCMCIA character devices
#
# CONFIG_SYNCLINK_CS is not set
# CONFIG_CARDMAN_4000 is not set
# CONFIG_CARDMAN_4040 is not set
# CONFIG_SCR24X is not set
# CONFIG_IPWIRELESS is not set
# end of PCMCIA character devices

CONFIG_DEVMEM=y
CONFIG_NVRAM=y
CONFIG_DEVPORT=y
# CONFIG_HANGCHECK_TIMER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_XILLYBUS is not set
# CONFIG_XILLYUSB is not set
# CONFIG_RANDOM_TRUST_CPU is not set
# CONFIG_RANDOM_TRUST_BOOTLOADER is not set
# end of Character devices

#
# I2C support
#
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=y
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
CONFIG_I2C_AMD8111=y
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_ISCH is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_NVIDIA_GPU is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_DESIGNWARE_PLATFORM is not set
# CONFIG_I2C_DESIGNWARE_PCI is not set
# CONFIG_I2C_MPC is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set

#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_DIOLAN_U2C is not set
# CONFIG_I2C_CP2615 is not set
# CONFIG_I2C_ROBOTFUZZ_OSIF is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Other I2C/SMBus bus drivers
#
CONFIG_I2C_OPAL=y
# CONFIG_I2C_VIRTIO is not set
# end of I2C Hardware Bus support

# CONFIG_I2C_STUB is not set
# CONFIG_I2C_SLAVE is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# end of I2C support

# CONFIG_I3C is not set
# CONFIG_SPI is not set
# CONFIG_SPMI is not set
# CONFIG_HSI is not set
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set
# CONFIG_NTP_PPS is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
# CONFIG_PPS_CLIENT_LDISC is not set
# CONFIG_PPS_CLIENT_GPIO is not set

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=y
CONFIG_PTP_1588_CLOCK_OPTIONAL=y

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# CONFIG_PTP_1588_CLOCK_IDT82P33 is not set
# CONFIG_PTP_1588_CLOCK_IDTCM is not set
# end of PTP clock support

# CONFIG_PINCTRL is not set
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
# CONFIG_POWER_RESET is not set
# CONFIG_POWER_SUPPLY is not set
CONFIG_HWMON=y
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM1177 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7410 is not set
# CONFIG_SENSORS_ADT7411 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_AHT10 is not set
# CONFIG_SENSORS_AQUACOMPUTER_D5NEXT is not set
# CONFIG_SENSORS_AS370 is not set
# CONFIG_SENSORS_ASC7621 is not set
# CONFIG_SENSORS_AXI_FAN_CONTROL is not set
# CONFIG_SENSORS_ASPEED is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_CORSAIR_CPRO is not set
# CONFIG_SENSORS_CORSAIR_PSU is not set
# CONFIG_SENSORS_DRIVETEMP is not set
# CONFIG_SENSORS_DS620 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_G760A is not set
# CONFIG_SENSORS_G762 is not set
# CONFIG_SENSORS_HIH6130 is not set
CONFIG_SENSORS_IBMPOWERNV=y
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_POWR1220 is not set
# CONFIG_SENSORS_LINEAGE is not set
# CONFIG_SENSORS_LTC2945 is not set
# CONFIG_SENSORS_LTC2947_I2C is not set
# CONFIG_SENSORS_LTC2990 is not set
# CONFIG_SENSORS_LTC4151 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4222 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LTC4260 is not set
# CONFIG_SENSORS_LTC4261 is not set
# CONFIG_SENSORS_MAX127 is not set
# CONFIG_SENSORS_MAX16065 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX1668 is not set
# CONFIG_SENSORS_MAX197 is not set
# CONFIG_SENSORS_MAX31730 is not set
# CONFIG_SENSORS_MAX6620 is not set
# CONFIG_SENSORS_MAX6621 is not set
# CONFIG_SENSORS_MAX6639 is not set
# CONFIG_SENSORS_MAX6642 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_MAX6697 is not set
# CONFIG_SENSORS_MAX31790 is not set
# CONFIG_SENSORS_MCP3021 is not set
# CONFIG_SENSORS_TC654 is not set
# CONFIG_SENSORS_TPS23861 is not set
# CONFIG_SENSORS_MR75203 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM73 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LM95234 is not set
# CONFIG_SENSORS_LM95241 is not set
# CONFIG_SENSORS_LM95245 is not set
# CONFIG_SENSORS_NCT7802 is not set
# CONFIG_SENSORS_NPCM7XX is not set
# CONFIG_SENSORS_NZXT_KRAKEN2 is not set
# CONFIG_SENSORS_NZXT_SMART2 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_PMBUS is not set
# CONFIG_SENSORS_SBTSI is not set
# CONFIG_SENSORS_SBRMI is not set
# CONFIG_SENSORS_SHT21 is not set
# CONFIG_SENSORS_SHT3x is not set
# CONFIG_SENSORS_SHT4x is not set
# CONFIG_SENSORS_SHTC1 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_EMC1403 is not set
# CONFIG_SENSORS_EMC2103 is not set
# CONFIG_SENSORS_EMC6W201 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_STTS751 is not set
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_ADC128D818 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_AMC6821 is not set
# CONFIG_SENSORS_INA209 is not set
# CONFIG_SENSORS_INA2XX is not set
# CONFIG_SENSORS_INA238 is not set
# CONFIG_SENSORS_INA3221 is not set
# CONFIG_SENSORS_TC74 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP102 is not set
# CONFIG_SENSORS_TMP103 is not set
# CONFIG_SENSORS_TMP108 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_TMP421 is not set
# CONFIG_SENSORS_TMP513 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83773G is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83795 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
# CONFIG_THERMAL is not set
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
CONFIG_BCMA_POSSIBLE=y
# CONFIG_BCMA is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_ACT8945A is not set
# CONFIG_MFD_AS3711 is not set
# CONFIG_MFD_AS3722 is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_ATMEL_FLEXCOM is not set
# CONFIG_MFD_ATMEL_HLCDC is not set
# CONFIG_MFD_BCM590XX is not set
# CONFIG_MFD_BD9571MWV is not set
# CONFIG_MFD_AXP20X_I2C is not set
# CONFIG_MFD_MADERA is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_MFD_DA9055 is not set
# CONFIG_MFD_DA9062 is not set
# CONFIG_MFD_DA9063 is not set
# CONFIG_MFD_DA9150 is not set
# CONFIG_MFD_DLN2 is not set
# CONFIG_MFD_GATEWORKS_GSC is not set
# CONFIG_MFD_MC13XXX_I2C is not set
# CONFIG_MFD_MP2629 is not set
# CONFIG_MFD_HI6421_PMIC is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_LPC_ICH is not set
# CONFIG_LPC_SCH is not set
# CONFIG_MFD_IQS62X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_88PM800 is not set
# CONFIG_MFD_88PM805 is not set
# CONFIG_MFD_88PM860X is not set
# CONFIG_MFD_MAX14577 is not set
# CONFIG_MFD_MAX77620 is not set
# CONFIG_MFD_MAX77650 is not set
# CONFIG_MFD_MAX77686 is not set
# CONFIG_MFD_MAX77693 is not set
# CONFIG_MFD_MAX77843 is not set
# CONFIG_MFD_MAX8907 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_MT6360 is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_MENF21BMC is not set
# CONFIG_MFD_VIPERBOARD is not set
# CONFIG_MFD_NTXEC is not set
# CONFIG_MFD_RETU is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_RT4831 is not set
# CONFIG_MFD_RT5033 is not set
# CONFIG_MFD_RC5T583 is not set
# CONFIG_MFD_RK808 is not set
# CONFIG_MFD_RN5T618 is not set
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_SI476X_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_MFD_SKY81452 is not set
# CONFIG_MFD_STMPE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_LP3943 is not set
# CONFIG_MFD_LP8788 is not set
# CONFIG_MFD_TI_LMU is not set
# CONFIG_MFD_PALMAS is not set
# CONFIG_TPS6105X is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TPS65086 is not set
# CONFIG_MFD_TPS65090 is not set
# CONFIG_MFD_TPS65217 is not set
# CONFIG_MFD_TI_LP873X is not set
# CONFIG_MFD_TI_LP87565 is not set
# CONFIG_MFD_TPS65218 is not set
# CONFIG_MFD_TPS6586X is not set
# CONFIG_MFD_TPS65912_I2C is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_TWL6040_CORE is not set
# CONFIG_MFD_WL1273_CORE is not set
# CONFIG_MFD_LM3533 is not set
# CONFIG_MFD_TC3589X is not set
# CONFIG_MFD_TQMX86 is not set
# CONFIG_MFD_VX855 is not set
# CONFIG_MFD_LOCHNAGAR is not set
# CONFIG_MFD_ARIZONA_I2C is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
# CONFIG_MFD_ROHM_BD718XX is not set
# CONFIG_MFD_ROHM_BD71828 is not set
# CONFIG_MFD_ROHM_BD957XMUF is not set
# CONFIG_MFD_STPMIC1 is not set
# CONFIG_MFD_STMFX is not set
# CONFIG_MFD_ATC260X_I2C is not set
# CONFIG_MFD_QCOM_PM8008 is not set
# CONFIG_MFD_RSMU_I2C is not set
# end of Multifunction device drivers

# CONFIG_REGULATOR is not set
# CONFIG_RC_CORE is not set

#
# CEC support
#
# CONFIG_MEDIA_CEC_SUPPORT is not set
# end of CEC support

# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
# CONFIG_AGP is not set
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
# CONFIG_DRM is not set

#
# ARM devices
#
# end of ARM devices

#
# Frame buffer Devices
#
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_DDC=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_MACMODES=y
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
CONFIG_FB_OF=y
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_OPENCORES is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I740 is not set
CONFIG_FB_MATROX=y
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
CONFIG_FB_RADEON=y
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_RADEON_BACKLIGHT=y
# CONFIG_FB_RADEON_DEBUG is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_SMSCUFX is not set
# CONFIG_FB_UDL is not set
CONFIG_FB_IBM_GXT4500=y
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_SIMPLE is not set
# CONFIG_FB_SM712 is not set
# end of Frame buffer Devices

#
# Backlight & LCD device support
#
CONFIG_LCD_CLASS_DEVICE=y
# CONFIG_LCD_PLATFORM is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_QCOM_WLED is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_ADP8870 is not set
# CONFIG_BACKLIGHT_LM3639 is not set
# CONFIG_BACKLIGHT_LV5207LP is not set
# CONFIG_BACKLIGHT_BD6107 is not set
# CONFIG_BACKLIGHT_ARCXCNN is not set
# CONFIG_BACKLIGHT_LED is not set
# end of Backlight & LCD device support

#
# Console display driver support
#
# CONFIG_VGA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_DUMMY_CONSOLE_COLUMNS=80
CONFIG_DUMMY_CONSOLE_ROWS=25
CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_LEGACY_ACCELERATION is not set
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER is not set
# end of Console display driver support

CONFIG_LOGO=y
CONFIG_LOGO_LINUX_MONO=y
CONFIG_LOGO_LINUX_VGA16=y
CONFIG_LOGO_LINUX_CLUT224=y
# end of Graphics support

CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_SEQ_DEVICE=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_PCM_TIMER=y
# CONFIG_SND_HRTIMER is not set
# CONFIG_SND_DYNAMIC_MINORS is not set
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_PROC_FS=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_SEQUENCER_OSS=m
CONFIG_SND_SEQ_MIDI_EVENT=m
CONFIG_SND_DRIVERS=y
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_ALOOP is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CTXFI is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_LOLA is not set
# CONFIG_SND_LX6464ES is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SE6X is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set

#
# HD-Audio
#
# CONFIG_SND_HDA_INTEL is not set
# end of HD-Audio

CONFIG_SND_HDA_PREALLOC_SIZE=64
CONFIG_SND_PPC=y
CONFIG_SND_USB=y
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_UA101 is not set
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
# CONFIG_SND_USB_6FIRE is not set
# CONFIG_SND_USB_HIFACE is not set
# CONFIG_SND_BCD2000 is not set
# CONFIG_SND_USB_POD is not set
# CONFIG_SND_USB_PODHD is not set
# CONFIG_SND_USB_TONEPORT is not set
# CONFIG_SND_USB_VARIAX is not set
CONFIG_SND_PCMCIA=y
# CONFIG_SND_VXPOCKET is not set
# CONFIG_SND_PDAUDIOCF is not set
# CONFIG_SND_SOC is not set
# CONFIG_SND_VIRTIO is not set

#
# HID support
#
CONFIG_HID=y
# CONFIG_HID_BATTERY_STRENGTH is not set
# CONFIG_HIDRAW is not set
# CONFIG_UHID is not set
CONFIG_HID_GENERIC=y

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
# CONFIG_HID_ACCUTOUCH is not set
# CONFIG_HID_ACRUX is not set
CONFIG_HID_APPLE=y
# CONFIG_HID_APPLEIR is not set
# CONFIG_HID_ASUS is not set
# CONFIG_HID_AUREAL is not set
CONFIG_HID_BELKIN=y
# CONFIG_HID_BETOP_FF is not set
# CONFIG_HID_BIGBEN_FF is not set
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
# CONFIG_HID_CORSAIR is not set
# CONFIG_HID_COUGAR is not set
# CONFIG_HID_MACALLY is not set
# CONFIG_HID_PRODIKEYS is not set
# CONFIG_HID_CMEDIA is not set
# CONFIG_HID_CREATIVE_SB0540 is not set
CONFIG_HID_CYPRESS=y
# CONFIG_HID_DRAGONRISE is not set
# CONFIG_HID_EMS_FF is not set
# CONFIG_HID_ELAN is not set
# CONFIG_HID_ELECOM is not set
# CONFIG_HID_ELO is not set
CONFIG_HID_EZKEY=y
# CONFIG_HID_GEMBIRD is not set
# CONFIG_HID_GFRM is not set
# CONFIG_HID_GLORIOUS is not set
# CONFIG_HID_HOLTEK is not set
# CONFIG_HID_VIVALDI is not set
# CONFIG_HID_GT683R is not set
# CONFIG_HID_KEYTOUCH is not set
# CONFIG_HID_KYE is not set
# CONFIG_HID_UCLOGIC is not set
# CONFIG_HID_WALTOP is not set
# CONFIG_HID_VIEWSONIC is not set
# CONFIG_HID_XIAOMI is not set
CONFIG_HID_GYRATION=y
# CONFIG_HID_ICADE is not set
CONFIG_HID_ITE=y
# CONFIG_HID_JABRA is not set
# CONFIG_HID_TWINHAN is not set
CONFIG_HID_KENSINGTON=y
# CONFIG_HID_LCPOWER is not set
# CONFIG_HID_LED is not set
# CONFIG_HID_LENOVO is not set
# CONFIG_HID_LETSKETCH is not set
CONFIG_HID_LOGITECH=m
# CONFIG_HID_LOGITECH_HIDPP is not set
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
# CONFIG_LOGIG940_FF is not set
# CONFIG_LOGIWHEELS_FF is not set
# CONFIG_HID_MAGICMOUSE is not set
# CONFIG_HID_MALTRON is not set
# CONFIG_HID_MAYFLASH is not set
CONFIG_HID_REDRAGON=y
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
# CONFIG_HID_MULTITOUCH is not set
# CONFIG_HID_NINTENDO is not set
# CONFIG_HID_NTI is not set
# CONFIG_HID_NTRIG is not set
# CONFIG_HID_ORTEK is not set
CONFIG_HID_PANTHERLORD=y
# CONFIG_PANTHERLORD_FF is not set
# CONFIG_HID_PENMOUNT is not set
CONFIG_HID_PETALYNX=y
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_PLANTRONICS is not set
# CONFIG_HID_PRIMAX is not set
# CONFIG_HID_RETRODE is not set
# CONFIG_HID_ROCCAT is not set
# CONFIG_HID_SAITEK is not set
CONFIG_HID_SAMSUNG=y
# CONFIG_HID_SEMITEK is not set
# CONFIG_HID_SONY is not set
# CONFIG_HID_SPEEDLINK is not set
# CONFIG_HID_STEAM is not set
# CONFIG_HID_STEELSERIES is not set
CONFIG_HID_SUNPLUS=y
# CONFIG_HID_RMI is not set
# CONFIG_HID_GREENASIA is not set
# CONFIG_HID_SMARTJOYPLUS is not set
# CONFIG_HID_TIVO is not set
# CONFIG_HID_TOPSEED is not set
# CONFIG_HID_THINGM is not set
# CONFIG_HID_THRUSTMASTER is not set
# CONFIG_HID_UDRAW_PS3 is not set
# CONFIG_HID_U2FZERO is not set
# CONFIG_HID_WACOM is not set
# CONFIG_HID_WIIMOTE is not set
# CONFIG_HID_XINMO is not set
# CONFIG_HID_ZEROPLUS is not set
# CONFIG_HID_ZYDACRON is not set
# CONFIG_HID_SENSOR_HUB is not set
# CONFIG_HID_ALPS is not set
# end of Special HID drivers

#
# USB HID support
#
CONFIG_USB_HID=y
# CONFIG_HID_PID is not set
CONFIG_USB_HIDDEV=y
# end of USB HID support

#
# I2C HID support
#
# CONFIG_I2C_HID_OF is not set
# CONFIG_I2C_HID_OF_GOODIX is not set
# end of I2C HID support
# end of HID support

CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
# CONFIG_USB_ULPI_BUS is not set
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_PCI=y
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEFAULT_PERSIST=y
# CONFIG_USB_FEW_INIT_RETRIES is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_OTG is not set
# CONFIG_USB_OTG_PRODUCTLIST is not set
CONFIG_USB_AUTOSUSPEND_DELAY=2
CONFIG_USB_MON=m

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
CONFIG_USB_EHCI_HCD=y
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=y
# CONFIG_USB_EHCI_FSL is not set
# CONFIG_USB_EHCI_HCD_PPC_OF is not set
# CONFIG_USB_EHCI_HCD_PLATFORM is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_FOTG210_HCD is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_HCD_PPC_OF_BE is not set
# CONFIG_USB_OHCI_HCD_PPC_OF_LE is not set
CONFIG_USB_OHCI_HCD_PCI=y
# CONFIG_USB_OHCI_HCD_PLATFORM is not set
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_HCD_TEST_MODE is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_REALTEK is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_STORAGE_ENE_UB6250 is not set
# CONFIG_USB_UAS is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USBIP_CORE is not set
# CONFIG_USB_CDNS_SUPPORT is not set
# CONFIG_USB_MUSB_HDRC is not set
# CONFIG_USB_DWC3 is not set
# CONFIG_USB_DWC2 is not set
# CONFIG_USB_CHIPIDEA is not set
# CONFIG_USB_ISP1760 is not set

#
# USB port drivers
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
CONFIG_USB_APPLEDISPLAY=m
# CONFIG_APPLE_MFI_FASTCHARGE is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_EHSET_TEST_FIXTURE is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_YUREX is not set
# CONFIG_USB_EZUSB_FX2 is not set
# CONFIG_USB_HUB_USB251XB is not set
# CONFIG_USB_HSIC_USB3503 is not set
# CONFIG_USB_HSIC_USB4604 is not set
# CONFIG_USB_LINK_LAYER_TEST is not set
# CONFIG_USB_CHAOSKEY is not set

#
# USB Physical Layer drivers
#
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_USB_ISP1301 is not set
# end of USB Physical Layer drivers

# CONFIG_USB_GADGET is not set
# CONFIG_TYPEC is not set
# CONFIG_USB_ROLE_SWITCH is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=m
# CONFIG_LEDS_CLASS_FLASH is not set
# CONFIG_LEDS_CLASS_MULTICOLOR is not set
# CONFIG_LEDS_BRIGHTNESS_HW_CHANGED is not set

#
# LED drivers
#
# CONFIG_LEDS_AN30259A is not set
# CONFIG_LEDS_AW2013 is not set
# CONFIG_LEDS_BCM6328 is not set
# CONFIG_LEDS_BCM6358 is not set
# CONFIG_LEDS_LM3530 is not set
# CONFIG_LEDS_LM3532 is not set
# CONFIG_LEDS_LM3642 is not set
# CONFIG_LEDS_LM3692X is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_LP50XX is not set
# CONFIG_LEDS_LP55XX_COMMON is not set
# CONFIG_LEDS_LP8860 is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_PCA963X is not set
# CONFIG_LEDS_BD2802 is not set
# CONFIG_LEDS_TCA6507 is not set
# CONFIG_LEDS_TLC591XX is not set
# CONFIG_LEDS_LM355x is not set
# CONFIG_LEDS_IS31FL319X is not set
# CONFIG_LEDS_IS31FL32XX is not set

#
# LED driver for blink(1) USB RGB LED is under Special HID drivers (HID_THINGM)
#
# CONFIG_LEDS_BLINKM is not set
CONFIG_LEDS_POWERNV=m
# CONFIG_LEDS_MLXREG is not set
# CONFIG_LEDS_USER is not set
# CONFIG_LEDS_TI_LMU_COMMON is not set

#
# Flash and Torch LED drivers
#

#
# LED Triggers
#
# CONFIG_LEDS_TRIGGERS is not set

#
# Simple LED drivers
#
# CONFIG_ACCESSIBILITY is not set
CONFIG_INFINIBAND=m
CONFIG_INFINIBAND_USER_MAD=m
CONFIG_INFINIBAND_USER_ACCESS=m
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ON_DEMAND_PAGING=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_VIRT_DMA=y
CONFIG_INFINIBAND_MTHCA=m
CONFIG_INFINIBAND_MTHCA_DEBUG=y
CONFIG_INFINIBAND_CXGB4=m
# CONFIG_INFINIBAND_EFA is not set
CONFIG_MLX4_INFINIBAND=m
# CONFIG_INFINIBAND_OCRDMA is not set
# CONFIG_RDMA_RXE is not set
# CONFIG_RDMA_SIW is not set
CONFIG_INFINIBAND_IPOIB=m
CONFIG_INFINIBAND_IPOIB_CM=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
# CONFIG_INFINIBAND_IPOIB_DEBUG_DATA is not set
CONFIG_INFINIBAND_SRP=m
CONFIG_INFINIBAND_ISER=m
# CONFIG_INFINIBAND_RTRS_CLIENT is not set
# CONFIG_INFINIBAND_RTRS_SERVER is not set
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
CONFIG_EDAC=y
CONFIG_EDAC_LEGACY_SYSFS=y
# CONFIG_EDAC_DEBUG is not set
# CONFIG_EDAC_CPC925 is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
CONFIG_RTC_SYSTOHC=y
CONFIG_RTC_SYSTOHC_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set
CONFIG_RTC_NVMEM=y

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_ABB5ZES3 is not set
# CONFIG_RTC_DRV_ABEOZ9 is not set
# CONFIG_RTC_DRV_ABX80X is not set
CONFIG_RTC_DRV_DS1307=y
# CONFIG_RTC_DRV_DS1307_CENTURY is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_HYM8563 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
# CONFIG_RTC_DRV_RS5C372 is not set
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_ISL12022 is not set
# CONFIG_RTC_DRV_ISL12026 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF8523 is not set
# CONFIG_RTC_DRV_PCF85063 is not set
# CONFIG_RTC_DRV_PCF85363 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_BQ32K is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8010 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set
# CONFIG_RTC_DRV_EM3027 is not set
# CONFIG_RTC_DRV_RV3028 is not set
# CONFIG_RTC_DRV_RV3032 is not set
# CONFIG_RTC_DRV_RV8803 is not set
# CONFIG_RTC_DRV_SD3078 is not set

#
# SPI RTC drivers
#
CONFIG_RTC_I2C_AND_SPI=y

#
# SPI and I2C RTC drivers
#
# CONFIG_RTC_DRV_DS3232 is not set
# CONFIG_RTC_DRV_PCF2127 is not set
# CONFIG_RTC_DRV_RV3029C2 is not set
# CONFIG_RTC_DRV_RX6110 is not set

#
# Platform RTC drivers
#
# CONFIG_RTC_DRV_CMOS is not set
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1685_FAMILY is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_DS2404 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_MSM6242 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_RP5C01 is not set
# CONFIG_RTC_DRV_V3020 is not set
CONFIG_RTC_DRV_OPAL=y
# CONFIG_RTC_DRV_ZYNQMP is not set

#
# on-CPU RTC drivers
#
# CONFIG_RTC_DRV_GENERIC is not set
# CONFIG_RTC_DRV_CADENCE is not set
# CONFIG_RTC_DRV_FTRTC010 is not set
# CONFIG_RTC_DRV_R7301 is not set

#
# HID Sensor RTC drivers
#
# CONFIG_RTC_DRV_GOLDFISH is not set
# CONFIG_DMADEVICES is not set

#
# DMABUF options
#
# CONFIG_SYNC_FILE is not set
# CONFIG_UDMABUF is not set
# CONFIG_DMABUF_MOVE_NOTIFY is not set
# CONFIG_DMABUF_DEBUG is not set
# CONFIG_DMABUF_SELFTESTS is not set
# CONFIG_DMABUF_HEAPS is not set
# CONFIG_DMABUF_SYSFS_STATS is not set
# end of DMABUF options

# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=m
# CONFIG_UIO_CIF is not set
# CONFIG_UIO_PDRV_GENIRQ is not set
# CONFIG_UIO_DMEM_GENIRQ is not set
# CONFIG_UIO_AEC is not set
# CONFIG_UIO_SERCOS3 is not set
# CONFIG_UIO_PCI_GENERIC is not set
# CONFIG_UIO_NETX is not set
# CONFIG_UIO_PRUSS is not set
# CONFIG_UIO_MF624 is not set
# CONFIG_VFIO is not set
CONFIG_IRQ_BYPASS_MANAGER=y
# CONFIG_VIRT_DRIVERS is not set
CONFIG_VIRTIO=y
CONFIG_VIRTIO_PCI_LIB=y
CONFIG_VIRTIO_PCI_LIB_LEGACY=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
# CONFIG_VIRTIO_PMEM is not set
CONFIG_VIRTIO_BALLOON=m
# CONFIG_VIRTIO_INPUT is not set
# CONFIG_VIRTIO_MMIO is not set
# CONFIG_VDPA is not set
CONFIG_VHOST_IOTLB=m
CONFIG_VHOST=m
CONFIG_VHOST_MENU=y
CONFIG_VHOST_NET=m
# CONFIG_VHOST_CROSS_ENDIAN_LEGACY is not set

#
# Microsoft Hyper-V guest support
#
# end of Microsoft Hyper-V guest support

# CONFIG_GREYBUS is not set
# CONFIG_COMEDI is not set
# CONFIG_STAGING is not set
# CONFIG_GOLDFISH is not set
# CONFIG_COMMON_CLK is not set
# CONFIG_HWSPINLOCK is not set

#
# Clock Source drivers
#
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
# CONFIG_MICROCHIP_PIT64B is not set
# end of Clock Source drivers

# CONFIG_MAILBOX is not set
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y

#
# Generic IOMMU Pagetable Support
#
# end of Generic IOMMU Pagetable Support

# CONFIG_IOMMU_DEBUGFS is not set
CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
# CONFIG_IOMMU_DEFAULT_DMA_LAZY is not set
# CONFIG_IOMMU_DEFAULT_PASSTHROUGH is not set
CONFIG_OF_IOMMU=y
CONFIG_SPAPR_TCE_IOMMU=y

#
# Remoteproc drivers
#
# CONFIG_REMOTEPROC is not set
# end of Remoteproc drivers

#
# Rpmsg drivers
#
# CONFIG_RPMSG_VIRTIO is not set
# end of Rpmsg drivers

# CONFIG_SOUNDWIRE is not set

#
# SOC (System On Chip) specific Drivers
#

#
# Amlogic SoC drivers
#
# end of Amlogic SoC drivers

#
# Broadcom SoC drivers
#
# end of Broadcom SoC drivers

#
# NXP/Freescale QorIQ SoC drivers
#
# CONFIG_QUICC_ENGINE is not set
# end of NXP/Freescale QorIQ SoC drivers

#
# i.MX SoC drivers
#
# end of i.MX SoC drivers

#
# Enable LiteX SoC Builder specific drivers
#
# CONFIG_LITEX_SOC_CONTROLLER is not set
# end of Enable LiteX SoC Builder specific drivers

#
# Qualcomm SoC drivers
#
# end of Qualcomm SoC drivers

# CONFIG_SOC_TI is not set

#
# Xilinx SoC drivers
#
# end of Xilinx SoC drivers
# end of SOC (System On Chip) specific Drivers

# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
# CONFIG_NTB is not set
# CONFIG_VME_BUS is not set
# CONFIG_PWM is not set

#
# IRQ chip support
#
CONFIG_IRQCHIP=y
# CONFIG_AL_FIC is not set
# end of IRQ chip support

# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set

#
# PHY Subsystem
#
CONFIG_GENERIC_PHY=y
# CONFIG_PHY_CAN_TRANSCEIVER is not set

#
# PHY drivers for Broadcom platforms
#
# CONFIG_BCM_KONA_USB2_PHY is not set
# end of PHY drivers for Broadcom platforms

# CONFIG_PHY_CADENCE_DPHY is not set
# CONFIG_PHY_CADENCE_SALVO is not set
# CONFIG_PHY_FSL_IMX8MQ_USB is not set
# CONFIG_PHY_MIXEL_MIPI_DPHY is not set
# CONFIG_PHY_FSL_IMX8M_PCIE is not set
# CONFIG_PHY_PXA_28NM_HSIC is not set
# CONFIG_PHY_PXA_28NM_USB2 is not set
# end of PHY Subsystem

# CONFIG_POWERCAP is not set
# CONFIG_MCB is not set

#
# Performance monitor support
#
# end of Performance monitor support

CONFIG_RAS=y
# CONFIG_USB4 is not set

#
# Android
#
# CONFIG_ANDROID is not set
# end of Android

CONFIG_LIBNVDIMM=y
CONFIG_BLK_DEV_PMEM=y
CONFIG_ND_BLK=y
CONFIG_ND_CLAIM=y
CONFIG_ND_BTT=y
CONFIG_BTT=y
CONFIG_OF_PMEM=y
CONFIG_DAX=y
# CONFIG_DEV_DAX is not set
CONFIG_NVMEM=y
CONFIG_NVMEM_SYSFS=y
# CONFIG_NVMEM_RMEM is not set

#
# HW tracing support
#
# CONFIG_STM is not set
# CONFIG_INTEL_TH is not set
# end of HW tracing support

# CONFIG_FPGA is not set
# CONFIG_FSI is not set
# CONFIG_SIOX is not set
# CONFIG_SLIMBUS is not set
# CONFIG_INTERCONNECT is not set
# CONFIG_COUNTER is not set
# end of Device Drivers

#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
# CONFIG_VALIDATE_FS_PARSER is not set
CONFIG_FS_IOMAP=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT3_FS is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
CONFIG_XFS_FS=y
CONFIG_XFS_SUPPORT_V4=y
# CONFIG_XFS_QUOTA is not set
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
# CONFIG_XFS_ONLINE_SCRUB is not set
# CONFIG_XFS_WARN is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_GFS2_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
# CONFIG_BTRFS_DEBUG is not set
# CONFIG_BTRFS_ASSERT is not set
# CONFIG_BTRFS_FS_REF_VERIFY is not set
CONFIG_NILFS2_FS=m
# CONFIG_F2FS_FS is not set
CONFIG_FS_DAX=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
# CONFIG_EXPORTFS_BLOCK_OPS is not set
CONFIG_FILE_LOCKING=y
# CONFIG_FS_ENCRYPTION is not set
# CONFIG_FS_VERITY is not set
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
# CONFIG_QUOTA is not set
CONFIG_AUTOFS4_FS=m
CONFIG_AUTOFS_FS=m
CONFIG_FUSE_FS=m
# CONFIG_CUSE is not set
# CONFIG_VIRTIO_FS is not set
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

#
# Caches
#
# CONFIG_FSCACHE is not set
# end of Caches

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
# CONFIG_JOLIET is not set
# CONFIG_ZISOFS is not set
CONFIG_UDF_FS=m
# end of CD-ROM/DVD Filesystems

#
# DOS/FAT/EXFAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_FAT_DEFAULT_UTF8 is not set
# CONFIG_EXFAT_FS is not set
# CONFIG_NTFS_FS is not set
# CONFIG_NTFS3_FS is not set
# end of DOS/FAT/EXFAT/NT Filesystems

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
# CONFIG_PROC_VMCORE_DEVICE_DUMP is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
# CONFIG_PROC_CHILDREN is not set
CONFIG_KERNFS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
# CONFIG_TMPFS_INODE64 is not set
CONFIG_ARCH_SUPPORTS_HUGETLBFS=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_MEMFD_CREATE=y
CONFIG_ARCH_HAS_GIGANTIC_PAGE=y
# CONFIG_CONFIGFS_FS is not set
# end of Pseudo filesystems

CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ORANGEFS_FS is not set
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=m
CONFIG_CRAMFS_BLOCKDEV=y
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_FILE_CACHE=y
# CONFIG_SQUASHFS_FILE_DIRECT is not set
CONFIG_SQUASHFS_DECOMP_SINGLE=y
# CONFIG_SQUASHFS_DECOMP_MULTI is not set
# CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU is not set
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_ZLIB=y
# CONFIG_SQUASHFS_LZ4 is not set
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_ZSTD is not set
# CONFIG_SQUASHFS_4K_DEVBLK_SIZE is not set
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX6FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_PSTORE is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_EROFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
CONFIG_ROOT_NFS=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFS_DISABLE_UDP_SUPPORT=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_BLOCKLAYOUT is not set
# CONFIG_NFSD_SCSILAYOUT is not set
# CONFIG_NFSD_FLEXFILELAYOUT is not set
CONFIG_GRACE_PERIOD=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
# CONFIG_SUNRPC_DEBUG is not set
CONFIG_SUNRPC_XPRT_RDMA=m
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=m
CONFIG_CIFS_STATS2=y
CONFIG_CIFS_ALLOW_INSECURE_LEGACY=y
# CONFIG_CIFS_UPCALL is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_CIFS_DEBUG=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DEBUG_DUMP_KEYS is not set
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_CIFS_SWN_UPCALL is not set
# CONFIG_CIFS_SMB_DIRECT is not set
# CONFIG_SMB_SERVER is not set
CONFIG_SMBFS_COMMON=m
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_MAC_ROMAN is not set
# CONFIG_NLS_MAC_CELTIC is not set
# CONFIG_NLS_MAC_CENTEURO is not set
# CONFIG_NLS_MAC_CROATIAN is not set
# CONFIG_NLS_MAC_CYRILLIC is not set
# CONFIG_NLS_MAC_GAELIC is not set
# CONFIG_NLS_MAC_GREEK is not set
# CONFIG_NLS_MAC_ICELAND is not set
# CONFIG_NLS_MAC_INUIT is not set
# CONFIG_NLS_MAC_ROMANIAN is not set
# CONFIG_NLS_MAC_TURKISH is not set
CONFIG_NLS_UTF8=y
# CONFIG_UNICODE is not set
CONFIG_IO_WQ=y
# end of File systems

#
# Security options
#
CONFIG_KEYS=y
# CONFIG_KEYS_REQUEST_CACHE is not set
# CONFIG_PERSISTENT_KEYRINGS is not set
# CONFIG_ENCRYPTED_KEYS is not set
# CONFIG_KEY_DH_OPERATIONS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y
# CONFIG_HARDENED_USERCOPY is not set
# CONFIG_STATIC_USERMODEHELPER is not set
CONFIG_DEFAULT_SECURITY_DAC=y
CONFIG_LSM="landlock,lockdown,yama,loadpin,safesetid,integrity,bpf"

#
# Kernel hardening options
#

#
# Memory initialization
#
CONFIG_CC_HAS_AUTO_VAR_INIT_PATTERN=y
CONFIG_CC_HAS_AUTO_VAR_INIT_ZERO=y
# CONFIG_INIT_STACK_NONE is not set
# CONFIG_INIT_STACK_ALL_PATTERN is not set
CONFIG_INIT_STACK_ALL_ZERO=y
# CONFIG_INIT_ON_ALLOC_DEFAULT_ON is not set
# CONFIG_INIT_ON_FREE_DEFAULT_ON is not set
# end of Memory initialization
# end of Kernel hardening options
# end of Security options

CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_SKCIPHER=m
CONFIG_CRYPTO_SKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=m
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_RNG_DEFAULT=m
CONFIG_CRYPTO_AKCIPHER2=y
CONFIG_CRYPTO_KPP2=y
CONFIG_CRYPTO_ACOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_USER is not set
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_NULL2=y
# CONFIG_CRYPTO_PCRYPT is not set
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m

#
# Public-key cryptography
#
# CONFIG_CRYPTO_RSA is not set
# CONFIG_CRYPTO_DH is not set
# CONFIG_CRYPTO_ECDH is not set
# CONFIG_CRYPTO_ECDSA is not set
# CONFIG_CRYPTO_ECRDSA is not set
# CONFIG_CRYPTO_SM2 is not set
# CONFIG_CRYPTO_CURVE25519 is not set

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
# CONFIG_CRYPTO_CHACHA20POLY1305 is not set
# CONFIG_CRYPTO_AEGIS128 is not set
CONFIG_CRYPTO_SEQIV=m
CONFIG_CRYPTO_ECHAINIV=m

#
# Block modes
#
CONFIG_CRYPTO_CBC=m
# CONFIG_CRYPTO_CFB is not set
CONFIG_CRYPTO_CTR=m
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=m
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_OFB is not set
CONFIG_CRYPTO_PCBC=m
# CONFIG_CRYPTO_XTS is not set
# CONFIG_CRYPTO_KEYWRAP is not set
# CONFIG_CRYPTO_ADIANTUM is not set
CONFIG_CRYPTO_ESSIV=m

#
# Hash modes
#
CONFIG_CRYPTO_CMAC=m
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set
# CONFIG_CRYPTO_VMAC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_VPMSUM=m
# CONFIG_CRYPTO_CRC32 is not set
CONFIG_CRYPTO_XXHASH=m
CONFIG_CRYPTO_BLAKE2B=m
# CONFIG_CRYPTO_BLAKE2S is not set
CONFIG_CRYPTO_CRCT10DIF=m
# CONFIG_CRYPTO_CRCT10DIF_VPMSUM is not set
CONFIG_CRYPTO_GHASH=m
# CONFIG_CRYPTO_POLY1305 is not set
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_MD5_PPC=m
CONFIG_CRYPTO_MICHAEL_MIC=m
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_SHA1 is not set
CONFIG_CRYPTO_SHA1_PPC=m
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=m
# CONFIG_CRYPTO_SHA3 is not set
# CONFIG_CRYPTO_SM3 is not set
# CONFIG_CRYPTO_STREEBOG is not set
CONFIG_CRYPTO_WP512=m

#
# Ciphers
#
CONFIG_CRYPTO_AES=m
# CONFIG_CRYPTO_AES_TI is not set
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
# CONFIG_CRYPTO_CAMELLIA is not set
CONFIG_CRYPTO_CAST_COMMON=m
# CONFIG_CRYPTO_CAST5 is not set
CONFIG_CRYPTO_CAST6=m
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_CHACHA20 is not set
CONFIG_CRYPTO_SERPENT=m
# CONFIG_CRYPTO_SM4 is not set
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=m
# CONFIG_CRYPTO_842 is not set
# CONFIG_CRYPTO_LZ4 is not set
# CONFIG_CRYPTO_LZ4HC is not set
# CONFIG_CRYPTO_ZSTD is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
CONFIG_CRYPTO_DRBG_MENU=m
CONFIG_CRYPTO_DRBG_HMAC=y
# CONFIG_CRYPTO_DRBG_HASH is not set
# CONFIG_CRYPTO_DRBG_CTR is not set
CONFIG_CRYPTO_DRBG=m
CONFIG_CRYPTO_JITTERENTROPY=m
# CONFIG_CRYPTO_USER_API_HASH is not set
# CONFIG_CRYPTO_USER_API_SKCIPHER is not set
# CONFIG_CRYPTO_USER_API_RNG is not set
# CONFIG_CRYPTO_USER_API_AEAD is not set
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_NX=y
CONFIG_CRYPTO_DEV_NX_COMPRESS=y
CONFIG_CRYPTO_DEV_NX_COMPRESS_PSERIES=y
CONFIG_CRYPTO_DEV_NX_COMPRESS_POWERNV=y
# CONFIG_CRYPTO_DEV_ATMEL_ECC is not set
# CONFIG_CRYPTO_DEV_ATMEL_SHA204A is not set
# CONFIG_CRYPTO_DEV_NITROX_CNN55XX is not set
CONFIG_CRYPTO_DEV_VMX=y
CONFIG_CRYPTO_DEV_VMX_ENCRYPT=m
# CONFIG_CRYPTO_DEV_CHELSIO is not set
# CONFIG_CRYPTO_DEV_VIRTIO is not set
# CONFIG_CRYPTO_DEV_SAFEXCEL is not set
# CONFIG_CRYPTO_DEV_CCREE is not set
# CONFIG_CRYPTO_DEV_AMLOGIC_GXL is not set
# CONFIG_ASYMMETRIC_KEY_TYPE is not set

#
# Certificates for signature checking
#
# CONFIG_SYSTEM_BLACKLIST_KEYRING is not set
# end of Certificates for signature checking

CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_RAID6_PQ_BENCHMARK=y
# CONFIG_PACKING is not set
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
# CONFIG_CORDIC is not set
# CONFIG_PRIME_NUMBERS is not set
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_ARCH_HAS_FAST_MULTIPLIER=y

#
# Crypto library routines
#
CONFIG_CRYPTO_LIB_AES=m
CONFIG_CRYPTO_LIB_BLAKE2S_GENERIC=y
# CONFIG_CRYPTO_LIB_CHACHA is not set
# CONFIG_CRYPTO_LIB_CURVE25519 is not set
CONFIG_CRYPTO_LIB_POLY1305_RSIZE=1
# CONFIG_CRYPTO_LIB_POLY1305 is not set
# CONFIG_CRYPTO_LIB_CHACHA20POLY1305 is not set
CONFIG_CRYPTO_LIB_SHA256=y
# end of Crypto library routines

CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=m
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC64 is not set
# CONFIG_CRC4 is not set
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
# CONFIG_CRC8 is not set
CONFIG_XXHASH=y
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_842_DECOMPRESS=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=m
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_DECOMPRESS=y
CONFIG_ZSTD_COMPRESS=m
CONFIG_ZSTD_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
# CONFIG_XZ_DEC_MICROLZMA is not set
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_DECOMPRESS_LZ4=y
CONFIG_DECOMPRESS_ZSTD=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_BTREE=y
CONFIG_INTERVAL_TREE=y
CONFIG_XARRAY_MULTI=y
CONFIG_ASSOCIATIVE_ARRAY=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT_MAP=y
CONFIG_HAS_DMA=y
CONFIG_DMA_OPS=y
CONFIG_DMA_OPS_BYPASS=y
CONFIG_ARCH_HAS_DMA_MAP_DIRECT=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DMA_DECLARE_COHERENT=y
CONFIG_ARCH_HAS_FORCE_DMA_UNENCRYPTED=y
CONFIG_SWIOTLB=y
# CONFIG_DMA_RESTRICTED_POOL is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_DMA_MAP_BENCHMARK is not set
CONFIG_SGL_ALLOC=y
CONFIG_IOMMU_HELPER=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_GLOB=y
# CONFIG_GLOB_SELFTEST is not set
CONFIG_NLATTR=y
CONFIG_IRQ_POLL=y
CONFIG_DIMLIB=y
CONFIG_LIBFDT=y
CONFIG_OID_REGISTRY=y
CONFIG_HAVE_GENERIC_VDSO=y
CONFIG_GENERIC_GETTIMEOFDAY=y
CONFIG_GENERIC_VDSO_TIME_NS=y
CONFIG_FONT_SUPPORT=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_SG_POOL=y
CONFIG_ARCH_HAS_PMEM_API=y
CONFIG_MEMREGION=y
CONFIG_ARCH_HAS_MEMREMAP_COMPAT_ALIGN=y
CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE=y
CONFIG_ARCH_HAS_COPY_MC=y
CONFIG_ARCH_STACKWALK=y
CONFIG_SBITMAP=y
# end of Library routines

#
# Kernel hacking
#

#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
CONFIG_PRINTK_CALLER=y
# CONFIG_STACKTRACE_BUILD_ID is not set
CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
CONFIG_CONSOLE_LOGLEVEL_QUIET=4
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DYNAMIC_DEBUG_CORE is not set
CONFIG_SYMBOLIC_ERRNAME=y
CONFIG_DEBUG_BUGVERBOSE=y
# end of printk and dmesg options

#
# Compile-time checks and compiler options
#
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
# CONFIG_DEBUG_INFO_BTF is not set
# CONFIG_GDB_SCRIPTS is not set
CONFIG_FRAME_WARN=2048
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_HEADERS_INSTALL is not set
CONFIG_SECTION_MISMATCH_WARN_ONLY=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# end of Compile-time checks and compiler options

#
# Generic Kernel Debugging Instruments
#
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_MAGIC_SYSRQ_SERIAL=y
CONFIG_MAGIC_SYSRQ_SERIAL_SEQUENCE=""
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_FS_ALLOW_ALL=y
# CONFIG_DEBUG_FS_DISALLOW_MOUNT is not set
# CONFIG_DEBUG_FS_ALLOW_NONE is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_ARCH_HAS_UBSAN_SANITIZE_ALL=y
# CONFIG_UBSAN is not set
CONFIG_HAVE_KCSAN_COMPILER=y
# end of Generic Kernel Debugging Instruments

CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_MISC=y

#
# Networking Debugging
#
# CONFIG_NET_DEV_REFCNT_TRACKER is not set
# CONFIG_NET_NS_REFCNT_TRACKER is not set
# end of Networking Debugging

#
# Memory Debugging
#
# CONFIG_PAGE_EXTENSION is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_PAGE_OWNER is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_PAGE_REF is not set
# CONFIG_DEBUG_RODATA_TEST is not set
CONFIG_ARCH_HAS_DEBUG_WX=y
# CONFIG_DEBUG_WX is not set
CONFIG_GENERIC_PTDUMP=y
# CONFIG_PTDUMP_DEBUGFS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
CONFIG_DEBUG_STACK_USAGE=y
# CONFIG_SCHED_STACK_END_CHECK is not set
CONFIG_ARCH_HAS_DEBUG_VM_PGTABLE=y
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VM_PGTABLE is not set
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
# CONFIG_DEBUG_VIRTUAL is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_CC_HAS_WORKING_NOSANITIZE_ADDRESS=y
# end of Memory Debugging

# CONFIG_DEBUG_SHIRQ is not set

#
# Debug Oops, Lockups and Hangs
#
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_LOCKUP_DETECTOR=y
CONFIG_SOFTLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_HARDLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_HARDLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
# CONFIG_WQ_WATCHDOG is not set
# CONFIG_TEST_LOCKUP is not set
# end of Debug Oops, Lockups and Hangs

#
# Scheduler Debugging
#
CONFIG_SCHED_DEBUG=y
CONFIG_SCHED_INFO=y
# CONFIG_SCHEDSTATS is not set
# end of Scheduler Debugging

# CONFIG_DEBUG_TIMEKEEPING is not set
CONFIG_DEBUG_PREEMPT=y

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_LOCK_DEBUGGING_SUPPORT=y
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_DEBUG_SPINLOCK is not set
CONFIG_DEBUG_MUTEXES=y
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
# CONFIG_DEBUG_RWSEMS is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_LOCK_TORTURE_TEST is not set
# CONFIG_WW_MUTEX_SELFTEST is not set
# CONFIG_SCF_TORTURE_TEST is not set
# CONFIG_CSD_LOCK_WAIT_DEBUG is not set
# end of Lock Debugging (spinlocks, mutexes, etc...)

# CONFIG_DEBUG_IRQFLAGS is not set
CONFIG_STACKTRACE=y
# CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set
# CONFIG_DEBUG_KOBJECT is not set

#
# Debug kernel data structures
#
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_PLIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_BUG_ON_DATA_CORRUPTION is not set
# end of Debug kernel data structures

# CONFIG_DEBUG_CREDENTIALS is not set

#
# RCU Debugging
#
CONFIG_TORTURE_TEST=y
# CONFIG_RCU_SCALE_TEST is not set
CONFIG_RCU_TORTURE_TEST=y
# CONFIG_RCU_REF_SCALE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=21
CONFIG_RCU_TRACE=y
# CONFIG_RCU_EQS_DEBUG is not set
# end of RCU Debugging

# CONFIG_DEBUG_WQ_FORCE_RR_CPU is not set
# CONFIG_CPU_HOTPLUG_STATE_CONTROL is not set
# CONFIG_LATENCYTOP is not set
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_BOOTTIME_TRACING is not set
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
# CONFIG_FUNCTION_PROFILER is not set
CONFIG_STACK_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_PREEMPT_TRACER is not set
CONFIG_SCHED_TRACER=y
# CONFIG_HWLAT_TRACER is not set
# CONFIG_OSNOISE_TRACER is not set
# CONFIG_TIMERLAT_TRACER is not set
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
# CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENTS=y
# CONFIG_KPROBE_EVENTS_ON_NOTRACE is not set
CONFIG_UPROBE_EVENTS=y
CONFIG_BPF_EVENTS=y
CONFIG_DYNAMIC_EVENTS=y
CONFIG_PROBE_EVENTS=y
# CONFIG_BPF_KPROBE_OVERRIDE is not set
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE_MCOUNT_USE_RECORDMCOUNT=y
# CONFIG_SYNTH_EVENTS is not set
# CONFIG_HIST_TRIGGERS is not set
# CONFIG_TRACE_EVENT_INJECT is not set
# CONFIG_TRACEPOINT_BENCHMARK is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_TRACE_EVAL_MAP_FILE is not set
# CONFIG_FTRACE_RECORD_RECURSION is not set
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_RING_BUFFER_STARTUP_TEST is not set
# CONFIG_RING_BUFFER_VALIDATE_TIME_DELTAS is not set
# CONFIG_PREEMPTIRQ_DELAY_TEST is not set
# CONFIG_KPROBE_EVENT_GEN_TEST is not set
# CONFIG_SAMPLES is not set
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
CONFIG_STRICT_DEVMEM=y
# CONFIG_IO_STRICT_DEVMEM is not set

#
# powerpc Debugging
#
# CONFIG_PPC_DISABLE_WERROR is not set
CONFIG_PPC_WERROR=y
CONFIG_PRINT_STACK_DEPTH=64
# CONFIG_HCALL_STATS is not set
# CONFIG_PPC_EMULATED_STATS is not set
CONFIG_CODE_PATCHING_SELFTEST=y
CONFIG_JUMP_LABEL_FEATURE_CHECKS=y
# CONFIG_JUMP_LABEL_FEATURE_CHECK_DEBUG is not set
CONFIG_FTR_FIXUP_SELFTEST=y
CONFIG_MSI_BITMAP_SELFTEST=y
# CONFIG_PPC_IRQ_SOFT_MASK_DEBUG is not set
# CONFIG_PPC_RFI_SRR_DEBUG is not set
CONFIG_XMON=y
# CONFIG_XMON_DEFAULT is not set
CONFIG_XMON_DISASSEMBLY=y
CONFIG_XMON_DEFAULT_RO_MODE=y
CONFIG_DEBUGGER=y
CONFIG_BOOTX_TEXT=y
# CONFIG_PPC_EARLY_DEBUG is not set
# CONFIG_PPC_FAST_ENDIAN_SWITCH is not set
# end of powerpc Debugging

#
# Kernel Testing and Coverage
#
# CONFIG_KUNIT is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
CONFIG_FUNCTION_ERROR_INJECTION=y
# CONFIG_FAULT_INJECTION is not set
CONFIG_ARCH_HAS_KCOV=y
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
# CONFIG_KCOV is not set
CONFIG_RUNTIME_TESTING_MENU=y
# CONFIG_LKDTM is not set
# CONFIG_TEST_MIN_HEAP is not set
# CONFIG_TEST_DIV64 is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_TEST_REF_TRACKER is not set
# CONFIG_RBTREE_TEST is not set
# CONFIG_REED_SOLOMON_TEST is not set
# CONFIG_INTERVAL_TREE_TEST is not set
# CONFIG_PERCPU_TEST is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_ASYNC_RAID6_TEST is not set
# CONFIG_TEST_HEXDUMP is not set
# CONFIG_STRING_SELFTEST is not set
# CONFIG_TEST_STRING_HELPERS is not set
# CONFIG_TEST_STRSCPY is not set
# CONFIG_TEST_KSTRTOX is not set
# CONFIG_TEST_PRINTF is not set
# CONFIG_TEST_SCANF is not set
# CONFIG_TEST_BITMAP is not set
# CONFIG_TEST_UUID is not set
# CONFIG_TEST_XARRAY is not set
# CONFIG_TEST_OVERFLOW is not set
# CONFIG_TEST_RHASHTABLE is not set
# CONFIG_TEST_SIPHASH is not set
# CONFIG_TEST_IDA is not set
# CONFIG_TEST_LKM is not set
# CONFIG_TEST_BITOPS is not set
# CONFIG_TEST_VMALLOC is not set
# CONFIG_TEST_USER_COPY is not set
# CONFIG_TEST_BPF is not set
# CONFIG_TEST_BLACKHOLE_DEV is not set
# CONFIG_FIND_BIT_BENCHMARK is not set
# CONFIG_TEST_FIRMWARE is not set
# CONFIG_TEST_SYSCTL is not set
# CONFIG_TEST_UDELAY is not set
# CONFIG_TEST_STATIC_KEYS is not set
# CONFIG_TEST_KMOD is not set
# CONFIG_TEST_MEMCAT_P is not set
# CONFIG_TEST_STACKINIT is not set
# CONFIG_TEST_MEMINIT is not set
# CONFIG_TEST_FREE_PAGES is not set
CONFIG_ARCH_USE_MEMTEST=y
# CONFIG_MEMTEST is not set
# end of Kernel Testing and Coverage
# end of Kernel hacking

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb)
  2022-02-16 13:19           ` BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb) Paul Menzel
@ 2022-02-17  1:16             ` Nathan Chancellor
  2022-02-21 11:17               ` Paul Menzel
  0 siblings, 1 reply; 6+ messages in thread
From: Nathan Chancellor @ 2022-02-17  1:16 UTC (permalink / raw)
  To: Paul Menzel
  Cc: Zhouyi Zhou, Paul E. McKenney, Josh Triplett, rcu, LKML,
	David S. Miller, Jakub Kicinski, netdev, Nick Desaulniers, llvm

Hi Paul,

On Wed, Feb 16, 2022 at 02:19:51PM +0100, Paul Menzel wrote:
> [Cc: +LLVM/clang build support folks]
> 
> 
> Dear Zhouyi, dear Nathan, dear Nick,
> 
> 
> To recap: On a ppc64le machine, building Linux in Ubuntu 21.10 with *llvm*
> and *clang* 1:13.0-53~exp1
> 
>     $ clang --version
>     Ubuntu clang version 13.0.0-2
>     Target: powerpc64le-unknown-linux-gnu
>     Thread model: posix
>     InstalledDir: /usr/bin
> 
> results in a segmentation fault, while it works when building with GCC.
> 
>     $ gcc --version
>     gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0

Thank you for keying us in. I am going to have a bit of a brain dump
here based on the information I have uncovered after a couple of hours
of debugging.

TL;DR: It seems like something is broken with __read_mostly + ld.lld
before 14.0.0.

My initial reproduction steps (boot-qemu.sh comes from
https://github.com/ClangBuiltLinux/boot-utils):

$ clang --version
clang version 13.0.1 (Fedora 13.0.1-1.fc37)
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

$ powerpc64le-linux-gnu-as --version
GNU assembler version 2.37-2.fc36
Copyright (C) 2021 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `powerpc64le-linux-gnu'.

$ curl -LSso .config https://lore.kernel.org/all/f41550c7-26c0-cf81-7de9-aa924434a565@molgen.mpg.de/3-linux-5.17-rc4-rcu-dev-config.txt

$ scripts/config --set-val INITRAMFS_SOURCE '""'

$ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- LLVM=1 LLVM_IAS=0 all

$ boot-qemu.sh -a ppc64le -k . -t 45s
QEMU location: /usr/bin

QEMU version: QEMU emulator version 6.2.0 (qemu-6.2.0-5.fc37)

+ timeout --foreground 45s stdbuf -oL -eL qemu-system-ppc64 -initrd \
/home/nathan/cbl/github/boot-utils-ro/images/ppc64le/rootfs.cpio -device \
ipmi-bmc-sim,id=bmc0 -device isa-ipmi-bt,bmc=bmc0,irq=10 -L \
/home/nathan/cbl/github/boot-utils-ro/images/ppc64le/ -bios skiboot.lid \
-machine powernv8 -display none -kernel \
/home/nathan/cbl/src/linux/arch/powerpc/boot/zImage.epapr -m 2G \
-nodefaults -serial mon:stdio
...
[    1.478028][    T1] BUG: Kernel NULL pointer dereference on read at 0x00000000
[    1.478630][    T1] Faulting instruction address: 0xc00000000090bee0
[    1.479521][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
[    1.480036][    T1] LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=16 NUMA PowerNV
[    1.480853][    T1] Modules linked in:
[    1.481265][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00001-gfa15c7cb550f #1
[    1.481967][    T1] NIP:  c00000000090bee0 LR: c000000000d96b60 CTR: c0000000000d5b4c
[    1.482596][    T1] REGS: c000000007443330 TRAP: 0380   Not tainted  (5.17.0-rc4-00001-gfa15c7cb550f)
[    1.483305][    T1] MSR:  9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE>  CR: 22800a87  XER: 00000000
[    1.484277][    T1] CFAR: c000000000d96b5c IRQMASK: 0
[    1.484277][    T1] GPR00: c000000000d96b54 c0000000074435d0 c0000000028bc600 0000000000000000
[    1.484277][    T1] GPR04: ffffffffffffffff ffffffffff1ea558 ffffffffff1ebfe4 c00000000261ae88
[    1.484277][    T1] GPR08: 0000000000000003 0000000000000004 c00000000261ae88 0000000000000000
[    1.484277][    T1] GPR12: 0000000000800000 c000000002a60000 c000000000012518 0000000000000000
[    1.484277][    T1] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[    1.484277][    T1] GPR20: 0000000000000000 c0000000027bff80 0000000000000cc0 0000000000000000
[    1.484277][    T1] GPR24: 0000000000000010 0000000000000000 0000000000000000 0000000000000000
[    1.484277][    T1] GPR28: c0000000028fcfd8 c000000007b83000 0000000000000000 c0000000074435d0
[    1.490325][    T1] NIP [c00000000090bee0] strlen+0x10/0x30
[    1.490788][    T1] LR [c000000000d96b60] if_nlmsg_size+0x2b0/0x390
[    1.491319][    T1] Call Trace:
[    1.491573][    T1] [c0000000074435d0] [c000000000d96b54] if_nlmsg_size+0x2a4/0x390 (unreliable)
[    1.492291][    T1] [c000000007443680] [c000000000d96790] rtmsg_ifinfo_build_skb+0x80/0x1a0
[    1.492958][    T1] [c000000007443740] [c000000000d97590] rtmsg_ifinfo+0x70/0xd0
[    1.493559][    T1] [c000000007443790] [c000000000d7d528] register_netdevice+0x5d8/0x670
[    1.494205][    T1] [c000000007443820] [c000000000d7d94c] register_netdev+0x4c/0x80
[    1.494823][    T1] [c000000007443850] [c000000000f826d8] sit_init_net+0x1b8/0x200
[    1.495426][    T1] [c0000000074438d0] [c000000000d63b5c] ops_init+0x14c/0x1c0
[    1.496014][    T1] [c000000007443930] [c000000000d6314c] register_pernet_operations+0xec/0x1e0
[    1.496716][    T1] [c000000007443990] [c000000000d633d0] register_pernet_device+0x60/0xd0
[    1.497372][    T1] [c0000000074439e0] [c000000002085194] sit_init+0x54/0x160
[    1.497950][    T1] [c000000007443a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
[    1.498573][    T1] [c000000007443c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
[    1.499219][    T1] [c000000007443cc0] [c00000000200604c] do_initcalls+0x84/0xe4
[    1.499799][    T1] [c000000007443d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
[    1.500444][    T1] [c000000007443da0] [c00000000001254c] kernel_init+0x3c/0x270
[    1.501042][    T1] [c000000007443e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
[    1.501721][    T1] Instruction dump:
[    1.502202][    T1] eb81ffe0 7c0803a6 4e800020 00000000 00000000 00000000 60000000 60000000
[    1.502934][    T1] 3883ffff 60000000 60000000 60000000 <8ca40001> 28050000 4082fff8 7c632050
[    1.504028][    T1] ---[ end trace 0000000000000000 ]---
...

First thing was figuring out where the NULL pointer dereference happens,
which appears to the "strlen(ops->kind)" in rtnl_link_get_size():

515 static size_t rtnl_link_get_size(const struct net_device *dev)
516 {
517 	const struct rtnl_link_ops *ops = dev->rtnl_link_ops;
518 	size_t size;
519 
520 	if (!ops)
521 		return 0;
522 
523 	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
524 	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */

which I confirmed some really rudimentary printk debugging:

[    1.476862][    T1] nathan: rtnl_link_get_size(): name: sit0, ops: c0000000028fcfd8, ops->kind: (null)

diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index 710da8a36729..c8d928e83aec 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c
@@ -520,6 +520,9 @@ static size_t rtnl_link_get_size(const struct net_device *dev)
 	if (!ops)
 		return 0;
 
+	pr_err("nathan: %s(): name: %s, ops: %px, ops->kind: %s\n", __func__,
+	       dev->name, ops, ops->kind);
+
 	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
 	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
 

Okay... how did sit0 end up with a NULL kind...? It is very clearly
defined as "sit":

1830 static struct rtnl_link_ops sit_link_ops __read_mostly = {
1831 	.kind		= "sit",

Adding some more debug prints to net/ipv6/sit.c:

diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
index c0b138c20992..7b9edbed2fcd 100644
--- a/net/ipv6/sit.c
+++ b/net/ipv6/sit.c
@@ -1920,6 +1920,12 @@ static int __net_init sit_init_net(struct net *net)
 	 */
 	sitn->fb_tunnel_dev->features |= NETIF_F_NETNS_LOCAL;
 
+	pr_err("nathan: %s(): &sit_link_ops: %px\n", __func__, &sit_link_ops);
+	pr_err("nathan: %s(): sit_link_ops.kind: %s\n", __func__, sit_link_ops.kind);
+	pr_err("nathan: %s(): sit_link_ops.maxtype: %u\n", __func__, sit_link_ops.maxtype);
+	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops: %px\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops);
+	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: %s\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops->kind);
+
 	err = register_netdev(sitn->fb_tunnel_dev);
 	if (err)
 		goto err_reg_dev;

reveals:

[    1.471920][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028fcfd8
[    1.472534][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: (null)
[    1.473088][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
[    1.473639][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028fcfd8
[    1.474370][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: (null)

This is super bizarre, as the maxtype member appears to have the correct
value, but how is kind's initial getting dropped on the floor?

Removing the __read_mostly annotation "fixes" it:

[    1.481708][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000027d3f60
[    1.482319][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
[    1.482878][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
[    1.483429][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000027d3f60
[    1.484174][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
...
Linux version 5.17.0-rc4-00001-g956f02ad5c31-dirty (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), LLD 13.0.1) #2 SMP PREEMPT Wed Feb 16 13:29:49 MST 2022
...

diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
index 7b9edbed2fcd..f109c7a0233b 100644
--- a/net/ipv6/sit.c
+++ b/net/ipv6/sit.c
@@ -70,7 +70,7 @@ static void ipip6_tunnel_setup(struct net_device *dev);
 static void ipip6_dev_free(struct net_device *dev);
 static bool check_6rd(struct ip_tunnel *tunnel, const struct in6_addr *v6dst,
 		      __be32 *v4dst);
-static struct rtnl_link_ops sit_link_ops __read_mostly;
+static struct rtnl_link_ops sit_link_ops;
 
 static unsigned int sit_net_id __read_mostly;
 struct sit_net {
@@ -1827,7 +1827,7 @@ static void ipip6_dellink(struct net_device *dev, struct list_head *head)
 		unregister_netdevice_queue(dev, head);
 }
 
-static struct rtnl_link_ops sit_link_ops __read_mostly = {
+static struct rtnl_link_ops sit_link_ops = {
 	.kind		= "sit",
 	.maxtype	= IFLA_IPTUN_MAX,
 	.policy		= ipip6_policy,

Switching to ld.bfd also resolves it:

[    1.470405][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028acfd8
[    1.471016][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
[    1.471534][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
[    1.472062][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028acfd8
[    1.472790][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
...
Linux version 5.17.0-rc4-00001-g956f02ad5c31 (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), GNU ld version 2.37-2.fc36) #3 SMP PREEMPT Wed Feb 16 13:33:42 MST 2022
...

I tested with ToT LLVM (or at least, close to it, since there is an
unrelated ld.lld regression there) and I could not reproduce it there,
so I did a reverse bisect to see what commit fixes this issue in LLVM 14
and I landed on:

commit 55c14d6dbfd8e7b86c15d2613fea3490078e2ae4
Author: Fangrui Song <i@maskray.me>
Date:   Thu Nov 25 14:12:34 2021 -0800

    [ELF] Simplify DynamicSection content computation. NFC

    The new code computes the content twice, but avoides the tricky
    std::function<uint64_t()>. Removed 13KiB code in a Release build.

 lld/ELF/SyntheticSections.cpp | 117 ++++++++++++++++--------------------------
 lld/ELF/SyntheticSections.h   |  12 +----
 2 files changed, 44 insertions(+), 85 deletions(-)

That's... interesting, given that commit title says No Functional
Change, even though there clearly is one. That commit has a couple
mentions of PowerPC synthetic sections, so it is possible that the
new content calculation lines up with ld.bfd?

I am not really sure where to go from here, as I don't fully understand
what the problem was before that LLD change. I'll see if I can do some
more investigation tomorrow (unless someone wants to beat me to it ;)

Cheers,
Nathan

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb)
  2022-02-17  1:16             ` Nathan Chancellor
@ 2022-02-21 11:17               ` Paul Menzel
  2022-02-21 15:29                 ` Nathan Chancellor
  0 siblings, 1 reply; 6+ messages in thread
From: Paul Menzel @ 2022-02-21 11:17 UTC (permalink / raw)
  To: Nathan Chancellor
  Cc: Zhouyi Zhou, Paul E. McKenney, Josh Triplett, rcu, LKML,
	David S. Miller, Jakub Kicinski, netdev, Nick Desaulniers, llvm,
	Fangrui Song

[Cc: +Fangrui]

Dear Nathan,


Am 17.02.22 um 02:16 schrieb Nathan Chancellor:

> On Wed, Feb 16, 2022 at 02:19:51PM +0100, Paul Menzel wrote:
>> [Cc: +LLVM/clang build support folks]

[…]

>> To recap: On a ppc64le machine, building Linux in Ubuntu 21.10 with *llvm*
>> and *clang* 1:13.0-53~exp1
>>
>>      $ clang --version
>>      Ubuntu clang version 13.0.0-2
>>      Target: powerpc64le-unknown-linux-gnu
>>      Thread model: posix
>>      InstalledDir: /usr/bin
>>
>> results in a segmentation fault, while it works when building with GCC.
>>
>>      $ gcc --version
>>      gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0
> 
> Thank you for keying us in. I am going to have a bit of a brain dump
> here based on the information I have uncovered after a couple of hours
> of debugging.
> 
> TL;DR: It seems like something is broken with __read_mostly + ld.lld
> before 14.0.0.
> 
> My initial reproduction steps (boot-qemu.sh comes from
> https://github.com/ClangBuiltLinux/boot-utils):
> 
> $ clang --version
> clang version 13.0.1 (Fedora 13.0.1-1.fc37)
> Target: x86_64-redhat-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin
> 
> $ powerpc64le-linux-gnu-as --version
> GNU assembler version 2.37-2.fc36
> Copyright (C) 2021 Free Software Foundation, Inc.
> This program is free software; you may redistribute it under the terms of
> the GNU General Public License version 3 or later.
> This program has absolutely no warranty.
> This assembler was configured for a target of `powerpc64le-linux-gnu'.
> 
> $ curl -LSso .config https://lore.kernel.org/all/f41550c7-26c0-cf81-7de9-aa924434a565@molgen.mpg.de/3-linux-5.17-rc4-rcu-dev-config.txt
> 
> $ scripts/config --set-val INITRAMFS_SOURCE '""'
> 
> $ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- LLVM=1 LLVM_IAS=0 all
> 
> $ boot-qemu.sh -a ppc64le -k . -t 45s
> QEMU location: /usr/bin
> 
> QEMU version: QEMU emulator version 6.2.0 (qemu-6.2.0-5.fc37)
> 
> + timeout --foreground 45s stdbuf -oL -eL qemu-system-ppc64 -initrd \
> /home/nathan/cbl/github/boot-utils-ro/images/ppc64le/rootfs.cpio -device \
> ipmi-bmc-sim,id=bmc0 -device isa-ipmi-bt,bmc=bmc0,irq=10 -L \
> /home/nathan/cbl/github/boot-utils-ro/images/ppc64le/ -bios skiboot.lid \
> -machine powernv8 -display none -kernel \
> /home/nathan/cbl/src/linux/arch/powerpc/boot/zImage.epapr -m 2G \
> -nodefaults -serial mon:stdio
> ...
> [    1.478028][    T1] BUG: Kernel NULL pointer dereference on read at 0x00000000
> [    1.478630][    T1] Faulting instruction address: 0xc00000000090bee0
> [    1.479521][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
> [    1.480036][    T1] LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=16 NUMA PowerNV
> [    1.480853][    T1] Modules linked in:
> [    1.481265][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00001-gfa15c7cb550f #1
> [    1.481967][    T1] NIP:  c00000000090bee0 LR: c000000000d96b60 CTR: c0000000000d5b4c
> [    1.482596][    T1] REGS: c000000007443330 TRAP: 0380   Not tainted  (5.17.0-rc4-00001-gfa15c7cb550f)
> [    1.483305][    T1] MSR:  9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE>  CR: 22800a87  XER: 00000000
> [    1.484277][    T1] CFAR: c000000000d96b5c IRQMASK: 0
> [    1.484277][    T1] GPR00: c000000000d96b54 c0000000074435d0 c0000000028bc600 0000000000000000
> [    1.484277][    T1] GPR04: ffffffffffffffff ffffffffff1ea558 ffffffffff1ebfe4 c00000000261ae88
> [    1.484277][    T1] GPR08: 0000000000000003 0000000000000004 c00000000261ae88 0000000000000000
> [    1.484277][    T1] GPR12: 0000000000800000 c000000002a60000 c000000000012518 0000000000000000
> [    1.484277][    T1] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [    1.484277][    T1] GPR20: 0000000000000000 c0000000027bff80 0000000000000cc0 0000000000000000
> [    1.484277][    T1] GPR24: 0000000000000010 0000000000000000 0000000000000000 0000000000000000
> [    1.484277][    T1] GPR28: c0000000028fcfd8 c000000007b83000 0000000000000000 c0000000074435d0
> [    1.490325][    T1] NIP [c00000000090bee0] strlen+0x10/0x30
> [    1.490788][    T1] LR [c000000000d96b60] if_nlmsg_size+0x2b0/0x390
> [    1.491319][    T1] Call Trace:
> [    1.491573][    T1] [c0000000074435d0] [c000000000d96b54] if_nlmsg_size+0x2a4/0x390 (unreliable)
> [    1.492291][    T1] [c000000007443680] [c000000000d96790] rtmsg_ifinfo_build_skb+0x80/0x1a0
> [    1.492958][    T1] [c000000007443740] [c000000000d97590] rtmsg_ifinfo+0x70/0xd0
> [    1.493559][    T1] [c000000007443790] [c000000000d7d528] register_netdevice+0x5d8/0x670
> [    1.494205][    T1] [c000000007443820] [c000000000d7d94c] register_netdev+0x4c/0x80
> [    1.494823][    T1] [c000000007443850] [c000000000f826d8] sit_init_net+0x1b8/0x200
> [    1.495426][    T1] [c0000000074438d0] [c000000000d63b5c] ops_init+0x14c/0x1c0
> [    1.496014][    T1] [c000000007443930] [c000000000d6314c] register_pernet_operations+0xec/0x1e0
> [    1.496716][    T1] [c000000007443990] [c000000000d633d0] register_pernet_device+0x60/0xd0
> [    1.497372][    T1] [c0000000074439e0] [c000000002085194] sit_init+0x54/0x160
> [    1.497950][    T1] [c000000007443a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
> [    1.498573][    T1] [c000000007443c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
> [    1.499219][    T1] [c000000007443cc0] [c00000000200604c] do_initcalls+0x84/0xe4
> [    1.499799][    T1] [c000000007443d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
> [    1.500444][    T1] [c000000007443da0] [c00000000001254c] kernel_init+0x3c/0x270
> [    1.501042][    T1] [c000000007443e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
> [    1.501721][    T1] Instruction dump:
> [    1.502202][    T1] eb81ffe0 7c0803a6 4e800020 00000000 00000000 00000000 60000000 60000000
> [    1.502934][    T1] 3883ffff 60000000 60000000 60000000 <8ca40001> 28050000 4082fff8 7c632050
> [    1.504028][    T1] ---[ end trace 0000000000000000 ]---
> ...
> 
> First thing was figuring out where the NULL pointer dereference happens,
> which appears to the "strlen(ops->kind)" in rtnl_link_get_size():
> 
> 515 static size_t rtnl_link_get_size(const struct net_device *dev)
> 516 {
> 517 	const struct rtnl_link_ops *ops = dev->rtnl_link_ops;
> 518 	size_t size;
> 519
> 520 	if (!ops)
> 521 		return 0;
> 522
> 523 	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
> 524 	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
> 
> which I confirmed some really rudimentary printk debugging:
> 
> [    1.476862][    T1] nathan: rtnl_link_get_size(): name: sit0, ops: c0000000028fcfd8, ops->kind: (null)
> 
> diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
> index 710da8a36729..c8d928e83aec 100644
> --- a/net/core/rtnetlink.c
> +++ b/net/core/rtnetlink.c
> @@ -520,6 +520,9 @@ static size_t rtnl_link_get_size(const struct net_device *dev)
>   	if (!ops)
>   		return 0;
>   
> +	pr_err("nathan: %s(): name: %s, ops: %px, ops->kind: %s\n", __func__,
> +	       dev->name, ops, ops->kind);
> +
>   	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
>   	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
>   
> 
> Okay... how did sit0 end up with a NULL kind...? It is very clearly
> defined as "sit":
> 
> 1830 static struct rtnl_link_ops sit_link_ops __read_mostly = {
> 1831 	.kind		= "sit",
> 
> Adding some more debug prints to net/ipv6/sit.c:
> 
> diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
> index c0b138c20992..7b9edbed2fcd 100644
> --- a/net/ipv6/sit.c
> +++ b/net/ipv6/sit.c
> @@ -1920,6 +1920,12 @@ static int __net_init sit_init_net(struct net *net)
>   	 */
>   	sitn->fb_tunnel_dev->features |= NETIF_F_NETNS_LOCAL;
>   
> +	pr_err("nathan: %s(): &sit_link_ops: %px\n", __func__, &sit_link_ops);
> +	pr_err("nathan: %s(): sit_link_ops.kind: %s\n", __func__, sit_link_ops.kind);
> +	pr_err("nathan: %s(): sit_link_ops.maxtype: %u\n", __func__, sit_link_ops.maxtype);
> +	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops: %px\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops);
> +	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: %s\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops->kind);
> +
>   	err = register_netdev(sitn->fb_tunnel_dev);
>   	if (err)
>   		goto err_reg_dev;
> 
> reveals:
> 
> [    1.471920][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028fcfd8
> [    1.472534][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: (null)
> [    1.473088][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> [    1.473639][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028fcfd8
> [    1.474370][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: (null)
> 
> This is super bizarre, as the maxtype member appears to have the correct
> value, but how is kind's initial getting dropped on the floor?
> 
> Removing the __read_mostly annotation "fixes" it:
> 
> [    1.481708][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000027d3f60
> [    1.482319][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
> [    1.482878][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> [    1.483429][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000027d3f60
> [    1.484174][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
> ...
> Linux version 5.17.0-rc4-00001-g956f02ad5c31-dirty (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), LLD 13.0.1) #2 SMP PREEMPT Wed Feb 16 13:29:49 MST 2022
> ...
> 
> diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
> index 7b9edbed2fcd..f109c7a0233b 100644
> --- a/net/ipv6/sit.c
> +++ b/net/ipv6/sit.c
> @@ -70,7 +70,7 @@ static void ipip6_tunnel_setup(struct net_device *dev);
>   static void ipip6_dev_free(struct net_device *dev);
>   static bool check_6rd(struct ip_tunnel *tunnel, const struct in6_addr *v6dst,
>   		      __be32 *v4dst);
> -static struct rtnl_link_ops sit_link_ops __read_mostly;
> +static struct rtnl_link_ops sit_link_ops;
>   
>   static unsigned int sit_net_id __read_mostly;
>   struct sit_net {
> @@ -1827,7 +1827,7 @@ static void ipip6_dellink(struct net_device *dev, struct list_head *head)
>   		unregister_netdevice_queue(dev, head);
>   }
>   
> -static struct rtnl_link_ops sit_link_ops __read_mostly = {
> +static struct rtnl_link_ops sit_link_ops = {
>   	.kind		= "sit",
>   	.maxtype	= IFLA_IPTUN_MAX,
>   	.policy		= ipip6_policy,
> 
> Switching to ld.bfd also resolves it:
> 
> [    1.470405][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028acfd8
> [    1.471016][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
> [    1.471534][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> [    1.472062][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028acfd8
> [    1.472790][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
> ...
> Linux version 5.17.0-rc4-00001-g956f02ad5c31 (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), GNU ld version 2.37-2.fc36) #3 SMP PREEMPT Wed Feb 16 13:33:42 MST 2022
> ...
> 
> I tested with ToT LLVM (or at least, close to it, since there is an
> unrelated ld.lld regression there) and I could not reproduce it there,
> so I did a reverse bisect to see what commit fixes this issue in LLVM 14
> and I landed on:
> 
> commit 55c14d6dbfd8e7b86c15d2613fea3490078e2ae4
> Author: Fangrui Song <i@maskray.me>
> Date:   Thu Nov 25 14:12:34 2021 -0800
> 
>      [ELF] Simplify DynamicSection content computation. NFC
> 
>      The new code computes the content twice, but avoides the tricky
>      std::function<uint64_t()>. Removed 13KiB code in a Release build.
> 
>   lld/ELF/SyntheticSections.cpp | 117 ++++++++++++++++--------------------------
>   lld/ELF/SyntheticSections.h   |  12 +----
>   2 files changed, 44 insertions(+), 85 deletions(-)
> 
> That's... interesting, given that commit title says No Functional
> Change, even though there clearly is one. That commit has a couple
> mentions of PowerPC synthetic sections, so it is possible that the
> new content calculation lines up with ld.bfd?
> 
> I am not really sure where to go from here, as I don't fully understand
> what the problem was before that LLD change. I'll see if I can do some
> more investigation tomorrow (unless someone wants to beat me to it ;)

Thank you for looking into this, and sharing your analysis.

I built LLVM/clang from the master branch, rebuilt, but can still 
reproduce this.

     $ git clone --depth=1 https://github.com/llvm/llvm-project.git
     $ cd llvm-project/
     $ git log --oneline
     41cb504b7 [mlir][linalg][bufferize][NFC] Move interface impl to 
Linalg Transforms
     $ mkdir build
     $ cd build
     $ cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles" 
-DCMAKE_BUILD_TYPE=Release -DLLVM_INSTALL_UTILS=ON 
-DCMAKE_INSTALL_PREFIX=/scratch/local2/llvm ../llvm
     $ make -j20
     $ make -j20 clang-check
     $ make install
     $ /scratch/local2/llvm/bin/clang --version
     clang version 15.0.0 (https://github.com/llvm/llvm-project.git 
41cb504b7c4b18ac15830107431a0c1eec73a6b2)
     Target: powerpc64le-unknown-linux-gnu
     Thread model: posix
     InstalledDir: /scratch/local2/llvm/bin

Then build Linux after `make clean` with `/scratch/local2/llvm/bin` in 
the path.

     $ LLVM=1 LLVM_IAS=0 eatmydata make -j20

     $ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=1 
-net none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial 
stdio -m 512 -kernel /dev/shm/linux/vmlinux -append 
"debug_boot_weak_hash panic=-1 console=ttyS0 
rcupdate.rcu_cpu_stall_suppress_at_boot=1 torture.disable_onoff_at_boot 
rcupdate.rcu_task_stall_timeout=30000 rcupdate.rcu_self_test=1 
rcutorture.onoff_interval=1000 rcutorture.onoff_holdoff=30 
rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15 
rcutorture.shutdown_secs=420 rcutorture.test_no_idle_hz=1 
rcutorture.verbose=1"
     […]
     Preparing to boot Linux version 5.17.0-rc5-00178-ga4b9a8fb20e7 
(pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (clang 
version 15.0.0 (https://github.com/llvm/llvm-project.git 
41cb504b7c4b18ac15830107431a0c1eec73a6b2), LLD 13.0.0) #29 SMP PREEMPT 
Mon Feb 21 10:58:54 CET 2022
     […]
     [    0.465889][    T1] BUG: Kernel NULL pointer dereference on read 
at 0x00000000
     [    0.466749][    T1] Faulting instruction address: 0xc0000000008fc300
     [    0.467507][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
     […]


Kind regards,

Paul

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb)
  2022-02-21 11:17               ` Paul Menzel
@ 2022-02-21 15:29                 ` Nathan Chancellor
  2022-02-21 17:33                   ` Paul Menzel
  2022-04-19 21:34                   ` Nathan Chancellor
  0 siblings, 2 replies; 6+ messages in thread
From: Nathan Chancellor @ 2022-02-21 15:29 UTC (permalink / raw)
  To: Paul Menzel
  Cc: Zhouyi Zhou, Paul E. McKenney, Josh Triplett, rcu, LKML,
	David S. Miller, Jakub Kicinski, netdev, Nick Desaulniers, llvm,
	Fangrui Song

Hi Paul,

On Mon, Feb 21, 2022 at 12:17:40PM +0100, Paul Menzel wrote:
> Am 17.02.22 um 02:16 schrieb Nathan Chancellor:
> 
> > On Wed, Feb 16, 2022 at 02:19:51PM +0100, Paul Menzel wrote:
> > > [Cc: +LLVM/clang build support folks]
> 
> […]
> 
> > > To recap: On a ppc64le machine, building Linux in Ubuntu 21.10 with *llvm*
> > > and *clang* 1:13.0-53~exp1
> > > 
> > >      $ clang --version
> > >      Ubuntu clang version 13.0.0-2
> > >      Target: powerpc64le-unknown-linux-gnu
> > >      Thread model: posix
> > >      InstalledDir: /usr/bin
> > > 
> > > results in a segmentation fault, while it works when building with GCC.
> > > 
> > >      $ gcc --version
> > >      gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0
> > 
> > Thank you for keying us in. I am going to have a bit of a brain dump
> > here based on the information I have uncovered after a couple of hours
> > of debugging.
> > 
> > TL;DR: It seems like something is broken with __read_mostly + ld.lld
> > before 14.0.0.
> > 
> > My initial reproduction steps (boot-qemu.sh comes from
> > https://github.com/ClangBuiltLinux/boot-utils):
> > 
> > $ clang --version
> > clang version 13.0.1 (Fedora 13.0.1-1.fc37)
> > Target: x86_64-redhat-linux-gnu
> > Thread model: posix
> > InstalledDir: /usr/bin
> > 
> > $ powerpc64le-linux-gnu-as --version
> > GNU assembler version 2.37-2.fc36
> > Copyright (C) 2021 Free Software Foundation, Inc.
> > This program is free software; you may redistribute it under the terms of
> > the GNU General Public License version 3 or later.
> > This program has absolutely no warranty.
> > This assembler was configured for a target of `powerpc64le-linux-gnu'.
> > 
> > $ curl -LSso .config https://lore.kernel.org/all/f41550c7-26c0-cf81-7de9-aa924434a565@molgen.mpg.de/3-linux-5.17-rc4-rcu-dev-config.txt
> > 
> > $ scripts/config --set-val INITRAMFS_SOURCE '""'
> > 
> > $ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- LLVM=1 LLVM_IAS=0 all
> > 
> > $ boot-qemu.sh -a ppc64le -k . -t 45s
> > QEMU location: /usr/bin
> > 
> > QEMU version: QEMU emulator version 6.2.0 (qemu-6.2.0-5.fc37)
> > 
> > + timeout --foreground 45s stdbuf -oL -eL qemu-system-ppc64 -initrd \
> > /home/nathan/cbl/github/boot-utils-ro/images/ppc64le/rootfs.cpio -device \
> > ipmi-bmc-sim,id=bmc0 -device isa-ipmi-bt,bmc=bmc0,irq=10 -L \
> > /home/nathan/cbl/github/boot-utils-ro/images/ppc64le/ -bios skiboot.lid \
> > -machine powernv8 -display none -kernel \
> > /home/nathan/cbl/src/linux/arch/powerpc/boot/zImage.epapr -m 2G \
> > -nodefaults -serial mon:stdio
> > ...
> > [    1.478028][    T1] BUG: Kernel NULL pointer dereference on read at 0x00000000
> > [    1.478630][    T1] Faulting instruction address: 0xc00000000090bee0
> > [    1.479521][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
> > [    1.480036][    T1] LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=16 NUMA PowerNV
> > [    1.480853][    T1] Modules linked in:
> > [    1.481265][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00001-gfa15c7cb550f #1
> > [    1.481967][    T1] NIP:  c00000000090bee0 LR: c000000000d96b60 CTR: c0000000000d5b4c
> > [    1.482596][    T1] REGS: c000000007443330 TRAP: 0380   Not tainted  (5.17.0-rc4-00001-gfa15c7cb550f)
> > [    1.483305][    T1] MSR:  9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE>  CR: 22800a87  XER: 00000000
> > [    1.484277][    T1] CFAR: c000000000d96b5c IRQMASK: 0
> > [    1.484277][    T1] GPR00: c000000000d96b54 c0000000074435d0 c0000000028bc600 0000000000000000
> > [    1.484277][    T1] GPR04: ffffffffffffffff ffffffffff1ea558 ffffffffff1ebfe4 c00000000261ae88
> > [    1.484277][    T1] GPR08: 0000000000000003 0000000000000004 c00000000261ae88 0000000000000000
> > [    1.484277][    T1] GPR12: 0000000000800000 c000000002a60000 c000000000012518 0000000000000000
> > [    1.484277][    T1] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> > [    1.484277][    T1] GPR20: 0000000000000000 c0000000027bff80 0000000000000cc0 0000000000000000
> > [    1.484277][    T1] GPR24: 0000000000000010 0000000000000000 0000000000000000 0000000000000000
> > [    1.484277][    T1] GPR28: c0000000028fcfd8 c000000007b83000 0000000000000000 c0000000074435d0
> > [    1.490325][    T1] NIP [c00000000090bee0] strlen+0x10/0x30
> > [    1.490788][    T1] LR [c000000000d96b60] if_nlmsg_size+0x2b0/0x390
> > [    1.491319][    T1] Call Trace:
> > [    1.491573][    T1] [c0000000074435d0] [c000000000d96b54] if_nlmsg_size+0x2a4/0x390 (unreliable)
> > [    1.492291][    T1] [c000000007443680] [c000000000d96790] rtmsg_ifinfo_build_skb+0x80/0x1a0
> > [    1.492958][    T1] [c000000007443740] [c000000000d97590] rtmsg_ifinfo+0x70/0xd0
> > [    1.493559][    T1] [c000000007443790] [c000000000d7d528] register_netdevice+0x5d8/0x670
> > [    1.494205][    T1] [c000000007443820] [c000000000d7d94c] register_netdev+0x4c/0x80
> > [    1.494823][    T1] [c000000007443850] [c000000000f826d8] sit_init_net+0x1b8/0x200
> > [    1.495426][    T1] [c0000000074438d0] [c000000000d63b5c] ops_init+0x14c/0x1c0
> > [    1.496014][    T1] [c000000007443930] [c000000000d6314c] register_pernet_operations+0xec/0x1e0
> > [    1.496716][    T1] [c000000007443990] [c000000000d633d0] register_pernet_device+0x60/0xd0
> > [    1.497372][    T1] [c0000000074439e0] [c000000002085194] sit_init+0x54/0x160
> > [    1.497950][    T1] [c000000007443a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
> > [    1.498573][    T1] [c000000007443c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
> > [    1.499219][    T1] [c000000007443cc0] [c00000000200604c] do_initcalls+0x84/0xe4
> > [    1.499799][    T1] [c000000007443d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
> > [    1.500444][    T1] [c000000007443da0] [c00000000001254c] kernel_init+0x3c/0x270
> > [    1.501042][    T1] [c000000007443e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
> > [    1.501721][    T1] Instruction dump:
> > [    1.502202][    T1] eb81ffe0 7c0803a6 4e800020 00000000 00000000 00000000 60000000 60000000
> > [    1.502934][    T1] 3883ffff 60000000 60000000 60000000 <8ca40001> 28050000 4082fff8 7c632050
> > [    1.504028][    T1] ---[ end trace 0000000000000000 ]---
> > ...
> > 
> > First thing was figuring out where the NULL pointer dereference happens,
> > which appears to the "strlen(ops->kind)" in rtnl_link_get_size():
> > 
> > 515 static size_t rtnl_link_get_size(const struct net_device *dev)
> > 516 {
> > 517 	const struct rtnl_link_ops *ops = dev->rtnl_link_ops;
> > 518 	size_t size;
> > 519
> > 520 	if (!ops)
> > 521 		return 0;
> > 522
> > 523 	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
> > 524 	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
> > 
> > which I confirmed some really rudimentary printk debugging:
> > 
> > [    1.476862][    T1] nathan: rtnl_link_get_size(): name: sit0, ops: c0000000028fcfd8, ops->kind: (null)
> > 
> > diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
> > index 710da8a36729..c8d928e83aec 100644
> > --- a/net/core/rtnetlink.c
> > +++ b/net/core/rtnetlink.c
> > @@ -520,6 +520,9 @@ static size_t rtnl_link_get_size(const struct net_device *dev)
> >   	if (!ops)
> >   		return 0;
> > +	pr_err("nathan: %s(): name: %s, ops: %px, ops->kind: %s\n", __func__,
> > +	       dev->name, ops, ops->kind);
> > +
> >   	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
> >   	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
> > 
> > Okay... how did sit0 end up with a NULL kind...? It is very clearly
> > defined as "sit":
> > 
> > 1830 static struct rtnl_link_ops sit_link_ops __read_mostly = {
> > 1831 	.kind		= "sit",
> > 
> > Adding some more debug prints to net/ipv6/sit.c:
> > 
> > diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
> > index c0b138c20992..7b9edbed2fcd 100644
> > --- a/net/ipv6/sit.c
> > +++ b/net/ipv6/sit.c
> > @@ -1920,6 +1920,12 @@ static int __net_init sit_init_net(struct net *net)
> >   	 */
> >   	sitn->fb_tunnel_dev->features |= NETIF_F_NETNS_LOCAL;
> > +	pr_err("nathan: %s(): &sit_link_ops: %px\n", __func__, &sit_link_ops);
> > +	pr_err("nathan: %s(): sit_link_ops.kind: %s\n", __func__, sit_link_ops.kind);
> > +	pr_err("nathan: %s(): sit_link_ops.maxtype: %u\n", __func__, sit_link_ops.maxtype);
> > +	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops: %px\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops);
> > +	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: %s\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops->kind);
> > +
> >   	err = register_netdev(sitn->fb_tunnel_dev);
> >   	if (err)
> >   		goto err_reg_dev;
> > 
> > reveals:
> > 
> > [    1.471920][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028fcfd8
> > [    1.472534][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: (null)
> > [    1.473088][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> > [    1.473639][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028fcfd8
> > [    1.474370][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: (null)
> > 
> > This is super bizarre, as the maxtype member appears to have the correct
> > value, but how is kind's initial getting dropped on the floor?
> > 
> > Removing the __read_mostly annotation "fixes" it:
> > 
> > [    1.481708][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000027d3f60
> > [    1.482319][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
> > [    1.482878][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> > [    1.483429][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000027d3f60
> > [    1.484174][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
> > ...
> > Linux version 5.17.0-rc4-00001-g956f02ad5c31-dirty (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), LLD 13.0.1) #2 SMP PREEMPT Wed Feb 16 13:29:49 MST 2022
> > ...
> > 
> > diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
> > index 7b9edbed2fcd..f109c7a0233b 100644
> > --- a/net/ipv6/sit.c
> > +++ b/net/ipv6/sit.c
> > @@ -70,7 +70,7 @@ static void ipip6_tunnel_setup(struct net_device *dev);
> >   static void ipip6_dev_free(struct net_device *dev);
> >   static bool check_6rd(struct ip_tunnel *tunnel, const struct in6_addr *v6dst,
> >   		      __be32 *v4dst);
> > -static struct rtnl_link_ops sit_link_ops __read_mostly;
> > +static struct rtnl_link_ops sit_link_ops;
> >   static unsigned int sit_net_id __read_mostly;
> >   struct sit_net {
> > @@ -1827,7 +1827,7 @@ static void ipip6_dellink(struct net_device *dev, struct list_head *head)
> >   		unregister_netdevice_queue(dev, head);
> >   }
> > -static struct rtnl_link_ops sit_link_ops __read_mostly = {
> > +static struct rtnl_link_ops sit_link_ops = {
> >   	.kind		= "sit",
> >   	.maxtype	= IFLA_IPTUN_MAX,
> >   	.policy		= ipip6_policy,
> > 
> > Switching to ld.bfd also resolves it:
> > 
> > [    1.470405][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028acfd8
> > [    1.471016][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
> > [    1.471534][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> > [    1.472062][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028acfd8
> > [    1.472790][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
> > ...
> > Linux version 5.17.0-rc4-00001-g956f02ad5c31 (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), GNU ld version 2.37-2.fc36) #3 SMP PREEMPT Wed Feb 16 13:33:42 MST 2022
> > ...
> > 
> > I tested with ToT LLVM (or at least, close to it, since there is an
> > unrelated ld.lld regression there) and I could not reproduce it there,
> > so I did a reverse bisect to see what commit fixes this issue in LLVM 14
> > and I landed on:
> > 
> > commit 55c14d6dbfd8e7b86c15d2613fea3490078e2ae4
> > Author: Fangrui Song <i@maskray.me>
> > Date:   Thu Nov 25 14:12:34 2021 -0800
> > 
> >      [ELF] Simplify DynamicSection content computation. NFC
> > 
> >      The new code computes the content twice, but avoides the tricky
> >      std::function<uint64_t()>. Removed 13KiB code in a Release build.
> > 
> >   lld/ELF/SyntheticSections.cpp | 117 ++++++++++++++++--------------------------
> >   lld/ELF/SyntheticSections.h   |  12 +----
> >   2 files changed, 44 insertions(+), 85 deletions(-)
> > 
> > That's... interesting, given that commit title says No Functional
> > Change, even though there clearly is one. That commit has a couple
> > mentions of PowerPC synthetic sections, so it is possible that the
> > new content calculation lines up with ld.bfd?
> > 
> > I am not really sure where to go from here, as I don't fully understand
> > what the problem was before that LLD change. I'll see if I can do some
> > more investigation tomorrow (unless someone wants to beat me to it ;)
> 
> Thank you for looking into this, and sharing your analysis.
> 
> I built LLVM/clang from the master branch, rebuilt, but can still reproduce
> this.
> 
>     $ git clone --depth=1 https://github.com/llvm/llvm-project.git
>     $ cd llvm-project/
>     $ git log --oneline
>     41cb504b7 [mlir][linalg][bufferize][NFC] Move interface impl to Linalg
> Transforms
>     $ mkdir build
>     $ cd build
>     $ cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles"

Since this is something related to ld.lld, not clang, this should be:

... -DLLVM_ENABLE_PROJECTS="clang;lld" ...

> -DCMAKE_BUILD_TYPE=Release -DLLVM_INSTALL_UTILS=ON
> -DCMAKE_INSTALL_PREFIX=/scratch/local2/llvm ../llvm
>     $ make -j20
>     $ make -j20 clang-check

You can also do 'check-lld' if you want.

>     $ make install
>     $ /scratch/local2/llvm/bin/clang --version
>     clang version 15.0.0 (https://github.com/llvm/llvm-project.git
> 41cb504b7c4b18ac15830107431a0c1eec73a6b2)
>     Target: powerpc64le-unknown-linux-gnu
>     Thread model: posix
>     InstalledDir: /scratch/local2/llvm/bin
> 
> Then build Linux after `make clean` with `/scratch/local2/llvm/bin` in the
> path.
> 
>     $ LLVM=1 LLVM_IAS=0 eatmydata make -j20
> 
>     $ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=1 -net
> none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial stdio -m
> 512 -kernel /dev/shm/linux/vmlinux -append "debug_boot_weak_hash panic=-1
> console=ttyS0 rcupdate.rcu_cpu_stall_suppress_at_boot=1
> torture.disable_onoff_at_boot rcupdate.rcu_task_stall_timeout=30000
> rcupdate.rcu_self_test=1 rcutorture.onoff_interval=1000
> rcutorture.onoff_holdoff=30 rcutorture.n_barrier_cbs=4
> rcutorture.stat_interval=15 rcutorture.shutdown_secs=420
> rcutorture.test_no_idle_hz=1 rcutorture.verbose=1"
>     […]
>     Preparing to boot Linux version 5.17.0-rc5-00178-ga4b9a8fb20e7
> (pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (clang version
> 15.0.0 (https://github.com/llvm/llvm-project.git
> 41cb504b7c4b18ac15830107431a0c1eec73a6b2), LLD 13.0.0) #29 SMP PREEMPT Mon

                                             ^ still using ld.lld 13.0.0.

If you want to test the master branch, I would checkout LLVM at
460830a9c664e8cce959c660648faa7747ad8bdc, as the next commit introduces
a boot regression unrelated to this issue:

https://github.com/ClangBuiltLinux/linux/issues/1581

That should at least confirm this is resolved in a newer release.

> Feb 21 10:58:54 CET 2022
>     […]
>     [    0.465889][    T1] BUG: Kernel NULL pointer dereference on read at
> 0x00000000
>     [    0.466749][    T1] Faulting instruction address: 0xc0000000008fc300
>     [    0.467507][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
>     […]

I do intend to do further analysis at some point over the next few days
to see if I can figure out exactly why that commit that I mentioned
above fixes the issue then we can look into what we should do about it
in the kernel sources.

Cheers,
Nathan

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb)
  2022-02-21 15:29                 ` Nathan Chancellor
@ 2022-02-21 17:33                   ` Paul Menzel
  2022-04-19 21:34                   ` Nathan Chancellor
  1 sibling, 0 replies; 6+ messages in thread
From: Paul Menzel @ 2022-02-21 17:33 UTC (permalink / raw)
  To: Nathan Chancellor
  Cc: Zhouyi Zhou, Paul E. McKenney, Josh Triplett, rcu, LKML,
	David S. Miller, Jakub Kicinski, netdev, Nick Desaulniers, llvm,
	Fangrui Song

Dear Nathan,


Am 21.02.22 um 16:29 schrieb Nathan Chancellor:

> On Mon, Feb 21, 2022 at 12:17:40PM +0100, Paul Menzel wrote:
>> Am 17.02.22 um 02:16 schrieb Nathan Chancellor:
>>
>>> On Wed, Feb 16, 2022 at 02:19:51PM +0100, Paul Menzel wrote:
>>>> [Cc: +LLVM/clang build support folks]
>>
>> […]
>>
>>>> To recap: On a ppc64le machine, building Linux in Ubuntu 21.10 with *llvm*
>>>> and *clang* 1:13.0-53~exp1
>>>>
>>>>       $ clang --version
>>>>       Ubuntu clang version 13.0.0-2
>>>>       Target: powerpc64le-unknown-linux-gnu
>>>>       Thread model: posix
>>>>       InstalledDir: /usr/bin
>>>>
>>>> results in a segmentation fault, while it works when building with GCC.
>>>>
>>>>       $ gcc --version
>>>>       gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0
>>>
>>> Thank you for keying us in. I am going to have a bit of a brain dump
>>> here based on the information I have uncovered after a couple of hours
>>> of debugging.
>>>
>>> TL;DR: It seems like something is broken with __read_mostly + ld.lld
>>> before 14.0.0.
>>>
>>> My initial reproduction steps (boot-qemu.sh comes from
>>> https://github.com/ClangBuiltLinux/boot-utils):
>>>
>>> $ clang --version
>>> clang version 13.0.1 (Fedora 13.0.1-1.fc37)
>>> Target: x86_64-redhat-linux-gnu
>>> Thread model: posix
>>> InstalledDir: /usr/bin
>>>
>>> $ powerpc64le-linux-gnu-as --version
>>> GNU assembler version 2.37-2.fc36
>>> Copyright (C) 2021 Free Software Foundation, Inc.
>>> This program is free software; you may redistribute it under the terms of
>>> the GNU General Public License version 3 or later.
>>> This program has absolutely no warranty.
>>> This assembler was configured for a target of `powerpc64le-linux-gnu'.
>>>
>>> $ curl -LSso .config https://lore.kernel.org/all/f41550c7-26c0-cf81-7de9-aa924434a565@molgen.mpg.de/3-linux-5.17-rc4-rcu-dev-config.txt
>>>
>>> $ scripts/config --set-val INITRAMFS_SOURCE '""'
>>>
>>> $ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- LLVM=1 LLVM_IAS=0 all
>>>
>>> $ boot-qemu.sh -a ppc64le -k . -t 45s
>>> QEMU location: /usr/bin
>>>
>>> QEMU version: QEMU emulator version 6.2.0 (qemu-6.2.0-5.fc37)
>>>
>>> + timeout --foreground 45s stdbuf -oL -eL qemu-system-ppc64 -initrd \
>>> /home/nathan/cbl/github/boot-utils-ro/images/ppc64le/rootfs.cpio -device \
>>> ipmi-bmc-sim,id=bmc0 -device isa-ipmi-bt,bmc=bmc0,irq=10 -L \
>>> /home/nathan/cbl/github/boot-utils-ro/images/ppc64le/ -bios skiboot.lid \
>>> -machine powernv8 -display none -kernel \
>>> /home/nathan/cbl/src/linux/arch/powerpc/boot/zImage.epapr -m 2G \
>>> -nodefaults -serial mon:stdio
>>> ...
>>> [    1.478028][    T1] BUG: Kernel NULL pointer dereference on read at 0x00000000
>>> [    1.478630][    T1] Faulting instruction address: 0xc00000000090bee0
>>> [    1.479521][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
>>> [    1.480036][    T1] LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=16 NUMA PowerNV
>>> [    1.480853][    T1] Modules linked in:
>>> [    1.481265][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00001-gfa15c7cb550f #1
>>> [    1.481967][    T1] NIP:  c00000000090bee0 LR: c000000000d96b60 CTR: c0000000000d5b4c
>>> [    1.482596][    T1] REGS: c000000007443330 TRAP: 0380   Not tainted  (5.17.0-rc4-00001-gfa15c7cb550f)
>>> [    1.483305][    T1] MSR:  9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE>  CR: 22800a87  XER: 00000000
>>> [    1.484277][    T1] CFAR: c000000000d96b5c IRQMASK: 0
>>> [    1.484277][    T1] GPR00: c000000000d96b54 c0000000074435d0 c0000000028bc600 0000000000000000
>>> [    1.484277][    T1] GPR04: ffffffffffffffff ffffffffff1ea558 ffffffffff1ebfe4 c00000000261ae88
>>> [    1.484277][    T1] GPR08: 0000000000000003 0000000000000004 c00000000261ae88 0000000000000000
>>> [    1.484277][    T1] GPR12: 0000000000800000 c000000002a60000 c000000000012518 0000000000000000
>>> [    1.484277][    T1] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>>> [    1.484277][    T1] GPR20: 0000000000000000 c0000000027bff80 0000000000000cc0 0000000000000000
>>> [    1.484277][    T1] GPR24: 0000000000000010 0000000000000000 0000000000000000 0000000000000000
>>> [    1.484277][    T1] GPR28: c0000000028fcfd8 c000000007b83000 0000000000000000 c0000000074435d0
>>> [    1.490325][    T1] NIP [c00000000090bee0] strlen+0x10/0x30
>>> [    1.490788][    T1] LR [c000000000d96b60] if_nlmsg_size+0x2b0/0x390
>>> [    1.491319][    T1] Call Trace:
>>> [    1.491573][    T1] [c0000000074435d0] [c000000000d96b54] if_nlmsg_size+0x2a4/0x390 (unreliable)
>>> [    1.492291][    T1] [c000000007443680] [c000000000d96790] rtmsg_ifinfo_build_skb+0x80/0x1a0
>>> [    1.492958][    T1] [c000000007443740] [c000000000d97590] rtmsg_ifinfo+0x70/0xd0
>>> [    1.493559][    T1] [c000000007443790] [c000000000d7d528] register_netdevice+0x5d8/0x670
>>> [    1.494205][    T1] [c000000007443820] [c000000000d7d94c] register_netdev+0x4c/0x80
>>> [    1.494823][    T1] [c000000007443850] [c000000000f826d8] sit_init_net+0x1b8/0x200
>>> [    1.495426][    T1] [c0000000074438d0] [c000000000d63b5c] ops_init+0x14c/0x1c0
>>> [    1.496014][    T1] [c000000007443930] [c000000000d6314c] register_pernet_operations+0xec/0x1e0
>>> [    1.496716][    T1] [c000000007443990] [c000000000d633d0] register_pernet_device+0x60/0xd0
>>> [    1.497372][    T1] [c0000000074439e0] [c000000002085194] sit_init+0x54/0x160
>>> [    1.497950][    T1] [c000000007443a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
>>> [    1.498573][    T1] [c000000007443c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
>>> [    1.499219][    T1] [c000000007443cc0] [c00000000200604c] do_initcalls+0x84/0xe4
>>> [    1.499799][    T1] [c000000007443d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
>>> [    1.500444][    T1] [c000000007443da0] [c00000000001254c] kernel_init+0x3c/0x270
>>> [    1.501042][    T1] [c000000007443e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
>>> [    1.501721][    T1] Instruction dump:
>>> [    1.502202][    T1] eb81ffe0 7c0803a6 4e800020 00000000 00000000 00000000 60000000 60000000
>>> [    1.502934][    T1] 3883ffff 60000000 60000000 60000000 <8ca40001> 28050000 4082fff8 7c632050
>>> [    1.504028][    T1] ---[ end trace 0000000000000000 ]---
>>> ...
>>>
>>> First thing was figuring out where the NULL pointer dereference happens,
>>> which appears to the "strlen(ops->kind)" in rtnl_link_get_size():
>>>
>>> 515 static size_t rtnl_link_get_size(const struct net_device *dev)
>>> 516 {
>>> 517 	const struct rtnl_link_ops *ops = dev->rtnl_link_ops;
>>> 518 	size_t size;
>>> 519
>>> 520 	if (!ops)
>>> 521 		return 0;
>>> 522
>>> 523 	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
>>> 524 	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
>>>
>>> which I confirmed some really rudimentary printk debugging:
>>>
>>> [    1.476862][    T1] nathan: rtnl_link_get_size(): name: sit0, ops: c0000000028fcfd8, ops->kind: (null)
>>>
>>> diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
>>> index 710da8a36729..c8d928e83aec 100644
>>> --- a/net/core/rtnetlink.c
>>> +++ b/net/core/rtnetlink.c
>>> @@ -520,6 +520,9 @@ static size_t rtnl_link_get_size(const struct net_device *dev)
>>>    	if (!ops)
>>>    		return 0;
>>> +	pr_err("nathan: %s(): name: %s, ops: %px, ops->kind: %s\n", __func__,
>>> +	       dev->name, ops, ops->kind);
>>> +
>>>    	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
>>>    	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
>>>
>>> Okay... how did sit0 end up with a NULL kind...? It is very clearly
>>> defined as "sit":
>>>
>>> 1830 static struct rtnl_link_ops sit_link_ops __read_mostly = {
>>> 1831 	.kind		= "sit",
>>>
>>> Adding some more debug prints to net/ipv6/sit.c:
>>>
>>> diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
>>> index c0b138c20992..7b9edbed2fcd 100644
>>> --- a/net/ipv6/sit.c
>>> +++ b/net/ipv6/sit.c
>>> @@ -1920,6 +1920,12 @@ static int __net_init sit_init_net(struct net *net)
>>>    	 */
>>>    	sitn->fb_tunnel_dev->features |= NETIF_F_NETNS_LOCAL;
>>> +	pr_err("nathan: %s(): &sit_link_ops: %px\n", __func__, &sit_link_ops);
>>> +	pr_err("nathan: %s(): sit_link_ops.kind: %s\n", __func__, sit_link_ops.kind);
>>> +	pr_err("nathan: %s(): sit_link_ops.maxtype: %u\n", __func__, sit_link_ops.maxtype);
>>> +	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops: %px\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops);
>>> +	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: %s\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops->kind);
>>> +
>>>    	err = register_netdev(sitn->fb_tunnel_dev);
>>>    	if (err)
>>>    		goto err_reg_dev;
>>>
>>> reveals:
>>>
>>> [    1.471920][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028fcfd8
>>> [    1.472534][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: (null)
>>> [    1.473088][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
>>> [    1.473639][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028fcfd8
>>> [    1.474370][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: (null)
>>>
>>> This is super bizarre, as the maxtype member appears to have the correct
>>> value, but how is kind's initial getting dropped on the floor?
>>>
>>> Removing the __read_mostly annotation "fixes" it:
>>>
>>> [    1.481708][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000027d3f60
>>> [    1.482319][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
>>> [    1.482878][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
>>> [    1.483429][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000027d3f60
>>> [    1.484174][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
>>> ...
>>> Linux version 5.17.0-rc4-00001-g956f02ad5c31-dirty (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), LLD 13.0.1) #2 SMP PREEMPT Wed Feb 16 13:29:49 MST 2022
>>> ...
>>>
>>> diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
>>> index 7b9edbed2fcd..f109c7a0233b 100644
>>> --- a/net/ipv6/sit.c
>>> +++ b/net/ipv6/sit.c
>>> @@ -70,7 +70,7 @@ static void ipip6_tunnel_setup(struct net_device *dev);
>>>    static void ipip6_dev_free(struct net_device *dev);
>>>    static bool check_6rd(struct ip_tunnel *tunnel, const struct in6_addr *v6dst,
>>>    		      __be32 *v4dst);
>>> -static struct rtnl_link_ops sit_link_ops __read_mostly;
>>> +static struct rtnl_link_ops sit_link_ops;
>>>    static unsigned int sit_net_id __read_mostly;
>>>    struct sit_net {
>>> @@ -1827,7 +1827,7 @@ static void ipip6_dellink(struct net_device *dev, struct list_head *head)
>>>    		unregister_netdevice_queue(dev, head);
>>>    }
>>> -static struct rtnl_link_ops sit_link_ops __read_mostly = {
>>> +static struct rtnl_link_ops sit_link_ops = {
>>>    	.kind		= "sit",
>>>    	.maxtype	= IFLA_IPTUN_MAX,
>>>    	.policy		= ipip6_policy,
>>>
>>> Switching to ld.bfd also resolves it:
>>>
>>> [    1.470405][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028acfd8
>>> [    1.471016][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
>>> [    1.471534][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
>>> [    1.472062][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028acfd8
>>> [    1.472790][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
>>> ...
>>> Linux version 5.17.0-rc4-00001-g956f02ad5c31 (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), GNU ld version 2.37-2.fc36) #3 SMP PREEMPT Wed Feb 16 13:33:42 MST 2022
>>> ...
>>>
>>> I tested with ToT LLVM (or at least, close to it, since there is an
>>> unrelated ld.lld regression there) and I could not reproduce it there,
>>> so I did a reverse bisect to see what commit fixes this issue in LLVM 14
>>> and I landed on:
>>>
>>> commit 55c14d6dbfd8e7b86c15d2613fea3490078e2ae4
>>> Author: Fangrui Song <i@maskray.me>
>>> Date:   Thu Nov 25 14:12:34 2021 -0800
>>>
>>>       [ELF] Simplify DynamicSection content computation. NFC
>>>
>>>       The new code computes the content twice, but avoides the tricky
>>>       std::function<uint64_t()>. Removed 13KiB code in a Release build.
>>>
>>>    lld/ELF/SyntheticSections.cpp | 117 ++++++++++++++++--------------------------
>>>    lld/ELF/SyntheticSections.h   |  12 +----
>>>    2 files changed, 44 insertions(+), 85 deletions(-)
>>>
>>> That's... interesting, given that commit title says No Functional
>>> Change, even though there clearly is one. That commit has a couple
>>> mentions of PowerPC synthetic sections, so it is possible that the
>>> new content calculation lines up with ld.bfd?
>>>
>>> I am not really sure where to go from here, as I don't fully understand
>>> what the problem was before that LLD change. I'll see if I can do some
>>> more investigation tomorrow (unless someone wants to beat me to it ;)
>>
>> Thank you for looking into this, and sharing your analysis.
>>
>> I built LLVM/clang from the master branch, rebuilt, but can still reproduce
>> this.
>>
>>      $ git clone --depth=1 https://github.com/llvm/llvm-project.git
>>      $ cd llvm-project/
>>      $ git log --oneline
>>      41cb504b7 [mlir][linalg][bufferize][NFC] Move interface impl to Linalg Transforms
>>      $ mkdir build
>>      $ cd build
>>      $ cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles"
> 
> Since this is something related to ld.lld, not clang, this should be:
> 
> ... -DLLVM_ENABLE_PROJECTS="clang;lld" ...
> 
>> -DCMAKE_BUILD_TYPE=Release -DLLVM_INSTALL_UTILS=ON
>> -DCMAKE_INSTALL_PREFIX=/scratch/local2/llvm ../llvm
>>      $ make -j20
>>      $ make -j20 clang-check
> 
> You can also do 'check-lld' if you want.
> 
>>      $ make install
>>      $ /scratch/local2/llvm/bin/clang --version
>>      clang version 15.0.0 (https://github.com/llvm/llvm-project.git 41cb504b7c4b18ac15830107431a0c1eec73a6b2)
>>      Target: powerpc64le-unknown-linux-gnu
>>      Thread model: posix
>>      InstalledDir: /scratch/local2/llvm/bin
>>
>> Then build Linux after `make clean` with `/scratch/local2/llvm/bin` in the
>> path.
>>
>>      $ LLVM=1 LLVM_IAS=0 eatmydata make -j20
>>
>>      $ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=1 -net none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial stdio -m 512 -kernel /dev/shm/linux/vmlinux -append "debug_boot_weak_hash panic=-1 console=ttyS0 rcupdate.rcu_cpu_stall_suppress_at_boot=1 torture.disable_onoff_at_boot rcupdate.rcu_task_stall_timeout=30000 rcupdate.rcu_self_test=1 rcutorture.onoff_interval=1000 rcutorture.onoff_holdoff=30 rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15 rcutorture.shutdown_secs=420 rcutorture.test_no_idle_hz=1 rcutorture.verbose=1"
>>      […]
>>      Preparing to boot Linux version 5.17.0-rc5-00178-ga4b9a8fb20e7 (pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (clang version 15.0.0 (https://github.com/llvm/llvm-project.git 41cb504b7c4b18ac15830107431a0c1eec73a6b2), LLD 13.0.0) #29 SMP PREEMPT Mon
> 
>                                               ^ still using ld.lld 13.0.0.
> 
> If you want to test the master branch, I would checkout LLVM at
> 460830a9c664e8cce959c660648faa7747ad8bdc, as the next commit introduces
> a boot regression unrelated to this issue:
> 
> https://github.com/ClangBuiltLinux/linux/issues/1581
> 
> That should at least confirm this is resolved in a newer release.

Sorry for missing to update ld.lld. Indeed with the commit you 
mentioned, the segmentation fault is gone.

     $ /scratch/local2/llvm/bin/ld.lld --version
     LLD 14.0.0 (compatible with GNU linkers)

>> Feb 21 10:58:54 CET 2022
>>      […]
>>      [    0.465889][    T1] BUG: Kernel NULL pointer dereference on read at 0x00000000
>>      [    0.466749][    T1] Faulting instruction address: 0xc0000000008fc300
>>      [    0.467507][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
>>      […]
> 
> I do intend to do further analysis at some point over the next few days
> to see if I can figure out exactly why that commit that I mentioned
> above fixes the issue then we can look into what we should do about it
> in the kernel sources.

Awesome. Thank you for working on that.


Kind regards,

Paul

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb)
  2022-02-21 15:29                 ` Nathan Chancellor
  2022-02-21 17:33                   ` Paul Menzel
@ 2022-04-19 21:34                   ` Nathan Chancellor
  1 sibling, 0 replies; 6+ messages in thread
From: Nathan Chancellor @ 2022-04-19 21:34 UTC (permalink / raw)
  To: Paul Menzel
  Cc: Zhouyi Zhou, Paul E. McKenney, Josh Triplett, rcu, LKML,
	David S. Miller, Jakub Kicinski, netdev, Nick Desaulniers, llvm,
	Fangrui Song

On Mon, Feb 21, 2022 at 08:29:46AM -0700, Nathan Chancellor wrote:
> Hi Paul,
> 
> On Mon, Feb 21, 2022 at 12:17:40PM +0100, Paul Menzel wrote:
> > Am 17.02.22 um 02:16 schrieb Nathan Chancellor:
> > 
> > > On Wed, Feb 16, 2022 at 02:19:51PM +0100, Paul Menzel wrote:
> > > > [Cc: +LLVM/clang build support folks]
> > 
> > […]
> > 
> > > > To recap: On a ppc64le machine, building Linux in Ubuntu 21.10 with *llvm*
> > > > and *clang* 1:13.0-53~exp1
> > > > 
> > > >      $ clang --version
> > > >      Ubuntu clang version 13.0.0-2
> > > >      Target: powerpc64le-unknown-linux-gnu
> > > >      Thread model: posix
> > > >      InstalledDir: /usr/bin
> > > > 
> > > > results in a segmentation fault, while it works when building with GCC.
> > > > 
> > > >      $ gcc --version
> > > >      gcc (Ubuntu 11.2.0-7ubuntu2) 11.2.0
> > > 
> > > Thank you for keying us in. I am going to have a bit of a brain dump
> > > here based on the information I have uncovered after a couple of hours
> > > of debugging.
> > > 
> > > TL;DR: It seems like something is broken with __read_mostly + ld.lld
> > > before 14.0.0.
> > > 
> > > My initial reproduction steps (boot-qemu.sh comes from
> > > https://github.com/ClangBuiltLinux/boot-utils):
> > > 
> > > $ clang --version
> > > clang version 13.0.1 (Fedora 13.0.1-1.fc37)
> > > Target: x86_64-redhat-linux-gnu
> > > Thread model: posix
> > > InstalledDir: /usr/bin
> > > 
> > > $ powerpc64le-linux-gnu-as --version
> > > GNU assembler version 2.37-2.fc36
> > > Copyright (C) 2021 Free Software Foundation, Inc.
> > > This program is free software; you may redistribute it under the terms of
> > > the GNU General Public License version 3 or later.
> > > This program has absolutely no warranty.
> > > This assembler was configured for a target of `powerpc64le-linux-gnu'.
> > > 
> > > $ curl -LSso .config https://lore.kernel.org/all/f41550c7-26c0-cf81-7de9-aa924434a565@molgen.mpg.de/3-linux-5.17-rc4-rcu-dev-config.txt
> > > 
> > > $ scripts/config --set-val INITRAMFS_SOURCE '""'
> > > 
> > > $ make -skj"$(nproc)" ARCH=powerpc CROSS_COMPILE=powerpc64le-linux-gnu- LLVM=1 LLVM_IAS=0 all
> > > 
> > > $ boot-qemu.sh -a ppc64le -k . -t 45s
> > > QEMU location: /usr/bin
> > > 
> > > QEMU version: QEMU emulator version 6.2.0 (qemu-6.2.0-5.fc37)
> > > 
> > > + timeout --foreground 45s stdbuf -oL -eL qemu-system-ppc64 -initrd \
> > > /home/nathan/cbl/github/boot-utils-ro/images/ppc64le/rootfs.cpio -device \
> > > ipmi-bmc-sim,id=bmc0 -device isa-ipmi-bt,bmc=bmc0,irq=10 -L \
> > > /home/nathan/cbl/github/boot-utils-ro/images/ppc64le/ -bios skiboot.lid \
> > > -machine powernv8 -display none -kernel \
> > > /home/nathan/cbl/src/linux/arch/powerpc/boot/zImage.epapr -m 2G \
> > > -nodefaults -serial mon:stdio
> > > ...
> > > [    1.478028][    T1] BUG: Kernel NULL pointer dereference on read at 0x00000000
> > > [    1.478630][    T1] Faulting instruction address: 0xc00000000090bee0
> > > [    1.479521][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
> > > [    1.480036][    T1] LE PAGE_SIZE=64K MMU=Hash PREEMPT SMP NR_CPUS=16 NUMA PowerNV
> > > [    1.480853][    T1] Modules linked in:
> > > [    1.481265][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc4-00001-gfa15c7cb550f #1
> > > [    1.481967][    T1] NIP:  c00000000090bee0 LR: c000000000d96b60 CTR: c0000000000d5b4c
> > > [    1.482596][    T1] REGS: c000000007443330 TRAP: 0380   Not tainted  (5.17.0-rc4-00001-gfa15c7cb550f)
> > > [    1.483305][    T1] MSR:  9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE>  CR: 22800a87  XER: 00000000
> > > [    1.484277][    T1] CFAR: c000000000d96b5c IRQMASK: 0
> > > [    1.484277][    T1] GPR00: c000000000d96b54 c0000000074435d0 c0000000028bc600 0000000000000000
> > > [    1.484277][    T1] GPR04: ffffffffffffffff ffffffffff1ea558 ffffffffff1ebfe4 c00000000261ae88
> > > [    1.484277][    T1] GPR08: 0000000000000003 0000000000000004 c00000000261ae88 0000000000000000
> > > [    1.484277][    T1] GPR12: 0000000000800000 c000000002a60000 c000000000012518 0000000000000000
> > > [    1.484277][    T1] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> > > [    1.484277][    T1] GPR20: 0000000000000000 c0000000027bff80 0000000000000cc0 0000000000000000
> > > [    1.484277][    T1] GPR24: 0000000000000010 0000000000000000 0000000000000000 0000000000000000
> > > [    1.484277][    T1] GPR28: c0000000028fcfd8 c000000007b83000 0000000000000000 c0000000074435d0
> > > [    1.490325][    T1] NIP [c00000000090bee0] strlen+0x10/0x30
> > > [    1.490788][    T1] LR [c000000000d96b60] if_nlmsg_size+0x2b0/0x390
> > > [    1.491319][    T1] Call Trace:
> > > [    1.491573][    T1] [c0000000074435d0] [c000000000d96b54] if_nlmsg_size+0x2a4/0x390 (unreliable)
> > > [    1.492291][    T1] [c000000007443680] [c000000000d96790] rtmsg_ifinfo_build_skb+0x80/0x1a0
> > > [    1.492958][    T1] [c000000007443740] [c000000000d97590] rtmsg_ifinfo+0x70/0xd0
> > > [    1.493559][    T1] [c000000007443790] [c000000000d7d528] register_netdevice+0x5d8/0x670
> > > [    1.494205][    T1] [c000000007443820] [c000000000d7d94c] register_netdev+0x4c/0x80
> > > [    1.494823][    T1] [c000000007443850] [c000000000f826d8] sit_init_net+0x1b8/0x200
> > > [    1.495426][    T1] [c0000000074438d0] [c000000000d63b5c] ops_init+0x14c/0x1c0
> > > [    1.496014][    T1] [c000000007443930] [c000000000d6314c] register_pernet_operations+0xec/0x1e0
> > > [    1.496716][    T1] [c000000007443990] [c000000000d633d0] register_pernet_device+0x60/0xd0
> > > [    1.497372][    T1] [c0000000074439e0] [c000000002085194] sit_init+0x54/0x160
> > > [    1.497950][    T1] [c000000007443a70] [c000000000011c58] do_one_initcall+0x108/0x3e0
> > > [    1.498573][    T1] [c000000007443c70] [c000000002006190] do_initcall_level+0xe4/0x1c4
> > > [    1.499219][    T1] [c000000007443cc0] [c00000000200604c] do_initcalls+0x84/0xe4
> > > [    1.499799][    T1] [c000000007443d40] [c000000002005da8] kernel_init_freeable+0x160/0x1ec
> > > [    1.500444][    T1] [c000000007443da0] [c00000000001254c] kernel_init+0x3c/0x270
> > > [    1.501042][    T1] [c000000007443e10] [c00000000000cd64] ret_from_kernel_thread+0x5c/0x64
> > > [    1.501721][    T1] Instruction dump:
> > > [    1.502202][    T1] eb81ffe0 7c0803a6 4e800020 00000000 00000000 00000000 60000000 60000000
> > > [    1.502934][    T1] 3883ffff 60000000 60000000 60000000 <8ca40001> 28050000 4082fff8 7c632050
> > > [    1.504028][    T1] ---[ end trace 0000000000000000 ]---
> > > ...
> > > 
> > > First thing was figuring out where the NULL pointer dereference happens,
> > > which appears to the "strlen(ops->kind)" in rtnl_link_get_size():
> > > 
> > > 515 static size_t rtnl_link_get_size(const struct net_device *dev)
> > > 516 {
> > > 517 	const struct rtnl_link_ops *ops = dev->rtnl_link_ops;
> > > 518 	size_t size;
> > > 519
> > > 520 	if (!ops)
> > > 521 		return 0;
> > > 522
> > > 523 	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
> > > 524 	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
> > > 
> > > which I confirmed some really rudimentary printk debugging:
> > > 
> > > [    1.476862][    T1] nathan: rtnl_link_get_size(): name: sit0, ops: c0000000028fcfd8, ops->kind: (null)
> > > 
> > > diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
> > > index 710da8a36729..c8d928e83aec 100644
> > > --- a/net/core/rtnetlink.c
> > > +++ b/net/core/rtnetlink.c
> > > @@ -520,6 +520,9 @@ static size_t rtnl_link_get_size(const struct net_device *dev)
> > >   	if (!ops)
> > >   		return 0;
> > > +	pr_err("nathan: %s(): name: %s, ops: %px, ops->kind: %s\n", __func__,
> > > +	       dev->name, ops, ops->kind);
> > > +
> > >   	size = nla_total_size(sizeof(struct nlattr)) + /* IFLA_LINKINFO */
> > >   	       nla_total_size(strlen(ops->kind) + 1);  /* IFLA_INFO_KIND */
> > > 
> > > Okay... how did sit0 end up with a NULL kind...? It is very clearly
> > > defined as "sit":
> > > 
> > > 1830 static struct rtnl_link_ops sit_link_ops __read_mostly = {
> > > 1831 	.kind		= "sit",
> > > 
> > > Adding some more debug prints to net/ipv6/sit.c:
> > > 
> > > diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
> > > index c0b138c20992..7b9edbed2fcd 100644
> > > --- a/net/ipv6/sit.c
> > > +++ b/net/ipv6/sit.c
> > > @@ -1920,6 +1920,12 @@ static int __net_init sit_init_net(struct net *net)
> > >   	 */
> > >   	sitn->fb_tunnel_dev->features |= NETIF_F_NETNS_LOCAL;
> > > +	pr_err("nathan: %s(): &sit_link_ops: %px\n", __func__, &sit_link_ops);
> > > +	pr_err("nathan: %s(): sit_link_ops.kind: %s\n", __func__, sit_link_ops.kind);
> > > +	pr_err("nathan: %s(): sit_link_ops.maxtype: %u\n", __func__, sit_link_ops.maxtype);
> > > +	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops: %px\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops);
> > > +	pr_err("nathan: %s(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: %s\n", __func__, sitn->fb_tunnel_dev->rtnl_link_ops->kind);
> > > +
> > >   	err = register_netdev(sitn->fb_tunnel_dev);
> > >   	if (err)
> > >   		goto err_reg_dev;
> > > 
> > > reveals:
> > > 
> > > [    1.471920][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028fcfd8
> > > [    1.472534][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: (null)
> > > [    1.473088][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> > > [    1.473639][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028fcfd8
> > > [    1.474370][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: (null)
> > > 
> > > This is super bizarre, as the maxtype member appears to have the correct
> > > value, but how is kind's initial getting dropped on the floor?
> > > 
> > > Removing the __read_mostly annotation "fixes" it:
> > > 
> > > [    1.481708][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000027d3f60
> > > [    1.482319][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
> > > [    1.482878][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> > > [    1.483429][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000027d3f60
> > > [    1.484174][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
> > > ...
> > > Linux version 5.17.0-rc4-00001-g956f02ad5c31-dirty (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), LLD 13.0.1) #2 SMP PREEMPT Wed Feb 16 13:29:49 MST 2022
> > > ...
> > > 
> > > diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
> > > index 7b9edbed2fcd..f109c7a0233b 100644
> > > --- a/net/ipv6/sit.c
> > > +++ b/net/ipv6/sit.c
> > > @@ -70,7 +70,7 @@ static void ipip6_tunnel_setup(struct net_device *dev);
> > >   static void ipip6_dev_free(struct net_device *dev);
> > >   static bool check_6rd(struct ip_tunnel *tunnel, const struct in6_addr *v6dst,
> > >   		      __be32 *v4dst);
> > > -static struct rtnl_link_ops sit_link_ops __read_mostly;
> > > +static struct rtnl_link_ops sit_link_ops;
> > >   static unsigned int sit_net_id __read_mostly;
> > >   struct sit_net {
> > > @@ -1827,7 +1827,7 @@ static void ipip6_dellink(struct net_device *dev, struct list_head *head)
> > >   		unregister_netdevice_queue(dev, head);
> > >   }
> > > -static struct rtnl_link_ops sit_link_ops __read_mostly = {
> > > +static struct rtnl_link_ops sit_link_ops = {
> > >   	.kind		= "sit",
> > >   	.maxtype	= IFLA_IPTUN_MAX,
> > >   	.policy		= ipip6_policy,
> > > 
> > > Switching to ld.bfd also resolves it:
> > > 
> > > [    1.470405][    T1] sit: nathan: sit_init_net(): &sit_link_ops: c0000000028acfd8
> > > [    1.471016][    T1] sit: nathan: sit_init_net(): sit_link_ops.kind: sit
> > > [    1.471534][    T1] sit: nathan: sit_init_net(): sit_link_ops.maxtype: 20
> > > [    1.472062][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops: c0000000028acfd8
> > > [    1.472790][    T1] sit: nathan: sit_init_net(): sitn->fb_tunnel_dev->rtnl_link_ops->kind: sit
> > > ...
> > > Linux version 5.17.0-rc4-00001-g956f02ad5c31 (nathan@dev-fedora.archlinux-ax161) (clang version 13.0.1 (Fedora 13.0.1-1.fc37), GNU ld version 2.37-2.fc36) #3 SMP PREEMPT Wed Feb 16 13:33:42 MST 2022
> > > ...
> > > 
> > > I tested with ToT LLVM (or at least, close to it, since there is an
> > > unrelated ld.lld regression there) and I could not reproduce it there,
> > > so I did a reverse bisect to see what commit fixes this issue in LLVM 14
> > > and I landed on:
> > > 
> > > commit 55c14d6dbfd8e7b86c15d2613fea3490078e2ae4
> > > Author: Fangrui Song <i@maskray.me>
> > > Date:   Thu Nov 25 14:12:34 2021 -0800
> > > 
> > >      [ELF] Simplify DynamicSection content computation. NFC
> > > 
> > >      The new code computes the content twice, but avoides the tricky
> > >      std::function<uint64_t()>. Removed 13KiB code in a Release build.
> > > 
> > >   lld/ELF/SyntheticSections.cpp | 117 ++++++++++++++++--------------------------
> > >   lld/ELF/SyntheticSections.h   |  12 +----
> > >   2 files changed, 44 insertions(+), 85 deletions(-)
> > > 
> > > That's... interesting, given that commit title says No Functional
> > > Change, even though there clearly is one. That commit has a couple
> > > mentions of PowerPC synthetic sections, so it is possible that the
> > > new content calculation lines up with ld.bfd?
> > > 
> > > I am not really sure where to go from here, as I don't fully understand
> > > what the problem was before that LLD change. I'll see if I can do some
> > > more investigation tomorrow (unless someone wants to beat me to it ;)
> > 
> > Thank you for looking into this, and sharing your analysis.
> > 
> > I built LLVM/clang from the master branch, rebuilt, but can still reproduce
> > this.
> > 
> >     $ git clone --depth=1 https://github.com/llvm/llvm-project.git
> >     $ cd llvm-project/
> >     $ git log --oneline
> >     41cb504b7 [mlir][linalg][bufferize][NFC] Move interface impl to Linalg
> > Transforms
> >     $ mkdir build
> >     $ cd build
> >     $ cmake -DLLVM_ENABLE_PROJECTS=clang -G "Unix Makefiles"
> 
> Since this is something related to ld.lld, not clang, this should be:
> 
> ... -DLLVM_ENABLE_PROJECTS="clang;lld" ...
> 
> > -DCMAKE_BUILD_TYPE=Release -DLLVM_INSTALL_UTILS=ON
> > -DCMAKE_INSTALL_PREFIX=/scratch/local2/llvm ../llvm
> >     $ make -j20
> >     $ make -j20 clang-check
> 
> You can also do 'check-lld' if you want.
> 
> >     $ make install
> >     $ /scratch/local2/llvm/bin/clang --version
> >     clang version 15.0.0 (https://github.com/llvm/llvm-project.git
> > 41cb504b7c4b18ac15830107431a0c1eec73a6b2)
> >     Target: powerpc64le-unknown-linux-gnu
> >     Thread model: posix
> >     InstalledDir: /scratch/local2/llvm/bin
> > 
> > Then build Linux after `make clean` with `/scratch/local2/llvm/bin` in the
> > path.
> > 
> >     $ LLVM=1 LLVM_IAS=0 eatmydata make -j20
> > 
> >     $ qemu-system-ppc64 -enable-kvm -nographic -smp cores=1,threads=1 -net
> > none -enable-kvm -M pseries -nodefaults -device spapr-vscsi -serial stdio -m
> > 512 -kernel /dev/shm/linux/vmlinux -append "debug_boot_weak_hash panic=-1
> > console=ttyS0 rcupdate.rcu_cpu_stall_suppress_at_boot=1
> > torture.disable_onoff_at_boot rcupdate.rcu_task_stall_timeout=30000
> > rcupdate.rcu_self_test=1 rcutorture.onoff_interval=1000
> > rcutorture.onoff_holdoff=30 rcutorture.n_barrier_cbs=4
> > rcutorture.stat_interval=15 rcutorture.shutdown_secs=420
> > rcutorture.test_no_idle_hz=1 rcutorture.verbose=1"
> >     […]
> >     Preparing to boot Linux version 5.17.0-rc5-00178-ga4b9a8fb20e7
> > (pmenzel@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (clang version
> > 15.0.0 (https://github.com/llvm/llvm-project.git
> > 41cb504b7c4b18ac15830107431a0c1eec73a6b2), LLD 13.0.0) #29 SMP PREEMPT Mon
> 
>                                              ^ still using ld.lld 13.0.0.
> 
> If you want to test the master branch, I would checkout LLVM at
> 460830a9c664e8cce959c660648faa7747ad8bdc, as the next commit introduces
> a boot regression unrelated to this issue:
> 
> https://github.com/ClangBuiltLinux/linux/issues/1581
> 
> That should at least confirm this is resolved in a newer release.
> 
> > Feb 21 10:58:54 CET 2022
> >     […]
> >     [    0.465889][    T1] BUG: Kernel NULL pointer dereference on read at
> > 0x00000000
> >     [    0.466749][    T1] Faulting instruction address: 0xc0000000008fc300
> >     [    0.467507][    T1] Oops: Kernel access of bad area, sig: 11 [#1]
> >     […]
> 
> I do intend to do further analysis at some point over the next few days
> to see if I can figure out exactly why that commit that I mentioned
> above fixes the issue then we can look into what we should do about it
> in the kernel sources.

Sorry for taking so long to get back to this. For me, commit
d79976918852 ("powerpc/64: Add UADDR64 relocation support") resolves
this for ld.lld 13.x. I have started a separate thread about whether or
not this commit is suitable for stable, specifically 5.17 and 5.15:

https://lore.kernel.org/Yl8pNxSGUgeHZ1FT@dev-arch.thelio-3990X/

Cheers,
Nathan

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2022-04-19 21:34 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <159db05f-539c-fe29-608b-91b036588033@molgen.mpg.de>
     [not found] ` <CAABZP2xampOLo8k93OLgaOfv9LreJ+f0g0_1mXwqtrv_LKewQg@mail.gmail.com>
     [not found]   ` <3534d781-7d01-b42a-8974-0b1c367946f0@molgen.mpg.de>
     [not found]     ` <CAABZP2zFDY-hrZqE=-c0uW8vFMH+Q9XezYd2DcBX4Wm+sxzK1g@mail.gmail.com>
     [not found]       ` <04a597dc-64aa-57e6-f7fb-17bd2ec58159@molgen.mpg.de>
     [not found]         ` <CAABZP2yb7-xa4F_2c6tuzkv7x902wU-hqgD_pqRooGC6C7S20A@mail.gmail.com>
2022-02-16 13:19           ` BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb) Paul Menzel
2022-02-17  1:16             ` Nathan Chancellor
2022-02-21 11:17               ` Paul Menzel
2022-02-21 15:29                 ` Nathan Chancellor
2022-02-21 17:33                   ` Paul Menzel
2022-04-19 21:34                   ` Nathan Chancellor

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox