[BUG][s390x] mm: system crashed

[BUG][s390x] mm: system crashed

[Zhouping Liu]

Hi All,

I hit the below crashed when doing memory related tests[1] on s390x:

--------------- snip ---------------------
� 15929.351639¨  � <000000000021c0a6>¨ shrink_inactive_list+0x1c6/0x56c 
� 15929.351647¨  � <000000000021c69e>¨ shrink_lruvec+0x252/0x56c 
� 15929.351654¨  � <000000000021ca44>¨ shrink_zone+0x8c/0x1bc 
� 15929.351662¨  � <000000000021d080>¨ balance_pgdat+0x50c/0x658 
� 15929.351671¨  � <000000000021d318>¨ kswapd+0x14c/0x470 
� 15929.351680¨  � <0000000000158292>¨ kthread+0xda/0xe4 
� 15929.351690¨  � <000000000062a5de>¨ kernel_thread_starter+0x6/0xc 
� 15929.351700¨  � <000000000062a5d8>¨ kernel_thread_starter+0x0/0xc 
� 16109.346061¨ INFO: rcu_sched self-detected stall on CPU { 0}  (t=24006 jiffies 
 g=89766 c=89765 q=10544) 
� 16109.346101¨ CPU: 0 Tainted: G      D      3.9.0-rc6+ #1 
� 16109.346106¨ Process kswapd0 (pid: 28, task: 000000003b2a0000, ksp: 000000003b 
2ab8c0) 
� 16109.346110¨        000000000001bb60 000000000001bb70 0000000000000002 0000000 
000000000 
       000000000001bc00 000000000001bb78 000000000001bb78 00000000001009ca 
       0000000000000000 0000000000002930 000000000000000a 000000000000000a 
       000000000001bbc0 000000000001bb60 0000000000000000 0000000000000000 
       000000000063bb18 00000000001009ca 000000000001bb60 000000000001bbb0 
� 16109.346170¨ Call Trace: 
� 16109.346179¨ (� <0000000000100920>¨ show_trace+0x128/0x12c) 
� 16109.346195¨  � <00000000001cd320>¨ rcu_check_callbacks+0x458/0xccc 
� 16109.346209¨  � <0000000000140f2e>¨ update_process_times+0x4a/0x74 
� 16109.346222¨  � <0000000000199452>¨ tick_sched_handle.isra.12+0x5e/0x70 
� 16109.346235¨  � <00000000001995aa>¨ tick_sched_timer+0x6a/0x98 
� 16109.346247¨  � <000000000015c1ea>¨ __run_hrtimer+0x8e/0x200 
� 16109.346381¨  � <000000000015d1b2>¨ hrtimer_interrupt+0x212/0x2b0 
� 16109.346385¨  � <00000000001040f6>¨ clock_comparator_work+0x4a/0x54 
� 16109.346390¨  � <000000000010d658>¨ do_extint+0x158/0x15c 
� 16109.346396¨  � <000000000062aa24>¨ ext_skip+0x38/0x3c 
� 16109.346404¨  � <00000000001153c8>¨ smp_yield_cpu+0x44/0x48 
� 16109.346412¨ (� <000003d10051aec0>¨ 0x3d10051aec0) 
� 16109.346457¨  � <000000000024206a>¨ __page_check_address+0x16a/0x170 
� 16109.346466¨  � <00000000002423a2>¨ page_referenced_one+0x3e/0xa0 
� 16109.346501¨  � <000000000024427c>¨ page_referenced+0x32c/0x41c 
� 16109.346510¨  � <000000000021b1dc>¨ shrink_page_list+0x380/0xb9c 
� 16109.346521¨  � <000000000021c0a6>¨ shrink_inactive_list+0x1c6/0x56c 
� 16109.346532¨  � <000000000021c69e>¨ shrink_lruvec+0x252/0x56c 
� 16109.346542¨  � <000000000021ca44>¨ shrink_zone+0x8c/0x1bc 
� 16109.346553¨  � <000000000021d080>¨ balance_pgdat+0x50c/0x658 
� 16109.346564¨  � <000000000021d318>¨ kswapd+0x14c/0x470 
� 16109.346576¨  � <0000000000158292>¨ kthread+0xda/0xe4 
� 16109.346656¨  � <000000000062a5de>¨ kernel_thread_starter+0x6/0xc 
� 16109.346682¨  � <000000000062a5d8>¨ kernel_thread_starter+0x0/0xc 
[-- MARK -- Fri Apr 12 06:15:00 2013] 
� 16289.386061¨ INFO: rcu_sched self-detected stall on CPU { 0}  (t=42010 jiffies 
 g=89766 c=89765 q=10627) 
-------------- snip ----------------------

The testing system has 1Gb RAM, kernel is new latest mainline.
please let me know if you need any more info.

[1] reproducer is come from LTP: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/mem/mtest06/mmap2.c
    and execute it using this command: `./mmap2 -x 0.002 -a -p`

-- 
Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG][s390x] mm: system crashed

[Heiko Carstens]

On Sun, Apr 14, 2013 at 11:28:40PM -0400, Zhouping Liu wrote:
> Hi All,
> 
> I hit the below crashed when doing memory related tests[1] on s390x:
> 
> --------------- snip ---------------------
> i? 1/2  15929.351639A?  i? 1/2  <000000000021c0a6>A? shrink_inactive_list+0x1c6/0x56c 
> i? 1/2  15929.351647A?  i? 1/2  <000000000021c69e>A? shrink_lruvec+0x252/0x56c 
> i? 1/2  15929.351654A?  i? 1/2  <000000000021ca44>A? shrink_zone+0x8c/0x1bc 
> i? 1/2  15929.351662A?  i? 1/2  <000000000021d080>A? balance_pgdat+0x50c/0x658 
> i? 1/2  15929.351671A?  i? 1/2  <000000000021d318>A? kswapd+0x14c/0x470 
> i? 1/2  15929.351680A?  i? 1/2  <0000000000158292>A? kthread+0xda/0xe4 
> i? 1/2  15929.351690A?  i? 1/2  <000000000062a5de>A? kernel_thread_starter+0x6/0xc 
> i? 1/2  15929.351700A?  i? 1/2  <000000000062a5d8>A? kernel_thread_starter+0x0/0xc 
> i? 1/2  16109.346061A? INFO: rcu_sched self-detected stall on CPU { 0}  (t=24006 jiffies 
>  g=89766 c=89765 q=10544) 
> i? 1/2  16109.346101A? CPU: 0 Tainted: G      D      3.9.0-rc6+ #1 
> i? 1/2  16109.346106A? Process kswapd0 (pid: 28, task: 000000003b2a0000, ksp: 000000003b 
> 2ab8c0) 
> i? 1/2  16109.346110A?        000000000001bb60 000000000001bb70 0000000000000002 0000000 
> 000000000 
>        000000000001bc00 000000000001bb78 000000000001bb78 00000000001009ca 
>        0000000000000000 0000000000002930 000000000000000a 000000000000000a 
>        000000000001bbc0 000000000001bb60 0000000000000000 0000000000000000 
>        000000000063bb18 00000000001009ca 000000000001bb60 000000000001bbb0 
> i? 1/2  16109.346170A? Call Trace: 
> i? 1/2  16109.346179A? (i? 1/2  <0000000000100920>A? show_trace+0x128/0x12c) 
> i? 1/2  16109.346195A?  i? 1/2  <00000000001cd320>A? rcu_check_callbacks+0x458/0xccc 
> i? 1/2  16109.346209A?  i? 1/2  <0000000000140f2e>A? update_process_times+0x4a/0x74 
> i? 1/2  16109.346222A?  i? 1/2  <0000000000199452>A? tick_sched_handle.isra.12+0x5e/0x70 
> i? 1/2  16109.346235A?  i? 1/2  <00000000001995aa>A? tick_sched_timer+0x6a/0x98 
> i? 1/2  16109.346247A?  i? 1/2  <000000000015c1ea>A? __run_hrtimer+0x8e/0x200 
> i? 1/2  16109.346381A?  i? 1/2  <000000000015d1b2>A? hrtimer_interrupt+0x212/0x2b0 
> i? 1/2  16109.346385A?  i? 1/2  <00000000001040f6>A? clock_comparator_work+0x4a/0x54 
> i? 1/2  16109.346390A?  i? 1/2  <000000000010d658>A? do_extint+0x158/0x15c 
> i? 1/2  16109.346396A?  i? 1/2  <000000000062aa24>A? ext_skip+0x38/0x3c 
> i? 1/2  16109.346404A?  i? 1/2  <00000000001153c8>A? smp_yield_cpu+0x44/0x48 
> i? 1/2  16109.346412A? (i? 1/2  <000003d10051aec0>A? 0x3d10051aec0) 
> i? 1/2  16109.346457A?  i? 1/2  <000000000024206a>A? __page_check_address+0x16a/0x170 
> i? 1/2  16109.346466A?  i? 1/2  <00000000002423a2>A? page_referenced_one+0x3e/0xa0 
> i? 1/2  16109.346501A?  i? 1/2  <000000000024427c>A? page_referenced+0x32c/0x41c 
> i? 1/2  16109.346510A?  i? 1/2  <000000000021b1dc>A? shrink_page_list+0x380/0xb9c 
> i? 1/2  16109.346521A?  i? 1/2  <000000000021c0a6>A? shrink_inactive_list+0x1c6/0x56c 
> i? 1/2  16109.346532A?  i? 1/2  <000000000021c69e>A? shrink_lruvec+0x252/0x56c 
> i? 1/2  16109.346542A?  i? 1/2  <000000000021ca44>A? shrink_zone+0x8c/0x1bc 
> i? 1/2  16109.346553A?  i? 1/2  <000000000021d080>A? balance_pgdat+0x50c/0x658 
> i? 1/2  16109.346564A?  i? 1/2  <000000000021d318>A? kswapd+0x14c/0x470 
> i? 1/2  16109.346576A?  i? 1/2  <0000000000158292>A? kthread+0xda/0xe4 
> i? 1/2  16109.346656A?  i? 1/2  <000000000062a5de>A? kernel_thread_starter+0x6/0xc 
> i? 1/2  16109.346682A?  i? 1/2  <000000000062a5d8>A? kernel_thread_starter+0x0/0xc 
> [-- MARK -- Fri Apr 12 06:15:00 2013] 
> i? 1/2  16289.386061A? INFO: rcu_sched self-detected stall on CPU { 0}  (t=42010 jiffies 
>  g=89766 c=89765 q=10627) 

Did the system really crash or did you just see the rcu related warning(s)?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG][s390x] mm: system crashed

[Zhouping Liu]

On 04/15/2013 01:56 PM, Heiko Carstens wrote:
> On Sun, Apr 14, 2013 at 11:28:40PM -0400, Zhouping Liu wrote:
>> Hi All,
>>
>> I hit the below crashed when doing memory related tests[1] on s390x:
>>
>> --------------- snip ---------------------
>> i? 1/2  15929.351639A?  i? 1/2  <000000000021c0a6>A? shrink_inactive_list+0x1c6/0x56c
>> i? 1/2  15929.351647A?  i? 1/2  <000000000021c69e>A? shrink_lruvec+0x252/0x56c
>> i? 1/2  15929.351654A?  i? 1/2  <000000000021ca44>A? shrink_zone+0x8c/0x1bc
>> i? 1/2  15929.351662A?  i? 1/2  <000000000021d080>A? balance_pgdat+0x50c/0x658
>> i? 1/2  15929.351671A?  i? 1/2  <000000000021d318>A? kswapd+0x14c/0x470
>> i? 1/2  15929.351680A?  i? 1/2  <0000000000158292>A? kthread+0xda/0xe4
>> i? 1/2  15929.351690A?  i? 1/2  <000000000062a5de>A? kernel_thread_starter+0x6/0xc
>> i? 1/2  15929.351700A?  i? 1/2  <000000000062a5d8>A? kernel_thread_starter+0x0/0xc
>> i? 1/2  16109.346061A? INFO: rcu_sched self-detected stall on CPU { 0}  (t=24006 jiffies
>>   g=89766 c=89765 q=10544)
>> i? 1/2  16109.346101A? CPU: 0 Tainted: G      D      3.9.0-rc6+ #1
>> i? 1/2  16109.346106A? Process kswapd0 (pid: 28, task: 000000003b2a0000, ksp: 000000003b
>> 2ab8c0)
>> i? 1/2  16109.346110A?        000000000001bb60 000000000001bb70 0000000000000002 0000000
>> 000000000
>>         000000000001bc00 000000000001bb78 000000000001bb78 00000000001009ca
>>         0000000000000000 0000000000002930 000000000000000a 000000000000000a
>>         000000000001bbc0 000000000001bb60 0000000000000000 0000000000000000
>>         000000000063bb18 00000000001009ca 000000000001bb60 000000000001bbb0
>> i? 1/2  16109.346170A? Call Trace:
>> i? 1/2  16109.346179A? (i? 1/2  <0000000000100920>A? show_trace+0x128/0x12c)
>> i? 1/2  16109.346195A?  i? 1/2  <00000000001cd320>A? rcu_check_callbacks+0x458/0xccc
>> i? 1/2  16109.346209A?  i? 1/2  <0000000000140f2e>A? update_process_times+0x4a/0x74
>> i? 1/2  16109.346222A?  i? 1/2  <0000000000199452>A? tick_sched_handle.isra.12+0x5e/0x70
>> i? 1/2  16109.346235A?  i? 1/2  <00000000001995aa>A? tick_sched_timer+0x6a/0x98
>> i? 1/2  16109.346247A?  i? 1/2  <000000000015c1ea>A? __run_hrtimer+0x8e/0x200
>> i? 1/2  16109.346381A?  i? 1/2  <000000000015d1b2>A? hrtimer_interrupt+0x212/0x2b0
>> i? 1/2  16109.346385A?  i? 1/2  <00000000001040f6>A? clock_comparator_work+0x4a/0x54
>> i? 1/2  16109.346390A?  i? 1/2  <000000000010d658>A? do_extint+0x158/0x15c
>> i? 1/2  16109.346396A?  i? 1/2  <000000000062aa24>A? ext_skip+0x38/0x3c
>> i? 1/2  16109.346404A?  i? 1/2  <00000000001153c8>A? smp_yield_cpu+0x44/0x48
>> i? 1/2  16109.346412A? (i? 1/2  <000003d10051aec0>A? 0x3d10051aec0)
>> i? 1/2  16109.346457A?  i? 1/2  <000000000024206a>A? __page_check_address+0x16a/0x170
>> i? 1/2  16109.346466A?  i? 1/2  <00000000002423a2>A? page_referenced_one+0x3e/0xa0
>> i? 1/2  16109.346501A?  i? 1/2  <000000000024427c>A? page_referenced+0x32c/0x41c
>> i? 1/2  16109.346510A?  i? 1/2  <000000000021b1dc>A? shrink_page_list+0x380/0xb9c
>> i? 1/2  16109.346521A?  i? 1/2  <000000000021c0a6>A? shrink_inactive_list+0x1c6/0x56c
>> i? 1/2  16109.346532A?  i? 1/2  <000000000021c69e>A? shrink_lruvec+0x252/0x56c
>> i? 1/2  16109.346542A?  i? 1/2  <000000000021ca44>A? shrink_zone+0x8c/0x1bc
>> i? 1/2  16109.346553A?  i? 1/2  <000000000021d080>A? balance_pgdat+0x50c/0x658
>> i? 1/2  16109.346564A?  i? 1/2  <000000000021d318>A? kswapd+0x14c/0x470
>> i? 1/2  16109.346576A?  i? 1/2  <0000000000158292>A? kthread+0xda/0xe4
>> i? 1/2  16109.346656A?  i? 1/2  <000000000062a5de>A? kernel_thread_starter+0x6/0xc
>> i? 1/2  16109.346682A?  i? 1/2  <000000000062a5d8>A? kernel_thread_starter+0x0/0xc
>> [-- MARK -- Fri Apr 12 06:15:00 2013]
>> i? 1/2  16289.386061A? INFO: rcu_sched self-detected stall on CPU { 0}  (t=42010 jiffies
>>   g=89766 c=89765 q=10627)
> Did the system really crash or did you just see the rcu related warning(s)?

I just check it again, actually at first the system didn't really crash, 
but the system is very slow in response.
and the reproducer process can't be killed, after I did some common 
actions such as 'ls' 'vim' etc, the system
seemed to be really crashed, no any response.

also in the previous testing, I can remember that the system would be no 
any response for a long time, just only
repeatedly print out the such above 'Call Trace' into console.

Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG][s390x] mm: system crashed

[Heiko Carstens]

On Mon, Apr 15, 2013 at 02:16:55PM +0800, Zhouping Liu wrote:
> On 04/15/2013 01:56 PM, Heiko Carstens wrote:
> >On Sun, Apr 14, 2013 at 11:28:40PM -0400, Zhouping Liu wrote:
> >>i? 1/2  16109.346170A? Call Trace:
> >>i? 1/2  16109.346179A? (i? 1/2  <0000000000100920>A? show_trace+0x128/0x12c)
> >>i? 1/2  16109.346195A?  i? 1/2  <00000000001cd320>A? rcu_check_callbacks+0x458/0xccc
> >>i? 1/2  16109.346209A?  i? 1/2  <0000000000140f2e>A? update_process_times+0x4a/0x74
> >>i? 1/2  16109.346222A?  i? 1/2  <0000000000199452>A? tick_sched_handle.isra.12+0x5e/0x70
> >>i? 1/2  16109.346235A?  i? 1/2  <00000000001995aa>A? tick_sched_timer+0x6a/0x98
> >>i? 1/2  16109.346247A?  i? 1/2  <000000000015c1ea>A? __run_hrtimer+0x8e/0x200
> >>i? 1/2  16109.346381A?  i? 1/2  <000000000015d1b2>A? hrtimer_interrupt+0x212/0x2b0
> >>i? 1/2  16109.346385A?  i? 1/2  <00000000001040f6>A? clock_comparator_work+0x4a/0x54
> >>i? 1/2  16109.346390A?  i? 1/2  <000000000010d658>A? do_extint+0x158/0x15c
> >>i? 1/2  16109.346396A?  i? 1/2  <000000000062aa24>A? ext_skip+0x38/0x3c
> >>i? 1/2  16109.346404A?  i? 1/2  <00000000001153c8>A? smp_yield_cpu+0x44/0x48
> >>i? 1/2  16109.346412A? (i? 1/2  <000003d10051aec0>A? 0x3d10051aec0)
> >>i? 1/2  16109.346457A?  i? 1/2  <000000000024206a>A? __page_check_address+0x16a/0x170
> >>i? 1/2  16109.346466A?  i? 1/2  <00000000002423a2>A? page_referenced_one+0x3e/0xa0
> >>i? 1/2  16109.346501A?  i? 1/2  <000000000024427c>A? page_referenced+0x32c/0x41c
> >>i? 1/2  16109.346510A?  i? 1/2  <000000000021b1dc>A? shrink_page_list+0x380/0xb9c
> >>i? 1/2  16109.346521A?  i? 1/2  <000000000021c0a6>A? shrink_inactive_list+0x1c6/0x56c
> >>i? 1/2  16109.346532A?  i? 1/2  <000000000021c69e>A? shrink_lruvec+0x252/0x56c
> >>i? 1/2  16109.346542A?  i? 1/2  <000000000021ca44>A? shrink_zone+0x8c/0x1bc
> >>i? 1/2  16109.346553A?  i? 1/2  <000000000021d080>A? balance_pgdat+0x50c/0x658
> >>i? 1/2  16109.346564A?  i? 1/2  <000000000021d318>A? kswapd+0x14c/0x470
> >>i? 1/2  16109.346576A?  i? 1/2  <0000000000158292>A? kthread+0xda/0xe4
> >>i? 1/2  16109.346656A?  i? 1/2  <000000000062a5de>A? kernel_thread_starter+0x6/0xc
> >>i? 1/2  16109.346682A?  i? 1/2  <000000000062a5d8>A? kernel_thread_starter+0x0/0xc
> >>[-- MARK -- Fri Apr 12 06:15:00 2013]
> >>i? 1/2  16289.386061A? INFO: rcu_sched self-detected stall on CPU { 0}  (t=42010 jiffies
> >>  g=89766 c=89765 q=10627)
> >Did the system really crash or did you just see the rcu related warning(s)?
> 
> I just check it again, actually at first the system didn't really
> crash, but the system is very slow in response.
> and the reproducer process can't be killed, after I did some common
> actions such as 'ls' 'vim' etc, the system
> seemed to be really crashed, no any response.
> 
> also in the previous testing, I can remember that the system would
> be no any response for a long time, just only
> repeatedly print out the such above 'Call Trace' into console.

Ok, thanks.
Just a couple of more questions: did you see this also on other archs, or just
s390 (if you tried other platforms at all).

If you have some time, could you please repeat your test with the kernel
command line option " user_mode=home "?

As far as I can tell there was only one s390 patch merged that was
mmap related: 486c0a0bc80d370471b21662bf03f04fbb37cdc6 "s390/mm: Fix crst
upgrade of mmap with MAP_FIXED".
Even though I don't think it explains the bug you've seen it might be worth
to try to revert it.

And at last, can you share your kernel config?

Thanks,
Heiko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG][s390x] mm: system crashed

[Simon Jeons]

Hi Heiko,
On 04/16/2013 03:50 PM, Heiko Carstens wrote:
> On Mon, Apr 15, 2013 at 02:16:55PM +0800, Zhouping Liu wrote:
>> On 04/15/2013 01:56 PM, Heiko Carstens wrote:
>>> On Sun, Apr 14, 2013 at 11:28:40PM -0400, Zhouping Liu wrote:
>>>> i? 1/2  16109.346170A? Call Trace:
>>>> i? 1/2  16109.346179A? (i? 1/2  <0000000000100920>A? show_trace+0x128/0x12c)
>>>> i? 1/2  16109.346195A?  i? 1/2  <00000000001cd320>A? rcu_check_callbacks+0x458/0xccc
>>>> i? 1/2  16109.346209A?  i? 1/2  <0000000000140f2e>A? update_process_times+0x4a/0x74
>>>> i? 1/2  16109.346222A?  i? 1/2  <0000000000199452>A? tick_sched_handle.isra.12+0x5e/0x70
>>>> i? 1/2  16109.346235A?  i? 1/2  <00000000001995aa>A? tick_sched_timer+0x6a/0x98
>>>> i? 1/2  16109.346247A?  i? 1/2  <000000000015c1ea>A? __run_hrtimer+0x8e/0x200
>>>> i? 1/2  16109.346381A?  i? 1/2  <000000000015d1b2>A? hrtimer_interrupt+0x212/0x2b0
>>>> i? 1/2  16109.346385A?  i? 1/2  <00000000001040f6>A? clock_comparator_work+0x4a/0x54
>>>> i? 1/2  16109.346390A?  i? 1/2  <000000000010d658>A? do_extint+0x158/0x15c
>>>> i? 1/2  16109.346396A?  i? 1/2  <000000000062aa24>A? ext_skip+0x38/0x3c
>>>> i? 1/2  16109.346404A?  i? 1/2  <00000000001153c8>A? smp_yield_cpu+0x44/0x48
>>>> i? 1/2  16109.346412A? (i? 1/2  <000003d10051aec0>A? 0x3d10051aec0)
>>>> i? 1/2  16109.346457A?  i? 1/2  <000000000024206a>A? __page_check_address+0x16a/0x170
>>>> i? 1/2  16109.346466A?  i? 1/2  <00000000002423a2>A? page_referenced_one+0x3e/0xa0
>>>> i? 1/2  16109.346501A?  i? 1/2  <000000000024427c>A? page_referenced+0x32c/0x41c
>>>> i? 1/2  16109.346510A?  i? 1/2  <000000000021b1dc>A? shrink_page_list+0x380/0xb9c
>>>> i? 1/2  16109.346521A?  i? 1/2  <000000000021c0a6>A? shrink_inactive_list+0x1c6/0x56c
>>>> i? 1/2  16109.346532A?  i? 1/2  <000000000021c69e>A? shrink_lruvec+0x252/0x56c
>>>> i? 1/2  16109.346542A?  i? 1/2  <000000000021ca44>A? shrink_zone+0x8c/0x1bc
>>>> i? 1/2  16109.346553A?  i? 1/2  <000000000021d080>A? balance_pgdat+0x50c/0x658
>>>> i? 1/2  16109.346564A?  i? 1/2  <000000000021d318>A? kswapd+0x14c/0x470
>>>> i? 1/2  16109.346576A?  i? 1/2  <0000000000158292>A? kthread+0xda/0xe4
>>>> i? 1/2  16109.346656A?  i? 1/2  <000000000062a5de>A? kernel_thread_starter+0x6/0xc
>>>> i? 1/2  16109.346682A?  i? 1/2  <000000000062a5d8>A? kernel_thread_starter+0x0/0xc
>>>> [-- MARK -- Fri Apr 12 06:15:00 2013]
>>>> i? 1/2  16289.386061A? INFO: rcu_sched self-detected stall on CPU { 0}  (t=42010 jiffies
>>>>   g=89766 c=89765 q=10627)
>>> Did the system really crash or did you just see the rcu related warning(s)?
>> I just check it again, actually at first the system didn't really
>> crash, but the system is very slow in response.
>> and the reproducer process can't be killed, after I did some common
>> actions such as 'ls' 'vim' etc, the system
>> seemed to be really crashed, no any response.
>>
>> also in the previous testing, I can remember that the system would
>> be no any response for a long time, just only
>> repeatedly print out the such above 'Call Trace' into console.
> Ok, thanks.
> Just a couple of more questions: did you see this also on other archs, or just
> s390 (if you tried other platforms at all).
>
> If you have some time, could you please repeat your test with the kernel
> command line option " user_mode=home "?

What's the meaning of this command line? I can't find it in 
Documentation/kernel-parameters.txt/

>
> As far as I can tell there was only one s390 patch merged that was
> mmap related: 486c0a0bc80d370471b21662bf03f04fbb37cdc6 "s390/mm: Fix crst
> upgrade of mmap with MAP_FIXED".
> Even though I don't think it explains the bug you've seen it might be worth
> to try to revert it.
>
> And at last, can you share your kernel config?
>
> Thanks,
> Heiko
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG][s390x] mm: system crashed

[Heiko Carstens]

On Tue, Apr 16, 2013 at 03:56:59PM +0800, Simon Jeons wrote:
> Hi Heiko,
> >If you have some time, could you please repeat your test with the kernel
> >command line option " user_mode=home "?
> 
> What's the meaning of this command line? I can't find it in
> Documentation/kernel-parameters.txt/

It switches the architectural address space where kernel and user space
reside in.
We only recently switched the default address space for user space from
home space to primary space, since that's needed for kvm.
The user space runs in home space mode will be removed in the future; we
keep it currently as fallback, just in case something breaks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG][s390x] mm: system crashed

[Zhouping Liu]

[-- Attachment #1: Type: text/plain, Size: 3927 bytes --]

On 04/16/2013 03:50 PM, Heiko Carstens wrote:
> On Mon, Apr 15, 2013 at 02:16:55PM +0800, Zhouping Liu wrote:
>> On 04/15/2013 01:56 PM, Heiko Carstens wrote:
>>> On Sun, Apr 14, 2013 at 11:28:40PM -0400, Zhouping Liu wrote:
>>>> i? 1/2  16109.346170A? Call Trace:
>>>> i? 1/2  16109.346179A? (i? 1/2  <0000000000100920>A? show_trace+0x128/0x12c)
>>>> i? 1/2  16109.346195A?  i? 1/2  <00000000001cd320>A? rcu_check_callbacks+0x458/0xccc
>>>> i? 1/2  16109.346209A?  i? 1/2  <0000000000140f2e>A? update_process_times+0x4a/0x74
>>>> i? 1/2  16109.346222A?  i? 1/2  <0000000000199452>A? tick_sched_handle.isra.12+0x5e/0x70
>>>> i? 1/2  16109.346235A?  i? 1/2  <00000000001995aa>A? tick_sched_timer+0x6a/0x98
>>>> i? 1/2  16109.346247A?  i? 1/2  <000000000015c1ea>A? __run_hrtimer+0x8e/0x200
>>>> i? 1/2  16109.346381A?  i? 1/2  <000000000015d1b2>A? hrtimer_interrupt+0x212/0x2b0
>>>> i? 1/2  16109.346385A?  i? 1/2  <00000000001040f6>A? clock_comparator_work+0x4a/0x54
>>>> i? 1/2  16109.346390A?  i? 1/2  <000000000010d658>A? do_extint+0x158/0x15c
>>>> i? 1/2  16109.346396A?  i? 1/2  <000000000062aa24>A? ext_skip+0x38/0x3c
>>>> i? 1/2  16109.346404A?  i? 1/2  <00000000001153c8>A? smp_yield_cpu+0x44/0x48
>>>> i? 1/2  16109.346412A? (i? 1/2  <000003d10051aec0>A? 0x3d10051aec0)
>>>> i? 1/2  16109.346457A?  i? 1/2  <000000000024206a>A? __page_check_address+0x16a/0x170
>>>> i? 1/2  16109.346466A?  i? 1/2  <00000000002423a2>A? page_referenced_one+0x3e/0xa0
>>>> i? 1/2  16109.346501A?  i? 1/2  <000000000024427c>A? page_referenced+0x32c/0x41c
>>>> i? 1/2  16109.346510A?  i? 1/2  <000000000021b1dc>A? shrink_page_list+0x380/0xb9c
>>>> i? 1/2  16109.346521A?  i? 1/2  <000000000021c0a6>A? shrink_inactive_list+0x1c6/0x56c
>>>> i? 1/2  16109.346532A?  i? 1/2  <000000000021c69e>A? shrink_lruvec+0x252/0x56c
>>>> i? 1/2  16109.346542A?  i? 1/2  <000000000021ca44>A? shrink_zone+0x8c/0x1bc
>>>> i? 1/2  16109.346553A?  i? 1/2  <000000000021d080>A? balance_pgdat+0x50c/0x658
>>>> i? 1/2  16109.346564A?  i? 1/2  <000000000021d318>A? kswapd+0x14c/0x470
>>>> i? 1/2  16109.346576A?  i? 1/2  <0000000000158292>A? kthread+0xda/0xe4
>>>> i? 1/2  16109.346656A?  i? 1/2  <000000000062a5de>A? kernel_thread_starter+0x6/0xc
>>>> i? 1/2  16109.346682A?  i? 1/2  <000000000062a5d8>A? kernel_thread_starter+0x0/0xc
>>>> [-- MARK -- Fri Apr 12 06:15:00 2013]
>>>> i? 1/2  16289.386061A? INFO: rcu_sched self-detected stall on CPU { 0}  (t=42010 jiffies
>>>>   g=89766 c=89765 q=10627)
>>> Did the system really crash or did you just see the rcu related warning(s)?
>> I just check it again, actually at first the system didn't really
>> crash, but the system is very slow in response.
>> and the reproducer process can't be killed, after I did some common
>> actions such as 'ls' 'vim' etc, the system
>> seemed to be really crashed, no any response.
>>
>> also in the previous testing, I can remember that the system would
>> be no any response for a long time, just only
>> repeatedly print out the such above 'Call Trace' into console.
> Ok, thanks.
> Just a couple of more questions: did you see this also on other archs, or just
> s390 (if you tried other platforms at all).

Didn't meet the issue on other arches, just only on s390x, I will check 
further on other arches.

>
> If you have some time, could you please repeat your test with the kernel
> command line option " user_mode=home "?

Sure, but the testing machine is not in my hand now, I will test it ASAP 
when I get it, also I'll
try it on other s390x machine.

>
> As far as I can tell there was only one s390 patch merged that was
> mmap related: 486c0a0bc80d370471b21662bf03f04fbb37cdc6 "s390/mm: Fix crst
> upgrade of mmap with MAP_FIXED".
> Even though I don't think it explains the bug you've seen it might be worth
> to try to revert it.

OK, I'll try it later.

>
> And at last, can you share your kernel config?

Of course.

Thanks,
Zhouping

>
> Thanks,
> Heiko
>


[-- Attachment #2: config_s390x --]
[-- Type: text/plain, Size: 46781 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/s390 3.9.0-rc6 Kernel Configuration
#
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_KEXEC=y
CONFIG_AUDIT_ARCH=y
CONFIG_NO_IOPORT=y
# CONFIG_PCI_QUIRKS is not set
CONFIG_S390=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_FHANDLE=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y
CONFIG_AUDIT_LOGINUID_IMMUTABLE=y
# CONFIG_ALWAYS_USE_PERSISTENT_CLOCK is not set
CONFIG_GENERIC_TIME_VSYSCALL_OLD=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

#
# CPU/Task time and stats accounting
#
CONFIG_VIRT_CPU_ACCOUNTING=y
CONFIG_VIRT_CPU_ACCOUNTING_NATIVE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_FANOUT=64
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_RCU_FAST_NO_HZ is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_RCU_NOCB_CPU is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=17
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
CONFIG_MEMCG_KMEM=y
# CONFIG_CGROUP_HUGETLB is not set
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_CHECKPOINT_RESTORE is not set
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_SCHED_AUTOGROUP=y
CONFIG_MM_OWNER=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EXPERT is not set
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_HAVE_64BIT_ALIGNED_ACCESS=y
CONFIG_HAVE_SYSCALL_WRAPPERS=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
# CONFIG_HAVE_DMA_ATTRS is not set
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_MUTEX_CPU_RELAX=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_CLONE_BACKWARDS2=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_OLD_SIGACTION=y
CONFIG_COMPAT_OLD_SIGACTION=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_MODULE_SIG=y
# CONFIG_MODULE_SIG_FORCE is not set
CONFIG_MODULE_SIG_ALL=y
CONFIG_MODULE_SIG_SHA1=y
# CONFIG_MODULE_SIG_SHA224 is not set
# CONFIG_MODULE_SIG_SHA256 is not set
# CONFIG_MODULE_SIG_SHA384 is not set
# CONFIG_MODULE_SIG_SHA512 is not set
CONFIG_MODULE_SIG_HASH="sha1"
CONFIG_INIT_ALL_POSSIBLE=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_THROTTLING=y

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_IBM_PARTITION=y
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_DEFAULT_DEADLINE=y
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_PADATA=y
CONFIG_ASN1=y
CONFIG_ARCH_INLINE_SPIN_TRYLOCK=y
CONFIG_ARCH_INLINE_SPIN_TRYLOCK_BH=y
CONFIG_ARCH_INLINE_SPIN_LOCK=y
CONFIG_ARCH_INLINE_SPIN_LOCK_BH=y
CONFIG_ARCH_INLINE_SPIN_LOCK_IRQ=y
CONFIG_ARCH_INLINE_SPIN_LOCK_IRQSAVE=y
CONFIG_ARCH_INLINE_SPIN_UNLOCK=y
CONFIG_ARCH_INLINE_SPIN_UNLOCK_BH=y
CONFIG_ARCH_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_ARCH_INLINE_SPIN_UNLOCK_IRQRESTORE=y
CONFIG_ARCH_INLINE_READ_TRYLOCK=y
CONFIG_ARCH_INLINE_READ_LOCK=y
CONFIG_ARCH_INLINE_READ_LOCK_BH=y
CONFIG_ARCH_INLINE_READ_LOCK_IRQ=y
CONFIG_ARCH_INLINE_READ_LOCK_IRQSAVE=y
CONFIG_ARCH_INLINE_READ_UNLOCK=y
CONFIG_ARCH_INLINE_READ_UNLOCK_BH=y
CONFIG_ARCH_INLINE_READ_UNLOCK_IRQ=y
CONFIG_ARCH_INLINE_READ_UNLOCK_IRQRESTORE=y
CONFIG_ARCH_INLINE_WRITE_TRYLOCK=y
CONFIG_ARCH_INLINE_WRITE_LOCK=y
CONFIG_ARCH_INLINE_WRITE_LOCK_BH=y
CONFIG_ARCH_INLINE_WRITE_LOCK_IRQ=y
CONFIG_ARCH_INLINE_WRITE_LOCK_IRQSAVE=y
CONFIG_ARCH_INLINE_WRITE_UNLOCK=y
CONFIG_ARCH_INLINE_WRITE_UNLOCK_BH=y
CONFIG_ARCH_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_ARCH_INLINE_WRITE_UNLOCK_IRQRESTORE=y
CONFIG_INLINE_SPIN_TRYLOCK=y
CONFIG_INLINE_SPIN_TRYLOCK_BH=y
CONFIG_INLINE_SPIN_LOCK=y
CONFIG_INLINE_SPIN_LOCK_BH=y
CONFIG_INLINE_SPIN_LOCK_IRQ=y
CONFIG_INLINE_SPIN_LOCK_IRQSAVE=y
CONFIG_INLINE_SPIN_UNLOCK_BH=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE=y
CONFIG_INLINE_READ_TRYLOCK=y
CONFIG_INLINE_READ_LOCK=y
CONFIG_INLINE_READ_LOCK_BH=y
CONFIG_INLINE_READ_LOCK_IRQ=y
CONFIG_INLINE_READ_LOCK_IRQSAVE=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_BH=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK_IRQRESTORE=y
CONFIG_INLINE_WRITE_TRYLOCK=y
CONFIG_INLINE_WRITE_LOCK=y
CONFIG_INLINE_WRITE_LOCK_BH=y
CONFIG_INLINE_WRITE_LOCK_IRQ=y
CONFIG_INLINE_WRITE_LOCK_IRQSAVE=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_BH=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_HAVE_MARCH_Z900_FEATURES=y
CONFIG_HAVE_MARCH_Z990_FEATURES=y
CONFIG_HAVE_MARCH_Z9_109_FEATURES=y
# CONFIG_HAVE_MARCH_Z10_FEATURES is not set
# CONFIG_HAVE_MARCH_Z196_FEATURES is not set
# CONFIG_HAVE_MARCH_ZEC12_FEATURES is not set
# CONFIG_MARCH_Z900 is not set
# CONFIG_MARCH_Z990 is not set
CONFIG_MARCH_Z9_109=y
# CONFIG_MARCH_Z10 is not set
# CONFIG_MARCH_Z196 is not set
# CONFIG_MARCH_ZEC12 is not set
CONFIG_64BIT=y
CONFIG_COMPAT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_KEYS_COMPAT=y
CONFIG_SMP=y
CONFIG_NR_CPUS=64
CONFIG_HOTPLUG_CPU=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_BOOK=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y

#
# Memory setup
#
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_FORCE_MAX_ZONEORDER=9
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_MEMORY_ISOLATION=y
# CONFIG_HAVE_BOOTMEM_INFO_NODE is not set
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_BALLOON_COMPACTION=y
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
# CONFIG_TRANSPARENT_HUGEPAGE is not set
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_PACK_STACK=y
# CONFIG_SMALL_STACK is not set
CONFIG_CHECK_STACK=y
CONFIG_STACK_GUARD=256
# CONFIG_WARN_DYNAMIC_STACK is not set

#
# I/O subsystem
#
CONFIG_QDIO=m
# CONFIG_PCI is not set
# CONFIG_PCI_DOMAINS is not set
# CONFIG_HAS_IOMEM is not set
# CONFIG_IOMMU_HELPER is not set
# CONFIG_HAS_DMA is not set
# CONFIG_NEED_SG_DMA_LENGTH is not set
# CONFIG_NEED_DMA_MAP_STATE is not set
CONFIG_CHSC_SCH=m
CONFIG_SCM_BUS=y
CONFIG_EADM_SCH=m

#
# Dump support
#
CONFIG_CRASH_DUMP=y
# CONFIG_ZFCPDUMP is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=m
CONFIG_COREDUMP=y
CONFIG_SECCOMP=y

#
# Power Management
#
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_ARCH_SAVE_PAGE_KEYS=y
CONFIG_PM_STD_PARTITION="/dev/jokes"
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM_RUNTIME=y
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_DIAG is not set
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=y
CONFIG_XFRM_USER=y
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_STATISTICS=y
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_IUCV=y
CONFIG_AFIUCV=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_FIB_TRIE_STATS=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_NET_IPVTI=m
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_INET_UDP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_MIP6=m
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_SIT_6RD=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
# CONFIG_IPV6_GRE is not set
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y
CONFIG_IPV6_PIMSM_V2=y
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETWORK_PHY_TIMESTAMPING=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
# CONFIG_NF_CONNTRACK_TIMEOUT is not set
CONFIG_NF_CONNTRACK_TIMESTAMP=y
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
# CONFIG_NF_CT_NETLINK_TIMEOUT is not set
CONFIG_NF_CT_NETLINK_HELPER=m
CONFIG_NETFILTER_NETLINK_QUEUE_CT=y
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XTABLES=y

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_SET=m

#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_AUDIT=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_HMARK=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LOG=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NETMAP=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_REDIRECT=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
# CONFIG_NETFILTER_XT_MATCH_CONNLABEL is not set
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_NFACCT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
CONFIG_IP_SET=m
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=m
CONFIG_IP_SET_BITMAP_IPMAC=m
CONFIG_IP_SET_BITMAP_PORT=m
CONFIG_IP_SET_HASH_IP=m
CONFIG_IP_SET_HASH_IPPORT=m
CONFIG_IP_SET_HASH_IPPORTIP=m
CONFIG_IP_SET_HASH_IPPORTNET=m
CONFIG_IP_SET_HASH_NET=m
CONFIG_IP_SET_HASH_NETPORT=m
CONFIG_IP_SET_HASH_NETIFACE=m
CONFIG_IP_SET_LIST_SET=m
CONFIG_IP_VS=m
CONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
# CONFIG_NF_CONNTRACK_PROC_COMPAT is not set
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_RPFILTER=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT_IPV4=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RPFILTER=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
CONFIG_NF_NAT_IPV6=m
# CONFIG_IP6_NF_TARGET_MASQUERADE is not set
# CONFIG_IP6_NF_TARGET_NPT is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m

#
# DCCP CCIDs Configuration
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_TFRC_LIB=y

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
# CONFIG_NET_DCCPPROBE is not set
CONFIG_IP_SCTP=m
CONFIG_NET_SCTPPROBE=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5 is not set
CONFIG_SCTP_DEFAULT_COOKIE_HMAC_SHA1=y
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_NONE is not set
CONFIG_SCTP_COOKIE_HMAC_MD5=y
CONFIG_SCTP_COOKIE_HMAC_SHA1=y
CONFIG_RDS=m
CONFIG_RDS_TCP=m
# CONFIG_RDS_DEBUG is not set
# CONFIG_TIPC is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
CONFIG_L2TP_V3=y
CONFIG_L2TP_IP=m
CONFIG_L2TP_ETH=m
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_BRIDGE_VLAN_FILTERING is not set
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
# CONFIG_VLAN_8021Q_MVRP is not set
# CONFIG_DECNET is not set
CONFIG_LLC=m
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
CONFIG_IEEE802154=m
CONFIG_IEEE802154_6LOWPAN=m
CONFIG_MAC802154=m
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
CONFIG_NET_SCH_MQPRIO=m
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_QFQ=m
CONFIG_NET_SCH_CODEL=m
CONFIG_NET_SCH_FQ_CODEL=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_SCH_PLUG=m

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_EMATCH_IPSET=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
CONFIG_DCB=y
CONFIG_DNS_RESOLVER=m
CONFIG_BATMAN_ADV=m
CONFIG_BATMAN_ADV_BLA=y
CONFIG_BATMAN_ADV_DAT=y
# CONFIG_BATMAN_ADV_DEBUG is not set
CONFIG_OPENVSWITCH=m
# CONFIG_VSOCKETS is not set
CONFIG_RPS=y
CONFIG_XPS=y
CONFIG_NETPRIO_CGROUP=m
CONFIG_BQL=y
CONFIG_BPF_JIT=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
CONFIG_NET_DROP_MONITOR=y
# CONFIG_CAN is not set
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
CONFIG_WIMAX=m
CONFIG_WIMAX_DEBUG_LEVEL=8
CONFIG_RFKILL=m
CONFIG_RFKILL_INPUT=y
# CONFIG_NET_9P is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
CONFIG_NFC=m
CONFIG_NFC_NCI=m
CONFIG_NFC_HCI=m
CONFIG_NFC_SHDLC=y
CONFIG_NFC_LLCP=y

#
# Near Field Communication (NFC) devices
#
# CONFIG_NFC_PN544 is not set
# CONFIG_NFC_MICROREAD is not set
CONFIG_HAVE_BPF_JIT=y
# CONFIG_PCMCIA is not set
CONFIG_CCW=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_FIRMWARE_IN_KERNEL is not set
CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
CONFIG_SYS_HYPERVISOR=y
# CONFIG_GENERIC_CPU_DEVICES is not set
# CONFIG_DMA_SHARED_BUFFER is not set

#
# Bus devices
#
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP_MIN_COUNT=8
CONFIG_BLK_DEV_CRYPTOLOOP=m
# CONFIG_BLK_DEV_DRBD is not set
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_OSD=m
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set

#
# S/390 block device drivers
#
CONFIG_BLK_DEV_XPRAM=m
CONFIG_DCSSBLK=m
CONFIG_DASD=m
CONFIG_DASD_PROFILE=y
CONFIG_DASD_ECKD=m
CONFIG_DASD_FBA=m
CONFIG_DASD_DIAG=m
CONFIG_DASD_EER=y
CONFIG_SCM_BLOCK=m
CONFIG_SCM_BLOCK_CLUSTER_WRITE=y
CONFIG_VIRTIO_BLK=m
# CONFIG_BLK_DEV_RBD is not set

#
# Misc devices
#
# CONFIG_SENSORS_LIS3LV02D is not set
CONFIG_ENCLOSURE_SERVICES=m
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_93CX6 is not set

#
# Texas Instruments shared transport line discipline
#

#
# Altera FPGA firmware download module
#

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
# CONFIG_SCSI_DMA is not set
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_FC_TGT_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=m
CONFIG_SCSI_SRP_TGT_ATTRS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
# CONFIG_SCSI_UFSHCD is not set
# CONFIG_LIBFC is not set
# CONFIG_LIBFCOE is not set
CONFIG_SCSI_DEBUG=m
CONFIG_ZFCP=m
CONFIG_SCSI_VIRTIO=m
CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_RDAC=y
CONFIG_SCSI_DH_HP_SW=y
CONFIG_SCSI_DH_EMC=y
CONFIG_SCSI_DH_ALUA=y
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
# CONFIG_SCSI_OSD_DEBUG is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
# CONFIG_MD_MULTIPATH is not set
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=m
CONFIG_DM_BIO_PRISON=m
CONFIG_DM_PERSISTENT_DATA=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
# CONFIG_DM_DEBUG_BLOCK_STACK_TRACING is not set
# CONFIG_DM_CACHE is not set
CONFIG_DM_MIRROR=m
CONFIG_DM_RAID=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
CONFIG_DM_DELAY=m
CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
CONFIG_TCM_PSCSI=m
CONFIG_LOOPBACK_TARGET=m
CONFIG_ISCSI_TARGET=m
CONFIG_NETDEVICES=y
CONFIG_NET_CORE=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
# CONFIG_EQUALIZER is not set
# CONFIG_MII is not set
CONFIG_IFB=m
CONFIG_NET_TEAM=m
CONFIG_NET_TEAM_MODE_BROADCAST=m
CONFIG_NET_TEAM_MODE_ROUNDROBIN=m
CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=m
CONFIG_NET_TEAM_MODE_LOADBALANCE=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
CONFIG_VXLAN=m
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=m
CONFIG_VETH=m
CONFIG_VIRTIO_NET=m
CONFIG_ATM_DRIVERS=y
# CONFIG_ATM_DUMMY is not set
CONFIG_ATM_TCP=m

#
# CAIF transport drivers
#
CONFIG_ETHERNET=y
CONFIG_NET_VENDOR_INTEL=y
# CONFIG_NET_VENDOR_I825XX is not set
# CONFIG_NET_VENDOR_MARVELL is not set
CONFIG_NET_VENDOR_NATSEMI=y
CONFIG_NET_VENDOR_8390=y
# CONFIG_PHYLIB is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set

#
# S/390 network device drivers
#
CONFIG_LCS=m
CONFIG_CTCM=m
# CONFIG_NETIUCV is not set
CONFIG_SMSGIUCV=m
CONFIG_SMSGIUCV_EVENT=m
# CONFIG_CLAW is not set
CONFIG_QETH=m
CONFIG_QETH_L2=m
CONFIG_QETH_L3=m
CONFIG_QETH_IPV6=y
CONFIG_CCWGROUP=m

#
# WiMAX Wireless Broadband devices
#

#
# Enable USB support to see WiMAX USB drivers
#
CONFIG_WAN=y
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
# CONFIG_HDLC_RAW_ETH is not set
CONFIG_HDLC_CISCO=m
CONFIG_HDLC_FR=m
CONFIG_HDLC_PPP=m

#
# X.25/LAPB support is disabled
#
CONFIG_DLCI=m
CONFIG_DLCI_MAX=8
CONFIG_IEEE802154_DRIVERS=m
CONFIG_IEEE802154_FAKEHARD=m
CONFIG_IEEE802154_FAKELB=m

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
# CONFIG_INPUT_SPARSEKMAP is not set
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_SERPORT=y
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_SERIO_ALTERA_PS2 is not set
# CONFIG_SERIO_PS2MULT is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_TTY=y
CONFIG_UNIX98_PTYS=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_N_GSM=m
# CONFIG_TRACE_SINK is not set
# CONFIG_DEVKMEM is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IUCV=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_VIRTIO=m
# CONFIG_R3964 is not set
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=8192
CONFIG_HANGCHECK_TIMER=m

#
# S/390 character device drivers
#
CONFIG_TN3270=y
CONFIG_TN3270_TTY=y
CONFIG_TN3270_FS=m
CONFIG_TN3270_CONSOLE=y
CONFIG_TN3215=y
CONFIG_TN3215_CONSOLE=y
CONFIG_CCW_CONSOLE=y
CONFIG_SCLP_TTY=y
CONFIG_SCLP_CONSOLE=y
CONFIG_SCLP_VT220_TTY=y
CONFIG_SCLP_VT220_CONSOLE=y
CONFIG_SCLP_CPI=m
CONFIG_SCLP_ASYNC=m
CONFIG_S390_TAPE=m

#
# S/390 tape hardware support
#
CONFIG_S390_TAPE_34XX=m
CONFIG_S390_TAPE_3590=m
CONFIG_VMLOGRDR=m
CONFIG_VMCP=y
CONFIG_MONREADER=m
CONFIG_MONWRITER=m
CONFIG_S390_VMUR=m
# CONFIG_I2C is not set
# CONFIG_HSI is not set

#
# PPS support
#
CONFIG_PPS=m
# CONFIG_PPS_DEBUG is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=m

#
# Enable PHYLIB and NETWORK_PHY_TIMESTAMPING to see the additional clocks.
#
# CONFIG_PTP_1588_CLOCK_PCH is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_BQ27x00 is not set
CONFIG_POWER_RESET=y
# CONFIG_POWER_AVS is not set
# CONFIG_THERMAL is not set
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_ZVM_WATCHDOG=m
# CONFIG_REGULATOR is not set

#
# HID support
#
CONFIG_HID=y
# CONFIG_HID_BATTERY_STRENGTH is not set
# CONFIG_HIDRAW is not set
# CONFIG_UHID is not set
CONFIG_HID_GENERIC=y

#
# Special HID drivers
#
# CONFIG_USB_ARCH_HAS_OHCI is not set
# CONFIG_USB_ARCH_HAS_EHCI is not set
# CONFIG_USB_ARCH_HAS_XHCI is not set
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set

#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
# CONFIG_MSPRO_BLOCK is not set

#
# MemoryStick Host Controller Drivers
#
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
CONFIG_AUXDISPLAY=y
# CONFIG_UIO is not set
CONFIG_VIRTIO=y

#
# Virtio drivers
#
CONFIG_VIRTIO_BALLOON=m

#
# Microsoft Hyper-V guest support
#
CONFIG_STAGING=y
# CONFIG_ECHO is not set
# CONFIG_COMEDI is not set
# CONFIG_ZSMALLOC is not set
# CONFIG_FT1000 is not set

#
# Speakup console speech
#
CONFIG_STAGING_MEDIA=y

#
# Android
#
# CONFIG_ANDROID is not set
# CONFIG_WIMAX_GDM72XX is not set
# CONFIG_DGRP is not set
# CONFIG_ZCACHE is not set

#
# Hardware Spinlock drivers
#
# CONFIG_MAILBOX is not set
CONFIG_IOMMU_SUPPORT=y

#
# Remoteproc drivers
#

#
# Rpmsg drivers
#
# CONFIG_VIRT_DRIVERS is not set
# CONFIG_PM_DEVFREQ is not set
# CONFIG_EXTCON is not set
# CONFIG_MEMORY is not set
# CONFIG_PWM is not set

#
# File systems
#
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT23=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_NILFS2_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_AUTOFS4_FS=y
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_GENERIC_ACL=y

#
# Caches
#
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
CONFIG_FSCACHE_OBJECT_LIST=y
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=y
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_LOGFS is not set
CONFIG_CRAMFS=m
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_4K_DEVBLK_SIZE is not set
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_MINIX_FS_NATIVE_ENDIAN is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX6FS_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_PSTORE=y
# CONFIG_PSTORE_CONSOLE is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_EXOFS_FS is not set
# CONFIG_F2FS_FS is not set
CONFIG_ORE=m
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V2=m
CONFIG_NFS_V3=m
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_PNFS_OBJLAYOUT=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
CONFIG_NFS_FSCACHE=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFS_DEBUG=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_SUNRPC_DEBUG=y
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=m
CONFIG_CIFS_STATS=y
# CONFIG_CIFS_STATS2 is not set
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_CIFS_ACL=y
CONFIG_CIFS_DEBUG=y
# CONFIG_CIFS_DEBUG2 is not set
CONFIG_CIFS_DFS_UPCALL=y
CONFIG_CIFS_SMB2=y
# CONFIG_CIFS_FSCACHE is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_MAC_ROMAN=m
CONFIG_NLS_MAC_CELTIC=m
CONFIG_NLS_MAC_CENTEURO=m
CONFIG_NLS_MAC_CROATIAN=m
CONFIG_NLS_MAC_CYRILLIC=m
CONFIG_NLS_MAC_GAELIC=m
CONFIG_NLS_MAC_GREEK=m
CONFIG_NLS_MAC_ICELAND=m
CONFIG_NLS_MAC_INUIT=m
CONFIG_NLS_MAC_ROMANIAN=m
CONFIG_NLS_MAC_TURKISH=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4
# CONFIG_ENABLE_WARN_DEPRECATED is not set
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_READABLE_ASM is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_DEBUG_KERNEL=y
# CONFIG_PANIC_ON_OOPS is not set
CONFIG_PANIC_ON_OOPS_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_ATOMIC_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_VM=y
# CONFIG_DEBUG_VM_RB is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set

#
# RCU Debugging
#
CONFIG_SPARSE_RCU_POINTER=y
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=60
# CONFIG_RCU_CPU_STALL_INFO is not set
# CONFIG_RCU_TRACE is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_LKDTM is not set
# CONFIG_NOTIFIER_ERROR_INJECTION is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_LATENCYTOP=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_FUNCTION_TRACER is not set
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SCHED_TRACER=y
# CONFIG_FTRACE_SYSCALLS is not set
# CONFIG_TRACER_SNAPSHOT is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_STACK_TRACER is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_PROBE_EVENTS=y
# CONFIG_FTRACE_STARTUP_TEST is not set
CONFIG_RING_BUFFER_BENCHMARK=m
# CONFIG_RBTREE_TEST is not set
# CONFIG_INTERVAL_TREE_TEST is not set
CONFIG_BUILD_DOCSRC=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_ATOMIC64_SELFTEST=y
CONFIG_ASYNC_RAID6_TEST=m
# CONFIG_SAMPLES is not set
CONFIG_TEST_KSTRTOX=y
CONFIG_STRICT_DEVMEM=y
# CONFIG_DEBUG_STRICT_USER_COPY_CHECKS is not set
# CONFIG_S390_PTDUMP is not set
CONFIG_DEBUG_SET_MODULE_RONX=y

#
# Security options
#
CONFIG_KEYS=y
CONFIG_ENCRYPTED_KEYS=m
CONFIG_KEYS_DEBUG_PROC_KEYS=y
# CONFIG_SECURITY_DMESG_RESTRICT is not set
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
# CONFIG_SECURITY_PATH is not set
CONFIG_LSM_MMAP_MIN_ADDR=65535
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
# CONFIG_SECURITY_APPARMOR is not set
# CONFIG_SECURITY_YAMA is not set
# CONFIG_IMA is not set
# CONFIG_EVM is not set
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=m
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
# CONFIG_CRYPTO_MANAGER_DISABLE_TESTS is not set
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=y

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32 is not set
CONFIG_CRYPTO_GHASH=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_ZLIB=m
CONFIG_CRYPTO_LZO=m

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_USER_API=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_USER_API_SKCIPHER=y
CONFIG_CRYPTO_HW=y
CONFIG_ZCRYPT=m
CONFIG_CRYPTO_SHA1_S390=m
CONFIG_CRYPTO_SHA256_S390=m
CONFIG_CRYPTO_SHA512_S390=m
CONFIG_CRYPTO_DES_S390=m
CONFIG_CRYPTO_AES_S390=m
CONFIG_S390_PRNG=m
CONFIG_CRYPTO_GHASH_S390=m
CONFIG_ASYMMETRIC_KEY_TYPE=y
CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y
CONFIG_PUBLIC_KEY_ALGO_RSA=y
CONFIG_X509_CERTIFICATE_PARSER=y
CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
# CONFIG_GENERIC_IO is not set
CONFIG_CRC_CCITT=m
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=m
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=m
CONFIG_CRC8=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_DQL=y
CONFIG_NLATTR=y
CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y
# CONFIG_AVERAGE is not set
CONFIG_CLZ_TAB=y
CONFIG_CORDIC=m
# CONFIG_DDR is not set
CONFIG_MPILIB=y
CONFIG_OID_REGISTRY=y

#
# Virtualization
#
CONFIG_PFAULT=y
CONFIG_CMM=m
CONFIG_CMM_IUCV=y
CONFIG_APPLDATA_BASE=y
CONFIG_APPLDATA_MEM=m
CONFIG_APPLDATA_OS=m
CONFIG_APPLDATA_NET_SUM=m
CONFIG_S390_HYPFS_FS=y
CONFIG_HAVE_KVM=y
CONFIG_VIRTUALIZATION=y
# CONFIG_KVM is not set
CONFIG_VHOST_NET=m
CONFIG_TCM_VHOST=m
CONFIG_S390_GUEST=y

Re: [BUG][s390x] mm: system crashed

[Zhouping Liu]

Hello Heiko,

----- Original Message -----
> From: "Heiko Carstens" <heiko.carstens@de.ibm.com>
> To: "Zhouping Liu" <zliu@redhat.com>
> Cc: linux-mm@kvack.org, "LKML" <linux-kernel@vger.kernel.org>, "caiqian" <caiqian@redhat.com>, "Caspar Zhang"
> <czhang@redhat.com>, "Martin Schwidefsky" <schwidefsky@de.ibm.com>
> Sent: Tuesday, April 16, 2013 3:50:47 PM
> Subject: Re: [BUG][s390x] mm: system crashed
> 
> On Mon, Apr 15, 2013 at 02:16:55PM +0800, Zhouping Liu wrote:
> > On 04/15/2013 01:56 PM, Heiko Carstens wrote:
> > >On Sun, Apr 14, 2013 at 11:28:40PM -0400, Zhouping Liu wrote:
> > >>� 16109.346170¨ Call Trace:
> > >>� 16109.346179¨ (� <0000000000100920>¨ show_trace+0x128/0x12c)
> > >>� 16109.346195¨  � <00000000001cd320>¨ rcu_check_callbacks+0x458/0xccc
> > >>� 16109.346209¨  � <0000000000140f2e>¨ update_process_times+0x4a/0x74
> > >>� 16109.346222¨  � <0000000000199452>¨
> > >>tick_sched_handle.isra.12+0x5e/0x70
> > >>� 16109.346235¨  � <00000000001995aa>¨ tick_sched_timer+0x6a/0x98
> > >>� 16109.346247¨  � <000000000015c1ea>¨ __run_hrtimer+0x8e/0x200
> > >>� 16109.346381¨  � <000000000015d1b2>¨ hrtimer_interrupt+0x212/0x2b0
> > >>� 16109.346385¨  � <00000000001040f6>¨ clock_comparator_work+0x4a/0x54
> > >>� 16109.346390¨  � <000000000010d658>¨ do_extint+0x158/0x15c
> > >>� 16109.346396¨  � <000000000062aa24>¨ ext_skip+0x38/0x3c
> > >>� 16109.346404¨  � <00000000001153c8>¨ smp_yield_cpu+0x44/0x48
> > >>� 16109.346412¨ (� <000003d10051aec0>¨ 0x3d10051aec0)
> > >>� 16109.346457¨  � <000000000024206a>¨ __page_check_address+0x16a/0x170
> > >>� 16109.346466¨  � <00000000002423a2>¨ page_referenced_one+0x3e/0xa0
> > >>� 16109.346501¨  � <000000000024427c>¨ page_referenced+0x32c/0x41c
> > >>� 16109.346510¨  � <000000000021b1dc>¨ shrink_page_list+0x380/0xb9c
> > >>� 16109.346521¨  � <000000000021c0a6>¨ shrink_inactive_list+0x1c6/0x56c
> > >>� 16109.346532¨  � <000000000021c69e>¨ shrink_lruvec+0x252/0x56c
> > >>� 16109.346542¨  � <000000000021ca44>¨ shrink_zone+0x8c/0x1bc
> > >>� 16109.346553¨  � <000000000021d080>¨ balance_pgdat+0x50c/0x658
> > >>� 16109.346564¨  � <000000000021d318>¨ kswapd+0x14c/0x470
> > >>� 16109.346576¨  � <0000000000158292>¨ kthread+0xda/0xe4
> > >>� 16109.346656¨  � <000000000062a5de>¨ kernel_thread_starter+0x6/0xc
> > >>� 16109.346682¨  � <000000000062a5d8>¨ kernel_thread_starter+0x0/0xc
> > >>[-- MARK -- Fri Apr 12 06:15:00 2013]
> > >>� 16289.386061¨ INFO: rcu_sched self-detected stall on CPU { 0}  (t=42010
> > >>jiffies
> > >>  g=89766 c=89765 q=10627)
> > >Did the system really crash or did you just see the rcu related
> > >warning(s)?
> > 
> > I just check it again, actually at first the system didn't really
> > crash, but the system is very slow in response.
> > and the reproducer process can't be killed, after I did some common
> > actions such as 'ls' 'vim' etc, the system
> > seemed to be really crashed, no any response.
> > 
> > also in the previous testing, I can remember that the system would
> > be no any response for a long time, just only
> > repeatedly print out the such above 'Call Trace' into console.
> 
> Ok, thanks.
> Just a couple of more questions: did you see this also on other archs, or
> just
> s390 (if you tried other platforms at all).
> 
> If you have some time, could you please repeat your test with the kernel
> command line option " user_mode=home "?

I tested the system with the kernel parameter, but the issue still appeared,
I just to say it takes longer time to reproduce the issue than the before.

> 
> As far as I can tell there was only one s390 patch merged that was
> mmap related: 486c0a0bc80d370471b21662bf03f04fbb37cdc6 "s390/mm: Fix crst
> upgrade of mmap with MAP_FIXED".

also I tested the revert commit, unluckily, the same issue as the before.


-- 
Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [BUG][s390x] mm: system crashed

[Heiko Carstens]

On Thu, Apr 18, 2013 at 02:27:45AM -0400, Zhouping Liu wrote:
> Hello Heiko,
> > If you have some time, could you please repeat your test with the kernel
> > command line option " user_mode=home "?
> 
> I tested the system with the kernel parameter, but the issue still appeared,
> I just to say it takes longer time to reproduce the issue than the before.
> 
> > 
> > As far as I can tell there was only one s390 patch merged that was
> > mmap related: 486c0a0bc80d370471b21662bf03f04fbb37cdc6 "s390/mm: Fix crst
> > upgrade of mmap with MAP_FIXED".
> 
> also I tested the revert commit, unluckily, the same issue as the before.

Ok, thanks for verifying! I'll look into it; hopefully I can reproduce it
here as well.

Thanks!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[v3.9-rc8]: kernel BUG at mm/memcontrol.c:3994! (was: Re: [BUG][s390x] mm: system crashed)

[Heiko Carstens]

On Thu, Apr 18, 2013 at 09:13:03AM +0200, Heiko Carstens wrote:
> Ok, thanks for verifying! I'll look into it; hopefully I can reproduce it
> here as well.

That seems to be a common code bug. I can easily trigger the VM_BUG_ON()
below (when I force the system to swap):

[   48.347963] ------------[ cut here ]------------
[   48.347972] kernel BUG at mm/memcontrol.c:3994!
[   48.348012] illegal operation: 0001 [#1] SMP 
[   48.348015] Modules linked in:
[   48.348017] CPU: 1 Not tainted 3.9.0-rc8+ #38
[   48.348020] Process mmap2 (pid: 635, task: 0000000029476100, ksp: 000000002e91b938)
[   48.348022] Krnl PSW : 0704f00180000000 000000000026552c (__mem_cgroup_uncharge_common+0x2c4/0x33c)
[   48.348032]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 EA:3
               Krnl GPRS: 0000000000000008 0000000000000009 000003d1002a9200 0000000000000000
[   48.348039]            0000000000000000 00000000006812d8 000003ffdf339000 00000000321a6f98
[   48.348043]            000003fffce11000 0000000000000000 0000000000000001 000003d1002a9200
[   48.348046]            0000000000000001 0000000000681b88 000000002e91bc18 000000002e91bbd0
[   48.348057] Krnl Code: 000000000026551e: c0e5fffaa2a1        brasl   %r14,1b9a60
                          0000000000265524: a7f4ff7d            brc     15,26541e
                         #0000000000265528: a7f40001            brc     15,26552a
                         >000000000026552c: e3c0b8200124        stg     %r12,6176(%r11)
                          0000000000265532: a7f4ff57            brc     15,2653e0
                          0000000000265536: e310b8280104        lg      %r1,6184(%r11)
                          000000000026553c: a71b0001            aghi    %r1,1
                          0000000000265540: e310b8280124        stg     %r1,6184(%r11)
[   48.348099] Call Trace:
[   48.348100] ([<000003d1002a91c0>] 0x3d1002a91c0)
[   48.348102]  [<00000000002404aa>] page_remove_rmap+0xf2/0x16c
[   48.348106]  [<0000000000232dc8>] unmap_single_vma+0x494/0x7d8
[   48.348107]  [<0000000000233ac0>] unmap_vmas+0x50/0x74
[   48.348109]  [<00000000002396ec>] unmap_region+0x9c/0x110
[   48.348110]  [<000000000023bd18>] do_munmap+0x284/0x470
[   48.348111]  [<000000000023bf56>] vm_munmap+0x52/0x70
[   48.348113]  [<000000000023cf32>] SyS_munmap+0x3a/0x4c
[   48.348114]  [<0000000000665e14>] sysc_noemu+0x22/0x28
[   48.348118]  [<000003fffcf187b2>] 0x3fffcf187b2
[   48.348119] Last Breaking-Event-Address:
[   48.348120]  [<0000000000265528>] __mem_cgroup_uncharge_common+0x2c0/0x33c

Looking at the code, the code flow is:

page_remove_rmap() -> mem_cgroup_uncharge_page() -> __mem_cgroup_uncharge_common()

Note that in mem_cgroup_uncharge_page() the page in question passed the check:

[...]
        if (PageSwapCache(page))
                return;
[...]

and just a couple of instructions later the VM_BUG_ON() within
__mem_cgroup_uncharge_common() triggers:

[...]
        if (mem_cgroup_disabled())
                return NULL;

        VM_BUG_ON(PageSwapCache(page));
[...]

Which means that another cpu changed the pageflags concurrently. In fact,
looking at the dump a different cpu is indeed busy with running kswapd.

So.. this seems to be somewhat broken. Anyone familiar with memcontrol?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [v3.9-rc8]: kernel BUG at mm/memcontrol.c:3994! (was: Re: [BUG][s390x] mm: system crashed)

[Michal Hocko]

On Wed 24-04-13 12:42:55, Heiko Carstens wrote:
> On Thu, Apr 18, 2013 at 09:13:03AM +0200, Heiko Carstens wrote:
> > Ok, thanks for verifying! I'll look into it; hopefully I can reproduce it
> > here as well.
> 
> That seems to be a common code bug. I can easily trigger the VM_BUG_ON()
> below (when I force the system to swap):
> 
> [   48.347963] ------------[ cut here ]------------
> [   48.347972] kernel BUG at mm/memcontrol.c:3994!
> [   48.348012] illegal operation: 0001 [#1] SMP 
> [   48.348015] Modules linked in:
> [   48.348017] CPU: 1 Not tainted 3.9.0-rc8+ #38
> [   48.348020] Process mmap2 (pid: 635, task: 0000000029476100, ksp: 000000002e91b938)
> [   48.348022] Krnl PSW : 0704f00180000000 000000000026552c (__mem_cgroup_uncharge_common+0x2c4/0x33c)
> [   48.348032]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 EA:3
>                Krnl GPRS: 0000000000000008 0000000000000009 000003d1002a9200 0000000000000000
> [   48.348039]            0000000000000000 00000000006812d8 000003ffdf339000 00000000321a6f98
> [   48.348043]            000003fffce11000 0000000000000000 0000000000000001 000003d1002a9200
> [   48.348046]            0000000000000001 0000000000681b88 000000002e91bc18 000000002e91bbd0
> [   48.348057] Krnl Code: 000000000026551e: c0e5fffaa2a1        brasl   %r14,1b9a60
>                           0000000000265524: a7f4ff7d            brc     15,26541e
>                          #0000000000265528: a7f40001            brc     15,26552a
>                          >000000000026552c: e3c0b8200124        stg     %r12,6176(%r11)
>                           0000000000265532: a7f4ff57            brc     15,2653e0
>                           0000000000265536: e310b8280104        lg      %r1,6184(%r11)
>                           000000000026553c: a71b0001            aghi    %r1,1
>                           0000000000265540: e310b8280124        stg     %r1,6184(%r11)
> [   48.348099] Call Trace:
> [   48.348100] ([<000003d1002a91c0>] 0x3d1002a91c0)
> [   48.348102]  [<00000000002404aa>] page_remove_rmap+0xf2/0x16c
> [   48.348106]  [<0000000000232dc8>] unmap_single_vma+0x494/0x7d8
> [   48.348107]  [<0000000000233ac0>] unmap_vmas+0x50/0x74
> [   48.348109]  [<00000000002396ec>] unmap_region+0x9c/0x110
> [   48.348110]  [<000000000023bd18>] do_munmap+0x284/0x470
> [   48.348111]  [<000000000023bf56>] vm_munmap+0x52/0x70
> [   48.348113]  [<000000000023cf32>] SyS_munmap+0x3a/0x4c
> [   48.348114]  [<0000000000665e14>] sysc_noemu+0x22/0x28
> [   48.348118]  [<000003fffcf187b2>] 0x3fffcf187b2
> [   48.348119] Last Breaking-Event-Address:
> [   48.348120]  [<0000000000265528>] __mem_cgroup_uncharge_common+0x2c0/0x33c
> 
> Looking at the code, the code flow is:
> 
> page_remove_rmap() -> mem_cgroup_uncharge_page() -> __mem_cgroup_uncharge_common()
> 
> Note that in mem_cgroup_uncharge_page() the page in question passed the check:
> 
> [...]
>         if (PageSwapCache(page))
>                 return;
> [...]
> 
> and just a couple of instructions later the VM_BUG_ON() within
> __mem_cgroup_uncharge_common() triggers:
> 
> [...]
>         if (mem_cgroup_disabled())
>                 return NULL;
> 
>         VM_BUG_ON(PageSwapCache(page));
> [...]
> 
> Which means that another cpu changed the pageflags concurrently. In fact,
> looking at the dump a different cpu is indeed busy with running kswapd.

Hmm, maybe I am missing something but it really looks like we can race
here. Reclaim path takes the page lock while zap_pte takes page table
lock so nothing prevents them from racing here:
shrink_page_list		zap_pte_range
  trylock_page			  pte_offset_map_lock
  add_to_swap			    page_remove_rmap
    /* Page can be still mapped */
    add_to_swap_cache		      atomic_add_negative(_mapcount)
      __add_to_swap_cache	        mem_cgroup_uncharge_page
      				          (PageSwapCache(page)) && return
        SetPageSwapCache
				          __mem_cgroup_uncharge_common
					    VM_BUG_ON(PageSwapCache(page))

Maybe not many people run with CONFIG_DEBUG_VM enabled these days so we
do not this more often (even me testing configs are not consistent in
that regards and only few have it on). The only thing that changed in
this area recently is 0c59b89c which made the test VM_BUG_ON rather then
simple return in 3.6
And maybe the BUG_ON is too harsh as CgroupUsed should guarantee that
the uncharge will eventually go away. What do you think Johannes?
 
> So.. this seems to be somewhat broken. Anyone familiar with memcontrol?

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [v3.9-rc8]: kernel BUG at mm/memcontrol.c:3994! (was: Re: [BUG][s390x] mm: system crashed)

[Johannes Weiner]

On Wed, Apr 24, 2013 at 03:18:51PM +0200, Michal Hocko wrote:
> On Wed 24-04-13 12:42:55, Heiko Carstens wrote:
> > On Thu, Apr 18, 2013 at 09:13:03AM +0200, Heiko Carstens wrote:
> > > Ok, thanks for verifying! I'll look into it; hopefully I can reproduce it
> > > here as well.
> > 
> > That seems to be a common code bug. I can easily trigger the VM_BUG_ON()
> > below (when I force the system to swap):
> > 
> > [   48.347963] ------------[ cut here ]------------
> > [   48.347972] kernel BUG at mm/memcontrol.c:3994!
> > [   48.348012] illegal operation: 0001 [#1] SMP 
> > [   48.348015] Modules linked in:
> > [   48.348017] CPU: 1 Not tainted 3.9.0-rc8+ #38
> > [   48.348020] Process mmap2 (pid: 635, task: 0000000029476100, ksp: 000000002e91b938)
> > [   48.348022] Krnl PSW : 0704f00180000000 000000000026552c (__mem_cgroup_uncharge_common+0x2c4/0x33c)
> > [   48.348032]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 EA:3
> >                Krnl GPRS: 0000000000000008 0000000000000009 000003d1002a9200 0000000000000000
> > [   48.348039]            0000000000000000 00000000006812d8 000003ffdf339000 00000000321a6f98
> > [   48.348043]            000003fffce11000 0000000000000000 0000000000000001 000003d1002a9200
> > [   48.348046]            0000000000000001 0000000000681b88 000000002e91bc18 000000002e91bbd0
> > [   48.348057] Krnl Code: 000000000026551e: c0e5fffaa2a1        brasl   %r14,1b9a60
> >                           0000000000265524: a7f4ff7d            brc     15,26541e
> >                          #0000000000265528: a7f40001            brc     15,26552a
> >                          >000000000026552c: e3c0b8200124        stg     %r12,6176(%r11)
> >                           0000000000265532: a7f4ff57            brc     15,2653e0
> >                           0000000000265536: e310b8280104        lg      %r1,6184(%r11)
> >                           000000000026553c: a71b0001            aghi    %r1,1
> >                           0000000000265540: e310b8280124        stg     %r1,6184(%r11)
> > [   48.348099] Call Trace:
> > [   48.348100] ([<000003d1002a91c0>] 0x3d1002a91c0)
> > [   48.348102]  [<00000000002404aa>] page_remove_rmap+0xf2/0x16c
> > [   48.348106]  [<0000000000232dc8>] unmap_single_vma+0x494/0x7d8
> > [   48.348107]  [<0000000000233ac0>] unmap_vmas+0x50/0x74
> > [   48.348109]  [<00000000002396ec>] unmap_region+0x9c/0x110
> > [   48.348110]  [<000000000023bd18>] do_munmap+0x284/0x470
> > [   48.348111]  [<000000000023bf56>] vm_munmap+0x52/0x70
> > [   48.348113]  [<000000000023cf32>] SyS_munmap+0x3a/0x4c
> > [   48.348114]  [<0000000000665e14>] sysc_noemu+0x22/0x28
> > [   48.348118]  [<000003fffcf187b2>] 0x3fffcf187b2
> > [   48.348119] Last Breaking-Event-Address:
> > [   48.348120]  [<0000000000265528>] __mem_cgroup_uncharge_common+0x2c0/0x33c
> > 
> > Looking at the code, the code flow is:
> > 
> > page_remove_rmap() -> mem_cgroup_uncharge_page() -> __mem_cgroup_uncharge_common()
> > 
> > Note that in mem_cgroup_uncharge_page() the page in question passed the check:
> > 
> > [...]
> >         if (PageSwapCache(page))
> >                 return;
> > [...]
> > 
> > and just a couple of instructions later the VM_BUG_ON() within
> > __mem_cgroup_uncharge_common() triggers:
> > 
> > [...]
> >         if (mem_cgroup_disabled())
> >                 return NULL;
> > 
> >         VM_BUG_ON(PageSwapCache(page));
> > [...]
> > 
> > Which means that another cpu changed the pageflags concurrently. In fact,
> > looking at the dump a different cpu is indeed busy with running kswapd.
> 
> Hmm, maybe I am missing something but it really looks like we can race
> here. Reclaim path takes the page lock while zap_pte takes page table
> lock so nothing prevents them from racing here:
> shrink_page_list		zap_pte_range
>   trylock_page			  pte_offset_map_lock
>   add_to_swap			    page_remove_rmap
>     /* Page can be still mapped */
>     add_to_swap_cache		      atomic_add_negative(_mapcount)
>       __add_to_swap_cache	        mem_cgroup_uncharge_page
>       				          (PageSwapCache(page)) && return
>         SetPageSwapCache
> 				          __mem_cgroup_uncharge_common
> 					    VM_BUG_ON(PageSwapCache(page))
> 
> Maybe not many people run with CONFIG_DEBUG_VM enabled these days so we
> do not this more often (even me testing configs are not consistent in
> that regards and only few have it on). The only thing that changed in
> this area recently is 0c59b89c which made the test VM_BUG_ON rather then
> simple return in 3.6
> And maybe the BUG_ON is too harsh as CgroupUsed should guarantee that
> the uncharge will eventually go away. What do you think Johannes?

Interesting.  We need to ensure there is ordering between setting
PG_swapcache and installing swap entries because I think we are the
only ones looking at PG_swapcache without the page lock held.  So we
don't have a safe way to check for PG_swapcache but if we get it
wrong, we may steal an uncharge that uncharge_swapcache() should be
doing instead and that means we mess up the swap statistics
accounting.

So how can we, without holding the page lock, either safely back off
from a page in swapcache or make sure we do the swap statistics
accounting when uncharging a swapcache page from the final unmap?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [v3.9-rc8]: kernel BUG at mm/memcontrol.c:3994! (was: Re: [BUG][s390x] mm: system crashed)

[Hugh Dickins]

On Wed, 24 Apr 2013, Johannes Weiner wrote:
> On Wed, Apr 24, 2013 at 03:18:51PM +0200, Michal Hocko wrote:
> > On Wed 24-04-13 12:42:55, Heiko Carstens wrote:
> > > On Thu, Apr 18, 2013 at 09:13:03AM +0200, Heiko Carstens wrote:
> > > > Ok, thanks for verifying! I'll look into it; hopefully I can reproduce it
> > > > here as well.
> > > 
> > > That seems to be a common code bug. I can easily trigger the VM_BUG_ON()
> > > below (when I force the system to swap):
> > > 
> > > [   48.347963] ------------[ cut here ]------------
> > > [   48.347972] kernel BUG at mm/memcontrol.c:3994!
> > > [   48.348012] illegal operation: 0001 [#1] SMP 
> > > [   48.348015] Modules linked in:
> > > [   48.348017] CPU: 1 Not tainted 3.9.0-rc8+ #38
> > > [   48.348020] Process mmap2 (pid: 635, task: 0000000029476100, ksp: 000000002e91b938)
> > > [   48.348022] Krnl PSW : 0704f00180000000 000000000026552c (__mem_cgroup_uncharge_common+0x2c4/0x33c)
> > > [   48.348032]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 EA:3
> > >                Krnl GPRS: 0000000000000008 0000000000000009 000003d1002a9200 0000000000000000
> > > [   48.348039]            0000000000000000 00000000006812d8 000003ffdf339000 00000000321a6f98
> > > [   48.348043]            000003fffce11000 0000000000000000 0000000000000001 000003d1002a9200
> > > [   48.348046]            0000000000000001 0000000000681b88 000000002e91bc18 000000002e91bbd0
> > > [   48.348057] Krnl Code: 000000000026551e: c0e5fffaa2a1        brasl   %r14,1b9a60
> > >                           0000000000265524: a7f4ff7d            brc     15,26541e
> > >                          #0000000000265528: a7f40001            brc     15,26552a
> > >                          >000000000026552c: e3c0b8200124        stg     %r12,6176(%r11)
> > >                           0000000000265532: a7f4ff57            brc     15,2653e0
> > >                           0000000000265536: e310b8280104        lg      %r1,6184(%r11)
> > >                           000000000026553c: a71b0001            aghi    %r1,1
> > >                           0000000000265540: e310b8280124        stg     %r1,6184(%r11)
> > > [   48.348099] Call Trace:
> > > [   48.348100] ([<000003d1002a91c0>] 0x3d1002a91c0)
> > > [   48.348102]  [<00000000002404aa>] page_remove_rmap+0xf2/0x16c
> > > [   48.348106]  [<0000000000232dc8>] unmap_single_vma+0x494/0x7d8
> > > [   48.348107]  [<0000000000233ac0>] unmap_vmas+0x50/0x74
> > > [   48.348109]  [<00000000002396ec>] unmap_region+0x9c/0x110
> > > [   48.348110]  [<000000000023bd18>] do_munmap+0x284/0x470
> > > [   48.348111]  [<000000000023bf56>] vm_munmap+0x52/0x70
> > > [   48.348113]  [<000000000023cf32>] SyS_munmap+0x3a/0x4c
> > > [   48.348114]  [<0000000000665e14>] sysc_noemu+0x22/0x28
> > > [   48.348118]  [<000003fffcf187b2>] 0x3fffcf187b2
> > > [   48.348119] Last Breaking-Event-Address:
> > > [   48.348120]  [<0000000000265528>] __mem_cgroup_uncharge_common+0x2c0/0x33c
> > > 
> > > Looking at the code, the code flow is:
> > > 
> > > page_remove_rmap() -> mem_cgroup_uncharge_page() -> __mem_cgroup_uncharge_common()
> > > 
> > > Note that in mem_cgroup_uncharge_page() the page in question passed the check:
> > > 
> > > [...]
> > >         if (PageSwapCache(page))
> > >                 return;
> > > [...]
> > > 
> > > and just a couple of instructions later the VM_BUG_ON() within
> > > __mem_cgroup_uncharge_common() triggers:
> > > 
> > > [...]
> > >         if (mem_cgroup_disabled())
> > >                 return NULL;
> > > 
> > >         VM_BUG_ON(PageSwapCache(page));
> > > [...]
> > > 
> > > Which means that another cpu changed the pageflags concurrently. In fact,
> > > looking at the dump a different cpu is indeed busy with running kswapd.
> > 
> > Hmm, maybe I am missing something but it really looks like we can race
> > here. Reclaim path takes the page lock while zap_pte takes page table
> > lock so nothing prevents them from racing here:
> > shrink_page_list		zap_pte_range
> >   trylock_page			  pte_offset_map_lock
> >   add_to_swap			    page_remove_rmap
> >     /* Page can be still mapped */
> >     add_to_swap_cache		      atomic_add_negative(_mapcount)
> >       __add_to_swap_cache	        mem_cgroup_uncharge_page
> >       				          (PageSwapCache(page)) && return
> >         SetPageSwapCache
> > 				          __mem_cgroup_uncharge_common
> > 					    VM_BUG_ON(PageSwapCache(page))
> > 
> > Maybe not many people run with CONFIG_DEBUG_VM enabled these days so we
> > do not this more often (even me testing configs are not consistent in
> > that regards and only few have it on). The only thing that changed in
> > this area recently is 0c59b89c which made the test VM_BUG_ON rather then
> > simple return in 3.6
> > And maybe the BUG_ON is too harsh as CgroupUsed should guarantee that
> > the uncharge will eventually go away. What do you think Johannes?
> 
> Interesting.  We need to ensure there is ordering between setting
> PG_swapcache and installing swap entries because I think we are the
> only ones looking at PG_swapcache without the page lock held.  So we
> don't have a safe way to check for PG_swapcache but if we get it
> wrong, we may steal an uncharge that uncharge_swapcache() should be
> doing instead and that means we mess up the swap statistics
> accounting.
> 
> So how can we, without holding the page lock, either safely back off
> from a page in swapcache or make sure we do the swap statistics
> accounting when uncharging a swapcache page from the final unmap?

Awkward.

I agree that the actual memcg uncharging should be okay, but the memsw
swap stats will go wrong (doesn't matter toooo much), and mem_cgroup_put
get missed (leaking a struct mem_cgroup).

I confess to having seen this myself, just once, on x86_64 3.8-rc6-mm1
plus some patches I was testing; but I never got back to look into it.

I wonder if something gone into 3.9 has changed the timing to make it
more easily reproduced now; but the underlying problem has been there
a long time - long before your VM_BUG_ON was there to expose it.

For now I guess the best is just to remove the VM_BUG_ON, or restore
it to a (usually redundant) return NULL, and live with the old leak.

But the real fix... I fear it may involve adding another PageCgroup
flag, just for memsw, to ensure that its counts are kept in balance
even when SwapCache races cause what's been charged in one way to
get uncharged in an unexpected way.

All those PageSwapCache and page_mapped tests are suspect: I expect
most will turn out to be okay, one way or another, but they are
suspicious and deserve audit.

(But I won't be working on this myself, sorry.)

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [v3.9-rc8]: kernel BUG at mm/memcontrol.c:3994! (was: Re: [BUG][s390x] mm: system crashed)

[Johannes Weiner]

On Wed, Apr 24, 2013 at 08:50:01PM -0700, Hugh Dickins wrote:
> On Wed, 24 Apr 2013, Johannes Weiner wrote:
> > On Wed, Apr 24, 2013 at 03:18:51PM +0200, Michal Hocko wrote:
> > > On Wed 24-04-13 12:42:55, Heiko Carstens wrote:
> > > > On Thu, Apr 18, 2013 at 09:13:03AM +0200, Heiko Carstens wrote:
> > > > > Ok, thanks for verifying! I'll look into it; hopefully I can reproduce it
> > > > > here as well.
> > > > 
> > > > That seems to be a common code bug. I can easily trigger the VM_BUG_ON()
> > > > below (when I force the system to swap):
> > > > 
> > > > [   48.347963] ------------[ cut here ]------------
> > > > [   48.347972] kernel BUG at mm/memcontrol.c:3994!
> > > > [   48.348012] illegal operation: 0001 [#1] SMP 
> > > > [   48.348015] Modules linked in:
> > > > [   48.348017] CPU: 1 Not tainted 3.9.0-rc8+ #38
> > > > [   48.348020] Process mmap2 (pid: 635, task: 0000000029476100, ksp: 000000002e91b938)
> > > > [   48.348022] Krnl PSW : 0704f00180000000 000000000026552c (__mem_cgroup_uncharge_common+0x2c4/0x33c)
> > > > [   48.348032]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:3 PM:0 EA:3
> > > >                Krnl GPRS: 0000000000000008 0000000000000009 000003d1002a9200 0000000000000000
> > > > [   48.348039]            0000000000000000 00000000006812d8 000003ffdf339000 00000000321a6f98
> > > > [   48.348043]            000003fffce11000 0000000000000000 0000000000000001 000003d1002a9200
> > > > [   48.348046]            0000000000000001 0000000000681b88 000000002e91bc18 000000002e91bbd0
> > > > [   48.348057] Krnl Code: 000000000026551e: c0e5fffaa2a1        brasl   %r14,1b9a60
> > > >                           0000000000265524: a7f4ff7d            brc     15,26541e
> > > >                          #0000000000265528: a7f40001            brc     15,26552a
> > > >                          >000000000026552c: e3c0b8200124        stg     %r12,6176(%r11)
> > > >                           0000000000265532: a7f4ff57            brc     15,2653e0
> > > >                           0000000000265536: e310b8280104        lg      %r1,6184(%r11)
> > > >                           000000000026553c: a71b0001            aghi    %r1,1
> > > >                           0000000000265540: e310b8280124        stg     %r1,6184(%r11)
> > > > [   48.348099] Call Trace:
> > > > [   48.348100] ([<000003d1002a91c0>] 0x3d1002a91c0)
> > > > [   48.348102]  [<00000000002404aa>] page_remove_rmap+0xf2/0x16c
> > > > [   48.348106]  [<0000000000232dc8>] unmap_single_vma+0x494/0x7d8
> > > > [   48.348107]  [<0000000000233ac0>] unmap_vmas+0x50/0x74
> > > > [   48.348109]  [<00000000002396ec>] unmap_region+0x9c/0x110
> > > > [   48.348110]  [<000000000023bd18>] do_munmap+0x284/0x470
> > > > [   48.348111]  [<000000000023bf56>] vm_munmap+0x52/0x70
> > > > [   48.348113]  [<000000000023cf32>] SyS_munmap+0x3a/0x4c
> > > > [   48.348114]  [<0000000000665e14>] sysc_noemu+0x22/0x28
> > > > [   48.348118]  [<000003fffcf187b2>] 0x3fffcf187b2
> > > > [   48.348119] Last Breaking-Event-Address:
> > > > [   48.348120]  [<0000000000265528>] __mem_cgroup_uncharge_common+0x2c0/0x33c
> > > > 
> > > > Looking at the code, the code flow is:
> > > > 
> > > > page_remove_rmap() -> mem_cgroup_uncharge_page() -> __mem_cgroup_uncharge_common()
> > > > 
> > > > Note that in mem_cgroup_uncharge_page() the page in question passed the check:
> > > > 
> > > > [...]
> > > >         if (PageSwapCache(page))
> > > >                 return;
> > > > [...]
> > > > 
> > > > and just a couple of instructions later the VM_BUG_ON() within
> > > > __mem_cgroup_uncharge_common() triggers:
> > > > 
> > > > [...]
> > > >         if (mem_cgroup_disabled())
> > > >                 return NULL;
> > > > 
> > > >         VM_BUG_ON(PageSwapCache(page));
> > > > [...]
> > > > 
> > > > Which means that another cpu changed the pageflags concurrently. In fact,
> > > > looking at the dump a different cpu is indeed busy with running kswapd.
> > > 
> > > Hmm, maybe I am missing something but it really looks like we can race
> > > here. Reclaim path takes the page lock while zap_pte takes page table
> > > lock so nothing prevents them from racing here:
> > > shrink_page_list		zap_pte_range
> > >   trylock_page			  pte_offset_map_lock
> > >   add_to_swap			    page_remove_rmap
> > >     /* Page can be still mapped */
> > >     add_to_swap_cache		      atomic_add_negative(_mapcount)
> > >       __add_to_swap_cache	        mem_cgroup_uncharge_page
> > >       				          (PageSwapCache(page)) && return
> > >         SetPageSwapCache
> > > 				          __mem_cgroup_uncharge_common
> > > 					    VM_BUG_ON(PageSwapCache(page))
> > > 
> > > Maybe not many people run with CONFIG_DEBUG_VM enabled these days so we
> > > do not this more often (even me testing configs are not consistent in
> > > that regards and only few have it on). The only thing that changed in
> > > this area recently is 0c59b89c which made the test VM_BUG_ON rather then
> > > simple return in 3.6
> > > And maybe the BUG_ON is too harsh as CgroupUsed should guarantee that
> > > the uncharge will eventually go away. What do you think Johannes?
> > 
> > Interesting.  We need to ensure there is ordering between setting
> > PG_swapcache and installing swap entries because I think we are the
> > only ones looking at PG_swapcache without the page lock held.  So we
> > don't have a safe way to check for PG_swapcache but if we get it
> > wrong, we may steal an uncharge that uncharge_swapcache() should be
> > doing instead and that means we mess up the swap statistics
> > accounting.
> > 
> > So how can we, without holding the page lock, either safely back off
> > from a page in swapcache or make sure we do the swap statistics
> > accounting when uncharging a swapcache page from the final unmap?
> 
> Awkward.
> 
> I agree that the actual memcg uncharging should be okay, but the memsw
> swap stats will go wrong (doesn't matter toooo much), and mem_cgroup_put
> get missed (leaking a struct mem_cgroup).

Ok, so I just went over this again.  For the swapout path the memsw
uncharge is deferred, but if we "steal" this uncharge from the swap
code, we actually do uncharge memsw in mem_cgroup_do_uncharge(), so we
may prematurely unaccount the swap page, but we never leak a charge.
Good.

Because of this stealing, we also don't do the following:

	if (do_swap_account && ctype == MEM_CGROUP_CHARGE_TYPE_SWAPOUT) {
		mem_cgroup_swap_statistics(memcg, true);
		mem_cgroup_get(memcg);
	}

I.e. it does not matter that mem_cgroup_uncharge_swap() doesn't do the
put, we are also not doing the get.  We should not leak references.

So the only thing that I can see go wrong is that we may have a
swapped out page that is not charged to memsw and not accounted as
MEM_CGROUP_STAT_SWAP.  But I don't know how likely that is, because we
check for PG_swapcache in this uncharge path after the last pte is
torn down, so even though the page is put on swap cache, it probably
won't be swapped.  It would require that the PG_swapcache setting
would become visible only after the page has been added to the swap
cache AND rmap has established at least one swap pte for us to
uncharge a page that actually continues to be used.  And that's a bit
of a stretch, I think.

Did I miss something?  If not, I'll just send a patch that removes the
VM_BUG_ON() and adds a comment describing the scenarios and a note
that we may want to fix this in the future.

Thanks!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [v3.9-rc8]: kernel BUG at mm/memcontrol.c:3994! (was: Re: [BUG][s390x] mm: system crashed)

[Hugh Dickins]

On Tue, 30 Apr 2013, Johannes Weiner wrote:
> On Wed, Apr 24, 2013 at 08:50:01PM -0700, Hugh Dickins wrote:
> > On Wed, 24 Apr 2013, Johannes Weiner wrote:
> > > On Wed, Apr 24, 2013 at 03:18:51PM +0200, Michal Hocko wrote:
> > > > On Wed 24-04-13 12:42:55, Heiko Carstens wrote:
> > > > > On Thu, Apr 18, 2013 at 09:13:03AM +0200, Heiko Carstens wrote:
> > > > > 
> > > > > [   48.347963] ------------[ cut here ]------------
> > > > > [   48.347972] kernel BUG at mm/memcontrol.c:3994!
> > > > > __mem_cgroup_uncharge_common() triggers:
> > > > > 
> > > > > [...]
> > > > >         if (mem_cgroup_disabled())
> > > > >                 return NULL;
> > > > > 
> > > > >         VM_BUG_ON(PageSwapCache(page));
> > > > > [...]
> > 
> > I agree that the actual memcg uncharging should be okay, but the memsw
> > swap stats will go wrong (doesn't matter toooo much), and mem_cgroup_put
> > get missed (leaking a struct mem_cgroup).
> 
> Ok, so I just went over this again.  For the swapout path the memsw
> uncharge is deferred, but if we "steal" this uncharge from the swap
> code, we actually do uncharge memsw in mem_cgroup_do_uncharge(), so we
> may prematurely unaccount the swap page, but we never leak a charge.
> Good.
> 
> Because of this stealing, we also don't do the following:
> 
> 	if (do_swap_account && ctype == MEM_CGROUP_CHARGE_TYPE_SWAPOUT) {
> 		mem_cgroup_swap_statistics(memcg, true);
> 		mem_cgroup_get(memcg);
> 	}
> 
> I.e. it does not matter that mem_cgroup_uncharge_swap() doesn't do the
> put, we are also not doing the get.  We should not leak references.
> 
> So the only thing that I can see go wrong is that we may have a
> swapped out page that is not charged to memsw and not accounted as
> MEM_CGROUP_STAT_SWAP.  But I don't know how likely that is, because we
> check for PG_swapcache in this uncharge path after the last pte is
> torn down, so even though the page is put on swap cache, it probably
> won't be swapped.  It would require that the PG_swapcache setting
> would become visible only after the page has been added to the swap
> cache AND rmap has established at least one swap pte for us to
> uncharge a page that actually continues to be used.  And that's a bit
> of a stretch, I think.

Sorry, our minds seem to work in different ways,
I understood very little of what you wrote above :-(

But once I try to disprove you with a counter-example, I seem to
arrive at the same conclusion as you have (well, I haven't quite
arrived there yet, but cannot give it any more time).

Looking at it from my point of view, I concentrate on the racy
	if (PageSwapCache(page))
		return;
	__mem_cgroup_uncharge_common(page, MEM_CGROUP_CHARGE_TYPE_ANON, false);
in mem_cgroup_uncharge_page().

Now, that may or may not catch the case where last reference to page
is unmapped at the same time as the page is added to swap: but being
a MEM_CGROUP_CHARGE_TYPE_ANON call, it does not interfere with the
memsw stats and get/put at all, those remain in balance.

And mem_cgroup_uncharge_swap() has all along been prepared to get
a zero id from swap_cgroup_record(), if a SwapCache page should be
uncharged when it was never quite charged as such.

Yes, we may occasionally fail to charge a SwapCache page as such
if its final unmap from userspace races with its being added to swap;
but it's heading towards swap_writepage()'s try_to_free_swap() anyway,
so I don't think that's anything to worry about.

(If I had time to stop and read through that, I'd probably find it
just as hard to understand as what you wrote!)

> 
> Did I miss something?  If not, I'll just send a patch that removes the
> VM_BUG_ON() and adds a comment describing the scenarios and a note
> that we may want to fix this in the future.

I don't think you missed something.  Yes, please just send Linus and
Andrew a patch to remove the VM_BUG_ON() (with Cc stable tag), I now
agree that's all that's really needed - thanks.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [v3.9-rc8]: kernel BUG at mm/memcontrol.c:3994! (was: Re: [BUG][s390x] mm: system crashed)

[Johannes Weiner]

On Wed, May 01, 2013 at 08:28:30AM -0700, Hugh Dickins wrote:
> On Tue, 30 Apr 2013, Johannes Weiner wrote:
> > On Wed, Apr 24, 2013 at 08:50:01PM -0700, Hugh Dickins wrote:
> > > On Wed, 24 Apr 2013, Johannes Weiner wrote:
> > > > On Wed, Apr 24, 2013 at 03:18:51PM +0200, Michal Hocko wrote:
> > > > > On Wed 24-04-13 12:42:55, Heiko Carstens wrote:
> > > > > > On Thu, Apr 18, 2013 at 09:13:03AM +0200, Heiko Carstens wrote:
> > > > > > 
> > > > > > [   48.347963] ------------[ cut here ]------------
> > > > > > [   48.347972] kernel BUG at mm/memcontrol.c:3994!
> > > > > > __mem_cgroup_uncharge_common() triggers:
> > > > > > 
> > > > > > [...]
> > > > > >         if (mem_cgroup_disabled())
> > > > > >                 return NULL;
> > > > > > 
> > > > > >         VM_BUG_ON(PageSwapCache(page));
> > > > > > [...]
> > > 
> > > I agree that the actual memcg uncharging should be okay, but the memsw
> > > swap stats will go wrong (doesn't matter toooo much), and mem_cgroup_put
> > > get missed (leaking a struct mem_cgroup).
> > 
> > Ok, so I just went over this again.  For the swapout path the memsw
> > uncharge is deferred, but if we "steal" this uncharge from the swap
> > code, we actually do uncharge memsw in mem_cgroup_do_uncharge(), so we
> > may prematurely unaccount the swap page, but we never leak a charge.
> > Good.
> > 
> > Because of this stealing, we also don't do the following:
> > 
> > 	if (do_swap_account && ctype == MEM_CGROUP_CHARGE_TYPE_SWAPOUT) {
> > 		mem_cgroup_swap_statistics(memcg, true);
> > 		mem_cgroup_get(memcg);
> > 	}
> > 
> > I.e. it does not matter that mem_cgroup_uncharge_swap() doesn't do the
> > put, we are also not doing the get.  We should not leak references.
> > 
> > So the only thing that I can see go wrong is that we may have a
> > swapped out page that is not charged to memsw and not accounted as
> > MEM_CGROUP_STAT_SWAP.  But I don't know how likely that is, because we
> > check for PG_swapcache in this uncharge path after the last pte is
> > torn down, so even though the page is put on swap cache, it probably
> > won't be swapped.  It would require that the PG_swapcache setting
> > would become visible only after the page has been added to the swap
> > cache AND rmap has established at least one swap pte for us to
> > uncharge a page that actually continues to be used.  And that's a bit
> > of a stretch, I think.
> 
> Sorry, our minds seem to work in different ways,
> I understood very little of what you wrote above :-(
> 
> But once I try to disprove you with a counter-example, I seem to
> arrive at the same conclusion as you have (well, I haven't quite
> arrived there yet, but cannot give it any more time).

I might be losing my mind.  But since you are reaching the same
conclusion, and I see the same mental milestones in your thought
process described below, it's more likely that I suck at describing my
train of thought coherently.  Or the third possibility: we're both
losing it!

> Looking at it from my point of view, I concentrate on the racy
> 	if (PageSwapCache(page))
> 		return;
> 	__mem_cgroup_uncharge_common(page, MEM_CGROUP_CHARGE_TYPE_ANON, false);
> in mem_cgroup_uncharge_page().
> 
> Now, that may or may not catch the case where last reference to page
> is unmapped at the same time as the page is added to swap: but being
> a MEM_CGROUP_CHARGE_TYPE_ANON call, it does not interfere with the
> memsw stats and get/put at all, those remain in balance.

Yes, exactly.

> And mem_cgroup_uncharge_swap() has all along been prepared to get
> a zero id from swap_cgroup_record(), if a SwapCache page should be
> uncharged when it was never quite charged as such.
> 
> Yes, we may occasionally fail to charge a SwapCache page as such
> if its final unmap from userspace races with its being added to swap;
> but it's heading towards swap_writepage()'s try_to_free_swap() anyway,
> so I don't think that's anything to worry about.

Agreed as well.  If there are no pte references to the swap slot, it
will be freed either way.  I didn't even think of the
try_to_free_swap() in the writeout call, but was looking at the
__remove_mapping later on in reclaim that will do a swapcache_free().

The only case I was worried about is the following:

#0                                      #1
page_remove_rmap()                      shrink_page_list()
  if --page->mapcount == 0:               add_to_swap()
    mem_cgroup_uncharge_page()              __add_to_swap_cache()
      if PageSwapCache:                       SetPageSwapCache()
        return                            try_to_unmap()
      __mem_cgroup_uncharge_common()        for each pte:
                                              install swp_entry_t
                                              page->mapcount--

Looking at #1, I don't see anything that would force concurrent
threads to observe SetSwapCache ordered against the page->mapcount--.
My concern was that if those get reordered, #0 may see page->mapcount
== 1 AND !PageSwapcache, and then go ahead and uncharge the page while
there is actually a swp_entry_t pointing to it.  The page will be a
proper long-term swap page without being charged as such.

> (If I had time to stop and read through that, I'd probably find it
> just as hard to understand as what you wrote!)
> 
> > 
> > Did I miss something?  If not, I'll just send a patch that removes the
> > VM_BUG_ON() and adds a comment describing the scenarios and a note
> > that we may want to fix this in the future.
> 
> I don't think you missed something.  Yes, please just send Linus and
> Andrew a patch to remove the VM_BUG_ON() (with Cc stable tag), I now
> agree that's all that's really needed - thanks.

Will do, thanks for taking them time to think through it again, even
after failing to decipher my ramblings...

Johannes

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [v3.9-rc8]: kernel BUG at mm/memcontrol.c:3994! (was: Re: [BUG][s390x] mm: system crashed)

[Hugh Dickins]

[-- Attachment #1: Type: text/plain, Size: 6713 bytes --]

On Wed, May 1, 2013 at 12:10 PM, Johannes Weiner <hannes@cmpxchg.org> wrote:

> On Wed, May 01, 2013 at 08:28:30AM -0700, Hugh Dickins wrote:
> > On Tue, 30 Apr 2013, Johannes Weiner wrote:
> > > On Wed, Apr 24, 2013 at 08:50:01PM -0700, Hugh Dickins wrote:
> > > > On Wed, 24 Apr 2013, Johannes Weiner wrote:
> > > > > On Wed, Apr 24, 2013 at 03:18:51PM +0200, Michal Hocko wrote:
> > > > > > On Wed 24-04-13 12:42:55, Heiko Carstens wrote:
> > > > > > > On Thu, Apr 18, 2013 at 09:13:03AM +0200, Heiko Carstens wrote:
> > > > > > >
> > > > > > > [   48.347963] ------------[ cut here ]------------
> > > > > > > [   48.347972] kernel BUG at mm/memcontrol.c:3994!
> > > > > > > __mem_cgroup_uncharge_common() triggers:
> > > > > > >
> > > > > > > [...]
> > > > > > >         if (mem_cgroup_disabled())
> > > > > > >                 return NULL;
> > > > > > >
> > > > > > >         VM_BUG_ON(PageSwapCache(page));
> > > > > > > [...]
> > > >
> > > > I agree that the actual memcg uncharging should be okay, but the
> memsw
> > > > swap stats will go wrong (doesn't matter toooo much), and
> mem_cgroup_put
> > > > get missed (leaking a struct mem_cgroup).
> > >
> > > Ok, so I just went over this again.  For the swapout path the memsw
> > > uncharge is deferred, but if we "steal" this uncharge from the swap
> > > code, we actually do uncharge memsw in mem_cgroup_do_uncharge(), so we
> > > may prematurely unaccount the swap page, but we never leak a charge.
> > > Good.
> > >
> > > Because of this stealing, we also don't do the following:
> > >
> > >     if (do_swap_account && ctype == MEM_CGROUP_CHARGE_TYPE_SWAPOUT) {
> > >             mem_cgroup_swap_statistics(memcg, true);
> > >             mem_cgroup_get(memcg);
> > >     }
> > >
> > > I.e. it does not matter that mem_cgroup_uncharge_swap() doesn't do the
> > > put, we are also not doing the get.  We should not leak references.
> > >
> > > So the only thing that I can see go wrong is that we may have a
> > > swapped out page that is not charged to memsw and not accounted as
> > > MEM_CGROUP_STAT_SWAP.  But I don't know how likely that is, because we
> > > check for PG_swapcache in this uncharge path after the last pte is
> > > torn down, so even though the page is put on swap cache, it probably
> > > won't be swapped.  It would require that the PG_swapcache setting
> > > would become visible only after the page has been added to the swap
> > > cache AND rmap has established at least one swap pte for us to
> > > uncharge a page that actually continues to be used.  And that's a bit
> > > of a stretch, I think.
> >
> > Sorry, our minds seem to work in different ways,
> > I understood very little of what you wrote above :-(
> >
> > But once I try to disprove you with a counter-example, I seem to
> > arrive at the same conclusion as you have (well, I haven't quite
> > arrived there yet, but cannot give it any more time).
>
> I might be losing my mind.  But since you are reaching the same
> conclusion, and I see the same mental milestones in your thought
> process described below, it's more likely that I suck at describing my
> train of thought coherently.  Or the third possibility: we're both
> losing it!
>
> > Looking at it from my point of view, I concentrate on the racy
> >       if (PageSwapCache(page))
> >               return;
> >       __mem_cgroup_uncharge_common(page, MEM_CGROUP_CHARGE_TYPE_ANON,
> false);
> > in mem_cgroup_uncharge_page().
> >
> > Now, that may or may not catch the case where last reference to page
> > is unmapped at the same time as the page is added to swap: but being
> > a MEM_CGROUP_CHARGE_TYPE_ANON call, it does not interfere with the
> > memsw stats and get/put at all, those remain in balance.
>
> Yes, exactly.
>
> > And mem_cgroup_uncharge_swap() has all along been prepared to get
> > a zero id from swap_cgroup_record(), if a SwapCache page should be
> > uncharged when it was never quite charged as such.
> >
> > Yes, we may occasionally fail to charge a SwapCache page as such
> > if its final unmap from userspace races with its being added to swap;
> > but it's heading towards swap_writepage()'s try_to_free_swap() anyway,
> > so I don't think that's anything to worry about.
>
> Agreed as well.  If there are no pte references to the swap slot, it
> will be freed either way.  I didn't even think of the
> try_to_free_swap() in the writeout call, but was looking at the
> __remove_mapping later on in reclaim that will do a swapcache_free().
>
> The only case I was worried about is the following:
>
> #0                                      #1
> page_remove_rmap()                      shrink_page_list()
>   if --page->mapcount == 0:               add_to_swap()
>     mem_cgroup_uncharge_page()              __add_to_swap_cache()
>       if PageSwapCache:                       SetPageSwapCache()
>         return                            try_to_unmap()
>       __mem_cgroup_uncharge_common()        for each pte:
>                                               install swp_entry_t
>                                               page->mapcount--
>

Thanks for spelling it out for me in more detail, this time I think I do
grasp your concern.


>
> Looking at #1, I don't see anything that would force concurrent
> threads to observe SetSwapCache ordered against the page->mapcount--.
> My concern was that if those get reordered, #0 may see page->mapcount
> == 1 AND !PageSwapcache, and then go ahead and uncharge the page while
> there is actually a swp_entry_t pointing to it.  The page will be a
> proper long-term swap page without being charged as such.
>

But I don't see any problem with ordering here.  #0 is using an atomic
operation which returns a result on page->mapcount, so that amounts to
(more than) an smp_rmb ensuring it reads mapcount before reading
PageSwapCache flag.  And in #1, there's at least an unlock of the
radix_tree lock (after adding to swap tree) and a lock of the page table
lock (before unmapping the page), and that pairing amounts to (more than)
an smp_wmb.

Hugh


> > (If I had time to stop and read through that, I'd probably find it
> > just as hard to understand as what you wrote!)
> >
> > >
> > > Did I miss something?  If not, I'll just send a patch that removes the
> > > VM_BUG_ON() and adds a comment describing the scenarios and a note
> > > that we may want to fix this in the future.
> >
> > I don't think you missed something.  Yes, please just send Linus and
> > Andrew a patch to remove the VM_BUG_ON() (with Cc stable tag), I now
> > agree that's all that's really needed - thanks.
>
> Will do, thanks for taking them time to think through it again, even
> after failing to decipher my ramblings...
>
> Johannes
>

[-- Attachment #2: Type: text/html, Size: 8812 bytes --]