Re: [RFC Patch 0/2] mm: Add parameters to make kernel behavior at memory error on dirty cache selectable

[Simon Jeons <simon.jeons@gmail.com>]

Hi Mitsuhiro,
On 04/11/2013 11:26 AM, Mitsuhiro Tanino wrote:
> Hi All,
> Please find a patch set that introduces these new sysctl interfaces,
> to handle a case when an memory error is detected on dirty page cache.
>
> - vm.memory_failure_dirty_panic
> - vm.memory_failure_print_ratelimit
> - vm.memory_failure_print_ratelimit_burst
>
> Problem
> ---------
> Recently, it is common that enterprise servers likely have a large
> amount of memory, especially for cloud environment. This means that
> possibility of memory failures is increased.
>
> To handle memory failure, Linux has a hwpoison feature. When a memory
> error is detected by memory scrub, the error is reported as machine
> check, uncorrected recoverable (UCR), to OS. Then OS isolates the memory
> region with memory failure if the memory page can be isolated.
> The hwpoison handles it according to the memory region, such as kernel,
> dirty cache, clean cache. If the memory region can be isolated, the
> page is marked "hwpoison" and it is not used again.
>
> When SRAO machine check is reported on a page which is included dirty
> page cache, the page is truncated because the memory is corrupted and
> data of the page cannot be written to a disk any more.
>
> As a result, if the dirty cache includes user data, the data is lost,
> and data corruption occurs if an application uses old data.

One question against mce instead of the patchset. ;-)

When check memory is bad? Before memory access? Is there a process scan 
it period?

>
>
> Solution
> ---------
> The patch proposes a new sysctl interface, vm.memory_failure_dirty_panic,
> in order to prevent data corruption comes from data lost problem.
> Also this patch displays information of affected file such as device name,
> inode number, file offset and file type if the file is mapped on a memory
> and the page is dirty cache.
>
> When SRAO machine check occurs on a dirty page cache, corresponding
> data cannot be recovered any more. Therefore, the patch proposes a kernel
> option to keep a system running or force system panic in order
> to avoid further trouble such as data corruption problem of application.
>
> System administrator can select an error action using this option
> according to characteristics of target system.
>
>
>
> Use Case
> ---------
> This option is intended to be adopted in KVM guest because it is
> supposed that Linux on KVM guest operates customers business and
> it is big impact to lost or corrupt customers data by memory failure.
>
> On the other hand, this option does not recommend to apply KVM host
> as following reasons.
>
> - Making KVM host panic has a big impact because all virtual guests are
>    affected by their host panic. Affected virtual guests are forced to stop
>    and have to be restarted on the other hypervisor.
>
> - If disk cached model of qemu is set to "none", I/O type of virtual
>    guests becomes O_DIRECT and KVM host does not cache guest's disk I/O.
>    Therefore, if SRAO machine check is reported on a dirty page cache
>    in KVM host, its virtual machines are not affected by the machine check.
>    So the host is expected to keep operating instead of kernel panic.
>
>
> Past discussion
> --------------------
> This problem was previously discussed in the kernel community,
> (refer: mail threads pertaining to
> http://marc.info/?l=linux-kernel&m=135187403804934&w=4).
>
>>> - I worry that if a hardware error occurs, it might affect a large
>>>    amount of memory all at the same time.  For example, if a 4G memory
>>>    block goes bad, this message will be printed a million times?
>
> As Andrew mentioned in the above threads, if 4GB memory blocks goes bad,
> error messages will be printed a million times and this behavior loses
> a system reliability.
>
> Therefore, the second patch introduces two sysctl parameters for
> __ratelimit() which is used at mce_notify_irq() in order to notify
> occurrence of machine check event to system administrator.
> The use of __ratelimit(), this patch can limit quantity of messages
> per interval to be output at syslog or terminal console.
>
> If system administrator needs to limit quantity of messages,
> these parameters are available.
>
> - vm.memory_failure_print_ratelimit:
>    Specifies the minimum length of time between messages.
>    By default the rate limiting is disabled.
>
> - vm.memory_failure_print_ratelimit_burst:
>    Specifies the number of messages we can send before rate limiting.
>
>
>
> Test Results
> ---------
> These patches are tested on 3.8.1 kernel(FC18) using software pseudo MCE
> injection from KVM host to guest.
>
>
> ******** Host OS Screen logs(SRAO Machine Check injection) ********
> Inject software pseudo MCE into guest qemu process.
>
> (1) Load mce-inject module
> # modprobe mce-inject
>
> (2) Find a PID of target qemu-kvm and page struct
> # ps -C qemu-kvm -o pid=
>   8176
>
> (3) Edit software pseudo MCE data
> Choose a offset of page struct and insert the offset to ADDR line in mce-file.
>
> #  ./page-types -p 8176 -LN | grep "___UDlA____Ma_b___________________"
> voffset         offset  flags
> ...
> 7fd25eb77       344d77  ___UDlA____Ma_b___________________
> 7fd25eb78       344d78  ___UDlA____Ma_b___________________
> 7fd25eb79       344d79  ___UDlA____Ma_b___________________
> 7fd25eb7a       344d7a  ___UDlA____Ma_b___________________
> 7fd25eb7b       344d7b  ___UDlA____Ma_b___________________
> 7fd25eb7c       344d7c  ___UDlA____Ma_b___________________
> 7fd25eb7d       344d7d  ___UDlA____Ma_b___________________
> ...
>
> # vi mce-file
> CPU 0 BANK 2
> STATUS UNCORRECTED SRAO 0x17a
> MCGSTATUS MCIP RIPV
> MISC 0x8c
> ADDR 0x344d77000
> EOF
>
> (4) Inject MCE
> # mce-inject mce-file
>
> Try step (3) to (4) a couple of times
>
>
>
>
> ******** Guest OS Screen logs(kdump) ********
> Receive MCE from KVM host
>
> (1) Set vm.memory_failure_dirty_panic parameter to 1.
>
> (2) When guest catches MCE injection from qemu and
>      MCE hit dirty page cache, hwpoison dirty cache handler
>      displays information of affected file such as Device name,
>      Inode Number, Offset and File Type.
>      And then, system goes a panic.
>
> ex.
> -------------
> [root@host /]# sysctl -a | grep memory_failure
> vm.memory_failure_dirty_panic = 1
> vm.memory_failure_early_kill = 0
> vm.memory_failure_recovery = 1
> [root@host /]#
> [  517.975220] MCE 0x326e6: clean LRU page recovery: Recovered
> [  521.969218] MCE 0x34df8: clean LRU page recovery: Recovered
> [  525.769171] MCE 0x37509: corrupted page was clean: dropped without side effects
> [  525.771070] MCE 0x37509: clean LRU page recovery: Recovered
> [  529.969246] MCE 0x39c18: File was corrupted: Dev:vda3 Inode:808998 Offset:6561
> [  529.969995] Kernel panic - not syncing: MCE 0x39c18: Force a panic because of dirty page cache was corrupted : File type:0x81a4
> [  529.969995]
> [  529.970055] Pid: 245, comm: kworker/0:2 Tainted: G   M         3.8.1 #22
> [  529.970055] Call Trace:
> [  529.970055]  [<ffffffff81645d1e>] panic+0xc1/0x1d0
> [  529.970055]  [<ffffffff811991fa>] me_pagecache_dirty+0xda/0x1a0
> [  529.970055]  [<ffffffff8119a0ab>] memory_failure+0x4eb/0xca0
> [  529.970055]  [<ffffffff8102f34d>] mce_process_work+0x3d/0x60
> [  529.970055]  [<ffffffff8107a5f7>] process_one_work+0x147/0x490
> [  529.970055]  [<ffffffff8102f310>] ? mce_schedule_work+0x50/0x50
> [  529.970055]  [<ffffffff8107ce8e>] worker_thread+0x15e/0x450
> [  529.970055]  [<ffffffff8107cd30>] ? busy_worker_rebind_fn+0x110/0x110
> [  529.970055]  [<ffffffff81081f50>] kthread+0xc0/0xd0
> [  529.970055]  [<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0xa0/0xf0
> [  529.970055]  [<ffffffff81081e90>] ? kthread_create_on_node+0x120/0x120
> [  529.970055]  [<ffffffff81657cec>] ret_from_fork+0x7c/0xb0
> [  529.970055]  [<ffffffff81081e90>] ? kthread_create_on_node+0x120/0x120
> -------------
>
> (3) Case of a number of MCE occurs
> If a number of MCE occurs during a minute, error messages are suppressed to output
> by __ratelimit() with following message.
>
> [  414.815303] me_pagecache_dirty: 3 callbacks suppressed
>
> ex.
> -------------
> [root@host /]# sysctl -a | grep memory_failure
> vm.memory_failure_dirty_panic = 0
> vm.memory_failure_early_kill = 0
> vm.memory_failure_print_ratelimit = 30
> vm.memory_failure_print_ratelimit_burst = 2
> vm.memory_failure_recovery = 1
>
> [root@host /]#
>
> [  181.565534] MCE 0xc38c: File was corrupted: Dev:vda3 Inode:808998 Offset:9566
> [  181.566310] MCE 0xc38c: dirty LRU page recovery: Recovered
> [  183.525425] MCE 0xc45a: Unknown page state
> [  183.527225] MCE 0xc45a: unknown page state page recovery: Failed
> [  183.527907] MCE 0xc45a: unknown page state page still referenced by -1 users
> [  185.000329] MCE 0xc524: dirty LRU page recovery: Recovered
> [  186.065231] MCE 0xc5ef: dirty LRU page recovery: Recovered
> [  188.054096] MCE 0xc6ba: clean LRU page recovery: Recovered
> [  189.565275] MCE 0xc783: clean LRU page recovery: Recovered
> [  191.692628] MCE 0xc84c: clean LRU page recovery: Recovered
> [  193.000257] MCE 0xc91d: File was corrupted: Dev:vda3 Inode:808998 Offset:6201
> [  193.001222] MCE 0xc91d: dirty LRU page recovery: Recovered
> [  194.065314] MCE 0xc9e6: dirty LRU page recovery: Recovered
> [  195.711211] MCE 0xcaaf: clean LRU page recovery: Recovered
> [  197.565339] MCE 0xcb78: dirty LRU page recovery: Recovered
> [  200.054177] MCE 0xcc41: dirty LRU page recovery: Recovered
> [  201.000272] MCE 0xcd0a: clean LRU page recovery: Recovered
> [  204.054109] MCE 0xcdd3: clean LRU page recovery: Recovered
> [  205.283189] MCE 0xcf65: clean LRU page recovery: Recovered
> [  207.110339] MCE 0xd02f: Unknown page state
> [  207.110787] MCE 0xd02f: unknown page state page recovery: Failed
> [  207.111427] MCE 0xd02f: unknown page state page still referenced by -1 users
> [  209.000134] MCE 0xd0f9: dirty LRU page recovery: Recovered
> [  210.106360] MCE 0xd1c5: dirty LRU page recovery: Recovered
> [  211.796333] me_pagecache_dirty: 3 callbacks suppressed
> [  211.796961] MCE 0xd296: File was corrupted: Dev:vda3 Inode:808998 Offset:9320
> [  211.798091] MCE 0xd296: dirty LRU page recovery: Recovered
> [  213.565288] MCE 0xd35f: clean LRU page recovery: Recovered
> -------------
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>