Re: Pausing md check hangs

[Guoqing Jiang <guoqing.jiang@cloud.ionos.com>]

On 1/27/20 2:42 PM, Georgi Nikolov wrote:
> Hi,
> 
> I posted a kernel bug about this a month ago but it did not receive any attention: 
> https://bugzilla.kernel.org/show_bug.cgi?id=205929
> Here is a copy of the bug report and I hope that this is the correct place to discuss this:
> 
> I have a Supermicro server with 10 md raid6 arrays each consisting of 8 SATA drives. SATA drives are 
> Hitachi/HGST Ultrastar 7K4000 8T.
> When i try to pause array check with "echo idle > "/sys/block/<md_dev>/md/sync_action" it randomly 
> hangs at different md device.
> Process "mdX_raid6" is at 100% cpu usage.cat /sys/block/mdX/md/journal_mode hungs forever.

Hmm, "echo idle > /sys/block/<md_dev>/md/sync_action" can't get reconfig_mutex, but seems
it should acquire mddev->lock instead of reconfig_mutex to align with other read functions
of raid5_attrs. I think we need to change it.

diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c
index 9b6da759dca2..a961d8eed73e 100644
--- a/drivers/md/raid5-cache.c
+++ b/drivers/md/raid5-cache.c
@@ -2532,13 +2532,10 @@ static ssize_t r5c_journal_mode_show(struct mddev *mddev, char *page)
         struct r5conf *conf;
         int ret;

-       ret = mddev_lock(mddev);
-       if (ret)
-               return ret;
-
+       spin_lock(&mddev->lock);
         conf = mddev->private;
         if (!conf || !conf->log) {
-               mddev_unlock(mddev);
+               spin_unlock(&mddev->lock);
                 return 0;
         }

@@ -2558,7 +2555,7 @@ static ssize_t r5c_journal_mode_show(struct mddev *mddev, char *page)
         default:
                 ret = 0;
         }
-       mddev_unlock(mddev);
+       spin_unlock(&mddev->lock);
         return ret;
  }


> 
> Here is the state at the moment of crash for one of the md devices:
> 
> root@supermicro:/sys/block/mdX/md# find -mindepth 1 -maxdepth 1 -type f|sort|grep -v 
> journal_mode|xargs -r egrep .
> ./array_size:default
> ./array_state:write-pending

MD_SB_CHANGE_PENDING was set, so md_update_sb() didn't clear it yet.

> grep: ./bitmap_set_bits: Permission denied
> ./chunk_size:524288
> ./component_size:7813895168
> ./consistency_policy:resync
> ./degraded:0
> ./group_thread_cnt:4

To narrow down the issue, I'd suggest to disable multiple raid5 workers.

> ./last_sync_action:check
> ./layout:2
> ./level:raid6
> ./max_read_errors:20
> ./metadata_version:1.2
> ./mismatch_cnt:0
> grep: ./new_dev: Permission denied
> ./preread_bypass_threshold:1
> ./raid_disks:8
> ./reshape_direction:forwards
> ./reshape_position:none
> ./resync_start:none
> ./rmw_level:1
> ./safe_mode_delay:0.204
> ./skip_copy:0
> ./stripe_cache_active:13173
> ./stripe_cache_size:8192
> ./suspend_hi:0
> ./suspend_lo:0
> ./sync_action:check

Since it was 'check' for sync_action means the array was set with RECOVERY_RUNNING,
RECOVERY_SYNC and RECOVERY_CHECK.

> ./sync_completed:3566405120 / 15627790336
> ./sync_force_parallel:0
> ./sync_max:max
> ./sync_min:1821385984
> ./sync_speed:126
> ./sync_speed_max:1000 (local)
> ./sync_speed_min:1000 (system)

The sync_speed is really low which means the system was under heavy pressure.

> 
> root@supermicro:~# cat /proc/mdstat
> Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
> md4 : active raid6 sdaa[2] sdab[3] sdy[0] sdae[6] sdac[4] sdad[5] sdaf[7] sdz[1]
>        46883371008 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
>        [====>................]  check = 22.8% (1784112640/7813895168) finish=20571.7min speed=4884K/sec
> 

When check was in progress, I guess write 'idle' to sync_action was not quit in below
section. Specifically, I think it was waiting the completion of md_misc_wq, otherwise
the check should be interrupted after set MD_RECOVERY_INTR.

         if (cmd_match(page, "idle") || cmd_match(page, "frozen")) {
                 if (cmd_match(page, "frozen"))
                         set_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
                 else
                         clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery);
                 if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) &&
                     mddev_lock(mddev) == 0) {
                         flush_workqueue(md_misc_wq);
                         if (mddev->sync_thread) {
                                 set_bit(MD_RECOVERY_INTR, &mddev->recovery);
                                 md_reap_sync_thread(mddev);
                         }
                         mddev_unlock(mddev);



But it is not clear to me why flush_workqueue was not finished. Maybe the below could
help but I am not sure.

--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -4779,7 +4779,8 @@ action_store(struct mddev *mddev, const char *page, size_t len)
                 if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) &&
                     mddev_lock(mddev) == 0) {
                         flush_workqueue(md_misc_wq);
-                       if (mddev->sync_thread) {
+                       if (mddev->sync_thread ||
+                           test_bit(MD_RECOVERY_RUNNING,&mddev->recovery)) {


Thanks,
Guoqing