From: Adam Hamsik <adam.hamsik@chillisys.com>
To: linux-raid@vger.kernel.org
Cc: kernel-team@lists.ubuntu.com,
"Korbelář Jakub" <j.korbelar@radiokomunikace.cz>,
"Kouřil Přemysl" <P.Kouril@radiokomunikace.cz>,
cra@elitecode.cz
Subject: mdadm raid soft lock-ups ubuntu kernel 4.13.0-36
Date: Fri, 8 Jun 2018 15:20:57 +0200 [thread overview]
Message-ID: <CAOmBuBNmzD9PpGhKA6PdrqNLMFsXZ9GcbNiL8mxqkUNmGvYmsw@mail.gmail.com> (raw)
[-- Attachment #1.1: Type: text/plain, Size: 1770 bytes --]
Hi,
we're running Ubuntu 16.04.4, mdadm - v3.3 and Kernel 4.13.0-36.
We have created raid10 using 22 960GB SSDs [1] . The problem we're
experiencing is that /usr/share/mdadm/checkarray
(executed by cron, included in a mdadm pkg) results in (soft?)
deadlock - load on the node spikes up to 500-700 and all I/O operations
are blocked for a period of time. We can see traces liek these [2] in
our kernel log.
e.g. it ends up in static state like
test@os-node1:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md1 : active raid10 dm-23[9] dm-22[8] dm-21[7] dm-20[6] dm-18[4] dm-19[5]
dm-17[3]
dm-16[21] dm-15[20] dm-14[2] dm-13[19] dm-12[18]
dm-11[17]
dm-10[16] dm-9[15] dm-8[14] dm-7[13] dm-6[12] dm-5[11]
dm-4[10] dm-3[1] dm-2[0]
10313171968 blocks super 1.2 512K chunks 2 near-copies [22/22]
[UUUUUUUUUUUUUUUUUUUUUU]
[===>.................] check = 19.0% (1965748032/10313171968)
finish=1034728.8min speed=134K/sec
bitmap: 0/39 pages [0KB], 131072KB chunk
unused devices: <none>
and the only solution is to hard reboot the node. What we found out is that
it
doesn't happen on idle raid, we have to generate some significant load
(10 VMs running fio[3] with 500GB HDDs.) to be able to reproduce the issue.
Anyone ever experienced similar issues? Do you have any suggestions how to
better trouble shoot this issue and maybe identify if disks or software
layer
is responsible for this behaviour
[1] http://www.samsung.com/us/dell/pdfs/PM1633a_Flyer_2016_v4.pdf
[2] https://gist.github.com/haad/09213bab1bc30a00c7d255c0bc60897b
[3] https://github.com/axboe/fio
Regards
Adam.
Adam Hamsik
00421 904 937 495
adam.hamsik@chillisys.com
haad@netbsd.org
[-- Attachment #1.2: Type: text/html, Size: 2424 bytes --]
[-- Attachment #2: Type: text/plain, Size: 112 bytes --]
--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
reply other threads:[~2018-06-08 13:20 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAOmBuBNmzD9PpGhKA6PdrqNLMFsXZ9GcbNiL8mxqkUNmGvYmsw@mail.gmail.com \
--to=adam.hamsik@chillisys.com \
--cc=P.Kouril@radiokomunikace.cz \
--cc=cra@elitecode.cz \
--cc=j.korbelar@radiokomunikace.cz \
--cc=kernel-team@lists.ubuntu.com \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).