* Re: Fwd: RAID6 Array crash during reshape.....now will not re-assemble.
From: Sarah Newman @ 2016-03-03 17:48 UTC (permalink / raw)
To: Another Sillyname, Linux-RAID
In-Reply-To: <CAOS+5GHUKD63M90zWfiv14QkJunj2QF+gYqDMrg208-S4oXE2Q@mail.gmail.com>
On 03/03/2016 06:07 AM, Another Sillyname wrote:
> Plenty of hits....but no clear fixes and as I said I'm not willing to
> 'try' things from google hits until someone with better insight can
> give me a view.
>
> Trying things is fine in many instances, but not with a 20+TB data set
> as I'm sure you can understand.
>
I have not tried this personally, but it may be of interest:
https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file I found this explanation
of the dmsetup commands to be more easy to follow: http://www.flaterco.com/kb/sandbox.html
^ permalink raw reply
* Re: [PATCH RESEND v2 11/18] fs: Ensure the mounter of a filesystem is privileged towards its inodes
From: Seth Forshee @ 2016-03-03 17:02 UTC (permalink / raw)
To: Eric W. Biederman
Cc: Alexander Viro, Serge Hallyn, Richard Weinberger,
Austin S Hemmelgarn, Miklos Szeredi, linux-kernel, linux-bcache,
dm-devel, linux-raid, linux-mtd, linux-fsdevel, fuse-devel,
linux-security-module, selinux
In-Reply-To: <1451930639-94331-12-git-send-email-seth.forshee@canonical.com>
On Mon, Jan 04, 2016 at 12:03:50PM -0600, Seth Forshee wrote:
> The mounter of a filesystem should be privileged towards the
> inodes of that filesystem. Extend the checks in
> inode_owner_or_capable() and capable_wrt_inode_uidgid() to
> permit access by users priviliged in the user namespace of the
> inode's superblock.
Eric - I've discovered a problem related to this patch. The patches
you've already applied to your testing branch make it so that s_user_ns
can be an unprivileged user for proc and kernfs-based mounts. In some
cases DAC is the only thing protecting files in these mounts (ignoring
MAC), and with this patch an unprivileged user could bypass DAC.
There's a simple solution - always set s_user_ns to &init_user_ns for
those filesystems. I think this is the right thing to do, since the
backing store behind these filesystems are really kernel objects. But
this would break the assumption behind your patch "userns: Simpilify
MNT_NODEV handling" and cause a regression in mounting behavior.
I've come up with several possible solutions for this conflict.
1. Drop this patch and keep on setting s_user_ns to unprivilged users.
This would be unfortunate because I think this patch does make sense
for most filesystems.
2. Restrict this patch so that a user privileged towards s_user_ns is
only privileged towards the super blocks inodes if s_user_ns has a
mapping for both i_uid and i_gid. This is better than (1) but still
not ideal in my mind.
3. Drop your patch and maintain the current MNT_NODEV behavior.
4. Add a new s_iflags flag to indicate a super block is from an
unprivileged mount, and use this in your patch instead of s_user_ns.
Any preference, or any other ideas?
Thanks,
Seth
^ permalink raw reply
* Re: RAID5/6 slow due to a member device -- how to diagnose?
From: Roman Mamedov @ 2016-03-03 15:00 UTC (permalink / raw)
To: Mateusz Korniak; +Cc: linux-raid
In-Reply-To: <4472129.pebMKxQj7W@matkor-toshiba>
[-- Attachment #1: Type: text/plain, Size: 875 bytes --]
On Thu, 03 Mar 2016 15:35:21 +0100
Mateusz Korniak <mateusz-lists@ant.gliwice.pl> wrote:
> On Thursday 03 March 2016 19:30:26 Roman Mamedov wrote:
> > But is that broken down somewhere to show which particular block device is
> > being the most 'waited for'?
>
> iostat ?
>
> r_await w_await svctm %util columns?
Cheers -- never knew there was this 'extended' output mode in iostat.
Indeed it pinpoints the culprit nicely.
Device: [...] wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sde1 [...] 53748.00 969.01 1.95 17.46 68.00 17.00 4.09 45.40
sdf1 [...] 49410.75 1176.97 32.09 393.95 106.00 397.42 11.90 100.00
sdg1 [...] 50230.00 995.17 7.12 66.81 88.00 66.60 6.08 61.40
sdh1 [...] 53110.00 970.89 2.46 22.21 46.40 21.64 4.64 50.80
--
With respect,
Roman
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* ∣ iphone 6s,388euro
From: hi @ 2016-03-03 14:37 UTC (permalink / raw)
To: marinogiorgio
good news, good news
iphone 6s,388euro,huawei mate7,200euro,dj,gultar.....free shipping
s ite: tooooel .com
^ permalink raw reply
* Re: RAID5/6 slow due to a member device -- how to diagnose?
From: Carlos Carvalho @ 2016-03-03 14:37 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20160303193026.7d08e64b@natsu>
Roman Mamedov (rm@romanrm.net) wrote on Thu, Mar 03, 2016 at 11:30:26AM BRT:
> Let's suppose I am seeing slow write performance on a RAID5 or RAID6, and I am
> suspecting that it's because of one of its devices operating slower than
> others (maybe a soft-failing or just low-performing hard drive, in case with
> non-identical member drives). Nothing in dmesg, no timeouts or ATA errors or
> anything like that.
>
> How would I go about diagnosing which device that is (if any) is slowing down
> the array, aside from performance-testing each individual device (which seems
> rather difficult if it's only slow on writes, not reads).
I use sar -d -p 5 |grep -v md
^ permalink raw reply
* Re: RAID5/6 slow due to a member device -- how to diagnose?
From: Mateusz Korniak @ 2016-03-03 14:35 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-raid
In-Reply-To: <20160303193026.7d08e64b@natsu>
On Thursday 03 March 2016 19:30:26 Roman Mamedov wrote:
> But is that broken down somewhere to show which particular block device is
> being the most 'waited for'?
iostat ?
r_await w_await svctm %util columns?
Regards,
--
Mateusz Korniak
"(...) mam brata - poważny, domator, liczykrupa, hipokryta, pobożniś,
krótko mówiąc - podpora społeczeństwa."
Nikos Kazantzakis - "Grek Zorba"
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RAID5/6 slow due to a member device -- how to diagnose?
From: Roman Mamedov @ 2016-03-03 14:30 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 773 bytes --]
Hello,
Let's suppose I am seeing slow write performance on a RAID5 or RAID6, and I am
suspecting that it's because of one of its devices operating slower than
others (maybe a soft-failing or just low-performing hard drive, in case with
non-identical member drives). Nothing in dmesg, no timeouts or ATA errors or
anything like that.
How would I go about diagnosing which device that is (if any) is slowing down
the array, aside from performance-testing each individual device (which seems
rather difficult if it's only slow on writes, not reads).
In 'top' there's of course high iowait, 50-70% all the time during writes. But
is that broken down somewhere to show which particular block device is being
the most 'waited for'?
--
With respect,
Roman
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Fwd: RAID6 Array crash during reshape.....now will not re-assemble.
From: Another Sillyname @ 2016-03-03 14:07 UTC (permalink / raw)
To: Linux-RAID
In-Reply-To: <CAOS+5GH1Rcu8zGk1dQ+aSNmVzjo=irH65KfPuq1ZGruzqX_=vg@mail.gmail.com>
Plenty of hits....but no clear fixes and as I said I'm not willing to
'try' things from google hits until someone with better insight can
give me a view.
Trying things is fine in many instances, but not with a 20+TB data set
as I'm sure you can understand.
On 3 March 2016 at 12:56, Wols Lists <antlists@youngman.org.uk> wrote:
> On 03/03/16 11:37, Another Sillyname wrote:
>> I'd even be happy to just bring the array up in readonly mode and
>> transfer the data off, but it will NOT let me reassemble the array
>> without the 'need data backup file to finish reshape, sorry' error and
>> will not reassemble.
>
> Seeing as no-one else has joined in, search the list archives for that
> error message, and you should get plenty of hits ...
>
> Cheers,
> Wol
^ permalink raw reply
* Re: RAID6 Array crash during reshape.....now will not re-assemble.
From: Wols Lists @ 2016-03-03 12:56 UTC (permalink / raw)
To: Another Sillyname, Linux-RAID
In-Reply-To: <CAOS+5GGB5gVSFJbYwwJu+z0C0oyYPzsLRhnjD6FkmLEWFFquFA@mail.gmail.com>
On 03/03/16 11:37, Another Sillyname wrote:
> I'd even be happy to just bring the array up in readonly mode and
> transfer the data off, but it will NOT let me reassemble the array
> without the 'need data backup file to finish reshape, sorry' error and
> will not reassemble.
Seeing as no-one else has joined in, search the list archives for that
error message, and you should get plenty of hits ...
Cheers,
Wol
^ permalink raw reply
* Re: RAID6 Array crash during reshape.....now will not re-assemble.
From: Another Sillyname @ 2016-03-03 11:37 UTC (permalink / raw)
To: Linux-RAID
In-Reply-To: <CAOS+5GGX8QBo3=s+Mcj=Kd6VSmZyatfZ30NXBy8E=4WJE6z0sA@mail.gmail.com>
Just to add a bit more to this.....
It looks like the backup file is just full of EOLs (I haven't looked
at it with a bit editor admittedly).
So I'm absolutely stuck now and would really appreciate some help.
I'd even be happy to just bring the array up in readonly mode and
transfer the data off, but it will NOT let me reassemble the array
without the 'need data backup file to finish reshape, sorry' error and
will not reassemble.
Anyone?
On 2 March 2016 at 15:59, Another Sillyname <anothersname@googlemail.com> wrote:
> I've found out more info....and now have a theory.......but do not
> know how best to proceed.
>
>>sudo mdadm -A --scan --verbose
>
> mdadm: looking for devices for further assembly
> mdadm: No super block found on /dev/sdh (Expected magic a92b4efc, got 00000000)
> mdadm: no RAID superblock on /dev/sdh
> mdadm: No super block found on /dev/sdg (Expected magic a92b4efc, got 00000000)
> mdadm: no RAID superblock on /dev/sdg
> mdadm: No super block found on /dev/sdf (Expected magic a92b4efc, got 00000000)
> mdadm: no RAID superblock on /dev/sdf
> mdadm: No super block found on /dev/sde (Expected magic a92b4efc, got 00000000)
> mdadm: no RAID superblock on /dev/sde
> mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
> mdadm: no RAID superblock on /dev/sdd
> mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
> mdadm: no RAID superblock on /dev/sdc
> mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
> mdadm: no RAID superblock on /dev/sdb
> mdadm: No super block found on /dev/sda6 (Expected magic a92b4efc, got 00000000)
> mdadm: no RAID superblock on /dev/sda6
> mdadm: No super block found on /dev/sda5 (Expected magic a92b4efc, got 75412023)
> mdadm: no RAID superblock on /dev/sda5
> mdadm: /dev/sda4 is too small for md: size is 2 sectors.
> mdadm: no RAID superblock on /dev/sda4
> mdadm: No super block found on /dev/sda3 (Expected magic a92b4efc, got 00000401)
> mdadm: no RAID superblock on /dev/sda3
> mdadm: No super block found on /dev/sda2 (Expected magic a92b4efc, got 00000401)
> mdadm: no RAID superblock on /dev/sda2
> mdadm: No super block found on /dev/sda1 (Expected magic a92b4efc, got 0000007e)
> mdadm: no RAID superblock on /dev/sda1
> mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got e71e974a)
> mdadm: no RAID superblock on /dev/sda
> mdadm: /dev/sdh1 is identified as a member of /dev/md/server187:1, slot 5.
> mdadm: /dev/sdg1 is identified as a member of /dev/md/server187:1, slot 0.
> mdadm: /dev/sdf1 is identified as a member of /dev/md/server187:1, slot 2.
> mdadm: /dev/sde1 is identified as a member of /dev/md/server187:1, slot 3.
> mdadm: /dev/sdd1 is identified as a member of /dev/md/server187:1, slot 6.
> mdadm: /dev/sdc1 is identified as a member of /dev/md/server187:1, slot 1.
> mdadm: /dev/sdb1 is identified as a member of /dev/md/server187:1, slot 4.
> mdadm: /dev/md/server187:1 has an active reshape - checking if
> critical section needs to be restored
> mdadm: Failed to find backup of critical section
> mdadm: Failed to restore critical section for reshape, sorry.
> Possibly you needed to specify the --backup-file
> mdadm: looking for devices for further assembly
> mdadm: No arrays found in config file or automatically
>
> As I stated in my original posting I do not know where the server187
> stuff came from when I tried the original assemble and two of the
> drives (sdg & sdh) reported as busy.
>
> So my theory is this......
>
> This 30TB array has been up and active since about August 2015, fully
> functional without any major issues, except performance was sometimes
> a bit iffy.
>
> It is possible that drives sdg and sdh were used in a temporary box in
> a different array that was only active for about 10 days, before they
> were moved to the new 30TB array that was cleanly built. That array
> may well have been called server187 (it was a temp box so no reason to
> remember it).
>
> When the reshape of the current array 'died' during initialisation or
> immediately thereafter, even though cat /proc/mdstat showed the
> reshape active after 12 hours it was still stuck on 0.0%.
>
> When the machine was rebooted and the array didn't come up...is it
> possible that drives sdh and sdg still thought they were in the old
> server187 array and that is why they reported themselves busy? I'm
> not sure why this would happen, but am just theorising.
>
> When I tried the assemble command where it reported it was merging
> with the already existing server187 array, even though there
> wasn't/isn't a server187 array as prior to that assemble cat
> /proc/mdstat reported the offline md127 array.
>
> Somehow therefore the array names have got confused/transposed and
> that's why the backup file is now not seen as the correct one? This
> would seem to be borne out by all the drives now seeing themselves as
> part of server187 array rather then md127 array and also the reshape
> seems to be attached to this server187 array.
>
> I still believe/hope the data is all still intact and complete,
> however I am averse to just hacking around using google to 'try
> commands' hoping I hit a solution before someone with much more
> experience casts an eye over this to give me a little guidance.
>
> Help!!
>
>
>
> On 2 March 2016 at 13:42, Another Sillyname <anothersname@googlemail.com> wrote:
>> Kernel is latest Fedora x86_64 4.3.5-300, can't get too much newer
>> then that (latest is 4.4.x), mdadm is 3.3.4-2.
>>
>> I agree that the data is likely still intact, doesn't stop me being
>> nervous till I see it though!!
>>
>>
>>
>> On 2 March 2016 at 13:20, Wols Lists <antlists@youngman.org.uk> wrote:
>>> On 02/03/16 03:46, Another Sillyname wrote:
>>>> Any help and guidance would be appreciated, the drives showing clean
>>>> gives me comfort that the data is likely intact and complete (crossed
>>>> fingers) however I can't re-assemble the array as I keep getting the
>>>> 'critical information for reshape, sorry' warning.
>>>>
>>>> Help???
>>>
>>> Someone else will chip in what to do, but this doesn't seem alarming at
>>> all. Reshapes stuck at zero is a recent bug, but all the data is
>>> probably safe and sound.
>>>
>>> Wait for one of the experts to chip in what to do, but you might find
>>> mdadm --resume --invalid-backup will get it going again.
>>>
>>> Otherwise it's likely to be an "upgrade your kernel and mdadm" job ...
>>>
>>> Cheers,
>>> Wol
^ permalink raw reply
* [PATCH 4/4] md: dm-crypt: Initialize the sector number for one request
From: Baolin Wang @ 2016-03-03 5:19 UTC (permalink / raw)
To: herbert, davem, agk, snitzer, axboe, dm-devel
Cc: akpm, david.s.gordon, thomas.lendacky, robert.jarzmik,
yamada.masahiro, smueller, tadeusz.struk, standby24x7, shli,
broonie, linus.walleij, arnd, baolin.wang, linux-kernel,
linux-crypto, linux-raid
In-Reply-To: <cover.1456981314.git.baolin.wang@linaro.org>
If the crypto engine can support the bulk mode, that means the contiguous
requests from one block can be merged into one request to be handled by
crypto engine. If so, the crypto engine need the sector number of one request
to do merging action.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/md/dm-crypt.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 3147c8d..9e2dbfd 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -866,6 +866,7 @@ static int crypt_convert_block(struct crypt_config *cc,
return r;
}
+ req->sector = ctx->cc_sector;
ablkcipher_request_set_crypt(req, &dmreq->sg_in, &dmreq->sg_out,
1 << SECTOR_SHIFT, iv);
--
1.7.9.5
^ permalink raw reply related
* [PATCH 3/4] crypto: Introduce the bulk mode for crypto engine framework
From: Baolin Wang @ 2016-03-03 5:19 UTC (permalink / raw)
To: herbert, davem, agk, snitzer, axboe, dm-devel
Cc: akpm, david.s.gordon, thomas.lendacky, robert.jarzmik,
yamada.masahiro, smueller, tadeusz.struk, standby24x7, shli,
broonie, linus.walleij, arnd, baolin.wang, linux-kernel,
linux-crypto, linux-raid
In-Reply-To: <cover.1456981314.git.baolin.wang@linaro.org>
Now some cipher hardware engines prefer to handle bulk block by merging
requests to increase the block size and thus increase the hardware engine
processing speed.
This patch introduces request bulk mode to help the crypto hardware drivers
improve in efficiency, and chooses the suitable mode (SECTOR_MODE) for
initializing aes engine.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
crypto/Kconfig | 1 +
crypto/crypto_engine.c | 122 +++++++++++++++++++++++++++++++++++++++++++--
drivers/crypto/omap-aes.c | 2 +-
include/crypto/algapi.h | 23 ++++++++-
4 files changed, 143 insertions(+), 5 deletions(-)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index c844227..6a2f9a6 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -229,6 +229,7 @@ config CRYPTO_GLUE_HELPER_X86
config CRYPTO_ENGINE
tristate
+ select CRYPTO_ABLK_HELPER
comment "Authenticated Encryption with Associated Data"
diff --git a/crypto/crypto_engine.c b/crypto/crypto_engine.c
index a55c82d..0de5829 100644
--- a/crypto/crypto_engine.c
+++ b/crypto/crypto_engine.c
@@ -14,6 +14,7 @@
#include <linux/err.h>
#include <linux/delay.h>
+#include <crypto/ablk_helper.h>
#include "internal.h"
#define CRYPTO_ENGINE_MAX_QLEN 10
@@ -84,6 +85,17 @@ static void crypto_pump_requests(struct crypto_engine *engine,
req = ablkcipher_request_cast(async_req);
+ /*
+ * If the engine supports the bulk mode and the request is allocated the
+ * sg table to expand scatterlists entries, then it need to point the
+ * scatterlists from the sg table.
+ */
+ if (engine->mode == SECTOR_BULK_MODE && req->sgt_src.orig_nents &&
+ req->sgt_dst.orig_nents) {
+ req->src = req->sgt_src.sgl;
+ req->dst = req->sgt_dst.sgl;
+ }
+
engine->cur_req = req;
if (backlog)
backlog->complete(backlog, -EINPROGRESS);
@@ -137,9 +149,46 @@ static void crypto_pump_work(struct kthread_work *work)
}
/**
+ * crypto_merge_request_to_engine - try to merge one request into previous one
+ * @engine: the hardware engine
+ * @req: the request need to be merged
+ *
+ * If the crypto engine supports bulk mode, then try to merge the new request
+ * into the listed one from engine queue to handle them together.
+ *
+ * Return 0 on success and others are failed.
+ */
+static bool crypto_merge_request_to_engine(struct crypto_engine *engine,
+ struct ablkcipher_request *req)
+{
+ /*
+ * The request is allocated from memory pool in dm-crypt, here need to
+ * do initialization for sg table in case some random values.
+ */
+ req->sgt_src.orig_nents = 0;
+ req->sgt_dst.orig_nents = 0;
+
+ /*
+ * If the hardware engine can not support the bulk mode encryption,
+ * just return 1 means merging failed.
+ */
+ if (engine->mode != SECTOR_BULK_MODE)
+ return 1;
+
+ return ablk_try_merge(&engine->queue, req);
+}
+
+/**
* crypto_transfer_request - transfer the new request into the engine queue
* @engine: the hardware engine
* @req: the request need to be listed into the engine queue
+ *
+ * Firstly it will check if the new request can be merged into previous one
+ * if their secotr numbers are continuous, if not should list it into engine
+ * queue.
+ *
+ * If the new request can be merged into the previous request, then just
+ * finalize the new request.
*/
int crypto_transfer_request(struct crypto_engine *engine,
struct ablkcipher_request *req, bool need_pump)
@@ -154,6 +203,26 @@ int crypto_transfer_request(struct crypto_engine *engine,
return -ESHUTDOWN;
}
+ /*
+ * Here need to check if the request can be merged into previous
+ * request. If the hardware engine can support encryption with
+ * bulk block, we can merge the new request into previous request
+ * if their secotr numbers are continuous, which can be handled
+ * together by engine to improve the encryption efficiency.
+ * Return -EINPROGRESS means it has been merged into previous request,
+ * so just end up this request.
+ */
+ ret = crypto_merge_request_to_engine(engine, req);
+ if (!ret) {
+ spin_unlock_irqrestore(&engine->queue_lock, flags);
+ crypto_finalize_request(engine, req, 0);
+ return -EINPROGRESS;
+ }
+
+ /*
+ * If the request can not be merged into previous request, then list it
+ * into the queue of engine, and will be handled by kthread worker.
+ */
ret = ablkcipher_enqueue_request(&engine->queue, req);
if (!engine->busy && need_pump)
@@ -178,7 +247,8 @@ int crypto_transfer_request_to_engine(struct crypto_engine *engine,
EXPORT_SYMBOL_GPL(crypto_transfer_request_to_engine);
/**
- * crypto_finalize_request - finalize one request if the request is done
+ * crypto_finalize_request - finalize one request if the request is done or
+ * merged into previous request
* @engine: the hardware engine
* @req: the request need to be finalized
* @err: error number
@@ -208,9 +278,18 @@ void crypto_finalize_request(struct crypto_engine *engine,
spin_unlock_irqrestore(&engine->queue_lock, flags);
}
+ sg_free_table(&req->sgt_src);
+ sg_free_table(&req->sgt_dst);
req->base.complete(&req->base, err);
- queue_kthread_work(&engine->kworker, &engine->pump_requests);
+ /*
+ * If the request is finalized by merging into the previous request from
+ * the engine queue, then it is no need to queue the kthread work.
+ * Cause now maybe there are other requests need to be merged into the
+ * listed request from one block, just wait for merging action.
+ */
+ if (finalize_cur_req)
+ queue_kthread_work(&engine->kworker, &engine->pump_requests);
}
EXPORT_SYMBOL_GPL(crypto_finalize_request);
@@ -279,15 +358,45 @@ int crypto_engine_stop(struct crypto_engine *engine)
EXPORT_SYMBOL_GPL(crypto_engine_stop);
/**
+ * crypto_engine_change_mode - Change the mode for hardware engine
+ * @engine: the hardware engine
+ * @mode: engine mode to be set
+ *
+ * This function can change the hardware engine mode when the engine is running.
+ * Return 0 on success, else on fail.
+ */
+int crypto_engine_change_mode(struct crypto_engine *engine,
+ enum engine_mode mode)
+{
+ unsigned long flags;
+ int ret;
+
+ ret = crypto_engine_stop(engine);
+ if (ret) {
+ pr_warn("could not change engine mode now\n");
+ return ret;
+ }
+
+ spin_lock_irqsave(&engine->queue_lock, flags);
+ engine->mode = mode;
+ spin_unlock_irqrestore(&engine->queue_lock, flags);
+
+ return crypto_engine_start(engine);
+}
+EXPORT_SYMBOL_GPL(crypto_engine_change_mode);
+
+/**
* crypto_engine_alloc_init - allocate crypto hardware engine structure and
* initialize it.
* @dev: the device attached with one hardware engine
+ * @mode: crypto engine mode
* @rt: whether this queue is set to run as a realtime task
*
* This must be called from context that can sleep.
* Return: the crypto engine structure on success, else NULL.
*/
-struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt)
+struct crypto_engine *crypto_engine_alloc_init(struct device *dev,
+ enum engine_mode mode, bool rt)
{
struct sched_param param = { .sched_priority = MAX_RT_PRIO - 1 };
struct crypto_engine *engine;
@@ -299,6 +408,13 @@ struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt)
if (!engine)
return NULL;
+ /*
+ * If the hardware engine can handle the IV by itself, that means it
+ * just need one initial IV for multiple requests from one block. So
+ * we can merge requests from one block into one request to handle,
+ * which can improve the hardware engine efficiency.
+ */
+ engine->mode = mode;
engine->rt = rt;
engine->running = false;
engine->busy = false;
diff --git a/drivers/crypto/omap-aes.c b/drivers/crypto/omap-aes.c
index d420ec7..946f11f 100644
--- a/drivers/crypto/omap-aes.c
+++ b/drivers/crypto/omap-aes.c
@@ -1230,7 +1230,7 @@ static int omap_aes_probe(struct platform_device *pdev)
}
/* Initialize crypto engine */
- dd->engine = crypto_engine_alloc_init(dev, 1);
+ dd->engine = crypto_engine_alloc_init(dev, SECTOR_MODE, 1);
if (!dd->engine)
goto err_algs;
diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index b09d43f..69fb43e 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -130,6 +130,22 @@ struct ablkcipher_walk {
};
#define ENGINE_NAME_LEN 30
+
+/*
+ * enum engine_mode - crypto engine mode
+ * @SECTOR_MODE: should do encryption/decryption one request by one (one
+ * request length is one sector size), and it will not coalesce requests.
+ * @SECTOR_BULK_MODE: it can coalesce the contiguous requests (one request
+ * length is one sector size) together to be one bulk request, which can be
+ * handled by crypto engine at one time.
+ * @MAX_MODE: engine mode numbers
+ */
+enum engine_mode {
+ SECTOR_MODE,
+ SECTOR_BULK_MODE,
+ MAX_MODE,
+};
+
/*
* struct crypto_engine - crypto hardware engine
* @name: the engine name
@@ -140,6 +156,7 @@ struct ablkcipher_walk {
* @list: link with the global crypto engine list
* @queue_lock: spinlock to syncronise access to request queue
* @queue: the crypto queue of the engine
+ * @mode: crypto engine mode
* @rt: whether this queue is set to run as a realtime task
* @prepare_crypt_hardware: a request will soon arrive from the queue
* so the subsystem requests the driver to prepare the hardware
@@ -167,6 +184,7 @@ struct crypto_engine {
spinlock_t queue_lock;
struct crypto_queue queue;
+ enum engine_mode mode;
bool rt;
int (*prepare_crypt_hardware)(struct crypto_engine *engine);
@@ -195,7 +213,10 @@ void crypto_finalize_request(struct crypto_engine *engine,
struct ablkcipher_request *req, int err);
int crypto_engine_start(struct crypto_engine *engine);
int crypto_engine_stop(struct crypto_engine *engine);
-struct crypto_engine *crypto_engine_alloc_init(struct device *dev, bool rt);
+int crypto_engine_change_mode(struct crypto_engine *engine,
+ enum engine_mode mode);
+struct crypto_engine *crypto_engine_alloc_init(struct device *dev,
+ enum engine_mode mode, bool rt);
int crypto_engine_exit(struct crypto_engine *engine);
extern const struct crypto_type crypto_ablkcipher_type;
--
1.7.9.5
^ permalink raw reply related
* [PATCH 2/4] crypto: Introduce some helper functions to help to merge requests
From: Baolin Wang @ 2016-03-03 5:19 UTC (permalink / raw)
To: herbert, davem, agk, snitzer, axboe, dm-devel
Cc: akpm, david.s.gordon, thomas.lendacky, robert.jarzmik,
yamada.masahiro, smueller, tadeusz.struk, standby24x7, shli,
broonie, linus.walleij, arnd, baolin.wang, linux-kernel,
linux-crypto, linux-raid
In-Reply-To: <cover.1456981314.git.baolin.wang@linaro.org>
Usually the dm-crypt subsystem will send encryption/descryption requests to
the crypto layer one block at a time, making each request 512 bytes long,
which is a much smaller size for hardware engine, that means the hardware
engine can not play its best performance.
Now some cipher hardware engines prefer to handle bulk block rather than one
sector (512 bytes) created by dm-crypt, cause these cipher engines can handle
the intermediate values (IV) by themselves in one bulk block. This means we
can increase the size of the request by merging request rather than always 512
bytes and thus increase the hardware engine processing speed.
This patch introduces some helper functions to help to merge requests to improve
hardware engine efficiency.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
crypto/ablk_helper.c | 135 ++++++++++++++++++++++++++++++++++++++++++
include/crypto/ablk_helper.h | 3 +
include/linux/crypto.h | 5 ++
3 files changed, 143 insertions(+)
diff --git a/crypto/ablk_helper.c b/crypto/ablk_helper.c
index e1fcf53..3cf15cb 100644
--- a/crypto/ablk_helper.c
+++ b/crypto/ablk_helper.c
@@ -26,6 +26,7 @@
#include <linux/kernel.h>
#include <linux/crypto.h>
+#include <linux/device-mapper.h>
#include <linux/init.h>
#include <linux/module.h>
#include <linux/hardirq.h>
@@ -34,6 +35,140 @@
#include <crypto/ablk_helper.h>
#include <asm/simd.h>
+/**
+ * ablk_link_request_if_contigous - try to link one request into previous one
+ * if the page address is contiguous.
+ * @list_req: the request from queue list
+ * @req: the new request need to be merged
+ *
+ * If the listed request and new request's pages of the scatterlists are
+ * contiguous, then merge the scatterlists of new request into the listed one.
+ *
+ * Return true on success, others means failed.
+ */
+static bool ablk_link_request_if_contigous(struct ablkcipher_request *list_req,
+ struct ablkcipher_request *req)
+{
+ struct scatterlist *last_src_sg =
+ sg_last(list_req->sgt_src.sgl, list_req->sgt_src.nents);
+ struct scatterlist *last_dst_sg =
+ sg_last(list_req->sgt_dst.sgl, list_req->sgt_dst.nents);
+ struct scatterlist *req_src_sg = req->src;
+ struct scatterlist *req_dst_sg = req->dst;
+
+ if (!last_src_sg || !last_dst_sg)
+ return false;
+
+ /* Check if the src/dst scatterlists are contiguous */
+ if (!sg_is_contiguous(last_src_sg, req_src_sg) ||
+ !sg_is_contiguous(last_dst_sg, req_dst_sg))
+ return false;
+
+ /*
+ * If the request can be merged into the listed request after the
+ * checking, then expand the listed request scatterlists' length.
+ */
+ last_src_sg->length += req_src_sg->length;
+ last_dst_sg->length += req_dst_sg->length;
+ list_req->nbytes += req->nbytes;
+
+ return true;
+}
+
+/**
+ * ablk_merge_request - try to merge one request into previous one
+ * @list_req: the request from queue list
+ * @req: the request need to be merged
+ *
+ * This function will create a dynamic scatterlist table for both source
+ * and destination if the request is the first coming in.
+ *
+ * Return true on success, others means failed.
+ */
+static bool ablk_merge_request(struct ablkcipher_request *list_req,
+ struct ablkcipher_request *req)
+{
+ struct sg_table *sgt_src = &list_req->sgt_src;
+ struct sg_table *sgt_dst = &list_req->sgt_dst;
+ unsigned int nents = SG_MAX_SINGLE_ALLOC;
+
+ if (sg_table_is_empty(sgt_src)) {
+ if (sg_alloc_empty_table(sgt_src, nents, GFP_ATOMIC))
+ return false;
+
+ if (sg_add_sg_to_table(sgt_src, list_req->src))
+ return false;
+ }
+
+ if (sg_table_is_empty(sgt_dst)) {
+ if (sg_alloc_empty_table(sgt_dst, nents, GFP_ATOMIC))
+ return false;
+
+ if (sg_add_sg_to_table(sgt_dst, list_req->dst))
+ return false;
+ }
+
+ /*
+ * Check if the new request is contiguous for the listed request,
+ * if it is contiguous then merge the new request into the listed one.
+ */
+ if (ablk_link_request_if_contigous(list_req, req))
+ return true;
+
+ if (sg_add_sg_to_table(sgt_src, req->src))
+ return false;
+
+ if (sg_add_sg_to_table(sgt_dst, req->dst))
+ return false;
+
+ list_req->nbytes += req->nbytes;
+ return true;
+}
+
+/**
+ * ablk_try_merge - try to merge one request into previous one
+ * @queue: the crypto queue list
+ * @req: the request need to be merged
+ *
+ * Note: The merging action should be under the spinlock or mutex protection.
+ *
+ * Return 0 on success and others are failed.
+ */
+int ablk_try_merge(struct crypto_queue *queue,
+ struct ablkcipher_request *req)
+{
+ struct ablkcipher_request *list_req;
+ struct crypto_async_request *async_req;
+
+ list_for_each_entry(async_req, &queue->list, list) {
+ list_req = ablkcipher_request_cast(async_req);
+
+ if (list_req->base.flags != req->base.flags)
+ continue;
+
+ /* Check that the request adds up to an even number of sectors */
+ if (!IS_ALIGNED(list_req->nbytes, (1U << SECTOR_SHIFT)))
+ continue;
+
+ if (list_req->nbytes + req->nbytes > UINT_MAX)
+ continue;
+
+ /*
+ * We first check that the sectors are adjacent so we don't
+ * mistadly coalesce something that is contigous in memory but
+ * not contigous on disk.
+ */
+ if (list_req->sector + list_req->nbytes /
+ (1U << SECTOR_SHIFT) == req->sector) {
+ if (ablk_merge_request(list_req, req))
+ return 0;
+ }
+ }
+
+ return 1;
+}
+EXPORT_SYMBOL_GPL(ablk_try_merge);
+
int ablk_set_key(struct crypto_ablkcipher *tfm, const u8 *key,
unsigned int key_len)
{
diff --git a/include/crypto/ablk_helper.h b/include/crypto/ablk_helper.h
index 4f93df5..12ae00d 100644
--- a/include/crypto/ablk_helper.h
+++ b/include/crypto/ablk_helper.h
@@ -8,6 +8,7 @@
#include <linux/crypto.h>
#include <linux/kernel.h>
#include <crypto/cryptd.h>
+#include <crypto/algapi.h>
struct async_helper_ctx {
struct cryptd_ablkcipher *cryptd_tfm;
@@ -28,4 +29,6 @@ extern int ablk_init_common(struct crypto_tfm *tfm, const char *drv_name);
extern int ablk_init(struct crypto_tfm *tfm);
+extern int ablk_try_merge(struct crypto_queue *queue,
+ struct ablkcipher_request *req);
#endif /* _CRYPTO_ABLK_HELPER_H */
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index e71cb70..f878bb1 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -21,6 +21,7 @@
#include <linux/kernel.h>
#include <linux/list.h>
#include <linux/bug.h>
+#include <linux/scatterlist.h>
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/uaccess.h>
@@ -170,6 +171,10 @@ struct ablkcipher_request {
struct scatterlist *src;
struct scatterlist *dst;
+ struct sg_table sgt_src;
+ struct sg_table sgt_dst;
+ sector_t sector;
+
void *__ctx[] CRYPTO_MINALIGN_ATTR;
};
--
1.7.9.5
^ permalink raw reply related
* [PATCH 1/4] scatterlist: Introduce some helper functions
From: Baolin Wang @ 2016-03-03 5:19 UTC (permalink / raw)
To: herbert, davem, agk, snitzer, axboe, dm-devel
Cc: akpm, david.s.gordon, thomas.lendacky, robert.jarzmik,
yamada.masahiro, smueller, tadeusz.struk, standby24x7, shli,
broonie, linus.walleij, arnd, baolin.wang, linux-kernel,
linux-crypto, linux-raid
In-Reply-To: <cover.1456981314.git.baolin.wang@linaro.org>
In crypto engine framework, one request can combine other requests'
scatterlists into its sg table to improve engine efficency with
handling bulk block. Thus we need some helper functions to manage
dynamic scattertables.
This patch introduces 'sg_is_contiguous()' function to check if two
scatterlists are contiguous, 'sg_alloc_empty_table()' function to
allocate one empty sg table, 'sg_add_sg_to_table()' function to add
one scatterlist into sg table and 'sg_table_is_empty' function to
check if the sg table is empty.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
include/linux/scatterlist.h | 33 ++++++++++++++++++++++++
lib/scatterlist.c | 59 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+)
diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h
index 556ec1e..ae9aee6 100644
--- a/include/linux/scatterlist.h
+++ b/include/linux/scatterlist.h
@@ -212,6 +212,37 @@ static inline void sg_unmark_end(struct scatterlist *sg)
}
/**
+ * sg_is_contiguous - Check if the scatterlists are contiguous
+ * @sga: SG entry
+ * @sgb: SG entry
+ *
+ * Description:
+ * If the sga scatterlist is contiguous with the sgb scatterlist,
+ * that means they can be merged together.
+ *
+ **/
+static inline bool sg_is_contiguous(struct scatterlist *sga,
+ struct scatterlist *sgb)
+{
+ return ((sga->page_link & ~0x3UL) + sga->offset + sga->length ==
+ (sgb->page_link & ~0x3UL));
+}
+
+/**
+ * sg_table_is_empty - Check if the sg table is empty
+ * @sgt: sg table
+ *
+ * Description:
+ * The 'orig_nents' member of one sg table is used to indicate how many
+ * scatterlists in the sg table.
+ *
+ **/
+static inline bool sg_table_is_empty(struct sg_table *sgt)
+{
+ return !sgt->orig_nents;
+}
+
+/**
* sg_phys - Return physical address of an sg entry
* @sg: SG entry
*
@@ -261,6 +292,8 @@ void sg_free_table(struct sg_table *);
int __sg_alloc_table(struct sg_table *, unsigned int, unsigned int,
struct scatterlist *, gfp_t, sg_alloc_fn *);
int sg_alloc_table(struct sg_table *, unsigned int, gfp_t);
+int sg_alloc_empty_table(struct sg_table *, unsigned int, gfp_t);
+int sg_add_sg_to_table(struct sg_table *, struct scatterlist *);
int sg_alloc_table_from_pages(struct sg_table *sgt,
struct page **pages, unsigned int n_pages,
unsigned long offset, unsigned long size,
diff --git a/lib/scatterlist.c b/lib/scatterlist.c
index 004fc70..e9e5d5a 100644
--- a/lib/scatterlist.c
+++ b/lib/scatterlist.c
@@ -370,6 +370,65 @@ int sg_alloc_table(struct sg_table *table, unsigned int nents, gfp_t gfp_mask)
EXPORT_SYMBOL(sg_alloc_table);
/**
+ * sg_add_sg_to_table - Add one scatterlist into sg table
+ * @sgt: The sg table header to use
+ * @src: The sg need to be added into sg table
+ *
+ * Description:
+ * The 'nents' member indicates how many scatterlists added in the sg table.
+ * Copy the @src@ scatterlist into sg table and increase 'nents' member.
+ *
+ **/
+int sg_add_sg_to_table(struct sg_table *sgt, struct scatterlist *src)
+{
+ unsigned int i = 0, orig_nents = sgt->orig_nents;
+ struct scatterlist *sgl = sgt->sgl;
+ struct scatterlist *sg;
+
+ /* Check if there are enough space for the new sg to be added */
+ if (sgt->nents >= sgt->orig_nents)
+ return -EINVAL;
+
+ for_each_sg(sgl, sg, orig_nents, i) {
+ if (sgt->nents > 0 && i == (sgt->nents - 1)) {
+ sg_unmark_end(sg);
+ } else if (i == sgt->nents) {
+ memcpy(sg, src, sizeof(struct scatterlist));
+ sg_mark_end(sg);
+ sgt->nents++;
+ break;
+ }
+ }
+
+ return 0;
+}
+
+/**
+ * sg_alloc_empty_table - Allocate one empty sg table
+ * @sgt: The sg table header to use
+ * @nents: Number of entries in sg list
+ * @gfp_mask: GFP allocation mask
+ *
+ * Description:
+ * Allocate and initialize an sg table. The 'nents' member of sg_table
+ * indicates how many scatterlists added in the sg table. It should set
+ * 0 which means there are no scatterlists added in this sg table now.
+ *
+ **/
+int sg_alloc_empty_table(struct sg_table *sgt, unsigned int nents,
+ gfp_t gfp_mask)
+{
+ int ret;
+
+ ret = sg_alloc_table(sgt, nents, gfp_mask);
+ if (ret)
+ return ret;
+
+ sgt->nents = 0;
+ return 0;
+}
+
+/**
* sg_alloc_table_from_pages - Allocate and initialize an sg table from
* an array of pages
* @sgt: The sg table header to use
--
1.7.9.5
^ permalink raw reply related
* [PATCH 0/4] Introduce bulk mode for crypto engine framework
From: Baolin Wang @ 2016-03-03 5:19 UTC (permalink / raw)
To: herbert, davem, agk, snitzer, axboe, dm-devel
Cc: akpm, david.s.gordon, thomas.lendacky, robert.jarzmik,
yamada.masahiro, smueller, tadeusz.struk, standby24x7, shli,
broonie, linus.walleij, arnd, baolin.wang, linux-kernel,
linux-crypto, linux-raid
Now some cipher hardware engines prefer to handle bulk block by merging requests
to increase the block size and thus increase the hardware engine processing speed.
This patchset introduces request bulk mode to help the crypto hardware drivers
improve in efficiency.
Baolin Wang (4):
scatterlist: Introduce some helper functions
crypto: Introduce some helper functions to help to merge requests
crypto: Introduce the bulk mode for crypto engine framework
md: dm-crypt: Initialize the sector number for one request
crypto/Kconfig | 1 +
crypto/ablk_helper.c | 135 ++++++++++++++++++++++++++++++++++++++++++
crypto/crypto_engine.c | 122 +++++++++++++++++++++++++++++++++++++-
drivers/crypto/omap-aes.c | 2 +-
drivers/md/dm-crypt.c | 1 +
include/crypto/ablk_helper.h | 3 +
include/crypto/algapi.h | 23 ++++++-
include/linux/crypto.h | 5 ++
include/linux/scatterlist.h | 33 +++++++++++
lib/scatterlist.c | 59 ++++++++++++++++++
10 files changed, 379 insertions(+), 5 deletions(-)
--
1.7.9.5
^ permalink raw reply
* RE: Checkarray doesn't seem to do anything
From: rodman @ 2016-03-02 19:53 UTC (permalink / raw)
To: linux-raid
I, too, get idle on 4.2.0-30.
----
rodman@nas:/media/nas$ dd if=/sys/block/md0/md/sync_action bs=1 count=1
idle
1+0 records in
0+1 records out
5 bytes (5 B) copied, 7.9147e-05 s, 63.2 kB/s
-------
Regards,
Rodman Frowert
-------- Original Message --------
Subject: Re: Checkarray doesn't seem to do anything
From: Roman Mamedov <rm@romanrm.net>
Date: Wed, March 02, 2016 11:47 am
To: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
Cc: rodman@shellport.com, linux-raid@vger.kernel.org
On Wed, 2 Mar 2016 18:44:54 +0100
Piergiorgio Sartor <piergiorgio.sartor@nexgo.de> wrote:
> Just tested:
>
> dd if=/sys/block/md0/md/sync_action bs=1 count=1
>
> On kernel 3.x I get "i", on kernel 4.x I get "idle"... :-)
I am getting 'i' on 4.4.1.
# dd if=/sys/block/md0/md/sync_action bs=1 count=1
i1+0 records in
1+0 records out
1 byte (1 B) copied, 0.000218199 s, 4.6 kB/s
--
With respect,
Roman
^ permalink raw reply
* Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor @ 2016-03-02 18:08 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Piergiorgio Sartor, rodman, linux-raid
In-Reply-To: <20160302224745.32db9c07@natsu>
On Wed, Mar 02, 2016 at 10:47:45PM +0500, Roman Mamedov wrote:
> On Wed, 2 Mar 2016 18:44:54 +0100
> Piergiorgio Sartor <piergiorgio.sartor@nexgo.de> wrote:
>
> > Just tested:
> >
> > dd if=/sys/block/md0/md/sync_action bs=1 count=1
> >
> > On kernel 3.x I get "i", on kernel 4.x I get "idle"... :-)
>
> I am getting 'i' on 4.4.1.
>
> # dd if=/sys/block/md0/md/sync_action bs=1 count=1
> i1+0 records in
> 1+0 records out
> 1 byte (1 B) copied, 0.000218199 s, 4.6 kB/s
Interesting.
I tested kernel 4.1.x (Ubuntu) and now 4.3.6 (Fedora 22).
They both deliver "idle".
On kernel 4.4.2 (Fedora 23) I get back the "i" alone...
Really funny... :-)
bye,
>
> --
> With respect,
> Roman
--
piergiorgio
^ permalink raw reply
* RE: Checkarray doesn't seem to do anything
From: rodman @ 2016-03-02 17:56 UTC (permalink / raw)
To: Piergiorgio Sartor; +Cc: linux-raid
Yup, I wasn't root. My fault!
Rodman
-------- Original Message --------
Subject: Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
Date: Wed, March 02, 2016 11:44 am
To: rodman@shellport.com
Cc: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>,
linux-raid@vger.kernel.org
On Tue, Mar 01, 2016 at 05:50:09PM -0700, rodman@shellport.com wrote:
>
> It appears this is not an Mdadm bug but is in fact kernel/dash issue
as
> you noted, Piergiorgio.
>
> >From the bug tracker here:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787950
>
> "This is a kernel/dash issue. In kernel 4.0, the md sysfs
implementation
> does not work with partial reads from /sys/block/md0/md/sync_action;
> the read syscall returns the entire contents of the file even if you
> try to read less. Dash has an inefficient implementation which reads
> input one byte at a time, and fails when it hits the kernel
> misbehavior."
Just tested:
dd if=/sys/block/md0/md/sync_action bs=1 count=1
On kernel 3.x I get "i", on kernel 4.x I get "idle"... :-)
>
> The work around is what you said. Substitute bash for sh at the
> beginning of the script. Unfortunately that doesn't work for me as I
> get this error when trying to run checkarray now:
> "checkarray: E: /sys/block/md0/md/sync_action not writeable."
Well, first check the permissions with:
ls -l /sys/block/md0/md/sync_action
Then make sure you're root, otherwise it will
not be possible to write, of course.
bye,
pg
>
> I'll do some research to see what's that's about. I'm glad to know
its
> not a Mdadm issue per say.
>
> Rodman Frowert
>
> -------- Original Message --------
> Subject: Re: Checkarray doesn't seem to do anything
> From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
> Date: Tue, March 01, 2016 4:25 pm
> To: rodman@shellport.com
> Cc: linux-raid@vger.kernel.org
>
> On Tue, Mar 01, 2016 at 02:57:21PM -0700, rodman@shellport.com wrote:
> > Thank you for that information. My kernel is 4.2.0.
>
> I guess the problem is the same.
>
> >
> > Is this it's intended behavior or is it simply a bug that has not
been
> > corrected?
>
> Well, I think it is a bug.
> I'm not sure if "dash" bug or kernel bug.
> I suspect "dash", since "bash" works...
>
> >
> > I only bring this up as nearly all mdadm config tutorials at some
point
> > mention the checkarray function. They also point out that
checkarray is
> > also auto configured in the cron tab (cron.d) to run once a month.
But
> > of course, if checkarray doesn't work then people may think it is
> > running once a month when it really isn't.
>
> Exactly!
> I've some Ubuntu 14.04.x with old kernel, and one
> with new kernel.
> This one has a "weak" HDD in RAID (one / some blocks
> waiting for relocation).
> In my sytems the check occurs weekly, but it did not!
> Tried from command line to no avail...
> Debugging the script led to the previous result.
>
> I'm not sure, maybe someone should file a bug, but I
> have not to much faith in Ubuntu md/mdadm support.
> In the past they were slow or no reacting, and "mdadm"
> was always an obsolete version (like it is in 14.04.x).
> Maybe Debian could be a better place, not sure.
>
> bye,
>
> pg
>
> >
> >
> > Rodman Frowert -------- Original Message --------
> > Subject: Re: Checkarray doesn't seem to do anything
> > From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
> > Date: Tue, March 01, 2016 3:33 pm
> > To: rodman@shellport.com
> > Cc: linux-raid@vger.kernel.org
> >
> > On Tue, Mar 01, 2016 at 02:09:44PM -0700, rodman@shellport.com
wrote:
> > >
> > > Hello,
> > >
> > > I have a question as it pertains to the checkarray function of
Mdadm.
> > I
> > > can't actually get it to do anything. I'm running Ubuntu 14.04.4
> > Server
> > > LTS with the latest mdadm installed. I have a single RAID 1
mirror
> > > configured and working properly near as I can tell.
> > >
> > > I can force a check if I issue "echo check >
> > > /sys/block/md0/md/sync_action" as root. I can then see the status
of
> > > the check via /proc/mdstat or using the --detail switch on mdadm
> > itself.
> > >
> > > However, nothing happens if I type in:
> > > /usr/share/mdadm/checkarray -a
> > > or
> > > /usr/share/mdadm/checkarray -a /dev/mdX
> > > or
> > > /usr/share/mdadm/checkarray -as
> > >
> > > I've tried the commands as root and using sudo. The array sits
idle
> > > after all of the above commands. The only thing I can actually
get
> > > checkarray to perform is to give me the help output if I pass it
-h.
> > I
> > > have attached, below, the current status of the mdadm managed
array.
> > > Doing some Googling I haven't been able to find anyone else that
> > isn't
> > > able to use the checkarray function so I must be doing something
> > wrong.
> > >
> > > Any ideas?
> >
> > Well, yes, I've the same problem.
> >
> > Ubuntu 14.04.x with kernel 3.13, "checkarray" works.
> > Ubuntu 14.04.x with kernel 4.11 (I think), "checkarray"
> > does not work.
> >
> > The script runs with "dash" (not "bash") and it has
> > a "read cur_status < something_from_sysfs" in order to check
> > the array status.
> > With kernel 3.13, the "read" returns the proper value,
> > with kernel 4.11 (I think) the very same "read" exits
> > the script without any message or else.
> >
> > One option is to use "bash", that is change the top
> > of the script from: #!/bin/sh to #!/bin/bash
> >
> > Another is to change the "read", with a "cat", that is
> > something like: cur_status=$( cat something_from_sysfs )
> >
> > You can check yourself, there is only one "read" in
> > the script (if I recall correctly).
> >
> > Hope this helps,
> >
> > bye,
> >
> > pg
> >
> > >
> > > Rodman Frowert
> > >
> >
> > >
> > > cat /proc/mdstat shows:
> > > ------------------
> > > rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
> > > Personalities : [linear] [multipath] [raid0] [raid1] [raid6]
[raid5]
> > > [raid4] [raid10]
> > > md0 : active raid1 sdc1[1] sdb1[0]
> > > 5860389696 blocks super 1.2 [2/2] [UU]
> > > -----------------
> > >
> > > sudo mdadm --detail /dev/md0 shows:
> > > ------------------
> > > rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
> > > /dev/md0:
> > > Version : 1.2
> > > Creation Time : Fri Feb 26 14:07:40 2016
> > > Raid Level : raid1
> > > Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > > Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > > Raid Devices : 2
> > > Total Devices : 2
> > > Persistence : Superblock is persistent
> > >
> > > Update Time : Tue Mar 1 07:40:19 2016
> > > State : clean
> > > Active Devices : 2
> > > Working Devices : 2
> > > Failed Devices : 0
> > > Spare Devices : 0
> > >
> > > Name : nas:0 (local to host nas)
> > > UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
> > > Events : 164
> > >
> > > Number Major Minor RaidDevice State
> > > 0 8 17 0 active sync /dev/sdb1
> > > 1 8 33 1 active sync /dev/sdc1
> > >
> > > -----------------
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
linux-raid"
> > in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> >
> > piergiorgio
> > --
> > To unsubscribe from this list: send the line "unsubscribe
linux-raid"
> > in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe
linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
>
> piergiorgio
--
piergiorgio
--
To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Checkarray doesn't seem to do anything
From: Roman Mamedov @ 2016-03-02 17:47 UTC (permalink / raw)
To: Piergiorgio Sartor; +Cc: rodman, linux-raid
In-Reply-To: <20160302174454.GA2567@lazy.lzy>
[-- Attachment #1: Type: text/plain, Size: 425 bytes --]
On Wed, 2 Mar 2016 18:44:54 +0100
Piergiorgio Sartor <piergiorgio.sartor@nexgo.de> wrote:
> Just tested:
>
> dd if=/sys/block/md0/md/sync_action bs=1 count=1
>
> On kernel 3.x I get "i", on kernel 4.x I get "idle"... :-)
I am getting 'i' on 4.4.1.
# dd if=/sys/block/md0/md/sync_action bs=1 count=1
i1+0 records in
1+0 records out
1 byte (1 B) copied, 0.000218199 s, 4.6 kB/s
--
With respect,
Roman
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor @ 2016-03-02 17:44 UTC (permalink / raw)
To: rodman; +Cc: Piergiorgio Sartor, linux-raid
In-Reply-To: <20160301175009.0a91db92e0d9ae1e5e3fde0508678719.a1548d3694.wbe@email03.secureserver.net>
On Tue, Mar 01, 2016 at 05:50:09PM -0700, rodman@shellport.com wrote:
>
> It appears this is not an Mdadm bug but is in fact kernel/dash issue as
> you noted, Piergiorgio.
>
> >From the bug tracker here:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787950
>
> "This is a kernel/dash issue. In kernel 4.0, the md sysfs implementation
> does not work with partial reads from /sys/block/md0/md/sync_action;
> the read syscall returns the entire contents of the file even if you
> try to read less. Dash has an inefficient implementation which reads
> input one byte at a time, and fails when it hits the kernel
> misbehavior."
Just tested:
dd if=/sys/block/md0/md/sync_action bs=1 count=1
On kernel 3.x I get "i", on kernel 4.x I get "idle"... :-)
>
> The work around is what you said. Substitute bash for sh at the
> beginning of the script. Unfortunately that doesn't work for me as I
> get this error when trying to run checkarray now:
> "checkarray: E: /sys/block/md0/md/sync_action not writeable."
Well, first check the permissions with:
ls -l /sys/block/md0/md/sync_action
Then make sure you're root, otherwise it will
not be possible to write, of course.
bye,
pg
>
> I'll do some research to see what's that's about. I'm glad to know its
> not a Mdadm issue per say.
>
> Rodman Frowert
>
> -------- Original Message --------
> Subject: Re: Checkarray doesn't seem to do anything
> From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
> Date: Tue, March 01, 2016 4:25 pm
> To: rodman@shellport.com
> Cc: linux-raid@vger.kernel.org
>
> On Tue, Mar 01, 2016 at 02:57:21PM -0700, rodman@shellport.com wrote:
> > Thank you for that information. My kernel is 4.2.0.
>
> I guess the problem is the same.
>
> >
> > Is this it's intended behavior or is it simply a bug that has not been
> > corrected?
>
> Well, I think it is a bug.
> I'm not sure if "dash" bug or kernel bug.
> I suspect "dash", since "bash" works...
>
> >
> > I only bring this up as nearly all mdadm config tutorials at some point
> > mention the checkarray function. They also point out that checkarray is
> > also auto configured in the cron tab (cron.d) to run once a month. But
> > of course, if checkarray doesn't work then people may think it is
> > running once a month when it really isn't.
>
> Exactly!
> I've some Ubuntu 14.04.x with old kernel, and one
> with new kernel.
> This one has a "weak" HDD in RAID (one / some blocks
> waiting for relocation).
> In my sytems the check occurs weekly, but it did not!
> Tried from command line to no avail...
> Debugging the script led to the previous result.
>
> I'm not sure, maybe someone should file a bug, but I
> have not to much faith in Ubuntu md/mdadm support.
> In the past they were slow or no reacting, and "mdadm"
> was always an obsolete version (like it is in 14.04.x).
> Maybe Debian could be a better place, not sure.
>
> bye,
>
> pg
>
> >
> >
> > Rodman Frowert -------- Original Message --------
> > Subject: Re: Checkarray doesn't seem to do anything
> > From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
> > Date: Tue, March 01, 2016 3:33 pm
> > To: rodman@shellport.com
> > Cc: linux-raid@vger.kernel.org
> >
> > On Tue, Mar 01, 2016 at 02:09:44PM -0700, rodman@shellport.com wrote:
> > >
> > > Hello,
> > >
> > > I have a question as it pertains to the checkarray function of Mdadm.
> > I
> > > can't actually get it to do anything. I'm running Ubuntu 14.04.4
> > Server
> > > LTS with the latest mdadm installed. I have a single RAID 1 mirror
> > > configured and working properly near as I can tell.
> > >
> > > I can force a check if I issue "echo check >
> > > /sys/block/md0/md/sync_action" as root. I can then see the status of
> > > the check via /proc/mdstat or using the --detail switch on mdadm
> > itself.
> > >
> > > However, nothing happens if I type in:
> > > /usr/share/mdadm/checkarray -a
> > > or
> > > /usr/share/mdadm/checkarray -a /dev/mdX
> > > or
> > > /usr/share/mdadm/checkarray -as
> > >
> > > I've tried the commands as root and using sudo. The array sits idle
> > > after all of the above commands. The only thing I can actually get
> > > checkarray to perform is to give me the help output if I pass it -h.
> > I
> > > have attached, below, the current status of the mdadm managed array.
> > > Doing some Googling I haven't been able to find anyone else that
> > isn't
> > > able to use the checkarray function so I must be doing something
> > wrong.
> > >
> > > Any ideas?
> >
> > Well, yes, I've the same problem.
> >
> > Ubuntu 14.04.x with kernel 3.13, "checkarray" works.
> > Ubuntu 14.04.x with kernel 4.11 (I think), "checkarray"
> > does not work.
> >
> > The script runs with "dash" (not "bash") and it has
> > a "read cur_status < something_from_sysfs" in order to check
> > the array status.
> > With kernel 3.13, the "read" returns the proper value,
> > with kernel 4.11 (I think) the very same "read" exits
> > the script without any message or else.
> >
> > One option is to use "bash", that is change the top
> > of the script from: #!/bin/sh to #!/bin/bash
> >
> > Another is to change the "read", with a "cat", that is
> > something like: cur_status=$( cat something_from_sysfs )
> >
> > You can check yourself, there is only one "read" in
> > the script (if I recall correctly).
> >
> > Hope this helps,
> >
> > bye,
> >
> > pg
> >
> > >
> > > Rodman Frowert
> > >
> >
> > >
> > > cat /proc/mdstat shows:
> > > ------------------
> > > rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
> > > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> > > [raid4] [raid10]
> > > md0 : active raid1 sdc1[1] sdb1[0]
> > > 5860389696 blocks super 1.2 [2/2] [UU]
> > > -----------------
> > >
> > > sudo mdadm --detail /dev/md0 shows:
> > > ------------------
> > > rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
> > > /dev/md0:
> > > Version : 1.2
> > > Creation Time : Fri Feb 26 14:07:40 2016
> > > Raid Level : raid1
> > > Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > > Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > > Raid Devices : 2
> > > Total Devices : 2
> > > Persistence : Superblock is persistent
> > >
> > > Update Time : Tue Mar 1 07:40:19 2016
> > > State : clean
> > > Active Devices : 2
> > > Working Devices : 2
> > > Failed Devices : 0
> > > Spare Devices : 0
> > >
> > > Name : nas:0 (local to host nas)
> > > UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
> > > Events : 164
> > >
> > > Number Major Minor RaidDevice State
> > > 0 8 17 0 active sync /dev/sdb1
> > > 1 8 33 1 active sync /dev/sdc1
> > >
> > > -----------------
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> > in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > --
> >
> > piergiorgio
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> > in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
>
> piergiorgio
--
piergiorgio
^ permalink raw reply
* Re: RAID6 Array crash during reshape.....now will not re-assemble.
From: Another Sillyname @ 2016-03-02 15:59 UTC (permalink / raw)
To: Linux-RAID
In-Reply-To: <CAOS+5GEfQ2p0xguguKC3uXW8GYBDCxt2vk=-Ci6+nTKwkcWtdQ@mail.gmail.com>
I've found out more info....and now have a theory.......but do not
know how best to proceed.
>sudo mdadm -A --scan --verbose
mdadm: looking for devices for further assembly
mdadm: No super block found on /dev/sdh (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdh
mdadm: No super block found on /dev/sdg (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdg
mdadm: No super block found on /dev/sdf (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdf
mdadm: No super block found on /dev/sde (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sde
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda6 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda6
mdadm: No super block found on /dev/sda5 (Expected magic a92b4efc, got 75412023)
mdadm: no RAID superblock on /dev/sda5
mdadm: /dev/sda4 is too small for md: size is 2 sectors.
mdadm: no RAID superblock on /dev/sda4
mdadm: No super block found on /dev/sda3 (Expected magic a92b4efc, got 00000401)
mdadm: no RAID superblock on /dev/sda3
mdadm: No super block found on /dev/sda2 (Expected magic a92b4efc, got 00000401)
mdadm: no RAID superblock on /dev/sda2
mdadm: No super block found on /dev/sda1 (Expected magic a92b4efc, got 0000007e)
mdadm: no RAID superblock on /dev/sda1
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got e71e974a)
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sdh1 is identified as a member of /dev/md/server187:1, slot 5.
mdadm: /dev/sdg1 is identified as a member of /dev/md/server187:1, slot 0.
mdadm: /dev/sdf1 is identified as a member of /dev/md/server187:1, slot 2.
mdadm: /dev/sde1 is identified as a member of /dev/md/server187:1, slot 3.
mdadm: /dev/sdd1 is identified as a member of /dev/md/server187:1, slot 6.
mdadm: /dev/sdc1 is identified as a member of /dev/md/server187:1, slot 1.
mdadm: /dev/sdb1 is identified as a member of /dev/md/server187:1, slot 4.
mdadm: /dev/md/server187:1 has an active reshape - checking if
critical section needs to be restored
mdadm: Failed to find backup of critical section
mdadm: Failed to restore critical section for reshape, sorry.
Possibly you needed to specify the --backup-file
mdadm: looking for devices for further assembly
mdadm: No arrays found in config file or automatically
As I stated in my original posting I do not know where the server187
stuff came from when I tried the original assemble and two of the
drives (sdg & sdh) reported as busy.
So my theory is this......
This 30TB array has been up and active since about August 2015, fully
functional without any major issues, except performance was sometimes
a bit iffy.
It is possible that drives sdg and sdh were used in a temporary box in
a different array that was only active for about 10 days, before they
were moved to the new 30TB array that was cleanly built. That array
may well have been called server187 (it was a temp box so no reason to
remember it).
When the reshape of the current array 'died' during initialisation or
immediately thereafter, even though cat /proc/mdstat showed the
reshape active after 12 hours it was still stuck on 0.0%.
When the machine was rebooted and the array didn't come up...is it
possible that drives sdh and sdg still thought they were in the old
server187 array and that is why they reported themselves busy? I'm
not sure why this would happen, but am just theorising.
When I tried the assemble command where it reported it was merging
with the already existing server187 array, even though there
wasn't/isn't a server187 array as prior to that assemble cat
/proc/mdstat reported the offline md127 array.
Somehow therefore the array names have got confused/transposed and
that's why the backup file is now not seen as the correct one? This
would seem to be borne out by all the drives now seeing themselves as
part of server187 array rather then md127 array and also the reshape
seems to be attached to this server187 array.
I still believe/hope the data is all still intact and complete,
however I am averse to just hacking around using google to 'try
commands' hoping I hit a solution before someone with much more
experience casts an eye over this to give me a little guidance.
Help!!
On 2 March 2016 at 13:42, Another Sillyname <anothersname@googlemail.com> wrote:
> Kernel is latest Fedora x86_64 4.3.5-300, can't get too much newer
> then that (latest is 4.4.x), mdadm is 3.3.4-2.
>
> I agree that the data is likely still intact, doesn't stop me being
> nervous till I see it though!!
>
>
>
> On 2 March 2016 at 13:20, Wols Lists <antlists@youngman.org.uk> wrote:
>> On 02/03/16 03:46, Another Sillyname wrote:
>>> Any help and guidance would be appreciated, the drives showing clean
>>> gives me comfort that the data is likely intact and complete (crossed
>>> fingers) however I can't re-assemble the array as I keep getting the
>>> 'critical information for reshape, sorry' warning.
>>>
>>> Help???
>>
>> Someone else will chip in what to do, but this doesn't seem alarming at
>> all. Reshapes stuck at zero is a recent bug, but all the data is
>> probably safe and sound.
>>
>> Wait for one of the experts to chip in what to do, but you might find
>> mdadm --resume --invalid-backup will get it going again.
>>
>> Otherwise it's likely to be an "upgrade your kernel and mdadm" job ...
>>
>> Cheers,
>> Wol
^ permalink raw reply
* Fwd: RAID6 Array crash during reshape.....now will not re-assemble.
From: Another Sillyname @ 2016-03-02 13:42 UTC (permalink / raw)
To: Linux-RAID
In-Reply-To: <CAOS+5GHof=F94x58SKqFojV26hGpDSLF85dFfm8Xc6M43sN6jA@mail.gmail.com>
Kernel is latest Fedora x86_64 4.3.5-300, can't get too much newer
then that (latest is 4.4.x), mdadm is 3.3.4-2.
I agree that the data is likely still intact, doesn't stop me being
nervous till I see it though!!
On 2 March 2016 at 13:20, Wols Lists <antlists@youngman.org.uk> wrote:
> On 02/03/16 03:46, Another Sillyname wrote:
>> Any help and guidance would be appreciated, the drives showing clean
>> gives me comfort that the data is likely intact and complete (crossed
>> fingers) however I can't re-assemble the array as I keep getting the
>> 'critical information for reshape, sorry' warning.
>>
>> Help???
>
> Someone else will chip in what to do, but this doesn't seem alarming at
> all. Reshapes stuck at zero is a recent bug, but all the data is
> probably safe and sound.
>
> Wait for one of the experts to chip in what to do, but you might find
> mdadm --resume --invalid-backup will get it going again.
>
> Otherwise it's likely to be an "upgrade your kernel and mdadm" job ...
>
> Cheers,
> Wol
^ permalink raw reply
* Re: RAID6 Array crash during reshape.....now will not re-assemble.
From: Wols Lists @ 2016-03-02 13:20 UTC (permalink / raw)
To: Another Sillyname, Linux-RAID
In-Reply-To: <CAOS+5GEd4E2wzYhw0rPdrZnjK6UwAUdKCQLLTxXrdPO8d7hS7w@mail.gmail.com>
On 02/03/16 03:46, Another Sillyname wrote:
> Any help and guidance would be appreciated, the drives showing clean
> gives me comfort that the data is likely intact and complete (crossed
> fingers) however I can't re-assemble the array as I keep getting the
> 'critical information for reshape, sorry' warning.
>
> Help???
Someone else will chip in what to do, but this doesn't seem alarming at
all. Reshapes stuck at zero is a recent bug, but all the data is
probably safe and sound.
Wait for one of the experts to chip in what to do, but you might find
mdadm --resume --invalid-backup will get it going again.
Otherwise it's likely to be an "upgrade your kernel and mdadm" job ...
Cheers,
Wol
^ permalink raw reply
* RAID6 Array crash during reshape.....now will not re-assemble.
From: Another Sillyname @ 2016-03-02 3:46 UTC (permalink / raw)
To: Linux-RAID
I have a 30TB RAID6 array using 7 x 6TB drives that I wanted to
migrate to RAID5 to take one of the drives offline and use in a new
array for a migration.
sudo mdadm --grow /dev/md127 --level=raid5 --raid-device=6
--backup-file=mdadm_backupfile
I watched this using cat /proc/mdstat and even after an hour the
percentage of the reshape was still 0.0%.
I know from previous experience that reshaping can be slow, but did
not expect it to be this slow frankly. But erring on the side of
caution I decided to leave the array for 12 hours and see what was
happening then.
Sure enough, 12 hours later cat /proc/mdstat still shows reshape at 0.0%
Looking at CPU usage the reshape process is using 0% of the CPU.
So reading a bit more......if you reboot a server the reshape should continue.
Reboot.....
Array will not come back online at all.
Bring the server up without the array trying to automount.
cat /proc/mdstat shows the array offline.
Personalities :
md127 : inactive sdf1[2](S) sde1[3](S) sdg1[0](S) sdb1[8](S)
sdh1[7](S) sdc1[1](S) sdd1[6](S)
41022733300 blocks super 1.2
unused devices: <none>
Try to reassemble the array.
>sudo mdadm --assemble /dev/md127 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
mdadm: /dev/sdg1 is busy - skipping
mdadm: /dev/sdh1 is busy - skipping
mdadm: Merging with already-assembled /dev/md/server187.internallan.com:1
mdadm: Failed to restore critical section for reshape, sorry.
Possibly you needed to specify the --backup-file
Have no idea where the server187 stuff has come from.
stop the array.
>sudo mdadm --stop /dev/md127
try to re-assemble
>sudo mdadm --assemble /dev/md127 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1
mdadm: Failed to restore critical section for reshape, sorry.
Possibly you needed to specify the --backup-file
try to re-assemble using the backup file
>sudo mdadm --assemble /dev/md127 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 --backup-file=mdadm_backupfile
mdadm: Failed to restore critical section for reshape, sorry.
have a look at the individual drives
>sudo mdadm --examine /dev/sd[b-h]1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : server187.internallan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 1152bdeb:15546156:1918b67d:37d68b1f
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
New Layout : left-symmetric-6
Update Time : Wed Mar 2 01:19:42 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 3a66db58 - correct
Events : 369282
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : server187.internallan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 140e09af:56e14b4e:5035d724:c2005f0b
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
New Layout : left-symmetric-6
Update Time : Wed Mar 2 01:19:42 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 88916c56 - correct
Events : 369282
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : server187.internallan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : a50dd0a1:eeb0b3df:76200476:818e004d
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
New Layout : left-symmetric-6
Update Time : Wed Mar 2 01:19:42 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9f8eb46a - correct
Events : 369282
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : server187.internallan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 7d0b65b3:d2ba2023:4625c287:1db2de9b
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
New Layout : left-symmetric-6
Update Time : Wed Mar 2 01:19:42 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 552ce48f - correct
Events : 369282
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : server187.internallan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : cda4f5e5:a489dbb9:5c1ab6a0:b257c984
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
New Layout : left-symmetric-6
Update Time : Wed Mar 2 01:19:42 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2056e75c - correct
Events : 369282
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : server187.internallan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : df5af6ce:9017c863:697da267:046c9709
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
New Layout : left-symmetric-6
Update Time : Wed Mar 2 01:19:42 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : fefea2b5 - correct
Events : 369282
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : da29a06f:f8cf1409:bc52afb2:6945ba08
Name : server187.internallan.com:1
Creation Time : Sun May 10 14:47:51 2015
Raid Level : raid6
Raid Devices : 7
Avail Dev Size : 11720780943 (5588.90 GiB 6001.04 GB)
Array Size : 29301952000 (27944.52 GiB 30005.20 GB)
Used Dev Size : 11720780800 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=143 sectors
State : clean
Device UUID : 9d98af83:243c3e02:94de20c7:293de111
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
New Layout : left-symmetric-6
Update Time : Wed Mar 2 01:19:42 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : b9f6375e - correct
Events : 369282
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
As all the drives are showing Reshape pos'n 0 I'm assuming the reshape
never got started (even though cat /proc/mdstat showed the array
reshaping)?
So now I'm well out of my comfort zone so instead of flapping around
have decided to sleep for a few hours before revisiting this.
Any help and guidance would be appreciated, the drives showing clean
gives me comfort that the data is likely intact and complete (crossed
fingers) however I can't re-assemble the array as I keep getting the
'critical information for reshape, sorry' warning.
Help???
^ permalink raw reply
* RE: Checkarray doesn't seem to do anything
From: rodman @ 2016-03-02 0:50 UTC (permalink / raw)
To: Piergiorgio Sartor; +Cc: linux-raid
It appears this is not an Mdadm bug but is in fact kernel/dash issue as
you noted, Piergiorgio.
From the bug tracker here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787950
"This is a kernel/dash issue. In kernel 4.0, the md sysfs implementation
does not work with partial reads from /sys/block/md0/md/sync_action;
the read syscall returns the entire contents of the file even if you
try to read less. Dash has an inefficient implementation which reads
input one byte at a time, and fails when it hits the kernel
misbehavior."
The work around is what you said. Substitute bash for sh at the
beginning of the script. Unfortunately that doesn't work for me as I
get this error when trying to run checkarray now:
"checkarray: E: /sys/block/md0/md/sync_action not writeable."
I'll do some research to see what's that's about. I'm glad to know its
not a Mdadm issue per say.
Rodman Frowert
-------- Original Message --------
Subject: Re: Checkarray doesn't seem to do anything
From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
Date: Tue, March 01, 2016 4:25 pm
To: rodman@shellport.com
Cc: linux-raid@vger.kernel.org
On Tue, Mar 01, 2016 at 02:57:21PM -0700, rodman@shellport.com wrote:
> Thank you for that information. My kernel is 4.2.0.
I guess the problem is the same.
>
> Is this it's intended behavior or is it simply a bug that has not been
> corrected?
Well, I think it is a bug.
I'm not sure if "dash" bug or kernel bug.
I suspect "dash", since "bash" works...
>
> I only bring this up as nearly all mdadm config tutorials at some point
> mention the checkarray function. They also point out that checkarray is
> also auto configured in the cron tab (cron.d) to run once a month. But
> of course, if checkarray doesn't work then people may think it is
> running once a month when it really isn't.
Exactly!
I've some Ubuntu 14.04.x with old kernel, and one
with new kernel.
This one has a "weak" HDD in RAID (one / some blocks
waiting for relocation).
In my sytems the check occurs weekly, but it did not!
Tried from command line to no avail...
Debugging the script led to the previous result.
I'm not sure, maybe someone should file a bug, but I
have not to much faith in Ubuntu md/mdadm support.
In the past they were slow or no reacting, and "mdadm"
was always an obsolete version (like it is in 14.04.x).
Maybe Debian could be a better place, not sure.
bye,
pg
>
>
> Rodman Frowert -------- Original Message --------
> Subject: Re: Checkarray doesn't seem to do anything
> From: Piergiorgio Sartor <piergiorgio.sartor@nexgo.de>
> Date: Tue, March 01, 2016 3:33 pm
> To: rodman@shellport.com
> Cc: linux-raid@vger.kernel.org
>
> On Tue, Mar 01, 2016 at 02:09:44PM -0700, rodman@shellport.com wrote:
> >
> > Hello,
> >
> > I have a question as it pertains to the checkarray function of Mdadm.
> I
> > can't actually get it to do anything. I'm running Ubuntu 14.04.4
> Server
> > LTS with the latest mdadm installed. I have a single RAID 1 mirror
> > configured and working properly near as I can tell.
> >
> > I can force a check if I issue "echo check >
> > /sys/block/md0/md/sync_action" as root. I can then see the status of
> > the check via /proc/mdstat or using the --detail switch on mdadm
> itself.
> >
> > However, nothing happens if I type in:
> > /usr/share/mdadm/checkarray -a
> > or
> > /usr/share/mdadm/checkarray -a /dev/mdX
> > or
> > /usr/share/mdadm/checkarray -as
> >
> > I've tried the commands as root and using sudo. The array sits idle
> > after all of the above commands. The only thing I can actually get
> > checkarray to perform is to give me the help output if I pass it -h.
> I
> > have attached, below, the current status of the mdadm managed array.
> > Doing some Googling I haven't been able to find anyone else that
> isn't
> > able to use the checkarray function so I must be doing something
> wrong.
> >
> > Any ideas?
>
> Well, yes, I've the same problem.
>
> Ubuntu 14.04.x with kernel 3.13, "checkarray" works.
> Ubuntu 14.04.x with kernel 4.11 (I think), "checkarray"
> does not work.
>
> The script runs with "dash" (not "bash") and it has
> a "read cur_status < something_from_sysfs" in order to check
> the array status.
> With kernel 3.13, the "read" returns the proper value,
> with kernel 4.11 (I think) the very same "read" exits
> the script without any message or else.
>
> One option is to use "bash", that is change the top
> of the script from: #!/bin/sh to #!/bin/bash
>
> Another is to change the "read", with a "cat", that is
> something like: cur_status=$( cat something_from_sysfs )
>
> You can check yourself, there is only one "read" in
> the script (if I recall correctly).
>
> Hope this helps,
>
> bye,
>
> pg
>
> >
> > Rodman Frowert
> >
>
> >
> > cat /proc/mdstat shows:
> > ------------------
> > rodman@nas:/usr/share/mdadm$ cat /proc/mdstat
> > Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> > [raid4] [raid10]
> > md0 : active raid1 sdc1[1] sdb1[0]
> > 5860389696 blocks super 1.2 [2/2] [UU]
> > -----------------
> >
> > sudo mdadm --detail /dev/md0 shows:
> > ------------------
> > rodman@nas:/usr/share/mdadm$ sudo mdadm --detail /dev/md0
> > /dev/md0:
> > Version : 1.2
> > Creation Time : Fri Feb 26 14:07:40 2016
> > Raid Level : raid1
> > Array Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > Used Dev Size : 5860389696 (5588.90 GiB 6001.04 GB)
> > Raid Devices : 2
> > Total Devices : 2
> > Persistence : Superblock is persistent
> >
> > Update Time : Tue Mar 1 07:40:19 2016
> > State : clean
> > Active Devices : 2
> > Working Devices : 2
> > Failed Devices : 0
> > Spare Devices : 0
> >
> > Name : nas:0 (local to host nas)
> > UUID : 4765e7df:0e1a6ad2:1321765f:a2d9f2c9
> > Events : 164
> >
> > Number Major Minor RaidDevice State
> > 0 8 17 0 active sync /dev/sdb1
> > 1 8 33 1 active sync /dev/sdc1
> >
> > -----------------
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> --
>
> piergiorgio
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
piergiorgio
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox