* Re: Raid 5 rebuild and Raid 6 reshape question
From: Mikael Abrahamsson @ 2015-04-18 13:13 UTC (permalink / raw)
To: Thomas Fjellstrom; +Cc: linux-raid
In-Reply-To: <6744821.j7csV3RJVE@balsa>
On Fri, 17 Apr 2015, Thomas Fjellstrom wrote:
> I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
> seemed to work), and i decided to replace it. I actually bought two drives,
If you have a not-ancient kernel, you should have issued the replace
command. You should have added one new drive as spare, then (with a new
mdadm) issued the --replace command. This would have started copying the
drive you wanted replaced onto the spare. Then after all this was done,
you could have added another spare and told it to re-shape to raid6.
That's at least what I would have done.
http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Raid 5 rebuild and Raid 6 reshape question
From: Thomas Fjellstrom @ 2015-04-17 17:55 UTC (permalink / raw)
To: linux-raid
This is a potentially dumb question...
I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
seemed to work), and i decided to replace it. I actually bought two drives,
thinking I'd keep one as a cold spare, but when i got to replacing the drive
(after days of load testing to see if I'd gotten a dud), I figured maybe I'd
use the spare as a hot spare, or make a Raid6 array. In the end I decided on a
Raid6, as it is the recommended way to go these days for large disks.
When I did the add for the first disk, it started rebuilding the raid5, and
said it'd only take like 5 hours, then I added the second disk and it
skyrocketed to about 116 hours (7000s or so). I presume this is expected, due
to having to re-stripe all of the data, and re-calculate the parity at the
same time.
How much faster would this have finished if I had just waited for the first
drive to finish rebuilding and done the reshape as a separate step? I decided
to just let this reshape continue, its about 30% the way through, and this is
actually the backup array for my main NAS, so it isn't actually a problem if
it dies mid-reshape. I'm just curious about the alternatives to how I did it.
I know i could have just copied the old disk to one of the new ones, and
skipped the entire rebuild on one disk, but i didn't feel like it ;D
--
Thomas Fjellstrom
thomas@fjellstrom.ca
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Alex Elsayed @ 2015-04-17 15:53 UTC (permalink / raw)
To: linux-kernel; +Cc: dm-devel, linux-raid, linux-pm
In-Reply-To: <20150417075211.GC403@redhat.com>
Mike Snitzer wrote:
> On Thu, Apr 16 2015 at 5:23am -0400,
> Alex Elsayed <eternaleye@gmail.com> wrote:
>
>> Mike Snitzer wrote:
>>
>> > On Thu, Apr 09 2015 at 9:28am -0400,
>> > Pali Rohár <pali.rohar@gmail.com> wrote:
>> >
>> >> On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
>> >> > On Mon, Apr 06 2015 at 9:29am -0400,
>> >> > Pali Rohár <pali.rohar@gmail.com> wrote:
>> >> >
>> >> > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
>> >> > > > On Sun, Apr 05 2015 at 1:20pm -0400,
>> >> > > >
>> >> > > > Pali Rohár <pali.rohar@gmail.com> wrote:
>> >> > > > > This patch series increase security of suspend and hibernate
>> >> > > > > actions. It allows user to safely wipe crypto keys before
>> >> > > > > suspend and hibernate actions starts without race
>> >> > > > > conditions on userspace process with heavy I/O.
>> >> > > > >
>> >> > > > > To automatically wipe cryto key for <device> before
>> >> > > > > hibernate action call: $ dmsetup message <device> 0 key
>> >> > > > > wipe_on_hibernation 1
>> >> > > > >
>> >> > > > > To automatically wipe cryto key for <device> before suspend
>> >> > > > > action call: $ dmsetup message <device> 0 key
>> >> > > > > wipe_on_suspend 1
>> >> > > > >
>> >> > > > > (Value 0 after wipe_* string reverts original behaviour - to
>> >> > > > > not wipe key)
>> >> > > >
>> >> > > > Can you elaborate on the attack vector your changes are meant
>> >> > > > to protect against? The user already authorized access, why
>> >> > > > is it inherently dangerous to _not_ wipe the associated key
>> >> > > > across these events?
>> >> > >
>> >> > > Hi,
>> >> > >
>> >> > > yes, I will try to explain current problems with cryptsetup
>> >> > > luksSuspend command and hibernation.
>> >> > >
>> >> > > First, sometimes it is needed to put machine into other hands.
>> >> > > You can still watch other person what is doing with machine, but
>> >> > > once if you let machine unlocked (e.g opened luks disk), she/he
>> >> > > can access encrypted data.
>> >> > >
>> >> > > If you turn off machine, it could be safe, because luks disk
>> >> > > devices are locked. But if you enter machine into suspend or
>> >> > > hibernate state luks devices are still open. And my patches try
>> >> > > to achieve similar security as when machine is off (= no crypto
>> >> > > keys in RAM or on swap).
>> >> > >
>> >> > > When doing hibernate on unencrypted swap it is to prevent leaking
>> >> > > crypto keys to hibernate image (which is stored in swap).
>> >> > >
>> >> > > When doing suspend action it is again to prevent leaking crypto
>> >> > > keys. E.g when you suspend laptop and put it off (somebody can
>> >> > > remove RAMs and do some cold boot attack).
>> >> > >
>> >> > > The most common situation is:
>> >> > > You have mounted partition from dm-crypt device (e.g. /home/),
>> >> > > some userspace processes access it (e.g opened firefox which
>> >> > > still reads/writes to cache ~/.firefox/) and you want to drop
>> >> > > crypto keys from kernel for some time.
>> >> > >
>> >> > > For that operation there is command cryptsetup luksSuspend, which
>> >> > > suspend dm device and then tell kernel to wipe crypto keys. All
>> >> > > I/O operations are then stopped and userspace processes which
>> >> > > want to do some those I/O operations are stopped too (until you
>> >> > > call cryptsetup luksResume and enter correct key).
>> >> > >
>> >> > > Now if you want to suspend/hiberate your machine (when some of dm
>> >> > > devices are suspeneded and some processes are stopped due to
>> >> > > pending I/O) it is not possible. Kernel freeze_processes function
>> >> > > will fail because userspace processes are still stopped inside
>> >> > > some I/O syscall (read/write, etc,...).
>> >> > >
>> >> > > My patches fixes this problem and do those operations (suspend dm
>> >> > > device, wipe crypto keys, enter suspend/hiberate) in correct
>> >> > > order and without race condition.
>> >> > >
>> >> > > dm device is suspended *after* userspace processes are freezed
>> >> > > and after that are crypto keys wiped. And then computer/laptop
>> >> > > enters into suspend/hibernate state.
>> >> >
>> >> > Wouldn't it be better to fix freeze_processes() to be tolerant of
>> >> > processes that are hung as a side-effect of their backing storage
>> >> > being
>> >> > suspended? A hibernate shouldn't fail simply because a user chose
>> >> > to suspend a DM device.
>> >> >
>> >> > Then this entire problem goes away and the key can be wiped from
>> >> > userspace (like you said above).
>> >>
>> >> Still there will be race condition. Before hibernation (and device
>> >> poweroff) we should have synced disks and filesystems to prevent data
>> >> lose (or other damage) as more as we can. And if there will be some
>> >> application which using lot of I/O (e.g normal firefox) then there
>> >> always will be race condtion.
>> >
>> > The DM suspend will take care of flushing any pending I/O. So I don't
>> > see where the supposed race is...
>> >
>> > Anything else that is trapped in userspace memory will be there when
>> > the machine resumes.
>> >
>> >> So proper way is to wipe luks crypto keys *after* userspace processes
>> >> are freezed.
>> >
>> > I know you believe that I'm just not accepting that at face value.
>>
>> Um, pardon me if I'm being naive, but what about the case of hibernation
>> where the swapdev and the root device are both LVs on the same dm_crypt
>> device?
>>
>> The kernel is writing to swap _after_ userspace processes are all frozen;
>> that seems to me like an ordering dependency entirely incompatible with
>> userspace dropping the key...
>
> Good point, definitely not compatible with the Pali's approach.
>
> (but is swap really configured ontop of the same dm-crypt device like
> this in practice? I've not heard of that being a common pattern but I
> could just be sheltered)
Every laptop I've owned in the past five years has been set up as follows:
- GPT partition table (mostly for the redundant table at the end in case of
fuckups)
- 1GB ESP as /boot (first with grub2, then gummiboot) - It's there _anyway_
- 32MB BIOS Boot Partition (for a traditional BIOS bootloader, so I can pop
the drive in a non-efi machine if the laptop dies - this has happened)
- The rest of the drive is a single dm-crypt volume, with LVM on top. What
goes on top of LVM has varied, but these days it's just swap and btrfs.
The main reason for this is that I find dealing with crypttab / multiple
LUKS devices on boot (or resume from hibernate) to be an incredible hassle;
and it's vastly simpler to just have a single dm-crypt device and let
Dracut unlock it from a single boot prompt.
I haven't set up custom-key secure boot yet, so the evil maid attack is
still on the table, but I do this more out of "Eh, why not" (and originally,
"I should at least know _how_ to set it up") than actually having stuff I
need the security for anyway.
^ permalink raw reply
* [PATCH 3/3] raid0: conditional mddev->queue access to suit dm-raid
From: heinzm @ 2015-04-17 12:40 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch is a prerequisite for dm-raid "raid0" support to allow
dm-raid to access the MD RAID0 personality doing unconditional
accesses to mddev->queue, which is NULL in case of dm-raid stacked
on top of MD.
This patch is here for completeness; it has been send to Neil Brown
and linux-raid, Neil said he applied it on Feb 23rd but it did not
show up in upstream as yet.
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/raid0.c | 48 +++++++++++++++++++++++++++---------------------
1 file changed, 27 insertions(+), 21 deletions(-)
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index 3ed9f42..e074813 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -271,14 +271,16 @@ static int create_strip_zones(struct mddev *mddev, struct r0conf **private_conf)
goto abort;
}
- blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
- blk_queue_io_opt(mddev->queue,
- (mddev->chunk_sectors << 9) * mddev->raid_disks);
-
- if (!discard_supported)
- queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
- else
- queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
+ if (mddev->queue) {
+ blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
+ blk_queue_io_opt(mddev->queue,
+ (mddev->chunk_sectors << 9) * mddev->raid_disks);
+
+ if (!discard_supported)
+ queue_flag_clear_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
+ else
+ queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, mddev->queue);
+ }
pr_debug("md/raid0:%s: done.\n", mdname(mddev));
*private_conf = conf;
@@ -429,9 +431,12 @@ static int raid0_run(struct mddev *mddev)
}
if (md_check_no_bitmap(mddev))
return -EINVAL;
- blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
- blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
- blk_queue_max_discard_sectors(mddev->queue, mddev->chunk_sectors);
+
+ if (mddev->queue) {
+ blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
+ blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
+ blk_queue_max_discard_sectors(mddev->queue, mddev->chunk_sectors);
+ }
/* if private is not null, we are here after takeover */
if (mddev->private == NULL) {
@@ -448,16 +453,17 @@ static int raid0_run(struct mddev *mddev)
printk(KERN_INFO "md/raid0:%s: md_size is %llu sectors.\n",
mdname(mddev),
(unsigned long long)mddev->array_sectors);
- /* calculate the max read-ahead size.
- * For read-ahead of large files to be effective, we need to
- * readahead at least twice a whole stripe. i.e. number of devices
- * multiplied by chunk size times 2.
- * If an individual device has an ra_pages greater than the
- * chunk size, then we will not drive that device as hard as it
- * wants. We consider this a configuration error: a larger
- * chunksize should be used in that case.
- */
- {
+
+ if (mddev->queue) {
+ /* calculate the max read-ahead size.
+ * For read-ahead of large files to be effective, we need to
+ * readahead at least twice a whole stripe. i.e. number of devices
+ * multiplied by chunk size times 2.
+ * If an individual device has an ra_pages greater than the
+ * chunk size, then we will not drive that device as hard as it
+ * wants. We consider this a configuration error: a larger
+ * chunksize should be used in that case.
+ */
int stripe = mddev->raid_disks *
(mddev->chunk_sectors << 9) / PAGE_SIZE;
if (mddev->queue->backing_dev_info.ra_pages < 2* stripe)
--
2.1.0
^ permalink raw reply related
* [PATCH 2/3] dm core: export dm_disk to allow for dm-raid to pass it on to the MD raid0 personality
From: heinzm @ 2015-04-17 12:40 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch is a prerequisite for dm-raid "raid0" support to allow
dm-raid to set rs->md.gendisk, which is required by the MD RAID0
pesonality to set disk stacking limits.
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/dm.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 8001fe9..f11e2f5 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -3206,6 +3206,7 @@ struct gendisk *dm_disk(struct mapped_device *md)
{
return md->disk;
}
+EXPORT_SYMBOL_GPL(dm_disk);
struct kobject *dm_kobject(struct mapped_device *md)
{
--
2.1.0
^ permalink raw reply related
* [PATCH 1/3] dm-raid: Add support for the MD RAID0 personality
From: heinzm @ 2015-04-17 12:40 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patch adds access to the MD RAID0 personality to dm-raid
(in addition to the already supported raid1/10/4/5/6 personalities)
to enable single zone striping.
The following changes enable that access:
- add type definition to raid_types array
- make bitmap creation conditonal in super_validat(), because
it is not allowed in RAID0
- set rdev->sectors to the data image size in super_validate()
to allow the RAID0 personality to calculate the MD array
size properly
- set rs->md.gendisk, because it is required by RAID0
to set disk stack limits
- enhance raid_resume() to not load bitmap in case of "raid0"
- enhance raid_status() to always report full sync for "raid0"
so that userspace checks for 100% sync will succeed and allow
for resize (and takeover/reshape once added in future paches)
- use mdddev(un)lock() functions instead of direct mutex_(un)lock()
(wrapped in here because it's a trivial change)
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
---
drivers/md/dm-raid.c | 115 ++++++++++++++++++++++++++++++---------------------
1 file changed, 68 insertions(+), 47 deletions(-)
diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
index 88e4c7f..14890dd 100644
--- a/drivers/md/dm-raid.c
+++ b/drivers/md/dm-raid.c
@@ -18,6 +18,7 @@
#define DM_MSG_PREFIX "raid"
+
static bool devices_handle_discard_safely = false;
/*
@@ -81,6 +82,7 @@ static struct raid_type {
const unsigned level; /* RAID level. */
const unsigned algorithm; /* RAID algorithm. */
} raid_types[] = {
+ {"raid0", "RAID0 (striping)", 0, 2, 0, 0 /* NONE */},
{"raid1", "RAID1 (mirroring)", 0, 2, 1, 0 /* NONE */},
{"raid10", "RAID10 (striped mirrors)", 0, 2, 10, UINT_MAX /* Varies */},
{"raid4", "RAID4 (dedicated parity disk)", 1, 2, 5, ALGORITHM_PARITY_0},
@@ -720,7 +722,7 @@ static int parse_raid_params(struct raid_set *rs, char **argv,
rs->md.layout = raid10_format_to_md_layout(raid10_format,
raid10_copies);
rs->md.new_layout = rs->md.layout;
- } else if ((rs->raid_type->level > 1) &&
+ } else if ((!rs->raid_type->level || rs->raid_type->level > 1) &&
sector_div(sectors_per_dev,
(rs->md.raid_disks - rs->raid_type->parity_devs))) {
rs->ti->error = "Target length not divisible by number of data devices";
@@ -1026,8 +1028,9 @@ static int super_init_validation(struct mddev *mddev, struct md_rdev *rdev)
return 0;
}
-static int super_validate(struct mddev *mddev, struct md_rdev *rdev)
+static int super_validate(struct raid_set *rs, struct md_rdev *rdev)
{
+ struct mddev *mddev = &rs->md;
struct dm_raid_superblock *sb = page_address(rdev->sb_page);
/*
@@ -1037,8 +1040,10 @@ static int super_validate(struct mddev *mddev, struct md_rdev *rdev)
if (!mddev->events && super_init_validation(mddev, rdev))
return -EINVAL;
- mddev->bitmap_info.offset = 4096 >> 9; /* Enable bitmap creation */
- rdev->mddev->bitmap_info.default_offset = 4096 >> 9;
+ /* Enable bitmap creation for RAID levels != 0 */
+ mddev->bitmap_info.offset = (rs->raid_type->level) ? 4096 >> 9 : 0;
+ rdev->mddev->bitmap_info.default_offset = mddev->bitmap_info.offset;
+
if (!test_bit(FirstUse, &rdev->flags)) {
rdev->recovery_offset = le64_to_cpu(sb->disk_recovery_offset);
if (rdev->recovery_offset != MaxSector)
@@ -1082,6 +1087,8 @@ static int analyse_superblocks(struct dm_target *ti, struct raid_set *rs)
* that the "sync" directive is disallowed during the
* reshape.
*/
+ rdev->sectors = to_sector(i_size_read(rdev->bdev->bd_inode));
+
if (rs->print_flags & DMPF_SYNC)
continue;
@@ -1140,11 +1147,11 @@ static int analyse_superblocks(struct dm_target *ti, struct raid_set *rs)
* validation for the remaining devices.
*/
ti->error = "Unable to assemble array: Invalid superblocks";
- if (super_validate(mddev, freshest))
+ if (super_validate(rs, freshest))
return -EINVAL;
rdev_for_each(rdev, mddev)
- if ((rdev != freshest) && super_validate(mddev, rdev))
+ if ((rdev != freshest) && super_validate(rs, rdev))
return -EINVAL;
return 0;
@@ -1282,10 +1289,14 @@ static int raid_ctr(struct dm_target *ti, unsigned argc, char **argv)
*/
configure_discard_support(ti, rs);
- mutex_lock(&rs->md.reconfig_mutex);
+ /* "raid0" calls disk_stack_limits() requiring gendisk access */
+ rs->md.gendisk = dm_disk(dm_table_get_md(ti->table));
+
+ /* Has to be held on running the array */
+ mddev_lock_nointr(&rs->md);
ret = md_run(&rs->md);
rs->md.in_sync = 0; /* Assume already marked dirty */
- mutex_unlock(&rs->md.reconfig_mutex);
+ mddev_unlock(&rs->md);
if (ret) {
ti->error = "Fail to run raid array";
@@ -1368,34 +1379,40 @@ static void raid_status(struct dm_target *ti, status_type_t type,
case STATUSTYPE_INFO:
DMEMIT("%s %d ", rs->raid_type->name, rs->md.raid_disks);
- if (test_bit(MD_RECOVERY_RUNNING, &rs->md.recovery))
- sync = rs->md.curr_resync_completed;
- else
- sync = rs->md.recovery_cp;
-
- if (sync >= rs->md.resync_max_sectors) {
- /*
- * Sync complete.
- */
+ if (rs->raid_type->level) {
+ if (test_bit(MD_RECOVERY_RUNNING, &rs->md.recovery))
+ sync = rs->md.curr_resync_completed;
+ else
+ sync = rs->md.recovery_cp;
+
+ if (sync >= rs->md.resync_max_sectors) {
+ /*
+ * Sync complete.
+ */
+ array_in_sync = 1;
+ sync = rs->md.resync_max_sectors;
+ } else if (test_bit(MD_RECOVERY_REQUESTED, &rs->md.recovery)) {
+ /*
+ * If "check" or "repair" is occurring, the array has
+ * undergone and initial sync and the health characters
+ * should not be 'a' anymore.
+ */
+ array_in_sync = 1;
+ } else {
+ /*
+ * The array may be doing an initial sync, or it may
+ * be rebuilding individual components. If all the
+ * devices are In_sync, then it is the array that is
+ * being initialized.
+ */
+ for (i = 0; i < rs->md.raid_disks; i++)
+ if (!test_bit(In_sync, &rs->dev[i].rdev.flags))
+ array_in_sync = 1;
+ }
+ } else {
+ /* RAID0 */
array_in_sync = 1;
sync = rs->md.resync_max_sectors;
- } else if (test_bit(MD_RECOVERY_REQUESTED, &rs->md.recovery)) {
- /*
- * If "check" or "repair" is occurring, the array has
- * undergone and initial sync and the health characters
- * should not be 'a' anymore.
- */
- array_in_sync = 1;
- } else {
- /*
- * The array may be doing an initial sync, or it may
- * be rebuilding individual components. If all the
- * devices are In_sync, then it is the array that is
- * being initialized.
- */
- for (i = 0; i < rs->md.raid_disks; i++)
- if (!test_bit(In_sync, &rs->dev[i].rdev.flags))
- array_in_sync = 1;
}
/*
@@ -1684,26 +1701,30 @@ static void raid_resume(struct dm_target *ti)
{
struct raid_set *rs = ti->private;
- set_bit(MD_CHANGE_DEVS, &rs->md.flags);
- if (!rs->bitmap_loaded) {
- bitmap_load(&rs->md);
- rs->bitmap_loaded = 1;
- } else {
- /*
- * A secondary resume while the device is active.
- * Take this opportunity to check whether any failed
- * devices are reachable again.
- */
- attempt_restore_of_faulty_devices(rs);
+ if (rs->raid_type->level) {
+ set_bit(MD_CHANGE_DEVS, &rs->md.flags);
+
+ if (!rs->bitmap_loaded) {
+ bitmap_load(&rs->md);
+ rs->bitmap_loaded = 1;
+ } else {
+ /*
+ * A secondary resume while the device is active.
+ * Take this opportunity to check whether any failed
+ * devices are reachable again.
+ */
+ attempt_restore_of_faulty_devices(rs);
+ }
+
+ clear_bit(MD_RECOVERY_FROZEN, &rs->md.recovery);
}
- clear_bit(MD_RECOVERY_FROZEN, &rs->md.recovery);
mddev_resume(&rs->md);
}
static struct target_type raid_target = {
.name = "raid",
- .version = {1, 6, 0},
+ .version = {1, 7, 0},
.module = THIS_MODULE,
.ctr = raid_ctr,
.dtr = raid_dtr,
--
2.1.0
^ permalink raw reply related
* [PATCH 0/3] dm-raid: Add support for the MD RAID0 personality
From: heinzm @ 2015-04-17 12:40 UTC (permalink / raw)
To: linux-raid; +Cc: Heinz Mauelshagen
From: Heinz Mauelshagen <heinzm@redhat.com>
This patchset of 3 adds access to the MD RAID0 personality to dm-raid
(in addition to the already supported raid1/10/4/5/6 personalities)
to enable single zone striping.
The added functionality is the first step to allow full support
of MD kernel functionality via dm-raid and to eventually replace
the dm "striped" target altogether. Adding shrink/takeover and
reshape functionality to dm-raid is subject to future patches.
Signed-off-by: Heinz Mauelshagen <heinzm@redhat.com>
Tested-by: Heinz Mauelshagen <heinzm@redhat.com>
Heinz Mauelshagen (3):
dm-raid: Add support for the MD RAID0 personality
dm core: export dm_disk to allow for dm-raid to pass it on to the MD
raid0 personality
raid0: conditional mddev->queue access to suit dm-raid
drivers/md/dm-raid.c | 115 ++++++++++++++++++++++++++++++---------------------
drivers/md/dm.c | 1 +
drivers/md/raid0.c | 48 +++++++++++----------
3 files changed, 96 insertions(+), 68 deletions(-)
--
2.1.0
^ permalink raw reply
* Re: mdadm hang when one subdev error(raid1)
From: hui jiao @ 2015-04-17 9:56 UTC (permalink / raw)
To: 席智勇; +Cc: linux-raid
In-Reply-To: <55fb513d.5f1b.14cc6426c67.Coremail.xizhiyong18@163.com>
The md42 is trying to update on-disk superblock, but the operation
doesn't finish. During this peroid, the configuration lock is holded
by md42_raid1 thread, all other operations which need the lock will
wait, such as -D --fail --remove.
what's the status of the subdev now? is it suspended?
check and resume it:
[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
1
[root@node0 ~]# dmsetup resume /dev/dm-0
[root@node0 ~]# cat /sys/block/dm-0/dm/suspended
0
On Fri, Apr 17, 2015 at 3:23 PM, 席智勇 <xizhiyong18@163.com> wrote:
> hi all:
>
> I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
> The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
> I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
> Can anyone give me a help？
> thanks.
>
> uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
> mdadm --version:mdadm - v3.2.5 - 18th May 2012
> kern.log:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
> Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
>
>
> -------------
>
> Zhiyong Xi
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Ondrej Kozina @ 2015-04-17 8:52 UTC (permalink / raw)
To: device-mapper development, Alex Elsayed, Mike Snitzer
Cc: linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150417075211.GC403@redhat.com>
On 04/17/2015 09:52 AM, Mike Snitzer wrote:
> On Thu, Apr 16 2015 at 5:23am -0400,
> Alex Elsayed <eternaleye@gmail.com> wrote:
>
>> Mike Snitzer wrote:
>>
>>> On Thu, Apr 09 2015 at 9:28am -0400,
>>> Pali Rohár <pali.rohar@gmail.com> wrote:
>>>
>>>> On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
>>>>> On Mon, Apr 06 2015 at 9:29am -0400,
>>>>> Pali Rohár <pali.rohar@gmail.com> wrote:
>>>>>
>>>>>> On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
>>>>>>> On Sun, Apr 05 2015 at 1:20pm -0400,
>>>>>>>
>>>>>>> Pali Rohár <pali.rohar@gmail.com> wrote:
>>>>>>>> This patch series increase security of suspend and hibernate
>>>>>>>> actions. It allows user to safely wipe crypto keys before
>>>>>>>> suspend and hibernate actions starts without race
>>>>>>>> conditions on userspace process with heavy I/O.
>>>>>>>>
>>>>>>>> To automatically wipe cryto key for <device> before
>>>>>>>> hibernate action call: $ dmsetup message <device> 0 key
>>>>>>>> wipe_on_hibernation 1
>>>>>>>>
>>>>>>>> To automatically wipe cryto key for <device> before suspend
>>>>>>>> action call: $ dmsetup message <device> 0 key
>>>>>>>> wipe_on_suspend 1
>>>>>>>>
>>>>>>>> (Value 0 after wipe_* string reverts original behaviour - to
>>>>>>>> not wipe key)
>>>>>>>
>>>>>>> Can you elaborate on the attack vector your changes are meant
>>>>>>> to protect against? The user already authorized access, why
>>>>>>> is it inherently dangerous to _not_ wipe the associated key
>>>>>>> across these events?
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> yes, I will try to explain current problems with cryptsetup
>>>>>> luksSuspend command and hibernation.
>>>>>>
>>>>>> First, sometimes it is needed to put machine into other hands.
>>>>>> You can still watch other person what is doing with machine, but
>>>>>> once if you let machine unlocked (e.g opened luks disk), she/he
>>>>>> can access encrypted data.
>>>>>>
>>>>>> If you turn off machine, it could be safe, because luks disk
>>>>>> devices are locked. But if you enter machine into suspend or
>>>>>> hibernate state luks devices are still open. And my patches try
>>>>>> to achieve similar security as when machine is off (= no crypto
>>>>>> keys in RAM or on swap).
>>>>>>
>>>>>> When doing hibernate on unencrypted swap it is to prevent leaking
>>>>>> crypto keys to hibernate image (which is stored in swap).
>>>>>>
>>>>>> When doing suspend action it is again to prevent leaking crypto
>>>>>> keys. E.g when you suspend laptop and put it off (somebody can
>>>>>> remove RAMs and do some cold boot attack).
>>>>>>
>>>>>> The most common situation is:
>>>>>> You have mounted partition from dm-crypt device (e.g. /home/),
>>>>>> some userspace processes access it (e.g opened firefox which
>>>>>> still reads/writes to cache ~/.firefox/) and you want to drop
>>>>>> crypto keys from kernel for some time.
>>>>>>
>>>>>> For that operation there is command cryptsetup luksSuspend, which
>>>>>> suspend dm device and then tell kernel to wipe crypto keys. All
>>>>>> I/O operations are then stopped and userspace processes which
>>>>>> want to do some those I/O operations are stopped too (until you
>>>>>> call cryptsetup luksResume and enter correct key).
>>>>>>
>>>>>> Now if you want to suspend/hiberate your machine (when some of dm
>>>>>> devices are suspeneded and some processes are stopped due to
>>>>>> pending I/O) it is not possible. Kernel freeze_processes function
>>>>>> will fail because userspace processes are still stopped inside
>>>>>> some I/O syscall (read/write, etc,...).
>>>>>>
>>>>>> My patches fixes this problem and do those operations (suspend dm
>>>>>> device, wipe crypto keys, enter suspend/hiberate) in correct
>>>>>> order and without race condition.
>>>>>>
>>>>>> dm device is suspended *after* userspace processes are freezed
>>>>>> and after that are crypto keys wiped. And then computer/laptop
>>>>>> enters into suspend/hibernate state.
>>>>>
>>>>> Wouldn't it be better to fix freeze_processes() to be tolerant of
>>>>> processes that are hung as a side-effect of their backing storage being
>>>>> suspended? A hibernate shouldn't fail simply because a user chose to
>>>>> suspend a DM device.
>>>>>
>>>>> Then this entire problem goes away and the key can be wiped from
>>>>> userspace (like you said above).
>>>>
>>>> Still there will be race condition. Before hibernation (and device
>>>> poweroff) we should have synced disks and filesystems to prevent data
>>>> lose (or other damage) as more as we can. And if there will be some
>>>> application which using lot of I/O (e.g normal firefox) then there
>>>> always will be race condtion.
>>>
>>> The DM suspend will take care of flushing any pending I/O. So I don't
>>> see where the supposed race is...
>>>
>>> Anything else that is trapped in userspace memory will be there when the
>>> machine resumes.
>>>
>>>> So proper way is to wipe luks crypto keys *after* userspace processes
>>>> are freezed.
>>>
>>> I know you believe that I'm just not accepting that at face value.
>>
>> Um, pardon me if I'm being naive, but what about the case of hibernation
>> where the swapdev and the root device are both LVs on the same dm_crypt
>> device?
>>
>> The kernel is writing to swap _after_ userspace processes are all frozen;
>> that seems to me like an ordering dependency entirely incompatible with
>> userspace dropping the key...
>
> Good point, definitely not compatible with the Pali's approach.
Ouch! I'm afraid this effectively killed one of my experiments with
dm-crypt suspend. Good to get reminded sooner than later!
> (but is swap really configured ontop of the same dm-crypt device like
> this in practice? I've not heard of that being a common pattern but I
> could just be sheltered)
yes. It's one among many perfectly valid setups.
(For some the goal here would be to have whole disk encrypted including
boot partition unlocked during boot)
^ permalink raw reply
* WARNING: at kernel/sched/core.c:7331 __might_sleep+0xa6/0xb0() during boot
From: Petr Janecek @ 2015-04-17 7:56 UTC (permalink / raw)
To: linux-raid; +Cc: linux-kernel
Hello,
I'm getting these warnings once during boot since 3.19.0. The fs
is on lvm on md raid1.
[ 57.217167] ------------[ cut here ]------------
[ 57.221800] WARNING: CPU: 4 PID: 1071 at kernel/sched/core.c:7331 __might_sleep+0xa6/0xb0()
[ 57.230184] do not call blocking ops when !TASK_RUNNING; state=2 set at [<ffffffff81085d1b>] prepare_to_wait+0x2b/0xa0
[ 57.240886] Modules linked in: xt_multiport iptable_filter ip_tables x_tables cn dummy bridge stp llc xfs libcrc32c coretemp kvm_intel kvm iTCO_wdt microcode evdev iTCO_vendor_support psmouse serio_raw tpm_infineon tpm_tis tpm i7core_edac edac_core i2c_i801 button lpc_ich mfd_core i2c_core acpi_cpufreq processor thermal_sys hwmon_vid loop ipmi_watchdog ipmi_si ipmi_poweroff ipmi_devintf ipmi_msghandler fuse autofs4 btrfs xor raid6_pq dm_mod raid1 md_mod sg sd_mod hid_generic usbhid hid e1000e ptp crc32c_intel ahci libahci libata ehci_pci ehci_hcd scsi_mod pps_core usbcore usb_common
[ 57.274503] CPU: 4 PID: 1071 Comm: freshclam Not tainted 3.19.4 #72
[ 57.274503] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.1 05/27/2010
[ 57.274504] ffffffff8179723f ffff88022fedb2a8 ffffffff814f70f3 ffffffff81089bf9
[ 57.274507] ffff88022fedb2f8 ffff88022fedb2e8 ffffffff81049bc5 ffff88023466e540
[ 57.274509] ffffffff817af863 00000000000006f6 0000000000000000 ffff88023625d800
[ 57.274511] Call Trace:
[ 57.274512] [<ffffffff814f70f3>] dump_stack+0x4c/0x65
[ 57.274518] [<ffffffff81089bf9>] ? down_trylock+0x29/0x40
[ 57.274521] [<ffffffff81049bc5>] warn_slowpath_common+0x85/0xc0
[ 57.274524] [<ffffffff81049c41>] warn_slowpath_fmt+0x41/0x50
[ 57.274526] [<ffffffff81085d1b>] ? prepare_to_wait+0x2b/0xa0
[ 57.274527] [<ffffffff81085d1b>] ? prepare_to_wait+0x2b/0xa0
[ 57.274529] [<ffffffff8106df86>] __might_sleep+0xa6/0xb0
[ 57.274532] [<ffffffff81282beb>] generic_make_request_checks+0x2b/0x400
[ 57.274534] [<ffffffff8108eb6d>] ? trace_hardirqs_on_caller+0x15d/0x200
[ 57.274537] [<ffffffff8108ec1d>] ? trace_hardirqs_on+0xd/0x10
[ 57.274539] [<ffffffff81282fd3>] generic_make_request+0x13/0x120
[ 57.274541] [<ffffffffa02f95e8>] raid1_unplug+0x108/0x160 [raid1]
[ 57.274544] [<ffffffff812878d2>] blk_flush_plug_list+0xa2/0x240
[ 57.274546] [<ffffffff8108ec1d>] ? trace_hardirqs_on+0xd/0x10
[ 57.274548] [<ffffffff814f9510>] ? bit_wait_timeout+0x80/0x80
[ 57.274550] [<ffffffff814f8cad>] io_schedule+0x7d/0x150
[ 57.274552] [<ffffffff814f953f>] bit_wait_io+0x2f/0x50
[ 57.274553] [<ffffffff814f90cf>] __wait_on_bit+0x5f/0x90
[ 57.274555] [<ffffffff814f9510>] ? bit_wait_timeout+0x80/0x80
[ 57.274556] [<ffffffff814f916d>] out_of_line_wait_on_bit+0x6d/0x80
[ 57.274558] [<ffffffff810860e0>] ? autoremove_wake_function+0x40/0x40
[ 57.274559] [<ffffffff811b0b2f>] __wait_on_buffer+0x3f/0x50
[ 57.274562] [<ffffffff811f64e0>] ext4_wait_block_bitmap.part.10+0xc0/0xd0
[ 57.274566] [<ffffffff811f6509>] ext4_wait_block_bitmap+0x19/0x20
[ 57.274567] [<ffffffff81230ef3>] ext4_mb_init_cache+0x203/0x780
[ 57.274570] [<ffffffff811207e9>] ? lru_cache_add+0x9/0x10
[ 57.274574] [<ffffffff8123158e>] ext4_mb_init_group+0x11e/0x210
[ 57.274575] [<ffffffff812317ab>] ext4_mb_good_group+0x12b/0x1a0
[ 57.274577] [<ffffffff81234b38>] ext4_mb_regular_allocator+0x1b8/0x460
[ 57.274579] [<ffffffff8122ffe0>] ? ext4_mb_use_preallocated.constprop.27+0x40/0x350
[ 57.274580] [<ffffffff8122f6fd>] ? ext4_mb_initialize_context+0x6d/0x1b0
[ 57.274582] [<ffffffff81236830>] ext4_mb_new_blocks+0x480/0x600
[ 57.274584] [<ffffffff8122b27d>] ext4_ext_map_blocks+0x62d/0xeb0
[ 57.274586] [<ffffffff811feb53>] ext4_map_blocks+0x133/0x560
[ 57.274588] [<ffffffff81201ed0>] ext4_writepages+0x560/0xdb0
[ 57.274590] [<ffffffff8111e3ac>] do_writepages+0x1c/0x30
[ 57.274592] [<ffffffff811128d1>] __filemap_fdatawrite_range+0x51/0x60
[ 57.274595] [<ffffffff81112977>] filemap_flush+0x17/0x20
[ 57.274597] [<ffffffff811ff851>] ext4_alloc_da_blocks+0x41/0x90
[ 57.274598] [<ffffffff8120ce0c>] ext4_rename+0x62c/0x880
[ 57.274601] [<ffffffff8108e9e1>] ? mark_held_locks+0x71/0xa0
[ 57.274603] [<ffffffff814faa3e>] ? mutex_lock_nested+0x22e/0x3b0
[ 57.274605] [<ffffffff8120d078>] ext4_rename2+0x18/0x40
[ 57.274606] [<ffffffff811897c2>] vfs_rename+0x3a2/0x7e0
[ 57.274609] [<ffffffff8118c8b0>] SYSC_renameat2+0x5b0/0x620
[ 57.274610] [<ffffffff814fe3e1>] ? _raw_spin_unlock_irqrestore+0x31/0x50
[ 57.274613] [<ffffffff81085bd8>] ? remove_wait_queue+0x48/0x50
[ 57.274615] [<ffffffff8104cb98>] ? do_wait+0x198/0x2c0
[ 57.274617] [<ffffffff8108eb6d>] ? trace_hardirqs_on_caller+0x15d/0x200
[ 57.274618] [<ffffffff812b7b1b>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 57.274621] [<ffffffff8118dba9>] SyS_rename+0x19/0x20
[ 57.274623] [<ffffffff814fecb2>] system_call_fastpath+0x12/0x17
[ 57.274626] ---[ end trace b4ab91b16fa597a6 ]---
Thanks,
Petr Janecek
^ permalink raw reply
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Mike Snitzer @ 2015-04-17 7:52 UTC (permalink / raw)
To: Alex Elsayed; +Cc: dm-devel, linux-raid, linux-kernel, linux-pm
In-Reply-To: <mgnv2g$if5$2@ger.gmane.org>
On Thu, Apr 16 2015 at 5:23am -0400,
Alex Elsayed <eternaleye@gmail.com> wrote:
> Mike Snitzer wrote:
>
> > On Thu, Apr 09 2015 at 9:28am -0400,
> > Pali Rohár <pali.rohar@gmail.com> wrote:
> >
> >> On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
> >> > On Mon, Apr 06 2015 at 9:29am -0400,
> >> > Pali Rohár <pali.rohar@gmail.com> wrote:
> >> >
> >> > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
> >> > > > On Sun, Apr 05 2015 at 1:20pm -0400,
> >> > > >
> >> > > > Pali Rohár <pali.rohar@gmail.com> wrote:
> >> > > > > This patch series increase security of suspend and hibernate
> >> > > > > actions. It allows user to safely wipe crypto keys before
> >> > > > > suspend and hibernate actions starts without race
> >> > > > > conditions on userspace process with heavy I/O.
> >> > > > >
> >> > > > > To automatically wipe cryto key for <device> before
> >> > > > > hibernate action call: $ dmsetup message <device> 0 key
> >> > > > > wipe_on_hibernation 1
> >> > > > >
> >> > > > > To automatically wipe cryto key for <device> before suspend
> >> > > > > action call: $ dmsetup message <device> 0 key
> >> > > > > wipe_on_suspend 1
> >> > > > >
> >> > > > > (Value 0 after wipe_* string reverts original behaviour - to
> >> > > > > not wipe key)
> >> > > >
> >> > > > Can you elaborate on the attack vector your changes are meant
> >> > > > to protect against? The user already authorized access, why
> >> > > > is it inherently dangerous to _not_ wipe the associated key
> >> > > > across these events?
> >> > >
> >> > > Hi,
> >> > >
> >> > > yes, I will try to explain current problems with cryptsetup
> >> > > luksSuspend command and hibernation.
> >> > >
> >> > > First, sometimes it is needed to put machine into other hands.
> >> > > You can still watch other person what is doing with machine, but
> >> > > once if you let machine unlocked (e.g opened luks disk), she/he
> >> > > can access encrypted data.
> >> > >
> >> > > If you turn off machine, it could be safe, because luks disk
> >> > > devices are locked. But if you enter machine into suspend or
> >> > > hibernate state luks devices are still open. And my patches try
> >> > > to achieve similar security as when machine is off (= no crypto
> >> > > keys in RAM or on swap).
> >> > >
> >> > > When doing hibernate on unencrypted swap it is to prevent leaking
> >> > > crypto keys to hibernate image (which is stored in swap).
> >> > >
> >> > > When doing suspend action it is again to prevent leaking crypto
> >> > > keys. E.g when you suspend laptop and put it off (somebody can
> >> > > remove RAMs and do some cold boot attack).
> >> > >
> >> > > The most common situation is:
> >> > > You have mounted partition from dm-crypt device (e.g. /home/),
> >> > > some userspace processes access it (e.g opened firefox which
> >> > > still reads/writes to cache ~/.firefox/) and you want to drop
> >> > > crypto keys from kernel for some time.
> >> > >
> >> > > For that operation there is command cryptsetup luksSuspend, which
> >> > > suspend dm device and then tell kernel to wipe crypto keys. All
> >> > > I/O operations are then stopped and userspace processes which
> >> > > want to do some those I/O operations are stopped too (until you
> >> > > call cryptsetup luksResume and enter correct key).
> >> > >
> >> > > Now if you want to suspend/hiberate your machine (when some of dm
> >> > > devices are suspeneded and some processes are stopped due to
> >> > > pending I/O) it is not possible. Kernel freeze_processes function
> >> > > will fail because userspace processes are still stopped inside
> >> > > some I/O syscall (read/write, etc,...).
> >> > >
> >> > > My patches fixes this problem and do those operations (suspend dm
> >> > > device, wipe crypto keys, enter suspend/hiberate) in correct
> >> > > order and without race condition.
> >> > >
> >> > > dm device is suspended *after* userspace processes are freezed
> >> > > and after that are crypto keys wiped. And then computer/laptop
> >> > > enters into suspend/hibernate state.
> >> >
> >> > Wouldn't it be better to fix freeze_processes() to be tolerant of
> >> > processes that are hung as a side-effect of their backing storage being
> >> > suspended? A hibernate shouldn't fail simply because a user chose to
> >> > suspend a DM device.
> >> >
> >> > Then this entire problem goes away and the key can be wiped from
> >> > userspace (like you said above).
> >>
> >> Still there will be race condition. Before hibernation (and device
> >> poweroff) we should have synced disks and filesystems to prevent data
> >> lose (or other damage) as more as we can. And if there will be some
> >> application which using lot of I/O (e.g normal firefox) then there
> >> always will be race condtion.
> >
> > The DM suspend will take care of flushing any pending I/O. So I don't
> > see where the supposed race is...
> >
> > Anything else that is trapped in userspace memory will be there when the
> > machine resumes.
> >
> >> So proper way is to wipe luks crypto keys *after* userspace processes
> >> are freezed.
> >
> > I know you believe that I'm just not accepting that at face value.
>
> Um, pardon me if I'm being naive, but what about the case of hibernation
> where the swapdev and the root device are both LVs on the same dm_crypt
> device?
>
> The kernel is writing to swap _after_ userspace processes are all frozen;
> that seems to me like an ordering dependency entirely incompatible with
> userspace dropping the key...
Good point, definitely not compatible with the Pali's approach.
(but is swap really configured ontop of the same dm-crypt device like
this in practice? I've not heard of that being a common pattern but I
could just be sheltered)
^ permalink raw reply
* mdadm hang when one subdev error(raid1)
From: 席智勇 @ 2015-04-17 7:23 UTC (permalink / raw)
To: linux-raid
hi all:
I create some raid1-device by mdadm, when one subdev error, all mdadm related operation just hang there, process state was D.
The backgroud is a physical disk was error, so a subdev which is part of the error disk created by device mapper must be errorred, then i did the command 'mdadm --fail' to fail the subdev from the md device, I found it not responsable, just hang there、I tryed 'mdadm --remove', even 'mdadm -D', all hang there. Later, I found not just mdadm operation hang on the problem md device, all mdadm operation on the machine connot be excute.
I wana find out what's the problem is, is it a bug of raid when disk error occur， or a problem of my system, because when i found the mdadm hang, the errored disk(/dev/sdp)just missing from my system, I said the disk was error judging from the error log in raid card log.
Can anyone give me a help？
thanks.
uname -a :Linux **-***-***-** 3.10.45-****-amd64 #1 SMP Tue Jul 1 01:52:20 UTC 2014 x86_64 GNU/Linux
mdadm --version:mdadm - v3.2.5 - 18th May 2012
kern.log:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561205] kvm D ffff88407f313f40 0 11581 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561209] ffff88356e846080 0000000000000082 0000000000000092 ffff881fe2d6a080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561215] 0000000000013f40 ffff882849dfdfd8 ffff882849dfdfd8 ffff88356e846080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561219] ffffffff8139958c ffff881cac478000 ffff882849dfdcb0 ffff881cac478290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561224] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561233] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561266] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561271] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561275] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561284] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561295] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561300] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561304] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561307] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561311] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561315] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561318] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561356] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.561367] INFO: task md52_raid1:39767 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.563976] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566971] md52_raid1 D ffff88407f233f40 0 39767 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566975] ffff883e60fa9810 0000000000000046 ffff883e60fa9810 ffff881fe2d620c0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566981] 0000000000013f40 ffff883d61523fd8 ffff883d61523fd8 ffff883e60fa9810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566985] ffffffff8139958c ffff883d61523c60 ffff881cac478000 ffff881cac478290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566990] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.566995] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567007] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567011] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567019] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567023] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567030] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567034] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567038] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567042] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567045] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567048] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567088] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567092] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567100] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567107] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567110] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567113] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567117] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567120] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.567125] INFO: task kvm:19762 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.570192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573492] kvm D ffff88407f273f40 0 19762 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573496] ffff883fc90f97d0 0000000000000082 0000000000011200 ffff881fe2d64040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573502] 0000000000013f40 ffff882aad01dfd8 ffff882aad01dfd8 ffff883fc90f97d0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573507] ffffffff8139958c ffff881fb5475800 ffff882aad01dcb0 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573512] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573517] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573529] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573545] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573549] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573557] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573561] [<ffffffff81100cbd>] ? ____cache_alloc+0x25d/0x293
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573569] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573576] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573580] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573583] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573586] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573590] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573593] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573596] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573600] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.573603] INFO: task kvm:9147 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.576936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580590] kvm D ffff88407f3f3f40 0 9147 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580594] ffff883fd9bfd080 0000000000000082 0000000000000096 ffff881fe2db5040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580631] 0000000000013f40 ffff882da06a1fd8 ffff882da06a1fd8 ffff883fd9bfd080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580635] ffffffff8139958c ffff881fb5475800 ffff882da06a1820 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580648] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580653] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580666] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580670] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580674] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580678] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580685] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580689] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580697] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580704] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580708] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580712] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580716] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580719] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580723] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580752] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580755] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580759] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580763] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580767] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580770] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580773] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.580778] INFO: task md42_raid1:36156 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.584560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588685] md42_raid1 D ffff88207fa33f40 0 36156 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588690] ffff881d0b324080 0000000000000046 ffff881d0b324080 ffff881fe2d62810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588695] 0000000000013f40 ffff881fe1157fd8 ffff881fe1157fd8 ffff881d0b324080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588699] ffffffff8139958c ffff881fe1157c60 ffff881fb5475800 ffff881fb5475a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588703] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588710] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588734] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588738] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588745] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588750] [<ffffffff8100c02f>] ? load_TLS+0x7/0xa
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588758] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588761] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588766] [<ffffffff81049389>] ? lock_timer_base.isra.35+0x23/0x48
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588769] [<ffffffff810490d4>] ? detach_if_pending+0x18/0x6c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588772] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588775] [<ffffffff810494ae>] ? try_to_del_timer_sync+0x4e/0x59
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588778] [<ffffffff810494e0>] ? del_timer_sync+0x27/0x44
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588781] [<ffffffff8139796c>] ? schedule_timeout+0xf5/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588784] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588787] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588794] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588797] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588804] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588810] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588813] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588816] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588819] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588822] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.588834] INFO: task kvm:14262 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.592943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597362] kvm D ffff88207fbb3f40 0 14262 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597365] ffff88361cc91080 0000000000000082 0000000000000092 ffff881fe2db3810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597371] 0000000000013f40 ffff882849fd9fd8 ffff882849fd9fd8 ffff88361cc91080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597374] ffffffff8139958c ffff883fe1297000 ffff882849fd9cb0 ffff883fe1297290
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597378] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597383] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597400] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597403] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597407] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597415] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597422] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597426] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597430] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597433] [<ffffffff811b6b0a>] ? blkdev_issue_flush+0x86/0xc4
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597438] [<ffffffff8113b298>] ? blkdev_fsync+0x2b/0x37
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597442] [<ffffffff81134a43>] ? do_fsync+0x2b/0x50
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597445] [<ffffffff81134c4f>] ? SyS_fdatasync+0xb/0xf
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597449] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.597452] INFO: task kvm:14266 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.602000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606850] kvm D ffff88207fa53f40 0 14266 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606854] ffff883e6f7a3810 0000000000000082 0000000000000096 ffff881fe2d63850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606860] 0000000000013f40 ffff883c8fc49fd8 ffff883c8fc49fd8 ffff883e6f7a3810
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606865] ffffffff8139958c ffff881b5fa52800 ffff883c8fc49700 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606870] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606877] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606897] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606901] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606906] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606909] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606917] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606925] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606932] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606936] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606940] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606944] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606947] [<ffffffff8113d115>] ? dio_send_cur_page+0x7d/0xa8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606951] [<ffffffff8113d1e8>] ? submit_page_section+0xa8/0x112
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606954] [<ffffffff8113da99>] ? do_blockdev_direct_IO+0x7bb/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606958] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606961] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606964] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606969] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606972] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606976] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606980] [<ffffffff81112038>] ? do_sync_readv_writev+0x50/0x76
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606983] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606987] [<ffffffff811130a6>] ? do_readv_writev+0xbf/0x135
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606990] [<ffffffff8113b5d3>] ? bd_may_claim+0x2c/0x2c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606993] [<ffffffff8111205e>] ? do_sync_readv_writev+0x76/0x76
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.606997] [<ffffffff81126eee>] ? fget_light+0x6b/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607000] [<ffffffff81111fbb>] ? fdget+0xe/0x17
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607004] [<ffffffff811133fe>] ? SyS_pwritev+0x65/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607007] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.607010] INFO: task kvm:14306 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.612040] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617441] kvm D ffff88207fa53f40 0 14306 1 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617445] ffff880eedbaa080 0000000000000082 0000000000011200 ffff881fe2d63850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617451] 0000000000013f40 ffff881d8e5edfd8 ffff881d8e5edfd8 ffff880eedbaa080
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617456] ffffffff8139958c ffff881b5fa52800 ffff881d8e5ed820 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617460] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617467] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617486] [<ffffffffa02efff0>] ? md_write_start+0x131/0x147 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617490] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617509] [<ffffffffa05c2762>] ? make_request+0x37/0xa63 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617513] [<ffffffff81399449>] ? _raw_read_lock_irqsave+0x21/0x2a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617531] [<ffffffffa000679f>] ? __split_and_process_bio+0x40d/0x420 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617536] [<ffffffff81102005>] ? kmem_cache_alloc+0xe1/0x154
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617552] [<ffffffffa02f4b7e>] ? md_make_request+0xee/0x1df [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617559] [<ffffffffa0006902>] ? dm_request+0x150/0x163 [dm_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617563] [<ffffffff811b3efc>] ? generic_make_request+0x96/0xd5
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617573] [<ffffffff811b4c79>] ? submit_bio+0x10a/0x13b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617577] [<ffffffff8113ce89>] ? dio_bio_submit+0x68/0x88
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617580] [<ffffffff8113dc35>] ? do_blockdev_direct_IO+0x957/0xae8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617584] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617587] [<ffffffff8113b0e8>] ? blkdev_direct_IO+0x4e/0x53
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617590] [<ffffffff8113aef1>] ? I_BDEV+0x8/0x8
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617594] [<ffffffff810c800a>] ? generic_file_direct_write+0xe3/0x14a
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617597] [<ffffffff810c818c>] ? __generic_file_aio_write+0x11b/0x1ff
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617601] [<ffffffff8113b617>] ? blkdev_aio_write+0x44/0x93
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617604] [<ffffffff811120b3>] ? do_sync_write+0x55/0x7c
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617608] [<ffffffff81112ab0>] ? vfs_write+0x9d/0x103
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617611] [<ffffffff81112eb9>] ? SyS_pwrite64+0x61/0x87
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617614] [<ffffffff8139ea69>] ? system_call_fastpath+0x16/0x1b
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.617629] INFO: task md71_raid1:14478 blocked for more than 120 seconds.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.622944] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628476] md71_raid1 D ffff88207fad3f40 0 14478 2 0x00000000
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628480] ffff883ed1a09040 0000000000000046 ffff883ed1a09040 ffff881fe2d68850
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628486] 0000000000013f40 ffff883fbd0b7fd8 ffff883fbd0b7fd8 ffff883ed1a09040
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628490] ffffffff8139958c ffff883fbd0b7c60 ffff881b5fa52800 ffff881b5fa52a90
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628498] Call Trace:
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628504] [<ffffffff8139958c>] ? _raw_spin_unlock_irqrestore+0xc/0xd
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628522] [<ffffffffa02f4d65>] ? md_super_wait+0x69/0x7f [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628526] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628534] [<ffffffffa02f5131>] ? md_update_sb+0x3b6/0x4b8 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628542] [<ffffffffa02f59ee>] ? md_check_recovery+0x1c6/0x3d1 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628546] [<ffffffffa05c31cc>] ? raid1d+0x3e/0xb22 [raid1]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628550] [<ffffffff813988db>] ? __schedule+0x4e7/0x53d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628553] [<ffffffff813978a3>] ? schedule_timeout+0x2c/0x123
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628557] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628560] [<ffffffff813995cb>] ? _raw_spin_lock_irqsave+0x14/0x35
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628568] [<ffffffffa02f02ed>] ? md_thread+0x114/0x132 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628571] [<ffffffff81059c2f>] ? abort_exclusive_wait+0x79/0x79
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628578] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628585] [<ffffffffa02f01d9>] ? signal_pending+0x10/0x10 [md_mod]
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628588] [<ffffffff81059295>] ? kthread+0x81/0x89
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628591] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628594] [<ffffffff8139e9bc>] ? ret_from_fork+0x7c/0xb0
Apr 12 19:33:47 10-120-202-67 kernel: [10636897.628597] [<ffffffff81059214>] ? __kthread_parkme+0x5d/0x5d
-------------
Zhiyong Xi
^ permalink raw reply
* [PATCH] Manage: prevent re-adding a raid1 device that was assembled on its own
From: nhdailey @ 2015-04-16 20:03 UTC (permalink / raw)
To: linux-raid; +Cc: Nate Dailey
From: Nate Dailey <nate.dailey@stratus.com>
When re-adding a device to a RAID1, check if it appears to have been assembled
on its own and prevent the re-add if it has (unless --force is given). This
prevents overwriting unique data on the device to be added, and prevents
incomplete resync if a bitmap is in use.
To determine this, get the raid_disk numbers for devices already in the array,
and check those against the superblock of the device-to-be-added. If any active
disk shows as faulty/removed, then the device being added was likely assembled
on its own.
Signed-off-by: Nate Dailey <nate.dailey@stratus.com>
---
Manage.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 52 insertions(+), 3 deletions(-)
diff --git a/Manage.c b/Manage.c
index d3cfb55..f1df608 100644
--- a/Manage.c
+++ b/Manage.c
@@ -593,17 +593,45 @@ static void add_set(struct mddev_dev *dv, int fd, char set_char)
}
}
+int assembled_separately(int fd, struct supertype *tst, char *devmap)
+{
+ mdu_disk_info_t disc;
+ int j;
+
+ /* This device may have been assembled on its own if the superblock
+ * indicates that some device in the running array is faulty/removed.
+ */
+ for (j = 0; j < tst->max_devs; j++) {
+ disc.number = j;
+ if (ioctl(fd, GET_DISK_INFO, &disc))
+ continue;
+ if (disc.major == 0 && disc.minor == 0)
+ continue;
+ if (disc.state & (1<<MD_DISK_FAULTY | 1<<MD_DISK_REMOVED))
+ continue;
+ if (disc.raid_disk < 0)
+ continue;
+ if (devmap[disc.raid_disk] == 0)
+ return 1;
+ }
+
+ return 0;
+}
+
int attempt_re_add(int fd, int tfd, struct mddev_dev *dv,
struct supertype *dev_st, struct supertype *tst,
unsigned long rdev,
- char *update, char *devname, int verbose,
+ char *update, char *devname, int force, int verbose,
mdu_array_info_t *array)
{
struct mdinfo mdi;
int duuid[4];
int ouuid[4];
+ char *devmap = NULL;
- dev_st->ss->getinfo_super(dev_st, &mdi, NULL);
+ devmap = calloc(array->raid_disks, 1);
+ mdi.array.raid_disks = array->raid_disks;
+ dev_st->ss->getinfo_super(dev_st, &mdi, devmap);
dev_st->ss->uuid_from_super(dev_st, ouuid);
if (tst->sb)
tst->ss->uuid_from_super(tst, duuid);
@@ -630,6 +658,22 @@ int attempt_re_add(int fd, int tfd, struct mddev_dev *dv,
|| disc.major != 0 || disc.minor != 0
)
goto skip_re_add;
+
+ if (array->level == 1 &&
+ assembled_separately(fd, tst, devmap)) {
+ if (!force) {
+ pr_err("not adding %s as it reports a device in the active array as failed.\n"
+ " Add --force if you really want to add this device.\n",
+ dv->devname);
+ free(devmap);
+ return 1;
+ }
+ pr_err("%s reports a device in the active array as failed.\n"
+ " Adding anyway as --force was given.\n",
+ dv->devname);
+ goto skip_re_add;
+ }
+
disc.major = major(rdev);
disc.minor = minor(rdev);
disc.number = mdi.disk.number;
@@ -645,6 +689,7 @@ int attempt_re_add(int fd, int tfd, struct mddev_dev *dv,
tfd = dev_open(dv->devname, O_RDWR);
if (tfd < 0) {
pr_err("failed to open %s for superblock update during re-add\n", dv->devname);
+ free(devmap);
return -1;
}
@@ -665,6 +710,7 @@ int attempt_re_add(int fd, int tfd, struct mddev_dev *dv,
close(tfd);
if (rv != 0) {
pr_err("failed to update superblock during re-add\n");
+ free(devmap);
return -1;
}
}
@@ -673,17 +719,20 @@ int attempt_re_add(int fd, int tfd, struct mddev_dev *dv,
if (ioctl(fd, ADD_NEW_DISK, &disc) == 0) {
if (verbose >= 0)
pr_err("re-added %s\n", dv->devname);
+ free(devmap);
return 1;
}
if (errno == ENOMEM || errno == EROFS) {
pr_err("add new device failed for %s: %s\n",
dv->devname, strerror(errno));
+ free(devmap);
if (dv->disposition == 'M')
return 0;
return -1;
}
}
skip_re_add:
+ free(devmap);
return 0;
}
@@ -805,7 +854,7 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
dev_st, tst,
rdev,
update, devname,
- verbose,
+ force, verbose,
array);
dev_st->ss->free_super(dev_st);
if (rv)
--
1.8.3.1
^ permalink raw reply related
* Re: RAID 5 "magicaly" become a RAID0
From: Phil Turmel @ 2015-04-16 12:25 UTC (permalink / raw)
To: Thomas MARCHESSEAU, linux-raid; +Cc: Roger Heflin
In-Reply-To: <D1554172.8F19F%marchesseau@gmail.com>
Good morning Thomas,
On 04/16/2015 04:53 AM, Thomas MARCHESSEAU wrote:
> Hi Team ,
>
>>
>> Very good to hear this.
> It crashed once , but after a reboot and re-assemble , i¹ve backuped
> _everything_
>
> -> Thank you very much for you support !
You're welcome.
>> Yes, just --add a spare device to the array, then --grow --level=6 to
>> start a live reshape.
>
> I tryed , but it failed with many buffer I/O error , ( mostly on the last
> 4 drives , ( not sure , but seems hosted on the same chipset)
> And now , i have tons of :
> Buffer I/O error on device sdf , logical block 12
> Š logical block 11 Š and so on
>
> Same on SDH
You mentioned a possibly loose sata cable causing the initial problem.
I'm not so sure. I think you should check your power supply.
The smartctl reports you supplied eliminate the most common problem,
timeout mismatch, as you have nice raid-rated drives. The next most
common problem I've seen is insufficient power, which is most apparent
when writing.
Add up all of the power consumption ratings of your hardware to make
sure a new power supply is suitable. And if you have/get a multiple
rail supply, make sure you spread the load across the independent circuits.
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] dm crypt: fix error return code in crypt_ctr()
From: weiyj_lk @ 2015-04-16 12:21 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Neil Brown
Cc: Wei Yongjun, dm-devel, linux-raid
From: Wei Yongjun <yongjun_wei@trendmicro.com.cn>
Fix to return a negative error code from the error handling
case instead of 0, as done elsewhere in this function.
Signed-off-by: Wei Yongjun <yongjun_wei@trendmicro.com.cn>
---
drivers/md/dm-crypt.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 713a962..8154f58 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -1816,6 +1816,7 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
if (ret)
goto bad;
+ ret = -EINVAL;
while (opt_params--) {
opt_string = dm_shift_arg(&as);
if (!opt_string) {
^ permalink raw reply related
* Re: [PATCH 0/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation
From: Alex Elsayed @ 2015-04-16 9:23 UTC (permalink / raw)
To: dm-devel; +Cc: linux-raid, linux-kernel, linux-pm
In-Reply-To: <20150409140842.GA9779@redhat.com>
Mike Snitzer wrote:
> On Thu, Apr 09 2015 at 9:28am -0400,
> Pali Rohár <pali.rohar@gmail.com> wrote:
>
>> On Thursday 09 April 2015 09:12:08 Mike Snitzer wrote:
>> > On Mon, Apr 06 2015 at 9:29am -0400,
>> > Pali Rohár <pali.rohar@gmail.com> wrote:
>> >
>> > > On Monday 06 April 2015 15:00:46 Mike Snitzer wrote:
>> > > > On Sun, Apr 05 2015 at 1:20pm -0400,
>> > > >
>> > > > Pali Rohár <pali.rohar@gmail.com> wrote:
>> > > > > This patch series increase security of suspend and hibernate
>> > > > > actions. It allows user to safely wipe crypto keys before
>> > > > > suspend and hibernate actions starts without race
>> > > > > conditions on userspace process with heavy I/O.
>> > > > >
>> > > > > To automatically wipe cryto key for <device> before
>> > > > > hibernate action call: $ dmsetup message <device> 0 key
>> > > > > wipe_on_hibernation 1
>> > > > >
>> > > > > To automatically wipe cryto key for <device> before suspend
>> > > > > action call: $ dmsetup message <device> 0 key
>> > > > > wipe_on_suspend 1
>> > > > >
>> > > > > (Value 0 after wipe_* string reverts original behaviour - to
>> > > > > not wipe key)
>> > > >
>> > > > Can you elaborate on the attack vector your changes are meant
>> > > > to protect against? The user already authorized access, why
>> > > > is it inherently dangerous to _not_ wipe the associated key
>> > > > across these events?
>> > >
>> > > Hi,
>> > >
>> > > yes, I will try to explain current problems with cryptsetup
>> > > luksSuspend command and hibernation.
>> > >
>> > > First, sometimes it is needed to put machine into other hands.
>> > > You can still watch other person what is doing with machine, but
>> > > once if you let machine unlocked (e.g opened luks disk), she/he
>> > > can access encrypted data.
>> > >
>> > > If you turn off machine, it could be safe, because luks disk
>> > > devices are locked. But if you enter machine into suspend or
>> > > hibernate state luks devices are still open. And my patches try
>> > > to achieve similar security as when machine is off (= no crypto
>> > > keys in RAM or on swap).
>> > >
>> > > When doing hibernate on unencrypted swap it is to prevent leaking
>> > > crypto keys to hibernate image (which is stored in swap).
>> > >
>> > > When doing suspend action it is again to prevent leaking crypto
>> > > keys. E.g when you suspend laptop and put it off (somebody can
>> > > remove RAMs and do some cold boot attack).
>> > >
>> > > The most common situation is:
>> > > You have mounted partition from dm-crypt device (e.g. /home/),
>> > > some userspace processes access it (e.g opened firefox which
>> > > still reads/writes to cache ~/.firefox/) and you want to drop
>> > > crypto keys from kernel for some time.
>> > >
>> > > For that operation there is command cryptsetup luksSuspend, which
>> > > suspend dm device and then tell kernel to wipe crypto keys. All
>> > > I/O operations are then stopped and userspace processes which
>> > > want to do some those I/O operations are stopped too (until you
>> > > call cryptsetup luksResume and enter correct key).
>> > >
>> > > Now if you want to suspend/hiberate your machine (when some of dm
>> > > devices are suspeneded and some processes are stopped due to
>> > > pending I/O) it is not possible. Kernel freeze_processes function
>> > > will fail because userspace processes are still stopped inside
>> > > some I/O syscall (read/write, etc,...).
>> > >
>> > > My patches fixes this problem and do those operations (suspend dm
>> > > device, wipe crypto keys, enter suspend/hiberate) in correct
>> > > order and without race condition.
>> > >
>> > > dm device is suspended *after* userspace processes are freezed
>> > > and after that are crypto keys wiped. And then computer/laptop
>> > > enters into suspend/hibernate state.
>> >
>> > Wouldn't it be better to fix freeze_processes() to be tolerant of
>> > processes that are hung as a side-effect of their backing storage being
>> > suspended? A hibernate shouldn't fail simply because a user chose to
>> > suspend a DM device.
>> >
>> > Then this entire problem goes away and the key can be wiped from
>> > userspace (like you said above).
>>
>> Still there will be race condition. Before hibernation (and device
>> poweroff) we should have synced disks and filesystems to prevent data
>> lose (or other damage) as more as we can. And if there will be some
>> application which using lot of I/O (e.g normal firefox) then there
>> always will be race condtion.
>
> The DM suspend will take care of flushing any pending I/O. So I don't
> see where the supposed race is...
>
> Anything else that is trapped in userspace memory will be there when the
> machine resumes.
>
>> So proper way is to wipe luks crypto keys *after* userspace processes
>> are freezed.
>
> I know you believe that I'm just not accepting that at face value.
Um, pardon me if I'm being naive, but what about the case of hibernation
where the swapdev and the root device are both LVs on the same dm_crypt
device?
The kernel is writing to swap _after_ userspace processes are all frozen;
that seems to me like an ordering dependency entirely incompatible with
userspace dropping the key...
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply
* Re: RAID 5 "magicaly" become a RAID0
From: Thomas MARCHESSEAU @ 2015-04-16 8:53 UTC (permalink / raw)
To: Phil Turmel, linux-raid; +Cc: Roger Heflin
In-Reply-To: <552D08C0.1010106@turmel.org>
Hi Team ,
>
>Very good to hear this.
It crashed once , but after a reboot and re-assemble , i¹ve backuped
_everything_
-> Thank you very much for you support !
>
>Yes, just --add a spare device to the array, then --grow --level=6 to
>start a live reshape.
I tryed , but it failed with many buffer I/O error , ( mostly on the last
4 drives , ( not sure , but seems hosted on the same chipset)
And now , i have tons of :
Buffer I/O error on device sdf , logical block 12
Š logical block 11 Š and so on
Same on SDH
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: mdadm raid 5 one disk overwritten file system failed
From: Wols Lists @ 2015-04-15 18:27 UTC (permalink / raw)
To: Mikael Abrahamsson, John Andre Taule; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1504151432250.16871@uplift.swm.pp.se>
On 15/04/15 13:38, Mikael Abrahamsson wrote:
> On Wed, 15 Apr 2015, John Andre Taule wrote:
>
>> The guy that did this to us got 3 months jail.
>>
>> His argument was that we should have failed the system manually
>> (removed the
>> disk that he targeted with "dd"), and the raid should have magically
>> fixed
>> itself. Anyone think this would have worked?
>> It was 5 hours of heavy write and deletes to the file system (ext4)
>> and all
>> that time the dd command where running.
>
> Not a chance, after 5 hours dd basically had overwritten 1/3 of the data
> spread out across a large portion of the volume. We're talking massive
> file and filesystem corruption.
Wouldn't failing the drive and then adding it "as new" (triggering a
rebuild) recover any files that hadn't been modified while the dd was
running?
Of course, that still means any directories that had been modified would
likely have also been corrupted, in all probability landing the files in
them into "lost+found" and necessitating a massive recataloging of all
the files in there.
The data would have been recovered, but the directory structure ... not
a nice recovery job.
Cheers,
Wol
^ permalink raw reply
* Re: mdadm raid 5 one disk overwritten file system failed
From: Mikael Abrahamsson @ 2015-04-15 12:38 UTC (permalink / raw)
To: John Andre Taule; +Cc: linux-raid
In-Reply-To: <00b901d07771$f56c0010$e0440030$@johnandre.net>
On Wed, 15 Apr 2015, John Andre Taule wrote:
> The guy that did this to us got 3 months jail.
>
> His argument was that we should have failed the system manually (removed the
> disk that he targeted with "dd"), and the raid should have magically fixed
> itself. Anyone think this would have worked?
> It was 5 hours of heavy write and deletes to the file system (ext4) and all
> that time the dd command where running.
Not a chance, after 5 hours dd basically had overwritten 1/3 of the data
spread out across a large portion of the volume. We're talking massive
file and filesystem corruption.
I don't know enough about zfs, but I am under the impression that zfs
perhaps could have detected the bad information (because checksum would no
longer match on those blocks) if you would have had native zfs to create
the raid5. I don't have personal experience with zfs though, someone else
might be able to answer that part.
It's really hard to protect against this kind of intentional sabotage.
Even if you would have run zfs instead, he could have just dd:ed to the
actual raid volume instead.
^ permalink raw reply
* SV: mdadm raid 5 one disk overwritten file system failed
From: John Andre Taule @ 2015-04-15 11:47 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <alpine.DEB.2.02.1502191521450.4007@uplift.swm.pp.se>
The guy that did this to us got 3 months jail.
His argument was that we should have failed the system manually (removed the
disk that he targeted with "dd"), and the raid should have magically fixed
itself. Anyone think this would have worked?
It was 5 hours of heavy write and deletes to the file system (ext4) and all
that time the dd command where running.
Later I also found this exact "test" of raid in mdadm documentation marking
it as not something you should do (will fail data integriy, eg corrupt
filesystem, period).
/regards
-----Opprinnelig melding-----
Fra: Mikael Abrahamsson [mailto:swmike@swm.pp.se]
Sendt: 19. februar 2015 15:24
Til: John Andre Taule
Kopi: linux-raid@vger.kernel.org
Emne: Re: mdadm raid 5 one disk overwritten file system failed
On Thu, 19 Feb 2015, John Andre Taule wrote:
> I'm a bit surprised that overwriting anything on the physical disk
> should corrupt the file system on the raid. I would think that would
> be similar to a disk crashing or failing in other ways.
Errr, in raid5 you have data blocks and parity blocks. WHen you overwrite
one of the component drives with zeroes, you're effectively doing the same
as writing 0:es to a non-raid drive every 3 $stripesize. You're zero:ing a
lot of the filesystem information.
> What you say that Linux might not have seen the disk as failing is
> interesting. This could explain why the file system got corrupted.
Correct. There is no mechanism that periodically checks the contents of the
superblock and fails the drive if it's not there anymore. So the drive is
never failed.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: BUG: unable to handle kernel NULL pointer dereference at sysfs_do_create_link_sd (after mdadm)
From: Azat Khuzhin @ 2015-04-15 8:47 UTC (permalink / raw)
To: Guoqing Jiang; +Cc: Kernel.org-Linux-RAID, neilb
In-Reply-To: <552DD097.9010504@suse.com>
On Wed, Apr 15, 2015 at 10:44:39AM +0800, Guoqing Jiang wrote:
> Azat Khuzhin wrote:
> > $ git describe
> > v4.0-2620-gb79013b
> >
> > During setting up partitions with mdadm, mdadm hung, after attaching to mdadm with strace I got next:
> >
> > # pgrep mdadm | xargs strace -fp
> > Process 27389 attached - interrupt to quit
> > unlink("/dev/.tmp.md.27389:9:127") = 0
> > mknod("/tmp/.tmp.md.27389:9:127", S_IFBLK|0600, makedev(9, 127)) = 0
> > open("/tmp/.tmp.md.27389:9:127", O_RDWR|O_EXCL|O_DIRECT) <-- *hung*
> >
> > After, I looked into dmesg, and found this:
> > [ 9627.630018] ------------[ cut here ]------------
> > [ 9627.630029] WARNING: CPU: 18 PID: 3330 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x5a/0x70()
> > [ 9627.630032] sysfs: cannot create duplicate filename '/devices/virtual/bdi/9:127'
> >
> sysfs complains about duplicate filename, maybe you tried to create the
> array with same node.
Yes, my steps was close to this:
$ mdadm --create /dev/md/foo --raid-devices=2 /dev/sda /dev/sdb
$ mdadm --stop --scan
$ mdadm --create /dev/md/foo --raid-devices=2 /dev/sda /dev/sdb
But it is not always reproduces.
Anyway the most important is not WARNING but BUG (also after that BUG
you can't umount all file systems normally -- i.e. on reboot).
^ permalink raw reply
* Re: BUG: unable to handle kernel NULL pointer dereference at sysfs_do_create_link_sd (after mdadm)
From: Guoqing Jiang @ 2015-04-15 2:44 UTC (permalink / raw)
To: Azat Khuzhin; +Cc: Kernel.org-Linux-RAID, neilb
In-Reply-To: <20150414171537.GH25394@azat>
Azat Khuzhin wrote:
> $ git describe
> v4.0-2620-gb79013b
>
> During setting up partitions with mdadm, mdadm hung, after attaching to mdadm with strace I got next:
>
> # pgrep mdadm | xargs strace -fp
> Process 27389 attached - interrupt to quit
> unlink("/dev/.tmp.md.27389:9:127") = 0
> mknod("/tmp/.tmp.md.27389:9:127", S_IFBLK|0600, makedev(9, 127)) = 0
> open("/tmp/.tmp.md.27389:9:127", O_RDWR|O_EXCL|O_DIRECT) <-- *hung*
>
> After, I looked into dmesg, and found this:
> [ 9627.630018] ------------[ cut here ]------------
> [ 9627.630029] WARNING: CPU: 18 PID: 3330 at fs/sysfs/dir.c:31 sysfs_warn_dup+0x5a/0x70()
> [ 9627.630032] sysfs: cannot create duplicate filename '/devices/virtual/bdi/9:127'
>
sysfs complains about duplicate filename, maybe you tried to create the
array with same node.
> [ 9627.630033] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables nfsd nfs lockd grace sunrpc ipmi_devintf netconsole configfs loop hid_generic usbhid hid x86_pkg_temp_thermal coretemp ghash_clmulni_intel aesni_intel ioatdma ehci_pci aes_x86_64 iTCO_wdt iTCO_ve
> [ 9627.630074] CPU: 18 PID: 3330 Comm: mdadm Not tainted 4.0.0bl-azat-v6+ #1
> [ 9627.630076] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
> [ 9627.630077] 0000000000000000 ffffffff814e3fcc ffffffff813e590f ffff885f9bcd3808
> [ 9627.630079] ffffffff8104575c ffff885f96acb000 ffff885fa4b3e3c0 ffff885fa2fec780
> [ 9627.630081] ffff885fa4bc4000 0000000000000000 ffffffff810457d5 ffffffff814e5d78
> [ 9627.630083] Call Trace:
> [ 9627.630091] [<ffffffff813e590f>] ? dump_stack+0x40/0x50
> [ 9627.630096] [<ffffffff8104575c>] ? warn_slowpath_common+0x7c/0xb0
> [ 9627.630098] [<ffffffff810457d5>] ? warn_slowpath_fmt+0x45/0x50
> [ 9627.630100] [<ffffffff81185092>] ? kernfs_path+0x42/0x50
> [ 9627.630102] [<ffffffff811883da>] ? sysfs_warn_dup+0x5a/0x70
> [ 9627.630104] [<ffffffff8118846e>] ? sysfs_create_dir_ns+0x7e/0x90
> [ 9627.630108] [<ffffffff811d94ab>] ? kobject_add_internal+0x9b/0x2f0
> [ 9627.630109] [<ffffffff811d9af6>] ? kobject_add+0x66/0xb0
> [ 9627.630114] [<ffffffff812bb2e3>] ? device_add+0x263/0x620
> [ 9627.630116] [<ffffffff812bb8a8>] ? device_create_groups_vargs+0xe8/0x100
> [ 9627.630118] [<ffffffff812bb8d3>] ? device_create_vargs+0x13/0x20
> [ 9627.630124] [<ffffffff810ed128>] ? bdi_register+0x68/0x150
> [ 9627.630129] [<ffffffff811c535d>] ? add_disk+0x14d/0x4a0
> [ 9627.630132] [<ffffffff811c585f>] ? alloc_disk_node+0xaf/0x100
> [ 9627.630137] [<ffffffffa0252269>] ? md_alloc+0x1e9/0x350 [md_mod]
> [ 9627.630141] [<ffffffffa02523db>] ? md_probe+0xb/0x20 [md_mod]
> [ 9627.630143] [<ffffffff812c0654>] ? kobj_lookup+0x104/0x170
> [ 9627.630147] [<ffffffffa02523d0>] ? md_alloc+0x350/0x350 [md_mod]
> [ 9627.630149] [<ffffffff811c4da8>] ? get_gendisk+0x28/0xf0
> [ 9627.630153] [<ffffffff8115fb74>] ? __blkdev_get+0x114/0x3c0
> [ 9627.630156] [<ffffffff8115e590>] ? bdev_direct_access+0xa0/0xa0
> [ 9627.630158] [<ffffffff8115e5a0>] ? bdev_test+0x10/0x10
> [ 9627.630160] [<ffffffff8115fe58>] ? blkdev_get+0x38/0x310
> [ 9627.630162] [<ffffffff81160170>] ? blkdev_get_by_dev+0x40/0x40
> [ 9627.630167] [<ffffffff8112b3d3>] ? do_dentry_open.isra.16+0x153/0x320
> [ 9627.630170] [<ffffffff811380f3>] ? do_last.isra.51+0x323/0xd50
> [ 9627.630172] [<ffffffff8111f5b3>] ? kmem_cache_alloc+0x123/0x130
> [ 9627.630174] [<ffffffff8113a97f>] ? path_openat+0x7f/0x610
> [ 9627.630177] [<ffffffff810f7480>] ? tlb_flush_mmu_free+0x30/0x50
> [ 9627.630180] [<ffffffff810fe800>] ? unmap_region+0xb0/0xf0
> [ 9627.630182] [<ffffffff8113bb3b>] ? do_filp_open+0x2b/0x90
> [ 9627.630187] [<ffffffff811472ec>] ? __alloc_fd+0x7c/0x120
> [ 9627.630189] [<ffffffff8112c531>] ? do_sys_open+0x121/0x210
> [ 9627.630193] [<ffffffff813ea097>] ? system_call_fastpath+0x12/0x6a
> [ 9627.630195] ---[ end trace b7a3e9c6f05c2666 ]---
> [ 9627.630196] ------------[ cut here ]------------
> [ 9627.630198] WARNING: CPU: 18 PID: 3330 at lib/kobject.c:240 kobject_add_internal+0x274/0x2f0()
> [ 9627.630200] kobject_add_internal failed for 9:127 with -EEXIST, don't try to register things with the same name in the same directory.
>
Ditto, seems the same issue.
Thanks,
Guoqing
^ permalink raw reply
* Re: [PATCH 2/4] md-cluster: remove capabilities
From: gary @ 2015-04-15 2:24 UTC (permalink / raw)
To: Goldwyn Rodrigues, Guoqing Jiang; +Cc: neilb, linux-raid
In-Reply-To: <552CF96A.7000306@suse.de>
Hi Goldwyn,
>>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>>> index bc11551..0c65e51 100644
>>> --- a/drivers/md/md.c
>>> +++ b/drivers/md/md.c
>>> @@ -2291,11 +2291,12 @@ static void export_rdev(struct md_rdev * rdev)
>>> kobject_put(&rdev->kobj);
>>> }
>>>
>>> -static void kick_rdev_from_array(struct md_rdev * rdev)
>>> +void md_kick_rdev_from_array(struct md_rdev * rdev)
>>> {
>>> unbind_rdev_from_array(rdev);
>>> export_rdev(rdev);
>>> }
>>> +EXPORT_SYMBOL_GPL(md_kick_rdev_from_array);
>>>
>>> static void export_array(struct mddev *mddev)
>>> {
>>> @@ -2306,7 +2307,7 @@ static void export_array(struct mddev *mddev)
>>> MD_BUG();
>>> continue;
>>> }
>>> - kick_rdev_from_array(rdev);
>>> + md_kick_rdev_from_array(rdev);
>>> }
>>> if (!list_empty(&mddev->disks))
>>> MD_BUG();
>>> @@ -2750,9 +2751,11 @@ state_store(struct md_rdev *rdev, const char
>>> *buf, size_t len)
>>> err = -EBUSY;
>>> else {
>>> struct mddev *mddev = rdev->mddev;
>>> - if (mddev_is_clustered(mddev))
>>> + if (mddev_is_clustered(mddev)) {
>>> md_cluster_ops->metadata_update_start(mddev);
>>> - kick_rdev_from_array(rdev);
>>> + md_cluster_ops->remove_disk(mddev, rdev);
>>> + }
>>> + md_kick_rdev_from_array(rdev);
>>>
>> For md-cluster, seems it is possible that md_kick_rdev_from_array could
>> be called twice,
>> is this what you want? Thanks.
>
>
> No, it would be called only once. There are two types of nodes in this
> case: one is sender and other received. The sender calls
> md_kick_rdev_from_array() in the regular flow of
> state_store/hot_remove_disk() while the receiver calls
> md_kick_rdev_from_array() in the process_remove_disk().
>
Thanks for explanation, both sender node and receiver node need to kick
the disk from array. I misunderstood it, :(.
Regards,
Guoqing
^ permalink raw reply
* Re: Recovery Raid6 Degraded
From: John Stoffel @ 2015-04-14 20:27 UTC (permalink / raw)
To: Emanuel Domingos; +Cc: Mikael Abrahamsson, linux-raid
In-Reply-To: <CACsoFmiCoZGRoUWpxg_PLw6hyhW1mAB0TZFO94MZ6Kix-DxMOA@mail.gmail.com>
Make sure you add in a bitmap to your array, to make rebuilds quicker
after a system crash.
mdadm --grow /dev/md126 --bitmap internal
^ permalink raw reply
* Re: Recovery Raid6 Degraded
From: Emanuel Domingos @ 2015-04-14 18:06 UTC (permalink / raw)
To: Mikael Abrahamsson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1504141617220.16871@uplift.swm.pp.se>
Thanks! worked perfectly
Follow results
#mdadm --stop /dev/md/ubuntu\:md-raid6_primary
#mdadm --assemble --force /dev/md/ubuntu\:md-raid6_primary /dev/sdaa
/dev/sdac /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdz
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md124 : active raid6 sdf[0] sdz[5] sdj[8] sdi[3] sdh[2] sdg[1]
11721077760 blocks super 1.2 level 6, 512k chunk, algorithm 2
[8/6] [UUUUUU__]
md125 : active (auto-read-only) raid1 sdag[1]
97684688 blocks super 1.2 [2/1] [_U]
md126 : active raid6 sdad[1] sdv[2] sdy[5] sdw[3] sdx[4]
7814051840 blocks super 1.2 level 6, 512k chunk, algorithm 2
[6/5] [_UUUUU]
md127 : inactive sdd[13] sdc[15] sde[14] sda[10] sdb[11] sdt[4] sdr[2]
sdq[1] sdm[7] sdk[5] sdo[9] sdn[8]
23442162720 blocks super 1.2
unused devices: <none>
Adding two disks that were removed [8/6]
mdadm --manage /dev/md124 --add /dev/sdaa
mdadm --manage /dev/md124 --add /dev/sdac
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md124 : active raid6 sdac[10] sdaa[9] sdf[0] sdz[5] sdj[8] sdi[3] sdh[2] sdg[1]
11721077760 blocks super 1.2 level 6, 512k chunk, algorithm 2
[8/6] [UUUUUU__]
[>....................] recovery = 0.4% (8112408/1953512960)
finish=6234.6min speed=5200K/sec
md125 : active (auto-read-only) raid1 sdag[1]
97684688 blocks super 1.2 [2/1] [_U]
md126 : active raid6 sdad[1] sdv[2] sdy[5] sdw[3] sdx[4]
7814051840 blocks super 1.2 level 6, 512k chunk, algorithm 2
[6/5] [_UUUUU]
md127 : inactive sdd[13] sdc[15] sde[14] sda[10] sdb[11] sdt[4] sdr[2]
sdq[1] sdm[7] sdk[5] sdo[9] sdn[8]
23442162720 blocks super 1.2
unused devices: <none>
mdadm --detail /dev/md124
/dev/md124:
Version : 1.2
Creation Time : Thu Sep 27 11:39:20 2012
Raid Level : raid6
Array Size : 11721077760 (11178.09 GiB 12002.38 GB)
Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistent
Update Time : Tue Apr 14 14:37:56 2015
State : clean, degraded, recovering
Active Devices : 6
Working Devices : 8
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 512K
Rebuild Status : 0% complete
Name : ubuntu:md-raid6_primary (local to host ubuntu)
UUID : cee8d180:d6275a41:599064c6:8894819e
Events : 1093711
Number Major Minor RaidDevice State
0 8 80 0 active sync /dev/sdf
1 8 96 1 active sync /dev/sdg
2 8 112 2 active sync /dev/sdh
3 8 128 3 active sync /dev/sdi
8 8 144 4 active sync /dev/sdj
5 65 144 5 active sync /dev/sdz
9 65 160 6 spare rebuilding /dev/sdaa
10 65 192 7 spare rebuilding /dev/sdac
Just wait for the complete rebuild! Great work. Thanks
2015-04-14 11:19 GMT-03:00 Mikael Abrahamsson <swmike@swm.pp.se>:
> On Tue, 14 Apr 2015, Emanuel Domingos wrote:
>
>> Hi, Mikael, hi guys!
>>
>>
>> Follow the errors found after building the new version of mdadm:
>>
>> #mdadm --stop /dev/md/ubuntu\:md-raid6_primary
>>
>> #mdadm --assemble --force /dev/md/ubuntu\:md-raid6_primary /dev/sdaa
>> /dev/sdac
>> /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdz
>>
>> mdadm: device /dev/md/ubuntu:md-raid6_primary exists but is not an md
>> array.
>
>
> Please call it /dev/md124 and not the above ubuntu: name. I have no idea
> what it is you're trying to do with that name. Also use --verbose when
> trying to do the assembly and see if you get any further information.
>
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se
--
Emanuel Domingos
Cursando Bacharelado em Ciência da Computação - IFCE Campus Maracanaú
Técnico em Conectividade - IFCE Campus Maracanaú
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox