* Re: RAID5 Performance
From: Peter Grandi @ 2016-08-03 21:23 UTC (permalink / raw)
To: Linux RAID
In-Reply-To: <36cb010d-0b77-964e-fae5-5494e5f7766b@websitemanagers.com.au>
[ ... ]
> However, I can't get the budget for those really awesome
> drives up the top of the list, that would require around
> $20k... or more.
> For now, I've got 16 x 545s TB drives, and have replaced the
> first half (ie, all drives in one server). Now I can see that
> the drives themselves don't seem to be the bottleneck (the
> drives don't run at 100% util, while the DRBD device does run
> at 100%).
The "%util" number is not that easy to interpret, especially for
flash SSD and in some situations which probably include this
one:
https://brooker.co.za/blog/2014/07/04/iostat-pct.html
> Hopefully that will line up right !
It is hard to read, and I don't understand what the numbers are,
but it does not matter a lot.
> So I can only presume the new drives are much better than the
> 530 series, but still not as good as the 520 series.
The 540s have an SLC write buffer as I mentioned previously,
which should help.
> However, the point of note is that DRBD devices are showing
> high util levels much more frequently than the underlying
> devices, so I can only assume that the current limitation is
> caused by DRBD rather than the drives.
I like guessing, but this assumptions seems to me a bit
excessive.
> From my understanding, the times these settings can cause a
> problem: [ ... ]
If you don't have reliable sync barriers at all levels (not just
DRBD) *any* crash (e.g. bug crash, mistake-crash, memory-full
crash, not just power crash) is going to cause massive trouble,
especially in a mostly-write workload where what is being
written is cache spill. Some interesting pages:
http://blog.2ndquadrant.com/intel_ssd_now_off_the_sherr_sh/
http://wiki.postgresql.org/wiki/Reliable_Writes
http://archive.is/WTeAE
https://news.ycombinator.com/item?id=6973179
http://lkcl.net/reports/ssd_analysis.html
>>> Do you have any other suggestions or ideas that might
>>> assist?
Another one that would likely give a bit of relief as you can't
budget for write-optimized "enterprise" flash SSDs is a SATA/SAS
host adapter with a very large battery-backed RAM buffer. As the
tests that I previously mentioned show, longer writes result in
much improved write rates on "consumer" flash SSD devices, and
hopefully the large buffer results in:
#1 When the large write buffer flushes, *hopefully* much longer
writes to the flash SSD will happen on average.
#2 Thanks to the battery backing, writes are reporte completed
to the OS when they reach the host adapter's buffer, rather
than the flash SSD layer.
If #1 does not happen #2 won't help much if writes are at the
flash SSD saturation level, only if they are bursty and on
average below it.
>> * Smaller RAID5 stripes, as in 4+1 or 2+1, are cheaper in
>> space than RAID10 and enormously raise the chances that a
>> full stripe-write can happen (it still has the write-hole
>> problem of parity RAID).
> I was planning to upgrade to the 4.4.x kernel, which would
> kind of solve this, [ ... ]
The write hole workaround in MD RAID relies on a mostly-write
journal device like for DRBD.
>> * Make sure the DRBD journal is also on a separate device
>> that allows fast small sync writes.
> I think this would be the next option to investigate.
> Currently the DRBD journal is on the same devices.
That means that every sync'ed write becomes two writes to the
same device.
> 2 x P3700 400GB is probably around $2500,
The Samsung SM863 I ahve already mentioned are write-optimized
too and much cheaper, at around $300-350 for the 480GB model.
> while 12 x 545s 1000GB is around $4800, [ ... ]
Many people try to use "consumer" drives to build manager wowing
systems that have huge capacity and low cost, but vendors are not
stupid, and make sure that premium priced "enteprise" drives have
some critical advantage for at least some important workloads
(usually write heavy, guessing that "enterprise" workloads that
can command premium prices are transactional); sometimes like for
SSDs the advantages are based on real stuff, capacitors and
overprovisioning, which do cost money, sometimes artificial like
disabling SCT/ERC control.
^ permalink raw reply
* Re: Inactive arrays
From: Wols Lists @ 2016-08-03 21:31 UTC (permalink / raw)
To: Daniel Sanabria; +Cc: linux-raid
In-Reply-To: <CAHscji3CMO=ZREJypdUhsyKN0TmHWsEybdVLwWCbMZqWGdngOA@mail.gmail.com>
On 03/08/16 20:18, Daniel Sanabria wrote:
> Ok,
>
> Unfortunately the PSU replacement didn't help and I could be facing a
> failed motherboard/cpu :( My question now is, is it possible to
> restore the arrays in a new machine?
Your comment about the system starting up and shutting down almost
immediately implies to me a faulty on/off switch. It could be that
simple. Or of course, it could be worse. Like so many things nowadays,
on/off is controlled by software and if the switch is stuck ...
You might find all it needs is a new case. Or you might not.
But yes, if it's all linux software raid, then sticking the disks into a
new machine should work fine - the new machine should just come straight
up unless you're running an optimised kernel.
(That doesn't imply the arrays will :-( you might need a bit of fixing
for them, but if the machine boots successfully you're most of the way
there.)
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH 37/45] drivers: use req op accessor
From: Ross Zwisler @ 2016-08-03 22:33 UTC (permalink / raw)
To: Mike Christie, Ross Zwisler, Dave Chinner, axboe
Cc: linux-f2fs-devel, linux-ext4, konrad.wilk, drbd-dev,
philipp.reisner, lars.ellenberg, linux-raid, dm-devel,
linux-fsdevel, linux-bcache, linux-block, LKML, linux-scsi,
linux-mtd, target-devel, linux-btrfs, osd-dev, XFS Developers,
ocfs2-devel
In-Reply-To: <1465155145-10812-38-git-send-email-mchristi@redhat.com>
On Sun, Jun 5, 2016 at 1:32 PM, <mchristi@redhat.com> wrote:
> From: Mike Christie <mchristi@redhat.com>
>
> The req operation REQ_OP is separated from the rq_flag_bits
> definition. This converts the block layer drivers to
> use req_op to get the op from the request struct.
>
> Signed-off-by: Mike Christie <mchristi@redhat.com>
> ---
> drivers/block/loop.c | 6 +++---
> drivers/block/mtip32xx/mtip32xx.c | 2 +-
> drivers/block/nbd.c | 2 +-
> drivers/block/rbd.c | 4 ++--
> drivers/block/xen-blkfront.c | 8 +++++---
> drivers/ide/ide-floppy.c | 2 +-
> drivers/md/dm.c | 2 +-
> drivers/mmc/card/block.c | 7 +++----
> drivers/mmc/card/queue.c | 6 ++----
Dave Chinner reported a deadlock with XFS + DAX, which I reproduced
and bisected to this commit:
commit c2df40dfb8c015211ec55f4b1dd0587f875c7b34
Author: Mike Christie <mchristi@redhat.com>
Date: Sun Jun 5 14:32:17 2016 -0500
drivers: use req op accessor
Here are the steps to reproduce the deadlock with a BRD ramdisk:
mkfs.xfs -f /dev/ram0
mount -o dax /dev/ram0 /mnt/scratch
xfs_io -f -c "truncate 1g" /mnt/scratch/test.img
losetup -f --show /mnt/scratch/test.img
mkfs.xfs -f /dev/loop0
At this point the mkfs.xfs deadlocks. Here is the stack trace
gathered via "echo w > /proc/sysrq-trigger" and passed through
kasan_symbolize.py:
brd: module loaded
XFS (ram0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
XFS (ram0): Mounting V5 Filesystem
XFS (ram0): Ending clean mount
sysrq: SysRq : Show Blocked State
task PC stack pid father
mkfs.xfs D ffff88060ae47b38 0 1482 1287 0x00000000
ffff88060ae47b38 00000000000079e8 ffff880610fd8d98 ffff880036011a40
ffff8800aa6dcec0 ffff88060ae48000 ffff880610fd8d80 7fffffffffffffff
ffff8800aa6dcec0 00000000024000c0 ffff88060ae47b50 ffffffff81aca775
Call Trace:
[<ffffffff81aca775>] schedule+0x35/0x80 kernel/sched/core.c:3360
[<ffffffff81acf431>] schedule_timeout+0x271/0x460 kernel/time/timer.c:1493
[<ffffffff81ac9c34>] io_schedule_timeout+0xa4/0x110 kernel/sched/core.c:4969
[< inline >] do_wait_for_common kernel/sched/completion.c:75
[< inline >] __wait_for_common kernel/sched/completion.c:93
[< inline >] wait_for_common_io kernel/sched/completion.c:107
[<ffffffff81acb33f>] wait_for_completion_io+0xdf/0x120
kernel/sched/completion.c:155
[<ffffffff81573206>] submit_bio_wait+0x66/0x90 block/bio.c:870
[<ffffffff81588016>] blkdev_issue_discard+0x86/0xc0 block/blk-lib.c:115
[<ffffffff8158ea23>] blk_ioctl_discard+0xa3/0xd0 block/ioctl.c:221
[<ffffffff8158f5da>] blkdev_ioctl+0x60a/0x9e0 block/ioctl.c:510
[<ffffffff812bddb3>] block_ioctl+0x43/0x50 fs/block_dev.c:1714
[< inline >] vfs_ioctl fs/ioctl.c:43
[<ffffffff8128ec72>] do_vfs_ioctl+0xa2/0x6a0 fs/ioctl.c:674
[< inline >] SYSC_ioctl fs/ioctl.c:689
[<ffffffff8128f2e9>] SyS_ioctl+0x79/0x90 fs/ioctl.c:680
[<ffffffff81ad0abc>] entry_SYSCALL_64_fastpath+0x1f/0xbd
arch/x86/entry/entry_64.S:207
The line numbers are for the commit above, not for linux/master. This
occurs 100% as of this commit, and 0% with the previous commit.
This doesn't occur if you don't use DAX, but based on the content of
the commit I'm guessing that difference is due to variations in the
way the two paths use discard.
- Ross
^ permalink raw reply
* Re: [PATCH] raid5: fix incorrectly counter of conf->empty_inactive_list_nr
From: NeilBrown @ 2016-08-03 22:44 UTC (permalink / raw)
To: liuzhengyuan; +Cc: shli, liuzhengyuang521, linux-raid, linux-kernel
In-Reply-To: <tencent_6BA8E587713443DB1CBA0A47@qq.com>
[-- Attachment #1: Type: text/plain, Size: 328 bytes --]
On Wed, Aug 03 2016, liuzhengyuan wrote:
> Thanks for you replay.
> I think it may be on the temp inactive list. An active sh was handled
> and put to temp inactive list firstly, then moved to inactive list.
> If sh is on the temp inactive list, sh->count is zero too.
Hmmm... yes, you are probably right.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 37/45] drivers: use req op accessor
From: Mike Christie @ 2016-08-03 23:47 UTC (permalink / raw)
To: Ross Zwisler, Ross Zwisler, Dave Chinner, axboe
Cc: linux-bcache, linux-block, XFS Developers, ocfs2-devel,
linux-scsi, konrad.wilk, LKML, philipp.reisner, linux-f2fs-devel,
linux-raid, dm-devel, target-devel, linux-mtd, osd-dev,
linux-fsdevel, lars.ellenberg, linux-ext4, linux-btrfs, drbd-dev
In-Reply-To: <CAOxpaSUMsgfTYsL0bL6VCnwxz7bpgpKO-ZfC5J8dFG9PN-hzvg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1470 bytes --]
On 08/03/2016 05:33 PM, Ross Zwisler wrote:
> On Sun, Jun 5, 2016 at 1:32 PM, <mchristi@redhat.com> wrote:
>> From: Mike Christie <mchristi@redhat.com>
>>
>> The req operation REQ_OP is separated from the rq_flag_bits
>> definition. This converts the block layer drivers to
>> use req_op to get the op from the request struct.
>>
>> Signed-off-by: Mike Christie <mchristi@redhat.com>
>> ---
>> drivers/block/loop.c | 6 +++---
>> drivers/block/mtip32xx/mtip32xx.c | 2 +-
>> drivers/block/nbd.c | 2 +-
>> drivers/block/rbd.c | 4 ++--
>> drivers/block/xen-blkfront.c | 8 +++++---
>> drivers/ide/ide-floppy.c | 2 +-
>> drivers/md/dm.c | 2 +-
>> drivers/mmc/card/block.c | 7 +++----
>> drivers/mmc/card/queue.c | 6 ++----
>
> Dave Chinner reported a deadlock with XFS + DAX, which I reproduced
> and bisected to this commit:
>
> commit c2df40dfb8c015211ec55f4b1dd0587f875c7b34
> Author: Mike Christie <mchristi@redhat.com>
> Date: Sun Jun 5 14:32:17 2016 -0500
> drivers: use req op accessor
>
> Here are the steps to reproduce the deadlock with a BRD ramdisk:
>
> mkfs.xfs -f /dev/ram0
> mount -o dax /dev/ram0 /mnt/scratch
When using ramdisks, we need the attached patch like in your other bug
report. I think it will fix some hangs people are seeing.
I do not think that it should cause the failure to run issue you saw
when doing generic/008 and ext2.
[-- Attachment #2: convert-rw_page.patch --]
[-- Type: text/x-patch, Size: 12108 bytes --]
diff --git a/drivers/block/brd.c b/drivers/block/brd.c
index 3022dad..9fbbeba 100644
--- a/drivers/block/brd.c
+++ b/drivers/block/brd.c
@@ -300,20 +300,20 @@ static void copy_from_brd(void *dst, struct brd_device *brd,
* Process a single bvec of a bio.
*/
static int brd_do_bvec(struct brd_device *brd, struct page *page,
- unsigned int len, unsigned int off, int rw,
+ unsigned int len, unsigned int off, int op,
sector_t sector)
{
void *mem;
int err = 0;
- if (rw != READ) {
+ if (op_is_write(op)) {
err = copy_to_brd_setup(brd, sector, len);
if (err)
goto out;
}
mem = kmap_atomic(page);
- if (rw == READ) {
+ if (!op_is_write(op)) {
copy_from_brd(mem + off, brd, sector, len);
flush_dcache_page(page);
} else {
@@ -330,7 +330,6 @@ static blk_qc_t brd_make_request(struct request_queue *q, struct bio *bio)
{
struct block_device *bdev = bio->bi_bdev;
struct brd_device *brd = bdev->bd_disk->private_data;
- int rw;
struct bio_vec bvec;
sector_t sector;
struct bvec_iter iter;
@@ -347,14 +346,12 @@ static blk_qc_t brd_make_request(struct request_queue *q, struct bio *bio)
goto out;
}
- rw = bio_data_dir(bio);
-
bio_for_each_segment(bvec, bio, iter) {
unsigned int len = bvec.bv_len;
int err;
err = brd_do_bvec(brd, bvec.bv_page, len,
- bvec.bv_offset, rw, sector);
+ bvec.bv_offset, bio_op(bio), sector);
if (err)
goto io_error;
sector += len >> SECTOR_SHIFT;
@@ -369,11 +366,11 @@ io_error:
}
static int brd_rw_page(struct block_device *bdev, sector_t sector,
- struct page *page, int rw)
+ struct page *page, int op, int op_flags)
{
struct brd_device *brd = bdev->bd_disk->private_data;
- int err = brd_do_bvec(brd, page, PAGE_SIZE, 0, rw, sector);
- page_endio(page, rw & WRITE, err);
+ int err = brd_do_bvec(brd, page, PAGE_SIZE, 0, op, sector);
+ page_endio(page, op, err);
return err;
}
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 7454cf1..f0e126c 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -843,15 +843,15 @@ static void zram_bio_discard(struct zram *zram, u32 index,
}
static int zram_bvec_rw(struct zram *zram, struct bio_vec *bvec, u32 index,
- int offset, int rw)
+ int offset, int op)
{
unsigned long start_time = jiffies;
int ret;
- generic_start_io_acct(rw, bvec->bv_len >> SECTOR_SHIFT,
+ generic_start_io_acct(op, bvec->bv_len >> SECTOR_SHIFT,
&zram->disk->part0);
- if (rw == READ) {
+ if (!op_is_write(op)) {
atomic64_inc(&zram->stats.num_reads);
ret = zram_bvec_read(zram, bvec, index, offset);
} else {
@@ -859,10 +859,10 @@ static int zram_bvec_rw(struct zram *zram, struct bio_vec *bvec, u32 index,
ret = zram_bvec_write(zram, bvec, index, offset);
}
- generic_end_io_acct(rw, &zram->disk->part0, start_time);
+ generic_end_io_acct(op, &zram->disk->part0, start_time);
if (unlikely(ret)) {
- if (rw == READ)
+ if (!op_is_write(op))
atomic64_inc(&zram->stats.failed_reads);
else
atomic64_inc(&zram->stats.failed_writes);
@@ -873,7 +873,7 @@ static int zram_bvec_rw(struct zram *zram, struct bio_vec *bvec, u32 index,
static void __zram_make_request(struct zram *zram, struct bio *bio)
{
- int offset, rw;
+ int offset;
u32 index;
struct bio_vec bvec;
struct bvec_iter iter;
@@ -888,7 +888,6 @@ static void __zram_make_request(struct zram *zram, struct bio *bio)
return;
}
- rw = bio_data_dir(bio);
bio_for_each_segment(bvec, bio, iter) {
int max_transfer_size = PAGE_SIZE - offset;
@@ -903,15 +902,18 @@ static void __zram_make_request(struct zram *zram, struct bio *bio)
bv.bv_len = max_transfer_size;
bv.bv_offset = bvec.bv_offset;
- if (zram_bvec_rw(zram, &bv, index, offset, rw) < 0)
+ if (zram_bvec_rw(zram, &bv, index, offset,
+ bio_op(bio)) < 0)
goto out;
bv.bv_len = bvec.bv_len - max_transfer_size;
bv.bv_offset += max_transfer_size;
- if (zram_bvec_rw(zram, &bv, index + 1, 0, rw) < 0)
+ if (zram_bvec_rw(zram, &bv, index + 1, 0,
+ bio_op(bio)) < 0)
goto out;
} else
- if (zram_bvec_rw(zram, &bvec, index, offset, rw) < 0)
+ if (zram_bvec_rw(zram, &bvec, index, offset,
+ bio_op(bio)) < 0)
goto out;
update_position(&index, &offset, &bvec);
@@ -968,7 +970,7 @@ static void zram_slot_free_notify(struct block_device *bdev,
}
static int zram_rw_page(struct block_device *bdev, sector_t sector,
- struct page *page, int rw)
+ struct page *page, int op, int op_flags)
{
int offset, err = -EIO;
u32 index;
@@ -992,7 +994,7 @@ static int zram_rw_page(struct block_device *bdev, sector_t sector,
bv.bv_len = PAGE_SIZE;
bv.bv_offset = 0;
- err = zram_bvec_rw(zram, &bv, index, offset, rw);
+ err = zram_bvec_rw(zram, &bv, index, offset, op);
put_zram:
zram_meta_put(zram);
out:
@@ -1005,7 +1007,7 @@ out:
* (e.g., SetPageError, set_page_dirty and extra works).
*/
if (err == 0)
- page_endio(page, rw, 0);
+ page_endio(page, op, 0);
return err;
}
diff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c
index 9dce03f..6a6208d 100644
--- a/drivers/nvdimm/btt.c
+++ b/drivers/nvdimm/btt.c
@@ -1133,11 +1133,11 @@ static int btt_write_pg(struct btt *btt, struct bio_integrity_payload *bip,
static int btt_do_bvec(struct btt *btt, struct bio_integrity_payload *bip,
struct page *page, unsigned int len, unsigned int off,
- int rw, sector_t sector)
+ int op, sector_t sector)
{
int ret;
- if (rw == READ) {
+ if (!op_is_write(op)) {
ret = btt_read_pg(btt, bip, page, off, sector, len);
flush_dcache_page(page);
} else {
@@ -1155,7 +1155,7 @@ static blk_qc_t btt_make_request(struct request_queue *q, struct bio *bio)
struct bvec_iter iter;
unsigned long start;
struct bio_vec bvec;
- int err = 0, rw;
+ int err = 0;
bool do_acct;
/*
@@ -1170,7 +1170,6 @@ static blk_qc_t btt_make_request(struct request_queue *q, struct bio *bio)
}
do_acct = nd_iostat_start(bio, &start);
- rw = bio_data_dir(bio);
bio_for_each_segment(bvec, bio, iter) {
unsigned int len = bvec.bv_len;
@@ -1181,11 +1180,12 @@ static blk_qc_t btt_make_request(struct request_queue *q, struct bio *bio)
BUG_ON(len % btt->sector_size);
err = btt_do_bvec(btt, bip, bvec.bv_page, len, bvec.bv_offset,
- rw, iter.bi_sector);
+ bio_op(bio), iter.bi_sector);
if (err) {
dev_info(&btt->nd_btt->dev,
"io error in %s sector %lld, len %d,\n",
- (rw == READ) ? "READ" : "WRITE",
+ (op_is_write(bio_op(bio))) ? "WRITE" :
+ "READ",
(unsigned long long) iter.bi_sector, len);
bio->bi_error = err;
break;
@@ -1200,12 +1200,12 @@ out:
}
static int btt_rw_page(struct block_device *bdev, sector_t sector,
- struct page *page, int rw)
+ struct page *page, int op, int op_flags)
{
struct btt *btt = bdev->bd_disk->private_data;
- btt_do_bvec(btt, NULL, page, PAGE_SIZE, 0, rw, sector);
- page_endio(page, rw & WRITE, 0);
+ btt_do_bvec(btt, NULL, page, PAGE_SIZE, 0, op, sector);
+ page_endio(page, op, 0);
return 0;
}
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index b511099..6a7b97d 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -67,7 +67,7 @@ static void pmem_clear_poison(struct pmem_device *pmem, phys_addr_t offset,
}
static int pmem_do_bvec(struct pmem_device *pmem, struct page *page,
- unsigned int len, unsigned int off, int rw,
+ unsigned int len, unsigned int off, int op,
sector_t sector)
{
int rc = 0;
@@ -79,7 +79,7 @@ static int pmem_do_bvec(struct pmem_device *pmem, struct page *page,
if (unlikely(is_bad_pmem(&pmem->bb, sector, len)))
bad_pmem = true;
- if (rw == READ) {
+ if (!op_is_write(op)) {
if (unlikely(bad_pmem))
rc = -EIO;
else {
@@ -134,7 +134,7 @@ static blk_qc_t pmem_make_request(struct request_queue *q, struct bio *bio)
do_acct = nd_iostat_start(bio, &start);
bio_for_each_segment(bvec, bio, iter) {
rc = pmem_do_bvec(pmem, bvec.bv_page, bvec.bv_len,
- bvec.bv_offset, bio_data_dir(bio),
+ bvec.bv_offset, bio_op(bio),
iter.bi_sector);
if (rc) {
bio->bi_error = rc;
@@ -152,12 +152,12 @@ static blk_qc_t pmem_make_request(struct request_queue *q, struct bio *bio)
}
static int pmem_rw_page(struct block_device *bdev, sector_t sector,
- struct page *page, int rw)
+ struct page *page, int op, int op_flags)
{
struct pmem_device *pmem = bdev->bd_queue->queuedata;
int rc;
- rc = pmem_do_bvec(pmem, page, PAGE_SIZE, 0, rw, sector);
+ rc = pmem_do_bvec(pmem, page, PAGE_SIZE, 0, op, sector);
/*
* The ->rw_page interface is subtle and tricky. The core
@@ -166,7 +166,7 @@ static int pmem_rw_page(struct block_device *bdev, sector_t sector,
* caused by double completion.
*/
if (rc == 0)
- page_endio(page, rw & WRITE, 0);
+ page_endio(page, op, 0);
return rc;
}
diff --git a/fs/block_dev.c b/fs/block_dev.c
index ada42cf..e790ced 100644
--- a/fs/block_dev.c
+++ b/fs/block_dev.c
@@ -416,7 +416,8 @@ int bdev_read_page(struct block_device *bdev, sector_t sector,
result = blk_queue_enter(bdev->bd_queue, false);
if (result)
return result;
- result = ops->rw_page(bdev, sector + get_start_sect(bdev), page, READ);
+ result = ops->rw_page(bdev, sector + get_start_sect(bdev), page,
+ REQ_OP_READ, 0);
blk_queue_exit(bdev->bd_queue);
return result;
}
@@ -445,7 +446,7 @@ int bdev_write_page(struct block_device *bdev, sector_t sector,
struct page *page, struct writeback_control *wbc)
{
int result;
- int rw = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : WRITE;
+ int op_flags = (wbc->sync_mode == WB_SYNC_ALL) ? WRITE_SYNC : 0;
const struct block_device_operations *ops = bdev->bd_disk->fops;
if (!ops->rw_page || bdev_get_integrity(bdev))
@@ -455,7 +456,8 @@ int bdev_write_page(struct block_device *bdev, sector_t sector,
return result;
set_page_writeback(page);
- result = ops->rw_page(bdev, sector + get_start_sect(bdev), page, rw);
+ result = ops->rw_page(bdev, sector + get_start_sect(bdev), page,
+ REQ_OP_WRITE, op_flags);
if (result)
end_page_writeback(page);
else
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index adf3307..3652408 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1673,7 +1673,8 @@ struct blk_dax_ctl {
struct block_device_operations {
int (*open) (struct block_device *, fmode_t);
void (*release) (struct gendisk *, fmode_t);
- int (*rw_page)(struct block_device *, sector_t, struct page *, int rw);
+ int (*rw_page)(struct block_device *, sector_t, struct page *,
+ int op, int op_flags);
int (*ioctl) (struct block_device *, fmode_t, unsigned, unsigned long);
int (*compat_ioctl) (struct block_device *, fmode_t, unsigned, unsigned long);
long (*direct_access)(struct block_device *, sector_t, void **, pfn_t *,
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 81363b8..4578637 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -510,7 +510,7 @@ static inline void wait_on_page_writeback(struct page *page)
extern void end_page_writeback(struct page *page);
void wait_for_stable_page(struct page *page);
-void page_endio(struct page *page, int rw, int err);
+void page_endio(struct page *page, int op, int err);
/*
* Add an arbitrary waiter to a page's wait queue
diff --git a/mm/filemap.c b/mm/filemap.c
index 3083ded..daef091 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -887,9 +887,9 @@ EXPORT_SYMBOL(end_page_writeback);
* After completing I/O on a page, call this routine to update the page
* flags appropriately
*/
-void page_endio(struct page *page, int rw, int err)
+void page_endio(struct page *page, int op, int err)
{
- if (rw == READ) {
+ if (!op_is_write(op)) {
if (!err) {
SetPageUptodate(page);
} else {
@@ -897,7 +897,7 @@ void page_endio(struct page *page, int rw, int err)
SetPageError(page);
}
unlock_page(page);
- } else { /* rw == WRITE */
+ } else {
if (err) {
SetPageError(page);
if (page->mapping)
[-- Attachment #3: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related
* Re: [PATCH 37/45] drivers: use req op accessor
From: Shaun Tancheff @ 2016-08-04 0:30 UTC (permalink / raw)
To: Mike Christie
Cc: philipp.reisner, dm-devel, target-devel, linux-mtd, drbd-dev,
linux-scsi, Ross Zwisler, linux-ext4, konrad.wilk, XFS Developers,
linux-raid, linux-bcache, linux-block, osd-dev, Ross Zwisler,
Jens Axboe, LKML, linux-f2fs-devel, ocfs2-devel, linux-fsdevel,
lars.ellenberg, linux-btrfs
In-Reply-To: <87fbe31f-27b3-ff7a-e381-34fc1aad337e@redhat.com>
On Wed, Aug 3, 2016 at 6:47 PM, Mike Christie <mchristi@redhat.com> wrote:
> On 08/03/2016 05:33 PM, Ross Zwisler wrote:
>> On Sun, Jun 5, 2016 at 1:32 PM, <mchristi@redhat.com> wrote:
>>> From: Mike Christie <mchristi@redhat.com>
>>>
>>> The req operation REQ_OP is separated from the rq_flag_bits
>>> definition. This converts the block layer drivers to
>>> use req_op to get the op from the request struct.
>>>
>>> Signed-off-by: Mike Christie <mchristi@redhat.com>
>>> ---
>>> drivers/block/loop.c | 6 +++---
>>> drivers/block/mtip32xx/mtip32xx.c | 2 +-
>>> drivers/block/nbd.c | 2 +-
>>> drivers/block/rbd.c | 4 ++--
>>> drivers/block/xen-blkfront.c | 8 +++++---
>>> drivers/ide/ide-floppy.c | 2 +-
>>> drivers/md/dm.c | 2 +-
>>> drivers/mmc/card/block.c | 7 +++----
>>> drivers/mmc/card/queue.c | 6 ++----
>>
>> Dave Chinner reported a deadlock with XFS + DAX, which I reproduced
>> and bisected to this commit:
>>
>> commit c2df40dfb8c015211ec55f4b1dd0587f875c7b34
>> Author: Mike Christie <mchristi@redhat.com>
>> Date: Sun Jun 5 14:32:17 2016 -0500
>> drivers: use req op accessor
>>
>> Here are the steps to reproduce the deadlock with a BRD ramdisk:
>>
>> mkfs.xfs -f /dev/ram0
>> mount -o dax /dev/ram0 /mnt/scratch
>
> When using ramdisks, we need the attached patch like in your other bug
> report. I think it will fix some hangs people are seeing.
>
> I do not think that it should cause the failure to run issue you saw
> when doing generic/008 and ext2.
>
I think the translation in loop.c is suspicious here:
"if use DIO && not (a flush_flag or discard_flag)"
should translate to:
"if use DIO && not ((a flush_flag) || op == discard)"
But in the patch I read:
"if use DIO && ((not a flush_flag) || op == discard)
Which would have DIO && discards follow the AIO path?
So I would humbly suggest something like the following
(on top of commit c2df40dfb8c015211ec55f4b1dd0587f875c7b34):
[Please excuse the messed up patch format ... gmail eats tabs]
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index b9b737c..0754d83 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1659,8 +1659,9 @@ static int loop_queue_rq(struct blk_mq_hw_ctx *hctx,
if (lo->lo_state != Lo_bound)
return -EIO;
- if (lo->use_dio && (!(cmd->rq->cmd_flags & REQ_FLUSH) ||
- req_op(cmd->rq) == REQ_OP_DISCARD))
+ if (lo->use_dio && !(
+ (cmd->rq->cmd_flags & REQ_FLUSH) ||
+ req_op(cmd->rq) == REQ_OP_DISCARD))
cmd->use_aio = true;
else
cmd->use_aio = false;
--
Shaun Tancheff
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related
* Re: [PATCH 1/3] MD: hold mddev lock for .quiesce in md_do_sync
From: NeilBrown @ 2016-08-04 3:16 UTC (permalink / raw)
To: shli, linux-raid; +Cc: Shaohua Li
In-Reply-To: <87y44epwb5.fsf@notabene.neil.brown.name>
[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]
On Wed, Aug 03 2016, NeilBrown wrote:
> [ Unknown signature status ]
> On Sun, Jul 31 2016, shli@kernel.org wrote:
>
>> From: Shaohua Li <shli@fb.com>
>>
>> .quiesce is called with mddev lock hold at most places. There are few
>> exceptions. Calling .quesce without the lock hold could create races. For
>> example, the .quesce of raid1 can't be recursively. The purpose of the patches
>> is to fix a race in raid5-cache. The raid5-cache .quesce will write md
>> superblock and should be called with mddev lock hold.
>>
>> Cc: NeilBrown <neilb@suse.com>
>> Signed-off-by: Shaohua Li <shli@fb.com>
>
> Acked-by: NeilBrown <neilb@suse.com>
>
> This should be safe but I'm not sure I really like it.
> The raid1 quiesce could be changed so that it can be called recursively.
> The raid5-cache situation would be harder to get right and maybe this is
> the best solution... It's just that 'quiesce' should be a fairly
> light-weight operation, just waiting for pending requests to flush. It
> shouldn't really *need* a lock.
Actually, the more I think about this, the less I like it.
I would much rather make .quiesce lighter weight so that no locking was
needed.
For r5l_quiesce, that probable means removed the "r5l_do_reclaim()".
Stopping and restarting the reclaim thread seems reasonable, but calling
r5l_do_reclaim() should not be needed. It should be done periodically
by the thread, and at 'stop' time, but otherwise isn't needed.
You would need to hold some mutex while calling md_register_thread, but
that could be probably be log->io_mutex, or maybe even some other new
mutex
Could you explore following that path instead?
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH 37/45] drivers: use req op accessor
From: Mike Christie @ 2016-08-04 5:47 UTC (permalink / raw)
To: Shaun Tancheff
Cc: Ross Zwisler, Ross Zwisler, Dave Chinner, Jens Axboe,
linux-f2fs-devel, linux-ext4, konrad.wilk, drbd-dev,
philipp.reisner, lars.ellenberg, linux-raid, dm-devel,
linux-fsdevel, linux-bcache, linux-block, LKML, linux-scsi,
linux-mtd, target-devel, linux-btrfs, osd-dev, XFS Developers,
ocfs2-devel
In-Reply-To: <CAJVOszAd9Eu88P2=QoNVArd0jwDH7KRe=OB82tCNMpjfv9B4KA@mail.gmail.com>
On 08/03/2016 07:30 PM, Shaun Tancheff wrote:
> On Wed, Aug 3, 2016 at 6:47 PM, Mike Christie <mchristi@redhat.com> wrote:
>> On 08/03/2016 05:33 PM, Ross Zwisler wrote:
>>> On Sun, Jun 5, 2016 at 1:32 PM, <mchristi@redhat.com> wrote:
>>>> From: Mike Christie <mchristi@redhat.com>
>>>>
>>>> The req operation REQ_OP is separated from the rq_flag_bits
>>>> definition. This converts the block layer drivers to
>>>> use req_op to get the op from the request struct.
>>>>
>>>> Signed-off-by: Mike Christie <mchristi@redhat.com>
>>>> ---
>>>> drivers/block/loop.c | 6 +++---
>>>> drivers/block/mtip32xx/mtip32xx.c | 2 +-
>>>> drivers/block/nbd.c | 2 +-
>>>> drivers/block/rbd.c | 4 ++--
>>>> drivers/block/xen-blkfront.c | 8 +++++---
>>>> drivers/ide/ide-floppy.c | 2 +-
>>>> drivers/md/dm.c | 2 +-
>>>> drivers/mmc/card/block.c | 7 +++----
>>>> drivers/mmc/card/queue.c | 6 ++----
>>>
>>> Dave Chinner reported a deadlock with XFS + DAX, which I reproduced
>>> and bisected to this commit:
>>>
>>> commit c2df40dfb8c015211ec55f4b1dd0587f875c7b34
>>> Author: Mike Christie <mchristi@redhat.com>
>>> Date: Sun Jun 5 14:32:17 2016 -0500
>>> drivers: use req op accessor
>>>
>>> Here are the steps to reproduce the deadlock with a BRD ramdisk:
>>>
>>> mkfs.xfs -f /dev/ram0
>>> mount -o dax /dev/ram0 /mnt/scratch
>>
>> When using ramdisks, we need the attached patch like in your other bug
>> report. I think it will fix some hangs people are seeing.
>>
>> I do not think that it should cause the failure to run issue you saw
>> when doing generic/008 and ext2.
>>
>
> I think the translation in loop.c is suspicious here:
>
> "if use DIO && not (a flush_flag or discard_flag)"
> should translate to:
> "if use DIO && not ((a flush_flag) || op == discard)"
>
> But in the patch I read:
> "if use DIO && ((not a flush_flag) || op == discard)
>
> Which would have DIO && discards follow the AIO path?
>
> So I would humbly suggest something like the following
> (on top of commit c2df40dfb8c015211ec55f4b1dd0587f875c7b34):
> [Please excuse the messed up patch format ... gmail eats tabs]
>
> diff --git a/drivers/block/loop.c b/drivers/block/loop.c
> index b9b737c..0754d83 100644
> --- a/drivers/block/loop.c
> +++ b/drivers/block/loop.c
> @@ -1659,8 +1659,9 @@ static int loop_queue_rq(struct blk_mq_hw_ctx *hctx,
> if (lo->lo_state != Lo_bound)
> return -EIO;
>
> - if (lo->use_dio && (!(cmd->rq->cmd_flags & REQ_FLUSH) ||
> - req_op(cmd->rq) == REQ_OP_DISCARD))
> + if (lo->use_dio && !(
> + (cmd->rq->cmd_flags & REQ_FLUSH) ||
> + req_op(cmd->rq) == REQ_OP_DISCARD))
> cmd->use_aio = true;
> else
> cmd->use_aio = false;
>
You are right. The translation was bad and your code above is correct.
I think we need my patch in the other mail though too, because for the
rw_page user case if WB_SYNC_ALL is set, then the IO gets sent down as a
read instead of a write.
^ permalink raw reply
* journal disk become the first of --detail list after fail/remove one disk
From: Yi Zhang @ 2016-08-04 10:52 UTC (permalink / raw)
To: linux-raid; +Cc: songliubraving, shli, Jes.Sorensen
In-Reply-To: <1597878292.11264554.1470307437177.JavaMail.zimbra@redhat.com>
Hi folks
patch[1] has move journal to end of --detail list, but I found after fail/remove the one of the disk,
the journal disk change to be the fisrt one, detailed steps pls refer[2]
[1]
commit 6fe4c6160399c94a15eda6092371e0ac9a38a4bc
Author: Song Liu <songliubraving@fb.com>
Date: Mon Dec 21 11:23:41 2015 -0800
move journal to end of --detail list
[2]
Reproduce steps
# uname -r
4.7.0
# mdadm --version
mdadm - v3.4-63-g52209d6 - 21st July 2016
# mdadm --create --run /dev/md0 --level 4 --metadata 1.2 --raid-devices 7 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 --write-journal /dev/loop0 --bitmap=internal --bitmap-chunk=64M --chunk 512
mdadm: array /dev/md0 started.
# mdadm --wait /dev/md0
# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Aug 4 18:42:19 2016
Raid Level : raid4
Array Size : 3139584 (2.99 GiB 3.21 GB)
Used Dev Size : 523264 (511.00 MiB 535.82 MB)
Raid Devices : 7
Total Devices : 8
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Aug 4 18:42:34 2016
State : clean
Active Devices : 7
Working Devices : 8
Failed Devices : 0
Spare Devices : 1
Chunk Size : 512K
Name : dhcp-12-125.nay.redhat.com:0 (local to host dhcp-12-125.nay.redhat.com)
UUID : 29a267c2:04e91998:639e6eef:64ddbab0
Events : 20
Number Major Minor RaidDevice State
0 7 1 0 active sync /dev/loop1
1 7 2 1 active sync /dev/loop2
2 7 3 2 active sync /dev/loop3
3 7 4 3 active sync /dev/loop4
4 7 5 4 active sync /dev/loop5
5 7 6 5 active sync /dev/loop6
8 7 7 6 active sync /dev/loop7
7 7 0 - journal /dev/loop0
# mdadm /dev/md0 -f /dev/loop1
mdadm: set /dev/loop1 faulty in /dev/md0
# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Aug 4 18:42:19 2016
Raid Level : raid4
Array Size : 3139584 (2.99 GiB 3.21 GB)
Used Dev Size : 523264 (511.00 MiB 535.82 MB)
Raid Devices : 7
Total Devices : 8
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Aug 4 18:43:07 2016
State : clean, degraded
Active Devices : 6
Working Devices : 7
Failed Devices : 1
Spare Devices : 1
Chunk Size : 512K
Name : dhcp-12-125.nay.redhat.com:0 (local to host dhcp-12-125.nay.redhat.com)
UUID : 29a267c2:04e91998:639e6eef:64ddbab0
Events : 22
Number Major Minor RaidDevice State
7 7 0 - journal /dev/loop0
1 7 2 1 active sync /dev/loop2
2 7 3 2 active sync /dev/loop3
3 7 4 3 active sync /dev/loop4
4 7 5 4 active sync /dev/loop5
5 7 6 5 active sync /dev/loop6
8 7 7 6 active sync /dev/loop7
0 7 1 - faulty /dev/loop1
# mdadm /dev/md0 -r /dev/loop1
mdadm: hot removed /dev/loop1 from /dev/md0
# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Aug 4 18:42:19 2016
Raid Level : raid4
Array Size : 3139584 (2.99 GiB 3.21 GB)
Used Dev Size : 523264 (511.00 MiB 535.82 MB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Thu Aug 4 18:43:22 2016
State : clean, degraded
Active Devices : 6
Working Devices : 7
Failed Devices : 0
Spare Devices : 1
Chunk Size : 512K
Name : dhcp-12-125.nay.redhat.com:0 (local to host dhcp-12-125.nay.redhat.com)
UUID : 29a267c2:04e91998:639e6eef:64ddbab0
Events : 23
Number Major Minor RaidDevice State
7 7 0 - journal /dev/loop0
1 7 2 1 active sync /dev/loop2
2 7 3 2 active sync /dev/loop3
3 7 4 3 active sync /dev/loop4
4 7 5 4 active sync /dev/loop5
5 7 6 5 active sync /dev/loop6
8 7 7 6 active sync /dev/loop7
Best Regards,
Yi Zhang
^ permalink raw reply
* No '--add-journal' helper page in "man mdadm" and "mdadm --manage --help"
From: Yi Zhang @ 2016-08-04 11:19 UTC (permalink / raw)
To: linux-raid; +Cc: songliubraving, shli, Jes.Sorensen
In-Reply-To: <1320987791.11266399.1470308772076.JavaMail.zimbra@redhat.com>
Hello shaohua/Song
Since write-journal has been integrated in mdadm/kernel, I found still no --add-journal usage on related pages, do you plan to add it?
Best Regards,
Yi Zhang
^ permalink raw reply
* Re: journal disk become the first of --detail list after fail/remove one disk
From: Adam Goryachev @ 2016-08-04 11:34 UTC (permalink / raw)
To: Yi Zhang, linux-raid; +Cc: songliubraving, shli, Jes.Sorensen
In-Reply-To: <1933083404.11265226.1470307932889.JavaMail.zimbra@redhat.com>
On 4/08/2016 20:52, Yi Zhang wrote:
> # mdadm -D /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Thu Aug 4 18:42:19 2016
> Raid Level : raid4
> Array Size : 3139584 (2.99 GiB 3.21 GB)
> Used Dev Size : 523264 (511.00 MiB 535.82 MB)
> Raid Devices : 7
> Total Devices : 7
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Thu Aug 4 18:43:22 2016
> State : clean, degraded
> Active Devices : 6
> Working Devices : 7
> Failed Devices : 0
> Spare Devices : 1
>
> Chunk Size : 512K
>
> Name : dhcp-12-125.nay.redhat.com:0 (local to host dhcp-12-125.nay.redhat.com)
> UUID : 29a267c2:04e91998:639e6eef:64ddbab0
> Events : 23
>
> Number Major Minor RaidDevice State
> 7 7 0 - journal /dev/loop0
> 1 7 2 1 active sync /dev/loop2
> 2 7 3 2 active sync /dev/loop3
> 3 7 4 3 active sync /dev/loop4
> 4 7 5 4 active sync /dev/loop5
> 5 7 6 5 active sync /dev/loop6
> 8 7 7 6 active sync /dev/loop7
>
I'm not really sure about the journal/your report, but also why does the
above show 1 spare device? I would expect spare should mean that there
is a device available to take over automatically if one of the active
devices becomes faulty, but that doesn't seem to be the case. Should a
journal device be counted as Active & Working (or faulty) instead of Spare?
Regards,
Adam
^ permalink raw reply
* Re: [PATCH 37/45] drivers: use req op accessor
From: Christoph Hellwig @ 2016-08-04 15:46 UTC (permalink / raw)
To: Shaun Tancheff
Cc: philipp.reisner, dm-devel, target-devel, linux-mtd, drbd-dev,
linux-scsi, Mike Christie, Ross Zwisler, linux-ext4, konrad.wilk,
XFS Developers, linux-block, linux-bcache, osd-dev, Ross Zwisler,
Jens Axboe, linux-raid, LKML, linux-f2fs-devel, linux-btrfs,
linux-fsdevel, lars.ellenberg, ocfs2-devel
In-Reply-To: <CAJVOszAd9Eu88P2=QoNVArd0jwDH7KRe=OB82tCNMpjfv9B4KA@mail.gmail.com>
On Wed, Aug 03, 2016 at 07:30:29PM -0500, Shaun Tancheff wrote:
> I think the translation in loop.c is suspicious here:
>
> "if use DIO && not (a flush_flag or discard_flag)"
> should translate to:
> "if use DIO && not ((a flush_flag) || op == discard)"
>
> But in the patch I read:
> "if use DIO && ((not a flush_flag) || op == discard)
>
> Which would have DIO && discards follow the AIO path?
Indeed. Sorry for missing out on your patch, I just sent a fix
in reply to Dave's other report earlier which is pretty similar to
yours.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply
* INFO: task systemd-udevd:794 blocked for more than 600 seconds observed after mdadm -G operation
From: Yi Zhang @ 2016-08-04 16:01 UTC (permalink / raw)
To: linux-raid; +Cc: shli, NeilBrown
In-Reply-To: <2047059297.11279583.1470314510395.JavaMail.zimbra@redhat.com>
Hello folks
Seems I hit one issue that task blocked issue on 4.7.0 during write journal testing, could anyone help check it?
I have list the log and reproduce steps below, thanks.
Log:
[ 106.688135] md: bind<sdb1>
[ 106.688301] md: bind<sdc1>
[ 106.689831] md: bind<sdd1>
[ 106.689897] md: bind<sdf1>
[ 106.710079] md: bind<sde1>
[ 106.804284] raid6: sse2x1 gen() 6707 MB/s
[ 106.821281] raid6: sse2x1 xor() 5667 MB/s
[ 106.838284] raid6: sse2x2 gen() 8875 MB/s
[ 106.855282] raid6: sse2x2 xor() 6230 MB/s
[ 106.872278] raid6: sse2x4 gen() 10359 MB/s
[ 106.889278] raid6: sse2x4 xor() 7697 MB/s
[ 106.889280] raid6: using algorithm sse2x4 gen() 10359 MB/s
[ 106.889281] raid6: .... xor() 7697 MB/s, rmw enabled
[ 106.889282] raid6: using ssse3x2 recovery algorithm
[ 106.907347] async_tx: api initialized (async)
[ 106.924571] xor: automatically using best checksumming function:
[ 106.934274] avx : 19356.000 MB/sec
[ 107.036697] md: raid6 personality registered for level 6
[ 107.036699] md: raid5 personality registered for level 5
[ 107.036700] md: raid4 personality registered for level 4
[ 107.037215] md/raid:md0: device sdd1 operational as raid disk 2
[ 107.037217] md/raid:md0: device sdc1 operational as raid disk 1
[ 107.037218] md/raid:md0: device sdb1 operational as raid disk 0
[ 107.037508] md/raid:md0: allocated 4374kB
[ 107.037533] md/raid:md0: raid level 5 active with 3 out of 4 devices, algorithm 2
[ 107.037534] RAID conf printout:
[ 107.037535] --- level:5 rd:4 wd:3
[ 107.037536] disk 0, o:1, dev:sdb1
[ 107.037537] disk 1, o:1, dev:sdc1
[ 107.037538] disk 2, o:1, dev:sdd1
[ 107.037548] md/raid:md0: using device sdf1 as journal
[ 107.037829] md0: detected capacity change from 0 to 32187088896
[ 107.037849] RAID conf printout:
[ 107.037853] --- level:5 rd:4 wd:3
[ 107.037855] disk 0, o:1, dev:sdb1
[ 107.037857] disk 1, o:1, dev:sdc1
[ 107.037858] disk 2, o:1, dev:sdd1
[ 107.037860] disk 3, o:1, dev:sde1
[ 107.037945] md: recovery of RAID array md0
[ 107.037948] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 107.037949] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 107.037954] md: using 128k window, over a total of 10477568k.
[ 216.232974] md: md0: recovery done.
[ 216.431212] RAID conf printout:
[ 216.431216] --- level:5 rd:4 wd:4
[ 216.431218] disk 0, o:1, dev:sdb1
[ 216.431219] disk 1, o:1, dev:sdc1
[ 216.431220] disk 2, o:1, dev:sdd1
[ 216.431221] disk 3, o:1, dev:sde1
[ 222.897854] md/raid:md0: Disk failure on sdf1, disabling device.
md/raid:md0: Operation continuing on 4 devices.
[ 223.174949] RAID conf printout:
[ 223.174952] --- level:5 rd:4 wd:4
[ 223.174953] disk 0, o:1, dev:sdb1
[ 223.174954] disk 1, o:1, dev:sdc1
[ 223.174955] disk 2, o:1, dev:sdd1
[ 223.174956] disk 3, o:1, dev:sde1
[ 241.640253] md: unbind<sdf1>
[ 241.643448] md: export_rdev(sdf1)
[ 261.329087] md: bind<sdf1>
[ 261.607691] RAID conf printout:
[ 261.607694] --- level:5 rd:4 wd:4
[ 261.607695] disk 0, o:1, dev:sdb1
[ 261.607696] disk 1, o:1, dev:sdc1
[ 261.607697] disk 2, o:1, dev:sdd1
[ 261.607698] disk 3, o:1, dev:sde1
[ 292.581113] RAID conf printout:
[ 292.581116] --- level:5 rd:5 wd:5
[ 292.581118] disk 0, o:1, dev:sdb1
[ 292.581119] disk 1, o:1, dev:sdc1
[ 292.581120] disk 2, o:1, dev:sdd1
[ 292.581121] disk 3, o:1, dev:sde1
[ 292.581121] disk 4, o:1, dev:sdf1
[ 292.581199] md: reshape of RAID array md0
[ 292.581202] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 292.581204] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
[ 292.581214] md: using 128k window, over a total of 10477568k.
[ 1201.734199] INFO: task systemd-udevd:794 blocked for more than 600 seconds.
[ 1201.734204] Not tainted 4.7.0 #5
[ 1201.734204] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1201.734206] systemd-udevd D ffff8800310a77b0 0 794 1 0x00000000
[ 1201.734209] ffff8800310a77b0 0000000000000001 ffff880031de0000 ffff8807cd3c5688
[ 1201.734211] ffff8800310a8000 0000000000000000 0000000000000002 ffff8807c3e4c100
[ 1201.734213] 0000000000000000 ffff8800310a77c8 ffffffff816e4935 ffff8807cd3c5400
[ 1201.734214] Call Trace:
[ 1201.734220] [<ffffffff816e4935>] schedule+0x35/0x80
[ 1201.734226] [<ffffffffa09a0ba9>] raid5_make_request+0x8b9/0xce0 [raid456]
[ 1201.734229] [<ffffffff810d0240>] ? prepare_to_wait_event+0xf0/0xf0
[ 1201.734233] [<ffffffff8156758e>] md_make_request+0xee/0x230
[ 1201.734236] [<ffffffff81320803>] generic_make_request+0x103/0x1d0
[ 1201.734237] [<ffffffff81320947>] submit_bio+0x77/0x150
[ 1201.734240] [<ffffffff813183bb>] ? bio_alloc_bioset+0x1ab/0x2d0
[ 1201.734242] [<ffffffff81254ecd>] ? alloc_page_buffers+0x5d/0xb0
[ 1201.734244] [<ffffffff81256a1f>] submit_bh_wbc+0x12f/0x160
[ 1201.734246] [<ffffffff81256d58>] block_read_full_page+0x208/0x380
[ 1201.734248] [<ffffffff81259000>] ? I_BDEV+0x20/0x20
[ 1201.734250] [<ffffffff812597e8>] blkdev_readpage+0x18/0x20
[ 1201.734253] [<ffffffff81197718>] do_read_cache_page+0x138/0x300
[ 1201.734255] [<ffffffff812597d0>] ? blkdev_writepages+0x40/0x40
[ 1201.734257] [<ffffffff811978f9>] read_cache_page+0x19/0x20
[ 1201.734259] [<ffffffff81333b50>] read_dev_sector+0x80/0xb0
[ 1201.734261] [<ffffffff81337f14>] read_lba+0x104/0x1c0
[ 1201.734263] [<ffffffff8133852a>] find_valid_gpt+0xfa/0x710
[ 1201.734266] [<ffffffff811d7d00>] ? vmap_page_range_noflush+0x190/0x380
[ 1201.734267] [<ffffffff81338b40>] ? find_valid_gpt+0x710/0x710
[ 1201.734269] [<ffffffff81338bc9>] efi_partition+0x89/0x440
[ 1201.734271] [<ffffffff81359209>] ? snprintf+0x49/0x70
[ 1201.734273] [<ffffffff81338b40>] ? find_valid_gpt+0x710/0x710
[ 1201.734275] [<ffffffff81335b31>] check_partition+0x101/0x1f0
[ 1201.734277] [<ffffffff81334462>] rescan_partitions+0xc2/0x2c0
[ 1201.734279] [<ffffffff8132f294>] __blkdev_reread_part+0x64/0x70
[ 1201.734281] [<ffffffff8132f2c3>] blkdev_reread_part+0x23/0x40
[ 1201.734282] [<ffffffff8132fd3a>] blkdev_ioctl+0x46a/0x8f0
[ 1201.734284] [<ffffffff81259aa1>] block_ioctl+0x41/0x50
[ 1201.734286] [<ffffffff812323b6>] do_vfs_ioctl+0xa6/0x5c0
[ 1201.734288] [<ffffffff81232949>] SyS_ioctl+0x79/0x90
[ 1201.734290] [<ffffffff81274d67>] ? SyS_flock+0x117/0x1a0
[ 1201.734292] [<ffffffff816e86b2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
[ 1201.734305] INFO: task md0_reshape:4089 blocked for more than 600 seconds.
[ 1201.734306] Not tainted 4.7.0 #5
[ 1201.734307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1201.734308] md0_reshape D ffff88080db77ac0 0 4089 2 0x00000080
[ 1201.734310] ffff88080db77ac0 0000000000000000 ffff8807d6724380 ffff8807cd3c5670
[ 1201.734311] ffff88080db78000 ffff88080db77b20 ffff8807cd3c5670 0000000000000c20
[ 1201.734313] ffff8807cd3c5418 ffff88080db77ad8 ffffffff816e4935 ffff8807cd3c5400
[ 1201.734314] Call Trace:
[ 1201.734316] [<ffffffff816e4935>] schedule+0x35/0x80
[ 1201.734319] [<ffffffffa099fe8b>] raid5_get_active_stripe+0x20b/0x670 [raid456]
[ 1201.734321] [<ffffffff810d0240>] ? prepare_to_wait_event+0xf0/0xf0
[ 1201.734323] [<ffffffffa09a4733>] reshape_request+0x4d3/0x8a0 [raid456]
[ 1201.734326] [<ffffffffa09a4e21>] raid5_sync_request+0x321/0x3a0 [raid456]
[ 1201.734327] [<ffffffff810cfc54>] ? __wake_up+0x44/0x50
[ 1201.734329] [<ffffffff8156ce97>] md_do_sync+0x9e7/0xf60
[ 1201.734332] [<ffffffff810c200d>] ? update_curr+0xed/0x180
[ 1201.734333] [<ffffffff810c025e>] ? account_entity_dequeue+0xae/0xd0
[ 1201.734336] [<ffffffff81098d43>] ? kernel_sigaction+0x43/0xe0
[ 1201.734339] [<ffffffff81566dc6>] md_thread+0x136/0x150
[ 1201.734341] [<ffffffff81566c90>] ? find_pers+0x70/0x70
[ 1201.734343] [<ffffffff810ab678>] kthread+0xd8/0xf0
[ 1201.734344] [<ffffffff816e88bf>] ret_from_fork+0x1f/0x40
[ 1201.734346] [<ffffffff810ab5a0>] ? kthread_park+0x60/0x60
Reproduce steps:
#mdadm -CR /dev/md0 -l5 -n4 /dev/sd[b-e]1 --write-journal /dev/sdf1
#mdadm --wait /dev/md0
#mdadm /dev/md0 -f /dev/sdf1
#mdadm /dev/md0 -r /dev/sdf1
#mdadm /dev/md0 -a /dev/sdf1
#mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Aug 4 20:37:41 2016
Raid Level : raid5
Array Size : 31432704 (29.98 GiB 32.19 GB)
Used Dev Size : 10477568 (9.99 GiB 10.73 GB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Thu Aug 4 20:40:16 2016
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Name : dhcp-12-125.nay.redhat.com:0 (local to host dhcp-12-125.nay.redhat.com)
UUID : 09607542:a8dccc81:f5d2f80c:c586923f
Events : 22
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
5 8 65 3 active sync /dev/sde1
4 8 81 - spare /dev/sdf1
#mdadm -G -n5 /dev/md0
Best Regards,
Yi Zhang
^ permalink raw reply
* Re: [PATCH 37/45] drivers: use req op accessor
From: Shaun Tancheff @ 2016-08-04 16:32 UTC (permalink / raw)
To: Christoph Hellwig
Cc: philipp.reisner, dm-devel, target-devel, linux-mtd, drbd-dev,
linux-scsi, Mike Christie, Ross Zwisler, linux-ext4, konrad.wilk,
XFS Developers, linux-block, linux-bcache, osd-dev, Ross Zwisler,
Jens Axboe, linux-raid, LKML, linux-f2fs-devel, linux-btrfs,
linux-fsdevel, lars.ellenberg, ocfs2-devel
In-Reply-To: <20160804154622.GB15656@infradead.org>
On Thu, Aug 4, 2016 at 10:46 AM, Christoph Hellwig <hch@infradead.org> wrote:
> On Wed, Aug 03, 2016 at 07:30:29PM -0500, Shaun Tancheff wrote:
>> I think the translation in loop.c is suspicious here:
>>
>> "if use DIO && not (a flush_flag or discard_flag)"
>> should translate to:
>> "if use DIO && not ((a flush_flag) || op == discard)"
>>
>> But in the patch I read:
>> "if use DIO && ((not a flush_flag) || op == discard)
>>
>> Which would have DIO && discards follow the AIO path?
>
> Indeed. Sorry for missing out on your patch, I just sent a fix
> in reply to Dave's other report earlier which is pretty similar to
> yours.
No worries. I prefer your switch to a an if conditional here.
--
Shaun Tancheff
^ permalink raw reply
* [PATCH RFC] mm, writeback: flush plugged IO in wakeup_flusher_threads()
From: Konstantin Khlebnikov @ 2016-08-04 18:36 UTC (permalink / raw)
To: linux-mm
Cc: Jens Axboe, Michal Hocko, Mel Gorman, linux-raid, Dave Chinner,
Tejun Heo, Andrew Morton, Shaohua Li
I've found funny live-lock between raid10 barriers during resync and memory
controller hard limits. Inside mpage_readpages() task holds on its plug bio
which blocks barrier in raid10. Its memory cgroup have no free memory thus
task goes into reclaimer but all reclaimable pages are dirty and cannot be
written because raid10 is rebuilding and stuck on barrier.
Common flush of such IO in schedule() never happens because machine where
that happened has a lot of free cpus and task never goes sleep.
Lock is 'live' because changing memory limit or killing tasks which holds
that stuck bio unblock whole progress.
That was happened in 3.18.x but I see no difference in upstream logic.
Theoretically this might happen even without memory cgroup.
Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
---
fs/fs-writeback.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 56c8fda436c0..ed58863cdb5d 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -1948,6 +1948,12 @@ void wakeup_flusher_threads(long nr_pages, enum wb_reason reason)
{
struct backing_dev_info *bdi;
+ /*
+ * If we are expecting writeback progress we must submit plugged IO.
+ */
+ if (blk_needs_flush_plug(current))
+ blk_schedule_flush_plug(current);
+
if (!nr_pages)
nr_pages = get_nr_dirty_pages();
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related
* Re: The subarray is loaded container by load_container
From: Jes Sorensen @ 2016-08-04 18:38 UTC (permalink / raw)
To: Artur Paszkiewicz
Cc: Xiao Ni, linux-raid, tomasz.majchrzak, aleksey.obitotskiy,
pawel.baldysiak
In-Reply-To: <5798B4DD.6030702@redhat.com>
Xiao Ni <xni@redhat.com> writes:
> On 07/07/2016 06:16 PM, Artur Paszkiewicz wrote:
>> Hi Xiao,
>>
>> I think you're right that mdadm -IRs should return 0 in this case.
>> IncrementalScan should not try loading a container from a member array,
>> because that will always fail. Can you check if this fixes the problem
>> for you?
>>
>> diff --git a/Incremental.c b/Incremental.c
>> index ba97b00..cc01d41 100644
>> --- a/Incremental.c
>> +++ b/Incremental.c
>> @@ -1347,8 +1347,12 @@ restart:
>> if (devnm && strcmp(devnm, me->devnm) != 0)
>> continue;
>> - if (devnm && me->metadata[0] == '/') {
>> + if (me->metadata[0] == '/') {
>> char *sl;
>> +
>> + if (!devnm)
>> + continue;
>> +
>> /* member array, need to work on container */
>> strncpy(container, me->metadata+1, 32);
>> container[31] = 0;
>>
>> Thanks,
>> Artur
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Hi Artur
>
> Sorry for late response. I have been waiting the result from customer
> and the customer haven't give me the answer. I tested this in my
> environment and this patch can fix this problem.
Artur,
Do you have a version of this patch you would like me to apply to the
official mdadm tree?
Cheers,
Jes
^ permalink raw reply
* Re: [PATCH RFC] mm, writeback: flush plugged IO in wakeup_flusher_threads()
From: Jens Axboe @ 2016-08-04 19:21 UTC (permalink / raw)
To: Konstantin Khlebnikov, linux-mm
Cc: Michal Hocko, Mel Gorman, linux-raid, Dave Chinner, Tejun Heo,
Andrew Morton, Shaohua Li
In-Reply-To: <147033576532.682609.2277943215598867297.stgit@buzz>
On 08/04/2016 12:36 PM, Konstantin Khlebnikov wrote:
> I've found funny live-lock between raid10 barriers during resync and memory
> controller hard limits. Inside mpage_readpages() task holds on its plug bio
> which blocks barrier in raid10. Its memory cgroup have no free memory thus
> task goes into reclaimer but all reclaimable pages are dirty and cannot be
> written because raid10 is rebuilding and stuck on barrier.
>
> Common flush of such IO in schedule() never happens because machine where
> that happened has a lot of free cpus and task never goes sleep.
>
> Lock is 'live' because changing memory limit or killing tasks which holds
> that stuck bio unblock whole progress.
>
> That was happened in 3.18.x but I see no difference in upstream logic.
> Theoretically this might happen even without memory cgroup.
So the issue is that the caller of wakeup_flusher_threads() ends up
never going to sleep, hence the plug is never auto-flushed. I didn't
quite understand your reasoning for why it never sleeps above, but that
must be the gist of it.
I don't see anything inherently wrong with the fix.
--
Jens Axboe
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* xosview - anybody willing to test it (the raid features :-)
From: Wols Lists @ 2016-08-04 22:27 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <22435.38657.908865.828712@lrd.home.lan>
Mike has pushed a test version to sourceforge and github. If anybody's
willing to clone the repository, compile and test that would be appreciated.
For those who don't know this utility, once you've built it just do
"xosview &" at the command line and you'll get a pretty little status
monitor appear on your screen.
I'm planning to test it tomorrow (wife permitting), so I'm not sure how
it's going to behave myself. As Mike says, all feedback will be welcome.
Cheers,
Wol
-------- Forwarded Message --------
This may help. I've turned the raid meter on by default and it is set to
auto which causes it to search for and display a meter for every md
device it
finds. The meter gives a pretty compact overview of the raids current state
in one glance.
If you are able to checkout and build a copy of the source from either of
these two repositories (I push to both) I'd be interested in your feedback.
https://sourceforge.net/p/xosview/git/ci/master/tree/
https://github.com/mromberg/xosview
I developed the new raid meter code using a few of raid1 devices
(two drives per raid) with a fedora 24 setup running on a VM. But I am not
an experienced raid administrator by any stretch of the imagination. I
have
no doubt that there are cases and configurations I have not tested or
accounted
for. So, I welcome any feedback on what is there so far.
Mike
^ permalink raw reply
* Re: [PATCH RFC] mm, writeback: flush plugged IO in wakeup_flusher_threads()
From: Konstantin Khlebnikov @ 2016-08-05 5:44 UTC (permalink / raw)
To: Jens Axboe
Cc: Konstantin Khlebnikov, linux-mm@kvack.org, Michal Hocko,
Mel Gorman, linux-raid, Dave Chinner, Tejun Heo, Andrew Morton,
Shaohua Li
In-Reply-To: <3ee639d7-2371-c27d-3639-e4b1315d6663@kernel.dk>
On Thu, Aug 4, 2016 at 10:21 PM, Jens Axboe <axboe@kernel.dk> wrote:
> On 08/04/2016 12:36 PM, Konstantin Khlebnikov wrote:
>>
>> I've found funny live-lock between raid10 barriers during resync and
>> memory
>> controller hard limits. Inside mpage_readpages() task holds on its plug
>> bio
>> which blocks barrier in raid10. Its memory cgroup have no free memory thus
>> task goes into reclaimer but all reclaimable pages are dirty and cannot be
>> written because raid10 is rebuilding and stuck on barrier.
>>
>> Common flush of such IO in schedule() never happens because machine where
>> that happened has a lot of free cpus and task never goes sleep.
>>
>> Lock is 'live' because changing memory limit or killing tasks which holds
>> that stuck bio unblock whole progress.
>>
>> That was happened in 3.18.x but I see no difference in upstream logic.
>> Theoretically this might happen even without memory cgroup.
>
>
> So the issue is that the caller of wakeup_flusher_threads() ends up never
> going to sleep, hence the plug is never auto-flushed. I didn't quite
> understand your reasoning for why it never sleeps above, but that must be
> the gist of it.
Ah right, simple context switch doesn't flush plug, so count of cpus
is irrelevant.
>
> I don't see anything inherently wrong with the fix.
>
> --
> Jens Axboe
>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: INFO: task systemd-udevd:794 blocked for more than 600 seconds observed after mdadm -G operation
From: Guoqing Jiang @ 2016-08-05 6:39 UTC (permalink / raw)
To: Yi Zhang, linux-raid; +Cc: shli, NeilBrown
In-Reply-To: <1363340670.11420734.1470326479339.JavaMail.zimbra@redhat.com>
On 08/04/2016 12:01 PM, Yi Zhang wrote:
> [ 1201.734199] INFO: task systemd-udevd:794 blocked for more than 600 seconds.
> [ 1201.734204] Not tainted 4.7.0 #5
> [ 1201.734204] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 1201.734206] systemd-udevd D ffff8800310a77b0 0 794 1 0x00000000
> [ 1201.734209] ffff8800310a77b0 0000000000000001 ffff880031de0000 ffff8807cd3c5688
> [ 1201.734211] ffff8800310a8000 0000000000000000 0000000000000002 ffff8807c3e4c100
> [ 1201.734213] 0000000000000000 ffff8800310a77c8 ffffffff816e4935 ffff8807cd3c5400
> [ 1201.734214] Call Trace:
> [ 1201.734220] [<ffffffff816e4935>] schedule+0x35/0x80
> [ 1201.734226] [<ffffffffa09a0ba9>] raid5_make_request+0x8b9/0xce0 [raid456]
> [ 1201.734229] [<ffffffff810d0240>] ? prepare_to_wait_event+0xf0/0xf0
> [ 1201.734233] [<ffffffff8156758e>] md_make_request+0xee/0x230
> [ 1201.734236] [<ffffffff81320803>] generic_make_request+0x103/0x1d0
> [ 1201.734237] [<ffffffff81320947>] submit_bio+0x77/0x150
> [ 1201.734240] [<ffffffff813183bb>] ? bio_alloc_bioset+0x1ab/0x2d0
> [ 1201.734242] [<ffffffff81254ecd>] ? alloc_page_buffers+0x5d/0xb0
> [ 1201.734244] [<ffffffff81256a1f>] submit_bh_wbc+0x12f/0x160
> [ 1201.734246] [<ffffffff81256d58>] block_read_full_page+0x208/0x380
> [ 1201.734248] [<ffffffff81259000>] ? I_BDEV+0x20/0x20
> [ 1201.734250] [<ffffffff812597e8>] blkdev_readpage+0x18/0x20
> [ 1201.734253] [<ffffffff81197718>] do_read_cache_page+0x138/0x300
> [ 1201.734255] [<ffffffff812597d0>] ? blkdev_writepages+0x40/0x40
> [ 1201.734257] [<ffffffff811978f9>] read_cache_page+0x19/0x20
> [ 1201.734259] [<ffffffff81333b50>] read_dev_sector+0x80/0xb0
> [ 1201.734261] [<ffffffff81337f14>] read_lba+0x104/0x1c0
> [ 1201.734263] [<ffffffff8133852a>] find_valid_gpt+0xfa/0x710
> [ 1201.734266] [<ffffffff811d7d00>] ? vmap_page_range_noflush+0x190/0x380
> [ 1201.734267] [<ffffffff81338b40>] ? find_valid_gpt+0x710/0x710
> [ 1201.734269] [<ffffffff81338bc9>] efi_partition+0x89/0x440
> [ 1201.734271] [<ffffffff81359209>] ? snprintf+0x49/0x70
> [ 1201.734273] [<ffffffff81338b40>] ? find_valid_gpt+0x710/0x710
> [ 1201.734275] [<ffffffff81335b31>] check_partition+0x101/0x1f0
> [ 1201.734277] [<ffffffff81334462>] rescan_partitions+0xc2/0x2c0
> [ 1201.734279] [<ffffffff8132f294>] __blkdev_reread_part+0x64/0x70
> [ 1201.734281] [<ffffffff8132f2c3>] blkdev_reread_part+0x23/0x40
> [ 1201.734282] [<ffffffff8132fd3a>] blkdev_ioctl+0x46a/0x8f0
> [ 1201.734284] [<ffffffff81259aa1>] block_ioctl+0x41/0x50
> [ 1201.734286] [<ffffffff812323b6>] do_vfs_ioctl+0xa6/0x5c0
> [ 1201.734288] [<ffffffff81232949>] SyS_ioctl+0x79/0x90
> [ 1201.734290] [<ffffffff81274d67>] ? SyS_flock+0x117/0x1a0
> [ 1201.734292] [<ffffffff816e86b2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
> [ 1201.734305] INFO: task md0_reshape:4089 blocked for more than 600 seconds.
> [ 1201.734306] Not tainted 4.7.0 #5
> [ 1201.734307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 1201.734308] md0_reshape D ffff88080db77ac0 0 4089 2 0x00000080
> [ 1201.734310] ffff88080db77ac0 0000000000000000 ffff8807d6724380 ffff8807cd3c5670
> [ 1201.734311] ffff88080db78000 ffff88080db77b20 ffff8807cd3c5670 0000000000000c20
> [ 1201.734313] ffff8807cd3c5418 ffff88080db77ad8 ffffffff816e4935 ffff8807cd3c5400
> [ 1201.734314] Call Trace:
> [ 1201.734316] [<ffffffff816e4935>] schedule+0x35/0x80
> [ 1201.734319] [<ffffffffa099fe8b>] raid5_get_active_stripe+0x20b/0x670 [raid456]
> [ 1201.734321] [<ffffffff810d0240>] ? prepare_to_wait_event+0xf0/0xf0
> [ 1201.734323] [<ffffffffa09a4733>] reshape_request+0x4d3/0x8a0 [raid456]
> [ 1201.734326] [<ffffffffa09a4e21>] raid5_sync_request+0x321/0x3a0 [raid456]
> [ 1201.734327] [<ffffffff810cfc54>] ? __wake_up+0x44/0x50
> [ 1201.734329] [<ffffffff8156ce97>] md_do_sync+0x9e7/0xf60
> [ 1201.734332] [<ffffffff810c200d>] ? update_curr+0xed/0x180
> [ 1201.734333] [<ffffffff810c025e>] ? account_entity_dequeue+0xae/0xd0
> [ 1201.734336] [<ffffffff81098d43>] ? kernel_sigaction+0x43/0xe0
> [ 1201.734339] [<ffffffff81566dc6>] md_thread+0x136/0x150
> [ 1201.734341] [<ffffffff81566c90>] ? find_pers+0x70/0x70
> [ 1201.734343] [<ffffffff810ab678>] kthread+0xd8/0xf0
> [ 1201.734344] [<ffffffff816e88bf>] ret_from_fork+0x1f/0x40
> [ 1201.734346] [<ffffffff810ab5a0>] ? kthread_park+0x60/0x60
>
>
> Reproduce steps:
>
> #mdadm -CR /dev/md0 -l5 -n4 /dev/sd[b-e]1 --write-journal /dev/sdf1
> #mdadm --wait /dev/md0
> #mdadm /dev/md0 -f /dev/sdf1
> #mdadm /dev/md0 -r /dev/sdf1
> #mdadm /dev/md0 -a /dev/sdf1
> #mdadm -D /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Thu Aug 4 20:37:41 2016
> Raid Level : raid5
> Array Size : 31432704 (29.98 GiB 32.19 GB)
> Used Dev Size : 10477568 (9.99 GiB 10.73 GB)
> Raid Devices : 4
> Total Devices : 5
> Persistence : Superblock is persistent
>
> Update Time : Thu Aug 4 20:40:16 2016
> State : clean
> Active Devices : 4
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 1
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Name : dhcp-12-125.nay.redhat.com:0 (local to host dhcp-12-125.nay.redhat.com)
> UUID : 09607542:a8dccc81:f5d2f80c:c586923f
> Events : 22
>
> Number Major Minor RaidDevice State
> 0 8 17 0 active sync /dev/sdb1
> 1 8 33 1 active sync /dev/sdc1
> 2 8 49 2 active sync /dev/sdd1
> 5 8 65 3 active sync /dev/sde1
>
> 4 8 81 - spare /dev/sdf1
>
> #mdadm -G -n5 /dev/md0
Also with 4.7+, I can't find the same issue, but md0_reshape is in 'D'
state from my side
(actually I can't find reshape is in-progress by 'cat /proc/mdstat').
linux241:~ # mdadm -G -n5 /dev/md0
mdadm: Need to backup 6144K of critical section..
linux241:~ # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 vdf1[4] vde1[5] vdd1[2] vdc1[1] vdb1[0]
9431040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5]
[UUUUU]
unused devices: <none>
linux241:~ # ps aux|grep md|grep D
root 1890 0.0 0.0 0 0 ? D 14:09 0:00
[md0_reshape]
linux241:~ # cat /proc/1890/stack
[<ffffffffa05423b9>] reshape_request+0x5c9/0x840 [raid456]
[<ffffffffa054283f>] raid5_sync_request+0x20f/0x3a0 [raid456]
[<ffffffffa043f03b>] md_do_sync+0x8fb/0xe60 [md_mod]
[<ffffffffa043a691>] md_thread+0x111/0x130 [md_mod]
[<ffffffff8108a649>] kthread+0xc9/0xe0
[<ffffffff815e87bf>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff
Then the array can't be stop successfully after '-G', and the stack of
reshape is
also changed,
linux241:~ # ps aux|grep md|grep D
root 1890 0.0 0.0 0 0 ? D 14:09 0:00
[md0_reshape]
root 1961 0.0 0.1 19864 2016 pts/1 D+ 14:13 0:00 mdadm -Ss
root 1962 0.0 0.2 43240 2288 ? D 14:13 0:00
/usr/lib/systemd/systemd-udevd
linux241:~ # cat /proc/1890/stack
[<ffffffffa043f2ab>] md_do_sync+0xb6b/0xe60 [md_mod]
[<ffffffffa043a691>] md_thread+0x111/0x130 [md_mod]
[<ffffffff8108a649>] kthread+0xc9/0xe0
[<ffffffff815e87bf>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff
linux241:~ # cat /proc/1961/stack
[<ffffffff8108a8da>] kthread_stop+0x4a/0x110
[<ffffffffa043dff4>] md_unregister_thread+0x44/0x80 [md_mod]
[<ffffffffa04431ab>] md_reap_sync_thread+0x1b/0x180 [md_mod]
[<ffffffffa0443451>] action_store+0x141/0x2a0 [md_mod]
[<ffffffffa043f619>] md_attr_store+0x79/0xb0 [md_mod]
[<ffffffff81269fca>] sysfs_kf_write+0x3a/0x50
[<ffffffff8126961b>] kernfs_fop_write+0x11b/0x1a0
[<ffffffff811f04e8>] __vfs_write+0x28/0x120
[<ffffffff811f1342>] vfs_write+0xb2/0x1b0
[<ffffffff811f2686>] SyS_write+0x46/0xa0
[<ffffffff815e85b2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
[<ffffffffffffffff>] 0xffffffffffffffff
Since the steps work for raid5 (add vdf1 as spare), I guess that '-G'
can't co-operate
with raid5-cache well so far. Just FYI.
Thanks,
Guoqing
^ permalink raw reply
* Re: INFO: task systemd-udevd:794 blocked for more than 600 seconds observed after mdadm -G operation
From: yizhan @ 2016-08-05 7:05 UTC (permalink / raw)
To: Guoqing Jiang, linux-raid; +Cc: shli, NeilBrown
In-Reply-To: <57A434A9.2020501@suse.com>
On 08/05/2016 02:39 PM, Guoqing Jiang wrote:
>
>
> On 08/04/2016 12:01 PM, Yi Zhang wrote:
>
>> [ 1201.734199] INFO: task systemd-udevd:794 blocked for more than 600
>> seconds.
>> [ 1201.734204] Not tainted 4.7.0 #5
>> [ 1201.734204] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 1201.734206] systemd-udevd D ffff8800310a77b0 0 794 1
>> 0x00000000
>> [ 1201.734209] ffff8800310a77b0 0000000000000001 ffff880031de0000
>> ffff8807cd3c5688
>> [ 1201.734211] ffff8800310a8000 0000000000000000 0000000000000002
>> ffff8807c3e4c100
>> [ 1201.734213] 0000000000000000 ffff8800310a77c8 ffffffff816e4935
>> ffff8807cd3c5400
>> [ 1201.734214] Call Trace:
>> [ 1201.734220] [<ffffffff816e4935>] schedule+0x35/0x80
>> [ 1201.734226] [<ffffffffa09a0ba9>] raid5_make_request+0x8b9/0xce0
>> [raid456]
>> [ 1201.734229] [<ffffffff810d0240>] ? prepare_to_wait_event+0xf0/0xf0
>> [ 1201.734233] [<ffffffff8156758e>] md_make_request+0xee/0x230
>> [ 1201.734236] [<ffffffff81320803>] generic_make_request+0x103/0x1d0
>> [ 1201.734237] [<ffffffff81320947>] submit_bio+0x77/0x150
>> [ 1201.734240] [<ffffffff813183bb>] ? bio_alloc_bioset+0x1ab/0x2d0
>> [ 1201.734242] [<ffffffff81254ecd>] ? alloc_page_buffers+0x5d/0xb0
>> [ 1201.734244] [<ffffffff81256a1f>] submit_bh_wbc+0x12f/0x160
>> [ 1201.734246] [<ffffffff81256d58>] block_read_full_page+0x208/0x380
>> [ 1201.734248] [<ffffffff81259000>] ? I_BDEV+0x20/0x20
>> [ 1201.734250] [<ffffffff812597e8>] blkdev_readpage+0x18/0x20
>> [ 1201.734253] [<ffffffff81197718>] do_read_cache_page+0x138/0x300
>> [ 1201.734255] [<ffffffff812597d0>] ? blkdev_writepages+0x40/0x40
>> [ 1201.734257] [<ffffffff811978f9>] read_cache_page+0x19/0x20
>> [ 1201.734259] [<ffffffff81333b50>] read_dev_sector+0x80/0xb0
>> [ 1201.734261] [<ffffffff81337f14>] read_lba+0x104/0x1c0
>> [ 1201.734263] [<ffffffff8133852a>] find_valid_gpt+0xfa/0x710
>> [ 1201.734266] [<ffffffff811d7d00>] ?
>> vmap_page_range_noflush+0x190/0x380
>> [ 1201.734267] [<ffffffff81338b40>] ? find_valid_gpt+0x710/0x710
>> [ 1201.734269] [<ffffffff81338bc9>] efi_partition+0x89/0x440
>> [ 1201.734271] [<ffffffff81359209>] ? snprintf+0x49/0x70
>> [ 1201.734273] [<ffffffff81338b40>] ? find_valid_gpt+0x710/0x710
>> [ 1201.734275] [<ffffffff81335b31>] check_partition+0x101/0x1f0
>> [ 1201.734277] [<ffffffff81334462>] rescan_partitions+0xc2/0x2c0
>> [ 1201.734279] [<ffffffff8132f294>] __blkdev_reread_part+0x64/0x70
>> [ 1201.734281] [<ffffffff8132f2c3>] blkdev_reread_part+0x23/0x40
>> [ 1201.734282] [<ffffffff8132fd3a>] blkdev_ioctl+0x46a/0x8f0
>> [ 1201.734284] [<ffffffff81259aa1>] block_ioctl+0x41/0x50
>> [ 1201.734286] [<ffffffff812323b6>] do_vfs_ioctl+0xa6/0x5c0
>> [ 1201.734288] [<ffffffff81232949>] SyS_ioctl+0x79/0x90
>> [ 1201.734290] [<ffffffff81274d67>] ? SyS_flock+0x117/0x1a0
>> [ 1201.734292] [<ffffffff816e86b2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
>> [ 1201.734305] INFO: task md0_reshape:4089 blocked for more than 600
>> seconds.
>> [ 1201.734306] Not tainted 4.7.0 #5
>> [ 1201.734307] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 1201.734308] md0_reshape D ffff88080db77ac0 0 4089 2
>> 0x00000080
>> [ 1201.734310] ffff88080db77ac0 0000000000000000 ffff8807d6724380
>> ffff8807cd3c5670
>> [ 1201.734311] ffff88080db78000 ffff88080db77b20 ffff8807cd3c5670
>> 0000000000000c20
>> [ 1201.734313] ffff8807cd3c5418 ffff88080db77ad8 ffffffff816e4935
>> ffff8807cd3c5400
>> [ 1201.734314] Call Trace:
>> [ 1201.734316] [<ffffffff816e4935>] schedule+0x35/0x80
>> [ 1201.734319] [<ffffffffa099fe8b>]
>> raid5_get_active_stripe+0x20b/0x670 [raid456]
>> [ 1201.734321] [<ffffffff810d0240>] ? prepare_to_wait_event+0xf0/0xf0
>> [ 1201.734323] [<ffffffffa09a4733>] reshape_request+0x4d3/0x8a0
>> [raid456]
>> [ 1201.734326] [<ffffffffa09a4e21>] raid5_sync_request+0x321/0x3a0
>> [raid456]
>> [ 1201.734327] [<ffffffff810cfc54>] ? __wake_up+0x44/0x50
>> [ 1201.734329] [<ffffffff8156ce97>] md_do_sync+0x9e7/0xf60
>> [ 1201.734332] [<ffffffff810c200d>] ? update_curr+0xed/0x180
>> [ 1201.734333] [<ffffffff810c025e>] ? account_entity_dequeue+0xae/0xd0
>> [ 1201.734336] [<ffffffff81098d43>] ? kernel_sigaction+0x43/0xe0
>> [ 1201.734339] [<ffffffff81566dc6>] md_thread+0x136/0x150
>> [ 1201.734341] [<ffffffff81566c90>] ? find_pers+0x70/0x70
>> [ 1201.734343] [<ffffffff810ab678>] kthread+0xd8/0xf0
>> [ 1201.734344] [<ffffffff816e88bf>] ret_from_fork+0x1f/0x40
>> [ 1201.734346] [<ffffffff810ab5a0>] ? kthread_park+0x60/0x60
>>
>>
>> Reproduce steps:
>>
>> #mdadm -CR /dev/md0 -l5 -n4 /dev/sd[b-e]1 --write-journal /dev/sdf1
>> #mdadm --wait /dev/md0
>> #mdadm /dev/md0 -f /dev/sdf1
>> #mdadm /dev/md0 -r /dev/sdf1
>> #mdadm /dev/md0 -a /dev/sdf1
>> #mdadm -D /dev/md0
>> /dev/md0:
>> Version : 1.2
>> Creation Time : Thu Aug 4 20:37:41 2016
>> Raid Level : raid5
>> Array Size : 31432704 (29.98 GiB 32.19 GB)
>> Used Dev Size : 10477568 (9.99 GiB 10.73 GB)
>> Raid Devices : 4
>> Total Devices : 5
>> Persistence : Superblock is persistent
>>
>> Update Time : Thu Aug 4 20:40:16 2016
>> State : clean
>> Active Devices : 4
>> Working Devices : 5
>> Failed Devices : 0
>> Spare Devices : 1
>>
>> Layout : left-symmetric
>> Chunk Size : 512K
>>
>> Name : dhcp-12-125.nay.redhat.com:0 (local to host
>> dhcp-12-125.nay.redhat.com)
>> UUID : 09607542:a8dccc81:f5d2f80c:c586923f
>> Events : 22
>>
>> Number Major Minor RaidDevice State
>> 0 8 17 0 active sync /dev/sdb1
>> 1 8 33 1 active sync /dev/sdc1
>> 2 8 49 2 active sync /dev/sdd1
>> 5 8 65 3 active sync /dev/sde1
>>
>> 4 8 81 - spare /dev/sdf1
>>
>> #mdadm -G -n5 /dev/md0
>
> Also with 4.7+, I can't find the same issue, but md0_reshape is in 'D'
> state from my side
> (actually I can't find reshape is in-progress by 'cat /proc/mdstat').
>
Hi guoqing
Thanks for you detailed log.
The trace cannot be print soon, I think you need wait for more time, the
time from below is more than 15m.
Aug 4 20:39:37 dhcp-12-125 kernel: md/raid:md0: Disk failure on sdf1,
disabling device.#012md/raid:md0: Operation continuing on 4 devices.
Aug 4 20:39:56 dhcp-12-125 kernel: md: unbind<sdf1>
Aug 4 20:39:56 dhcp-12-125 kernel: md: export_rdev(sdf1)
Aug 4 20:40:16 dhcp-12-125 kernel: md: bind<sdf1>
Aug 4 20:40:47 dhcp-12-125 kernel: md: reshape of RAID array md0
Aug 4 20:40:47 dhcp-12-125 kernel: md: minimum _guaranteed_ speed:
1000 KB/sec/disk.
Aug 4 20:40:47 dhcp-12-125 kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for reshape.
Aug 4 20:40:47 dhcp-12-125 kernel: md: using 128k window, over a total
of 10477568k
Aug 4 20:55:56 dhcp-12-125 kernel: INFO: task systemd-udevd:794 blocked
for more than 600 seconds.
> linux241:~ # mdadm -G -n5 /dev/md0
> mdadm: Need to backup 6144K of critical section..
> linux241:~ # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid5 vdf1[4] vde1[5] vdd1[2] vdc1[1] vdb1[0]
> 9431040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5]
> [UUUUU]
>
> unused devices: <none>
> linux241:~ # ps aux|grep md|grep D
> root 1890 0.0 0.0 0 0 ? D 14:09 0:00
> [md0_reshape]
> linux241:~ # cat /proc/1890/stack
> [<ffffffffa05423b9>] reshape_request+0x5c9/0x840 [raid456]
> [<ffffffffa054283f>] raid5_sync_request+0x20f/0x3a0 [raid456]
> [<ffffffffa043f03b>] md_do_sync+0x8fb/0xe60 [md_mod]
> [<ffffffffa043a691>] md_thread+0x111/0x130 [md_mod]
> [<ffffffff8108a649>] kthread+0xc9/0xe0
> [<ffffffff815e87bf>] ret_from_fork+0x1f/0x40
> [<ffffffffffffffff>] 0xffffffffffffffff
>
Same on my side:
[root@dhcp-12-125 ~]# ps aux | grep md0 | grep D
root 3824 0.0 0.0 0 0 ? D 14:56 0:00
[md0_reshape]
[root@dhcp-12-125 ~]# cat /proc/3824/stack
[<ffffffffa09a6e8b>] raid5_get_active_stripe+0x20b/0x670 [raid456]
[<ffffffffa09ab733>] reshape_request+0x4d3/0x8a0 [raid456]
[<ffffffffa09abe21>] raid5_sync_request+0x321/0x3a0 [raid456]
[<ffffffff8156ce97>] md_do_sync+0x9e7/0xf60
[<ffffffff81566dc6>] md_thread+0x136/0x150
[<ffffffff810ab678>] kthread+0xd8/0xf0
[<ffffffff816e88bf>] ret_from_fork+0x1f/0x40
[<ffffffffffffffff>] 0xffffffffffffffff
> Then the array can't be stop successfully after '-G', and the stack of
> reshape is
> also changed,
>
> linux241:~ # ps aux|grep md|grep D
> root 1890 0.0 0.0 0 0 ? D 14:09 0:00
> [md0_reshape]
> root 1961 0.0 0.1 19864 2016 pts/1 D+ 14:13 0:00 mdadm
> -Ss
> root 1962 0.0 0.2 43240 2288 ? D 14:13 0:00
> /usr/lib/systemd/systemd-udevd
> linux241:~ # cat /proc/1890/stack
> [<ffffffffa043f2ab>] md_do_sync+0xb6b/0xe60 [md_mod]
> [<ffffffffa043a691>] md_thread+0x111/0x130 [md_mod]
> [<ffffffff8108a649>] kthread+0xc9/0xe0
> [<ffffffff815e87bf>] ret_from_fork+0x1f/0x40
> [<ffffffffffffffff>] 0xffffffffffffffff
> linux241:~ # cat /proc/1961/stack
> [<ffffffff8108a8da>] kthread_stop+0x4a/0x110
> [<ffffffffa043dff4>] md_unregister_thread+0x44/0x80 [md_mod]
> [<ffffffffa04431ab>] md_reap_sync_thread+0x1b/0x180 [md_mod]
> [<ffffffffa0443451>] action_store+0x141/0x2a0 [md_mod]
> [<ffffffffa043f619>] md_attr_store+0x79/0xb0 [md_mod]
> [<ffffffff81269fca>] sysfs_kf_write+0x3a/0x50
> [<ffffffff8126961b>] kernfs_fop_write+0x11b/0x1a0
> [<ffffffff811f04e8>] __vfs_write+0x28/0x120
> [<ffffffff811f1342>] vfs_write+0xb2/0x1b0
> [<ffffffff811f2686>] SyS_write+0x46/0xa0
> [<ffffffff815e85b2>] entry_SYSCALL_64_fastpath+0x1a/0xa4
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> Since the steps work for raid5 (add vdf1 as spare), I guess that '-G'
> can't co-operate
> with raid5-cache well so far. Just FYI.
>
> Thanks,
> Guoqing
^ permalink raw reply
* Re: journal disk become the first of --detail list after fail/remove one disk
From: Guoqing Jiang @ 2016-08-05 8:49 UTC (permalink / raw)
To: Adam Goryachev, Yi Zhang, linux-raid; +Cc: songliubraving, shli, Jes.Sorensen
In-Reply-To: <8ca13969-9e4f-b01e-6456-68a23512836f@websitemanagers.com.au>
On 08/04/2016 07:34 AM, Adam Goryachev wrote:
>
>
> On 4/08/2016 20:52, Yi Zhang wrote:
>> # mdadm -D /dev/md0
>> /dev/md0:
>> Version : 1.2
>> Creation Time : Thu Aug 4 18:42:19 2016
>> Raid Level : raid4
>> Array Size : 3139584 (2.99 GiB 3.21 GB)
>> Used Dev Size : 523264 (511.00 MiB 535.82 MB)
>> Raid Devices : 7
>> Total Devices : 7
>> Persistence : Superblock is persistent
>>
>> Intent Bitmap : Internal
>>
>> Update Time : Thu Aug 4 18:43:22 2016
>> State : clean, degraded
>> Active Devices : 6
>> Working Devices : 7
>> Failed Devices : 0
>> Spare Devices : 1
>>
>> Chunk Size : 512K
>>
>> Name : dhcp-12-125.nay.redhat.com:0 (local to host
>> dhcp-12-125.nay.redhat.com)
>> UUID : 29a267c2:04e91998:639e6eef:64ddbab0
>> Events : 23
>>
>> Number Major Minor RaidDevice State
>> 7 7 0 - journal /dev/loop0
>> 1 7 2 1 active sync /dev/loop2
>> 2 7 3 2 active sync /dev/loop3
>> 3 7 4 3 active sync /dev/loop4
>> 4 7 5 4 active sync /dev/loop5
>> 5 7 6 5 active sync /dev/loop6
>> 8 7 7 6 active sync /dev/loop7
>>
>
> I'm not really sure about the journal/your report, but also why does
> the above show 1 spare device? I would expect spare should mean that
> there is a device available to take over automatically if one of the
> active devices becomes faulty, but that doesn't seem to be the case.
> Should a journal device be counted as Active & Working (or faulty)
> instead of Spare?
Agree, or maybe "Journal Devices" can be added for it (but both kernel
and mdadm need to modify md_u.h I think).
Regards,
Guoqing
^ permalink raw reply
* RAID 6 "Bad block number requested"
From: Jens-U. Mozdzen @ 2016-08-05 15:37 UTC (permalink / raw)
To: linux-raid
Hello list,
one of my server started getting many messages in dmesg, reporting
"Bad block number requested", like
--- cut here ---
[7749965.585075] sd 10:0:0:0: [sda] Bad block number requested
[7750561.279143] sd 10:0:2:0: [sdc] Bad block number requested
[7751481.566408] sd 10:0:2:0: [sdc] Bad block number requested
[7752774.458062] sd 10:0:2:0: [sdc] Bad block number requested
[7754296.938131] sd 10:0:5:0: [sdf] Bad block number requested
[7755557.728901] sd 10:0:4:0: [sde] Bad block number requested
[7756230.809538] sd 10:0:5:0: [sdf] Bad block number requested
--- cut here ---
I've had about 10k of those messages during the past 10 hours alone.
These messages appear only for those devices that run a MD-RAID 6 and
seem to be almost evenly distributed across all related devices
--- cut here ---
1680 [sda]
1790 [sdb]
1755 [sdc]
1855 [sdd]
1695 [sde]
1700 [sdf]
--- cut here ---
But interestingly. the incidents are not spread evenly across the day,
here's what I have for the last few hours:
--- cut here ---
9029 05
1224 06
59 07
33 09
31 10
29 11
53 12
5 13
2 14
7 15
--- cut here ---
So the majority of these occurred from 5:00 to 5:59 AM. There's no
rate increase while I run RAID checks.
Neither "mdadm --examine" for the devices nor "mdadm --detail" for the
RAID device show anything out of place. I'll add these infos once
somebody mentions that MD-RAID might be the source of the messages.
I had check runs on this RAID6 (echo check >
/sys/block/md126/md/sync_action), no problems were returned. No other
evident problems can be seen, either - no failing writes, no file
system corruption, nothing.
What baffles me: As it is a RAID6, MD should control all writes,
shouldn't it? Or can it be that some upper layer tries to write beyond
the end of the RAID device, resulting in the reported syslog messages?
Some words about the stack this server runs:
- sda1-sdf1 assemble to /dev/md126, RAID6 from 6 Seagate ST1000NX0323
(SAS HDDs)
- sdg1,sdh1 assemble to /dev/md127, RAID1 from two Toshiba PX02SMF020
(SAS SSDs)
- md126,127 joined as a bcache'd device (/dev/bcache0)
- /dev/bcache0 is the only PV in an LVM
- some LVs are used directly (OS partitions, Ceph OSDs)
- some LVs are local storage for a DRBD setup (this node being primary)
We have a second server with identical hardware and software stack,
just different LVs - I see none of these problems there.
Can someone confirm/deny that MD tests against writes beyond the end
of the /dev/md* devices? Or do I have to check the upper layers to see
if someone tries to write outside the available space?
Regards,
Jens
^ permalink raw reply
* Unable to convert raid1 to raid5
From: Wols Lists @ 2016-08-05 20:16 UTC (permalink / raw)
To: linux-raid
In my testing of xosview, I've been mucking about with a vm and raid.
xosview is looking quite promising (I've got a few comments about it,
but never mind).
BUT. In mucking about with raid 1, I increased my raid devices to three.
I now just can NOT convert the array to raid 5! I've been mucking around
with all sorts of things trying to get it to work, but finally two error
messages make things clear.
# mdadm /dev/md/testarray --grow --level=5 --backup-file=raidbackup
mdadm: Can only convert a 2-device array to RAID5
#
Okay. That shound't be a problem. BUT
# mdadm /dev/md/testarray --grow -n2
mdadm: failed to set raid disks
#
and I get that error even when I've failed and removed the third device.
Any ideas? Oh yes - I *have* downloaded the latest mdadm from Neil's git
repository :-)
I'm running SuSE leap 42.1, and I've disabled AppArmor (yes, I remember
SELinux mucking things up - I didn't want AppArmor doing the same ...)
Cheers,
Wol
^ permalink raw reply
* Re: migration of raid 5 to raid 6 and disk of 2TB to 4TB
From: bobzer @ 2016-08-05 23:32 UTC (permalink / raw)
To: Peter Grandi, Wols Lists; +Cc: Linux RAID
In-Reply-To: <22424.53061.186858.220134@tree.ty.sabi.co.uk>
thank you very much for all your help
I manage to borrow 2 3TB drive
So i was thinking of dd copie my image to one, use the other to
reconstruct my raid
so at this point i got my first raid5 clean
then i prepare my raid6 degraded with the the 3 4TB and do a rsync
another solution could be to move the image disk to one 3TB
create the raid6 degraded with the 3 4TB
move the image in the raid 6
replace the 2 2TB with the 2 3TB (with a DD copy ?)
reuse the 2 2TB to have a clean raid 6
rsync from the degraded raid5 to the clean raid6
I prefere the first solution because the raid 5 is clean and so it
should be safer but i worrying about the stress of rebuild the raid
i like the second solution because i don't need to rebuild the raid 5
and so i don't take the risk that my 2T disk died
I mean, they should be ok but i prefere to just have to read on them
than to write for the rebuild (last time it's the stress of the
rebuild which destroy my raid...)
the interest is that there if one 2TB died it more likely to happend
when is in the raid 6 not when i read from them to copy in the 3T, and
when my data are in 3tb they should be safe
there is plenty more solution available to me now that i got these 2
drives except that i have to give then back as soon as possible
anyway i still don't know which scenario is the best
but thanks for your help
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox