* [RFC PATCH 0/7] evacuate struct page from the block layer
From: Dan Williams @ 2015-03-16 20:25 UTC (permalink / raw)
To: linux-kernel
Cc: linux-arch, axboe, riel, linux-nvdimm, Dave Hansen, linux-raid,
mgorman, hch, linux-fsdevel, Matthew Wilcox
Avoid the impending disaster of requiring struct page coverage for what
is expected to be ever increasing capacities of persistent memory. In
conversations with Rik van Riel, Mel Gorman, and Jens Axboe at the
recently concluded Linux Storage Summit it became clear that struct page
is not required in many places, it was simply convenient to re-use.
Introduce helpers and infrastructure to remove struct page usage where
it is not necessary. One use case for these changes is to implement a
write-back-cache in persistent memory for software-RAID. Another use
case for the scatterlist changes is RDMA to a pfn-range.
This compiles and boots, but 0day-kbuild-robot coverage is needed before
this set exits "RFC". Obviously, the coccinelle script needs to be
re-run on the block updates for kernel.next. As is, this only includes
the resulting auto-generated-patch against 4.0-rc3.
---
Dan Williams (6):
block: add helpers for accessing a bio_vec page
block: convert bio_vec.bv_page to bv_pfn
dma-mapping: allow archs to optionally specify a ->map_pfn() operation
scatterlist: use sg_phys()
x86: support dma_map_pfn()
block: base support for pfn i/o
Matthew Wilcox (1):
scatterlist: support "page-less" (__pfn_t only) entries
arch/Kconfig | 3 +
arch/arm/mm/dma-mapping.c | 2 -
arch/microblaze/kernel/dma.c | 2 -
arch/powerpc/sysdev/axonram.c | 2 -
arch/x86/Kconfig | 12 +++
arch/x86/kernel/amd_gart_64.c | 22 ++++--
arch/x86/kernel/pci-nommu.c | 22 ++++--
arch/x86/kernel/pci-swiotlb.c | 4 +
arch/x86/pci/sta2x11-fixup.c | 4 +
arch/x86/xen/pci-swiotlb-xen.c | 4 +
block/bio-integrity.c | 8 +-
block/bio.c | 83 +++++++++++++++------
block/blk-core.c | 9 ++
block/blk-integrity.c | 7 +-
block/blk-lib.c | 2 -
block/blk-merge.c | 15 ++--
block/bounce.c | 26 +++----
drivers/block/aoe/aoecmd.c | 8 +-
drivers/block/brd.c | 2 -
drivers/block/drbd/drbd_bitmap.c | 5 +
drivers/block/drbd/drbd_main.c | 4 +
drivers/block/drbd/drbd_receiver.c | 4 +
drivers/block/drbd/drbd_worker.c | 3 +
drivers/block/floppy.c | 6 +-
drivers/block/loop.c | 8 +-
drivers/block/nbd.c | 8 +-
drivers/block/nvme-core.c | 2 -
drivers/block/pktcdvd.c | 11 ++-
drivers/block/ps3disk.c | 2 -
drivers/block/ps3vram.c | 2 -
drivers/block/rbd.c | 2 -
drivers/block/rsxx/dma.c | 3 +
drivers/block/umem.c | 2 -
drivers/block/zram/zram_drv.c | 10 +--
drivers/dma/ste_dma40.c | 5 -
drivers/iommu/amd_iommu.c | 21 ++++-
drivers/iommu/intel-iommu.c | 26 +++++--
drivers/iommu/iommu.c | 2 -
drivers/md/bcache/btree.c | 4 +
drivers/md/bcache/debug.c | 6 +-
drivers/md/bcache/movinggc.c | 2 -
drivers/md/bcache/request.c | 6 +-
drivers/md/bcache/super.c | 10 +--
drivers/md/bcache/util.c | 5 +
drivers/md/bcache/writeback.c | 2 -
drivers/md/dm-crypt.c | 12 ++-
drivers/md/dm-io.c | 2 -
drivers/md/dm-verity.c | 2 -
drivers/md/raid1.c | 50 +++++++------
drivers/md/raid10.c | 38 +++++-----
drivers/md/raid5.c | 6 +-
drivers/mmc/card/queue.c | 4 +
drivers/s390/block/dasd_diag.c | 2 -
drivers/s390/block/dasd_eckd.c | 14 ++--
drivers/s390/block/dasd_fba.c | 6 +-
drivers/s390/block/dcssblk.c | 2 -
drivers/s390/block/scm_blk.c | 2 -
drivers/s390/block/scm_blk_cluster.c | 2 -
drivers/s390/block/xpram.c | 2 -
drivers/scsi/mpt2sas/mpt2sas_transport.c | 6 +-
drivers/scsi/mpt3sas/mpt3sas_transport.c | 6 +-
drivers/scsi/sd_dif.c | 4 +
drivers/staging/android/ion/ion_chunk_heap.c | 4 +
drivers/staging/lustre/lustre/llite/lloop.c | 2 -
drivers/xen/biomerge.c | 4 +
drivers/xen/swiotlb-xen.c | 29 +++++--
fs/btrfs/check-integrity.c | 6 +-
fs/btrfs/compression.c | 12 ++-
fs/btrfs/disk-io.c | 4 +
fs/btrfs/extent_io.c | 8 +-
fs/btrfs/file-item.c | 8 +-
fs/btrfs/inode.c | 18 +++--
fs/btrfs/raid56.c | 4 +
fs/btrfs/volumes.c | 2 -
fs/buffer.c | 4 +
fs/direct-io.c | 2 -
fs/exofs/ore.c | 4 +
fs/exofs/ore_raid.c | 2 -
fs/ext4/page-io.c | 2 -
fs/f2fs/data.c | 4 +
fs/f2fs/segment.c | 2 -
fs/gfs2/lops.c | 4 +
fs/jfs/jfs_logmgr.c | 4 +
fs/logfs/dev_bdev.c | 10 +--
fs/mpage.c | 2 -
fs/splice.c | 2 -
include/asm-generic/dma-mapping-common.h | 30 ++++++++
include/asm-generic/memory_model.h | 4 +
include/asm-generic/scatterlist.h | 6 ++
include/crypto/scatterwalk.h | 10 +++
include/linux/bio.h | 24 +++---
include/linux/blk_types.h | 21 +++++
include/linux/blkdev.h | 2 +
include/linux/dma-debug.h | 23 +++++-
include/linux/dma-mapping.h | 8 ++
include/linux/scatterlist.h | 101 ++++++++++++++++++++++++--
include/linux/swiotlb.h | 5 +
kernel/power/block_io.c | 2 -
lib/dma-debug.c | 4 +
lib/swiotlb.c | 20 ++++-
mm/iov_iter.c | 22 +++---
mm/page_io.c | 8 +-
net/ceph/messenger.c | 2 -
103 files changed, 658 insertions(+), 335 deletions(-)
^ permalink raw reply
* Re: [PATCH] kernfs: handle poll correctly on 'direct_read' files.
From: Tejun Heo @ 2015-03-16 16:30 UTC (permalink / raw)
To: NeilBrown
Cc: Greg Kroah-Hartman, linux-kernel, Prakash Punnoor, Torsten Kaiser,
linux RAID
In-Reply-To: <20150316104452.366bffc0@notabene.brown>
On Mon, Mar 16, 2015 at 10:44:52AM +1100, NeilBrown wrote:
>
>
> Kernfs supports two styles of read: direct_read and seqfile_read.
>
> The latter supports 'poll' correctly thanks to the update of
> '->event' in kernfs_seq_show.
> The former does not as '->event' is never updated on a read.
>
> So add an appropriate update in kernfs_file_direct_read().
>
> This was noticed because some 'md' sysfs attributes were
> recently changed to use direct reads.
>
> Reported-by: Prakash Punnoor <prakash@punnoor.de>
> Reported-by: Torsten Kaiser <just.for.lkml@googlemail.com>
> Fixes: 750f199ee8b578062341e6ddfe36c59ac8ff2dcb
This fixes tag is kinda weird without further explanation.
> Signed-off-by: NeilBrown <neilb@suse.de>
Acked-by: Tejun Heo <tj@kernel.org>
Thanks.
--
tejun
^ permalink raw reply
* Bad Block Management in RAID1 and "Faulty"
From: Ankur Bose @ 2015-03-16 9:34 UTC (permalink / raw)
To: linux-raid; +Cc: neilb
Hi Group , I would like to understand more about bad block management (
esp RAID1 )that given the below link
http://neil.brown.name/blog/20100519043730 . Highly appreciate any kind
of help.
1) What happens when a write comes on a known bad block ?.
2) When exactly the raid1 decides to make the device "Faulty"? Does that
depends on the number of bad blocks in the list ie: 512?
Thanks,
Ankur
^ permalink raw reply
* [PATCH] kernfs: handle poll correctly on 'direct_read' files.
From: NeilBrown @ 2015-03-15 23:44 UTC (permalink / raw)
To: Greg Kroah-Hartman, Tejun Heo
Cc: linux-kernel, Prakash Punnoor, Torsten Kaiser, linux RAID
[-- Attachment #1: Type: text/plain, Size: 994 bytes --]
Kernfs supports two styles of read: direct_read and seqfile_read.
The latter supports 'poll' correctly thanks to the update of
'->event' in kernfs_seq_show.
The former does not as '->event' is never updated on a read.
So add an appropriate update in kernfs_file_direct_read().
This was noticed because some 'md' sysfs attributes were
recently changed to use direct reads.
Reported-by: Prakash Punnoor <prakash@punnoor.de>
Reported-by: Torsten Kaiser <just.for.lkml@googlemail.com>
Fixes: 750f199ee8b578062341e6ddfe36c59ac8ff2dcb
Signed-off-by: NeilBrown <neilb@suse.de>
diff --git a/fs/kernfs/file.c b/fs/kernfs/file.c
index b684e8a132e6..2bacb9988566 100644
--- a/fs/kernfs/file.c
+++ b/fs/kernfs/file.c
@@ -207,6 +207,7 @@ static ssize_t kernfs_file_direct_read(struct kernfs_open_file *of,
goto out_free;
}
+ of->event = atomic_read(&of->kn->attr.open->event);
ops = kernfs_ops(of->kn);
if (ops->read)
len = ops->read(of, buf, len, *ppos);
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* [PATCH] dm log userspace: base: fix compile warning
From: Nicholas Mc Guire @ 2015-03-15 17:09 UTC (permalink / raw)
To: Alasdair Kergon
Cc: Mike Snitzer, dm-devel, Neil Brown, linux-raid, linux-kernel,
Nicholas Mc Guire
This fixes up a compile warning [-Wunused-but-set-variable] - given the
comments in userspace_set_region_sync() the non-reporting of errors seems
intentional so the return value can be dropped to make gcc happy.
Signed-off-by: Nicholas Mc Guire <hofrat@osadl.org>
---
CC [M] drivers/md/dm-log-userspace-base.o
drivers/md/dm-log-userspace-base.c: In function 'userspace_set_region_sync':
drivers/md/dm-log-userspace-base.c:744:6: warning: variable 'r' set but not used [-Wunused-but-set-variable]
The comment here states:
/*
* It would be nice to be able to report failures.
* However, it is easy emough to detect and resolve.
*/
note though that userspace_do_request() calls dm_consult_userspace()
which can also return -EINVAL (which is though reported in
dm_consult_userspace via DMINFO if I got it right)
Patch was only compile tested with x86_64_defconfig + CONFIG_DM_LOG_USERSPACE
Patch is against 4.0-rc3 (localversion-next is -next-20150313)
drivers/md/dm-log-userspace-base.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/md/dm-log-userspace-base.c b/drivers/md/dm-log-userspace-base.c
index 39fa007..1e3038e 100644
--- a/drivers/md/dm-log-userspace-base.c
+++ b/drivers/md/dm-log-userspace-base.c
@@ -741,7 +741,6 @@ static int userspace_get_resync_work(struct dm_dirty_log *log, region_t *region)
static void userspace_set_region_sync(struct dm_dirty_log *log,
region_t region, int in_sync)
{
- int r;
struct log_c *lc = log->context;
struct {
region_t r;
@@ -751,7 +750,7 @@ static void userspace_set_region_sync(struct dm_dirty_log *log,
pkg.r = region;
pkg.i = (int64_t)in_sync;
- r = userspace_do_request(lc, lc->uuid, DM_ULOG_SET_REGION_SYNC,
+ userspace_do_request(lc, lc->uuid, DM_ULOG_SET_REGION_SYNC,
(char *)&pkg, sizeof(pkg), NULL, NULL);
/*
--
1.7.10.4
^ permalink raw reply related
* [PATCH] dm log userspace: transfer: match wait_for_completion_timeout return type
From: Nicholas Mc Guire @ 2015-03-15 17:09 UTC (permalink / raw)
To: Alasdair Kergon
Cc: Mike Snitzer, dm-devel, Neil Brown, linux-raid, linux-kernel,
Nicholas Mc Guire
Return type of wait_for_completion_timeout is unsigned long not int. An
appropriately named unsigned long is added and the assignments fixed up.
Signed-off-by: Nicholas Mc Guire <hofrat@osadl.org>
---
Patch was only compile tested with x86_64_defconfig + CONFIG_DM_LOG_USERSPACE
Patch is against 4.0-rc3 (localversion-next is -next-20150313)
drivers/md/dm-log-userspace-transfer.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-log-userspace-transfer.c b/drivers/md/dm-log-userspace-transfer.c
index 39ad966..c2050a1 100644
--- a/drivers/md/dm-log-userspace-transfer.c
+++ b/drivers/md/dm-log-userspace-transfer.c
@@ -172,6 +172,7 @@ int dm_consult_userspace(const char *uuid, uint64_t luid, int request_type,
char *rdata, size_t *rdata_size)
{
int r = 0;
+ unsigned long time_left;
size_t dummy = 0;
int overhead_size = sizeof(struct dm_ulog_request) + sizeof(struct cn_msg);
struct dm_ulog_request *tfr = prealloced_ulog_tfr;
@@ -236,11 +237,12 @@ resend:
goto out;
}
- r = wait_for_completion_timeout(&(pkg.complete), DM_ULOG_RETRY_TIMEOUT);
+ time_left = wait_for_completion_timeout(&(pkg.complete),
+ DM_ULOG_RETRY_TIMEOUT);
spin_lock(&receiving_list_lock);
list_del_init(&(pkg.list));
spin_unlock(&receiving_list_lock);
- if (!r) {
+ if (!time_left) {
DMWARN("[%s] Request timed out: [%u/%u] - retrying",
(strlen(uuid) > 8) ?
(uuid + (strlen(uuid) - 8)) : (uuid),
--
1.7.10.4
^ permalink raw reply related
* /dev/md/xxx sporadically unavailable after mdadm --assemble /dev/md/xxx
From: Andrei Borzenkov @ 2015-03-15 7:42 UTC (permalink / raw)
To: linux-raid
I hit this when running test suite that is using Linux MD over loop
devices. Sporadically it fails to mount raid after mdadm --assembly
with
mdadm: array /dev/md/mdraid10_raid1_6 started.
mdadm: stopped /dev/md/mdraid10_raid1_6
mdadm: /dev/md/mdraid10_raid1_6 has been started with 6 drives.
mount: special device /dev/md/mdraid10_raid1_6 does not exist
How naive is the assumption that when "mdadm --assemble" returns
devices are available? Should every use of mdadm be now followed by
waiting for device?
Code is simply
xmdraid*)
mdadm --assemble /dev/md/"${fs}_$NDEVICES" "${LODEVICES[@]}"
sleep 1
mount -t "$MOUNTFS" "${MOUNTDEVICE}" "$MNTPOINTRO" -o ${MOUNTOPTS}${SELINUXOPTS}ro ;;
but apparently even one second wait is not enough. But where is the
limit?
Of course when you look after test failure all devices are there.
Using mdadm 3.3.1 on openSUSE 13.2.
^ permalink raw reply
* PLEASE GET BACK TO ME AS SOON AS YOU RECEIVE THIS EMAIL
From: Mkhize Mthembu @ 2015-03-13 21:44 UTC (permalink / raw)
In-Reply-To: <1368102510.11588.1426283034729.JavaMail.yahoo@mail.yahoo.com>
[-- Attachment #1: Type: text/plain, Size: 1 bytes --]
[-- Attachment #2: Mkhize Mthembu.docx --]
[-- Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document, Size: 12966 bytes --]
^ permalink raw reply
* Re: Inject I/O latency for RAID5/6 read and writes
From: Sebastian Parschauer @ 2015-03-13 10:29 UTC (permalink / raw)
To: Alireza Haghdoost, Linux RAID; +Cc: Neil Brown
In-Reply-To: <CAB-428=f7eL3iD=0g9MsYV78KUng_dysEs+=70taPP9wh_UkbQ@mail.gmail.com>
On 12.03.2015 22:31, Alireza Haghdoost wrote:
> On Mon, Feb 23, 2015 at 12:32 PM, Alireza Haghdoost <alireza@cs.umn.edu> wrote:
>> I needed to inject I/O completion latency in the RAID5/6 codes for
>> test purpose. I was wondering where would be the good place in
>> md/raid5.c code to add delay ?
>>
>> So far I have tried adding mdelay/udelay in raid5_end_write_request()
>> and it seems it works to increase I/O completion of writes. However,
>> adding delay in raid5_end_read_request() does not really change the
>> read I/O latency. Any idea ?
>>
>> --Alireza
>
> Can Some one help me on this issue ?
Everything comes through md_make_request() in drivers/md/md.c but there
are multiple different bio->bi_end_io functions. We have implemented
custom I/O latency measurement. I've posted the patch to this list some
time ago. So feel free to look how it works as an inspiration.
http://www.spinics.net/lists/raid/msg46525.html
It introduces its own struct md_bio_private containing the original
bi_end_io function pointer and introduces the md_bio_endio() function as
a generic bi_end_io to be always called before calling the original
bi_end_io function.
I hope this helps! We've reworked this stuff since then a bit more but
it's really beneficial to detect disks which became slow.
Cheers,
Sebastian
^ permalink raw reply
* Bad block management in raid1
From: Ankur Bose @ 2015-03-13 10:06 UTC (permalink / raw)
To: linux-raid; +Cc: Suresh Babu Kandukuru
In-Reply-To: <55019940.4030104@turmel.org>
Hi There,
Can you conform the below scenario in which blocks are consider to be
a "bad" block.
1. A read error on a degraded array ( a state of raid when
array experiences the failure of one or more disks)for which the data
cannot be found from other legs is a "bad" block and gets recorded.
2. When recovering, from source to target leg, for any reason
if unable to read from source, the target leg's block gets recorded as
“bad” (thought the target block is writable and can be used in future).
3. Write to a block fails (Though it leads to degraded mode).
Are they all implemented and is there any other scenario?
When exactly the raid1 decides to make the device "Faulty"? Does that
depends on the number of bad blocks in the list ie: 512?
What is the size in the metadata for storing the bad block info.
Thanks,
Ankur
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Inject I/O latency for RAID5/6 read and writes
From: Alireza Haghdoost @ 2015-03-12 21:31 UTC (permalink / raw)
To: Linux RAID; +Cc: Neil Brown
In-Reply-To: <CAB-428=gcsgxa1k83h0QWmijhe_kXaLMQZqs+m=Zb0OWX06geA@mail.gmail.com>
On Mon, Feb 23, 2015 at 12:32 PM, Alireza Haghdoost <alireza@cs.umn.edu> wrote:
> I needed to inject I/O completion latency in the RAID5/6 codes for
> test purpose. I was wondering where would be the good place in
> md/raid5.c code to add delay ?
>
> So far I have tried adding mdelay/udelay in raid5_end_write_request()
> and it seems it works to increase I/O completion of writes. However,
> adding delay in raid5_end_read_request() does not really change the
> read I/O latency. Any idea ?
>
> --Alireza
Can Some one help me on this issue ?
^ permalink raw reply
* Re: Raid 5: all devices marked spare, cannot assemble
From: Paul Boven @ 2015-03-12 14:28 UTC (permalink / raw)
To: Phil Turmel, linux-raid
In-Reply-To: <55019940.4030104@turmel.org>
Hi Phil,
Good morning and thanks for your quick reply.
On 03/12/2015 02:48 PM, Phil Turmel wrote:
>> I have a rather curious issue with one of our storage machines. The
>> machine has 36x 4TB disks (SuperMicro 847 chassis) which are divided
>> over 4 dual SAS-HBAs and the on-board SAS. These disks are in RAID5
>> configurations, 6 raids of 6 disks each. Recently the machine ran out of
>> memory (it has 32GB, and no swapspace as it boots from SATA-DOM) and the
>> last entries in the syslog are from the OOM-killer. The machine is
>> running Ubuntu 14.04.02 LTS, mdadm 3.2.5-5ubuntu4.1.
>
> {BTW, I think raid5 is *insane* for this size array.}
It's 6 raid5s, not a single big one. This is only a temporary holding
space for data to be processed. In its original incarnation the machine
had 36 distinct file-systems that we would read from in a software
stripe, just to get enough IO performance. So this is a trade-off
between IO-speed and lost capacity versus convenience in case a drive
inevitably fails.
I guess you would recommend raid6? I would have liked a global hot
spare, maybe 7 arrays of 5 disks, but then we lose 8 disks in total
instead of the current 6.
> Wrong syntax. It's already assembled. Just try "mdadm --run /dev/md15"
Trying to 'run' md15 gives me the same errors as before:
md/raid:md15: not clean -- starting background reconstruction
md/raid:md15: device sdad1 operational as raid disk 0
md/raid:md15: device sdy1 operational as raid disk 3
md/raid:md15: device sdv1 operational as raid disk 4
md/raid:md15: device sdm1 operational as raid disk 2
md/raid:md15: device sdq1 operational as raid disk 1
md/raid:md15: allocated 0kB
md/raid:md15: cannot start dirty degraded array.
RAID conf printout:
--- level:5 rd:6 wd:5
disk 0, o:1, dev:sdad1
disk 1, o:1, dev:sdq1
disk 2, o:1, dev:sdm1
disk 3, o:1, dev:sdy1
disk 4, o:1, dev:sdv1
md/raid:md15: failed to run raid set.
md: pers->run() failed ...
> If the simple --run doesn't work, stop the array and force assemble the
> good drives:
>
> mdadm --stop /dev/md15
> mdadm --assemble --force --verbose /dev/md15 /dev/sd{ad,q,m,y,v}1
That worked!
mdadm: looking for devices for /dev/md15
mdadm: /dev/sdad1 is identified as a member of /dev/md15, slot 0.
mdadm: /dev/sdq1 is identified as a member of /dev/md15, slot 1.
mdadm: /dev/sdm1 is identified as a member of /dev/md15, slot 2.
mdadm: /dev/sdy1 is identified as a member of /dev/md15, slot 3.
mdadm: /dev/sdv1 is identified as a member of /dev/md15, slot 4.
mdadm: Marking array /dev/md15 as 'clean'
mdadm: added /dev/sdq1 to /dev/md15 as 1
mdadm: added /dev/sdm1 to /dev/md15 as 2
mdadm: added /dev/sdy1 to /dev/md15 as 3
mdadm: added /dev/sdv1 to /dev/md15 as 4
mdadm: no uptodate device for slot 5 of /dev/md15
mdadm: added /dev/sdad1 to /dev/md15 as 0
mdadm: /dev/md15 has been started with 5 drives (out of 6).
I've checked that the filesystem is in good shape, and added /dev/sdd1
back in, the array is now resyncing. 680 minutes to go, but there's a
few tricks I can do to speed that up a bit.
> In other words, unclean shutdowns should have manual intervention,
> unless the array in question contains the root filesystem, in which case
> the risky "start_dirty_degraded" may be appropriate. In that case, you
> probably would want your initramfs to have a special mdadm.conf,
> deferring assembly of bulk arrays to normal userspace.
I'm perfectly happy with doing the recovery in userspace, these drives
are not critical for booting. Except that Ubuntu, Plymouth and a few
other things conspire against booting a machine with any disk problems,
but that's a different rant for a different place.
Thank you very much for your very helpful reply, things look a lot
better now.
Regards, Paul Boven.
--
Paul Boven <boven@jive.nl> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
^ permalink raw reply
* Re: Raid 5: all devices marked spare, cannot assemble
From: Phil Turmel @ 2015-03-12 13:48 UTC (permalink / raw)
To: Paul Boven, linux-raid
In-Reply-To: <550184D4.8060104@jive.nl>
Good morning Paul,
On 03/12/2015 08:21 AM, Paul Boven wrote:
> Hi folks,
>
> I have a rather curious issue with one of our storage machines. The
> machine has 36x 4TB disks (SuperMicro 847 chassis) which are divided
> over 4 dual SAS-HBAs and the on-board SAS. These disks are in RAID5
> configurations, 6 raids of 6 disks each. Recently the machine ran out of
> memory (it has 32GB, and no swapspace as it boots from SATA-DOM) and the
> last entries in the syslog are from the OOM-killer. The machine is
> running Ubuntu 14.04.02 LTS, mdadm 3.2.5-5ubuntu4.1.
{BTW, I think raid5 is *insane* for this size array.}
> After doing a hard reset, the machine booted fine but one of the raids
> needed to resync. Worse, another of the raid5s will not assemble at all.
> All the drives are marked SPARE. Relevant output from /proc/mdstat (one
> working and the broken array):
>
> md14 : active raid5 sdc1[2] sdag1[6] sde1[4] sdi1[3] sdz1[0] sdu1[1]
> 19534425600 blocks super 1.2 level 5, 512k chunk, algorithm 2
> [6/6] [UUUUUU]
>
> md15 : inactive sdd1[6](S) sdad1[0](S) sdy1[3](S) sdv1[4](S) sdm1[2](S)
> sdq1[1](S)
> 23441313792 blocks super 1.2
Although (S) implies spare, that's only true if the array is active.
md15 is assembled but not assembled.
> Using 'mdadm --examine' on each of the drives from the broken md15, I get:
>
> sdd1: Spare, Events: 0
> sdad1: Active device 0, Events 194
> sdy1: Active device 3, Events 194
> sdv1: Active device 4, Events 194
> sdm1: Active device 2, Events 194
> sdq1: Active device 1, Events 194
Please don't trim the reports. This implies that your array simply
didn't --run as it is unexpected degraded.
[trim /]
> md: kicking non-fresh sdd1 from array!
> md: unbind<sdd1>
> md: export_rdev(sdd1)
> md/raid:md15: not clean -- starting background reconstruction
> md/raid:md15: device sdy1 operational as raid disk 3
> md/raid:md15: device sdv1 operational as raid disk 4
> md/raid:md15: device sdad1 operational as raid disk 0
> md/raid:md15: device sdq1 operational as raid disk 1
> md/raid:md15: device sdm1 operational as raid disk 2
> md/raid:md15: allocated 0kB
> md/raid:md15: cannot start dirty degraded array.
Exactly.
> * Why does this raid5 not assemble? Only one drive (sdd) seems to be
> missing (marked spare), although I see no real issues with it and can
> read from it fine. There should still be enough drives to start the array.
>
> # mdadm --assemble /dev/md15 --run
Wrong syntax. It's already assembled. Just try "mdadm --run /dev/md15"
> * How can the data be recovered, and the machine brought into production
> again
If the simple --run doesn't work, stop the array and force assemble the
good drives:
mdadm --stop /dev/md15
mdadm --assemble --force --verbose /dev/md15 /dev/sd{ad,q,m,y,v}1
If that doesn't work, show the complete output of the --assemble.
> * What went wrong, and how can we guard against this?
The crash prevented mdadm from writing the current state of the array to
the individual drives' metadata. You didn't provide complete --examine
output, so I'm speculating, but the drives must disagree on the last
known state of the array ==> "dirty". See "start_dirty_degraded" in man
md(4), and the --run and --no-degraded options to assemble.
In other words, unclean shutdowns should have manual intervention,
unless the array in question contains the root filesystem, in which case
the risky "start_dirty_degraded" may be appropriate. In that case, you
probably would want your initramfs to have a special mdadm.conf,
deferring assembly of bulk arrays to normal userspace.
Phil
^ permalink raw reply
* Raid 5: all devices marked spare, cannot assemble
From: Paul Boven @ 2015-03-12 12:21 UTC (permalink / raw)
To: linux-raid
Hi folks,
I have a rather curious issue with one of our storage machines. The
machine has 36x 4TB disks (SuperMicro 847 chassis) which are divided
over 4 dual SAS-HBAs and the on-board SAS. These disks are in RAID5
configurations, 6 raids of 6 disks each. Recently the machine ran out of
memory (it has 32GB, and no swapspace as it boots from SATA-DOM) and the
last entries in the syslog are from the OOM-killer. The machine is
running Ubuntu 14.04.02 LTS, mdadm 3.2.5-5ubuntu4.1.
After doing a hard reset, the machine booted fine but one of the raids
needed to resync. Worse, another of the raid5s will not assemble at all.
All the drives are marked SPARE. Relevant output from /proc/mdstat (one
working and the broken array):
md14 : active raid5 sdc1[2] sdag1[6] sde1[4] sdi1[3] sdz1[0] sdu1[1]
19534425600 blocks super 1.2 level 5, 512k chunk, algorithm 2
[6/6] [UUUUUU]
md15 : inactive sdd1[6](S) sdad1[0](S) sdy1[3](S) sdv1[4](S) sdm1[2](S)
sdq1[1](S)
23441313792 blocks super 1.2
Using 'mdadm --examine' on each of the drives from the broken md15, I get:
sdd1: Spare, Events: 0
sdad1: Active device 0, Events 194
sdy1: Active device 3, Events 194
sdv1: Active device 4, Events 194
sdm1: Active device 2, Events 194
sdq1: Active device 1, Events 194
This numbering corresponds to how the raid5 was created when I installed
the machine:
mdadm --create /dev/md15 -l 5 -n 6 /dev/sdad1 /dev/sdq1 /dev/sdm1
/dev/sdy1 /dev/sdv1 /dev/sdd1
Possible clues from /var/log/syslog:
md/raid:md13: not clean -- starting background reconstruction
(at 14 seconds uptime).
md15 isn't even mentioned in the boot-time syslog, only once I manually
try to assemble it did I get these errors:
md: kicking non-fresh sdd1 from array!
md: unbind<sdd1>
md: export_rdev(sdd1)
md/raid:md15: not clean -- starting background reconstruction
md/raid:md15: device sdy1 operational as raid disk 3
md/raid:md15: device sdv1 operational as raid disk 4
md/raid:md15: device sdad1 operational as raid disk 0
md/raid:md15: device sdq1 operational as raid disk 1
md/raid:md15: device sdm1 operational as raid disk 2
md/raid:md15: allocated 0kB
md/raid:md15: cannot start dirty degraded array.
RAID conf printout:
--- level:5 rd:6 wd:5
disk 0, o:1, dev:sdad1
disk 1, o:1, dev:sdq1
disk 2, o:1, dev:sdm1
disk 3, o:1, dev:sdy1
disk 4, o:1, dev:sdv1
md/raid:md15: failed to run raid set.
md: pers->run() failed ...
So the questions I'd like to pose are:
* Why does this raid5 not assemble? Only one drive (sdd) seems to be
missing (marked spare), although I see no real issues with it and can
read from it fine. There should still be enough drives to start the array.
# mdadm --assemble /dev/md15 --run
Returns without any error message, but leaves /proc/mdstat unchanged.
* How can the data be recovered, and the machine brought into production
again
And of course
* What went wrong, and how can we guard against this?
Any insights and help are much appreciated.
Regards, Paul Boven.
--
Paul Boven <boven@jive.nl> +31 (0)521-596547
Unix/Linux/Networking specialist
Joint Institute for VLBI in Europe - www.jive.nl
VLBI - It's a fringe science
^ permalink raw reply
* (unknown),
From: pepa6.es @ 2015-03-12 11:49 UTC (permalink / raw)
Proposal,
Respond to my personal email; mrs.zhangxiao1962@outlook.
com
Yours Sincerely.
Mrs. Zhang Xiao (Accounts book Keeper)
Angang
Steel Company Limited
396 Nan Zhong Hua Lu, Tie Dong District Anshan,
Liaoning 114021, China.
^ permalink raw reply
* Re: dm log userspace: use mempool_create_kmalloc_pool()
From: Mike Snitzer @ 2015-03-11 12:58 UTC (permalink / raw)
To: David Rientjes
Cc: Alasdair Kergon, Neil Brown, linux-kernel, linux-raid, dm-devel
In-Reply-To: <alpine.DEB.2.10.1503071642540.2772@chino.kir.corp.google.com>
On Sat, Mar 07 2015 at 7:43pm -0500,
David Rientjes <rientjes@google.com> wrote:
> Mempools created for kmalloc caches should use
> mempool_create_kmalloc_pool().
>
> Cc: Alasdair Kergon <agk@redhat.com>
> Cc: Mike Snitzer <snitzer@redhat.com>
> Cc: Neil Brown <neilb@suse.de>
> Signed-off-by: David Rientjes <rientjes@google.com>
Thanks David, I'll get this queued up.
^ permalink raw reply
* mdraid with device-mapper multipath devices
From: Sushma Gurram @ 2015-03-10 1:43 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
Hi,
Let's say I have a system with two multipath SSDs (dm-0 and dm-1) and a raid0 is created using both.
When the underlying drive for dm-0 is accidentally removed, the multipathd cannot flush the multipath map (because it's being used by mdraid) and hence the dm-0 entry continues to exist and hence mdraid also has the dm-0 entry in "active sync" state. However, I/O to dm-0 would fail since there is no drive underneath, while I/O to dm-1 would succeed.
Now if the failed drive is reinserted, device mapper/multipath would update the dm-0 maps and now dm-0 handles a valid/existing drive. Since mdraid still thinks dm-0 is active sync, I/O continues as if raid0 array was clean.
I haven't tried with other raid levels, but I suspect a similar behavior.
Here are prints in kernel log on drive removal, with drive being part of raid config and without it being part of config
***Removal with dm-X configured as path of md****
Mar 9 17:57:07 ion-ws1lvp4w kernel: [ 2173.868995] mpt2sas0: removing handle(0x0012), sas_addr(0x50011731001395ea)
Mar 9 17:57:07 ion-ws1lvp4w multipathd: sdq: remove path (uevent)
Mar 9 17:57:08 ion-ws1lvp4w multipathd: mpathh: map in use
Mar 9 17:57:08 ion-ws1lvp4w multipathd: mpathh: can't flush
Mar 9 17:57:08 ion-ws1lvp4w multipathd: mpathh: load table [0 3750748848 multipath 0 0 0 0]
Mar 9 17:57:08 ion-ws1lvp4w multipathd: sdq: path removed from map mpathh
***Removal with no configuration***
Mar 9 18:01:32 ion-ws1lvp4w kernel: [ 2438.314508] mpt2sas0: removing handle(0x0012), sas_addr(0x50011731001395ea)
Mar 9 18:01:33 ion-ws1lvp4w multipathd: 65:0: mark as failed
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: remaining active paths: 0
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391320] device-mapper: multipath: Failing path 65:0.
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391336] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391341] quiet_error: 7 callbacks suppressed
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391344] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391385] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.391391] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w multipathd: sdq: remove path (uevent)
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.395616] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.395619] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.395638] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.395640] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.396691] end_request: I/O error, dev dm-7, sector 3750748672
Mar 9 18:01:33 ion-ws1lvp4w kernel: [ 2438.396697] Buffer I/O error on device dm-7, logical block 468843584
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: map flushed
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: stop event checker thread (140284200109824)
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: removed map after removing all paths
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: adding map
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: devmap dm-7 added
Mar 9 18:01:33 ion-ws1lvp4w multipathd: mpathh: adding map
Ideally the dm-X (multipath device) should be removed so that mdraid updates its constituent drive information. dm-X is not being removed because its in use by raid. Any insight into how we could fix this issue?
Thanks,
Sushma
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
^ permalink raw reply
* Re: Raid 10 Issue - Swapping Data from Array to Array [SOLVED]
From: Stefan Lamby @ 2015-03-09 8:51 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
In-Reply-To: <54F9A421.2020907@turmel.org>
Hi Phil.
I did what you suggested and ended up with a working raid 10 array.
Thank you so much for your support and time.
God blees you.
Stefan
> Phil Turmel <philip@turmel.org> hat am 6. März 2015 um 13:57 geschrieben:
>
>
> On 03/06/2015 05:09 AM, Stefan Lamby wrote:
> >
> >
> > This is what I got right now.
> > What do you recommend to do?
>
> 1) pvcreate on the new array
> 2) vgextend to add the new array to the volume group
> 3) pvmove to get the data into the new array
> 4) vgreduce to disconnect the old array from lvm
> 5) pvremove to wipe the old array's lvm meta
> 6) stop the old array
> 7) mdadm --zero-superblock to clear the old members
> 8) mdadm --add to put those members into the new array
>
> Regards,
>
> Phil Turmel
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Raid 10 Issue
From: Wilson, Jonathan @ 2015-03-08 14:59 UTC (permalink / raw)
To: Phil Turmel; +Cc: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <54F8B792.9060206@turmel.org>
On Thu, 2015-03-05 at 15:07 -0500, Phil Turmel wrote:
> On 03/05/2015 12:56 PM, Stefan Lamby wrote:
> > Hello List.
> >
> > I was setting up a new machine using ubuntu 14.04.02 lts using its installer,
> > configuring a raid 10 with 2 disks and lvm on top of it. I was using 2 disks and
> > now I like to add 2 more disks to the array so i want to end up with 4 disks, no
> > spare.
> >
> > Searching the internet I found that I am not able to --grow the array with the
> > mdadm version this ubuntu is using (v3.2.5).
> > Is that right?
> >
> > So I decided to build a new array that way and try to move my data afterwards,
> > which failed:
> > (Is it OK to do it that way or do you recommend another?)
>
> No, you should be able to do this. Probably without any shutdown.
> Please show the full layout of your drives, partitions, and lvm.
>
> I suggest lsdrv[1] for working layouts. If your email is set to use
> utf8, just paste the result in a reply.
>
> Regards,
>
> Phil Turmel
>
> [1] https://github.com/pturmel/lsdrv
OT: What a fantastic little script, a couple of my biggest annoyances
with my set up was that it was a pain to track "os disk designation" to
"serial no." making identifying the physical device a chore.
Also one of my cheap "4 port sata" cards doesn't identify port number on
the device and while it was possible to work it out by tracing device
serial-sata cable-socket then mentally comparing sd*-device serial=port
it was something I was putting off as "a pain" but also being bugged by
my md member devices numbers being out of whack with my sd designations:
sda[4] sdb[2] sdc[1] sdd[3] and with 12 devices the last thing you want
when something is failing is to have to hunt around to work out which
device physically relates to which os identified device.
With this I can now easily mark my disks with a simple set of
identification, likewise the cable on both ends, and make a sketch of
ports and their sequences and which disk in which slot relates.
Obviously if I was starting my system from scratch I would have done all
this from the get go... but after 4 cases, various upgrades and
additions, differing numbers of MB sata ports, and various sata cards
and multiple variations of md's/partition layouts it all became a huge
muddle and one elegant script has made it simple to finally do what I
have put of for far to long.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: two raid issues
From: Phil Turmel @ 2015-03-08 13:52 UTC (permalink / raw)
To: Dave Stevens, linux-raid
In-Reply-To: <20150307144256.17321vlciup2v4kw@webmail.uniserve.com>
Good morning Dave,
On 03/07/2015 05:42 PM, Dave Stevens wrote:
> Hello the raid list,
>
> I have inherited a server set up by people who are no longer around. It
> worked fine until recently and then after a routine update refused to
> boot. I've got the machine in my office and have been examining the
> problem, or rather problems, I think there are two.
Three, at least.
> First, the bootable partition on /dev/sda1 won't successfully boot to a
> xen kernel, kernel version is 2.6.18.something-xen. The intent is to
> boot to a raid-10 array of four 750GB drives, each partitioned into a
> small and a large partition as detailed below.
>
> Boot proceeds normally according to on-screen messages until this:
>
> md: md0: raid array is not clean -- starting background reconstruction
> raid10: not enough operational mirrors for md0
> md: pers -> () failed
Yup. Degraded to the point of not running.
> Immediately after these messages is another stating that an attempt has
> been made to kill init, kernel panic and reboot.
No way to pivot to your root filesystem, so your initramfs gives up.
Depending on the distro, it may be possible to pass a kernel command
line option to drop into a repair shell at that point.
> I've tried to give these messages verbatim but have no way (I think) to
> reproduce them other than manually.
Repair shell, if available.
> So at first I looked around, read through the wiki and found advice to
> NOT write anything to the array, which seems reasonable. I looked at a
> live microknoppix distro called runtime live that gave me a command
> shell to run the examine command with the output below:
LiveCD boot is good, and the following report is very detailed, thanks:
> /dev/sda2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Wed Feb 18 19:28:22 2015
> State : active
> Active Devices : 2
> Working Devices : 2
> Failed Devices : 2
> Spare Devices : 0
> Checksum : 7c76593b - correct
> Events : 32945477
This is important: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 0 8 2 0 active sync /dev/sda2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 0 0 1 active sync
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdb2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Sat Nov 22 13:18:12 2014
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 7d850ca4 - correct
> Events : 32945477
With this: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 1 8 18 1 active sync /dev/sdb2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdc2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Wed Feb 18 19:30:13 2015
> State : active
> Active Devices : 1
> Working Devices : 1
> Failed Devices : 2
> Spare Devices : 0
> Checksum : 7c7659de - correct
> Events : 32945479
And this: ^^^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 2 8 34 2 active sync /dev/sdc2
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> /dev/sdd2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 5
> Preferred Minor : 0
>
> Update Time : Wed Sep 4 07:51:50 2013
> State : active
> Active Devices : 4
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 77c1a3a7 - correct
> Events : 53
Whoa! ^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 3 8 50 3 active sync /dev/sdd2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 8 50 3 active sync /dev/sdd2
> 4 4 8 66 4 spare /dev/sde2
> /dev/sde2:
> Magic : a92b4efc
> Version : 0.90.00
> UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
> Creation Time : Sun Nov 29 15:33:50 2009
> Raid Level : raid10
> Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
> Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
> Raid Devices : 4
> Total Devices : 5
> Preferred Minor : 0
>
> Update Time : Sun Sep 8 13:25:42 2013
> State : clean
> Active Devices : 4
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 0
> Checksum : 77c775be - correct
> Events : 7934
And Whoa again! ^^^^^^
> Layout : near=2
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 3 8 66 3 active sync /dev/sde2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 8 66 3 active sync /dev/sde2
>
> This makes sense to me as far as it goes but I don't see what to do
> next. As I understand it the four partitions from sda2 to sdd2 would
> form the array with sde as hot spare. It has been my assumption that if
> a drive failed that sde would sync and take over. I don't know if this
> is in fact the case and don't see a path forward. Of course the backups
> are inadequate.
Based on the events and update timestamps, sdd died sometime around Wed
Sep 4 07:51:50 2013, at which point sde stepped in. It too failed
shortly after ~ Sun Sep 8 13:25:42 2013. You then ran degraded for over
a year until sdb also failed ~ Sat Nov 22 13:18:12 2014. You were then
running doubly-degraded (luckily on non-adjacent members) until this Feb
14 when sda was booted out. Leaving only one running drive.
{ I wouldn't keep such people around, either. }
Your best bet is to force assembly of the last two working drives to get
the system running, then take an immediate backup of all critical files.
Do the forced assembly with the livecd, then do a clean shutdown. You
should then be able to boot the original OS and take your backup.
Then you need to completely rebuild your system with proper log
monitoring, array monitoring, and verification of your drives.
Phil
^ permalink raw reply
* [patch] dm log userspace: use mempool_create_kmalloc_pool()
From: David Rientjes @ 2015-03-08 0:43 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer
Cc: Neil Brown, linux-kernel, linux-raid, dm-devel
Mempools created for kmalloc caches should use
mempool_create_kmalloc_pool().
Cc: Alasdair Kergon <agk@redhat.com>
Cc: Mike Snitzer <snitzer@redhat.com>
Cc: Neil Brown <neilb@suse.de>
Signed-off-by: David Rientjes <rientjes@google.com>
---
drivers/md/dm-log-userspace-base.c | 19 ++++---------------
1 file changed, 4 insertions(+), 15 deletions(-)
diff --git a/drivers/md/dm-log-userspace-base.c b/drivers/md/dm-log-userspace-base.c
--- a/drivers/md/dm-log-userspace-base.c
+++ b/drivers/md/dm-log-userspace-base.c
@@ -74,18 +74,6 @@ struct log_c {
uint32_t integrated_flush;
};
-static mempool_t *flush_entry_pool;
-
-static void *flush_entry_alloc(gfp_t gfp_mask, void *pool_data)
-{
- return kmalloc(sizeof(struct flush_entry), gfp_mask);
-}
-
-static void flush_entry_free(void *element, void *pool_data)
-{
- kfree(element);
-}
-
static int userspace_do_request(struct log_c *lc, const char *uuid,
int request_type, char *data, size_t data_size,
char *rdata, size_t *rdata_size)
@@ -537,6 +525,8 @@ static int flush_by_group(struct log_c *lc, struct list_head *flush_list,
return r;
}
+static mempool_t *flush_entry_pool;
+
/*
* userspace_flush
*
@@ -886,9 +876,8 @@ static int __init userspace_dirty_log_init(void)
{
int r = 0;
- flush_entry_pool = mempool_create(100, flush_entry_alloc,
- flush_entry_free, NULL);
-
+ flush_entry_pool = mempool_create_kmalloc_pool(100,
+ sizeof(struct flush_entry));
if (!flush_entry_pool) {
DMWARN("Unable to create flush_entry_pool: No memory.");
return -ENOMEM;
^ permalink raw reply
* two raid issues
From: Dave Stevens @ 2015-03-07 22:42 UTC (permalink / raw)
To: linux-raid
Hello the raid list,
I have inherited a server set up by people who are no longer around.
It worked fine until recently and then after a routine update refused
to boot. I've got the machine in my office and have been examining the
problem, or rather problems, I think there are two.
First, the bootable partition on /dev/sda1 won't successfully boot to
a xen kernel, kernel version is 2.6.18.something-xen. The intent is to
boot to a raid-10 array of four 750GB drives, each partitioned into a
small and a large partition as detailed below.
Boot proceeds normally according to on-screen messages until this:
md: md0: raid array is not clean -- starting background reconstruction
raid10: not enough operational mirrors for md0
md: pers -> () failed
Immediately after these messages is another stating that an attempt
has been made to kill init, kernel panic and reboot.
I've tried to give these messages verbatim but have no way (I think)
to reproduce them other than manually.
So at first I looked around, read through the wiki and found advice to
NOT write anything to the array, which seems reasonable. I looked at a
live microknoppix distro called runtime live that gave me a command
shell to run the examine command with the output below:
/dev/sda2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Wed Feb 18 19:28:22 2015
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 2
Spare Devices : 0
Checksum : 7c76593b - correct
Events : 32945477
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 0 8 2 0 active sync /dev/sda2
0 0 8 2 0 active sync /dev/sda2
1 1 0 0 1 active sync
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
/dev/sdb2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sat Nov 22 13:18:12 2014
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 7d850ca4 - correct
Events : 32945477
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 1 8 18 1 active sync /dev/sdb2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
/dev/sdc2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Wed Feb 18 19:30:13 2015
State : active
Active Devices : 1
Working Devices : 1
Failed Devices : 2
Spare Devices : 0
Checksum : 7c7659de - correct
Events : 32945479
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 2 8 34 2 active sync /dev/sdc2
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
/dev/sdd2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Update Time : Wed Sep 4 07:51:50 2013
State : active
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Checksum : 77c1a3a7 - correct
Events : 53
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 3 8 50 3 active sync /dev/sdd2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 8 50 3 active sync /dev/sdd2
4 4 8 66 4 spare /dev/sde2
/dev/sde2:
Magic : a92b4efc
Version : 0.90.00
UUID : a9bde90a:77abaef6:6c6fe013:77d6cdaf
Creation Time : Sun Nov 29 15:33:50 2009
Raid Level : raid10
Used Dev Size : 732467456 (698.54 GiB 750.05 GB)
Array Size : 1464934912 (1397.07 GiB 1500.09 GB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Update Time : Sun Sep 8 13:25:42 2013
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 77c775be - correct
Events : 7934
Layout : near=2
Chunk Size : 256K
Number Major Minor RaidDevice State
this 3 8 66 3 active sync /dev/sde2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 8 66 3 active sync /dev/sde2
This makes sense to me as far as it goes but I don't see what to do
next. As I understand it the four partitions from sda2 to sdd2 would
form the array with sde as hot spare. It has been my assumption that
if a drive failed that sde would sync and take over. I don't know if
this is in fact the case and don't see a path forward. Of course the
backups are inadequate.
Any and all ideas welcome.
Dave
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: Raid 10 Issue - Booting in case raid failed
From: Phil Turmel @ 2015-03-06 20:12 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <322760937.22899.1425668800663.JavaMail.open-xchange@app04.ox.hosteurope.de>
On 03/06/2015 02:06 PM, Stefan Lamby wrote:
> Hi list.
>
> If everything will work out OK, I will end up with an raid 10 array with 4
> devices.
>
> My partition design and layout structure will be found at the end, if needed.
>
> There are a few questions left for me in case I have to boot with a failed disk:
>
> 1) As you might have seen from the partition design, only partition sda1 has the
> boot flag set. As far as I guess, the ubuntu installer was using grub-install
> only for sda. I am kind of afraid what will happen, in case sda will fail in the
> future. Will it be a good idea to grub-install to all the other devices also?
Yes, sort of.
> 2) What about the boot flag, if I need to grub-install the other devices also?
> Should it be O or 1? Do I have to leave it set to false and in case things go
> wrong boot from a live cd and set it to on to boot from another device?
Set to true.
But the "sort of" comes from your reliance on grub support for MD raid,
and having the appropriate mirrors containing the boot folder. You've
left enough space before your first partition (48g) to easily hold a
plain raid1 x4 boot partition and a raid6 root partition (use a small
chunk size for that). Then your system could boot with any two drives
missing, and let you know what's possible with the large raid10.
Also note that this kind of boot redundancy only helps if the bad drive
is entirely missing at boot time. If you really need boot redundancy,
you have fewer choices: BIOS fakeraid, or hardware raid with a BIOS
extension, or EFI boot with a monolithic kernel/initramfs on each device.
HTH,
Phil
^ permalink raw reply
* Re: raid 10 - Recovering raid - what stats are most helpfull?
From: Phil Turmel @ 2015-03-06 19:50 UTC (permalink / raw)
To: Stefan Lamby, linux-raid@vger.kernel.org
In-Reply-To: <270478433.23047.1425669790387.JavaMail.open-xchange@app04.ox.hosteurope.de>
On 03/06/2015 02:23 PM, Stefan Lamby wrote:
> What information about the raid would you suggest to collect and to save
> somewhere to have it handy in case it is needed?
Most important is the complete "mdadm -E" output for every member
device. With that you need a list that correlates the member device
names to the underlying drive serial numbers. An *excerpt* from "ls -l
/dev/disk/by-id/" would suffice. Or lsdrv output. I wrote lsdrv to
generate a functional summary that includes all of the useful serial
numbers.
Regards,
Phil Turmel
--
200 Creekwood Trail
Fayetteville, GA 30214
Mobile: (404) 713-7284
Home: (770) 719-0684
^ permalink raw reply
* Re: raid 10 - Recovering raid - what stats are most helpfull?
From: Stefan Lamby @ 2015-03-06 19:23 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
In-Reply-To: <54F85645.9000603@turmel.org>
> Phil Turmel <philip@turmel.org> hat am 5. März 2015 um 14:12 geschrieben:
>
>
> On 03/05/2015 03:18 AM, Stefan Lamby wrote:
> >
> > Hi.
> > Is it OK to ask for support here to this list or is it the wrong place?
> > If so, could you please give a hint where to go?
>
> This is the right place. Post your array details (mdadm -D), member
> details (mdadm -E) and a summary of what happened. Include excerpts
> from your dmesg and/or syslogs that look like they might be relevant.
>
> These should all be text files, so pasting them directly into your email
> would be preferred.
>
> Regards,
>
> Phil Turmel
>
>
Hi Phil,
hello List.
This would be also very interesting - just in case:
Hopefully it will never happen, but if the raid 10 array crashes, what
information will be the most usefull for later recovery? Is this the output of
cat /proc/mdstat or as mentioned by Phil the output of mdadm --detail or what
ever?
I would like to be prepared and write a small shell script which will write this
to a file to be included in the backups.
What information about the raid would you suggest to collect and to save
somewhere to have it handy in case it is needed?
Thank you very much for your help.
Stefan
P.S.: Phil, thank you so much for the hints regarding moving the data. It looks
very promising. I'll know more in detail tomorrow. pvmove is taking all night, I
guess.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox