* Re: /dev/md0 can't be created
From: NeilBrown @ 2015-03-25 6:35 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <12135556.2792023.1427264134774.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1415 bytes --]
On Wed, 25 Mar 2015 02:15:34 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> Hi all
>
> I have encountered so many times, the raid device is created successfully, but the directory
> /dev/md0 can't be created. It can't reproduce 100%.
>
> [root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
> Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
> 1788416 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> unused devices: <none>
> [root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
> ls: cannot access /dev/md0: No such file or directory
>
> The underline devices are loop devices which are created with big file.
>
> The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm - v3.3.2 - 21st August 2014)
> I'll try to reproduce this with upstream kernel and mdadm. But I think it shouldn't be the problem about kernel.
>
> What do you think I should check for this? And which tool is responsible for creating the directory? Maybe
> I can add some log to it to find the reason.
>
/dev/md0 is created by udev.
Run
udevadm monitor
to see the events that udev is processing. When and ADD event for "md0" is
processed, /dev/md0 should get created.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* /dev/md0 can't be created
From: Xiao Ni @ 2015-03-25 6:15 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1215809785.2787621.1427263359309.JavaMail.zimbra@redhat.com>
Hi all
I have encountered so many times, the raid device is created successfully, but the directory
/dev/md0 can't be created. It can't reproduce 100%.
[root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3] loop2[2] loop1[1] loop0[0]
1788416 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>
[root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
ls: cannot access /dev/md0: No such file or directory
The underline devices are loop devices which are created with big file.
The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm - v3.3.2 - 21st August 2014)
I'll try to reproduce this with upstream kernel and mdadm. But I think it shouldn't be the problem about kernel.
What do you think I should check for this? And which tool is responsible for creating the directory? Maybe
I can add some log to it to find the reason.
Best Regards
Xiao
^ permalink raw reply
* Re: [PATCH] Fix bitmap offset calculations
From: NeilBrown @ 2015-03-25 2:15 UTC (permalink / raw)
To: Goldwyn Rodrigues; +Cc: linux-raid
In-Reply-To: <20150324162905.GA16104@shrek.lan>
[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]
On Tue, 24 Mar 2015 11:29:05 -0500 Goldwyn Rodrigues <rgoldwyn@suse.de> wrote:
> The calculations of bitmap offset is incorrect with respect to bits to bytes
> conversion.
>
> Also, remove an irrelevant duplicate message.
>
> Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
> ---
> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
> index ac79fef..e98db04 100644
> --- a/drivers/md/bitmap.c
> +++ b/drivers/md/bitmap.c
> @@ -575,7 +575,9 @@ re_read:
>
> sector_div(bm_blocks,
> bitmap->mddev->bitmap_info.chunksize >> 9);
> - bm_blocks = bm_blocks << 3;
> + /* bits to bytes */
> + bm_blocks = ((bm_blocks+7) >> 3) + sizeof(bitmap_super_t);
> + /* to 4k blocks */
> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
> bitmap->mddev->bitmap_info.offset += bitmap->cluster_slot * (bm_blocks << 3);
> pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
> @@ -672,9 +674,6 @@ out:
> goto out_no_sb;
> }
> bitmap->cluster_slot = md_cluster_ops->slot_number(bitmap->mddev);
> - pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
> - bitmap->cluster_slot,
> - (unsigned long long)bitmap->mddev->bitmap_info.offset);
> goto re_read;
> }
>
Applied, thanks.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: MD request counter broken on 4.0-rc5 RAID1
From: NeilBrown @ 2015-03-25 0:46 UTC (permalink / raw)
To: Simon Kirby; +Cc: linux-raid
In-Reply-To: <20150325000403.GA27368@hostway.ca>
[-- Attachment #1: Type: text/plain, Size: 2085 bytes --]
On Tue, 24 Mar 2015 17:04:03 -0700 Simon Kirby <sim@hostway.ca> wrote:
> Hello :)
>
> I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
> It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
> the other two being write-mostly:
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
> md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
>
> This was working on 3.18. Shall I bisect, or is it obvious?
Almost certainly 18c0b223cf9901727ef3b02da6711ac930b4e5d4 in 3.19.
It adds 'part_inc_in_flight' and never decrements it.
Complain to the author.
Thanks,
NeilBrown
>
> It seems to just be leaking as the counters were going up gradually while
> it was rebuilding.
>
> Personalities : [linear] [raid0] [raid1] [raid10]
> md1 : active raid1 sdb2[1](W) sda2[0](W) sdc2[2]
> 530048 blocks [3/3] [UUU]
>
> md2 : active raid1 sdb3[0](W) sdc3[2] sda3[1](W)
> 153613440 blocks [3/3] [UUU]
>
> md0 : active raid1 sdb1[0](W) sdc1[2] sda1[1](W)
> 2104384 blocks [3/3] [UUU]
>
> unused devices: <none>
>
> Simo-
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* MD request counter broken on 4.0-rc5 RAID1
From: Simon Kirby @ 2015-03-25 0:04 UTC (permalink / raw)
To: linux-raid
Hello :)
I'm seeing "iostat -x -k 1" print this after a RAID1 rebuild on 4.0-rc5.
It's not abnormal other than it's 3-disk, with one being SSD (sdc) and
the other two being write-mostly:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 345.00 0.00 0.00 0.00 0.00 100.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 58779.00 0.00 0.00 0.00 0.00 100.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
This was working on 3.18. Shall I bisect, or is it obvious?
It seems to just be leaking as the counters were going up gradually while
it was rebuilding.
Personalities : [linear] [raid0] [raid1] [raid10]
md1 : active raid1 sdb2[1](W) sda2[0](W) sdc2[2]
530048 blocks [3/3] [UUU]
md2 : active raid1 sdb3[0](W) sdc3[2] sda3[1](W)
153613440 blocks [3/3] [UUU]
md0 : active raid1 sdb1[0](W) sdc1[2] sda1[1](W)
2104384 blocks [3/3] [UUU]
unused devices: <none>
Simo-
^ permalink raw reply
* Re: missing arrays after OS upgrade
From: Phil Turmel @ 2015-03-25 0:03 UTC (permalink / raw)
To: Daniel Sanabria, linux-raid
In-Reply-To: <CAHscji03jWu3EF0KUG45Hv4Y2t5beJmEoe9uWMfR=wGz_dkBbw@mail.gmail.com>
On 03/24/2015 06:48 PM, Daniel Sanabria wrote:
> Hi,
>
> One of our servers went trough an OS upgrade (from fedora20 to
> fedora21) and althou some of the arrays seem to be auto-detected one
> of them seem to be missing.
You may have had some device names reordered. Building a config that
relies on the names is common but unwise.
With such a thin report, there's not much to go on. Please run lsdrv
[1] and paste its output in a reply.
Then add the contents of /proc/mdstat, the relevant-looking parts of
dmesg, and your mdadm.conf file.
We may need more later, but this will permit intelligent follow-up
questions.
Phil
[1] https://github.com/pturmel/lsdrv
^ permalink raw reply
* missing arrays after OS upgrade
From: Daniel Sanabria @ 2015-03-24 22:48 UTC (permalink / raw)
To: linux-raid
Hi,
One of our servers went trough an OS upgrade (from fedora20 to
fedora21) and althou some of the arrays seem to be auto-detected one
of them seem to be missing.
I have some info related to the array in question:
115 /dev/md2:
116 Version : 0.90
117 Creation Time : Mon Feb 11 07:54:36 2013
118 Raid Level : raid5
119 Array Size : 511999872 (488.28 GiB 524.29 GB)
120 Used Dev Size : 255999936 (244.14 GiB 262.14 GB)
121 Raid Devices : 3
122 Total Devices : 3
123 Preferred Minor : 2
124 Persistence : Superblock is persistent
125
126 Update Time : Sun Mar 22 18:23:38 2015
127 State : clean
128 Active Devices : 3
129 Working Devices : 3
130 Failed Devices : 0
131 Spare Devices : 0
132
133 Layout : left-symmetric
134 Chunk Size : 64K
135
136 UUID : 2cff15d1:e411447b:fd5d4721:03e44022 (local to
host lamachine)
137 Events : 0.478
138
139 Number Major Minor RaidDevice State
140 0 8 3 0 active sync /dev/sda3
141 1 8 18 1 active sync /dev/sdb2
142 2 8 34 2 active sync /dev/sdc2
I haven't taken any actions to try to re-assemble the array and wanted
to check for ideas/suggestions here first.
Any advice you might regarding the steps to take to get the array up
and runnign again is higy appreciated.
Thanks in advance,
Daniel
^ permalink raw reply
* Monitor new badblocks
From: Serge Bartosh @ 2015-03-24 17:30 UTC (permalink / raw)
To: linux-raid
Hello all,
since md started to support of BadBlockList(Log) I'm trying to find
answer to question:
- is there a possibility to get alarm when md discovers new bad blocks?
Please correct me if I'm wrong but I suspect that answer will be "no".
According to man page possible Events are:
DeviceDisappeared, RebuildStarted, RebuildNN, RebuildFinished, Fail,
FailSpare, SpareActive, NewArray, DegradedArray, MoveSpare,
SparesMissing, TestMessage
There is no special event for new badblocks discover and I've checked
today that bad blocks existence is not the reason to mark device "Fail".
I've made experiments with dmsetup simulating "bad surface": even when
new disk containing 90 percents of unusable sectors is added to array md
makes and finishes recover without kicking disk out. Last event is
SpareActive. Syslog does not contain lines about md found bad blocks on
new disk. Array is clean and active.
I understand that manual command "mdadm --examine-badblocks" reveals
such hidden problem.
But is there a way to make "mdadm --monitor" to warn about bad blocks?
PS. System is Debian Jessie RC1
root@linux-test-vb:~# mdadm --version
mdadm - v3.3.2 - 21st August 2014
root@linux-test-vb:~# uname -a
Linux linux-test-vb 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt7-1
(2015-03-01) x86_64 GNU/Linux
root@linux-test-vb:~# cat /etc/debian_version
8.0
PPS. Script used for experiments:
-----8<-----
#!/bin/bash
MDNAME=/dev/md999
DISKSZ=$((64*1024*1024))
#cleanup
echo cleanup
mdadm --stop $MDNAME
dmsetup remove bad_disk4
echo making files and loops
for i in `seq 1 4` ; do
losetup -d /dev/loop$i;
dd if=/dev/zero of=f$i bs=1M count=$((DISKSZ/1024/1024))
losetup /dev/loop$i f$i;
done
echo
echo making faulty disk4 with bb somewhere in the middle...
DEV=/dev/loop4
SECTORS=`blockdev --getsz $DEV` #size in sectors
SECTSZ=`blockdev --getss $DEV` #size of sector
BBPOS=$((SECTORS/2)) #bad block position
BBLEN=$((SECTORS/4)) #bad block region length
echo badblock region from $BBPOS to $((BBPOS+BBLEN)) of $SECTORS
dmsetup create bad_disk4 << EOF
0 $BBPOS linear /dev/loop4 0
$BBPOS $BBLEN error
$((BBPOS+BBLEN)) $((SECTORS-BBLEN-BBPOS)) linear /dev/loop4
$((BBPOS+BBLEN))
EOF
echo
echo making initially degraded raid6
mdadm --create $MDNAME -l 6 -n 4 /dev/loop[123] missing
echo waiting for recover finish...
while [ `cat /sys/block/md999/md/sync_action` != 'idle' ]
do
sleep 1
done
echo adding bb disk
mdadm --add $MDNAME /dev/mapper/bad_disk4
-----8<-----
--
WBR,
Serge Bartosh
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Rik van Riel @ 2015-03-24 16:57 UTC (permalink / raw)
To: Boaz Harrosh, Matthew Wilcox, Andrew Morton
Cc: Dan Williams, linux-kernel, linux-arch, axboe, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel,
Michael S. Tsirkin
In-Reply-To: <55113151.7090304@plexistor.com>
On 03/24/2015 05:41 AM, Boaz Harrosh wrote:
> On 03/23/2015 05:19 PM, Rik van Riel wrote:
>> There are two things going on here:
>>
>> 1) You want to keep using struct page for now, while there are
>> subsystems that require it. This is perfectly legitimate.
>>
>> 2) Matthew and Dan are changing over some subsystems to no longer
>> require struct page. This is perfectly legitimate.
>>
>
> How is this legitimate when you need to Interface the [1] subsystems
> under the [2] subsystem? A subsystem that expects pages is now not
> usable by [2].
>
> Today *All* the Kernel subsystems are [1] Period.
That's not true. In the graphics subsystem it is very normal to
mmap graphics memory without ever using a struct page. There are
other callers of remap_pfn_range() too.
In these cases, refcounting is done by keeping a refcount on the
entire object, not on individual pages (since we have none).
> How does it become
> legitimate to now start *two* competing, do the same differently, abstraction,
> in our kernel. We have two much diversity not to little.
We are already able to refcount either the whole object, or an
individual page.
One issue is that not every subsystem can do the whole object
refcounting, and that it would be nice to have the refcounting
done by one single interface.
If we want the code to be the same everywhere, we could achieve
that just as well with an abstraction as with a single data
structure.
Maybe even something as simplistic as these, with the internals
automatically taking and releasing a refcount on the proper object:
get_reference(file, memory_address)
put_reference(file, memory_address)
--
All rights reversed
^ permalink raw reply
* [PATCH] Fix bitmap offset calculations
From: Goldwyn Rodrigues @ 2015-03-24 16:29 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
The calculations of bitmap offset is incorrect with respect to bits to bytes
conversion.
Also, remove an irrelevant duplicate message.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index ac79fef..e98db04 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -575,7 +575,9 @@ re_read:
sector_div(bm_blocks,
bitmap->mddev->bitmap_info.chunksize >> 9);
- bm_blocks = bm_blocks << 3;
+ /* bits to bytes */
+ bm_blocks = ((bm_blocks+7) >> 3) + sizeof(bitmap_super_t);
+ /* to 4k blocks */
bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
bitmap->mddev->bitmap_info.offset += bitmap->cluster_slot * (bm_blocks << 3);
pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
@@ -672,9 +674,6 @@ out:
goto out_no_sb;
}
bitmap->cluster_slot = md_cluster_ops->slot_number(bitmap->mddev);
- pr_info("%s:%d bm slot: %d offset: %llu\n", __func__, __LINE__,
- bitmap->cluster_slot,
- (unsigned long long)bitmap->mddev->bitmap_info.offset);
goto re_read;
}
^ permalink raw reply related
* Re: [PATCH 1/1] Make bm_blocks to match previous semantic
From: Goldwyn Rodrigues @ 2015-03-24 14:27 UTC (permalink / raw)
To: NeilBrown, Guoqing Jiang; +Cc: jgq516, linux-raid
In-Reply-To: <20150321093750.1f8c4075@notabene.brown>
On 03/20/2015 05:37 PM, NeilBrown wrote:
> On Thu, 19 Mar 2015 11:50:05 +0800 Guoqing Jiang <GQJiang@suse.com> wrote:
>
>> Hi Neil,
>>
>> NeilBrown wrote:
>>> On Tue, 17 Mar 2015 10:40:30 +0800 jgq516@gmail.com wrote:
>>>
>>>
>>>> From: Guoqing Jiang <gqjiang@suse.com>
>>>>
>>>> The bm_blocks is modified by commit fe60ce (md/bitmap: use
>>>> sector_div for sector_t divisions), but it makes bm_blocks
>>>> has different value which is changed from like "a/b" to "a%b",
>>>> need to correct this to make sure cluster-md still works.
>>>>
>>>
>>> One of us is confused here.
>>>
>>> This code is trying to find the start of the bitmap relevant to this host in
>>> a table of multiple bitmaps. So it first needs to find out the size of each
>>> bitmap. It then multiples the size by the index number of this host to get
>>> an offset.
>>>
>>>
>> Thanks for detailed description, it really helps. I quoted related lines
>> from bitmap.c.
>>
>> 574 sector_t bm_blocks;
>> 575 sector_t resync_sectors =
>> bitmap->mddev->resync_max_sectors;
>> 576
>> 577 bm_blocks = sector_div(resync_sectors,
>> 578
>> bitmap->mddev->bitmap_info.chunksize >> 9);
>> 579 bm_blocks = bm_blocks << 3;
>> 580 bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
>> 581 bitmap->mddev->bitmap_info.offset +=
>> bitmap->cluster_slot * (bm_blocks << 3);
>>
>>> So it take the total number of sectors (resync_max_sectors), divides by the
>>> chunksize (in sectors) to get a number of chunks. This is the number of bits.
>>>
>>>
>> L577 is supposed to do above job.
>>> Then it should div-round-up by 8 to get a number of bytes.
>>>
>> I guess what you mean is about L579, while it used "<<3" rather than
>> ">>3" now.
>>> Then div-round-up by 4096 to get number of 4-K blocks, because the bitmaps
>>> are always 4K aligned.
>>>
>> L580 did the job.
>>> Then this number is multiplied by 8 (or shifted by 3) to get a number of
>>> sectors to add to the start of the table.
>>>
>> L581 is for this, right? Is the shifted by 3 is to match the bitmap
>> format for each
>> nodes? Seems the relationship between slot and the bitmap region of the node
>> is like n <-----> [8*nK, 8*(n+1)K]. How about the following changes?
>>
>> diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
>> index 501f83f..b2a241b 100644
>> --- a/drivers/md/bitmap.c
>> +++ b/drivers/md/bitmap.c
>> @@ -571,12 +571,10 @@ static int bitmap_read_sb(struct bitmap *bitmap)
>> re_read:
>> /* If cluster_slot is set, the cluster is setup */
>> if (bitmap->cluster_slot >= 0) {
>> - sector_t bm_blocks;
>> - sector_t resync_sectors = bitmap->mddev->resync_max_sectors;
>> + sector_t bm_blocks = bitmap->mddev->resync_max_sectors;
>>
>> - bm_blocks = sector_div(resync_sectors,
>> -
>> bitmap->mddev->bitmap_info.chunksize >> 9);
>> - bm_blocks = bm_blocks << 3;
>> + sector_div(bm_blocks,
>> bitmap->mddev->bitmap_info.chunksize >> 9);
>
> Yes, of course. sector_div returns the remainder doesn't it!
> I was thinking that it returned the quotient and set the first arg to the
> remainder - and wonder why you wanted the remainder :-(
>
> I've updated the patch to do the right thing and credited you.
> Thanks.
>
>> + bm_blocks = bm_blocks >> 3;
>> bm_blocks = DIV_ROUND_UP_SECTOR_T(bm_blocks, 4096);
>>> So the original code in commit b97e92574c0bf335db1cd2ec491d8ff5cd5d0b49
>>> is wrong because it uses sector_div in a way which destroys
>>> resync_max_sectors.
>>> And is wrong because it multiplies by 8 (<<3) instead of divides by 8 to
>>> convert from bits to bytes.
>>>
>>> commit f9209a323547f054c7439a3bf67c45e64a054bd
>>> removes the abuse of sector_div, which is good, but uses a simple "a/b"
>>> division, which isn't allowed in the kernel.
>>>
>>> commit fe60ce80488a2a481ac175c4ff98f90df22e1e46
>>> then does the right thing with sector_div, but the "<< 3" is still the wrong
>>> way around.
>>>
>>> If you still think your code is correct, please explain in detail why.
>>>
>>> Goldwyn: if you agree that "<< 3" should be ">> 3" or even
>>> DIV_ROUND_UP_SECTOR_T( , 8);
>>> please send a patch. If you don't think so, please explain why.
>>>
>>>
>> But anyway, it is better wait for Goldwyn's back from vacation, :)
>
> I'll leave the other change (<<3 or >>8) until then.
Yes, you are right. It is working for small bitmaps which were covered
in 1 4k blocks, but was breaking for ones where the bitmap is larger.
It should be (bm_blocks+7) >> 3 or DV_ROUND_UP_SECTOR_T( ,8). Also, it
should account for bitmap_super_t size as well. I will send a patch
shortly to mimic how the userspace tools calculate it.
--
Goldwyn
^ permalink raw reply
* resize a 'far' RAID10 ?
From: Wolfgang Denk @ 2015-03-24 13:48 UTC (permalink / raw)
To: linux-raid
Hello all,
I am aware that it has been impossible to resize a 'far' RAID10; at
least for v3.4 or so - see for example [1]
Has this restriction been listed in the meantime?
If not: I have swapped disk drives for larger models and would like to
make use of the additional size. Is creating another RAID on the free
space the only available option?
Thanks in advance.
[1] http://www.spinics.net/lists/raid/msg38491.html
Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Everything should be made as simple as possible, but not simpler."
- Albert Einstein
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-24 9:41 UTC (permalink / raw)
To: Rik van Riel, Boaz Harrosh, Matthew Wilcox, Andrew Morton
Cc: Dan Williams, linux-kernel, linux-arch, axboe, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel,
Michael S. Tsirkin
In-Reply-To: <55102EEB.2090401@redhat.com>
On 03/23/2015 05:19 PM, Rik van Riel wrote:
>>> Michael Tsirkin and I have been doing some thinking about what
>>> it would take to allocate struct pages per 2MB area permanently,
>>> and allocate additional struct pages for 4kB pages on demand,
>>> when a 2MB area is broken up into 4kB pages.
>>
>> My thoughts as well, this need *not* be a huge evasive change. Is however
>> a careful surgery in very core code. And lots of sleepless scary nights
>> and testing to make sure all the side effects are wrinkled out.
>
> Even the above IS a huge invasive change, and I do not see it
> as much better than the work Dan and Matthew are doing.
>
You lost me again. Sorry for my slowness. The code I envision is not
invasive at all. Nothing is touched at all, except a few core places
at the page level.
The contract with Kernel stays the same:
page_to_pfn, pfn_to_page, page_address (which is kmap_atomic in 64bit)
virt_to_page, page_get/put and so on...
So none of the Kernel code need change at all. You were saying that we
might have a 2M page and on demand we can allocate a 4k page shove it down the
stack, which does not change at all, and once back from io, the 4k pages can be
freed and recycled for reuse with other IO. This is what I thought you said.
This is doable, and not that much work and for the life of me I do not see any
"invasive". (Yes a few core headers that make everything compile ;-))
That said I do not even think we need that (2M split to 4k on demand) we can even
do better and make sure 2M pages just work as is. It is very possible today
(Tested) to push a 2M page into a bio and write to a bdev. Yes lots of side
code will break, but the core path is clean. Let us fix that then.
(Need I send code to show you how a 2M page is written with a single
bvec?)
>> If we want copy-less, we need a common memory descriptor career. Today this
>> is page-struct. So for me your above statement means:
>> "still not convinced I care about copy-less pmem"
>>
>> Otherwise you either enhance what you have today or devise a new
>> system, which means change the all Kernel.
>
> We do not necessarily need a common descriptor, as much as
> one that abstracts out what is happening. Something like a
> struct bio could be a good I/O descriptor, and releasing the
> backing memory after IO completion could be a function of the
> bio freeing function itself.
>
Lost me again sorry. What backing memory. struct bio is already
an I/O descriptor which gets freed after use. How is that relevant
to pfn vs page ?
>> Lastly: Why does pmem need to wait out-of-tree. Even you say above that
>> machines with lots of DRAM can enjoy the HUGE-to-4k split. So why
>> not let pmem waist 4k pages like everyone else and fix it as above
>> down the line, both for pmem and ram. And save both ways.
>> Why do we need to first change the all Kernel, then have pmem. Why not
>> use current infra structure, for good or for worth, and incrementally
>> do better.
>
> There are two things going on here:
>
> 1) You want to keep using struct page for now, while there are
> subsystems that require it. This is perfectly legitimate.
>
> 2) Matthew and Dan are changing over some subsystems to no longer
> require struct page. This is perfectly legitimate.
>
How is this legitimate when you need to Interface the [1] subsystems
under the [2] subsystem? A subsystem that expects pages is now not
usable by [2].
Today *All* the Kernel subsystems are [1] Period. How does it become
legitimate to now start *two* competing, do the same differently, abstraction,
in our kernel. We have two much diversity not to little.
> I do not understand why either of you would have to object to what
> the other is doing. There is room to keep using struct page until
> the rest of the kernel no longer requires it.
>
So this is your vision "until the rest of the kernel no longer requires
pages" Really? Sigh, coming from other Kernels I thought pages were
a breeze of fresh air. I thought it was very clever. And BTW good luck
with that.
BTW: you have not solved the basic problem yet. for one pfn_kmap() given a
pfn what is its virtual address. would you like to loop through the
Kernel's range tables to look for the registered ioremap ? its a long
annoying loop. The page was invented exactly for this reason, to go through
the section object. And actually it is not that easy because if it is an ioremap
pointer it is in one list and if a page it is another way, and on top of
all this, it is ARCH dependent. And you are trashing highmem, because the
state and locks of that are at the page level. Not that I care about highmem
but I hate double coding. For god sake what do you guys have with poor old
pages, they were invented to exacly do this, abstract away management of a single
pfn-to-virt.
All I see is complains about page being 4K well it need not be. page can be any
size, and hell it can be variable size. (And no we do not need to add an extra size
member, all we need is the one bit)
Cheers
Boaz
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: lingli tang @ 2015-03-24 2:09 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid
In-Reply-To: <55100D9D.6080801@websitemanagers.com.au>
No, I did not shutdown the remote machine.
I just shutdown the machine of RAID1 disk with local disk and remote
disk, but not the remote machine.
And I think I have found the final reason after test with remove
iscsi/iscsid in chkconfig.
The final reason is :
when I issue command of 'reboot', linux will shutdown user process and
kernel module step by step. Because of iscsi register in chkconfig,
its session will be logout before mdadm shutdown. Therefore it is a
short time downgrade of mdadm(1-3 second), after that mdadm was
shutdown. During this time mysql will write binlog to a downgrade
mdadm which just contain the local disk (remote disk was kickout when
iscsi logout). So the remote disk lost 1-3 second binlog from the
'reboot' machine.
I have remove iscsi/iscsid from chkconfig with:
chkconfig --del iscsi
chkconfig --del iscsid
and found no data loss on the remote disk.
NeilBrown & Adam
Thanks very much for your help
2015-03-23 20:57 GMT+08:00 Adam Goryachev <mailinglists@websitemanagers.com.au>:
>
> On 23/03/2015 19:34, lingli tang wrote:
>>
>> I have test multi times of:
>> 1. mysql binlog write only on remote disk (without mdadm raid), there
>> are not any mysql binlog lost.
>> 2. mysql binlog write on RAID1 of only remote disk (no local disk),
>> there are not any mysql binlog lost.
>> mysql will return error immediately with error message "Error writing
>> file '/home/mysql/data/mysqldata1/binlog/mysql-bin.000001' (Errcode: 5
>> - Input/output error)" in the upper two case
>>
>> but when MySQL binlog run on RAID1 of local and remote disk, test
>> program which continued commit to mysql will run for 3 second and
>> hang in mysql_query() after reboot server. The error messge is also
>> not the same with upper case: "Lost connection to MySQL server during
>> query"
>>
>> Should it be iscsi exit before mdadm, So mysql continue to write
>> binlog to a downgrade RAID1, which has only a local disk but the
>> remote disk was just delete from mdadm.
>>
>> I will try to test it.
>> Thanks very much.
>>
> Silly question, which machine are you sending the shutdown command to?
>
> If you are doing this one the remote disk machine, then obviously it may not
> have received all of the data yet, and therefore may have lost some data,
> even if it is a clean reboot.
>
> Equally, as mentioned, if you shutdown the remote disk before MD shuts down
> (or shutdown the network prior to MD), then you have the same problem. You
> should check the MD status of each member disk to see if they think the
> other disk failed prior to MD being shutdown, and what is the event counter
> of each disk. You should see the local disk reporting the remote disk as
> failed, and the local disk should have a higher event count.
>
> Regards,
> Adam
>
>> 2015-03-22 20:51 GMT+08:00 Adam Goryachev
>> <mailinglists@websitemanagers.com.au>:
>>>
>>>
>>> On 22/03/2015 23:29, lingli tang wrote:
>>>>
>>>> Thanks very much.
>>>> I will try DRBD later
>>>> But I want to figure this out.
>>>>
>>>> I have export disk using tgtd and load disk on another server using
>>>> iscsiadm with infiniband of iser protocol.
>>>> Does ISCSI/Iser have any cache on it.
>>>
>>> Can you test that by removing the local disk from the MD array, or
>>> changing
>>> your test so writes are directly to the remote device. Then run the test,
>>> shutdown, and check the remote disk to see if it has all the expected
>>> data,
>>> or still only some of the expected data. This will remove MD as a
>>> suspect.
>>> Continue to try and get "closer" to the remote until you can find the
>>> culprit. You might also use tcpdump or similar to sniff the network,
>>> which
>>> will tell you if the expected data is being sent to the remote (and
>>> when).
>>>
>>> Sorry, I don't know anywhere near enough to comment on things like
>>> infiniband/iser, but these are the steps I would look into. Hope that it
>>> is
>>> helpful.
>>>
>>> PS, I do use DRBD, and iSCSI, and it has been working well in my
>>> environment
>>> for the last year or so, I have no commercial interest/benefit from you
>>> using it, just a happy customer.
>>>
>>> Regards,
>>> Adam
>>>>
>>>>
>>>> 2015-03-22 15:28 GMT+08:00 Adam Goryachev
>>>> <mailinglists@websitemanagers.com.au>:
>>>>>
>>>>>
>>>>> On 22/03/2015 16:00, lingli tang wrote:
>>>>>>
>>>>>> Thanks for reply.
>>>>>>
>>>>>> I have create a raid1 with two fusion io PCIe flash disk:
>>>>>> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
>>>>>> /dev/fioa2 /dev/mapper/mpathc
>>>>>> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
>>>>>> load disk export from server B.
>>>>>>
>>>>>> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option
>>>>>> on
>>>>>> /data1
>>>>>> and we will run mysql binlog on it.
>>>>>> In order to avoid data loss of mysql binlog we have set
>>>>>> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>>>>>>
>>>>>> according to your description. if we reboot the server A, the two disk
>>>>>> data on different server will be the same.
>>>>>> but after the server A restarted, we assemble the two disk on two
>>>>>> server, data is different on the two server, disk on server B lost
>>>>>> more than one sql commit.
>>>>>>
>>>>>> I have checked it with strace 'mysqld' on Server A.
>>>>>> I found a sql commit and fsync() on binlog file handle on server A but
>>>>>> this sql can not find in assembled disk on server B.
>>>>>>
>>>>>> I also test it with two SAS disk, Server B still has more than one sql
>>>>>> commit lost.
>>>>>
>>>>> Sounds like you might be better using something like DRBD
>>>>> (www.drbd.org)
>>>>> which has different modes, one of which will do what you are asking
>>>>> (not
>>>>> respond until both systems have confirmed the data is written to the
>>>>> local
>>>>> disk).
>>>>>
>>>>> In your current case, even if md is correctly writing to both
>>>>> underlying
>>>>> 'devices' you have multiple layers under one of the devices, so you
>>>>> should
>>>>> confirm that *all* of those layers are properly passing through the
>>>>> data
>>>>> without any caching/etc.
>>>>>
>>>>> Regards,
>>>>> Adam
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Christoph Hellwig @ 2015-03-23 19:30 UTC (permalink / raw)
To: Rik van Riel
Cc: Boaz Harrosh, Matthew Wilcox, Andrew Morton, Dan Williams,
linux-kernel, linux-arch, axboe, linux-nvdimm, Dave Hansen,
linux-raid, mgorman, hch, linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <55102EEB.2090401@redhat.com>
On Mon, Mar 23, 2015 at 11:19:07AM -0400, Rik van Riel wrote:
> There are two things going on here:
>
> 1) You want to keep using struct page for now, while there are
> subsystems that require it. This is perfectly legitimate.
>
> 2) Matthew and Dan are changing over some subsystems to no longer
> require struct page. This is perfectly legitimate.
>
> I do not understand why either of you would have to object to what
> the other is doing. There is room to keep using struct page until
> the rest of the kernel no longer requires it.
*nod*
I'd really like to merge the struct page based pmem driver ASAP. We can
then look into work that avoid the need for struct page, and I think Dan
is doing some good work in that direction.
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Rik van Riel @ 2015-03-23 15:19 UTC (permalink / raw)
To: Boaz Harrosh, Matthew Wilcox, Andrew Morton
Cc: Dan Williams, linux-kernel, linux-arch, axboe, linux-nvdimm,
Dave Hansen, linux-raid, mgorman, hch, linux-fsdevel,
Michael S. Tsirkin
In-Reply-To: <550EE4FE.1070009@plexistor.com>
On 03/22/2015 11:51 AM, Boaz Harrosh wrote:
> On 03/20/2015 06:21 PM, Rik van Riel wrote:
>> On 03/19/2015 09:43 AM, Matthew Wilcox wrote:
>>
>>> 1. Construct struct pages for persistent memory
>>> 1a. Permanently
>>> 1b. While the pages are under I/O
>>
>> Michael Tsirkin and I have been doing some thinking about what
>> it would take to allocate struct pages per 2MB area permanently,
>> and allocate additional struct pages for 4kB pages on demand,
>> when a 2MB area is broken up into 4kB pages.
>>
>> This should work for both DRAM and persistent memory.
>
> My thoughts as well, this need *not* be a huge evasive change. Is however
> a careful surgery in very core code. And lots of sleepless scary nights
> and testing to make sure all the side effects are wrinkled out.
Even the above IS a huge invasive change, and I do not see it
as much better than the work Dan and Matthew are doing.
> If we want copy-less, we need a common memory descriptor career. Today this
> is page-struct. So for me your above statement means:
> "still not convinced I care about copy-less pmem"
>
> Otherwise you either enhance what you have today or devise a new
> system, which means change the all Kernel.
We do not necessarily need a common descriptor, as much as
one that abstracts out what is happening. Something like a
struct bio could be a good I/O descriptor, and releasing the
backing memory after IO completion could be a function of the
bio freeing function itself.
> Lastly: Why does pmem need to wait out-of-tree. Even you say above that
> machines with lots of DRAM can enjoy the HUGE-to-4k split. So why
> not let pmem waist 4k pages like everyone else and fix it as above
> down the line, both for pmem and ram. And save both ways.
> Why do we need to first change the all Kernel, then have pmem. Why not
> use current infra structure, for good or for worth, and incrementally
> do better.
There are two things going on here:
1) You want to keep using struct page for now, while there are
subsystems that require it. This is perfectly legitimate.
2) Matthew and Dan are changing over some subsystems to no longer
require struct page. This is perfectly legitimate.
I do not understand why either of you would have to object to what
the other is doing. There is room to keep using struct page until
the rest of the kernel no longer requires it.
--
All rights reversed
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: Adam Goryachev @ 2015-03-23 12:57 UTC (permalink / raw)
To: lingli tang; +Cc: linux-raid
In-Reply-To: <CAN+bsqghYy1PyWpg9KRJan3RJZr0WMPLZCZtf01h0YR-_Jo_-w@mail.gmail.com>
On 23/03/2015 19:34, lingli tang wrote:
> I have test multi times of:
> 1. mysql binlog write only on remote disk (without mdadm raid), there
> are not any mysql binlog lost.
> 2. mysql binlog write on RAID1 of only remote disk (no local disk),
> there are not any mysql binlog lost.
> mysql will return error immediately with error message "Error writing
> file '/home/mysql/data/mysqldata1/binlog/mysql-bin.000001' (Errcode: 5
> - Input/output error)" in the upper two case
>
> but when MySQL binlog run on RAID1 of local and remote disk, test
> program which continued commit to mysql will run for 3 second and
> hang in mysql_query() after reboot server. The error messge is also
> not the same with upper case: "Lost connection to MySQL server during
> query"
>
> Should it be iscsi exit before mdadm, So mysql continue to write
> binlog to a downgrade RAID1, which has only a local disk but the
> remote disk was just delete from mdadm.
>
> I will try to test it.
> Thanks very much.
>
Silly question, which machine are you sending the shutdown command to?
If you are doing this one the remote disk machine, then obviously it may
not have received all of the data yet, and therefore may have lost some
data, even if it is a clean reboot.
Equally, as mentioned, if you shutdown the remote disk before MD shuts
down (or shutdown the network prior to MD), then you have the same
problem. You should check the MD status of each member disk to see if
they think the other disk failed prior to MD being shutdown, and what is
the event counter of each disk. You should see the local disk reporting
the remote disk as failed, and the local disk should have a higher event
count.
Regards,
Adam
> 2015-03-22 20:51 GMT+08:00 Adam Goryachev <mailinglists@websitemanagers.com.au>:
>>
>> On 22/03/2015 23:29, lingli tang wrote:
>>> Thanks very much.
>>> I will try DRBD later
>>> But I want to figure this out.
>>>
>>> I have export disk using tgtd and load disk on another server using
>>> iscsiadm with infiniband of iser protocol.
>>> Does ISCSI/Iser have any cache on it.
>> Can you test that by removing the local disk from the MD array, or changing
>> your test so writes are directly to the remote device. Then run the test,
>> shutdown, and check the remote disk to see if it has all the expected data,
>> or still only some of the expected data. This will remove MD as a suspect.
>> Continue to try and get "closer" to the remote until you can find the
>> culprit. You might also use tcpdump or similar to sniff the network, which
>> will tell you if the expected data is being sent to the remote (and when).
>>
>> Sorry, I don't know anywhere near enough to comment on things like
>> infiniband/iser, but these are the steps I would look into. Hope that it is
>> helpful.
>>
>> PS, I do use DRBD, and iSCSI, and it has been working well in my environment
>> for the last year or so, I have no commercial interest/benefit from you
>> using it, just a happy customer.
>>
>> Regards,
>> Adam
>>>
>>> 2015-03-22 15:28 GMT+08:00 Adam Goryachev
>>> <mailinglists@websitemanagers.com.au>:
>>>>
>>>> On 22/03/2015 16:00, lingli tang wrote:
>>>>> Thanks for reply.
>>>>>
>>>>> I have create a raid1 with two fusion io PCIe flash disk:
>>>>> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
>>>>> /dev/fioa2 /dev/mapper/mpathc
>>>>> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
>>>>> load disk export from server B.
>>>>>
>>>>> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option
>>>>> on
>>>>> /data1
>>>>> and we will run mysql binlog on it.
>>>>> In order to avoid data loss of mysql binlog we have set
>>>>> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>>>>>
>>>>> according to your description. if we reboot the server A, the two disk
>>>>> data on different server will be the same.
>>>>> but after the server A restarted, we assemble the two disk on two
>>>>> server, data is different on the two server, disk on server B lost
>>>>> more than one sql commit.
>>>>>
>>>>> I have checked it with strace 'mysqld' on Server A.
>>>>> I found a sql commit and fsync() on binlog file handle on server A but
>>>>> this sql can not find in assembled disk on server B.
>>>>>
>>>>> I also test it with two SAS disk, Server B still has more than one sql
>>>>> commit lost.
>>>> Sounds like you might be better using something like DRBD (www.drbd.org)
>>>> which has different modes, one of which will do what you are asking (not
>>>> respond until both systems have confirmed the data is written to the
>>>> local
>>>> disk).
>>>>
>>>> In your current case, even if md is correctly writing to both underlying
>>>> 'devices' you have multiple layers under one of the devices, so you
>>>> should
>>>> confirm that *all* of those layers are properly passing through the data
>>>> without any caching/etc.
>>>>
>>>> Regards,
>>>> Adam
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
^ permalink raw reply
* Thanks for the awesome
From: Phil Ingram @ 2015-03-23 11:34 UTC (permalink / raw)
To: linux-raid
To all who have contributed to mdadm I thank you. Last week for the
third time in my life I was able to recover from certain disaster.
This time it was a 4 disk complex raid10 that had lost 1 disk and
thrown another. Smartctl said that one disk was genuinely faulty and
in replacing the drive I hadn't appropriately re-secured the sata
cable to the drive above it. Long story short is the array started,
but disk 4 was flakey. Metadata was updated on 2 drives and upon
reboot (after seeing critical messages echoed to my screen about XFS
saying "nope") the array couldn't be automatically started.
mdadm --misc -E /dev/sdb and I could forcibly re-create the array,
after following the iteratively scary recovery guide on your wiki. I
wasn't writing to the array when it became corrupted so felt confident
in my decision - well, my choices were to either definitely lose
approx 4-5 TB of data or go for a seat of my pants hail mary...
I got to thank Neil in person at LCA 2013 in Canberra, but I had to
extend my thanks to the wider community after this third miracle. I
can't imagine such a recovery with a hardware based solution. i
probably wouldn't have had this issue with a proper drive array, but
this is a re-purposed pc->server conversion..
Thanks :)
p.s. the other 2 miracles were recovering a RAID5 that had thrown 2
disks (bad PSU) and reshaping a 2 disk RAID 1 array to a 3 disk RAID
5; amazing!!
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: lingli tang @ 2015-03-23 8:34 UTC (permalink / raw)
To: Adam Goryachev; +Cc: linux-raid
In-Reply-To: <550EBAEE.6070608@websitemanagers.com.au>
I have test multi times of:
1. mysql binlog write only on remote disk (without mdadm raid), there
are not any mysql binlog lost.
2. mysql binlog write on RAID1 of only remote disk (no local disk),
there are not any mysql binlog lost.
mysql will return error immediately with error message "Error writing
file '/home/mysql/data/mysqldata1/binlog/mysql-bin.000001' (Errcode: 5
- Input/output error)" in the upper two case
but when MySQL binlog run on RAID1 of local and remote disk, test
program which continued commit to mysql will run for 3 second and
hang in mysql_query() after reboot server. The error messge is also
not the same with upper case: "Lost connection to MySQL server during
query"
Should it be iscsi exit before mdadm, So mysql continue to write
binlog to a downgrade RAID1, which has only a local disk but the
remote disk was just delete from mdadm.
I will try to test it.
Thanks very much.
2015-03-22 20:51 GMT+08:00 Adam Goryachev <mailinglists@websitemanagers.com.au>:
>
>
> On 22/03/2015 23:29, lingli tang wrote:
>>
>> Thanks very much.
>> I will try DRBD later
>> But I want to figure this out.
>>
>> I have export disk using tgtd and load disk on another server using
>> iscsiadm with infiniband of iser protocol.
>> Does ISCSI/Iser have any cache on it.
>
> Can you test that by removing the local disk from the MD array, or changing
> your test so writes are directly to the remote device. Then run the test,
> shutdown, and check the remote disk to see if it has all the expected data,
> or still only some of the expected data. This will remove MD as a suspect.
> Continue to try and get "closer" to the remote until you can find the
> culprit. You might also use tcpdump or similar to sniff the network, which
> will tell you if the expected data is being sent to the remote (and when).
>
> Sorry, I don't know anywhere near enough to comment on things like
> infiniband/iser, but these are the steps I would look into. Hope that it is
> helpful.
>
> PS, I do use DRBD, and iSCSI, and it has been working well in my environment
> for the last year or so, I have no commercial interest/benefit from you
> using it, just a happy customer.
>
> Regards,
> Adam
>>
>>
>> 2015-03-22 15:28 GMT+08:00 Adam Goryachev
>> <mailinglists@websitemanagers.com.au>:
>>>
>>>
>>> On 22/03/2015 16:00, lingli tang wrote:
>>>>
>>>> Thanks for reply.
>>>>
>>>> I have create a raid1 with two fusion io PCIe flash disk:
>>>> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
>>>> /dev/fioa2 /dev/mapper/mpathc
>>>> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
>>>> load disk export from server B.
>>>>
>>>> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option
>>>> on
>>>> /data1
>>>> and we will run mysql binlog on it.
>>>> In order to avoid data loss of mysql binlog we have set
>>>> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
>>>>
>>>> according to your description. if we reboot the server A, the two disk
>>>> data on different server will be the same.
>>>> but after the server A restarted, we assemble the two disk on two
>>>> server, data is different on the two server, disk on server B lost
>>>> more than one sql commit.
>>>>
>>>> I have checked it with strace 'mysqld' on Server A.
>>>> I found a sql commit and fsync() on binlog file handle on server A but
>>>> this sql can not find in assembled disk on server B.
>>>>
>>>> I also test it with two SAS disk, Server B still has more than one sql
>>>> commit lost.
>>>
>>> Sounds like you might be better using something like DRBD (www.drbd.org)
>>> which has different modes, one of which will do what you are asking (not
>>> respond until both systems have confirmed the data is written to the
>>> local
>>> disk).
>>>
>>> In your current case, even if md is correctly writing to both underlying
>>> 'devices' you have multiple layers under one of the devices, so you
>>> should
>>> confirm that *all* of those layers are properly passing through the data
>>> without any caching/etc.
>>>
>>> Regards,
>>> Adam
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply
* Re: Hung RAID5 array with discard
From: NeilBrown @ 2015-03-23 2:57 UTC (permalink / raw)
To: Terry Hardie; +Cc: linux-raid
In-Reply-To: <CANHB05sb44ZD-31ceEWWSY0XdhWaU-Pq+NFRQSm-1qafHx1G2g@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 14489 bytes --]
On Wed, 4 Mar 2015 13:47:08 -0800 Terry Hardie <thardie@instartlogic.com>
wrote:
> Well, I'm dissapointed no one responded to this. This basically means
> linux RAID 4/5/6 and discard is fundamentally broken, and no one wants
> to acknowledge it.
It might just mean that no-one noticed your email, or that they were busy, or
were just about to leave on Christmas holidays or ......
If you don't get a response, resending after a reasonable period (couple of
weeks) is perfectly acceptable.
>
> I hope someone finds this post while I still have my lab available and
> I can help them troubleshoot this issue.
>
> I tried this again today on 3.13.0-44-generic (Ubuntu) and was easily
> able to reproduce it.
Can you try with a more recent kernel? 3.13.0 is over year old and there is
at least one raid5 bugfix that went into the 3.13-stable series.
If you can reproduce with 3.19, I'll definitely look into it.
Thanks for the report,
NeilBrown
>
> On Wed, Dec 17, 2014 at 7:08 PM, Terry Hardie <thardie@instartlogic.com> wrote:
> > Hi,
> >
> > I am testing 3 SSDs (1TB Crucial M550 with DRZAT, and I tested they do
> > return zeros after discard) with RAID5 and discard. I create the array
> > with a 64k chunk size, and it starts to sync. During it's initial
> > reconstruction, I do a mkfs.ext4, which starts to do the "Discarding
> > device blocks". After a short period (I believe when the mkfs reaches
> > the point where the reconstruction is at, all IO to the disks freezes,
> > and mkfs does not advance. iostat shows 2 of the 3 drives at 100%
> > utilization with no data read or written. After 2 minutes, I get the
> > hung task dump. Most CPUs are idle, and here are a few which are not,
> > which look like a deadlock to me:
> >
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154399] INFO:
> > rcu_sched detected stalls on CPUs/tasks: { 4 5} (detected by 3,
> > t=285032 jiffies, g=1160, c=1159, q=0)
> >
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154902] NMI
> > backtrace for cpu 4
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154904] CPU: 4 PID:
> > 2146 Comm: md3_raid5 Tainted: G W IOX 3.13.0-43-generic
> > #72~precise1
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154905] Hardware
> > name: Supermicro SYS-2028TP-HC1R/X10DRT-P, BIOS 1.0a 08/28/2014
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154906] task:
> > ffff88202594c800 ti: ffff8810245a0000 task.ti: ffff8810245a0000
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154907] RIP:
> > 0010:[<ffffffff817644c1>] [<ffffffff817644c1>]
> > _raw_spin_lock_irqsave+0x41/0x60
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154910] RSP:
> > 0018:ffff8810245a1cc8 EFLAGS: 00000006
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154911] RAX:
> > 0000000000002ec5 RBX: ffff882028a6ec00 RCX: 0000000000007b78
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154912] RDX:
> > 0000000000000202 RSI: 0000000000007b78 RDI: ffff882028a6ec10
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154913] RBP:
> > ffff8810245a1cc8 R08: 0000000000007b76 R09: ffff882023629170
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154914] R10:
> > 0000000000000000 R11: ffff882028a6ec00 R12: ffff882028a6ee68
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154915] R13:
> > 0000000000000003 R14: 0000000000000002 R15: ffff882028a6ec10
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154916] FS:
> > 0000000000000000(0000) GS:ffff88103fc80000(0000)
> > knlGS:0000000000000000
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154917] CS: 0010
> > DS: 0000 ES: 0000 CR0: 0000000080050033
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154918] CR2:
> > 00007f208c2d0000 CR3: 0000000001c0d000 CR4: 00000000001407e0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154919] Stack:
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154920]
> > ffff8810245a1d18 ffffffffa0149890 0000000000000002 ffff882028a6ee88
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154923]
> > ffff882028a6ee68 ffff882028a6ec00 0000000000000008 ffff882028a6ee68
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154926]
> > 0000000000000000 ffff882028a6ee50 ffff8810245a1d98 ffffffffa015212f
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154929] Call Trace:
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154933]
> > [<ffffffffa0149890>] release_inactive_stripe_list+0x50/0x160 [raid456]
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154937]
> > [<ffffffffa015212f>] handle_active_stripes.isra.38+0x7f/0x190
> > [raid456]
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154940]
> > [<ffffffffa0152758>] raid5d+0x198/0x2f0 [raid456]
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154942]
> > [<ffffffff815d30a7>] md_thread+0x117/0x150
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154945]
> > [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154947]
> > [<ffffffff815d2f90>] ? md_rdev_init+0x110/0x110
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154949]
> > [<ffffffff8108fb59>] kthread+0xc9/0xe0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154952]
> > [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154954]
> > [<ffffffff8176d5bc>] ret_from_fork+0x7c/0xb0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154956]
> > [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.154957] Code: 1f 44
> > 00 00 b8 00 00 02 00 f0 0f c1 07 89 c1 c1 e9 10 66 39 c1 75 05 48 89
> > d0 5d c3 83 e1 fe 0f b7 f1 b8 00 80 00 00 44 0f b7 07 <66> 44 39 c1 74
> > e6 f3 90 83 e8 01 75 ef 0f 1f 80 00 00 00 00 eb
> >
> >
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155001] NMI
> > backtrace for cpu 5
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155003] CPU: 5 PID:
> > 2147 Comm: md3_resync Tainted: G W IOX 3.13.0-43-generic
> > #72~precise1
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155004] Hardware
> > name: Supermicro SYS-2028TP-HC1R/X10DRT-P, BIOS 1.0a 08/28/2014
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155005] task:
> > ffff88202594b000 ti: ffff8810274a0000 task.ti: ffff8810274a0000
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155006] RIP:
> > 0010:[<ffffffffa01483b7>] [<ffffffffa01483b7>]
> > __find_stripe+0x57/0xa0 [raid456]
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155010] RSP:
> > 0018:ffff8810274a1b68 EFLAGS: 00000006
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155011] RAX:
> > ffff882027092da0 RBX: 0000000000a30c10 RCX: 0000000000000001
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155012] RDX:
> > 0000000000000c10 RSI: 0000000000a30c10 RDI: ffff882028a6ec00
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155013] RBP:
> > ffff8810274a1b88 R08: 0000000000000000 R09: 0000000000000000
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155014] R10:
> > 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155015] R13:
> > ffff882028a6ec00 R14: 0000000000000000 R15: ffff882028a6eda8
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155016] FS:
> > 0000000000000000(0000) GS:ffff88103fca0000(0000)
> > knlGS:0000000000000000
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155017] CS: 0010
> > DS: 0000 ES: 0000 CR0: 0000000080050033
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155018] CR2:
> > 00000000006e1dc8 CR3: 0000000001c0d000 CR4: 00000000001407e0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155019] Stack:
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155020]
> > ffff8810274a1ba8 ffff882028a6ec00 000000007b767b00 ffff882028a6ec10
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155023]
> > ffff8810274a1c28 ffffffffa0150555 ffff882023773b50 ffff882028a6eda8
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155026]
> > 0000000200000001 ffff882028a6ec08 0000000000000000 0000000000a30c10
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155029] Call Trace:
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155033]
> > [<ffffffffa0150555>] get_active_stripe+0x115/0x3e0 [raid456]
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155036]
> > [<ffffffffa014aea8>] ? release_stripe+0x68/0x100 [raid456]
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155040]
> > [<ffffffffa0154f3b>] sync_request+0x11b/0x2a0 [raid456]
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155042]
> > [<ffffffff815d5ccf>] md_do_sync+0x84f/0xdb0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155046]
> > [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155048]
> > [<ffffffff815d30a7>] md_thread+0x117/0x150
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155050]
> > [<ffffffff815d2f90>] ? md_rdev_init+0x110/0x110
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155052]
> > [<ffffffff8108fb59>] kthread+0xc9/0xe0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155054]
> > [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155057]
> > [<ffffffff8176d5bc>] ret_from_fork+0x7c/0xb0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155059]
> > [<ffffffff8108fa90>] ? flush_kthread_worker+0xb0/0xb0
> > Dec 18 00:57:41 unassigned-hostname kernel: [ 1606.155060] Code: e2 f8
> > 0f 00 00 48 8b 04 02 48 85 c0 75 25 f6 05 29 25 01 00 04 75 3e 31 c0
> > 48 83 c4 08 5b 41 5c 41 5d 5d c3 66 44 39 60 30 74 ee <48> 8b 00 48 85
> > c0 74 db 48 39 58 38 75 f2 eb e9 48 89 f2 48 c7
> >
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670082] INFO: task
> > mkfs.ext4:2235 blocked for more than 120 seconds.
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670109]
> > Tainted: G W IOX 3.13.0-43-generic #72~precise1
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670130] "echo 0 >
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670155] mkfs.ext4
> > D ffff881024fe39e0 0 2235 2080 0x00000000
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670158]
> > ffff882026eafa68 0000000000000082 ffff88103fc73480 ffff882026eaffd8
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670162]
> > 0000000000013480 0000000000013480 ffff8820293e8000 ffff88202208b000
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670165]
> > ffff882026eafa78 ffff882028a6ec00 ffff882028a6ed98 ffff882028a6ec0c
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670169] Call Trace:
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670175]
> > [<ffffffff81760ae9>] schedule+0x29/0x70
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670181]
> > [<ffffffffa01506e3>] get_active_stripe+0x2a3/0x3e0 [raid456]
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670185]
> > [<ffffffff8134c152>] ? blk_check_plugged+0x72/0xb0
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670189]
> > [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670193]
> > [<ffffffffa0155e44>] make_discard_request+0x108/0x12c4 [raid456]
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670196]
> > [<ffffffff810affe0>] ? __wake_up_sync+0x20/0x20
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670201]
> > [<ffffffffa0155c91>] make_request+0x581/0x590 [raid456]
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670204]
> > [<ffffffff8109cfd6>] ? ttwu_do_activate.constprop.82+0x66/0x70
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670207]
> > [<ffffffff8109d097>] ? ttwu_queue+0xb7/0xd0
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670210]
> > [<ffffffff8109f950>] ? try_to_wake_up+0x190/0x210
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670212]
> > [<ffffffff815d2c53>] md_make_request+0xd3/0x230
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670216]
> > [<ffffffff8115b085>] ? mempool_alloc_slab+0x15/0x20
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670219]
> > [<ffffffff8134ceb7>] generic_make_request.part.62+0x77/0xb0
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670222]
> > [<ffffffff8134d428>] generic_make_request+0x68/0x70
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670225]
> > [<ffffffff8134d4a8>] submit_bio+0x78/0x160
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670228]
> > [<ffffffff81202f80>] ? bio_alloc_bioset+0xa0/0x1d0
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670232]
> > [<ffffffff813578c0>] blkdev_issue_discard+0x1f0/0x2a0
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670235]
> > [<ffffffff8135c1f4>] blkdev_ioctl+0x354/0x810
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670238]
> > [<ffffffff8101361d>] ? __switch_to+0x16d/0x4d0
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670241]
> > [<ffffffff81204370>] block_ioctl+0x40/0x50
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670244]
> > [<ffffffff811dd5c5>] do_vfs_ioctl+0x75/0x2c0
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670247]
> > [<ffffffff817606be>] ? __schedule+0x38e/0x700
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670249]
> > [<ffffffff811dd8a1>] SyS_ioctl+0x91/0xb0
> > Dec 18 00:58:57 unassigned-hostname kernel: [ 1682.670252]
> > [<ffffffff8176d66d>] system_call_fastpath+0x1a/0x1f
> >
> >
> >
> >
> > If I do the mkfs.ext4 after the initial reconstruction is done, is
> > gets all the way through. I don't want to put this system into
> > production, since this could mean this condition could show up in the
> > future if the array needs to reconstruct again at a future point while
> > in service.
> >
> > This is a test system in a lab, so I'd be happy to try some tests.
> >
> > Terry
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: is mdadm RAID1 disk full sync
From: NeilBrown @ 2015-03-23 2:52 UTC (permalink / raw)
To: lingli tang; +Cc: linux-raid
In-Reply-To: <CAN+bsqg65AaPW1_fz=y62+7nsHNK3g1huyn3QrUEOgb-hrX=6A@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4206 bytes --]
On Sun, 22 Mar 2015 19:31:51 +0800 lingli tang <tanglingli001@gmail.com>
wrote:
> Yes, I just issue 'reboot' on Server A.
I really cannot explain that then.
If it is a completely clean "reboot" (not "reboot -f -n" for example),
then both devices should end up identical even if you weren't calling "fsync"
all the time.
> But I am curious about why 'some' request will lost to other server.
> Is It should be only one request lost(the last IO committed )
> according to full sync strategy.
Anything written since the last 'sync' or 'fsync' could be different. How
many individual requests that is might depend on the filesystem. But you
certainly shouldn't seem much difference, even if there is a crash.
Sorry I can't help more.
NeilBrown
>
> 2015-03-22 13:38 GMT+08:00 NeilBrown <neilb@suse.de>:
> > On Sun, 22 Mar 2015 13:00:54 +0800 lingli tang <tanglingli001@gmail.com>
> > wrote:
> >
> >> Thanks for reply.
> >>
> >> I have create a raid1 with two fusion io PCIe flash disk:
> >> mdadm --create /dev/md/master --name=master --level=1 --raid-devices=2
> >> /dev/fioa2 /dev/mapper/mpathc
> >> /dev/fioa2 is local disk on server A and /dev/mapper/mpathc is a iscsi
> >> load disk export from server B.
> >>
> >> After that we mkfs.ext4 on /dev/md/master and mount with 'sync' option on /data1
> >> and we will run mysql binlog on it.
> >> In order to avoid data loss of mysql binlog we have set
> >> sync_binlog=1. so every sql commit will call fsync() to flush to disk.
> >>
> >> according to your description. if we reboot the server A, the two disk
> >> data on different server will be the same.
> >> but after the server A restarted, we assemble the two disk on two
> >> server, data is different on the two server, disk on server B lost
> >> more than one sql commit.
> >
> > What exactly do you mean by "reboot"??
> > Is this a clean shutdown or do you remove the power or something like that.
> >
> > If you remove the power, then it is very possible that some requests will
> > have been submitted to one device but not the other.
> > If you have a clean shutdown, then the two devices should be identical.
> >
> > NeilBrown
> >
> >
> >>
> >> I have checked it with strace 'mysqld' on Server A.
> >> I found a sql commit and fsync() on binlog file handle on server A but
> >> this sql can not find in assembled disk on server B.
> >>
> >> I also test it with two SAS disk, Server B still has more than one sql
> >> commit lost.
> >>
> >>
> >> 2015-03-22 11:20 GMT+08:00 NeilBrown <neilb@suse.de>:
> >> > On Sat, 21 Mar 2015 19:01:54 +0800 lingli tang <tanglingli001@gmail.com>
> >> > wrote:
> >> >
> >> >> I am a newbie of mdadm. I have a question but find no answer in
> >> >> document or google for last 10 days.
> >> >>
> >> >> The question is : RAID1 made by mdadm is full sync? for example, I
> >> >> have two disk(sdb and sdc) to make RAID1 disk (/dev/md127), if I
> >> >> commit an IO to the RAID1 disk (md127), it will return back to me when
> >> >> all the two disk commit successfully or it will return back
> >> >> to me once just one of the disk successfully commit.
> >> >
> >> > The write request will not return until it has been submitted to all, and
> >> > returned by, all working devices.
> >> >
> >> >>
> >> >> I have test with xfs and ext4 with sync option, and it seems that two
> >> >> disk have lots of commit difference after reboot the server. is that
> >> >> means mdadm return success when one of the disk is commit
> >> >> successfully?
> >> >
> >> > That certainly shouldn't happen. I would need more details of the experiment
> >> > that you performed.
> >> >
> >> > NeilBrown
> >> >
> >> >
> >> >> --
> >> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> >> the body of a message to majordomo@vger.kernel.org
> >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* [GIT PULL REQUEST] one bugfix for md in 4.0-rc
From: NeilBrown @ 2015-03-22 21:17 UTC (permalink / raw)
To: Linus Torvalds; +Cc: linux RAID, lkml, Richard W.M. Jones
[-- Attachment #1: Type: text/plain, Size: 810 bytes --]
The following changes since commit 06e5801b8cb3fc057d88cb4dc03c0b64b2744cda:
Linux 4.0-rc4 (2015-03-15 17:38:20 -0700)
are available in the git repository at:
git://neil.brown.name/md/ tags/md/4.0-rc4-fix
for you to fetch changes up to 0c35bd4723e4a39ba2da4c13a22cb97986ee10c8:
md: fix problems with freeing private data after ->run failure. (2015-03-21 09:40:36 +1100)
----------------------------------------------------------------
One fix for md in 4.0-rc4
Regression in recent patch causes crash on error path.
----------------------------------------------------------------
NeilBrown (1):
md: fix problems with freeing private data after ->run failure.
drivers/md/md.c | 3 ++-
drivers/md/raid0.c | 2 --
2 files changed, 2 insertions(+), 3 deletions(-)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 17:39 UTC (permalink / raw)
To: Dan Williams
Cc: Rik van Riel, Matthew Wilcox, Andrew Morton,
linux-kernel@vger.kernel.org, linux-arch, Jens Axboe,
linux-nvdimm, Dave Hansen, linux-raid, Mel Gorman,
Christoph Hellwig, linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <CAPcyv4ieZjj6senh26BANA4KT54AdrU+O=o8zCAk7W=vA+SjnQ@mail.gmail.com>
On 03/22/2015 07:22 PM, Dan Williams wrote:
> On Sun, Mar 22, 2015 at 10:06 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
<>
>>
>> Moving to pfn's only means that all this unnamed code above that
>> "relies on struct page being PAGE_SIZE" is now not allowed to
>> interfaced with bio and sg list. Which in current code and in Dan's patches
>> means two tons of BUG_ONS and return -ENOTSUPP . For all these
>> subsystems below the bio and sglist that operate on page_structs
>
> I'm not convinced it will be that bad. In hyperbolic terms,
> continuing to overload struct page means we get to let floppy.c do i/o
> from pmem, who needs that level of compatibility?
>
But you do need to make sure it does not crash. right?
> Similar to sg_chain support I think it's fine to let sub-systems /
> archs add pmem i/o support over time. It's a scaling problem our
> development model is good at.
>
You are so eager to do all this massive change, and willing to do it
over a decade (Judging by your own example of sg-chain)
But you completely ignore the fact that what I'm saying is that
nothing needs to fundamentally change at all. No support over time
and no "scaling problem" at all. All we want to fix is that page-struct
means NOT PAGE_SIZE but some other size.
The much smaller change and full cross Kernel compatibility. What's
not to like ?
Cheers
Boaz
^ permalink raw reply
* Re: [RFC PATCH 0/7] evacuate struct page from the block layer
From: Dan Williams @ 2015-03-22 17:22 UTC (permalink / raw)
To: Boaz Harrosh
Cc: Rik van Riel, Matthew Wilcox, Andrew Morton,
linux-kernel@vger.kernel.org, linux-arch, Jens Axboe,
linux-nvdimm, Dave Hansen, linux-raid, Mel Gorman,
Christoph Hellwig, linux-fsdevel, Michael S. Tsirkin
In-Reply-To: <550EF6B2.70506@plexistor.com>
On Sun, Mar 22, 2015 at 10:06 AM, Boaz Harrosh <boaz@plexistor.com> wrote:
> On 03/20/2015 11:08 PM, Rik van Riel wrote:
>> On 03/20/2015 04:31 PM, Matthew Wilcox wrote:
> <>
>>> There's a lot of code out there that relies on struct page being PAGE_SIZE
>>> bytes. I'm cool with replacing 'struct page' with 'struct superpage'
>>> [1] in the biovec and auditing all of the code which touches it ... but
>>> that's going to be a lot of code! I'm not sure it's less code than
>>> going directly to 'just do I/O on PFNs'.
>>
>> Totally agreed here. I see absolutely no advantage to teaching the
>> IO layer about a "struct superpage" when it could operate on PFNs
>> just as easily.
>>
>
> Or teaching 'struct page' to be variable length, This is already so at
> bio and sg level so you fixed nothing.
>
> Moving to pfn's only means that all this unnamed code above that
> "relies on struct page being PAGE_SIZE" is now not allowed to
> interfaced with bio and sg list. Which in current code and in Dan's patches
> means two tons of BUG_ONS and return -ENOTSUPP . For all these
> subsystems below the bio and sglist that operate on page_structs
I'm not convinced it will be that bad. In hyperbolic terms,
continuing to overload struct page means we get to let floppy.c do i/o
from pmem, who needs that level of compatibility?
Similar to sg_chain support I think it's fine to let sub-systems /
archs add pmem i/o support over time. It's a scaling problem our
development model is good at.
^ permalink raw reply
* Re: [Linux-nvdimm] [RFC PATCH 0/7] evacuate struct page from the block layer
From: Boaz Harrosh @ 2015-03-22 17:22 UTC (permalink / raw)
To: Dan Williams, Andrew Morton
Cc: Boaz Harrosh, linux-arch, Jens Axboe, riel, linux-raid,
linux-nvdimm, Dave Hansen, linux-kernel@vger.kernel.org,
Christoph Hellwig, Mel Gorman, linux-fsdevel
In-Reply-To: <CAPcyv4j=M8V_36C-HhiJM7MHzNLFcpP=nec=LHnob5+qZ4xgYw@mail.gmail.com>
On 03/19/2015 10:59 PM, Dan Williams wrote:
>
> At least for block-i/o it seems the only place we really need struct
> page infrastructure is for kmap(). Given we already need a kmap_pfn()
> solution for option 2 a "dynamic allocation" stop along that
> development path may just naturally fall out.
Really? what about networked block-io, RDMA, FcOE emulated targets,
mmaped pointers. virtual-machine bdev drivers
Block layer sits in the middle of the stack not at the low end as you
make it appear. There are lots of below the bio subsystems that tie into
a page struct, which will now stop to operate, unless you do:
pfn_to_page() which means a page-less pfn will now crash or will need
to be rejected so any where you have a
if (page_less_pfn())
... /* Fail or do some other code like copy */
else
page = pfn_to_page()
Is a double code path in the Kernel and is a nightmare to maintain.
(I'm here for you believe me ;-) )
Thanks
Boaz
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox