* Re: raid1 - mismatches after resuming interrupted recovery
From: Jes Sorensen @ 2015-10-30 18:07 UTC (permalink / raw)
To: Nate Dailey; +Cc: NeilBrown, linux-raid
In-Reply-To: <5633A172.3020804@stratus.com>
Nate Dailey <nate.dailey@stratus.com> writes:
> This is the the same as "ignore recovery_offset if bitmap exists",
> describing how I hit the problem (before attempting to put a patch
> together to fix it).
Thanks for the clarification Nate, I thought it was two different
issues.
Clearly my grey hair is eating away at my brain. Neil please do your
best to ignore me :)
Jes
> On 10/30/2015 11:58 AM, Jes Sorensen wrote:
>> Nate Dailey <nate.dailey@stratus.com> writes:
>>> I've found that if I interrupt a recovery by removing the target
>>> device, do IO before the recovery checkpoint, then re-add the device
>>> and let the recovery complete, the mismatch_cnt is non-zero after
>>> doing a check.
>> Neil,
>>
>> While I am on the nagging path, here is another one.
>>
>> Jes
>>
>>> Here's exactly what I'm doing:
>>>
>>> - create a 5 GB raid1 with internal bitmap
>>>
>>> - do a check, verify zero mismatch_cnt
>>>
>>> - remove one member device
>>>
>>> - dd 256MB with 2GB seek
>>>
>>> - lower sync_speed_min/max to 500
>>>
>>> - re-add removed device
>>>
>>> - wait 15 sec
>>>
>>> - remove the same member device again
>>>
>>> - dd 1MB with 1 GB seek
>>>
>>> - restore sync_speed_min/max to system defaults
>>>
>>> - re-add removed device
>>>
>>> - when recovery competes, do another check
>>>
>>> At this point the mismatch_cnt is non-zero.
>>>
>>>
>>> I originally hit this on RHEL 7.1, but tested 4.1.1 from kernel.org
>>> and it happens there too.
>>>
>>> I'm out of my league in terms of trying to fix this, but would be
>>> happy to test a fix. I wonder if it's really necessary to resume a
>>> bitmap recovery from the checkpoint? Wouldn't the bitmap always
>>> reflect what needs to be copied?
>>>
>>> Nate
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: How to recover after md crash during reshape?
From: Phil Turmel @ 2015-10-30 18:12 UTC (permalink / raw)
To: Andras Tantos; +Cc: Linux-RAID
In-Reply-To: <56325086.4060104@tantosonline.com>
On 10/29/2015 12:59 PM, Andras Tantos wrote:
> Phil,
>
> On 10/28/2015 9:42 AM, Phil Turmel wrote:
>> If you stop the array cleanly and then manually re-assemble with
>> --update=metadata, you might get around it. (Specify all of the
>> devices explicitly to ensure you don't get burned by v0.90's problems
>> with last partitions.) You definitely don't want to stay on v0.90, but
>> you may need to for now to get out of trouble. Phil
>
> It seems that my mdadm doesn't have an --update=metadata option, which
> if I understand it right means I have to re-create the array with the
> no-bitmap option. How dangerous is that? Is it possible that things get
> overwritten during the re-create process in the data portion of the array?
Just clone and compile a local copy of the latest mdadm, then run it as
./mdadm for the --update operation.
git clone git://github.com/neilbrown/mdadm
> I've read that GRUB (which is my bootloader) didn't support v1.0
> superblocks for a while. It seems that 0.99 version of GRUB (which is
> what I have) has it, but how to make certain? I don't want to render my
> system un-bootable...
Old grub doesn't understand MD at all, which is why you needed a mirror
that has the content starting at the beginning of the partition. To
grub, it doesn't look like a mirror. This is true for v1.0 as well.
> Can you expand a little bit on the problems of v0.90 superblocks and why
> upgrading is advantageous? What I've read about the differences (lifted
> limit of number of devices/array and 2TB per device limit) don't really
> apply to my case.
v0.90 will screw up if you have it on the last partition of a device,
and that partition runs very close to the end of the device. v0.90
doesn't include size info in the metadata itself, so it is ambiguous in
that case whether the superblock belongs to the device as a whole or the
partition. That'll really scramble an array.
Just say no to v0.90.
Phil
^ permalink raw reply
* Re: Reconstruct a RAID 6 that has failed in a non typical manner
From: Phil Turmel @ 2015-10-30 18:31 UTC (permalink / raw)
To: Clement Parisot, linux-raid
In-Reply-To: <1874721715.14008052.1446134381481.JavaMail.zimbra@inria.fr>
Good afternoon, Clement, Marc,
On 10/29/2015 11:59 AM, Clement Parisot wrote:
> we've got a problem with our old RAID 6.
> After an electrical maintenance, 2 of our HDD came in fail state. An alert was sent that said everything was reconstructing.
> md1 reconstruction works but md2 failed as a 3rd HDD seems to be broked. A new disk has been successfully added to replace a failed one.
> All of the disks of md2 changed to Spare state. We rebooted the server but it was worse.
> As you can see, RAID is in "active, FAILED, Not Started" State. We tried to add the new disk, re-add the previously removed disks as they appears to have no errors.
> 2/3 of the disks should still contains the datas. We want to recover it.
Your subject is inaccurate. You've described a situation that is
extraordinarily common when using green drives. Or any modern desktop
drive -- they aren't rated for use in raid arrays. Please read the
references in the post-script.
> I tried procedure on RAID_Recovery wiki
> mdadm --assemble --force /dev/md2 /dev/sda /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp
> but it failed.
> mdadm: failed to add /dev/sdg to /dev/md2: Device or resource busy
> mdadm: failed to RUN_ARRAY /dev/md2: Input/output error
> mdadm: Not enough devices to start the array.
Did you run "mdadm --stop /dev/md2" first? That would explain the
"busy" reports.
Before proceeding, please supply more information:
for x in /dev/sd[a-p] ; mdadm -E $x ; smartctl -i -A -l scterc $x ; done
Paste the output inline in your response.
Phil
[1] http://marc.info/?l=linux-raid&m=139050322510249&w=2
[2] http://marc.info/?l=linux-raid&m=135863964624202&w=2
[3] http://marc.info/?l=linux-raid&m=135811522817345&w=1
[4] http://marc.info/?l=linux-raid&m=133761065622164&w=2
[5] http://marc.info/?l=linux-raid&m=132477199207506
[6] http://marc.info/?l=linux-raid&m=133665797115876&w=2
[7] http://marc.info/?l=linux-raid&m=142487508806844&w=3
[8] http://marc.info/?l=linux-raid&m=144535576302583&w=2
^ permalink raw reply
* Accidentally added as spare
From: Ian Radford @ 2015-10-30 19:07 UTC (permalink / raw)
To: linux-raid
I recently had a drive fail in a 3 drive RAID 5 array that had been up
for nearly 3 years. I've handled this several time before but this
time, whilst swapping out the failed drive, a 2nd drive dropped out of
the array for some reason and when I tried to add it back, became
marked as a spare. I haven't done anything with the new drive yet so
whenever I try to start the array, I get one active drive and one
spare and hence the array can't start. I can't work out how to prevent
the 2nd drive being detected as a spare and allow the array to start
with 2 of 3 drives. As far as I can tell from using --examine, the
event counts are identical so I'm hoping recovery is feasible.
Unfortunately I don't know what order the disks were originally in
(all of the original disks have failed and been replaced during the
array's life so far).
So, 2 questions: Is there a simple way to stop the 2nd drive thinking
that it's a spare so I can just assemble the array. If not, I believe
I should be able to recreate the array with --assume-clean: How do I
determine the correct order for the drives? (If it's a matter of trial
and error, how do I know when I get it right?)
The output from --examine on the two drives is:
/dev/sdb2:
Magic : a92b4efc
Version : 0.90.00
UUID : cd57d356:c19317bf:a7df5645:f34ebdb2
Creation Time : Fri Mar 20 17:53:13 2009
Raid Level : raid5
Used Dev Size : 1454894336 (1387.50 GiB 1489.81 GB)
Array Size : 2909788672 (2774.99 GiB 2979.62 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 1
Update Time : Thu Oct 29 18:37:53 2015
State : clean
Active Devices : 1
Working Devices : 2
Failed Devices : 2
Spare Devices : 1
Checksum : ca5c1582 - correct
Events : 3329174
Layout : left-symmetric
Chunk Size : 256K
Number Major Minor RaidDevice State
this 0 8 66 0 active sync
0 0 8 66 0 active sync
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 50 3 spare /dev/sdd2
/dev/sdd2:
Magic : a92b4efc
Version : 0.90.00
UUID : cd57d356:c19317bf:a7df5645:f34ebdb2
Creation Time : Fri Mar 20 17:53:13 2009
Raid Level : raid5
Used Dev Size : 1454894336 (1387.50 GiB 1489.81 GB)
Array Size : 2909788672 (2774.99 GiB 2979.62 GB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 1
Update Time : Thu Oct 29 18:37:53 2015
State : clean
Active Devices : 1
Working Devices : 2
Failed Devices : 2
Spare Devices : 1
Checksum : ca5c1572 - correct
Events : 3329174
Layout : left-symmetric
Chunk Size : 256K
Number Major Minor RaidDevice State
this 3 8 50 3 spare /dev/sdd2
0 0 8 66 0 active sync
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 50 3 spare /dev/sdd2
^ permalink raw reply
* GOOD NEWS
From: Carlos @ 2015-10-30 21:09 UTC (permalink / raw)
To: Recipients
GREETINGS,
My name is Carlos Slim Helu, A philanthropist the CEO and Chairman of the Carlos Slim Helu Foundation Charitable Foundation, one of the largest private foundations in the world. I believe strongly in ‘giving while living’ I had one idea that never changed in my mind — that you should use your wealth to help people and i have decided to secretly give {$2,500,000.00} Two Million Five Hundred Thousand United Dollars, to randomly selected individuals worldwide. On receipt of this email, you should count yourself as the lucky individual. Your email address was chosen online while searching at random. Kindly get back to me with your full address at your earliest convenience, so I know your email address is valid.
Visit the web page to know more about me: http://www.bornrich.com/carlos-slim-helu.html or you can Google me ( Carlos Slim Helu ) contact me at this email id (carlos.s.helu1@gmail.com) .
Regards,
Carlos Slim Helu.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] md/raid5: fix locking in handle_stripe_clean_event()
From: Neil Brown @ 2015-10-30 22:16 UTC (permalink / raw)
To: Shaohua Li, Roman Gushchin
Cc: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org
In-Reply-To: <20151030162443.GA31413@kernel.org>
[-- Attachment #1: Type: text/plain, Size: 2819 bytes --]
On Sat, Oct 31 2015, Shaohua Li wrote:
> On Fri, Oct 30, 2015 at 05:02:47PM +0300, Roman Gushchin wrote:
>> > Isn't the 4.1 fix just:
>> >
>> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> > index e5befa356dbe..6e4350a78257 100644
>> > --- a/drivers/md/raid5.c
>> > +++ b/drivers/md/raid5.c
>> > @@ -3522,16 +3522,16 @@ returnbi:
>> > * no updated data, so remove it from hash list and the stripe
>> > * will be reinitialized
>> > */
>> > - spin_lock_irq(&conf->device_lock);
>> > unhash:
>> > + spin_lock_irq(conf->hash_locks + sh->hash_lock_index);
>> > remove_hash(sh);
>> > + spin_unlock_irq(conf->hash_locks + sh->hash_lock_index);
>> > if (head_sh->batch_head) {
>> > sh = list_first_entry(&sh->batch_list,
>> > struct stripe_head, batch_list);
>> > if (sh != head_sh)
>> > goto unhash;
>> > }
>> > - spin_unlock_irq(&conf->device_lock);
>> > sh = head_sh;
>> >
>> > if (test_bit(STRIPE_SYNC_REQUESTED, &sh->state))
>> >
>> > ??
>>
>> In my opion, this patch looks correct, although it seems to me, that there is an another issue here.
>>
>> > if (head_sh->batch_head) {
>> > sh = list_first_entry(&sh->batch_list,
>> > struct stripe_head, batch_list);
>> > if (sh != head_sh)
>> > goto unhash;
>> > }
>>
>> With a patch above this code will be executed without taking any locks. It it correct?
>> In my opinion, we need to take at least sh->stripe_lock, which protects sh->batch_head.
>> Or do I miss something?
>>
>> If you want, we can handle this issue separately.
>
> The batch_list list doesn't need the protection. Only the remove_hash() need it.
Yes, that's my understanding too. The key to understanding is that
comment you (helpfully!) put in clear_batch_ready():
/*
* BATCH_READY is cleared, no new stripes can be added.
* batch_list can be accessed without lock
*/
I'll wrangle some patches...
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: [PATCH] drivers/md/md.c: ignore recovery_offset if bitmap exists
From: Neil Brown @ 2015-10-31 0:26 UTC (permalink / raw)
To: Nate Dailey, linux-raid; +Cc: Jes.Sorensen
In-Reply-To: <563370F8.3070100@stratus.com>
[-- Attachment #1: Type: text/plain, Size: 2611 bytes --]
On Sat, Oct 31 2015, Nate Dailey wrote:
> I first tested 4.3-rc6 that I already had laying around, and verified that the
> bug still happens.
>
> Then I reverted 7eb418851f3278de67126ea0c427641ab4792c57, rebuilt & installed,
> and tested again. Reverting this patch did indeed fix the bug.
>
> Thank you!
>
> Nate
>
Thanks a lot for testing.
The following with go to Linus soon, hopefully in time for 4.3-final.
It should then be picked up by stable.
If anyone wants to try their hand at the "future patch" I mentioned, I
wouldn't object. MD_FEATURE_RECOVERY_BITMAP is an important part of the
picture. I won't have an opportunity to work on it until December.
Thanks,
NeilBrown
commit d01552a76d71f9879af448e9142389ee9be6e95b
Author: NeilBrown <neilb@suse.com>
Date: Sat Oct 31 11:00:56 2015 +1100
Revert "md: allow a partially recovered device to be hot-added to an array."
This reverts commit 7eb418851f3278de67126ea0c427641ab4792c57.
This commit is poorly justified, I can find not discusison in email,
and it clearly causes a problem.
If a device which is being recovered fails and is subsequently
re-added to an array, there could easily have been changes to the
array *before* the point where the recovery was up to. So the
recovery must start again from the beginning.
If a spare is being recovered and fails, then when it is re-added we
really should do a bitmap-based recovery up to the recovery-offset,
and then a full recovery from there. Before this reversion, we only
did the "full recovery from there" which is not corect. After this
reversion with will do a full recovery from the start, which is safer
but not ideal.
It will be left to a future patch to arrange the two different styles
of recovery.
Reported-and-tested-by: Nate Dailey <nate.dailey@stratus.com>
Signed-off-by: NeilBrown <neilb@suse.com>
Cc: stable@vger.kernel.org (3.14+)
Fixes: 7eb418851f32 ("md: allow a partially recovered device to be hot-added to an array.")
diff --git a/drivers/md/md.c b/drivers/md/md.c
index c702de18207a..3fe3d04a968a 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -8040,8 +8040,7 @@ static int remove_and_add_spares(struct mddev *mddev,
!test_bit(Bitmap_sync, &rdev->flags)))
continue;
- if (rdev->saved_raid_disk < 0)
- rdev->recovery_offset = 0;
+ rdev->recovery_offset = 0;
if (mddev->pers->
hot_add_disk(mddev, rdev) == 0) {
if (sysfs_link_rdev(mddev, rdev))
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply related
* [GIT PULL REQUEST] last minute bug fixes for md.
From: Neil Brown @ 2015-10-31 1:26 UTC (permalink / raw)
To: Linus Torvalds
Cc: Nate Dailey, Roman Gushchin, Shaohua Li, linux-raid, LKML,
Jes.Sorensen
[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]
they just keep coming in :-(
The following changes since commit 8bce6d35b308d73cdb2ee273c95d711a55be688c:
md/raid10: fix the 'new' raid10 layout to work correctly. (2015-10-24 16:24:25 +1100)
are available in the git repository at:
git://neil.brown.name/md tags/md/4.3-rc7-fixes
for you to fetch changes up to d01552a76d71f9879af448e9142389ee9be6e95b:
Revert "md: allow a partially recovered device to be hot-added to an array." (2015-10-31 11:00:56 +1100)
----------------------------------------------------------------
two more bug fixes for md.
One bugfix for a list corruption in raid5 because of incorrect
locking.
Other for possible data corruption when a recovering device is failed,
removed, and re-added.
Both tagged for -stable.
----------------------------------------------------------------
NeilBrown (1):
Revert "md: allow a partially recovered device to be hot-added to an array."
Roman Gushchin (1):
md/raid5: fix locking in handle_stripe_clean_event()
drivers/md/md.c | 3 +--
drivers/md/raid5.c | 6 ++++--
2 files changed, 5 insertions(+), 4 deletions(-)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Advice requested
From: o1bigtenor @ 2015-10-31 11:55 UTC (permalink / raw)
To: linux-raid
Greetings
I had a raid 10 array where one drive threw an error (seemingly in a
superblock) and since I have not been able to get the array readable.
It was suggested to me that I remove 2 of the drives leaving the two
that were striped together.
So what I have is really a raid 0 array at present.
Info following from queries to system for system information.
mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Raid Level : raid0
Total Devices : 2
Persistence : Superblock is persistent
State : inactive
Name : debianbase:0 (local to host debianbase)
UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
Events : 47341
Number Major Minor RaidDevice
- 8 17 - /dev/sdb1
- 8 49 - /dev/sdd1
root@debianbase:/media/darald# echo $UUID
root@debianbase:/media/xxxxxx# cd /
root@debianbase:/# mdadm --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
Name : debianbase:0 (local to host debianbase)
Creation Time : Mon Mar 5 08:26:28 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1024 sectors
State : clean
Device UUID : a80c76db:eaea61af:bcb9cbbb:ac99e467
Update Time : Fri Aug 28 10:38:32 2015
Checksum : 29a4fa98 - correct
Events : 47341
Layout : near=2
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@debianbase:/# mdadm --examine /dev/sdd1
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
Name : debianbase:0 (local to host debianbase)
Creation Time : Mon Mar 5 08:26:28 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1024 sectors
State : clean
Device UUID : 9e749fa9:a0efe791:ea09d2e2:72b99f6c
Update Time : Fri Aug 28 10:38:32 2015
Checksum : 615d736 - correct
Events : 47341
Layout : near=2
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@debianbase:/#
Tried the following:
fsck.ext4 -N -b /dev/md0
e2fsck 1.42.13 (17-May-2015)
fsck.ext4: Invalid argument while trying to open /dev/md0
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
root@debianbase:/# dump2fs /dev/md0 | grep superblock
bash: dump2fs: command not found
root@debianbase:/# dumpe2fs /dev/md0 | grep superblock
dumpe2fs 1.42.13 (17-May-2015)
dumpe2fs: Invalid argument while trying to open /dev/md0
Couldn't find valid filesystem superblock.
root@debianbase:/# mdadm -A --force /dev/md0 /dev/sd[b1d1]2
mdadm: cannot open device /dev/sd[b1d1]2: No such file or directory
mdadm: /dev/sd[b1d1]2 has no superblock - assembly aborted
root@debianbase:/# mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdd1
mdadm: /dev/sdb1 is busy - skipping
mdadm: /dev/sdd1 is busy - skipping
root@debianbase:/# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
root@debianbase:/# mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdd1
mdadm: /dev/md0 assembled from 2 drives - need all 4 to start it (use
--run to insist).
root@debianbase:/# mdadm --assemble --run /dev/md0 /dev/sdb1 /dev/sdd1
mdadm: /dev/sdb1 is busy - skipping
mdadm: /dev/sdd1 is busy - skipping
drives were all ext4 when the array was setup.
Is there some way to take the 2 drives and convince mdadm that they
are a raid 0?
How do I get the superblock issue to go away?
TIA
Dee
^ permalink raw reply
* Re: [PATCH] md/raid5: fix locking in handle_stripe_clean_event()
From: Roman Gushchin @ 2015-10-31 12:25 UTC (permalink / raw)
To: Neil Brown, Shaohua Li
Cc: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org
In-Reply-To: <877fm4vw6j.fsf@notabene.neil.brown.name>
Ok, thank you for clarifications!
--
Roman
31.10.2015, 01:17, "Neil Brown" <neilb@suse.de>:
> On Sat, Oct 31 2015, Shaohua Li wrote:
>
>> On Fri, Oct 30, 2015 at 05:02:47PM +0300, Roman Gushchin wrote:
>>> > Isn't the 4.1 fix just:
>>> >
>>> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>>> > index e5befa356dbe..6e4350a78257 100644
>>> > --- a/drivers/md/raid5.c
>>> > +++ b/drivers/md/raid5.c
>>> > @@ -3522,16 +3522,16 @@ returnbi:
>>> > * no updated data, so remove it from hash list and the stripe
>>> > * will be reinitialized
>>> > */
>>> > - spin_lock_irq(&conf->device_lock);
>>> > unhash:
>>> > + spin_lock_irq(conf->hash_locks + sh->hash_lock_index);
>>> > remove_hash(sh);
>>> > + spin_unlock_irq(conf->hash_locks + sh->hash_lock_index);
>>> > if (head_sh->batch_head) {
>>> > sh = list_first_entry(&sh->batch_list,
>>> > struct stripe_head, batch_list);
>>> > if (sh != head_sh)
>>> > goto unhash;
>>> > }
>>> > - spin_unlock_irq(&conf->device_lock);
>>> > sh = head_sh;
>>> >
>>> > if (test_bit(STRIPE_SYNC_REQUESTED, &sh->state))
>>> >
>>> > ??
>>>
>>> In my opion, this patch looks correct, although it seems to me, that there is an another issue here.
>>>
>>> > if (head_sh->batch_head) {
>>> > sh = list_first_entry(&sh->batch_list,
>>> > struct stripe_head, batch_list);
>>> > if (sh != head_sh)
>>> > goto unhash;
>>> > }
>>>
>>> With a patch above this code will be executed without taking any locks. It it correct?
>>> In my opinion, we need to take at least sh->stripe_lock, which protects sh->batch_head.
>>> Or do I miss something?
>>>
>>> If you want, we can handle this issue separately.
>>
>> The batch_list list doesn't need the protection. Only the remove_hash() need it.
>
> Yes, that's my understanding too. The key to understanding is that
> comment you (helpfully!) put in clear_batch_ready():
>
> /*
> * BATCH_READY is cleared, no new stripes can be added.
> * batch_list can be accessed without lock
> */
>
> I'll wrangle some patches...
>
> Thanks,
> NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: quickest way for complex operation?
From: Jens-U. Mozdzen @ 2015-10-31 23:33 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20151030142606.Horde.D8WmIG47kX-kazg2Xb68-wV@www3.nde.ag>
Hi,
Zitat von "Jens-U. Mozdzen" <jmozdzen@nde.ag>:
> [...] Can I both replace and resize in a single (or almost single) step?
>
> I had hoped that something along the following might work:
>
> - fail one of the remaining 4 old drives
> - mark the other three old drives for replace
> - reshape the RAID to 6 disks (which probably will start right away)
> - insert the three new disks and add them as spares
I've tried, and it seems it's handled purely sequentially.
Unfortunately, I've run into totally unrelated problems during above
steps, that had me reinstall the node completely - and it was much
quicker to re-create the RAID set with just the new disks instead of
waiting for the reshape to complete.
Regards,
Jens
^ permalink raw reply
* Re: Advice requested
From: Phil Turmel @ 2015-11-01 20:55 UTC (permalink / raw)
To: o1bigtenor, linux-raid
In-Reply-To: <CAPpdf58rFk95KYGc3WkOQGg+ytJ3rcmWtu23ZU3-ss=vx_GfhQ@mail.gmail.com>
Hi Dee,
On 10/31/2015 07:55 AM, o1bigtenor wrote:
> Greetings
>
> I had a raid 10 array where one drive threw an error (seemingly in a
> superblock) and since I have not been able to get the array readable.
> It was suggested to me that I remove 2 of the drives leaving the two
> that were striped together.
[trim /]
What happened to the other two drives? If you still have them, please
supply "mdadm -E" report for them.
Also supply "smartctl -i -A -l scterc /dev/sdX" for each of your drives.
> Is there some way to take the 2 drives and convince mdadm that they
> are a raid 0?
Shouldn't be necessary. Whatever parts of your array that can be made
to run as originally is the safest way to proceed. Also, you haven't
provided any information about your original problem that suggests you
received good advice. (Probably not.)
You attempted a number of commands with invalid bracket syntax, so
received file not found errors that you ignored. You also attempted
filesystem diagnostics on md0 while it was inactive -- that's why those
failed.
You also attempted some operations after partial successes. When an
array is partially assembled but not running, before further attempts to
assemble you must stop the array:
mdadm --stop /dev/md0
Before I suggest anything else, I'd like to see the complete reports for
all four drives.
Please don't do anything else. Your report suggests you are trying
things almost at random in a desperate attempt to salvage your data.
Slow down. We can help if you don't ruin things in the meantime.
Phil
^ permalink raw reply
* [GIT PULL REQUEST] md updates for 4.4
From: Neil Brown @ 2015-11-02 2:37 UTC (permalink / raw)
To: Linus Torvalds
Cc: Bart Van Assche, Christoph Hellwig, Goldwyn Rodrigues,
Guoqing Jiang, Shaohua Li, Song Liu, kbuild test robot, LKML,
linux-raid
[-- Attachment #1: Type: text/plain, Size: 5770 bytes --]
The following changes since commit 25cb62b76430a91cc6195f902e61c2cb84ade622:
Linux 4.3-rc5 (2015-10-11 11:09:45 -0700)
are available in the git repository at:
git://neil.brown.name/md tags/md/4.4
for you to fetch changes up to 339421def582abb14c2217aa8c8f28bb2e299174:
MD: when RAID journal is missing/faulty, block RESTART_ARRAY_RW (2015-11-01 13:48:29 +1100)
----------------------------------------------------------------
md updates for 4.4.
Two major components to this update.
1/ the clustered-raid1 support from SUSE is nearly
complete. There are a few outstanding issues being
worked on. Maybe half a dozen patches will bring
this to a usable state.
2/ The first stage of journalled-raid5 support from
Facebook makes an appearance. With a journal
device configured (typically NVRAM or SSD), the
"RAID5 write hole" should be closed - a crash
during degraded operations cannot result in data
corruption.
The next stage will be to use the journal as a
write-behind cache so that latency can be reduced
and in some cases throughput increased by
performing more full-stripe writes.
----------------------------------------------------------------
Christoph Hellwig (12):
raid5-cache: move functionality out of __r5l_set_io_unit_state
raid5-cache: free I/O units earlier
raid5-cache: rename flushed_ios to finished_ios
raid5-cache: factor out a helper to run all stripes for an I/O unit
raid5-cache: simplify state machine when caches flushes are not needed
raid5-cache: clean up r5l_get_meta
raid5-cache: refactor bio allocation
raid5-cache: take rdev->data_offset into account early on
raid5-cache: inline r5l_alloc_io_unit into r5l_new_meta
raid5-cache: new helper: r5_reserve_log_entry
raid5-cache: small log->seq cleanup
raid5-cache: use bio chaining
Goldwyn Rodrigues (11):
md-cluster: complete all write requests before adding suspend_info
md: Increment version for clustered bitmaps
md-cluster: Use a small window for resync
md-cluster: Wake up suspended process
md: remove_and_add_spares() to activate specific rdev
md-cluster: Improve md_reload_sb to be less error prone
md-cluster: Perform a lazy update
md-cluster: Perform resync/recovery under a DLM lock
md-cluster: Fix adding of new disk with new reload code
md-cluster: Do not printk() every received message
md-cluster: Call update_raid_disks() if another node --grow's raid_disks
Guoqing Jiang (9):
md-cluster: send BITMAP_NEEDS_SYNC when node is leaving cluster
md-cluster: make other members of cluster_msg is handled by little endian funcs
md-cluster: remove unnecessary setting for slot
md-cluster: make sure the node do not receive it's own msg
md-cluster: zero cmsg before it was sent
md-cluster: Add 'SUSE' as author for md-cluster.c
md-cluster: only call kick_rdev_from_array after remove disk successfully
md: check the return value for metadata_update_start
md-cluster: Fix warnings when build with CF=-D__CHECK_ENDIAN__
NeilBrown (5):
Merge branch 'md-next' of git://github.com/goldwynr/linux into for-next
md-cluster: metadata_update_finish: consistently use cmsg.raid_slot as le32
md-cluster: discard unused sb_mutex.
md-cluster: don't cast void pointers when assigning them.
md-cluster: remove mddev arg from add_resync_info()
Shaohua Li (24):
md: override md superblock recovery_offset for journal device
raid5: export some functions
raid5: add a new state for stripe log handling
raid5: add basic stripe log
raid5: log reclaim support
raid5: log recovery
raid5-cache: use crc32c checksum
raid5: disable batch with log enabled
raid5: don't allow resize/reshape with cache(log) support
raid5: enable log for raid array with cache disk
raid5-cache: switching to state machine for log disk cache flush
raid5-cache: fix a user-after-free bug
raid5-cache: optimize FLUSH IO with log enabled
md: skip resync for raid array with journal
raid5-cache: check stripe finish out of order
raid5-cache: don't delay stripe captured in log
md: show journal for journal disk in disk state sysfs
raid5-cache: move reclaim stop to quiesce
MD: fix info output for journal disk
raid5-cache: add trim support for log
raid5: journal disk can't be removed
raid5-cache: IO error handling
raid5-cache: start raid5 readonly if journal is missing
MD: set journal disk ->raid_disk
Song Liu (6):
MD: replace special disk roles with macros
MD: add a new disk role to present write journal device
skip match_mddev_units check for special roles
MD: add new bit to indicate raid array with journal
MD: kick out journal disk if it's not fresh
MD: when RAID journal is missing/faulty, block RESTART_ARRAY_RW
drivers/md/Makefile | 2 +-
drivers/md/bitmap.c | 14 +-
drivers/md/bitmap.h | 4 +-
drivers/md/md-cluster.c | 228 ++++----
drivers/md/md-cluster.h | 12 +-
drivers/md/md.c | 495 ++++++++++++-----
drivers/md/md.h | 17 +-
drivers/md/raid1.c | 41 +-
drivers/md/raid1.h | 7 +
drivers/md/raid10.c | 2 +-
drivers/md/raid5-cache.c | 1191 ++++++++++++++++++++++++++++++++++++++++
drivers/md/raid5.c | 189 +++++--
drivers/md/raid5.h | 24 +
include/uapi/linux/raid/md_p.h | 73 ++-
14 files changed, 1989 insertions(+), 310 deletions(-)
create mode 100644 drivers/md/raid5-cache.c
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* [PATCH] md/raid5: remove redundant check in stripe_add_to_batch_list()
From: Roman Gushchin @ 2015-11-02 10:06 UTC (permalink / raw)
To: linux-kernel; +Cc: Roman Gushchin, Neil Brown, linux-raid
The stripe_add_to_batch_list() function is called only if
stripe_can_batch() returned true, so there is no need for double check.
Signed-off-by: Roman Gushchin <klamm@yandex-team.ru>
Cc: Neil Brown <neilb@suse.com>
Cc: linux-raid@vger.kernel.org
---
drivers/md/raid5.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 45933c1..f829ebc 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -768,8 +768,6 @@ static void stripe_add_to_batch_list(struct r5conf *conf, struct stripe_head *sh
int hash;
int dd_idx;
- if (!stripe_can_batch(sh))
- return;
/* Don't cross chunks, so stripe pd_idx/qd_idx is the same */
tmp_sec = sh->sector;
if (!sector_div(tmp_sec, conf->chunk_sectors))
--
2.4.3
^ permalink raw reply related
* [PATCH] mdadm: check bitmap first when reshape raid1 to raid0
From: Xiao Ni @ 2015-11-02 15:18 UTC (permalink / raw)
To: linux-raid
One raid1 with bitmap is composed by 4 disks. It'll fail when rashape to raid0
and lose 3 disks. It should check bitmap first when reshape raid1 to raid0.
And need to free subarray, close cfd in Grow_reshape.
Signed-off-by: Xiao Ni <xni@redhat.com>
---
Grow.c | 38 ++++++++++++++++++++------------------
mdadm.c | 16 ++++++++++++----
2 files changed, 32 insertions(+), 22 deletions(-)
diff --git a/Grow.c b/Grow.c
index d711884..51ed0c0 100644
--- a/Grow.c
+++ b/Grow.c
@@ -1600,15 +1600,14 @@ int Grow_reshape(char *devname, int fd,
cfd = open_dev_excl(st->container_devnm);
} else {
container = st->devnm;
- close(fd);
cfd = open_dev_excl(st->devnm);
fd = cfd;
}
if (cfd < 0) {
pr_err("Unable to open container for %s\n",
devname);
- free(subarray);
- return 1;
+ rv = 1;
+ goto release;
}
rv = st->ss->load_container(st, cfd, NULL);
@@ -1616,8 +1615,8 @@ int Grow_reshape(char *devname, int fd,
if (rv) {
pr_err("Cannot read superblock for %s\n",
devname);
- free(subarray);
- return 1;
+ rv = 1;
+ goto release;
}
/* check if operation is supported for metadata handler */
@@ -1641,8 +1640,8 @@ int Grow_reshape(char *devname, int fd,
pr_err("cannot reshape arrays in container with unsupported metadata: %s(%s)\n",
devname, container);
sysfs_free(cc);
- free(subarray);
- return 1;
+ rv = 1;
+ goto release;
}
}
sysfs_free(cc);
@@ -1662,7 +1661,8 @@ int Grow_reshape(char *devname, int fd,
s->raiddisks - array.raid_disks,
s->raiddisks - array.raid_disks == 1 ? "" : "s",
array.spare_disks + added_disks);
- return 1;
+ rv = 1;
+ goto release;
}
sra = sysfs_read(fd, NULL, GET_LEVEL | GET_DISKS | GET_DEVS
@@ -1675,17 +1675,18 @@ int Grow_reshape(char *devname, int fd,
} else {
pr_err("failed to read sysfs parameters for %s\n",
devname);
- return 1;
+ rv = 1;
+ goto release;
}
frozen = freeze(st);
if (frozen < -1) {
/* freeze() already spewed the reason */
- sysfs_free(sra);
- return 1;
+ rv = 1;
+ goto release;
} else if (frozen < 0) {
pr_err("%s is performing resync/recovery and cannot be reshaped\n", devname);
- sysfs_free(sra);
- return 1;
+ rv = 1;
+ goto release;
}
/* ========= set size =============== */
@@ -1901,8 +1902,6 @@ size_change_error:
if (array.state & (1<<MD_SB_BITMAP_PRESENT)) {
dprintf("Bitmap must be removed before level can be changed\n");
- if (cfd > -1)
- close(cfd);
rv = 1;
goto release;
}
@@ -1910,8 +1909,6 @@ size_change_error:
err = remove_disks_for_takeover(st, sra, array.layout);
if (err) {
dprintf("Array cannot be reshaped\n");
- if (cfd > -1)
- close(cfd);
rv = 1;
goto release;
}
@@ -2087,9 +2084,14 @@ size_change_error:
frozen = 0;
}
release:
- sysfs_free(sra);
if (frozen > 0)
unfreeze(st);
+ if (sra)
+ sysfs_free(sra);
+ if (cfd > -1)
+ close(cfd);
+ if (subarray)
+ free(subarray);
return rv;
}
diff --git a/mdadm.c b/mdadm.c
index f56a8cf..95db2a0 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -1299,7 +1299,8 @@ int main(int argc, char *argv[])
pr_err("'1' is an unusual number of drives for an array, so it is probably\n"
" a mistake. If you really mean it you will need to specify --force before\n"
" setting the number of drives.\n");
- exit(2);
+ rv = 2;
+ goto release;
}
}
@@ -1326,13 +1327,15 @@ int main(int argc, char *argv[])
rv = get_cluster_name(&c.homecluster);
if (rv) {
pr_err("The md can't get cluster name\n");
- exit(1);
+ rv = 1;
+ goto release;
}
}
if (c.backup_file && data_offset != INVALID_SECTORS) {
pr_err("--backup-file and --data-offset are incompatible\n");
- exit(2);
+ rv = 2;
+ goto release;
}
if ((mode == MISC && devmode == 'E')
@@ -1340,7 +1343,8 @@ int main(int argc, char *argv[])
/* Anyone may try this */;
else if (geteuid() != 0) {
pr_err("must be super-user to perform this action\n");
- exit(1);
+ rv = 1;
+ goto release;
}
ident.autof = c.autof;
@@ -1640,6 +1644,10 @@ int main(int argc, char *argv[])
autodetect();
break;
}
+
+release:
+ if (mdfd > -1)
+ close(mdfd);
exit(rv);
}
--
1.8.3.1
^ permalink raw reply related
* Re: Advice requested
From: Phil Turmel @ 2015-11-02 15:41 UTC (permalink / raw)
To: o1bigtenor, Linux-RAID
In-Reply-To: <CAPpdf5_PUpR7GhBHUaQfWb8Q+7FZenxXgFqbaN3f3r0r6NQNDA@mail.gmail.com>
Good morning Dee,
{ Added linux-raid back -- convention on kernel.org is reply-to-all }
On 11/02/2015 09:28 AM, o1bigtenor wrote:
> Reply is interleaved.
Thank you!
> On Sun, Nov 1, 2015 at 2:55 PM, Phil Turmel <philip@turmel.org> wrote:
>> What happened to the other two drives? If you still have them, please
>> supply "mdadm -E" report for them.
>
> root@debianbase:/# mdadm -E
> mdadm: No devices to examine
That means you didn't tell it what device(s) to examine, so it did
nothing. Based on your data below, I expect you need to examine
/dev/sdb1, /dev/sdc1, /dev/sde1, and /dev/sdf1.
{ A look at the section on --examine in "man mdadm" would have helped
here. Please spend some time reviewing "man mdadm" and "man md". }
It seems /dev/sdd1 is missing, so I guess the device names have changed.
Please supply the reports for all four array devices with their current
names (as shown in the smartctl reports).
Paste the result inline in your reply.
>> Also supply "smartctl -i -A -l scterc /dev/sdX" for each of your drives.
>
> please see attachment
I've converted your libreoffice file to text -- please just post text
in-line in the future, without wrapping. No attachments, no pastebin.
Some comments inline:
> root@debianbase:/# smartctl -i -A -l scterc /dev/sdb
> smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-2-amd64] (local build)
> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda 7200.12
> Device Model: ST31000524AS
> Serial Number: 9VPE3VX1
> LU WWN Device Id: 5 000c50 03ee8ad75
> Firmware Version: JC4B
> User Capacity: 1,000,204,886,016 bytes [1.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 7200 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
> Local Time is: Mon Nov 2 07:16:28 2015 CST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x000f 117 100 006 Pre-fail Always - 150964952
> 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0
> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 664
> 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
This ^^^ is good.
> 7 Seek_Error_Rate 0x000f 066 060 030 Pre-fail Always - 3829738
> 9 Power_On_Hours 0x0032 073 073 000 Old_age Always - 24498
> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 646
> 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
> 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
> 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
> 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
> 190 Airflow_Temperature_Cel 0x0022 069 052 045 Old_age Always - 31 (Min/Max 29/31)
> 194 Temperature_Celsius 0x0022 031 048 000 Old_age Always - 31 (0 17 0 0 0)
> 195 Hardware_ECC_Recovered 0x001a 035 009 000 Old_age Always - 150964952
> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
As is this ^^^
> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
> 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 26106 (151 195 0)
> 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 3874945427
> 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 3008180623
>
> SCT Error Recovery Control:
> Read: Disabled
> Write: Disabled
This ^^^ is *bad*. You have an older desktop drive that supports ERC,
but since it is a desktop drive, it's turned off by default. You *need*
a boot script that will turn it on every power cycle. Some references
on "timeout mismatch" are in the post-script.
> root@debianbase:/# smartctl -i -A -l scterc /dev/sdc
> smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-2-amd64] (local build)
> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda 7200.14 (AF)
> Device Model: ST1000DM003-1ER162
> Serial Number: Z4Y27SPH
> LU WWN Device Id: 5 000c50 079702992
> Firmware Version: CC45
> User Capacity: 1,000,204,886,016 bytes [1.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 7200 rpm
> Form Factor: 3.5 inches
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
> Local Time is: Mon Nov 2 07:16:38 2015 CST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x000f 115 099 006 Pre-fail Always - 98239568
> 3 Spin_Up_Time 0x0003 097 097 000 Pre-fail Always - 0
> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 89
> 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
> 7 Seek_Error_Rate 0x000f 100 253 030 Pre-fail Always - 308478
> 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 4258
> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 89
> 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
> 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
> 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
> 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
> 190 Airflow_Temperature_Cel 0x0022 073 053 045 Old_age Always - 27 (Min/Max 24/27)
> 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
> 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 20
> 193 Load_Cycle_Count 0x0032 099 099 000 Old_age Always - 3807
> 194 Temperature_Celsius 0x0022 027 047 000 Old_age Always - 27 (0 15 0 0 0)
> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
> 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 3830h+12m+39.793s
> 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2220627944
> 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 24802170836
>
> SCT Error Recovery Control command not supported
This ^^^ is *really bad*. You have a modern desktop drive that doesn't
support ERC at all. You *must* use a boot script to override linux's
low level driver timeout for this device. Please see the references.
> root@debianbase:/# smartctl -i -A -l scterc /dev/sde
> smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-2-amd64] (local build)
> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda 7200.12
> Device Model: ST31000524AS
> Serial Number: 9VPE44LL
> LU WWN Device Id: 5 000c50 03eed16df
> Firmware Version: JC4B
> User Capacity: 1,000,204,886,016 bytes [1.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 7200 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Mon Nov 2 07:16:45 2015 CST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x000f 118 100 006 Pre-fail Always - 184109834
> 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0
> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 663
> 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
> 7 Seek_Error_Rate 0x000f 065 060 030 Pre-fail Always - 3341907
> 9 Power_On_Hours 0x0032 073 073 000 Old_age Always - 24462
> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 644
> 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
> 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
> 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
> 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
> 190 Airflow_Temperature_Cel 0x0022 069 047 045 Old_age Always - 31 (Min/Max 25/31)
> 194 Temperature_Celsius 0x0022 031 053 000 Old_age Always - 31 (0 18 0 0 0)
> 195 Hardware_ECC_Recovered 0x001a 026 014 000 Old_age Always - 184109834
> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
> 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 26005 (154 33 0)
> 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 1062575833
> 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 3869056902
>
> SCT Error Recovery Control:
> Read: Disabled
> Write: Disabled
Again.
> root@debianbase:/# smartctl -i -A -l scterc /dev/sdf
> smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.1.0-2-amd64] (local build)
> Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda 7200.12
> Device Model: ST31000524AS
> Serial Number: 9VPE31PN
> LU WWN Device Id: 5 000c50 03eeadf56
> Firmware Version: JC4B
> User Capacity: 1,000,204,886,016 bytes [1.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 7200 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
> Local Time is: Mon Nov 2 07:17:53 2015 CST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x000f 116 099 006 Pre-fail Always - 104938837
> 3 Spin_Up_Time 0x0003 100 100 000 Pre-fail Always - 0
> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 594
> 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
> 7 Seek_Error_Rate 0x000f 066 060 030 Pre-fail Always - 3679723
> 9 Power_On_Hours 0x0032 073 073 000 Old_age Always - 24417
> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 579
> 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
> 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
> 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
> 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
> 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
> 190 Airflow_Temperature_Cel 0x0022 070 040 045 Old_age Always In_the_past 30 (19 143 30 29 0)
> 194 Temperature_Celsius 0x0022 030 060 000 Old_age Always - 30 (0 18 0 0 0)
> 195 Hardware_ECC_Recovered 0x001a 036 013 000 Old_age Always - 104938837
> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
> 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 25870 (135 125 0)
> 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 366206138
> 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 37669615
>
> SCT Error Recovery Control:
> Read: Disabled
> Write: Disabled
And again.
Looking for this is one of the reasons I asked for these reports --
non-raid rated drives producing timeout mismatch failures is a common
problem seen on this list. I also wanted to know if your drives are
generally healthy -- they are -- and how that might have impacted your
situation. { I had some of the same Seagate .12 drives -- they didn't
start failing until they had nearly 40k hours. }
Another reason I asked for this is that drive names can change on power
cycles and unplug/replug cycles. Knowing what device is what (versus
drive serial number) is important. Please verify drive serial number
versus device name after any power cycle or replug event and let us know
any changes.
> enclosed is also a copy of the kernel log which indicates that a drive error
> had occurred. My thinking is that my ups box (too small but all I could find
> when I bought the system) didn't provide enough power in a severe brown-
> out incident.
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.025162] scsi 3:0:0:0: Direct-Access ATA ST31000524AS JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.025768] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.026296] sd 3:0:0:0: [sdb] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.026304] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.026484] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.028273] sdb: sdb1
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.028698] sd 3:0:0:0: [sdb] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.086532] Switched to clocksource tsc
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.168919] md: bind<sdb1>
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.342285] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.343153] ata5.00: ATA-9: ST1000DM003-1ER162, CC45, max UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.343158] ata5.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344067] ata5.00: configured for UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344255] scsi 4:0:0:0: Direct-Access ATA ST1000DM003-1ER1 CC45 PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344627] sd 4:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344631] sd 4:0:0:0: [sdc] 4096-byte physical blocks
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344823] sd 4:0:0:0: [sdc] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344831] sd 4:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344946] sd 4:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.411364] sdc: sdc1
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.412317] sd 4:0:0:0: [sdc] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.501080] md: bind<sdc1>
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.662509] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.674767] ata6.00: ATA-8: Corsair Force 3 SSD, 1.3.3, max UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.674772] ata6.00: 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.684647] ata6.00: configured for UDMA/133
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.684933] scsi 5:0:0:0: Direct-Access ATA Corsair Force 3 3 PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.685504] sd 5:0:0:0: [sdd] 468862128 512-byte logical blocks: (240 GB/223 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.685975] sd 5:0:0:0: [sdd] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.685983] sd 5:0:0:0: [sdd] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.686186] sd 5:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.688051] sdd: sdd1 sdd4 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 >
> Aug 28 10:39:28 debiantestingbase kernel: [ 3.689305] sd 5:0:0:0: [sdd] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.002712] ata8: SATA link down (SStatus 0 SControl 300)
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003107] scsi 8:0:0:0: Direct-Access ATA ST31000524AS JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003597] sd 8:0:0:0: [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003843] scsi 9:0:0:0: Direct-Access ATA ST31000524AS JC4B PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003975] sd 8:0:0:0: [sde] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003980] sd 8:0:0:0: [sde] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004090] sd 8:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004478] sd 9:0:0:0: [sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004645] sd 9:0:0:0: [sdf] Write Protect is off
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004650] sd 9:0:0:0: [sdf] Mode Sense: 00 3a 00 00
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004737] sd 9:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004778] scsi 15:0:0:0: Processor Marvell 91xx Config 1.01 PQ: 0 ANSI: 5
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.006375] sdf: sdf1
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.006967] sd 9:0:0:0: [sdf] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.008855] sde: sde1
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.009704] sd 8:0:0:0: [sde] Attached SCSI disk
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018710] ata16.00: exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018753] ata16.00: irq_stat 0x40000001
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018783] ata16.00: cmd a0/01:00:00:00:01/00:00:00:00:
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018783] Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018868] ata16.00: status: { DRDY }
This ^^^ isn't really an error.
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.125325] random: nonblocking pool is initialized
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.125530] md: bind<sde1>
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.142140] md: bind<sdf1>
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.144984] md: raid10 personality registered for level 10
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.145397] md/raid10:md0: active with 4 out of 4 devices
If it was, you wouldn't get this ^^^ success.
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.145440] md0: detected capacity change from 0 to 2000403038208
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.208978] md0:
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.479305] device-mapper: uevent: version 1.0.3
> Aug 28 10:39:28 debiantestingbase kernel: [ 4.479536] device-mapper: ioctl: 4.30.0-ioctl (2014-12-22) initialised: dm-devel@redhat.com
This section of your syslog is too far back in time -- before the failure.
[trim /]
> Was following directions from one person on linux-raid and when he stopped
> responding turned to someone who is somewhat connected with redhat and
> hard drive stuff. There seemed to be a consensus that I needed to use low
> level tools in an attempt to recover files, if I could. As there are about 200k
> I wasn't looking forward to repairing things ;-( !
A link to the linux-raid archive of the prior discussion might help.
And you might still need low level tools -- we are still trying to get
your array started. Then we can look at the next layer on top of that.
Based on your prior mails, /dev/md0 was formatted as ext4, yes?
> Thanking you for your assistance.
You're welcome.
Phil
[1] http://marc.info/?l=linux-raid&m=139050322510249&w=2
[2] http://marc.info/?l=linux-raid&m=135863964624202&w=2
[3] http://marc.info/?l=linux-raid&m=135811522817345&w=1
[4] http://marc.info/?l=linux-raid&m=133761065622164&w=2
[5] http://marc.info/?l=linux-raid&m=132477199207506
[6] http://marc.info/?l=linux-raid&m=133665797115876&w=2
[7] http://marc.info/?l=linux-raid&m=142487508806844&w=3
[8] http://marc.info/?l=linux-raid&m=144535576302583&w=2
^ permalink raw reply
* Re: [PATCH] mdadm: check bitmap first when reshape raid1 to raid0
From: Xiao Ni @ 2015-11-02 16:04 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <1446477507-26755-1-git-send-email-xni@redhat.com>
Sorry, the patch is wrong. Please ignore this and I'll send another.
----- Original Message -----
> From: "Xiao Ni" <xni@redhat.com>
> To: linux-raid@vger.kernel.org
> Sent: Monday, November 2, 2015 11:18:27 PM
> Subject: [PATCH] mdadm: check bitmap first when reshape raid1 to raid0
>
> One raid1 with bitmap is composed by 4 disks. It'll fail when rashape to
> raid0
> and lose 3 disks. It should check bitmap first when reshape raid1 to raid0.
>
> And need to free subarray, close cfd in Grow_reshape.
>
> Signed-off-by: Xiao Ni <xni@redhat.com>
> ---
> Grow.c | 38 ++++++++++++++++++++------------------
> mdadm.c | 16 ++++++++++++----
> 2 files changed, 32 insertions(+), 22 deletions(-)
>
> diff --git a/Grow.c b/Grow.c
> index d711884..51ed0c0 100644
> --- a/Grow.c
> +++ b/Grow.c
> @@ -1600,15 +1600,14 @@ int Grow_reshape(char *devname, int fd,
> cfd = open_dev_excl(st->container_devnm);
> } else {
> container = st->devnm;
> - close(fd);
> cfd = open_dev_excl(st->devnm);
> fd = cfd;
> }
> if (cfd < 0) {
> pr_err("Unable to open container for %s\n",
> devname);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> rv = st->ss->load_container(st, cfd, NULL);
> @@ -1616,8 +1615,8 @@ int Grow_reshape(char *devname, int fd,
> if (rv) {
> pr_err("Cannot read superblock for %s\n",
> devname);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> /* check if operation is supported for metadata handler */
> @@ -1641,8 +1640,8 @@ int Grow_reshape(char *devname, int fd,
> pr_err("cannot reshape arrays in container with unsupported metadata:
> %s(%s)\n",
> devname, container);
> sysfs_free(cc);
> - free(subarray);
> - return 1;
> + rv = 1;
> + goto release;
> }
> }
> sysfs_free(cc);
> @@ -1662,7 +1661,8 @@ int Grow_reshape(char *devname, int fd,
> s->raiddisks - array.raid_disks,
> s->raiddisks - array.raid_disks == 1 ? "" : "s",
> array.spare_disks + added_disks);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> sra = sysfs_read(fd, NULL, GET_LEVEL | GET_DISKS | GET_DEVS
> @@ -1675,17 +1675,18 @@ int Grow_reshape(char *devname, int fd,
> } else {
> pr_err("failed to read sysfs parameters for %s\n",
> devname);
> - return 1;
> + rv = 1;
> + goto release;
> }
> frozen = freeze(st);
> if (frozen < -1) {
> /* freeze() already spewed the reason */
> - sysfs_free(sra);
> - return 1;
> + rv = 1;
> + goto release;
> } else if (frozen < 0) {
> pr_err("%s is performing resync/recovery and cannot be reshaped\n",
> devname);
> - sysfs_free(sra);
> - return 1;
> + rv = 1;
> + goto release;
> }
>
> /* ========= set size =============== */
> @@ -1901,8 +1902,6 @@ size_change_error:
>
> if (array.state & (1<<MD_SB_BITMAP_PRESENT)) {
> dprintf("Bitmap must be removed before level can be changed\n");
> - if (cfd > -1)
> - close(cfd);
> rv = 1;
> goto release;
> }
> @@ -1910,8 +1909,6 @@ size_change_error:
> err = remove_disks_for_takeover(st, sra, array.layout);
> if (err) {
> dprintf("Array cannot be reshaped\n");
> - if (cfd > -1)
> - close(cfd);
> rv = 1;
> goto release;
> }
> @@ -2087,9 +2084,14 @@ size_change_error:
> frozen = 0;
> }
> release:
> - sysfs_free(sra);
> if (frozen > 0)
> unfreeze(st);
> + if (sra)
> + sysfs_free(sra);
> + if (cfd > -1)
> + close(cfd);
> + if (subarray)
> + free(subarray);
> return rv;
> }
>
> diff --git a/mdadm.c b/mdadm.c
> index f56a8cf..95db2a0 100644
> --- a/mdadm.c
> +++ b/mdadm.c
> @@ -1299,7 +1299,8 @@ int main(int argc, char *argv[])
> pr_err("'1' is an unusual number of drives for an array, so it is
> probably\n"
> " a mistake. If you really mean it you will need to specify --force
> before\n"
> " setting the number of drives.\n");
> - exit(2);
> + rv = 2;
> + goto release;
> }
> }
>
> @@ -1326,13 +1327,15 @@ int main(int argc, char *argv[])
> rv = get_cluster_name(&c.homecluster);
> if (rv) {
> pr_err("The md can't get cluster name\n");
> - exit(1);
> + rv = 1;
> + goto release;
> }
> }
>
> if (c.backup_file && data_offset != INVALID_SECTORS) {
> pr_err("--backup-file and --data-offset are incompatible\n");
> - exit(2);
> + rv = 2;
> + goto release;
> }
>
> if ((mode == MISC && devmode == 'E')
> @@ -1340,7 +1343,8 @@ int main(int argc, char *argv[])
> /* Anyone may try this */;
> else if (geteuid() != 0) {
> pr_err("must be super-user to perform this action\n");
> - exit(1);
> + rv = 1;
> + goto release;
> }
>
> ident.autof = c.autof;
> @@ -1640,6 +1644,10 @@ int main(int argc, char *argv[])
> autodetect();
> break;
> }
> +
> +release:
> + if (mdfd > -1)
> + close(mdfd);
> exit(rv);
> }
>
> --
> 1.8.3.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Failed Raid 5 due to OS mbr written on one of the array drives
From: Nicolas Tellier @ 2015-11-02 17:02 UTC (permalink / raw)
To: linux-raid
I'm looking for advices regarding a failed (2 disc down) Raid 5 array
with 4 disks. It was running in a NAS for quite some time but after I
came back from a trip, I found out that the system disk was dead.
After replacing the drive, I reinstalled the OS (OpenMediaVault for
the curious). Sadly the mbr was written to one of the raid disk
instead of the OS one. This would not have been too critical if, after
booting the system, I didn't realized that the array was already
running in degraded mode prior to the OS disk problem. Luckily I have
a backup for most of the critical data on that array. There is nothing
that I cannot replace, but it would still be quite inconvenient. I
guess it's a good opportunity to learn more about raid :)
mdadm --stop /dev/md0 and removing the boot flag from the wrongly
written raid disk are the only stuff that I did so far.
Here are the informations I gathered :
root@NAStradamus:~# uname -a
Linux NAStradamus 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u5 x86_64 GNU/Linux
root@NAStradamus:~# mdadm --examine /dev/sd[a-z]1
mdadm: No md superblock detected on /dev/sda1.
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : a08bcee5:9fb42352:319ecab9:53d6277b
Name : ArchliNAS:0
Creation Time : Sun Jul 8 17:59:49 2012
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953005569 (931.27 GiB 999.94 GB)
Array Size : 2929507584 (2793.80 GiB 2999.82 GB)
Used Dev Size : 1953005056 (931.27 GiB 999.94 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9b28a04c:f2c3d6c9:6f76859d:624927f0
Update Time : Wed Sep 16 19:32:12 2015
Checksum : 4fb98985 - correct
Events : 108016
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 1
Array State : AAA. ('A' == active, '.' == missing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : a08bcee5:9fb42352:319ecab9:53d6277b
Name : ArchliNAS:0
Creation Time : Sun Jul 8 17:59:49 2012
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953005569 (931.27 GiB 999.94 GB)
Array Size : 2929507584 (2793.80 GiB 2999.82 GB)
Used Dev Size : 1953005056 (931.27 GiB 999.94 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1952e898:50043e66:8247a64d:72ffb6c0
Update Time : Wed Sep 16 19:32:12 2015
Checksum : b9197a85 - correct
Events : 108016
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 2
Array State : AAA. ('A' == active, '.' == missing)
root@NAStradamus:~# mdadm --examine /dev/sdd
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : a08bcee5:9fb42352:319ecab9:53d6277b
Name : ArchliNAS:0
Creation Time : Sun Jul 8 17:59:49 2012
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 1953523120 (931.51 GiB 1000.20 GB)
Array Size : 2929507584 (2793.80 GiB 2999.82 GB)
Used Dev Size : 1953005056 (931.27 GiB 999.94 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 425c85ec:0c038b4b:cd59b4b5:280bf233
Update Time : Mon May 4 08:18:04 2015
Checksum : cca22997 - correct
Events : 55532
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing)
root@NAStradamus:~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : inactive sdb1[1] sdc1[3]
1953005569 blocks super 1.2
unused devices: <none>
root@NAStradamus:~# mdadm --examine --scan
ARRAY /dev/md/0 metadata=1.2 UUID=a08bcee5:9fb42352:319ecab9:53d6277b
name=ArchliNAS:0
root@NAStradamus:~# fdisk -l
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
107 heads, 58 sectors/track, 314780 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00048269
Device Boot Start End Blocks Id System
/dev/sda1 2048 1953269760 976633856+ da Non-FS data
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdb1 2048 1953269760 976633856+ da Non-FS data
Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdc1 2048 1953269760 976633856+ da Non-FS data
Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdd doesn't contain a valid partition table
First off we can see that /dev/sdd is in a weird state. You need to
know that it was added later in order to grow the array. It seems that
I was drunk when I did that as I didn't put any partition on that
disk…
Besides the event count is way off. My understanding is that there is
nothing to be done here, apart from re-adding it to the array once it
is up and running again (hopefully) and once the disk has been checked
for error and reinitialized (with a proper partition this time !).
Now, /dev/sda is more interesting. The partition is still present and
looks intact, it seems like it's just missing the superblock because
of the mbr shenanigan. Also the two healthy drives still see it as
active.
After looking around on the internet, I found people suggesting to
re-create the raid. It seems a bit extreme to me, but I cannot find
any other solution…
Luckily I saved the original command used to create this array. Here
is the one I think would be relevant in this case :
mdadm --create --verbose --assume-clean /dev/md0 --level=5
--metadata=1.2 --chunk=128 --raid-devices=4 /dev/sda1 /dev/sdb1
/dev/sdc1 missing /dev/sdd
This would be followed by a backup, the re-addition of /dev/sdd and a
migration to raid 6 with 2 more disks.
The wiki advise to get an experienced person review the measures
you're about to take, I don't know anybody experienced in RAID, hence
this e-mail :) What do you think ?
Please, CC me on the answers/comments posted to the list in response
to this; I'm not subscribed to the mailing list.
Thanks in advance for your time !
Nico
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Failed Raid 5 due to OS mbr written on one of the array drives
From: Phil Turmel @ 2015-11-02 18:19 UTC (permalink / raw)
To: Nicolas Tellier, linux-raid
In-Reply-To: <CAArDwD-7=Vmo16K3GH0dxZie=RAiSavZNfMNvabf4X6G4fK34w@mail.gmail.com>
Good afternoon Nicolas,
On 11/02/2015 12:02 PM, Nicolas Tellier wrote:
> I'm looking for advices regarding a failed (2 disc down) Raid 5 array
> with 4 disks. It was running in a NAS for quite some time but after I
> came back from a trip, I found out that the system disk was dead.
> After replacing the drive, I reinstalled the OS (OpenMediaVault for
> the curious). Sadly the mbr was written to one of the raid disk
> instead of the OS one. This would not have been too critical if, after
> booting the system, I didn't realized that the array was already
> running in degraded mode prior to the OS disk problem. Luckily I have
> a backup for most of the critical data on that array. There is nothing
> that I cannot replace, but it would still be quite inconvenient. I
> guess it's a good opportunity to learn more about raid :)
Very good. Many people don't understand that RAID != Backup.
> mdadm --stop /dev/md0 and removing the boot flag from the wrongly
> written raid disk are the only stuff that I did so far.
Good.
[trim /]
> Now, /dev/sda is more interesting. The partition is still present and
> looks intact, it seems like it's just missing the superblock because
> of the mbr shenanigan. Also the two healthy drives still see it as
> active.
The superblock isn't anywhere near the MBR, so something more than just
grub-install must have happened. Be prepared to have lost more of
/dev/sda1 than you've yet discovered.
> After looking around on the internet, I found people suggesting to
> re-create the raid. It seems a bit extreme to me, but I cannot find
> any other solution…
Unfortunately, yes. You are past the point of a forced assembly or
other normal operations.
> Luckily I saved the original command used to create this array. Here
> is the one I think would be relevant in this case :
>
> mdadm --create --verbose --assume-clean /dev/md0 --level=5
> --metadata=1.2 --chunk=128 --raid-devices=4 /dev/sda1 /dev/sdb1
> /dev/sdc1 missing /dev/sdd
1) Leave off /dev/sdd
2) Include --data-offset=262144
If it runs and you can access the filesystem (I suspect not), set up a
partition on sdd and add that to your array. Use zero-superblock to
blow away the stale superblock on sdd.
If it doesn't work due to more sda1 damage and you end up starting over,
consider wiping all of those partition tables and creating the new array
on all bare devices. That'll minimize the chance of a misidentified
device later.
You'll want to investigate the health of sdd. If it's healthy, then its
drop-out must have been for some other reason. You'll want to ensure
that doesn't happen again.
Consider browsing the archives and/or subscribing to learn the many
pitfalls for the unwary (hint: try searching for "timeout mismatch").
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: performance issue
From: Rainer Fügenstein @ 2015-11-02 22:55 UTC (permalink / raw)
To: Neil Brown; +Cc: Linux-RAID
In-Reply-To: <87a8r6g188.fsf@notabene.neil.brown.name>
On 25.10.2015 21:08, Neil Brown wrote:
> mdadm --grow /dev/md0 --bitmap=intermnal --bitmap-chunk=256
not sure how to specify the chunks size:
[root@alfred ~]# mdadm --grow /dev/md0 --bitmap=internal --bitmap-chunk=256
mdadm: failed to create internal bitmap - chunksize problem.
[root@alfred ~]# mdadm --grow /dev/md0 --bitmap=internal --bitmap-chunk=256M
mdadm: invalid bitmap chunksize: 256M
^ permalink raw reply
* Re: Advice requested
From: o1bigtenor @ 2015-11-02 23:49 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux-RAID
In-Reply-To: <5637BCE4.8000006@turmel.org>
Rund again
On Mon, Nov 2, 2015 at 1:43 PM, Phil Turmel <philip@turmel.org> wrote:
> Hi Dee,
>
> On 11/02/2015 01:21 PM, o1bigtenor wrote:
>> On Mon, Nov 2, 2015 at 9:41 AM, Phil Turmel <philip@turmel.org> wrote:
>
>>> That means you didn't tell it what device(s) to examine, so it did
>>> nothing. Based on your data below, I expect you need to examine
>>> /dev/sdb1, /dev/sdc1, /dev/sde1, and /dev/sdf1.
>>
>> root@debianbase:/# mdadm -E /dev/sdb
>> /dev/sdb:
>> MBR Magic : aa55
>> Partition[0] : 1953521664 sectors at 2048 (type fd)
>> root@debianbase:/# mdadm -E /dev/sdc
>> /dev/sdc:
>> MBR Magic : aa55
>> Partition[0] : 1953525167 sectors at 1 (type ee)
>> root@debianbase:/# mdadm -E /dev/sde
>> /dev/sde:
>> MBR Magic : aa55
>> Partition[0] : 1953521664 sectors at 2048 (type fd)
>> root@debianbase:/# mdadm -E /dev/sdf
>> /dev/sdf:
>> MBR Magic : aa55
>> Partition[0] : 1953521664 sectors at 2048 (type fd)
>
> :-) You left off the partition numbers... Please try again.
mdadm -E /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
Name : debianbase:0 (local to host debianbase)
Creation Time : Mon Mar 5 08:26:28 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1024 sectors
State : clean
Device UUID : a80c76db:eaea61af:bcb9cbbb:ac99e467
Update Time : Fri Aug 28 10:38:32 2015
Checksum : 29a4fa98 - correct
Events : 47341
Layout : near=2
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
root@debianbase:/# mdadm -E /dev/sdc1
mdadm: No md superblock detected on /dev/sdc1.
root@debianbase:/# mdadm -E /dev/sdd1
mdadm: No md superblock detected on /dev/sdd1.
root@debianbase:/# mdadm -E /dev/sdf1
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
Name : debianbase:0 (local to host debianbase)
Creation Time : Mon Mar 5 08:26:28 2012
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=1024 sectors
State : clean
Device UUID : 9e749fa9:a0efe791:ea09d2e2:72b99f6c
Update Time : Fri Aug 28 10:38:32 2015
Checksum : 615d736 - correct
Events : 47341
Layout : near=2
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>
>> (I had some other system issues about 5 weeks ago an damaged the
>> superblock from sdc.)
>
> Hmmm. That makes me wonder if the correct superblock is available, but
> only if the partition is relocated back to sector 2048. We may revisit
> this.
>
>>> { A look at the section on --examine in "man mdadm" would have helped
>>> here. Please spend some time reviewing "man mdadm" and "man md". }
>
>> I have found that man pages are great resources when one does know or has
>> used the command, when you haven't - - - well there are most often NO examples,
>> either of what to do or what happens when something is wrong. They are most
>> often written by those intimately familiar with the area (never have a
>> problem with
>> anything) and so I don't bother looking at them - - - they just aren't useful.
>> (The whole RTFM thing is one of the biggest reasons that I find *nix very
>> frustrating - - - I have spent hours reading and re-reading a man page
>> and sometimes
>> four or five pages talking about it and its still not clear exactly
>> how to write the
>> command or even better if the result is not positive there is absolutely no idea
>> given as to how to troubleshoot things - - - sorry its a rant but you
>> did tell me to
>> read it and - - - - .)
>
> You may have to conquer this to succeed. While man pages certainly
> aren't perfect, they are the most complete documentation available on
> any linux system. Most people in your situation struggle with the
> 'synopsis' section, and how to decipher the brackets and braces and
> such. Google 'reading man pages' and sample the results for a wide
> overview. Save the deep dive on 'Backus-Naur' for last. :-)
Sounds like you like the present layout. If there were examples included I would
think that man pages are useful. As they are at present - - - well I
have an easier
time figuring out ancient Hebrew - - - at least there are tools that work.
Brackets and braces aren't the issue - - - good technical writing is. Others may
find that man pages are 'complete' - - - I would disagree. (I have worked
quite a few years in technical fields and find that computer documentation is
some of the most difficult to follow - - - I've even had to decipher
English written
in China - - - found that easier as they followed a standardized layout.)
>
>>> It seems /dev/sdd1 is missing, so I guess the device names have changed.
>>> Please supply the reports for all four array devices with their current
>>> names (as shown in the smartctl reports).
>>
>> sda is my current 'operating system' disc - - - or where I want my operating
>> system to reside. I wanted all my information to be on the raid array.
>>
>> sdd is a ssd drive that is presently not used and was used previously as
>> an 'operating system' disc.
>>
>> Therefore neither of these really has anything to do with the raid array.
>
> Well, your initial report referred to sdb1 and sdd1, so I clearly had to
> ask.
With two drives removed from access then sdb1 and sdd1 were the useful drives.
Now with all drives plugged in we are still looking at sdb1 and sdd1 as being
the most likely accessible drives.
>
>>> Looking for this is one of the reasons I asked for these reports --
>>> non-raid rated drives producing timeout mismatch failures is a common
>>> problem seen on this list. I also wanted to know if your drives are
>>> generally healthy -- they are -- and how that might have impacted your
>>> situation. { I had some of the same Seagate .12 drives -- they didn't
>>> start failing until they had nearly 40k hours. }
>>
>> When these drives were bought, late January 2012 there was no differentiation
>> between drives based upon intended usage. That happened some time in
>> about 2013 IIRC.
>
> Drive manufacturers have always pitched their enterprise drives for raid
> duty, but only started sabotaging raid support in desktop drives in the
> 2011 time frame (as shown in the email archives I referenced).
>
> But even before they started cutting ERC support, you had to activate it
> manually to be safe in a raid array. So no, desktop drives have never
> been safe to use in MD raid "out of the box". And still are not.
>
>>> Another reason I asked for this is that drive names can change on power
>>> cycles and unplug/replug cycles. Knowing what device is what (versus
>>> drive serial number) is important. Please verify drive serial number
>>> versus device name after any power cycle or replug event and let us know
>>> any changes.
>>
>> There have been no real changes in quite a long time.
>> The only UUID change was between before Aug 28th of this and after when the
>> UUID of the array was changed. It was not finding the array upon a reboot (used
>> to improve web speeds changed by Firefox's software) that informed me that I
>> had an issue.
>
> The device you originally referred to as /dev/sdd is now apparently
> something else. So yes, you are subject to device name changes.
Yes something changed from this morning to now. The way it is now makes more
sense but I didn't change anything.
>
> A UUID change only happens when an array is re-created. If you've lost
> the original superblocks, recovery may not be possible. Losing access
> at boot time is often a problem with updates to an initramfs, not to any
> actual problem with an array.
>
>>> This ^^^ isn't really an error.
>> If it isn't an error this was the only place where there were any
>> listed changes to the array and/or its discs.
>>
>> Subsequently I was unable to mount the array nor did it mount automatically as
>> it had for the previous time (at least 4 months).
>>>
>>>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.125325] random: nonblocking pool is initialized
>>>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.125530] md: bind<sde1>
>>>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.142140] md: bind<sdf1>
>>>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.144984] md: raid10 personality registered for level 10
>>>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.145397] md/raid10:md0: active with 4 out of 4 devices
>>>
>>> If it was, you wouldn't get this ^^^ success.
>> May have been a success but the UUID was changed and I could find no way
>> to access the array (still can't for that matter!).
>
> Which suggests that the new UUID array is not pointing at the data you
> need. If the other two drives still have the old superblocks, it might
> still be possible.
>
>>> This section of your syslog is too far back in time -- before the failure.
>> This section is AT the time of failure.
>> Md0 has not been accessible subsequently!
>
> Then all four drives have new superblocks due to some form of array
> creation that has blown away the others. Since the array won't mount,
> you had either incorrect array creation parameters and/or some operation
> stomping on the contents.
>
>>>
>>> [trim /]
>>>
>>>> Was following directions from one person on linux-raid and when he stopped
>>>> responding turned to someone who is somewhat connected with redhat and
>>>> hard drive stuff. There seemed to be a consensus that I needed to use low
>>>> level tools in an attempt to recover files, if I could. As there are about 200k
>>>> I wasn't looking forward to repairing things ;-( !
>>>
>>> A link to the linux-raid archive of the prior discussion might help.
>>> And you might still need low level tools -- we are still trying to get
>>> your array started. Then we can look at the next layer on top of that.
>>> Based on your prior mails, /dev/md0 was formatted as ext4, yes?
>>
>> Yes.
>
> Based on your comments below, NO.
>
>> o1bigtenor <o1bigtenor@gmail.com>
>> Sep 3
>> Reply
>> to linux-raid
>> Greetings
>>
>> Had updated a system to Debian 8 which also had a Raid 10 array that
>> has been in use for about 3 1/2 years. (Setup raid under Debian 6 then
>> ran it mostly under Debian 7 mounting the array each time after
>> booting using the command
>>
>> #mount /dev/dm-o /home/myspace/RAID
>
> This indicates that you had a device mapper layer on top of the raid
> array, with the filesystem in that layer.
>
> I trimmed the rest of this mail in favor of the archive link:
> http://marc.info/?l=linux-raid&m=144131963804698&w=2
>
snip
>> # mdadm -D
>> mdadm: No devices given.
>>
>>
>>
>>> member
>>> details (mdadm -E)
>>
>> # mdadm -E
>> mdadm: No devices to examine
>>
>>
>>> and a summary of what happened.
>>> Include any excerpts from your dmesg and/or syslogs that look like they
>>> might be relevant.
>>
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.025162] scsi 3:0:0:0:
>> Direct-Access ATA ST31000524AS JC4B PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.025768] sd 3:0:0:0:
>> [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.026296] sd 3:0:0:0:
>> [sdb] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.026304] sd 3:0:0:0:
>> [sdb] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.026484] sd 3:0:0:0:
>> [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO
>> or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.028273] sdb: sdb1
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.028698] sd 3:0:0:0:
>> [sdb] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.086532] Switched to
>> clocksource tsc
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.168919] md: bind<sdb1>
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.342285] ata5: SATA
>> link up 3.0 Gbps (SStatus 123 SControl 300)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.343153] ata5.00:
>> ATA-9: ST1000DM003-1ER162, CC45, max UDMA/133
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.343158] ata5.00:
>> 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344067] ata5.00:
>> configured for UDMA/133
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344255] scsi 4:0:0:0:
>> Direct-Access ATA ST1000DM003-1ER1 CC45 PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344627] sd 4:0:0:0:
>> [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344631] sd 4:0:0:0:
>> [sdc] 4096-byte physical blocks
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344823] sd 4:0:0:0:
>> [sdc] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344831] sd 4:0:0:0:
>> [sdc] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.344946] sd 4:0:0:0:
>> [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO
>> or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.411364] sdc: sdc1
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.412317] sd 4:0:0:0:
>> [sdc] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.501080] md: bind<sdc1>
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.662509] ata6: SATA
>> link up 3.0 Gbps (SStatus 123 SControl 300)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.674767] ata6.00:
>> ATA-8: Corsair Force 3 SSD, 1.3.3, max UDMA/133
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.674772] ata6.00:
>> 468862128 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.684647] ata6.00:
>> configured for UDMA/133
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.684933] scsi 5:0:0:0:
>> Direct-Access ATA Corsair Force 3 3 PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.685504] sd 5:0:0:0:
>> [sdd] 468862128 512-byte logical blocks: (240 GB/223 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.685975] sd 5:0:0:0:
>> [sdd] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.685983] sd 5:0:0:0:
>> [sdd] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.686186] sd 5:0:0:0:
>> [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO
>> or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.688051] sdd: sdd1
>> sdd4 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 >
>> Aug 28 10:39:28 debiantestingbase kernel: [ 3.689305] sd 5:0:0:0:
>> [sdd] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.002712] ata8: SATA
>> link down (SStatus 0 SControl 300)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003107] scsi 8:0:0:0:
>> Direct-Access ATA ST31000524AS JC4B PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003597] sd 8:0:0:0:
>> [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003843] scsi 9:0:0:0:
>> Direct-Access ATA ST31000524AS JC4B PQ: 0 ANSI: 5
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003975] sd 8:0:0:0:
>> [sde] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.003980] sd 8:0:0:0:
>> [sde] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004090] sd 8:0:0:0:
>> [sde] Write cache: enabled, read cache: enabled, doesn't support DPO
>> or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004478] sd 9:0:0:0:
>> [sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004645] sd 9:0:0:0:
>> [sdf] Write Protect is off
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004650] sd 9:0:0:0:
>> [sdf] Mode Sense: 00 3a 00 00
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004737] sd 9:0:0:0:
>> [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO
>> or FUA
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.004778] scsi
>> 15:0:0:0: Processor Marvell 91xx Config 1.01 PQ: 0 ANSI:
>> 5
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.006375] sdf: sdf1
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.006967] sd 9:0:0:0:
>> [sdf] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.008855] sde: sde1
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.009704] sd 8:0:0:0:
>> [sde] Attached SCSI disk
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018710] ata16.00:
>> exception Emask 0x1 SAct 0x0 SErr 0x0 action 0x0
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018753] ata16.00:
>> irq_stat 0x40000001
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018783] ata16.00: cmd
>> a0/01:00:00:00:01/00:00:00:00:00/a0 tag 1 dma 16640 in
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018783]
>> Inquiry 12 01 00 00 ff 00res 50/00:00:af:6d:70/00:00:74:00:00/e0 Emask
>> 0x1 (device error)
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.018868] ata16.00:
>> status: { DRDY }
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.125325] random:
>> nonblocking pool is initialized
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.125530] md: bind<sde1>
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.142140] md: bind<sdf1>
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.144984] md: raid10
>> personality registered for level 10
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.145397]
>> md/raid10:md0: active with 4 out of 4 devices
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.145440] md0: detected
>> capacity change from 0 to 2000403038208
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.208978] md0:
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.479305]
>> device-mapper: uevent: version 1.0.3
>> Aug 28 10:39:28 debiantestingbase kernel: [ 4.479536]
>> device-mapper: ioctl: 4.30.0-ioctl (2014-12-22) initialised:
>> dm-devel@redhat.com
snip
>>
>
> Lots of useful information, but no-one tried to drill deeper in /dev/dm0.
>
> Everything points to there being another missing layer between the raid
> device (md0) and your ext4 filesystem, that previously available as
> /dev/dm-0. We need to decipher the initial contents of /dev/md0 to find
> a lost signature.
>
> Please supply the updated mdadm -E reports as I asked. Trim all of the
> old mails from your reply, too. Makes for very difficult reading.
Sorry - - - you did ask for all of what had gone on.
>
> When we get your array assembled, we'll use hexdump to poke around.
Over to you
Dee
^ permalink raw reply
* Re: Advice requested
From: Phil Turmel @ 2015-11-03 0:11 UTC (permalink / raw)
To: o1bigtenor; +Cc: Linux-RAID
In-Reply-To: <CAPpdf58En1ARr9YiFSgvPN5Ke_t7=EfxiaN9cZtGVtvR4NUdmA@mail.gmail.com>
Hi Dee,
On 11/02/2015 06:49 PM, o1bigtenor wrote:
> mdadm -E /dev/sdb1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
> Name : debianbase:0 (local to host debianbase)
> Creation Time : Mon Mar 5 08:26:28 2012
> Raid Level : raid10
> Raid Devices : 4
>
> Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
> Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
> Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=1024 sectors
> State : clean
> Device UUID : a80c76db:eaea61af:bcb9cbbb:ac99e467
>
> Update Time : Fri Aug 28 10:38:32 2015
> Checksum : 29a4fa98 - correct
> Events : 47341
>
> Layout : near=2
> Chunk Size : 512K
>
> Device Role : Active device 3
> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
Useful disk.
> root@debianbase:/# mdadm -E /dev/sdc1
> mdadm: No md superblock detected on /dev/sdc1.
> root@debianbase:/# mdadm -E /dev/sdd1
> mdadm: No md superblock detected on /dev/sdd1.
Not useful disks.
> root@debianbase:/# mdadm -E /dev/sdf1
> /dev/sdf1:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
> Name : debianbase:0 (local to host debianbase)
> Creation Time : Mon Mar 5 08:26:28 2012
> Raid Level : raid10
> Raid Devices : 4
>
> Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
> Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
> Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
> Data Offset : 2048 sectors
> Super Offset : 8 sectors
> Unused Space : before=1968 sectors, after=1024 sectors
> State : clean
> Device UUID : 9e749fa9:a0efe791:ea09d2e2:72b99f6c
>
> Update Time : Fri Aug 28 10:38:32 2015
> Checksum : 615d736 - correct
> Events : 47341
>
> Layout : near=2
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
Yay! Useful disk. And with a device role that should make a functional
degraded array with /dev/sdb1
>> You may have to conquer this to succeed. While man pages certainly
>> aren't perfect, they are the most complete documentation available on
>> any linux system. Most people in your situation struggle with the
>> 'synopsis' section, and how to decipher the brackets and braces and
>> such. Google 'reading man pages' and sample the results for a wide
>> overview. Save the deep dive on 'Backus-Naur' for last. :-)
>
> Sounds like you like the present layout.
Not really. That doesn't change my advice.
[trim /]
> With two drives removed from access then sdb1 and sdd1 were the useful drives.
> Now with all drives plugged in we are still looking at sdb1 and sdd1 as being
> the most likely accessible drives.
/dev/sdb1 and /dev/sdf1 at the moment.
> Yes something changed from this morning to now. The way it is now makes more
> sense but I didn't change anything.
Rebooting with or without a USB device or any other missing disk is all
it takes.
>> Please supply the updated mdadm -E reports as I asked. Trim all of the
>> old mails from your reply, too. Makes for very difficult reading.
>
> Sorry - - - you did ask for all of what had gone on.
Oh, no. I wasn't complaining that you supplied them. Just reminding
you to trim them. I don't want to annoy the list managers any more than
necessary. :-)
>> When we get your array assembled, we'll use hexdump to poke around.
Ok. Make sure no array is active in /proc/mdstat. For any there, do:
mdadm --stop /dev/mdX
Force assemble the array in degraded mode, verbosely:
mdadm -Avf /dev/md0 /dev/sdb1 /dev/sdf1
If the above fails, cut and paste the entire report in your reply.
If it succeeds, paste the output of the following in your reply:
dd if=/dev/md0 bs=1M count=16 |hexdump -C |head -n 1000
That'll be the first 1000 lines of a hex report of the beginning of your
array.
Phil
^ permalink raw reply
* [PATCH] mdadm: check bitmap first when reshape raid1 to raid0
From: Xiao Ni @ 2015-11-03 0:25 UTC (permalink / raw)
To: linux-raid
One raid1 with bitmap is composed by 4 disks. It'll fail when rashape
to raid0 and lose 3 disks. It should check bitmap first when reshape
raid1 to raid0.
And need to free subarray, close cfd in Grow_reshape.
It can't remove bitmap in Grow_reshape, because it's already frozen. I
think it should not unfreeze in the process of the function. So I just
test the bitmap.
Signed-off-by: Xiao Ni <xni@redhat.com>
---
Grow.c | 45 ++++++++++++++++++++++++++++-----------------
mdadm.c | 16 ++++++++++++----
2 files changed, 40 insertions(+), 21 deletions(-)
diff --git a/Grow.c b/Grow.c
index 80d7b22..c4ea2a5 100644
--- a/Grow.c
+++ b/Grow.c
@@ -1600,15 +1600,14 @@ int Grow_reshape(char *devname, int fd,
cfd = open_dev_excl(st->container_devnm);
} else {
container = st->devnm;
- close(fd);
cfd = open_dev_excl(st->devnm);
fd = cfd;
}
if (cfd < 0) {
pr_err("Unable to open container for %s\n",
devname);
- free(subarray);
- return 1;
+ rv = 1;
+ goto release;
}
rv = st->ss->load_container(st, cfd, NULL);
@@ -1616,8 +1615,8 @@ int Grow_reshape(char *devname, int fd,
if (rv) {
pr_err("Cannot read superblock for %s\n",
devname);
- free(subarray);
- return 1;
+ rv = 1;
+ goto release;
}
/* check if operation is supported for metadata handler */
@@ -1641,8 +1640,8 @@ int Grow_reshape(char *devname, int fd,
pr_err("cannot reshape arrays in container with unsupported metadata: %s(%s)\n",
devname, container);
sysfs_free(cc);
- free(subarray);
- return 1;
+ rv = 1;
+ goto release;
}
}
sysfs_free(cc);
@@ -1662,7 +1661,8 @@ int Grow_reshape(char *devname, int fd,
s->raiddisks - array.raid_disks,
s->raiddisks - array.raid_disks == 1 ? "" : "s",
array.spare_disks + added_disks);
- return 1;
+ rv = 1;
+ goto release;
}
sra = sysfs_read(fd, NULL, GET_LEVEL | GET_DISKS | GET_DEVS
@@ -1675,17 +1675,18 @@ int Grow_reshape(char *devname, int fd,
} else {
pr_err("failed to read sysfs parameters for %s\n",
devname);
- return 1;
+ rv = 1;
+ goto release;
}
frozen = freeze(st);
if (frozen < -1) {
/* freeze() already spewed the reason */
- sysfs_free(sra);
- return 1;
+ rv = 1;
+ goto release;
} else if (frozen < 0) {
pr_err("%s is performing resync/recovery and cannot be reshaped\n", devname);
- sysfs_free(sra);
- return 1;
+ rv = 1;
+ goto release;
}
/* ========= set size =============== */
@@ -1898,11 +1899,16 @@ size_change_error:
array.layout == ((1 << 8) + 2) && !(array.raid_disks & 1)) ||
(s->level == 0 && array.level == 1 && sra)) {
int err;
+
+ if (array.state & (1<<MD_SB_BITMAP_PRESENT)) {
+ pr_err("Bitmap must be removed before level can be changed\n");
+ rv = 1;
+ goto release;
+ }
+
err = remove_disks_for_takeover(st, sra, array.layout);
if (err) {
- dprintf("Array cannot be reshaped\n");
- if (cfd > -1)
- close(cfd);
+ pr_err("Array cannot be reshaped\n");
rv = 1;
goto release;
}
@@ -2078,9 +2084,14 @@ size_change_error:
frozen = 0;
}
release:
- sysfs_free(sra);
if (frozen > 0)
unfreeze(st);
+ if (sra)
+ sysfs_free(sra);
+ if (cfd > -1)
+ close(cfd);
+ if (subarray)
+ free(subarray);
return rv;
}
diff --git a/mdadm.c b/mdadm.c
index f56a8cf..95db2a0 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -1299,7 +1299,8 @@ int main(int argc, char *argv[])
pr_err("'1' is an unusual number of drives for an array, so it is probably\n"
" a mistake. If you really mean it you will need to specify --force before\n"
" setting the number of drives.\n");
- exit(2);
+ rv = 2;
+ goto release;
}
}
@@ -1326,13 +1327,15 @@ int main(int argc, char *argv[])
rv = get_cluster_name(&c.homecluster);
if (rv) {
pr_err("The md can't get cluster name\n");
- exit(1);
+ rv = 1;
+ goto release;
}
}
if (c.backup_file && data_offset != INVALID_SECTORS) {
pr_err("--backup-file and --data-offset are incompatible\n");
- exit(2);
+ rv = 2;
+ goto release;
}
if ((mode == MISC && devmode == 'E')
@@ -1340,7 +1343,8 @@ int main(int argc, char *argv[])
/* Anyone may try this */;
else if (geteuid() != 0) {
pr_err("must be super-user to perform this action\n");
- exit(1);
+ rv = 1;
+ goto release;
}
ident.autof = c.autof;
@@ -1640,6 +1644,10 @@ int main(int argc, char *argv[])
autodetect();
break;
}
+
+release:
+ if (mdfd > -1)
+ close(mdfd);
exit(rv);
}
--
1.8.3.1
^ permalink raw reply related
* Re: Advice requested
From: o1bigtenor @ 2015-11-03 0:44 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux-RAID
In-Reply-To: <5637FBAC.8000900@turmel.org>
On Mon, Nov 2, 2015 at 6:11 PM, Phil Turmel <philip@turmel.org> wrote:
> Hi Dee,
>
> On 11/02/2015 06:49 PM, o1bigtenor wrote:
>
>> mdadm -E /dev/sdb1
>> /dev/sdb1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>> Name : debianbase:0 (local to host debianbase)
>> Creation Time : Mon Mar 5 08:26:28 2012
>> Raid Level : raid10
>> Raid Devices : 4
>>
>> Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
>> Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>> Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=1024 sectors
>> State : clean
>> Device UUID : a80c76db:eaea61af:bcb9cbbb:ac99e467
>>
>> Update Time : Fri Aug 28 10:38:32 2015
>> Checksum : 29a4fa98 - correct
>> Events : 47341
>>
>> Layout : near=2
>> Chunk Size : 512K
>>
>> Device Role : Active device 3
>> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>
> Useful disk.
>
>> root@debianbase:/# mdadm -E /dev/sdc1
>> mdadm: No md superblock detected on /dev/sdc1.
>> root@debianbase:/# mdadm -E /dev/sdd1
>> mdadm: No md superblock detected on /dev/sdd1.
>
> Not useful disks.
>
>> root@debianbase:/# mdadm -E /dev/sdf1
>> /dev/sdf1:
>> Magic : a92b4efc
>> Version : 1.2
>> Feature Map : 0x0
>> Array UUID : 79baaa2f:0aa2b9fa:18e2ea6b:6e2846b3
>> Name : debianbase:0 (local to host debianbase)
>> Creation Time : Mon Mar 5 08:26:28 2012
>> Raid Level : raid10
>> Raid Devices : 4
>>
>> Avail Dev Size : 1953519616 (931.51 GiB 1000.20 GB)
>> Array Size : 1953518592 (1863.02 GiB 2000.40 GB)
>> Used Dev Size : 1953518592 (931.51 GiB 1000.20 GB)
>> Data Offset : 2048 sectors
>> Super Offset : 8 sectors
>> Unused Space : before=1968 sectors, after=1024 sectors
>> State : clean
>> Device UUID : 9e749fa9:a0efe791:ea09d2e2:72b99f6c
>>
>> Update Time : Fri Aug 28 10:38:32 2015
>> Checksum : 615d736 - correct
>> Events : 47341
>>
>> Layout : near=2
>> Chunk Size : 512K
>>
>> Device Role : Active device 1
>> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>
> Yay! Useful disk. And with a device role that should make a functional
> degraded array with /dev/sdb1
>
>>> You may have to conquer this to succeed. While man pages certainly
>>> aren't perfect, they are the most complete documentation available on
>>> any linux system. Most people in your situation struggle with the
>>> 'synopsis' section, and how to decipher the brackets and braces and
>>> such. Google 'reading man pages' and sample the results for a wide
>>> overview. Save the deep dive on 'Backus-Naur' for last. :-)
>>
>> Sounds like you like the present layout.
>
> Not really. That doesn't change my advice.
>
> [trim /]
>
>> With two drives removed from access then sdb1 and sdd1 were the useful drives.
>> Now with all drives plugged in we are still looking at sdb1 and sdd1 as being
>> the most likely accessible drives.
>
> /dev/sdb1 and /dev/sdf1 at the moment.
>
>> Yes something changed from this morning to now. The way it is now makes more
>> sense but I didn't change anything.
>
> Rebooting with or without a USB device or any other missing disk is all
> it takes.
No reboots and machine has been on the whole time.
>
>
>>> When we get your array assembled, we'll use hexdump to poke around.
>
> Ok. Make sure no array is active in /proc/mdstat. For any there, do:
>
> mdadm --stop /dev/mdX
>
> Force assemble the array in degraded mode, verbosely:
>
> mdadm -Avf /dev/md0 /dev/sdb1 /dev/sdf1
>
> If the above fails, cut and paste the entire report in your reply.
>
> If it succeeds, paste the output of the following in your reply:
>
> dd if=/dev/md0 bs=1M count=16 |hexdump -C |head -n 1000
>
> That'll be the first 1000 lines of a hex report of the beginning of your
> array.
root@debianbase:/# mdadm --stop /dev/md0
mdadm: error opening /dev/md0: No such file or directory
root@debianbase:/# mdadm -Avf /dev/md0 /dev/sdb1 /dev/sdf1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md0
mdadm: no uptodate device for slot 4 of /dev/md0
mdadm: added /dev/sdb1 to /dev/md0 as 3
mdadm: added /dev/sdf1 to /dev/md0 as 1
mdadm: /dev/md0 has been started with 2 drives (out of 4).
root@debianbase:/# dd if=/dev/md0 bs=1M count=16 |hexdump -c |head -n 1000
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
00001c0 001 \0 � � � � 001 \0 \0 \0 � � � � \0 \0
00001d0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
00001f0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U �
0000200 E F I P A R T \0 \0 001 \0 \ \0 \0 \0
0000210 8 F L ] \0 \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0
0000220 � � � � \0 \0 \0 \0 " \0 \0 \0 \0 \0 \0 \0
0000230 � � � � \0 \0 \0 \0 223 w � 036 � \f 026 N
0000240 � 204 3 � 202 Z � H 002 \0 \0 \0 \0 \0 \0 \0
0000250 200 \0 \0 \0 200 \0 \0 \0 206 � T � \0 \0 \0 \0
0000260 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0100400 \0 G \a \0 024 034 035 231 232 t 001 � 200 r 030
0100410 203 � = \a \0 \0 \0 \0 002 \0 \0 \0 002 \0 \0 \0
0100420 \0 200 \0 \0 \0 200 \0 \0 \0 \0 \0 � F � U
0100430 x 200 � U Z \0 ' \0 S � 001 \0 001 \0 \0 \0
0100440 036 � m R \0 N � \0 \0 \0 \0 \0 001 \0 \0 \0
0100450 \0 \0 \0 \0 \v \0 \0 \0 \0 001 \0 \0 < \0 \0 \0
0100460 B 002 \0 \0 { \0 \0 \0 I U 6 � o I V
0100470 � � 5 A � � 177 \n \0 \0 \0 \0 \0 \0 \0 \0
0100480 \0 \0 \0 \0 \0 \0 \0 \0 / m e d i a / d
0100490 a r a l d / 4 9 5 5 2 0 3 6 - b
01004a0 4 6 f - 4 9 5 6 - a d e 9 - 3 5
01004b0 4 1 a 3 d d 7 f 0 a \0 \0 \0 \0 \0 \0
01004c0 \0 \0 � ` 017 � 017 210 \0 \0 \0 \0 \0 \0 213 003
01004d0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
01004e0 \b \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � * � �
01004f0 � o M \t 217 � + D � 225 � ^ 001 001 \0 \0
0100500 \0 \0 \0 \0 \0 \0 \0 \0 � % e O \n � 002 \0
0100510 004 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 177 \0 \0
0100520 \0 200 210 016 � 177 \0 \0 001 \0 \0 \0 � � 210 016
0100530 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0100540 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \b
0100550 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 034 \0 034 \0
0100560 001 \0 \0 \0 200 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0100570 \0 001 \0 \0 004 \0 \0 \0 � 226 � 022 \0 \0 \0 \0
0100580 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0101000 001 004 \0 \0 021 004 \0 \0 ! 004 \0 \0 � [ � 037
0101010 002 \0 004 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 037 � T
0101020 002 004 \0 \0 022 004 \0 \0 ! 006 \0 \0 E 001 \0
0101030 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � <
0101040 003 004 \0 \0 023 004 \0 \0 ! \b \0 \0 � 006 \0
0101050 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 020 �
0101060 004 004 \0 \0 024 004 \0 \0 ! \n \0 \0 � ( \0
0101070 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 004
0101080 005 004 \0 \0 025 004 \0 \0 ! \f \0 \0 \0 \0 \0
0101090 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \t
01010a0 006 004 \0 \0 026 004 \0 \0 ! 016 \0 \0 \v 002 \0
01010b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 H �
01010c0 \a 004 \0 \0 027 004 \0 \0 ! 020 \0 \0 \0 \0 \0
01010d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 237 �
01010e0 \b 004 \0 \0 030 004 \0 \0 ! 022 \0 \0 203 003 \0
01010f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � r
0101100 \t 004 \0 \0 031 004 \0 \0 ! 024 \0 \0 \0 \0 \0
0101110 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 017 \f
0101120 \n 004 \0 \0 032 004 \0 \0 ! 026 \0 \0 � 001 \0
0101130 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 t \a
0101140 \v 004 \0 \0 033 004 \0 \0 ! 030 \0 \0 \0 \0 \0
0101150 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 3 E
0101160 \f 004 \0 \0 034 004 \0 \0 ! 032 \0 \0 � 001 \0
0101170 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 3 024
0101180 \r 004 \0 \0 035 004 \0 \0 ! 034 \0 \0 � 003 \0
0101190 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 V 210
01011a0 016 004 \0 \0 036 004 \0 \0 ! 036 \0 \0 ) 006 \0
01011b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 t E
01011c0 017 004 \0 \0 037 004 \0 \0 ! \0 \0 ' 002 \0
01011d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 w j
01011e0 020 004 \0 \0 004 \0 \0 ! " \0 \0 003 001 \0
01011f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 b �
0101200 \0 \0 \b \0 020 \0 \b \0 \0 \b \0 � Y \0
0101210 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 & 006
0101220 001 \0 \b \0 021 \0 \b \0 002 \b \0 \ 001 \0
0101230 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101240 002 \0 \b \0 022 \0 \b \0 004 \b \0 \t 001 \0
0101250 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101260 003 \0 \b \0 023 \0 \b \0 006 \b \0 201 \0 \0
0101270 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \t
0101280 004 \0 \b \0 024 \0 \b \0 \b \b \0 � 002 \0
0101290 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 231
01012a0 005 \0 \b \0 025 \0 \b \0 \n \b \0 y 004 \0
01012b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 233
01012c0 006 \0 \b \0 026 \0 \b \0 \f \b \0 5 004 \0
01012d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 v �
01012e0 \a \0 \b \0 027 \0 \b \0 016 \b \0 L 001 \0
01012f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 w \t
0101300 \b \0 \b \0 030 \0 \b \0 020 \b \0 \0 \0 \0
0101310 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 �
0101320 \t \0 \b \0 031 \0 \b \0 022 \b \0 c 001 \0
0101330 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 D �
0101340 \n \0 \b \0 032 \0 \b \0 024 \b \0 \0 \0 \0
0101350 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 225 �
0101360 \v \0 \b \0 033 \0 \b \0 026 \b \0 � 001 \0
0101370 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101380 \f \0 \b \0 034 \0 \b \0 030 \b \0 ] 005 \0
0101390 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � V
01013a0 \r \0 \b \0 035 \0 \b \0 032 \b \0 � 003 \0
01013b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 205 201
01013c0 016 \0 \b \0 036 \0 \b \0 034 \b \0 H 003 \0
01013d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 230 �
01013e0 017 \0 \b \0 037 \0 \b \0 036 \b \0 S 004 \0
01013f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 207 214
0101400 \0 \0 020 \0 020 \0 020 \0 \0 020 \0 � _ \0
0101410 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � =
0101420 001 \0 020 \0 021 \0 020 \0 002 020 \0 203 \0 \0
0101430 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � %
0101440 002 \0 020 \0 022 \0 020 \0 004 020 \0 � 002 \0
0101450 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 o �
0101460 003 \0 020 \0 023 \0 020 \0 006 020 \0 \0 \0 \0
0101470 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 210
0101480 004 \0 020 \0 024 \0 020 \0 \b 020 \0 \0 \0 \0
0101490 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � *
01014a0 005 \0 020 \0 025 \0 020 \0 \n 020 \0 ` \0 \0
01014b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 a 224
01014c0 006 \0 020 \0 026 \0 020 \0 \f 020 \0 \0 \0 \0
01014d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 021 +
01014e0 \a \0 020 \0 027 \0 020 \0 016 020 \0 � \0 \0
01014f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � K
0101500 \b \0 020 \0 030 \0 020 \0 020 020 \0 \r 003 \0
0101510 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 x p
0101520 \t \0 020 \0 031 \0 020 \0 022 020 \0 � 003 \0
0101530 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101540 \n \0 020 \0 032 \0 020 \0 024 020 \0 � 001 \0
0101550 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 | �
0101560 \v \0 020 \0 033 \0 020 \0 026 020 \0 j 001 \0
0101570 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 030 �
0101580 \f \0 020 \0 034 \0 020 \0 030 020 \0 177 001 \0
0101590 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \b p
01015a0 \r \0 020 \0 035 \0 020 \0 032 020 \0 \0 \0 \0
01015b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 214
01015c0 016 \0 020 \0 036 \0 020 \0 034 020 \0 \0 \0 \0
01015d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � -
01015e0 017 \0 020 \0 037 \0 020 \0 036 020 \0 = \0 \0
01015f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 203 -
0101600 \0 \0 030 \0 020 \0 030 \0 \0 030 \0 7 ^ \0
0101610 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > R
0101620 001 \0 030 \0 021 \0 030 \0 002 030 \0 ( \0 \0
0101630 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 212 R
0101640 002 \0 030 \0 022 \0 030 \0 004 030 \0 W 002 \0
0101650 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 l _
0101660 003 \0 030 \0 023 \0 030 \0 006 030 \0 \0 \0 \0
0101670 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101680 004 \0 030 \0 024 \0 030 \0 \b 030 \0 \0 \0 \0
0101690 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � Y
01016a0 005 \0 030 \0 025 \0 030 \0 \n 030 \0 \0 \0 \0
01016b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 i �
01016c0 006 \0 030 \0 026 \0 030 \0 \f 030 \0 020 002 \0
01016d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01016e0 \a \0 030 \0 027 \0 030 \0 016 030 \0 \0 \0 \0
01016f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101700 \b \0 030 \0 030 \0 030 \0 020 030 \0 \0 \0 \0
0101710 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \f \
0101720 \t \0 030 \0 031 \0 030 \0 022 030 \0 M 004 \0
0101730 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 022 F
0101740 \n \0 030 \0 032 \0 030 \0 024 030 \0 217 005 \0
0101750 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101760 \v \0 030 \0 033 \0 030 \0 026 030 \0 023 002 \0
0101770 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 7 034
0101780 \f \0 030 \0 034 \0 030 \0 030 030 \0 � 001 \0
0101790 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 v �
01017a0 \r \0 030 \0 035 \0 030 \0 032 030 \0 034 004 \0
01017b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 8 �
01017c0 016 \0 030 \0 036 \0 030 \0 034 030 \0 \ 005 \0
01017d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 022 M
01017e0 017 \0 030 \0 037 \0 030 \0 036 030 \0 � 001 \0
01017f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 I 207
0101800 \0 \0 \0 020 \0 \0 \0 \0 � _ \0
0101810 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 207 W
0101820 001 \0 \0 021 \0 \0 002 \0 \0 \0 \0
0101830 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 b �
0101840 002 \0 \0 022 \0 \0 004 \0 \0 \0 \0
0101850 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \f B
0101860 003 \0 \0 023 \0 \0 006 \0 � 002 \0
0101870 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101880 004 \0 \0 024 \0 \0 \b \0 � 005 \0
0101890 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 ! `
01018a0 005 \0 \0 025 \0 \0 \n \0 6 \0 \0
01018b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � f
01018c0 006 \0 \0 026 \0 \0 \f \0 226 002 \0
01018d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \n 205
01018e0 \a \0 \0 027 \0 \0 016 \0 k 003 \0
01018f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 ! )
0101900 \b \0 \0 030 \0 \0 020 \0 \0 \0 \0
0101910 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 m E
0101920 \t \0 \0 031 \0 \0 022 \0 @ 001 \0
0101930 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101940 \n \0 \0 032 \0 \0 024 \0 � 002 \0
0101950 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � m
0101960 \v \0 \0 033 \0 \0 026 \0 \0 \0 \0
0101970 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 003 �
0101980 \f \0 \0 034 \0 \0 030 \0 \0 \0 \0
0101990 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \a F
01019a0 \r \0 \0 035 \0 \0 032 \0 \0 \0 \0
01019b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01019c0 016 \0 \0 036 \0 \0 034 \0 \0 \0 \0
01019d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � G
01019e0 017 \0 \0 037 \0 \0 036 \0 \0 001 \0
01019f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 8 w
0101a00 \0 \0 ( \0 020 \0 ( \0 \0 ( \0 � _ \0
0101a10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 221 $
0101a20 001 \0 ( \0 021 \0 ( \0 002 ( \0 � 003 \0
0101a30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 1 �
0101a40 002 \0 ( \0 022 \0 ( \0 004 ( \0 \0 \0 \0
0101a50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 032 1
0101a60 003 \0 ( \0 023 \0 ( \0 006 ( \0 231 002 \0
0101a70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 236 B
0101a80 004 \0 ( \0 024 \0 ( \0 \b ( \0 \0 \0 \0
0101a90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 3
0101aa0 005 \0 ( \0 025 \0 ( \0 \n ( \0 \0 \0 \0
0101ab0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 036 223
0101ac0 006 \0 ( \0 026 \0 ( \0 \f ( \0 \0 \0 \0
0101ad0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 p 2
0101ae0 \a \0 ( \0 027 \0 ( \0 016 ( \0 A 003 \0
0101af0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 037 J
0101b00 \b \0 ( \0 030 \0 ( \0 020 ( \0 \0 \0 \0
0101b10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 { 6
0101b20 \t \0 ( \0 031 \0 ( \0 022 ( \0 � 002 \0
0101b30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 222 �
0101b40 \n \0 ( \0 032 \0 ( \0 024 ( \0 \0 \0 \0
0101b50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 7
0101b60 \v \0 ( \0 033 \0 ( \0 026 ( \0 \0 \0 \0
0101b70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 025 227
0101b80 \f \0 ( \0 034 \0 ( \0 030 ( \0 � 003 \0
0101b90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 5
0101ba0 \r \0 ( \0 035 \0 ( \0 032 ( \0 \0 \0 \0
0101bb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 225
0101bc0 016 \0 ( \0 036 \0 ( \0 034 ( \0 \0 \0 \0
0101bd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 4
0101be0 017 \0 ( \0 037 \0 ( \0 036 ( \0 d 003 \0
0101bf0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � K
0101c00 \0 \0 0 \0 020 \0 0 \0 \0 0 \0 � _ \0
0101c10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101c20 001 \0 0 \0 021 \0 0 \0 002 0 \0 \0 \0 \0
0101c30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 N 005
0101c40 002 \0 0 \0 022 \0 0 \0 004 0 \0 \0 \0 \0
0101c50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 �
0101c60 003 \0 0 \0 023 \0 0 \0 006 0 \0 \0 \0 \0
0101c70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 004
0101c80 004 \0 0 \0 024 \0 0 \0 \b 0 \0 \0 \0 \0
0101c90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101ca0 005 \0 0 \0 025 \0 0 \0 \n 0 \0 \0 \0 \0
0101cb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 $ 006
0101cc0 006 \0 0 \0 026 \0 0 \0 \f 0 \0 230 001 \0
0101cd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � x
0101ce0 \a \0 0 \0 027 \0 0 \0 016 0 \0 \0 \0 \0
0101cf0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 221 \a
0101d00 \b \0 0 \0 030 \0 0 \0 020 0 \0 \0 \0 \0
0101d10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 A �
0101d20 \t \0 0 \0 031 \0 0 \0 022 0 \0 \0 \0 \0
0101d30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 232 003
0101d40 \n \0 0 \0 032 \0 0 \0 024 0 \0 � \0 \0
0101d50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � T
0101d60 \v \0 0 \0 033 \0 0 \0 026 0 \0 \0 \0 \0
0101d70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 / 002
0101d80 \f \0 0 \0 034 \0 0 \0 030 0 \0 \0 \0 \0
0101d90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 + �
0101da0 \r \0 0 \0 035 \0 0 \0 032 0 \0 \0 \0 \0
0101db0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \0
0101dc0 016 \0 0 \0 036 \0 0 \0 034 0 \0 \0 \0 \0
0101dd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 236 �
0101de0 017 \0 0 \0 037 \0 0 \0 036 0 \0 h \t \0
0101df0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 017 /
0101e00 \0 \0 8 \0 020 \0 8 \0 \0 8 \0 � _ \0
0101e10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101e20 001 \0 8 \0 021 \0 8 \0 002 8 \0 � \0 \0
0101e30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101e40 002 \0 8 \0 022 \0 8 \0 004 8 \0 \0 \0 \0
0101e50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 6 �
0101e60 003 \0 8 \0 023 \0 8 \0 006 8 \0 � 001 \0
0101e70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 1
0101e80 004 \0 8 \0 024 \0 8 \0 \b 8 \0 F 005 \0
0101e90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 017 �
0101ea0 005 \0 8 \0 025 \0 8 \0 \n 8 \0 232 005 \0
0101eb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 / �
0101ec0 006 \0 8 \0 026 \0 8 \0 \f 8 \0 e 003 \0
0101ed0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101ee0 \a \0 8 \0 027 \0 8 \0 016 8 \0 _ 004 \0
0101ef0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101f00 \b \0 8 \0 030 \0 8 \0 020 8 \0 232 001 \0
0101f10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \b �
0101f20 \t \0 8 \0 031 \0 8 \0 022 8 \0 m 005 \0
0101f30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 023 220
0101f40 \n \0 8 \0 032 \0 8 \0 024 8 \0 \0 \0 \0
0101f50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101f60 \v \0 8 \0 033 \0 8 \0 026 8 \0 \0 \0 \0
0101f70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 9 q
0101f80 \f \0 8 \0 034 \0 8 \0 030 8 \0 Z \a \0
0101f90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � )
0101fa0 \r \0 8 \0 035 \0 8 \0 032 8 \0 022 \b \0
0101fb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0101fc0 016 \0 8 \0 036 \0 8 \0 034 8 \0 � 004 \0
0101fd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 217 �
0101fe0 017 \0 8 \0 037 \0 8 \0 036 8 \0 � \v \0
0101ff0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 003 {
0102000 \0 \0 @ \0 020 \0 @ \0 \0 @ \0 � Q \0
0102010 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 222
0102020 001 \0 @ \0 021 \0 @ \0 002 @ \0 � 001 \0
0102030 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � `
0102040 002 \0 @ \0 022 \0 @ \0 004 @ \0 I \a \0
0102050 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 226 �
0102060 003 \0 @ \0 023 \0 @ \0 006 @ \0 � 001 \0
0102070 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 005
0102080 004 \0 @ \0 024 \0 @ \0 \b @ \0 � 005 \0
0102090 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 n \f
01020a0 005 \0 @ \0 025 \0 @ \0 \n @ \0 232 027 \0
01020b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 W 7
01020c0 006 \0 @ \0 026 \0 @ \0 \f @ \0 215 003 \0
01020d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � "
01020e0 \a \0 @ \0 027 \0 @ \0 016 @ \0 \0 \0 \0
01020f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 S 5
0102100 \b \0 @ \0 030 \0 @ \0 020 @ \0 + \b \0
0102110 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 } �
0102120 \t \0 @ \0 031 \0 @ \0 022 @ \0 \n 002 \0
0102130 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 213
0102140 \n \0 @ \0 032 \0 @ \0 024 @ \0 210 \0 \0
0102150 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 235 032
0102160 \v \0 @ \0 033 \0 @ \0 026 @ \0 216 002 \0
0102170 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \a S
0102180 \f \0 @ \0 034 \0 @ \0 030 @ \0 b \0 \0
0102190 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 V 4
01021a0 \r \0 @ \0 035 \0 @ \0 032 @ \0 \0 \0 \0
01021b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 2 2
01021c0 016 \0 @ \0 036 \0 @ \0 034 @ \0 b 005 \0
01021d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � d
01021e0 017 \0 @ \0 037 \0 @ \0 036 @ \0 � \r \0
01021f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102200 \0 \0 H \0 020 \0 H \0 \0 H \0 216 Y \0
0102210 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102220 001 \0 H \0 021 \0 H \0 002 H \0 026 002 \0
0102230 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � h
0102240 002 \0 H \0 022 \0 H \0 004 H \0 6 \r \0
0102250 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 217 �
0102260 003 \0 H \0 023 \0 H \0 006 H \0 177 001 \0
0102270 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 W 031
0102280 004 \0 H \0 024 \0 H \0 \b H \0 b 017 \0
0102290 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01022a0 005 \0 H \0 025 \0 H \0 \n H \0 � 035 \0
01022b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 024
01022c0 006 \0 H \0 026 \0 H \0 \f H \0 @ 021 \0
01022d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � g
01022e0 \a \0 H \0 027 \0 H \0 016 H \0 ` 017 \0
01022f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 n �
0102300 \b \0 H \0 030 \0 H \0 020 H \0 � 002 \0
0102310 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 003 t
0102320 \t \0 H \0 031 \0 H \0 022 H \0 231 001 \0
0102330 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � a
0102340 \n \0 H \0 032 \0 H \0 024 H \0 \v \a \0
0102350 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102360 \v \0 H \0 033 \0 H \0 026 H \0 � \b \0
0102370 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102380 \f \0 H \0 034 \0 H \0 030 H \0 { \a \0
0102390 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 % -
01023a0 \r \0 H \0 035 \0 H \0 032 H \0 � \f \0
01023b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 232
01023c0 016 \0 H \0 036 \0 H \0 034 H \0 � \0 \0
01023d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 ^ �
01023e0 017 \0 H \0 037 \0 H \0 036 H \0 � 002 \0
01023f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 q 5
0102400 \0 \0 P \0 020 \0 P \0 \0 P \0 � _ \0
0102410 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 E e
0102420 001 \0 P \0 021 \0 P \0 002 P \0 � \r \0
0102430 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102440 002 \0 P \0 022 \0 P \0 004 P \0 � 004 \0
0102450 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 y �
0102460 003 \0 P \0 023 \0 P \0 006 P \0 � 004 \0
0102470 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 026 004
0102480 004 \0 P \0 024 \0 P \0 \b P \0 } \b \0
0102490 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 036 �
01024a0 005 \0 P \0 025 \0 P \0 \n P \0 � 001 \0
01024b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 222
01024c0 006 \0 P \0 026 \0 P \0 \f P \0 202 \f \0
01024d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 �
01024e0 \a \0 P \0 027 \0 P \0 016 P \0 � 004 \0
01024f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 231
0102500 \b \0 P \0 030 \0 P \0 020 P \0 \0 \0 \0
0102510 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � w
0102520 \t \0 P \0 031 \0 P \0 022 P \0 \0 \a \0
0102530 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102540 \n \0 P \0 032 \0 P \0 024 P \0 \0 \0 \0
0102550 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 032 v
0102560 \v \0 P \0 033 \0 P \0 026 P \0 \0 \0 \0
0102570 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102580 \f \0 P \0 034 \0 P \0 030 P \0 \0 \0 \0
0102590 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � t
01025a0 \r \0 P \0 035 \0 P \0 032 P \0 \0 \0 \0
01025b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 036 �
01025c0 016 \0 P \0 036 \0 P \0 034 P \0 \0 \0 \0
01025d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 p u
01025e0 017 \0 P \0 037 \0 P \0 036 P \0 \0 \0 \0
01025f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102600 \0 \0 X \0 020 \0 X \0 \0 X \0 � _ \0
0102610 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 S 026
0102620 001 \0 X \0 021 \0 X \0 002 X \0 \0 \0 \0
0102630 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102640 002 \0 X \0 022 \0 X \0 004 X \0 \0 \0 \0
0102650 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 003
0102660 003 \0 X \0 023 \0 X \0 006 X \0 \0 \0 \0
0102670 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 003 �
0102680 004 \0 X \0 024 \0 X \0 \b X \0 \0 \0 \0
0102690 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \a 001
01026a0 005 \0 X \0 025 \0 X \0 \n X \0 \0 \0 \0
01026b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01026c0 006 \0 X \0 026 \0 X \0 \f X \0 \0 \0 \0
01026d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \0
01026e0 \a \0 X \0 027 \0 X \0 016 X \0 \0 \0 \0
01026f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 i �
0102700 \b \0 X \0 030 \0 X \0 020 X \0 \0 \0 \0
0102710 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 004
0102720 \t \0 X \0 031 \0 X \0 022 X \0 \0 \0 \0
0102730 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 b �
0102740 \n \0 X \0 032 \0 X \0 024 X \0 \0 \0 \0
0102750 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \f 005
0102760 \v \0 X \0 033 \0 X \0 026 X \0 \0 \0 \0
0102770 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102780 \f \0 X \0 034 \0 X \0 030 X \0 \0 \0 \0
0102790 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \a
01027a0 \r \0 X \0 035 \0 X \0 032 X \0 \0 \0 \0
01027b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \b �
01027c0 016 \0 X \0 036 \0 X \0 034 X \0 \0 \0 \0
01027d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 f 006
01027e0 017 \0 X \0 037 \0 X \0 036 X \0 \0 \0 \0
01027f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102800 \0 \0 ` \0 020 \0 ` \0 \0 ` \0 � _ \0
0102810 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 2 017
0102820 001 \0 ` \0 021 \0 ` \0 002 ` \0 \0 \0 \0
0102830 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102840 002 \0 ` \0 022 \0 ` \0 004 ` \0 \0 \0 \0
0102850 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 032
0102860 003 \0 ` \0 023 \0 ` \0 006 ` \0 \0 \0 \0
0102870 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 b �
0102880 004 \0 ` \0 024 \0 ` \0 \b ` \0 \0 \0 \0
0102890 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 f 030
01028a0 005 \0 ` \0 025 \0 ` \0 \n ` \0 \0 \0 \0
01028b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01028c0 006 \0 ` \0 026 \0 ` \0 \f ` \0 \0 \0 \0
01028d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 031
01028e0 \a \0 ` \0 027 \0 ` \0 016 ` \0 � 006 \0
01028f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102900 \b \0 ` \0 030 \0 ` \0 020 ` \0 D 001 \0
0102910 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � h
0102920 \t \0 ` \0 031 \0 ` \0 022 ` \0 � 005 \0
0102930 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 005 034
0102940 \n \0 ` \0 032 \0 ` \0 024 ` \0 m 003 \0
0102950 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 021 ]
0102960 \v \0 ` \0 033 \0 ` \0 026 ` \0 W 002 \0
0102970 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102980 \f \0 ` \0 034 \0 ` \0 030 ` \0 = 033 \0
0102990 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U I
01029a0 \r \0 ` \0 035 \0 ` \0 032 ` \0 \n \a \0
01029b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � U
01029c0 016 \0 ` \0 036 \0 ` \0 034 ` \0 � 006 \0
01029d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 N �
01029e0 017 \0 ` \0 037 \0 ` \0 036 ` \0 O \b \0
01029f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 } �
0102a00 \0 \0 h \0 020 \0 h \0 \0 h \0 � _ \0
0102a10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 $ |
0102a20 001 \0 h \0 021 \0 h \0 002 h \0 � \t \0
0102a30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 B 207
0102a40 002 \0 h \0 022 \0 h \0 004 h \0 � \b \0
0102a50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 016 L
0102a60 003 \0 h \0 023 \0 h \0 006 h \0 � 002 \0
0102a70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 q ,
0102a80 004 \0 h \0 024 \0 h \0 \b h \0 4 \0 \0
0102a90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 > U
0102aa0 005 \0 h \0 025 \0 h \0 \n h \0 \0 \0 \0
0102ab0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102ac0 006 \0 h \0 026 \0 h \0 \f h \0 \0 \0 \0
0102ad0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � j
0102ae0 \a \0 h \0 027 \0 h \0 016 h \0 \0 \0 \0
0102af0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 036 �
0102b00 \b \0 h \0 030 \0 h \0 020 h \0 \0 \0 \0
0102b10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � n
0102b20 \t \0 h \0 031 \0 h \0 022 h \0 � 006 \0
0102b30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 S 6
0102b40 \n \0 h \0 032 \0 h \0 024 h \0 \0 \0 \0
0102b50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 { o
0102b60 \v \0 h \0 033 \0 h \0 026 h \0 \0 \0 \0
0102b70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102b80 \f \0 h \0 034 \0 h \0 030 h \0 \0 \0 \0
0102b90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � m
0102ba0 \r \0 h \0 035 \0 h \0 032 h \0 \0 \0 \0
0102bb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 177 �
0102bc0 016 \0 h \0 036 \0 h \0 034 h \0 \0 \0 \0
0102bd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 021 l
0102be0 017 \0 h \0 037 \0 h \0 036 h \0 \0 \0 \0
0102bf0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102c00 \0 \0 p \0 020 \0 p \0 \0 p \0 � _ \0
0102c10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 036 �
0102c20 001 \0 p \0 021 \0 p \0 002 p \0 1 002 \0
0102c30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 004 �
0102c40 002 \0 p \0 022 \0 p \0 004 p \0 \0 \0 \0
0102c50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 225 �
0102c60 003 \0 p \0 023 \0 p \0 006 p \0 \0 \0 \0
0102c70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 N \
0102c80 004 \0 p \0 024 \0 p \0 \b p \0 \0 \0 \0
0102c90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 J �
0102ca0 005 \0 p \0 025 \0 p \0 \n p \0 \0 \0 \0
0102cb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 221 ^
0102cc0 006 \0 p \0 026 \0 p \0 \f p \0 \0 \0 \0
0102cd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102ce0 \a \0 p \0 027 \0 p \0 016 p \0 \0 \0 \0
0102cf0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 $ _
0102d00 \b \0 p \0 030 \0 p \0 020 p \0 \0 \0 \0
0102d10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102d20 \t \0 p \0 031 \0 p \0 022 p \0 \0 \0 \0
0102d30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 / [
0102d40 \n \0 p \0 032 \0 p \0 024 p \0 \0 \0 \0
0102d50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 A �
0102d60 \v \0 p \0 033 \0 p \0 026 p \0 \0 \0 \0
0102d70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 232 Z
0102d80 \f \0 p \0 034 \0 p \0 030 p \0 \0 \0 \0
0102d90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 236 �
0102da0 \r \0 p \0 035 \0 p \0 032 p \0 \0 \0 \0
0102db0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 E X
0102dc0 016 \0 p \0 036 \0 p \0 034 p \0 \0 \0 \0
0102dd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 + �
0102de0 017 \0 p \0 037 \0 p \0 036 p \0 \0 \0 \0
0102df0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � Y
0102e00 \0 \0 x \0 020 \0 x \0 \0 x \0 � _ \0
0102e10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \b 232
0102e20 001 \0 x \0 021 \0 x \0 002 x \0 \0 \0 \0
0102e30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � .
0102e40 002 \0 x \0 022 \0 x \0 004 x \0 \0 \0 \0
0102e50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 203 217
0102e60 003 \0 x \0 023 \0 x \0 006 x \0 � 003 \0
0102e70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102e80 004 \0 x \0 024 \0 x \0 \b x \0 235 004 \0
0102e90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102ea0 005 \0 x \0 025 \0 x \0 \n x \0 037 \f \0
0102eb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 u �
0102ec0 006 \0 x \0 026 \0 x \0 \f x \0 % * \0
0102ed0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0102ee0 \a \0 x \0 027 \0 x \0 016 x \0 - 023 \0
0102ef0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 V |
0102f00 \b \0 x \0 030 \0 x \0 020 x \0 2 002 \0
0102f10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � _
0102f20 \t \0 x \0 031 \0 x \0 022 x \0 \0 \0 \0
0102f30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 9 (
0102f40 \n \0 x \0 032 \0 x \0 024 x \0 \0 \0 \0
0102f50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 W 211
0102f60 \v \0 x \0 033 \0 x \0 026 x \0 \0 \0 \0
0102f70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 214 )
0102f80 \f \0 x \0 034 \0 x \0 030 x \0 \0 \0 \0
0102f90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 210 213
0102fa0 \r \0 x \0 035 \0 x \0 032 x \0 \0 \0 \0
0102fb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 S +
0102fc0 016 \0 x \0 036 \0 x \0 034 x \0 \0 \0 \0
0102fd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 = 212
0102fe0 017 \0 x \0 037 \0 x \0 036 x \0 \0 \0 \0
0102ff0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � *
0103000 \0 \0 200 \0 020 \0 200 \0 \0 200 \0 � _ \0
0103010 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � j
0103020 001 \0 200 \0 021 \0 200 \0 002 200 \0 \0 \0 \0
0103030 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 S �
0103040 002 \0 200 \0 022 \0 200 \0 004 200 \0 \0 \0 \0
0103050 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 = 177
0103060 003 \0 200 \0 023 \0 200 \0 006 200 \0 \0 \0 \0
0103070 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103080 004 \0 200 \0 024 \0 200 \0 \b 200 \0 \0 \0 \0
0103090 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � }
01030a0 005 \0 200 \0 025 \0 200 \0 \n 200 \0 \0 \0 \0
01030b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 9 �
01030c0 006 \0 200 \0 026 \0 200 \0 \f 200 \0 \0 \0 \0
01030d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 W |
01030e0 \a \0 200 \0 027 \0 200 \0 016 200 \0 \0 \0 \0
01030f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 214 �
0103100 \b \0 200 \0 030 \0 200 \0 020 200 \0 \0 \0 \0
0103110 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \ x
0103120 \t \0 200 \0 031 \0 200 \0 022 200 \0 \0 \0 \0
0103130 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 207 �
0103140 \n \0 200 \0 032 \0 200 \0 024 200 \0 \0 \0 \0
0103150 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � y
0103160 \v \0 200 \0 033 \0 200 \0 026 200 \0 h 005 \0
0103170 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103180 \f \0 200 \0 034 \0 200 \0 030 200 \0 \r \a \0
0103190 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 026 �
01031a0 \r \0 200 \0 035 \0 200 \0 032 200 \0 006 004 \0
01031b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 O 223
01031c0 016 \0 200 \0 036 \0 200 \0 034 200 \0 � \v \0
01031d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � c
01031e0 017 \0 200 \0 037 \0 200 \0 036 200 \0 T \t \0
01031f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103200 \0 \0 210 \0 020 \0 210 \0 \0 210 \0 � ^ \0
0103210 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 9 020
0103220 001 \0 210 \0 021 \0 210 \0 002 210 \0 s \f \0
0103230 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 j 1
0103240 002 \0 210 \0 022 \0 210 \0 004 210 \0 024 \a \0
0103250 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103260 003 \0 210 \0 023 \0 210 \0 006 210 \0 206 \b \0
0103270 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 017
0103280 004 \0 210 \0 024 \0 210 \0 \b 210 \0 ] \b \0
0103290 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � r
01032a0 005 \0 210 \0 025 \0 210 \0 \n 210 \0 � 004 \0
01032b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 Z 231
01032c0 006 \0 210 \0 026 \0 210 \0 \f 210 \0 � 006 \0
01032d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 C �
01032e0 \a \0 210 \0 027 \0 210 \0 016 210 \0 031 001 \0
01032f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 034 d
0103300 \b \0 210 \0 030 \0 210 \0 020 210 \0 \0 \0 \0
0103310 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 J \v
0103320 \t \0 210 \0 031 \0 210 \0 022 210 \0 \0 \0 \0
0103330 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 221 �
0103340 \n \0 210 \0 032 \0 210 \0 024 210 \0 \0 \0 \0
0103350 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \n
0103360 \v \0 210 \0 033 \0 210 \0 026 210 \0 \0 \0 \0
0103370 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 $ �
0103380 \f \0 210 \0 034 \0 210 \0 030 210 \0 m 005 \0
0103390 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01033a0 \r \0 210 \0 035 \0 210 \0 032 210 \0 \0 \0 \0
01033b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01033c0 016 \0 210 \0 036 \0 210 \0 034 210 \0 \0 \0 \0
01033d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 225 \t
01033e0 017 \0 210 \0 037 \0 210 \0 036 210 \0 \0 \0 \0
01033f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 N �
0103400 \0 \0 220 \0 020 \0 220 \0 \0 220 \0 � _ \0
0103410 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 232 214
0103420 001 \0 220 \0 021 \0 220 \0 002 220 \0 \0 \0 \0
0103430 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 177 8
0103440 002 \0 220 \0 022 \0 220 \0 004 220 \0 \0 \0 \0
0103450 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 021 231
0103460 003 \0 220 \0 023 \0 220 \0 006 220 \0 � 005 \0
0103470 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � {
0103480 004 \0 220 \0 024 \0 220 \0 \b 220 \0 \0 \0 \0
0103490 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 233
01034a0 005 \0 220 \0 025 \0 220 \0 \n 220 \0 \0 \0 \0
01034b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 025 ;
01034c0 006 \0 220 \0 026 \0 220 \0 \f 220 \0 \0 \0 \0
01034d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 { 232
01034e0 \a \0 220 \0 027 \0 220 \0 016 220 \0 \0 \0 \0
01034f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � :
0103500 \b \0 220 \0 030 \0 220 \0 020 220 \0 \0 \0 \0
0103510 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 p 236
0103520 \t \0 220 \0 031 \0 220 \0 022 220 \0 \0 \0 \0
0103530 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � >
0103540 \n \0 220 \0 032 \0 220 \0 024 220 \0 \0 \0 \0
0103550 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 237
0103560 \v \0 220 \0 033 \0 220 \0 026 220 \0 \0 003 \0
0103570 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103580 \f \0 220 \0 034 \0 220 \0 030 220 \0 \0 \0 \0
0103590 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 032 235
01035a0 \r \0 220 \0 035 \0 220 \0 032 220 \0 \0 \0 \0
01035b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � =
01035c0 016 \0 220 \0 036 \0 220 \0 034 220 \0 \0 \0 \0
01035d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 234
01035e0 017 \0 220 \0 037 \0 220 \0 036 220 \0 \0 \0 \0
01035f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 t <
0103600 \0 \0 230 \0 020 \0 230 \0 \0 230 \0 � _ \0
0103610 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 214 �
0103620 001 \0 230 \0 021 \0 230 \0 002 230 \0 \0 \0 \0
0103630 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 i K
0103640 002 \0 230 \0 022 \0 230 \0 004 230 \0 \0 \0 \0
0103650 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \a �
0103660 003 \0 230 \0 023 \0 230 \0 006 230 \0 1 002 \0
0103670 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 # �
0103680 004 \0 230 \0 024 \0 230 \0 \b 230 \0 \0 \0 \0
0103690 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01036a0 005 \0 230 \0 025 \0 230 \0 \n 230 \0 \0 \0 \0
01036b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 003 H
01036c0 006 \0 230 \0 026 \0 230 \0 \f 230 \0 � 004 \0
01036d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 003
01036e0 \a \0 230 \0 027 \0 230 \0 016 230 \0 \0 \0 \0
01036f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � I
0103700 \b \0 230 \0 030 \0 230 \0 020 230 \0 \0 002 \0
0103710 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 215
0103720 \t \0 230 \0 031 \0 230 \0 022 230 \0 � \0 \0
0103730 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 201 q
0103740 \n \0 230 \0 032 \0 230 \0 024 230 \0 \0 \0 \0
0103750 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103760 \v \0 230 \0 033 \0 230 \0 026 230 \0 \0 \0 \0
0103770 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \b L
0103780 \f \0 230 \0 034 \0 230 \0 030 230 \0 \0 \0 \0
0103790 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \f �
01037a0 \r \0 230 \0 035 \0 230 \0 032 230 \0 \0 \0 \0
01037b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � N
01037c0 016 \0 230 \0 036 \0 230 \0 034 230 \0 \0 \0 \0
01037d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01037e0 017 \0 230 \0 037 \0 230 \0 036 230 \0 \0 004 \0
01037f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 216
0103800 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0103810 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103820 001 \0 � \0 021 \0 � \0 002 � \0 r 006 \0
0103830 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 P 220
0103840 002 \0 � \0 022 \0 � \0 004 � \0 \0 \0 \0
0103850 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 f �
0103860 003 \0 � \0 023 \0 � \0 006 � \0 \0 \0 \0
0103870 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � S
0103880 004 \0 � \0 024 \0 � \0 \b � \0 � 002 \0
0103890 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 233 \v
01038a0 005 \0 � \0 025 \0 � \0 \n � \0 \0 \0 \0
01038b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 b Q
01038c0 006 \0 � \0 026 \0 � \0 \f � \0 0 002 \0
01038d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 237
01038e0 \a \0 � \0 027 \0 � \0 016 � \0 225 003 \0
01038f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 032 @
0103900 \b \0 � \0 030 \0 � \0 020 � \0 \0 \0 \0
0103910 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \a �
0103920 \t \0 � \0 031 \0 � \0 022 � \0 l 002 \0
0103930 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � y
0103940 \n \0 � \0 032 \0 � \0 024 � \0 � 003 \0
0103950 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 206
0103960 \v \0 � \0 033 \0 � \0 026 � \0 � \a \0
0103970 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 177
0103980 \f \0 � \0 034 \0 � \0 030 � \0 Q 002 \0
0103990 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 214 z
01039a0 \r \0 � \0 035 \0 � \0 032 � \0 > 017 \0
01039b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � @
01039c0 016 \0 � \0 036 \0 � \0 034 � \0 031 \t \0
01039d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01039e0 017 \0 � \0 037 \0 � \0 036 � \0 � 002 \0
01039f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � $
0103a00 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0103a10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 225
0103a20 001 \0 � \0 021 \0 � \0 002 � \0 ; \t \0
0103a30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � Z
0103a40 002 \0 � \0 022 \0 � \0 004 � \0 211 \v \0
0103a50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103a60 003 \0 � \0 023 \0 � \0 006 � \0 021 017 \0
0103a70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103a80 004 \0 � \0 024 \0 � \0 \b � \0 f 021 \0
0103a90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \r @
0103aa0 005 \0 � \0 025 \0 � \0 \n � \0 � 006 \0
0103ab0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 230
0103ac0 006 \0 � \0 026 \0 � \0 \f � \0 � 017 \0
0103ad0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 I z
0103ae0 \a \0 � \0 027 \0 � \0 016 � \0 W \0 \0
0103af0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 a G
0103b00 \b \0 � \0 030 \0 � \0 020 � \0 200 017 \0
0103b10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \r 234
0103b20 \t \0 � \0 031 \0 � \0 022 � \0 � \t \0
0103b30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 o �
0103b40 \n \0 � \0 032 \0 � \0 024 � \0 7 \v \0
0103b50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 m �
0103b60 \v \0 � \0 033 \0 � \0 026 � \0 \0 \0 \0
0103b70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 177 &
0103b80 \f \0 � \0 034 \0 � \0 030 � \0 \0 \0 \0
0103b90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 { 204
0103ba0 \r \0 � \0 035 \0 � \0 032 � \0 2 003 \0
0103bb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � c
0103bc0 016 \0 � \0 036 \0 � \0 034 � \0 � 003 \0
0103bd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 4
0103be0 017 \0 � \0 037 \0 � \0 036 � \0 S 004 \0
0103bf0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103c00 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0103c10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \0
0103c20 001 \0 � \0 021 \0 � \0 002 � \0 n 006 \0
0103c30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103c40 002 \0 � \0 022 \0 � \0 004 � \0 $ \a \0
0103c50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103c60 003 \0 � \0 023 \0 � \0 006 � \0 236 006 \0
0103c70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 = �
0103c80 004 \0 � \0 024 \0 � \0 \b � \0 � 002 \0
0103c90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 030 206
0103ca0 005 \0 � \0 025 \0 � \0 \n � \0 021 006 \0
0103cb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � /
0103cc0 006 \0 � \0 026 \0 � \0 \f � \0 \0 \0 \0
0103cd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 026
0103ce0 \a \0 � \0 027 \0 � \0 016 � \0 \0 \0 \0
0103cf0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103d00 \b \0 � \0 030 \0 � \0 020 � \0 \0 \0 \0
0103d10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 + 022
0103d20 \t \0 � \0 031 \0 � \0 022 � \0 \0 \0 \0
0103d30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103d40 \n \0 � \0 032 \0 � \0 024 � \0 \0 \0 \0
0103d50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 236 023
0103d60 \v \0 � \0 033 \0 � \0 026 � \0 � \a \0
0103d70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 R �
0103d80 \f \0 � \0 034 \0 � \0 030 � \0 \a \a \0
0103d90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 B U
0103da0 \r \0 � \0 035 \0 � \0 032 � \0 � 003 \0
0103db0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103dc0 016 \0 � \0 036 \0 � \0 034 � \0 � 005 \0
0103dd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � t
0103de0 017 \0 � \0 037 \0 � \0 036 � \0 � 006 \0
0103df0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 O 213
0103e00 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0103e10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � s
0103e20 001 \0 � \0 021 \0 � \0 002 � \0 i 006 \0
0103e30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 034 �
0103e40 002 \0 � \0 022 \0 � \0 004 � \0 � 002 \0
0103e50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 ; h
0103e60 003 \0 � \0 023 \0 � \0 006 � \0 021 \b \0
0103e70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U =
0103e80 004 \0 � \0 024 \0 � \0 \b � \0 016 \b \0
0103e90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 f M
0103ea0 005 \0 � \0 025 \0 � \0 \n � \0 k 006 \0
0103eb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � ]
0103ec0 006 \0 � \0 026 \0 � \0 \f � \0 Z \t \0
0103ed0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0103ee0 \a \0 � \0 027 \0 � \0 016 � \0 � 001 \0
0103ef0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 030 �
0103f00 \b \0 � \0 030 \0 � \0 020 � \0 � 003 \0
0103f10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 207 N
0103f20 \t \0 � \0 031 \0 � \0 022 � \0 B \0 \0
0103f30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 R �
0103f40 \n \0 � \0 032 \0 � \0 024 � \0 \0 \0 \0
0103f50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 210 `
0103f60 \v \0 � \0 033 \0 � \0 026 � \0 � 003 \0
0103f70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 o H
0103f80 \f \0 � \0 034 \0 � \0 030 � \0 211 005 \0
0103f90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � E
0103fa0 \r \0 � \0 035 \0 � \0 032 � \0 0 \0 \0
0103fb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 203 �
0103fc0 016 \0 � \0 036 \0 � \0 034 � \0 � \a \0
0103fd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \n @
0103fe0 017 \0 � \0 037 \0 � \0 036 � \0 H 004 \0
0103ff0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104000 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0104010 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 003 2
0104020 001 \0 � \0 021 \0 � \0 002 � \0 \0 \0 \0
0104030 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 206
0104040 002 \0 � \0 022 \0 � \0 004 � \0 \0 \0 \0
0104050 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 210 '
0104060 003 \0 � \0 023 \0 � \0 006 � \0 F 006 \0
0104070 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 E {
0104080 004 \0 � \0 024 \0 � \0 \b � \0 201 \n \0
0104090 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \t 223
01040a0 005 \0 � \0 025 \0 � \0 \n � \0 < 005 \0
01040b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 S 210
01040c0 006 \0 � \0 026 \0 � \0 \f � \0 z 003 \0
01040d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 + �
01040e0 \a \0 � \0 027 \0 � \0 016 � \0 4 \b \0
01040f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � x
0104100 \b \0 � \0 030 \0 � \0 020 � \0 4 005 \0
0104110 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � O
0104120 \t \0 � \0 031 \0 � \0 022 � \0 � 005 \0
0104130 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104140 \n \0 � \0 032 \0 � \0 024 � \0 223 001 \0
0104150 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104160 \v \0 � \0 033 \0 � \0 026 � \0 2 005 \0
0104170 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 : g
0104180 \f \0 � \0 034 \0 � \0 030 � \0 032 003 \0
0104190 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 T �
01041a0 \r \0 � \0 035 \0 � \0 032 � \0 b \a \0
01041b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U 024
01041c0 016 \0 � \0 036 \0 � \0 034 � \0 8 \t \0
01041d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 m 035
01041e0 017 \0 � \0 037 \0 � \0 036 � \0 � 003 \0
01041f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104200 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0104210 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 025 A
0104220 001 \0 � \0 021 \0 � \0 002 � \0 235 003 \0
0104230 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 207
0104240 002 \0 � \0 022 \0 � \0 004 � \0 � \a \0
0104250 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104260 003 \0 � \0 023 \0 � \0 006 � \0 M \a \0
0104270 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 p �
0104280 004 \0 � \0 024 \0 � \0 \b � \0 � 005 \0
0104290 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 026 \v
01042a0 005 \0 � \0 025 \0 � \0 \n � \0 7 \a \0
01042b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 226 �
01042c0 006 \0 � \0 026 \0 � \0 \f � \0 � \b \0
01042d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 K
01042e0 \a \0 � \0 027 \0 � \0 016 � \0 � 005 \0
01042f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 226 �
0104300 \b \0 � \0 030 \0 � \0 020 � \0 � \t \0
0104310 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 025 =
0104320 \t \0 � \0 031 \0 � \0 022 � \0 \0 \0 \0
0104330 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 $ �
0104340 \n \0 � \0 032 \0 � \0 024 � \0 � 003 \0
0104350 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 206
0104360 \v \0 � \0 033 \0 � \0 026 � \0 � 017 \0
0104370 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104380 \f \0 � \0 034 \0 � \0 030 � \0 ; \b \0
0104390 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 o �
01043a0 \r \0 � \0 035 \0 � \0 032 � \0 \ \0 \0
01043b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 234 �
01043c0 016 \0 � \0 036 \0 � \0 034 � \0 \0 \0 \0
01043d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 Q
01043e0 017 \0 � \0 037 \0 � \0 036 � \0 @ \b \0
01043f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 i �
0104400 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0104410 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 / �
0104420 001 \0 � \0 021 \0 � \0 002 � \0 \0 \0 \0
0104430 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � `
0104440 002 \0 � \0 022 \0 � \0 004 � \0 \0 \0 \0
0104450 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104460 003 \0 � \0 023 \0 � \0 006 � \0 \0 \0 \0
0104470 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 177 a
0104480 004 \0 � \0 024 \0 � \0 \b � \0 \0 \0 \0
0104490 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 { �
01044a0 005 \0 � \0 025 \0 � \0 \n � \0 \0 \0 \0
01044b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � c
01044c0 006 \0 � \0 026 \0 � \0 \f � \0 G 003 \0
01044d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 232 223
01044e0 \a \0 � \0 027 \0 � \0 016 � \0 \0 \0 \0
01044f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 025 b
0104500 \b \0 � \0 030 \0 � \0 020 � \0 \0 \0 \0
0104510 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104520 \t \0 � \0 031 \0 � \0 022 � \0 5 004 \0
0104530 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 C e
0104540 \n \0 � \0 032 \0 � \0 024 � \0 / 003 \0
0104550 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104560 \v \0 � \0 033 \0 � \0 026 � \0 \0 \0 \0
0104570 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � g
0104580 \f \0 � \0 034 \0 � \0 030 � \0 231 \a \0
0104590 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � G
01045a0 \r \0 � \0 035 \0 � \0 032 � \0 Z 002 \0
01045b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01045c0 016 \0 � \0 036 \0 � \0 034 � \0 \0 \0 \0
01045d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 032 �
01045e0 017 \0 � \0 037 \0 � \0 036 � \0 x 001 \0
01045f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \b M
0104600 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0104610 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 9 �
0104620 001 \0 � \0 021 \0 � \0 002 � \0 1 005 \0
0104630 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 221 �
0104640 002 \0 � \0 022 \0 � \0 004 � \0 205 004 \0
0104650 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � V
0104660 003 \0 � \0 023 \0 � \0 006 � \0 6 001 \0
0104670 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 004
0104680 004 \0 � \0 024 \0 � \0 \b � \0 226 \0 \0
0104690 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 024
01046a0 005 \0 � \0 025 \0 � \0 \n � \0 \n 001 \0
01046b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � Z
01046c0 006 \0 � \0 026 \0 � \0 \f � \0 Z 003 \0
01046d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 032 212
01046e0 \a \0 � \0 027 \0 � \0 016 � \0 223 001 \0
01046f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 217 �
0104700 \b \0 � \0 030 \0 � \0 020 � \0 � 002 \0
0104710 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 025 003
0104720 \t \0 � \0 031 \0 � \0 022 � \0 ' 001 \0
0104730 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � :
0104740 \n \0 � \0 032 \0 � \0 024 � \0 \0 \0 \0
0104750 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 f �
0104760 \v \0 � \0 033 \0 � \0 026 � \0 \0 \0 \0
0104770 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 024
0104780 \f \0 � \0 034 \0 � \0 030 � \0 \0 \0 \0
0104790 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01047a0 \r \0 � \0 035 \0 � \0 032 � \0 \0 \0 \0
01047b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 b 026
01047c0 016 \0 � \0 036 \0 � \0 034 � \0 \0 \0 \0
01047d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \f �
01047e0 017 \0 � \0 037 \0 � \0 036 � \0 \0 \0 \0
01047f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 027
0104800 \0 \0 � \0 020 \0 � \0 \0 � \0 � X \0
0104810 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � b
0104820 001 \0 � \0 021 \0 � \0 002 � \0 \0 \0 \0
0104830 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \n
0104840 002 \0 � \0 022 \0 � \0 004 � \0 \0 \0 \0
0104850 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104860 003 \0 � \0 023 \0 � \0 006 � \0 \0 \0 \0
0104870 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \b \v
0104880 004 \0 � \0 024 \0 � \0 \b � \0 \0 \0 \0
0104890 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \f �
01048a0 005 \0 � \0 025 \0 � \0 \n � \0 \0 \0 \0
01048b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \t
01048c0 006 \0 � \0 026 \0 � \0 \f � \0 \0 \0 \0
01048d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01048e0 \a \0 � \0 027 \0 � \0 016 � \0 \0 \0 \0
01048f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 b \b
0104900 \b \0 � \0 030 \0 � \0 020 � \0 \0 \0 \0
0104910 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104920 \t \0 � \0 031 \0 � \0 022 � \0 \0 \0 \0
0104930 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 i \f
0104940 \n \0 � \0 032 \0 � \0 024 � \0 \0 \0 \0
0104950 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \a �
0104960 \v \0 � \0 033 \0 � \0 026 � \0 \0 \0 \0
0104970 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � \r
0104980 \f \0 � \0 034 \0 � \0 030 � \0 \0 \0 \0
0104990 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
01049a0 \r \0 � \0 035 \0 � \0 032 � \0 \0 \0 \0
01049b0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 003 017
01049c0 016 \0 � \0 036 \0 � \0 034 � \0 \0 \0 \0
01049d0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 m �
01049e0 017 \0 � \0 037 \0 � \0 036 � \0 \0 \0 \0
01049f0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � 016
0104a00 \0 \0 � \0 020 \0 � \0 \0 � \0 � _ \0
0104a10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 N �
0104a20 001 \0 � \0 021 \0 � \0 002 � \0 \0 \0 \0
0104a30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � y
0104a40 002 \0 � \0 022 \0 � \0 004 � \0 \0 \0 \0
0104a50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104a60 003 \0 � \0 023 \0 � \0 006 � \0 \0 \0 \0
0104a70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 036 x
0104a80 004 \0 � \0 024 \0 � \0 \b � \0 \0 \0 \0
0104a90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 032 �
0104aa0 005 \0 � \0 025 \0 � \0 \n � \0 \0 \0 \0
0104ab0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � z
0104ac0 006 \0 � \0 026 \0 � \0 \f � \0 \0 \0 \0
0104ad0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104ae0 \a \0 � \0 027 \0 � \0 016 � \0 \0 \0 \0
0104af0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 t {
0104b00 \b \0 � \0 030 \0 � \0 020 � \0 \0 \0 \0
0104b10 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104b20 \t \0 � \0 031 \0 � \0 022 � \0 \0 \0 \0
0104b30 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 177 177
0104b40 \n \0 � \0 032 \0 � \0 024 � \0 \0 \0 \0
0104b50 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 021 �
0104b60 \v \0 � \0 033 \0 � \0 026 � \0 \0 \0 \0
0104b70 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � ~
0104b80 \f \0 � \0 034 \0 � \0 030 � \0 \0 \0 \0
0104b90 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � �
0104ba0 \r \0 � \0 035 \0 � \0 032 � \0 \0 \0 \0
0104bb0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 025 |
0104bc0 016 \0 � \0 036 \0 � \0 034 � \0 \0 \0 \0
0104bd0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 { �
0104be0 017 \0 � \0 037 \0 � \0 036 � \0 \0 \0 \0
0104bf0 \0 \0 005 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 � }
root@debianbase:/#
According to the responses I have something at md0 and you have
the first 1000 lines of hexdump. (Sorry the hexdump doesn't tell
me anything - - - unless its that there seem to be some lines missing
there isn't anything from 0000000 to 000001c0 and 00001e0 is missing,
then 0000260 to 0100400 are missing, also missing are 0100580 to
0101000 with each of the lists being non-inclusive of noted lines.)
I am going to be out for a few hours, will check in after that.
Dee
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Advice requested
From: Phil Turmel @ 2015-11-03 0:57 UTC (permalink / raw)
To: o1bigtenor; +Cc: Linux-RAID
In-Reply-To: <CAPpdf59mkRiyNYF1dT6Xp9tcofTU8hOdih+tz+i=wPJfRQrTnQ@mail.gmail.com>
On 11/02/2015 07:44 PM, o1bigtenor wrote:
> root@debianbase:/# dd if=/dev/md0 bs=1M count=16 |hexdump -c |head -n 1000
> 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
Please use a capital 'C' as the hexdump option.
Hexdump collapses repeated data as a convenience ... that I was counting on.
Phil
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox