RAID-5 creation crashes when files system installed while recovery going on

RAID-5 creation crashes when files system installed while recovery going on

[Tirumala Reddy Marri]

All,
 I started RAID-5 creation using mdadm command. While recovery is
happening I tried to install file system . I see intermittent crash   .
And crash doesn't happen at fixed location. Any clue what could be the
problem? How does the read/writes to MD driver handled while RAID-5 is
under building.
Thanks,
Marri

Re: RAID-5 creation crashes when files system installed while recovery going on

[NeilBrown]

On Wed, June 24, 2009 8:35 am, Tirumala Reddy Marri wrote:
> All,
>  I started RAID-5 creation using mdadm command. While recovery is
> happening I tried to install file system . I see intermittent crash   .
> And crash doesn't happen at fixed location. Any clue what could be the
> problem? How does the read/writes to MD driver handled while RAID-5 is
> under building.

Can you describe exactly what you mean by "crash" ??

The raid5 driver normally has no problems handling writes while
it is resyncing.  It has a 'stripe_cache' which is used to
synchronise regular IO with resync IO.

NeilBrown

RE: RAID-5 creation crashes when files system installed while recovery going on

[NeilBrown]

On Wed, June 24, 2009 4:39 pm, Tirumala Reddy Marri wrote:
> My meaning of crash was kernel panic happens.  BTW we have HW acceleration
> to do the XOR calculations which uses ADMA driver model.
> Thanks,
> MArri

Is it possible to get details of this panic?  A digital photo
of the screen perhaps?

NeilBrown

>
> ________________________________
>
> From: linux-raid-owner@vger.kernel.org on behalf of NeilBrown
> Sent: Tue 6/23/2009 7:31 PM
> To: Tirumala Reddy Marri
> Cc: linux-raid@vger.kernel.org
> Subject: Re: RAID-5 creation crashes when files system installed while
> recovery going on
>
>
>
> On Wed, June 24, 2009 8:35 am, Tirumala Reddy Marri wrote:
>> All,
>>  I started RAID-5 creation using mdadm command. While recovery is
>> happening I tried to install file system . I see intermittent crash   .
>> And crash doesn't happen at fixed location. Any clue what could be the
>> problem? How does the read/writes to MD driver handled while RAID-5 is
>> under building.
>
> Can you describe exactly what you mean by "crash" ??
>
> The raid5 driver normally has no problems handling writes while
> it is resyncing.  It has a 'stripe_cache' which is used to
> synchronise regular IO with resync IO.
>
> NeilBrown
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

RE: RAID-5 creation crashes when files system installed while recovery going on

[Tirumala Reddy Marri]

I could not reproduce the panic. But there is one other data point is
when I tried to mount the /dev/md0 after file system installation it
failed to mount. Once the MD recover is done I was able to mount the
drive.
Thanks,
Marri

------- here is log-----
# mkfs.ext3 -b 65536 /dev/md0
Warning: blocksize 65536 not usable on most systems.
mke2fs 1.38 (30-Jun-2005)
Filesystem label=
OS type: Linux
Block size=65536 (log=6)
Fragment size=65536 (log=6)
471040 inodes, 468750 blocks
23437 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536805376
8 block groups
65528 blocks per group, 65528 fragments per group
58880 inodes per group
Superblock backups stored on blocks: 
        65528, 196584, 327640, 458696

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
# mount -t ext3 /dev/md0 /mnt/tmpmnt/
EXT3-fs: invalid journal inode.
mount: mounting /dev/md0 on /mnt/tmpmnt/ failed: Invalid argument
# mount  /dev/md0 /mnt/tmpmnt/       
EXT3-fs: invalid journal inode.
EXT2-fs: corrupt root inode, run e2fsck
mount: mounting /dev/md0 on /mnt/tmpmnt/ failed: Invalid argument
# md: md0: recovery done.
RAID5 conf printout:
 --- rd:4 wd:4
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
 disk 3, o:1, dev:sdd

# 
# mount -t ext3 /dev/md0 /mnt/tmpmnt/
kjournald starting.  Commit interval 5 seconds
EXT3 FS on md0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/ram0                63461     44131     19330  70% /
/dev/md0              29939456    525568  27913920   2% /mnt/tmpmnt

-----------

-----Original Message-----
From: NeilBrown [mailto:neilb@suse.de] 
Sent: Tuesday, June 23, 2009 11:56 PM
To: Tirumala Reddy Marri
Cc: linux-raid@vger.kernel.org; Dan Williams
Subject: RE: RAID-5 creation crashes when files system installed while
recovery going on

On Wed, June 24, 2009 4:39 pm, Tirumala Reddy Marri wrote:
> My meaning of crash was kernel panic happens.  BTW we have HW
acceleration
> to do the XOR calculations which uses ADMA driver model.
> Thanks,
> MArri

Is it possible to get details of this panic?  A digital photo
of the screen perhaps?

NeilBrown

>
> ________________________________
>
> From: linux-raid-owner@vger.kernel.org on behalf of NeilBrown
> Sent: Tue 6/23/2009 7:31 PM
> To: Tirumala Reddy Marri
> Cc: linux-raid@vger.kernel.org
> Subject: Re: RAID-5 creation crashes when files system installed while
> recovery going on
>
>
>
> On Wed, June 24, 2009 8:35 am, Tirumala Reddy Marri wrote:
>> All,
>>  I started RAID-5 creation using mdadm command. While recovery is
>> happening I tried to install file system . I see intermittent crash
.
>> And crash doesn't happen at fixed location. Any clue what could be
the
>> problem? How does the read/writes to MD driver handled while RAID-5
is
>> under building.
>
> Can you describe exactly what you mean by "crash" ??
>
> The raid5 driver normally has no problems handling writes while
> it is resyncing.  It has a 'stripe_cache' which is used to
> synchronise regular IO with resync IO.
>
> NeilBrown
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

RE: RAID-5 creation crashes when files system installed while recovery going on

[Tirumala Reddy Marri]

Hi Dan,
   I am suspecting that ADMA driver has all the XOR operations queued up
for processing and file-system mount is not waiting till ADMA finishes
all the requests. Is this working fine on Intel IO-Processor ?
Thanks and Regards,
Marri

-----Original Message-----
From: Tirumala Reddy Marri 
Sent: Wednesday, June 24, 2009 3:36 PM
To: 'NeilBrown'
Cc: linux-raid@vger.kernel.org; Dan Williams
Subject: RE: RAID-5 creation crashes when files system installed while
recovery going on


I could not reproduce the panic. But there is one other data point is
when I tried to mount the /dev/md0 after file system installation it
failed to mount. Once the MD recover is done I was able to mount the
drive.
Thanks,
Marri

------- here is log-----
# mkfs.ext3 -b 65536 /dev/md0
Warning: blocksize 65536 not usable on most systems.
mke2fs 1.38 (30-Jun-2005)
Filesystem label=
OS type: Linux
Block size=65536 (log=6)
Fragment size=65536 (log=6)
471040 inodes, 468750 blocks
23437 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536805376
8 block groups
65528 blocks per group, 65528 fragments per group
58880 inodes per group
Superblock backups stored on blocks: 
        65528, 196584, 327640, 458696

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
# mount -t ext3 /dev/md0 /mnt/tmpmnt/
EXT3-fs: invalid journal inode.
mount: mounting /dev/md0 on /mnt/tmpmnt/ failed: Invalid argument
# mount  /dev/md0 /mnt/tmpmnt/       
EXT3-fs: invalid journal inode.
EXT2-fs: corrupt root inode, run e2fsck
mount: mounting /dev/md0 on /mnt/tmpmnt/ failed: Invalid argument
# md: md0: recovery done.
RAID5 conf printout:
 --- rd:4 wd:4
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
 disk 3, o:1, dev:sdd

# 
# mount -t ext3 /dev/md0 /mnt/tmpmnt/
kjournald starting.  Commit interval 5 seconds
EXT3 FS on md0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/ram0                63461     44131     19330  70% /
/dev/md0              29939456    525568  27913920   2% /mnt/tmpmnt

-----------

-----Original Message-----
From: NeilBrown [mailto:neilb@suse.de] 
Sent: Tuesday, June 23, 2009 11:56 PM
To: Tirumala Reddy Marri
Cc: linux-raid@vger.kernel.org; Dan Williams
Subject: RE: RAID-5 creation crashes when files system installed while
recovery going on

On Wed, June 24, 2009 4:39 pm, Tirumala Reddy Marri wrote:
> My meaning of crash was kernel panic happens.  BTW we have HW
acceleration
> to do the XOR calculations which uses ADMA driver model.
> Thanks,
> MArri

Is it possible to get details of this panic?  A digital photo
of the screen perhaps?

NeilBrown

>
> ________________________________
>
> From: linux-raid-owner@vger.kernel.org on behalf of NeilBrown
> Sent: Tue 6/23/2009 7:31 PM
> To: Tirumala Reddy Marri
> Cc: linux-raid@vger.kernel.org
> Subject: Re: RAID-5 creation crashes when files system installed while
> recovery going on
>
>
>
> On Wed, June 24, 2009 8:35 am, Tirumala Reddy Marri wrote:
>> All,
>>  I started RAID-5 creation using mdadm command. While recovery is
>> happening I tried to install file system . I see intermittent crash
.
>> And crash doesn't happen at fixed location. Any clue what could be
the
>> problem? How does the read/writes to MD driver handled while RAID-5
is
>> under building.
>
> Can you describe exactly what you mean by "crash" ??
>
> The raid5 driver normally has no problems handling writes while
> it is resyncing.  It has a 'stripe_cache' which is used to
> synchronise regular IO with resync IO.
>
> NeilBrown
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

Re: RAID-5 creation crashes when files system installed while recovery going on

[Dan Williams]

On Wed, Jun 24, 2009 at 4:45 PM, Tirumala Reddy Marri<tmarri@amcc.com> wrote:
> Hi Dan,
>   I am suspecting that ADMA driver has all the XOR operations queued up
> for processing and file-system mount is not waiting till ADMA finishes
> all the requests. Is this working fine on Intel IO-Processor ?

Yes, this works fine on iop13xx with 2.6.30, see the log below.  Check
to see if your driver can run dmatest (CONFIG_DMATEST).  Make sure
CONFIG_NET_DMA=n and the raid456 module is not loaded, otherwise
dmatest will not be able to acquire the channels.

Regards,
Dan

iq81340mc:~# mdadm --create /dev/md0 /dev/sd[a-d] -n 4 -l 5
md: bind<sda>
md: bind<sdb>
md: bind<sdc>
md: bind<sdd>
raid5: device sdc operational as raid disk 2
raid5: device sdb operational as raid disk 1
raid5: device sda operational as raid disk 0
raid5: allocated 4208kB for md0
raid5: raid level 5 set md0 active with 3 out of 4 devices, algorithm 2
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
 disk 3, o:1, dev:sdd
md0: detected capacity change from 0 to 240078815232
 md0:<6>md: recovery of RAID array md0
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than
200000 KB/sec) for recovery.
md: using 128k window, over a total of 78150656 blocks.
 unknown partition table
mdadm: array /dev/md/0 started.
iq81340mc:~# mkfs /dev/md0
mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
29310976 inodes, 58612992 blocks
2930649 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
1789 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
iq81340mc:~# cat /proc/interrupts
           CPU0
  0:       6929   IOP13xx-1  iop-adma
  1:     301173   IOP13xx-1  iop-adma
  2:     186895   IOP13xx-1  iop-adma
  3:     187957   IOP13xx-1  iop-adma
  4:          0   IOP13xx-1  iop-adma
  5:          5   IOP13xx-1  iop-adma
  8:      18175   IOP13xx-1  IOP Timer Tick
 10:          0   IOP13xx-1  IOP3xx-I2C
 11:          0   IOP13xx-1  IOP3xx-I2C
 25:     127234   IOP13xx-1  sata_vsc
 26:      31572   IOP13xx-1  eth0
 52:       3937   IOP13xx-2  serial
 57:          0   IOP13xx-2  iop-adma
 58:          0   IOP13xx-2  iop-adma
 59:          0   IOP13xx-2  iop-adma
 96:          0   IOP13xx-4  IOP3xx-I2C
Err:          0

iq81340mc:~# fsck /dev/md0
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
/dev/md0: clean, 12/29310976 files, 960553/58612992 blocks
iq81340mc:~# mdadm --fail /dev/md0 /dev/sdd
raid5: Disk failure on sdd, disabling device.
raid5: Operation continuing on 3 devices.
md: md0: recovery done.
mdadm: set /dev/sdd faulty in /dev/md0
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
 disk 3, o:0, dev:sdd
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
iq81340mc:~# echo 3 > /proc/sys/vm/drop_caches
iq81340mc:~# fsck /dev/md0
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
/dev/md0: clean, 12/29310976 files, 960553/58612992 blocks
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: RAID-5 creation crashes when files system installed while recovery going on

[Tirumala Reddy Marri]

Thanks Dan. 
   Looks like my ADMA driver has issue. I will debug more into it.
Regards,
Marri

-----Original Message-----
From: dan.j.williams@gmail.com [mailto:dan.j.williams@gmail.com] On Behalf Of Dan Williams
Sent: Thursday, June 25, 2009 6:26 PM
To: Tirumala Reddy Marri
Cc: NeilBrown; linux-raid@vger.kernel.org
Subject: Re: RAID-5 creation crashes when files system installed while recovery going on

On Wed, Jun 24, 2009 at 4:45 PM, Tirumala Reddy Marri<tmarri@amcc.com> wrote:
> Hi Dan,
>   I am suspecting that ADMA driver has all the XOR operations queued up
> for processing and file-system mount is not waiting till ADMA finishes
> all the requests. Is this working fine on Intel IO-Processor ?

Yes, this works fine on iop13xx with 2.6.30, see the log below.  Check
to see if your driver can run dmatest (CONFIG_DMATEST).  Make sure
CONFIG_NET_DMA=n and the raid456 module is not loaded, otherwise
dmatest will not be able to acquire the channels.

Regards,
Dan

iq81340mc:~# mdadm --create /dev/md0 /dev/sd[a-d] -n 4 -l 5
md: bind<sda>
md: bind<sdb>
md: bind<sdc>
md: bind<sdd>
raid5: device sdc operational as raid disk 2
raid5: device sdb operational as raid disk 1
raid5: device sda operational as raid disk 0
raid5: allocated 4208kB for md0
raid5: raid level 5 set md0 active with 3 out of 4 devices, algorithm 2
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
 disk 3, o:1, dev:sdd
md0: detected capacity change from 0 to 240078815232
 md0:<6>md: recovery of RAID array md0
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than
200000 KB/sec) for recovery.
md: using 128k window, over a total of 78150656 blocks.
 unknown partition table
mdadm: array /dev/md/0 started.
iq81340mc:~# mkfs /dev/md0
mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
29310976 inodes, 58612992 blocks
2930649 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
1789 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
iq81340mc:~# cat /proc/interrupts
           CPU0
  0:       6929   IOP13xx-1  iop-adma
  1:     301173   IOP13xx-1  iop-adma
  2:     186895   IOP13xx-1  iop-adma
  3:     187957   IOP13xx-1  iop-adma
  4:          0   IOP13xx-1  iop-adma
  5:          5   IOP13xx-1  iop-adma
  8:      18175   IOP13xx-1  IOP Timer Tick
 10:          0   IOP13xx-1  IOP3xx-I2C
 11:          0   IOP13xx-1  IOP3xx-I2C
 25:     127234   IOP13xx-1  sata_vsc
 26:      31572   IOP13xx-1  eth0
 52:       3937   IOP13xx-2  serial
 57:          0   IOP13xx-2  iop-adma
 58:          0   IOP13xx-2  iop-adma
 59:          0   IOP13xx-2  iop-adma
 96:          0   IOP13xx-4  IOP3xx-I2C
Err:          0

iq81340mc:~# fsck /dev/md0
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
/dev/md0: clean, 12/29310976 files, 960553/58612992 blocks
iq81340mc:~# mdadm --fail /dev/md0 /dev/sdd
raid5: Disk failure on sdd, disabling device.
raid5: Operation continuing on 3 devices.
md: md0: recovery done.
mdadm: set /dev/sdd faulty in /dev/md0
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
 disk 3, o:0, dev:sdd
RAID5 conf printout:
 --- rd:4 wd:3
 disk 0, o:1, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
iq81340mc:~# echo 3 > /proc/sys/vm/drop_caches
iq81340mc:~# fsck /dev/md0
fsck 1.40-WIP (14-Nov-2006)
e2fsck 1.40-WIP (14-Nov-2006)
/dev/md0: clean, 12/29310976 files, 960553/58612992 blocks
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html