Re: [git patch] 2.6.x libata fix

Re: [git patch] 2.6.x libata fix

[Matt Darcy]

Matt Darcy wrote:

> Jeff Garzik wrote:
>
>> Please pull from 'upstream-linus' branch of
>> master.kernel.org:/pub/scm/linux/kernel/git/jgarzik/libata-dev.git
>>
>> to receive the following updates:
>>
>> drivers/scsi/sata_nv.c |   30 ++++++++++++++++++++++++------
>> 1 files changed, 24 insertions(+), 6 deletions(-)
>>
>> Andrew Chew:
>>      sata_nv, spurious interrupts at system startup with MAXTOR 
>> 6H500F0 drive
>>
>> diff --git a/drivers/scsi/sata_nv.c b/drivers/scsi/sata_nv.c
>> index c0cf52c..bbbb55e 100644
>> --- a/drivers/scsi/sata_nv.c
>> +++ b/drivers/scsi/sata_nv.c
>>
>> <snip reset of patch>
>>  
>>
> Hi Jeff, et all
>
> I pulled this down last night from 
> git.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-dev.git
> as I don't have an account on master.kernel.org, I assume these are 
> the same physical tree.
>
> This built fine and seemed aware of all the SATA disks on my 
> controller (as did previous git branches).
>
> Again using my raid5 (7 disks 1.4tb) example, I am unable to build an 
> array without the machine hanging.
>
> I am able to access the individual disks for a short period of time 
> before the box hangs totally which I assume is why the raid array will 
> not build as the access for the disks hangs, not the actual building 
> of the array.
>
> I havn't got any output yet on the errors (if there are any) as I left 
> the array building overnight (300+ minutes) and when I woke up this 
> morning the box had hung, but power saver had come on the screen so I 
> couldn't see the messages.
>
> I'll do another test this afternoon and try to get some output for 
> you, but I just thought I'd let you know this was coming.
>
> Aslo FYI: I'm using maxtor SATA disks, so your patch was of particular 
> interesting to me.
>
> Many thanks,
>
> Matt


I can now provide further updates for this, although this are not really 
super useful.

I've copied the linux-raid list in as well, as after a little more 
testing on my part I'd appriciate some input from the raid guys also.

First of all, please ignore the comments above, there was a problem with 
grub and it actually "failed back" and booted into the older git 
release, so my initial test was actually done running the wrong kenel 
which I didn't notice. Appologies to all for this.

Last nights tests where done using the correct kernel (I fixed the grub 
typo) 2.6.15-g5367f2d6

The details I have are as follows.

I can run the machine accessing the 7 maxtor SATA disks as individual 
disks for around 12 hours now, without any hangs or errors or any real 
problems. I've not hit them very hard, but initial performance seems 
fine and more than usable.

The actual problems occurr when including these disks in a raid group.

root@berger:~# fdisk -l /dev/sdc

Disk /dev/sdc: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       30515   245111706   fd  Linux raid 
autodetect

root@berger:~# fdisk -l /dev/sde

Disk /dev/sde: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1       30515   245111706   fd  Linux raid 
autodetect


As you can see from my two random disks examples, they are partitioned 
and makred as raid auto detect.

I issue the mdadm command to build the raid 5 array

 mdadm -C /dev/md6 -l5 -n6 -x1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 
/dev/sdg1 /dev/sdh1 /dev/sdi1

and the array starts to build.......

md6 : active raid5 sdh1[7] sdi1[6](S) sdg1[4] sdf1[3] sde1[2] sdd1[1] 
sdc1[0]
      1225558080 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
      [>....................]  recovery =  0.1% (374272/245111616) 
finish=337.8min speed=12073K/sec


however at around %25 - %40 completion the box will simpley just hang - 
I'm getting no on screen messages and the sylog is not reporting anything.

SysRQ is unusable.

I'm open to options on how to resolve this and move the driver forward 
(assuming it is the drivers interfaction with the raid sub system)
or
how to get some meaningful debug out to report back to the appropriate 
development groups.

thanks.

Matt.

Re: [git patch] 2.6.x libata fix

[PFC]

	Hello,

	I've had a lot of problems with nv_sata (nforce3) and Maxtor harddrives.  
Basically it always boils down to :

Dec 24 23:04:34 apollo13 ata3: command 0x35 timeout, stat 0xd0 host_stat  
0x21
Dec 24 23:04:34 apollo13 ata3: translated ATA stat/err 0x35/00 to SCSI  
SK/ASC/ASCQ 0x4/00/00
Dec 24 23:04:34 apollo13 ata3: status=0x35 { DeviceFault SeekComplete  
CorrectedError Error }
Dec 24 23:04:34 apollo13 sd 2:0:0:0: SCSI error: return code = 0x8000002
Dec 24 23:04:34 apollo13 sdc: Current: sense key=0x4
Dec 24 23:04:34 apollo13 ASC=0x0 ASCQ=0x0
Dec 24 23:04:34 apollo13 end_request: I/O error, dev sdc, sector 490223295
Dec 24 23:04:34 apollo13 raid5: Disk failure on sdc1, disabling device.  
Operation continuing on 3 devices

	Plugging the Maxtor drives on the not-nv SATA sockets solved it (thanks  
god I have 2 maxtor and 2 seagate drives)...

	If you want more info, just ask.

Re: [git patch] 2.6.x libata fix

[Matt Darcy]

PFC wrote:

>
>     Hello,
>
>     I've had a lot of problems with nv_sata (nforce3) and Maxtor 
> harddrives.  Basically it always boils down to :
>
> Dec 24 23:04:34 apollo13 ata3: command 0x35 timeout, stat 0xd0 
> host_stat  0x21
> Dec 24 23:04:34 apollo13 ata3: translated ATA stat/err 0x35/00 to 
> SCSI  SK/ASC/ASCQ 0x4/00/00
> Dec 24 23:04:34 apollo13 ata3: status=0x35 { DeviceFault SeekComplete  
> CorrectedError Error }
> Dec 24 23:04:34 apollo13 sd 2:0:0:0: SCSI error: return code = 0x8000002
> Dec 24 23:04:34 apollo13 sdc: Current: sense key=0x4
> Dec 24 23:04:34 apollo13 ASC=0x0 ASCQ=0x0
> Dec 24 23:04:34 apollo13 end_request: I/O error, dev sdc, sector 
> 490223295
> Dec 24 23:04:34 apollo13 raid5: Disk failure on sdc1, disabling 
> device.  Operation continuing on 3 devices
>
>     Plugging the Maxtor drives on the not-nv SATA sockets solved it 
> (thanks  god I have 2 maxtor and 2 seagate drives)...
>
>

Thanks for the message,

I'm not using nv_sata, I'm using sata_mv,

just to clarify.

Could we please try to keep this thread on topic.

Regards,

Matt

Re: [git patch] 2.6.x libata fix

[Matt Darcy]

>
>
> I can now provide further updates for this, although this are not 
> really super useful.
>
> I've copied the linux-raid list in as well, as after a little more 
> testing on my part I'd appriciate some input from the raid guys also.
>
> First of all, please ignore the comments above, there was a problem 
> with grub and it actually "failed back" and booted into the older git 
> release, so my initial test was actually done running the wrong kenel 
> which I didn't notice. Appologies to all for this.
>
> Last nights tests where done using the correct kernel (I fixed the 
> grub typo) 2.6.15-g5367f2d6
>
> The details I have are as follows.
>
> I can run the machine accessing the 7 maxtor SATA disks as individual 
> disks for around 12 hours now, without any hangs or errors or any real 
> problems. I've not hit them very hard, but initial performance seems 
> fine and more than usable.
>
> The actual problems occurr when including these disks in a raid group.
>
> root@berger:~# fdisk -l /dev/sdc
>
> Disk /dev/sdc: 251.0 GB, 251000193024 bytes
> 255 heads, 63 sectors/track, 30515 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
>
>   Device Boot      Start         End      Blocks   Id  System
> /dev/sdc1               1       30515   245111706   fd  Linux raid 
> autodetect
>
> root@berger:~# fdisk -l /dev/sde
>
> Disk /dev/sde: 251.0 GB, 251000193024 bytes
> 255 heads, 63 sectors/track, 30515 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
>
>   Device Boot      Start         End      Blocks   Id  System
> /dev/sde1               1       30515   245111706   fd  Linux raid 
> autodetect
>
>
> As you can see from my two random disks examples, they are partitioned 
> and makred as raid auto detect.
>
> I issue the mdadm command to build the raid 5 array
>
> mdadm -C /dev/md6 -l5 -n6 -x1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 
> /dev/sdg1 /dev/sdh1 /dev/sdi1
>
> and the array starts to build.......
>
> md6 : active raid5 sdh1[7] sdi1[6](S) sdg1[4] sdf1[3] sde1[2] sdd1[1] 
> sdc1[0]
>      1225558080 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
>      [>....................]  recovery =  0.1% (374272/245111616) 
> finish=337.8min speed=12073K/sec
>
>
> however at around %25 - %40 completion the box will simpley just hang 
> - I'm getting no on screen messages and the sylog is not reporting 
> anything.
>
> SysRQ is unusable.
>
> I'm open to options on how to resolve this and move the driver forward 
> (assuming it is the drivers interfaction with the raid sub system)
> or
> how to get some meaningful debug out to report back to the appropriate 
> development groups.
>
> thanks.
>
> Matt.
>
>
>
Further further information

The speed that the raid array is being built att appears to drop as the 
array is created

[=====>...............]  recovery = 29.2% (71633360/245111616) 
finish=235.1min speed=12296K/sec
[=====>...............]  recovery = 29.3% (71874512/245111616) 
finish=235.2min speed=12269K/sec
[=====>...............]  recovery = 29.4% (72115872/245111616) 
finish=236.0min speed=12209K/sec
[=====>...............]  recovery = 29.7% (72839648/245111616) 
finish=237.4min speed=12091K/sec
[=====>...............]  recovery = 29.8% (73078560/245111616) 
finish=238.6min speed=12010K/sec
[=====>...............]  recovery = 29.8% (73139424/245111616) 
finish=350.5min speed=8176K/sec
[=====>...............]  recovery = 29.8% (73139424/245111616) 
finish=499.6min speed=5735K/sec
[=====>...............]  recovery = 29.8% (73139776/245111616) 
finish=691.0min speed=4147K/sec

Now the box is hung

I didn't notice this until about %20 through the creation of the array 
then I started paying attention to this. These snap shots are taken 
every 30 seconds

So the problem appears to sap bandwidth on the card to the point there 
the box hangs.

This may have some relevance, or it may not, but worth mentioning at least.

Matt










[=====>...............]  recovery = 29.8% (73139424/245111616) 
finish=350.5min speed=8176K/sec

Re: [git patch] 2.6.x libata fix more information (sata_mv problems continued)

[Matt Darcy]

Matt Darcy wrote:

>
>>
>>
>> I can now provide further updates for this, although this are not 
>> really super useful.
>>
>> I've copied the linux-raid list in as well, as after a little more 
>> testing on my part I'd appriciate some input from the raid guys also.
>>
>> First of all, please ignore the comments above, there was a problem 
>> with grub and it actually "failed back" and booted into the older git 
>> release, so my initial test was actually done running the wrong kenel 
>> which I didn't notice. Appologies to all for this.
>>
>> Last nights tests where done using the correct kernel (I fixed the 
>> grub typo) 2.6.15-g5367f2d6
>>
>> The details I have are as follows.
>>
>> I can run the machine accessing the 7 maxtor SATA disks as individual 
>> disks for around 12 hours now, without any hangs or errors or any 
>> real problems. I've not hit them very hard, but initial performance 
>> seems fine and more than usable.
>>
>> The actual problems occurr when including these disks in a raid group.
>>
>> root@berger:~# fdisk -l /dev/sdc
>>
>> Disk /dev/sdc: 251.0 GB, 251000193024 bytes
>> 255 heads, 63 sectors/track, 30515 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>>
>>   Device Boot      Start         End      Blocks   Id  System
>> /dev/sdc1               1       30515   245111706   fd  Linux raid 
>> autodetect
>>
>> root@berger:~# fdisk -l /dev/sde
>>
>> Disk /dev/sde: 251.0 GB, 251000193024 bytes
>> 255 heads, 63 sectors/track, 30515 cylinders
>> Units = cylinders of 16065 * 512 = 8225280 bytes
>>
>>   Device Boot      Start         End      Blocks   Id  System
>> /dev/sde1               1       30515   245111706   fd  Linux raid 
>> autodetect
>>
>>
>> As you can see from my two random disks examples, they are 
>> partitioned and makred as raid auto detect.
>>
>> I issue the mdadm command to build the raid 5 array
>>
>> mdadm -C /dev/md6 -l5 -n6 -x1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 
>> /dev/sdg1 /dev/sdh1 /dev/sdi1
>>
>> and the array starts to build.......
>>
>> md6 : active raid5 sdh1[7] sdi1[6](S) sdg1[4] sdf1[3] sde1[2] sdd1[1] 
>> sdc1[0]
>>      1225558080 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
>>      [>....................]  recovery =  0.1% (374272/245111616) 
>> finish=337.8min speed=12073K/sec
>>
>>
>> however at around %25 - %40 completion the box will simpley just hang 
>> - I'm getting no on screen messages and the sylog is not reporting 
>> anything.
>>
>> SysRQ is unusable.
>>
>> I'm open to options on how to resolve this and move the driver 
>> forward (assuming it is the drivers interfaction with the raid sub 
>> system)
>> or
>> how to get some meaningful debug out to report back to the 
>> appropriate development groups.
>>
>> thanks.
>>
>> Matt.
>>
>>
>>
> Further further information
>
> The speed that the raid array is being built att appears to drop as 
> the array is created
>
> [=====>...............]  recovery = 29.2% (71633360/245111616) 
> finish=235.1min speed=12296K/sec
> [=====>...............]  recovery = 29.3% (71874512/245111616) 
> finish=235.2min speed=12269K/sec
> [=====>...............]  recovery = 29.4% (72115872/245111616) 
> finish=236.0min speed=12209K/sec
> [=====>...............]  recovery = 29.7% (72839648/245111616) 
> finish=237.4min speed=12091K/sec
> [=====>...............]  recovery = 29.8% (73078560/245111616) 
> finish=238.6min speed=12010K/sec
> [=====>...............]  recovery = 29.8% (73139424/245111616) 
> finish=350.5min speed=8176K/sec
> [=====>...............]  recovery = 29.8% (73139424/245111616) 
> finish=499.6min speed=5735K/sec
> [=====>...............]  recovery = 29.8% (73139776/245111616) 
> finish=691.0min speed=4147K/sec
>
> Now the box is hung
>
> I didn't notice this until about %20 through the creation of the array 
> then I started paying attention to this. These snap shots are taken 
> every 30 seconds
>
> So the problem appears to sap bandwidth on the card to the point there 
> the box hangs.
>
> This may have some relevance, or it may not, but worth mentioning at 
> least.
>
> Matt
>
>
>
First - a quick response to John Stoffels comments.

Both disks and controller on the latest Bios/firmware versions (thanks 
for making me point this out)

I created a much smaller array (3 disks 1 spare) today and again around 
%35 through the creation of the array the whole machine hung, no warning 
no errors no logging.
The speed parameter from /proc/mdstat stayed constant to around %30 
(which explained why I perhaps didn't notice this earlier) and like the 
creation of the large raid 5 array took a massive nose dive in speed 
over about 180 seconds to the point where the box hung.

Its almost as if there is an "IO leak" which is the only way I can think 
of to describe it.the card / system performaces quite well as individual 
disks, but as soon as its entered into a raid 5 configuration using the 
any number of disks the creation of the array appears to be fine until 
around %20-%30 through the assembly, the speed of the arrays creations 
plummits and the machine hangs.

I'm not too sure how to take this further as I get no warnings (other 
than the arrays creation time slowing) - I can't use any tools like 
netdump or sysRQ.

I'll try some additional raid tests (such as raid0 or raid1 across more 
disks) to see how that works. But as it stands I'm not sure how to get 
additional information.

thanks,

Matt

Re: [git patch] 2.6.x libata fix more information (sata_mv problems continued)

[Jens Axboe]

On Fri, Jan 13 2006, Matt Darcy wrote:
> Matt Darcy wrote:
> 
> >
> >>
> >>
> >>I can now provide further updates for this, although this are not 
> >>really super useful.
> >>
> >>I've copied the linux-raid list in as well, as after a little more 
> >>testing on my part I'd appriciate some input from the raid guys also.
> >>
> >>First of all, please ignore the comments above, there was a problem 
> >>with grub and it actually "failed back" and booted into the older git 
> >>release, so my initial test was actually done running the wrong kenel 
> >>which I didn't notice. Appologies to all for this.
> >>
> >>Last nights tests where done using the correct kernel (I fixed the 
> >>grub typo) 2.6.15-g5367f2d6
> >>
> >>The details I have are as follows.
> >>
> >>I can run the machine accessing the 7 maxtor SATA disks as individual 
> >>disks for around 12 hours now, without any hangs or errors or any 
> >>real problems. I've not hit them very hard, but initial performance 
> >>seems fine and more than usable.
> >>
> >>The actual problems occurr when including these disks in a raid group.
> >>
> >>root@berger:~# fdisk -l /dev/sdc
> >>
> >>Disk /dev/sdc: 251.0 GB, 251000193024 bytes
> >>255 heads, 63 sectors/track, 30515 cylinders
> >>Units = cylinders of 16065 * 512 = 8225280 bytes
> >>
> >>  Device Boot      Start         End      Blocks   Id  System
> >>/dev/sdc1               1       30515   245111706   fd  Linux raid 
> >>autodetect
> >>
> >>root@berger:~# fdisk -l /dev/sde
> >>
> >>Disk /dev/sde: 251.0 GB, 251000193024 bytes
> >>255 heads, 63 sectors/track, 30515 cylinders
> >>Units = cylinders of 16065 * 512 = 8225280 bytes
> >>
> >>  Device Boot      Start         End      Blocks   Id  System
> >>/dev/sde1               1       30515   245111706   fd  Linux raid 
> >>autodetect
> >>
> >>
> >>As you can see from my two random disks examples, they are 
> >>partitioned and makred as raid auto detect.
> >>
> >>I issue the mdadm command to build the raid 5 array
> >>
> >>mdadm -C /dev/md6 -l5 -n6 -x1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 
> >>/dev/sdg1 /dev/sdh1 /dev/sdi1
> >>
> >>and the array starts to build.......
> >>
> >>md6 : active raid5 sdh1[7] sdi1[6](S) sdg1[4] sdf1[3] sde1[2] sdd1[1] 
> >>sdc1[0]
> >>     1225558080 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
> >>     [>....................]  recovery =  0.1% (374272/245111616) 
> >>finish=337.8min speed=12073K/sec
> >>
> >>
> >>however at around %25 - %40 completion the box will simpley just hang 
> >>- I'm getting no on screen messages and the sylog is not reporting 
> >>anything.
> >>
> >>SysRQ is unusable.
> >>
> >>I'm open to options on how to resolve this and move the driver 
> >>forward (assuming it is the drivers interfaction with the raid sub 
> >>system)
> >>or
> >>how to get some meaningful debug out to report back to the 
> >>appropriate development groups.
> >>
> >>thanks.
> >>
> >>Matt.
> >>
> >>
> >>
> >Further further information
> >
> >The speed that the raid array is being built att appears to drop as 
> >the array is created
> >
> >[=====>...............]  recovery = 29.2% (71633360/245111616) 
> >finish=235.1min speed=12296K/sec
> >[=====>...............]  recovery = 29.3% (71874512/245111616) 
> >finish=235.2min speed=12269K/sec
> >[=====>...............]  recovery = 29.4% (72115872/245111616) 
> >finish=236.0min speed=12209K/sec
> >[=====>...............]  recovery = 29.7% (72839648/245111616) 
> >finish=237.4min speed=12091K/sec
> >[=====>...............]  recovery = 29.8% (73078560/245111616) 
> >finish=238.6min speed=12010K/sec
> >[=====>...............]  recovery = 29.8% (73139424/245111616) 
> >finish=350.5min speed=8176K/sec
> >[=====>...............]  recovery = 29.8% (73139424/245111616) 
> >finish=499.6min speed=5735K/sec
> >[=====>...............]  recovery = 29.8% (73139776/245111616) 
> >finish=691.0min speed=4147K/sec
> >
> >Now the box is hung
> >
> >I didn't notice this until about %20 through the creation of the array 
> >then I started paying attention to this. These snap shots are taken 
> >every 30 seconds
> >
> >So the problem appears to sap bandwidth on the card to the point there 
> >the box hangs.
> >
> >This may have some relevance, or it may not, but worth mentioning at 
> >least.
> >
> >Matt
> >
> >
> >
> First - a quick response to John Stoffels comments.
> 
> Both disks and controller on the latest Bios/firmware versions (thanks 
> for making me point this out)
> 
> I created a much smaller array (3 disks 1 spare) today and again around 
> %35 through the creation of the array the whole machine hung, no warning 
> no errors no logging.
> The speed parameter from /proc/mdstat stayed constant to around %30 
> (which explained why I perhaps didn't notice this earlier) and like the 
> creation of the large raid 5 array took a massive nose dive in speed 
> over about 180 seconds to the point where the box hung.
> 
> Its almost as if there is an "IO leak" which is the only way I can think 
> of to describe it.the card / system performaces quite well as individual 
> disks, but as soon as its entered into a raid 5 configuration using the 
> any number of disks the creation of the array appears to be fine until 
> around %20-%30 through the assembly, the speed of the arrays creations 
> plummits and the machine hangs.
> 
> I'm not too sure how to take this further as I get no warnings (other 
> than the arrays creation time slowing) - I can't use any tools like 
> netdump or sysRQ.
> 
> I'll try some additional raid tests (such as raid0 or raid1 across more 
> disks) to see how that works. But as it stands I'm not sure how to get 
> additional information.

You could try and monitor /proc/meminfo and /proc/slabinfo as the system
begins to slow to a crawl, any leaks of io structures should be visible
there as well.

-- 
Jens Axboe

Re: [git patch] 2.6.x libata fix more information (sata_mv problems continued)

[Sebastian Kuzminsky]

[-- Attachment #1: Type: text/plain, Size: 1718 bytes --]

Matt Darcy <kernel-lists@projecthugo.co.uk> wrote:
> Its almost as if there is an "IO leak" which is the only way I can think 
> of to describe it.the card / system performaces quite well as individual 
> disks, but as soon as its entered into a raid 5 configuration using the 
> any number of disks the creation of the array appears to be fine until 
> around %20-%30 through the assembly, the speed of the arrays creations 
> plummits and the machine hangs.

You have 7x250G disks in Raid-5, so that's 6x250G or 1.5T total space.
In the beginning of raid recovery, when the system is good, you're
getting 12M/s.  It slows then dies after 25% to 40% of completion.

6x250G is 1536000M, at 12M/s that's about 35 hours.  You tested the
disks individually (without Raid) for ~12 hours, which is about 34%
of 35 hours.  So it's possible you'd see the the same slowdown & hang
if you tested the individual disks longer.

You're having these problems on a Marvell controller with 2.6.15 and the
in-kernel sata_mv driver, right?  I've got a very similar system with
unexplained hard hangs too.  On my system the individual disks seem to
work fine, Raid-6 of the disks seems work fine, LVM of the disks seems
to work fine, but LVM of a Raid-6 of the disks hangs.

One wierd thing I've discovered is that if I enable all the kernel
debugging options, the system is perfectly stable, and all the debug
tests report no warnings or errors to the logs.  Seems like a race
condition somewhere, I'm suspecting in the interaction of Raid-6 and
LVM, but it could be anywhere I suppose.  I've attached the .config of
the production (non-debug) kernel that hangs, and the diff to the debug
kernel that works.


-- 
Sebastian Kuzminsky


[-- Attachment #2: Type: text/plain, Size: 34582 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.15-minimal-debug
# Tue Jan  3 15:35:25 2006
#
CONFIG_X86_32=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
# CONFIG_CLEAN_COMPILE is not set
CONFIG_BROKEN=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
# CONFIG_IKCONFIG is not set
# CONFIG_CPUSETS is not set
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_EMBEDDED=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y

#
# Block layer
#
CONFIG_LBD=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_HPET_TIMER=y
CONFIG_SMP=y
CONFIG_NR_CPUS=4
CONFIG_SCHED_SMT=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_BKL is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m

#
# Firmware Drivers
#
CONFIG_EDD=m
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_HIGHMEM=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_HIGHPTE is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
CONFIG_IRQBALANCE=y
# CONFIG_REGPARM is not set
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_PHYSICAL_START=0x100000
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
CONFIG_PM_LEGACY=y
# CONFIG_PM_DEBUG is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
CONFIG_ACPI=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_HOTKEY=m
CONFIG_ACPI_FAN=m
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_THERMAL=m
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_IBM is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=m

#
# APM (Advanced Power Management) BIOS Support
#
# CONFIG_APM is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m

#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=m
# CONFIG_X86_POWERNOW_K6 is not set
# CONFIG_X86_POWERNOW_K7 is not set
# CONFIG_X86_POWERNOW_K8 is not set
# CONFIG_X86_GX_SUSPMOD is not set
CONFIG_X86_SPEEDSTEP_CENTRINO=m
CONFIG_X86_SPEEDSTEP_CENTRINO_ACPI=y
CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
CONFIG_X86_SPEEDSTEP_ICH=m
CONFIG_X86_SPEEDSTEP_SMI=m
CONFIG_X86_P4_CLOCKMOD=m
# CONFIG_X86_CPUFREQ_NFORCE2 is not set
# CONFIG_X86_LONGRUN is not set
# CONFIG_X86_LONGHAUL is not set

#
# shared options
#
CONFIG_X86_ACPI_CPUFREQ_PROC_INTF=y
CONFIG_X86_SPEEDSTEP_LIB=m
CONFIG_X86_SPEEDSTEP_RELAXED_CAP_CHECK=y

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY_PROC=y
# CONFIG_PCI_DEBUG is not set
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
# CONFIG_HOTPLUG_CPU is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=m
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=m
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
# CONFIG_NET_IPGRE_BROADCAST is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_BIC=y

#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_INET6_TUNNEL=m
CONFIG_IPV6_TUNNEL=m
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
CONFIG_IP_NF_CT_ACCT=y
CONFIG_IP_NF_CONNTRACK_MARK=y
# CONFIG_IP_NF_CONNTRACK_EVENTS is not set
CONFIG_IP_NF_CONNTRACK_NETLINK=m
CONFIG_IP_NF_CT_PROTO_SCTP=m
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_NETBIOS_NS=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_PPTP=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_REALM=m
CONFIG_IP_NF_MATCH_SCTP=m
CONFIG_IP_NF_MATCH_DCCP=m
CONFIG_IP_NF_MATCH_COMMENT=m
CONFIG_IP_NF_MATCH_CONNMARK=m
CONFIG_IP_NF_MATCH_CONNBYTES=m
CONFIG_IP_NF_MATCH_HASHLIMIT=m
CONFIG_IP_NF_MATCH_STRING=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_TARGET_NFQUEUE=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_NAT_PPTP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_TARGET_CONNMARK=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration (EXPERIMENTAL)
#
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_LIMIT=m
CONFIG_IP6_NF_MATCH_MAC=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_MULTIPORT=m
CONFIG_IP6_NF_MATCH_OWNER=m
CONFIG_IP6_NF_MATCH_MARK=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_AHESP=m
CONFIG_IP6_NF_MATCH_LENGTH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_TARGET_NFQUEUE=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_TARGET_MARK=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_RAW=m

#
# DCCP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set
CONFIG_NET_CLS_ROUTE=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_IEEE80211 is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
# CONFIG_DEBUG_DRIVER is not set

#
# Connector - unified userspace <-> kernelspace linker
#
CONFIG_CONNECTOR=m

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
CONFIG_PARPORT_PC_FIFO=y
CONFIG_PARPORT_PC_SUPERIO=y
CONFIG_PARPORT_NOT_PC=y
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_1284=y

#
# Plug and Play support
#
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
CONFIG_ISAPNP=y
CONFIG_PNPBIOS=y
CONFIG_PNPBIOS_PROC_FS=y
CONFIG_PNPACPI=y

#
# Block devices
#
CONFIG_BLK_DEV_FD=m
# CONFIG_BLK_DEV_XD is not set
CONFIG_PARIDE=m
CONFIG_PARIDE_PARPORT=m

#
# Parallel IDE high-level drivers
#
CONFIG_PARIDE_PD=m
CONFIG_PARIDE_PCD=m
CONFIG_PARIDE_PF=m
CONFIG_PARIDE_PT=m
CONFIG_PARIDE_PG=m

#
# Parallel IDE protocol modules
#
CONFIG_PARIDE_ATEN=m
CONFIG_PARIDE_BPCK=m
CONFIG_PARIDE_BPCK6=m
CONFIG_PARIDE_COMM=m
CONFIG_PARIDE_DSTR=m
CONFIG_PARIDE_FIT2=m
CONFIG_PARIDE_FIT3=m
CONFIG_PARIDE_EPAT=m
CONFIG_PARIDE_EPATC8=y
CONFIG_PARIDE_EPIA=m
CONFIG_PARIDE_FRIQ=m
CONFIG_PARIDE_FRPW=m
CONFIG_PARIDE_KBIC=m
CONFIG_PARIDE_KTTI=m
CONFIG_PARIDE_ON20=m
CONFIG_PARIDE_ON26=m
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=8192
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDETAPE=m
CONFIG_BLK_DEV_IDEFLOPPY=m
CONFIG_BLK_DEV_IDESCSI=m
# CONFIG_IDE_TASK_IOCTL is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_CS5535 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
CONFIG_BLK_DEV_PDC202XX_NEW=m
CONFIG_PDC202XX_FORCE=y
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
# CONFIG_IDE_CHIPSETS is not set
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_IVB=y
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=m
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y

#
# SCSI Transport Attributes
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=8
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
CONFIG_SCSI_AIC7XXX_OLD=m
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
CONFIG_SCSI_SATA=m
# CONFIG_SCSI_SATA_AHCI is not set
# CONFIG_SCSI_SATA_SVW is not set
# CONFIG_SCSI_ATA_PIIX is not set
CONFIG_SCSI_SATA_MV=m
# CONFIG_SCSI_SATA_NV is not set
# CONFIG_SCSI_PDC_ADMA is not set
# CONFIG_SCSI_SATA_QSTOR is not set
# CONFIG_SCSI_SATA_PROMISE is not set
# CONFIG_SCSI_SATA_SX4 is not set
# CONFIG_SCSI_SATA_SIL is not set
# CONFIG_SCSI_SATA_SIL24 is not set
# CONFIG_SCSI_SATA_SIS is not set
# CONFIG_SCSI_SATA_ULI is not set
# CONFIG_SCSI_SATA_VIA is not set
# CONFIG_SCSI_SATA_VITESSE is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=m
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA24XX is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_SEAGATE is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID5=m
CONFIG_MD_RAID6=m
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_EMC=m

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
# CONFIG_FUSION_SAS is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Network device support
#
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m
# CONFIG_NET_SB1000 is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# PHY device support
#

#
# Ethernet (10 or 100Mbit)
#
# CONFIG_NET_ETHERNET is not set
CONFIG_MII=m

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
CONFIG_E1000=m
# CONFIG_E1000_NAPI is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SK98LIN is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_CHELSIO_T1 is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
CONFIG_KEYBOARD_XTKBD=m
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_WISTRON_BTNS is not set
CONFIG_INPUT_UINPUT=m

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
CONFIG_SERIO_CT82C710=m
CONFIG_SERIO_PARKBD=m
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_ACPI=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
# CONFIG_SERIAL_8250_MANY_PORTS is not set
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
# CONFIG_SERIAL_8250_RSA is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
# CONFIG_TIPAR is not set

#
# IPMI
#
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
CONFIG_HW_RANDOM=m
CONFIG_NVRAM=m
CONFIG_RTC=m
CONFIG_GEN_RTC=m
CONFIG_GEN_RTC_X=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=m

#
# TPM devices
#
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set

#
# I2C support
#
CONFIG_I2C=m
CONFIG_I2C_CHARDEV=m

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCF=m
CONFIG_I2C_ALGOPCA=m

#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_ELEKTOR=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_STUB=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
CONFIG_I2C_PCA_ISA=m

#
# Miscellaneous I2C Chip support
#
CONFIG_SENSORS_DS1337=m
CONFIG_SENSORS_DS1374=m
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
CONFIG_SENSORS_PCA9539=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_RTC8564=m
CONFIG_SENSORS_MAX6875=m
CONFIG_RTC_X1205_I2C=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
CONFIG_W1=m
CONFIG_W1_MATROX=m
CONFIG_W1_DS9490=m
CONFIG_W1_DS9490_BRIDGE=m
CONFIG_W1_THERM=m
CONFIG_W1_SMEM=m
CONFIG_W1_DS2433=m
CONFIG_W1_DS2433_CRC=y

#
# Hardware Monitoring support
#
CONFIG_HWMON=m
CONFIG_HWMON_VID=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_FSCPOS=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_HDAPS=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia Capabilities Port drivers
#

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
# CONFIG_FB is not set
CONFIG_VIDEO_SELECT=y

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
# CONFIG_SOUND is not set

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_SPLIT_ISO=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#

#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_USBAT=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y

#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y

#
# USB HID Boot Protocol drivers
#
CONFIG_USB_KBD=m
CONFIG_USB_MOUSE=m
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_ACECAD is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_ITMTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_YEALINK is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set
# CONFIG_USB_KEYSPAN_REMOTE is not set
# CONFIG_USB_APPLETOUCH is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB Multimedia devices
#
# CONFIG_USB_DABUSB is not set

#
# Video4Linux support is needed for USB Multimedia device support
#

#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_CDCETHER=m
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
# CONFIG_USB_NET_PLUSB is not set
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
# CONFIG_USB_NET_ZAURUS is not set
CONFIG_USB_MON=y

#
# USB port drivers
#
CONFIG_USB_USS720=m

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRPRIME=m
CONFIG_USB_SERIAL_ANYDATA=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP2101=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KEYSPAN_MPR=y
CONFIG_USB_SERIAL_KEYSPAN_USA28=y
CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
CONFIG_USB_SERIAL_KEYSPAN_USA19=y
CONFIG_USB_SERIAL_KEYSPAN_USA18X=y
CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_EZUSB=y

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
CONFIG_USB_CYTHERM=m
CONFIG_USB_PHIDGETKIT=m
CONFIG_USB_PHIDGETSERVO=m
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
CONFIG_USB_TEST=m

#
# USB DSL modem support
#

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# SN Devices
#

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_JBD_DEBUG=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
CONFIG_REISERFS_PROC_INFO=y
# CONFIG_REISERFS_FS_XATTR is not set
CONFIG_JFS_FS=y
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
CONFIG_JFS_STATISTICS=y
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=y
CONFIG_XFS_EXPORT=y
CONFIG_XFS_QUOTA=y
CONFIG_XFS_SECURITY=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_MINIX_FS=m
CONFIG_ROMFS_FS=m
CONFIG_INOTIFY=y
CONFIG_QUOTA=y
CONFIG_QFMT_V1=m
CONFIG_QFMT_V2=m
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
CONFIG_FUSE_FS=y

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=m
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y
# CONFIG_RELAYFS_FS is not set

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=m
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_RPCSEC_GSS_SPKM3=m
CONFIG_SMB_FS=m
# CONFIG_SMB_NLS_DEFAULT is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_XATTR is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
# CONFIG_9P_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="cp437"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m

#
# Instrumentation Support
#
# CONFIG_PROFILING is not set
# CONFIG_KPROBES is not set

#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_LOG_BUF_SHIFT=16
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_FS is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_FRAME_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_EARLY_PRINTK is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_AES_586=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_TEST=m

#
# Hardware crypto devices
#
# CONFIG_CRYPTO_DEV_PADLOCK is not set

#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=m
CONFIG_ZLIB_DEFLATE=m
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y

[-- Attachment #3: Type: text/plain, Size: 1327 bytes --]

--- /boot/config-2.6.15-minimal-debug	2006-01-03 15:35:25.000000000 -0700
+++ /boot/config-2.6.15-full-debug	2006-01-03 15:49:01.000000000 -0700
@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
-# Linux kernel version: 2.6.15-minimal-debug
-# Tue Jan  3 15:35:25 2006
+# Linux kernel version: 2.6.15-full-debug
+# Tue Jan  3 15:49:01 2006
 #
 CONFIG_X86_32=y
 CONFIG_SEMAPHORE_SLEEPERS=y
@@ -1580,21 +1580,21 @@
 CONFIG_LOG_BUF_SHIFT=16
 CONFIG_DETECT_SOFTLOCKUP=y
 # CONFIG_SCHEDSTATS is not set
-# CONFIG_DEBUG_SLAB is not set
-# CONFIG_DEBUG_SPINLOCK is not set
-# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
+CONFIG_DEBUG_SLAB=y
+CONFIG_DEBUG_SPINLOCK=y
+CONFIG_DEBUG_SPINLOCK_SLEEP=y
 # CONFIG_DEBUG_KOBJECT is not set
-# CONFIG_DEBUG_HIGHMEM is not set
+CONFIG_DEBUG_HIGHMEM=y
 CONFIG_DEBUG_BUGVERBOSE=y
 # CONFIG_DEBUG_INFO is not set
 # CONFIG_DEBUG_FS is not set
-# CONFIG_DEBUG_VM is not set
+CONFIG_DEBUG_VM=y
 # CONFIG_FRAME_POINTER is not set
 # CONFIG_RCU_TORTURE_TEST is not set
 # CONFIG_EARLY_PRINTK is not set
-# CONFIG_DEBUG_STACKOVERFLOW is not set
-# CONFIG_DEBUG_STACK_USAGE is not set
-# CONFIG_DEBUG_PAGEALLOC is not set
+CONFIG_DEBUG_STACKOVERFLOW=y
+CONFIG_DEBUG_STACK_USAGE=y
+CONFIG_DEBUG_PAGEALLOC=y
 # CONFIG_4KSTACKS is not set
 CONFIG_X86_FIND_SMP_CONFIG=y
 CONFIG_X86_MPPARSE=y

Re: [git patch] 2.6.x libata fix more information (sata_mv problems continued)

[Matt Darcy]

Sebastian Kuzminsky wrote:

>Matt Darcy <kernel-lists@projecthugo.co.uk> wrote:
>  
>
>>Its almost as if there is an "IO leak" which is the only way I can think 
>>of to describe it.the card / system performaces quite well as individual 
>>disks, but as soon as its entered into a raid 5 configuration using the 
>>any number of disks the creation of the array appears to be fine until 
>>around %20-%30 through the assembly, the speed of the arrays creations 
>>plummits and the machine hangs.
>>    
>>
>
>You have 7x250G disks in Raid-5, so that's 6x250G or 1.5T total space.
>In the beginning of raid recovery, when the system is good, you're
>getting 12M/s.  It slows then dies after 25% to 40% of completion.
>
>6x250G is 1536000M, at 12M/s that's about 35 hours.  You tested the
>disks individually (without Raid) for ~12 hours, which is about 34%
>of 35 hours.  So it's possible you'd see the the same slowdown & hang
>if you tested the individual disks longer.
>
>You're having these problems on a Marvell controller with 2.6.15 and the
>in-kernel sata_mv driver, right?  I've got a very similar system with
>unexplained hard hangs too.  On my system the individual disks seem to
>work fine, Raid-6 of the disks seems work fine, LVM of the disks seems
>to work fine, but LVM of a Raid-6 of the disks hangs.
>
>One wierd thing I've discovered is that if I enable all the kernel
>debugging options, the system is perfectly stable, and all the debug
>tests report no warnings or errors to the logs.  Seems like a race
>condition somewhere, I'm suspecting in the interaction of Raid-6 and
>LVM, but it could be anywhere I suppose.  I've attached the .config of
>the production (non-debug) kernel that hangs, and the diff to the debug
>kernel that works.
>
>  
>


Just to clarify a few things,

using the 2.6.15 kernel I can use and assemble the raid 5 array without 
a problem, however using it lvm2 causes it to hang exactly as you have 
mentioned before.

When I first started working this problem through I started using some 
of he mm patches with the 2.6.15-rc's which made a good difference, in 
that I could build and use the array and even with lvm2 for a period of 
time, however there was a few quirky bugs with it, in that it couldn't 
maintain the arrays stability, on certain occasions, if I rebooted the 
box, most of the disks would be marked as unsuable and the array would 
refuse to start until it was rebuilt, to futher progress this I started 
using the libata git branch which again made things a "little" better, 
until the last 2 git versions where I have this problem with the raid 
array not being able to build.

from the results I have,  have a gut feeling that this is a driver 
issue, simpley due to the different results i get with the different 
kernels.

I've been given some good thoughts today (last mail in from Mark Haln 
has some good suggestions), so all I can do is run the tests Mark 
suggested and report back the results to try to progress this forward, 
although Marks tests seem to point to hardware issues, such as heat, 
vibration etc I still believe this lies at a software driver level, but 
its worth running the tests to see what additional data I can get, and 
to prove/disprove Marks suggestoins.

I shall report back later

thanks,

Matt

Re: [git patch] 2.6.x libata fix more information DEBUG INFO !!!

[Matt Darcy]

Matt Darcy wrote:

> Sebastian Kuzminsky wrote:
>
>> Matt Darcy <kernel-lists@projecthugo.co.uk> wrote:
>>  
>>
>>> Its almost as if there is an "IO leak" which is the only way I can 
>>> think of to describe it.the card / system performaces quite well as 
>>> individual disks, but as soon as its entered into a raid 5 
>>> configuration using the any number of disks the creation of the 
>>> array appears to be fine until around %20-%30 through the assembly, 
>>> the speed of the arrays creations plummits and the machine hangs.
>>>   
>>
>>
>> You have 7x250G disks in Raid-5, so that's 6x250G or 1.5T total space.
>> In the beginning of raid recovery, when the system is good, you're
>> getting 12M/s.  It slows then dies after 25% to 40% of completion.
>>
>> 6x250G is 1536000M, at 12M/s that's about 35 hours.  You tested the
>> disks individually (without Raid) for ~12 hours, which is about 34%
>> of 35 hours.  So it's possible you'd see the the same slowdown & hang
>> if you tested the individual disks longer.
>>
>> You're having these problems on a Marvell controller with 2.6.15 and the
>> in-kernel sata_mv driver, right?  I've got a very similar system with
>> unexplained hard hangs too.  On my system the individual disks seem to
>> work fine, Raid-6 of the disks seems work fine, LVM of the disks seems
>> to work fine, but LVM of a Raid-6 of the disks hangs.
>>
>> One wierd thing I've discovered is that if I enable all the kernel
>> debugging options, the system is perfectly stable, and all the debug
>> tests report no warnings or errors to the logs.  Seems like a race
>> condition somewhere, I'm suspecting in the interaction of Raid-6 and
>> LVM, but it could be anywhere I suppose.  I've attached the .config of
>> the production (non-debug) kernel that hangs, and the diff to the debug
>> kernel that works.
>>
>>  
>>
>
>
> Just to clarify a few things,
>
> using the 2.6.15 kernel I can use and assemble the raid 5 array 
> without a problem, however using it lvm2 causes it to hang exactly as 
> you have mentioned before.
>
> When I first started working this problem through I started using some 
> of he mm patches with the 2.6.15-rc's which made a good difference, in 
> that I could build and use the array and even with lvm2 for a period 
> of time, however there was a few quirky bugs with it, in that it 
> couldn't maintain the arrays stability, on certain occasions, if I 
> rebooted the box, most of the disks would be marked as unsuable and 
> the array would refuse to start until it was rebuilt, to futher 
> progress this I started using the libata git branch which again made 
> things a "little" better, until the last 2 git versions where I have 
> this problem with the raid array not being able to build.
>
> from the results I have,  have a gut feeling that this is a driver 
> issue, simpley due to the different results i get with the different 
> kernels.
>
> I've been given some good thoughts today (last mail in from Mark Haln 
> has some good suggestions), so all I can do is run the tests Mark 
> suggested and report back the results to try to progress this forward, 
> although Marks tests seem to point to hardware issues, such as heat, 
> vibration etc I still believe this lies at a software driver level, 
> but its worth running the tests to see what additional data I can get, 
> and to prove/disprove Marks suggestoins.
>
> I shall report back later
>
> thanks,
>

> Matt

Ok,

reverting back to 2.6.15-rc5-mm3 which was my "good" kernel

I started to rebuild me 3+1 spare raid 5 array (smaller test array) and 
it hung on about %50 through
however - from this kernel I got debug results. Bellow (I'll snip them 
in future mails)

I'm going to try the same test again with the latest git kernel to see 
what happens.


Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041321f>] mv_channel_reset+0xff/0x120

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041327a>] mv_stop_and_reset+0x3a/0x60

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c04128ab>] mv_host_intr+0x13b/0x180

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041298d>] mv_interrupt+0x9d/0x130

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c014080d>] handle_IRQ_event+0x3d/0x70

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c01408b6>] __do_IRQ+0x76/0x100

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c0105089>] do_IRQ+0x19/0x30

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c01033ee>] common_interrupt+0x1a/0x20

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041316c>] mv_channel_reset+0x4c/0x120

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c0512aeb>] schedule+0x31b/0x6a0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c03fb2b0>] scsi_error_handler+0x0/0xb0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041327a>] mv_stop_and_reset+0x3a/0x60

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041374f>] mv_eng_timeout+0x6f/0xb0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c040b4b7>] ata_scsi_error+0x17/0x30

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c03fb33b>] scsi_error_handler+0x8b/0xb0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c01325e6>] kthread+0xb6/0xc0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c0132530>] kthread+0x0/0xc0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c0101389>] kernel_thread_helper+0x5/0xc

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041365e>] __mv_phy_reset+0x3be/0x420

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041321f>] mv_channel_reset+0xff/0x120

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041328a>] mv_stop_and_reset+0x4a/0x60

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c04128ab>] mv_host_intr+0x13b/0x180

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041298d>] mv_interrupt+0x9d/0x130

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c014080d>] handle_IRQ_event+0x3d/0x70

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c01408b6>] __do_IRQ+0x76/0x100

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c0105089>] do_IRQ+0x19/0x30

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c01033ee>] common_interrupt+0x1a/0x20

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041316c>] mv_channel_reset+0x4c/0x120

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c0512aeb>] schedule+0x31b/0x6a0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c03fb2b0>] scsi_error_handler+0x0/0xb0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041327a>] mv_stop_and_reset+0x3a/0x60

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c041374f>] mv_eng_timeout+0x6f/0xb0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c040b4b7>] ata_scsi_error+0x17/0x30

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c03fb33b>] scsi_error_handler+0x8b/0xb0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c01325e6>] kthread+0xb6/0xc0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c0132530>] kthread+0x0/0xc0

Message from syslogd@berger at Fri Jan 13 20:59:05 2006 ...
berger kernel:  [<c0101389>] kernel_thread_helper+0x5/0xc