Kernel 2.6.32-5-amd64 fails to boot.

Kernel 2.6.32-5-amd64 fails to boot.

[Leslie Rhorer]

	I have been attempting to upgrade a Debian "Squeeze" Linux box from
2.6.32-3-amd64 to 2.6.32-5-amd64, but the upgrade is a non-starter.  GRUB2
comes up just fine, but when I select the new kernel version, a number of
announcements flash by too fast to seen.  I am not 100% certain, but I
believe the initrd starts to load OK. Some text flies by far too quickly to
be seen, but then an error pops up concerning an address space collision of
some PCI device. Then it shows three errors for RAID devices md1. md2, and
md3, saying they are already in use. Immediately thereafter the system shows
errors concerning  the RAID targets being already in use, after which point
the system complains it can't mount / (md2), /dev, /sys, or /proc (in that
order) because the sources do not exist (if /dev/md2 does not exist, how can
it be busy?).  Thereafter, of course, it fails to find init, since / is not
mounted.  It then tries to run BusyBox, but Busybox complains:

/bin/sh: can't access tty; job control turned off

After that, it attempts to put up an initramfs prompt, but of course with no
tty access, it just hangs completely.  Not surprisingly, recovery mode
doesn't boot, either.  It gives a bit more detail in the output, but nothing
illuminating.  The old kernel (2.6.32-3-amd64) boots just fine.

Obviously there is a problem in the initramfs, probably with mdadm, but
what?  What should I try to manipulate in the initrd so I can find out what
is failing?

Re: Kernel 2.6.32-5-amd64 fails to boot.

[Rudy Zijlstra]

Hi Leslie,



On 02/01/2011 09:23 PM, Leslie Rhorer wrote:
> 	I have been attempting to upgrade a Debian "Squeeze" Linux box from
> 2.6.32-3-amd64 to 2.6.32-5-amd64, but the upgrade is a non-starter.  GRUB2
> comes up just fine, but when I select the new kernel version, a number of
> announcements flash by too fast to seen.  I am not 100% certain, but I
> believe the initrd starts to load OK. Some text flies by far too quickly to
> be seen, but then an error pops up concerning an address space collision of
> some PCI device. Then it shows three errors for RAID devices md1. md2, and
> md3, saying they are already in use. Immediately thereafter the system shows
> errors concerning  the RAID targets being already in use, after which point
> the system complains it can't mount / (md2), /dev, /sys, or /proc (in that
> order) because the sources do not exist (if /dev/md2 does not exist, how can
> it be busy?).  Thereafter, of course, it fails to find init, since / is not
> mounted.  It then tries to run BusyBox, but Busybox complains:
>
> /bin/sh: can't access tty; job control turned off
>
> After that, it attempts to put up an initramfs prompt, but of course with no
> tty access, it just hangs completely.  Not surprisingly, recovery mode
> doesn't boot, either.  It gives a bit more detail in the output, but nothing
> illuminating.  The old kernel (2.6.32-3-amd64) boots just fine.
>    

Can you enable serial console and catch the console output that way?

Cheers

Rudy
> Obviously there is a problem in the initramfs, probably with mdadm, but
> what?  What should I try to manipulate in the initrd so I can find out what
> is failing?
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

RE: Kernel 2.6.32-5-amd64 fails to boot.

[Leslie Rhorer]

> On 02/01/2011 09:23 PM, Leslie Rhorer wrote:
> > 	I have been attempting to upgrade a Debian "Squeeze" Linux box from
> > 2.6.32-3-amd64 to 2.6.32-5-amd64, but the upgrade is a non-starter.
> GRUB2
> > comes up just fine, but when I select the new kernel version, a number
> of
> > announcements flash by too fast to seen.  I am not 100% certain, but I
> > believe the initrd starts to load OK. Some text flies by far too quickly
> to
> > be seen, but then an error pops up concerning an address space collision
> of
> > some PCI device. Then it shows three errors for RAID devices md1. md2,
> and
> > md3, saying they are already in use. Immediately thereafter the system
> shows
> > errors concerning  the RAID targets being already in use, after which
> point
> > the system complains it can't mount / (md2), /dev, /sys, or /proc (in
> that
> > order) because the sources do not exist (if /dev/md2 does not exist, how
> can
> > it be busy?).  Thereafter, of course, it fails to find init, since / is
> not
> > mounted.  It then tries to run BusyBox, but Busybox complains:
> >
> > /bin/sh: can't access tty; job control turned off
> >
> > After that, it attempts to put up an initramfs prompt, but of course
> with no
> > tty access, it just hangs completely.  Not surprisingly, recovery mode
> > doesn't boot, either.  It gives a bit more detail in the output, but
> nothing
> > illuminating.  The old kernel (2.6.32-3-amd64) boots just fine.
> >
> 
> Can you enable serial console and catch the console output that way?

	Well, not easily.  I don't have a serial terminal.  I suppose I
could set something up.  You're suggesting I poke around to see what md is
reporting?  Also, I'm not sure why - if the keyboard console is failing -
the serial console would work better.

 
> Cheers
> 
> Rudy
> > Obviously there is a problem in the initramfs, probably with mdadm, but
> > what?  What should I try to manipulate in the initrd so I can find out
> what
> > is failing?
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

Re: Kernel 2.6.32-5-amd64 fails to boot.

[Rudy Zijlstra]

On 02/01/2011 09:52 PM, Leslie Rhorer wrote:
>> On 02/01/2011 09:23 PM, Leslie Rhorer wrote:
>>      
>>
>>> doesn't boot, either.  It gives a bit more detail in the output, but
>>>        
>> nothing
>>      
>>> illuminating.  The old kernel (2.6.32-3-amd64) boots just fine.
>>>
>>>        
>> Can you enable serial console and catch the console output that way?
>>      
> 	Well, not easily.  I don't have a serial terminal.  I suppose I
> could set something up.  You're suggesting I poke around to see what md is
> reporting?  Also, I'm not sure why - if the keyboard console is failing -
> the serial console would work better.
>
>
>    
Serial console can be captured to a file and then the output analysed. 
So you can read afterwards the messages that are flying past.

What i do in such case, is that i take serial connector from a laptop, 
and run a terminal emulater like minicom. And capture it to a file so i 
can analyse what has been happening.

RE: Kernel 2.6.32-5-amd64 fails to boot.

[Leslie Rhorer]

> -----Original Message-----
> From: Rudy Zijlstra [mailto:rudy@grumpydevil.homelinux.org]
> Sent: Tuesday, February 01, 2011 3:02 PM
> To: lrhorer@satx.rr.com
> Cc: 'Linux RAID'
> Subject: Re: Kernel 2.6.32-5-amd64 fails to boot.
> 
> On 02/01/2011 09:52 PM, Leslie Rhorer wrote:
> >> On 02/01/2011 09:23 PM, Leslie Rhorer wrote:
> >>
> >>
> >>> doesn't boot, either.  It gives a bit more detail in the output, but
> >>>
> >> nothing
> >>
> >>> illuminating.  The old kernel (2.6.32-3-amd64) boots just fine.
> >>>
> >>>
> >> Can you enable serial console and catch the console output that way?
> >>
> > 	Well, not easily.  I don't have a serial terminal.  I suppose I
> > could set something up.  You're suggesting I poke around to see what md
> is
> > reporting?  Also, I'm not sure why - if the keyboard console is failing
> -
> > the serial console would work better.
> >
> >
> >
> Serial console can be captured to a file and then the output analysed.
> So you can read afterwards the messages that are flying past.
> 
> What i do in such case, is that i take serial connector from a laptop,
> and run a terminal emulater like minicom. And capture it to a file so i
> can analyse what has been happening.

	Yeah, I don't think there is anything pertinent prior to the md
failures, but it might be worth a shot.  I don't have a laptop, but I do
have another server sitting right below the one in question.  The main
problem is, neither one has a serial port.

Re: Kernel 2.6.32-5-amd64 fails to boot.

[Rudy Zijlstra]

On 02/01/2011 10:44 PM, Leslie Rhorer wrote:
>> -----Original Message-----
>> From: Rudy Zijlstra [mailto:rudy@grumpydevil.homelinux.org]
>> Sent: Tuesday, February 01, 2011 3:02 PM
>> To: lrhorer@satx.rr.com
>> Cc: 'Linux RAID'
>> Subject: Re: Kernel 2.6.32-5-amd64 fails to boot.
>>
>> On 02/01/2011 09:52 PM, Leslie Rhorer wrote:
>>      
>>>> On 02/01/2011 09:23 PM, Leslie Rhorer wrote:
>>>>
>>>>
>>>>          
>>>>> doesn't boot, either.  It gives a bit more detail in the output, but
>>>>>
>>>>>            
>>>> nothing
>>>>
>>>>          
>>>>> illuminating.  The old kernel (2.6.32-3-amd64) boots just fine.
>>>>>
>>>>>
>>>>>            
>>>> Can you enable serial console and catch the console output that way?
>>>>
>>>>          
>>> 	Well, not easily.  I don't have a serial terminal.  I suppose I
>>> could set something up.  You're suggesting I poke around to see what md
>>>        
>> is
>>      
>>> reporting?  Also, I'm not sure why - if the keyboard console is failing
>>>        
>> -
>>      
>>> the serial console would work better.
>>>
>>>
>>>
>>>        
>> Serial console can be captured to a file and then the output analysed.
>> So you can read afterwards the messages that are flying past.
>>
>> What i do in such case, is that i take serial connector from a laptop,
>> and run a terminal emulater like minicom. And capture it to a file so i
>> can analyse what has been happening.
>>      
> 	Yeah, I don't think there is anything pertinent prior to the md
> failures, but it might be worth a shot.  I don't have a laptop, but I do
> have another server sitting right below the one in question.  The main
> problem is, neither one has a serial port.
>
>    
Network console? I could tell you how to setup serial console, but i've 
never done the network console

RE: Kernel 2.6.32-5-amd64 fails to boot.

[Leslie Rhorer]

> -----Original Message-----
> From: Rudy Zijlstra [mailto:rudy@grumpydevil.homelinux.org]
> Sent: Tuesday, February 01, 2011 4:46 PM
> To: lrhorer@satx.rr.com
> Cc: 'Linux RAID'
> Subject: Re: Kernel 2.6.32-5-amd64 fails to boot.
> 
> On 02/01/2011 10:44 PM, Leslie Rhorer wrote:
> >> -----Original Message-----
> >> From: Rudy Zijlstra [mailto:rudy@grumpydevil.homelinux.org]
> >> Sent: Tuesday, February 01, 2011 3:02 PM
> >> To: lrhorer@satx.rr.com
> >> Cc: 'Linux RAID'
> >> Subject: Re: Kernel 2.6.32-5-amd64 fails to boot.
> >>
> >> On 02/01/2011 09:52 PM, Leslie Rhorer wrote:
> >>
> >>>> On 02/01/2011 09:23 PM, Leslie Rhorer wrote:
> >>>>
> >>>>
> >>>>
> >>>>> doesn't boot, either.  It gives a bit more detail in the output, but
> >>>>>
> >>>>>
> >>>> nothing
> >>>>
> >>>>
> >>>>> illuminating.  The old kernel (2.6.32-3-amd64) boots just fine.
> >>>>>
> >>>>>
> >>>>>
> >>>> Can you enable serial console and catch the console output that way?
> >>>>
> >>>>
> >>> 	Well, not easily.  I don't have a serial terminal.  I suppose I
> >>> could set something up.  You're suggesting I poke around to see what
> md
> >>>
> >> is
> >>
> >>> reporting?  Also, I'm not sure why - if the keyboard console is
> failing
> >>>
> >> -
> >>
> >>> the serial console would work better.
> >>>
> >>>
> >>>
> >>>
> >> Serial console can be captured to a file and then the output analysed.
> >> So you can read afterwards the messages that are flying past.
> >>
> >> What i do in such case, is that i take serial connector from a laptop,
> >> and run a terminal emulater like minicom. And capture it to a file so i
> >> can analyse what has been happening.
> >>
> > 	Yeah, I don't think there is anything pertinent prior to the md
> > failures, but it might be worth a shot.  I don't have a laptop, but I do
> > have another server sitting right below the one in question.  The main
> > problem is, neither one has a serial port.
> >
> >
> Network console? I could tell you how to setup serial console, but i've
> never done the network console

	Me either.  I tired to do a Google search, but I think the words
"network", "console", and "boot" are too common to return practical results,
even when combined with "initrd".  I waded through a ton of results, but
couldn't find a tutorial on how to enable a network console in the initrd.

RE: Kernel 2.6.32-5-amd64 fails to boot.

[Justin Piszcz]

On Wed, 2 Feb 2011, Leslie Rhorer wrote:

>>>> Serial console can be captured to a file and then the output analysed.
>>>> So you can read afterwards the messages that are flying past.
>>>>
>>>> What i do in such case, is that i take serial connector from a laptop,
>>>> and run a terminal emulater like minicom. And capture it to a file so i
>>>> can analyse what has been happening.
>>>>
>>> 	Yeah, I don't think there is anything pertinent prior to the md
>>> failures, but it might be worth a shot.  I don't have a laptop, but I do
>>> have another server sitting right below the one in question.  The main
>>> problem is, neither one has a serial port.
>>>
>>>
>> Network console? I could tell you how to setup serial console, but i've
>> never done the network console
>
> 	Me either.  I tired to do a Google search, but I think the words
> "network", "console", and "boot" are too common to return practical results,
> even when combined with "initrd".  I waded through a ton of results, but
> couldn't find a tutorial on how to enable a network console in the initrd.

Hi,

Check the network console documentation in the kernel.  Essentially, in 
LILO, run:

append="netconsole=4444@192.0.0.1/eth0,514@192.0.0.2/00:60:2D:CD:2A:B7"
                              ^^ src                  ^^dest / ^dest mac addr

On the remote host, run syslogd -r (to accespt remote syslogging)

This *is* clearly documented here in the kernel docs:
linux/Documentation/networking/netconsole.txt

Re:

It takes a string configuration parameter "netconsole" in the
following format:

  netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]

    where
         src-port      source for UDP packets (defaults to 6665)
         src-ip        source IP to use (interface address)
         dev           network interface (eth0)
         tgt-port      port for logging agent (6666)
         tgt-ip        IP address for logging agent
         tgt-macaddr   ethernet MAC address for logging agent (broadcast)

Examples:

Justin.

Re: Kernel 2.6.32-5-amd64 fails to boot.

[CoolCold]

On Wed, Feb 2, 2011 at 2:00 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
>
> On Wed, 2 Feb 2011, Leslie Rhorer wrote:
>
>>>>> Serial console can be captured to a file and then the output analysed.
>>>>> So you can read afterwards the messages that are flying past.
>>>>>
>>>>> What i do in such case, is that i take serial connector from a laptop,
>>>>> and run a terminal emulater like minicom. And capture it to a file so i
>>>>> can analyse what has been happening.
>>>>>
>>>>        Yeah, I don't think there is anything pertinent prior to the md
>>>> failures, but it might be worth a shot.  I don't have a laptop, but I do
>>>> have another server sitting right below the one in question.  The main
>>>> problem is, neither one has a serial port.
>>>>
>>>>
>>> Network console? I could tell you how to setup serial console, but i've
>>> never done the network console
>>
>>        Me either.  I tired to do a Google search, but I think the words
>> "network", "console", and "boot" are too common to return practical
>> results,
>> even when combined with "initrd".  I waded through a ton of results, but
>> couldn't find a tutorial on how to enable a network console in the initrd.
>
> Hi,
>
> Check the network console documentation in the kernel.  Essentially, in
> LILO, run:
>
> append="netconsole=4444@192.0.0.1/eth0,514@192.0.0.2/00:60:2D:CD:2A:B7"
>                             ^^ src                  ^^dest / ^dest mac addr
>
> On the remote host, run syslogd -r (to accespt remote syslogging)
>
> This *is* clearly documented here in the kernel docs:
> linux/Documentation/networking/netconsole.txt
>
> Re:
>
> It takes a string configuration parameter "netconsole" in the
> following format:
>
>  netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
>
>   where
>        src-port      source for UDP packets (defaults to 6665)
>        src-ip        source IP to use (interface address)
>        dev           network interface (eth0)
>        tgt-port      port for logging agent (6666)
>        tgt-ip        IP address for logging agent
>        tgt-macaddr   ethernet MAC address for logging agent (broadcast)
>
> Examples:
>
> Justin.
As for me, when i've tryed to use netconsole on debian, i've needed to
add such lines into /etc/initramfs-tools/modules
8139too
8139cp
netconsole netconsole=4444@87.118.x.y/eth0,514@95.x.x.x/00:18:74:z:z:z

and do "update-initramfs -u -k `uname -r`" then
where 8139too are my NIC modules - can be obtained with lspci -k and
00:18:74:z:z:z was mac of the gateway. Worked on .18, but not on .32
kernel, though..

> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Kernel 2.6.32-5-amd64 fails to boot.

[Leslie Rhorer]

> Check the network console documentation in the kernel.  Essentially, in
> LILO, run:

	I'm not using LILO.

> append="netconsole=4444@192.0.0.1/eth0,514@192.0.0.2/00:60:2D:CD:2A:B7"
>                               ^^ src                  ^^dest / ^dest mac
> addr
> 
> On the remote host, run syslogd -r (to accespt remote syslogging)
> 
> This *is* clearly documented here in the kernel docs:
> linux/Documentation/networking/netconsole.txt

	It is?  It doesn't look that clear to me:

Backup:/etc/mdadm# cd linux/Documentation/networking/
-bash: cd: linux/Documentation/networking/: No such file or directory

> It takes a string configuration parameter "netconsole" in the
> following format:
> 
>   netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
> 
>     where
>          src-port      source for UDP packets (defaults to 6665)
>          src-ip        source IP to use (interface address)
>          dev           network interface (eth0)
>          tgt-port      port for logging agent (6666)
>          tgt-ip        IP address for logging agent
>          tgt-macaddr   ethernet MAC address for logging agent (broadcast)

	Thanks.  I'll give it a shot.
 

RE: Kernel 2.6.32-5-amd64 fails to boot.

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Leslie Rhorer
> Sent: Tuesday, February 01, 2011 2:24 PM
> To: 'Linux RAID'
> Subject: Kernel 2.6.32-5-amd64 fails to boot.
> 
> 	I have been attempting to upgrade a Debian "Squeeze" Linux box from
> 2.6.32-3-amd64 to 2.6.32-5-amd64, but the upgrade is a non-starter.  GRUB2
> comes up just fine, but when I select the new kernel version, a number of
> announcements flash by too fast to seen.  I am not 100% certain, but I
> believe the initrd starts to load OK. Some text flies by far too quickly
> to
> be seen, but then an error pops up concerning an address space collision
> of
> some PCI device. Then it shows three errors for RAID devices md1. md2, and
> md3, saying they are already in use. Immediately thereafter the system
> shows
> errors concerning  the RAID targets being already in use, after which
> point
> the system complains it can't mount / (md2), /dev, /sys, or /proc (in that
> order) because the sources do not exist (if /dev/md2 does not exist, how
> can
> it be busy?).

	OK, I've made some progress with the help of one of the guys on
usenet, at least in terms of getting to a point where I can do some
troubleshooting.  I went into the initrd configuration file and changed the
MODULES= line to "most", so that now the server recognizes the keyboard.

	For some reason, the system is not properly assembling the arrays,
or perhaps more accurately is not starting them.  Now the first thing that
pops out at me is the fact under the old kernel the array members were
/dev/hdaX and /dev/hdbX, but under the new kernel they are /dev/sdkX and
/dev/sdlX (sdl and sdm when the usb flash drive is inserted as detailed
below).  I don't see why that should cause any issues, however.  The arrays
all show to be there, but all four report to be inactive.  If I manually
stop and then re-assemble the drives, I can mount them, but otherwise, not.
Before stopping the arrays, the command `mdadm --assemble --scan` reports
the arrays in use, but after stopping any of them, the command works on any
stopped array, and I am then able to mount the array.

	In the initrd, the mdadm.conf file is simply:

DEVICE partitions
HOMEHOST <system>
ARRAY /dev/md0 level=raid6 num-devices=10 metadata=01.2 name=Backup:0
UUID=431244d6:45d9635a:e88b3de5:92f30255
ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90
UUID=4cde286c:0687556a:4d9996dd:dd23e701
ARRAY /dev/md2 level=raid1 num-devices=2 metadata=01.2 name=Backup:2
UUID=d45ff663:9e53774c:6fcf9968:21692025
ARRAY /dev/md3 level=raid1 num-devices=2 metadata=01.2 name=Backup:3
UUID=51d22c47:10f58974:0b27ef04:5609d357

	...which should work.  Indeed, it is not a matter of mdadm not
finding the arrays.  It just does not seem to be starting them correctly.

	Here is the output of (most of) dmesg:

[    4.228353] ata4.04: hard resetting link
[    4.564319] ata4.04: SATA link down (SStatus 0 SControl 320)
[    4.564375] ata4.05: hard resetting link
[    4.900300] ata4.05: SATA link up 1.5 Gbps (SStatus 113 SControl 320)
[    4.939162] ata4.00: ATA-8: WDC WD15EADS-00R6B0, 01.00A01, max UDMA/133
[    4.939164] ata4.00: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    4.943192] ata4.00: configured for UDMA/100
[    4.946435] ata4.01: ATA-8: WDC WD15EADS-00R6B0, 01.00A01, max UDMA/133
[    4.946438] ata4.01: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    4.950727] ata4.01: configured for UDMA/100
[    4.955783] ata4.02: ATA-8: WDC WD15EADS-00R6B0, 01.00A01, max UDMA/133
[    4.955785] ata4.02: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    4.961792] ata4.02: configured for UDMA/100
[    4.963501] ata4.03: ATA-8: WDC WD15EADS-00P8B0, 01.00A01, max UDMA/133
[    4.963503] ata4.03: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    4.965305] ata4.03: configured for UDMA/100
[    4.965365] ata4: EH complete
[    4.965449] scsi 3:0:0:0: Direct-Access     ATA      WDC WD15EADS-00R
01.0 PQ: 0 ANSI: 5
[    4.965585] scsi 3:1:0:0: Direct-Access     ATA      WDC WD15EADS-00R
01.0 PQ: 0 ANSI: 5
[    4.965707] scsi 3:2:0:0: Direct-Access     ATA      WDC WD15EADS-00R
01.0 PQ: 0 ANSI: 5
[    4.965832] scsi 3:3:0:0: Direct-Access     ATA      WDC WD15EADS-00P
01.0 PQ: 0 ANSI: 5
[    6.380157] usb-storage: device scan complete
[    6.380626] scsi 13:0:0:0: Direct-Access     SanDisk  SanDisk Cruzer
8.02 PQ: 0 ANSI: 0 CCS
[    6.380996] scsi 13:0:0:1: CD-ROM            SanDisk  SanDisk Cruzer
8.02 PQ: 0 ANSI: 0
[    7.076030] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 0)
[    7.076258] ata5.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports,
feat 0x1/0x9
[    7.092219] ata5.00: hard resetting link
[    7.428322] ata5.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[    7.428352] ata5.01: hard resetting link
[    7.764293] ata5.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    7.764320] ata5.02: hard resetting link
[    8.100323] ata5.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    8.100353] ata5.03: hard resetting link
[    8.436319] ata5.03: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    8.436348] ata5.04: hard resetting link
[    8.772318] ata5.04: SATA link down (SStatus 0 SControl 320)
[    8.772374] ata5.05: hard resetting link
[    9.108293] ata5.05: SATA link up 1.5 Gbps (SStatus 113 SControl 320)
[    9.110465] ata5.00: ATA-8: WDC WD15EARS-00Z5B1, 80.00A80, max UDMA/133
[    9.110468] ata5.00: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    9.113055] ata5.00: configured for UDMA/100
[    9.115434] ata5.01: ATA-8: WDC WD15EARS-00Z5B1, 80.00A80, max UDMA/133
[    9.115436] ata5.01: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    9.117420] ata5.01: configured for UDMA/100
[    9.119999] ata5.02: ATA-8: WDC WD15EARS-00Z5B1, 80.00A80, max UDMA/133
[    9.120001] ata5.02: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    9.121969] ata5.02: configured for UDMA/100
[    9.124597] ata5.03: ATA-8: WDC WD15EADS-00P8B0, 01.00A01, max UDMA/133
[    9.124600] ata5.03: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    9.127144] ata5.03: configured for UDMA/100
[    9.127204] ata5: EH complete
[    9.127291] scsi 4:0:0:0: Direct-Access     ATA      WDC WD15EARS-00Z
80.0 PQ: 0 ANSI: 5
[    9.127469] scsi 4:1:0:0: Direct-Access     ATA      WDC WD15EARS-00Z
80.0 PQ: 0 ANSI: 5
[    9.127593] scsi 4:2:0:0: Direct-Access     ATA      WDC WD15EARS-00Z
80.0 PQ: 0 ANSI: 5
[    9.127705] scsi 4:3:0:0: Direct-Access     ATA      WDC WD15EADS-00P
01.0 PQ: 0 ANSI: 5
[   11.252029] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 0)
[   11.252255] ata6.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports,
feat 0x1/0x9
[   11.268197] ata6.00: hard resetting link
[   11.604322] ata6.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[   11.604353] ata6.01: hard resetting link
[   11.940321] ata6.01: SATA link down (SStatus 0 SControl 320)
[   11.940378] ata6.02: hard resetting link
[   12.276293] ata6.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[   12.276320] ata6.03: hard resetting link
[   12.612321] ata6.03: SATA link down (SStatus 0 SControl 320)
[   12.612377] ata6.04: hard resetting link
[   12.948321] ata6.04: SATA link down (SStatus 0 SControl 320)
[   12.948378] ata6.05: hard resetting link
[   13.284318] ata6.05: SATA link up 1.5 Gbps (SStatus 113 SControl 320)
[   13.287633] ata6.00: ATA-8: WDC WD15EADS-00P8B0, 01.00A01, max UDMA/133
[   13.287635] ata6.00: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[   13.290246] ata6.00: configured for UDMA/100
[   13.292122] ata6.02: ATA-8: WDC WD15EARS-00Z5B1, 80.00A80, max UDMA/133
[   13.292124] ata6.02: 2930277168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[   13.294114] ata6.02: configured for UDMA/100
[   13.294174] ata6: EH complete
[   13.294229] scsi 5:0:0:0: Direct-Access     ATA      WDC WD15EADS-00P
01.0 PQ: 0 ANSI: 5
[   13.294344] scsi 5:2:0:0: Direct-Access     ATA      WDC WD15EARS-00Z
80.0 PQ: 0 ANSI: 5
[   23.324019] ata7: softreset failed (timeout)
[   33.340017] ata7: softreset failed (timeout)
[   68.364019] ata7: softreset failed (timeout)
[   68.364053] ata7: limiting SATA link speed to 1.5 Gbps
[   73.388016] ata7: softreset failed (timeout)
[   73.388049] ata7: reset failed, giving up
[   73.388183] scsi 11:0:0:0: Direct-Access     ATA      ST3500641A
3.AA PQ: 0 ANSI: 5
[   73.388293] scsi 11:0:1:0: Direct-Access     ATA      ST3500641A
3.AA PQ: 0 ANSI: 5
[   73.562936] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   73.563158] r8169 0000:05:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ
18
[   73.563213] r8169 0000:05:00.0: setting latency timer to 64
[   73.563248]   alloc irq_desc for 28 on node 0
[   73.563250]   alloc kstat_irqs on node 0
[   73.563269] r8169 0000:05:00.0: irq 28 for MSI/MSI-X
[   73.567928] eth0: RTL8168c/8111c at 0xffffc90010ad0000,
00:22:15:59:63:1f, XID 1c4000c0 IRQ 28
[   73.581890] firewire_ohci 0000:04:00.0: PCI INT A -> GSI 17 (level, low)
-> IRQ 17
[   73.581898] firewire_ohci 0000:04:00.0: setting latency timer to 64
[   73.595412] sd 3:0:0:0: [sda] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.595470] sd 3:0:0:0: [sda] Write Protect is off
[   73.595473] sd 3:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   73.595483] sd 3:1:0:0: [sdb] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.595495] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.595533] sd 3:1:0:0: [sdb] Write Protect is off
[   73.595536] sd 3:1:0:0: [sdb] Mode Sense: 00 3a 00 00
[   73.595555] sd 3:1:0:0: [sdb] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.595651]  sda:
[   73.595737]  sdb:
[   73.595880] sd 3:2:0:0: [sdc] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.595923] sd 3:2:0:0: [sdc] Write Protect is off
[   73.595925] sd 3:2:0:0: [sdc] Mode Sense: 00 3a 00 00
[   73.595943] sd 3:2:0:0: [sdc] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.596044]  sdc:
[   73.596784] sd 3:3:0:0: [sdd] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.596853] sd 3:3:0:0: [sdd] Write Protect is off
[   73.596855] sd 3:3:0:0: [sdd] Mode Sense: 00 3a 00 00
[   73.596875] sd 3:3:0:0: [sdd] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.597002]  sdd:
[   73.597851] sd 13:0:0:0: [sde] 7856127 512-byte logical blocks: (4.02
GB/3.74 GiB)
[   73.598390] sd 13:0:0:0: [sde] Write Protect is off
[   73.598393] sd 13:0:0:0: [sde] Mode Sense: 45 00 00 08
[   73.598395] sd 13:0:0:0: [sde] Assuming drive cache: write through
[   73.598590] sd 4:0:0:0: [sdf] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.598630] sd 4:0:0:0: [sdf] Write Protect is off
[   73.598633] sd 4:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[   73.598714] sd 4:0:0:0: [sdf] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.598832]  sdf:
[   73.598906] sd 4:1:0:0: [sdg] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.598944] sd 4:1:0:0: [sdg] Write Protect is off
[   73.598946] sd 4:1:0:0: [sdg] Mode Sense: 00 3a 00 00
[   73.598962] sd 4:1:0:0: [sdg] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.599062]  sdg:
[   73.599116] sd 4:2:0:0: [sdh] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.599169] sd 4:2:0:0: [sdh] Write Protect is off
[   73.599171] sd 4:2:0:0: [sdh] Mode Sense: 00 3a 00 00
[   73.599187] sd 4:2:0:0: [sdh] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.599286]  sdh:
[   73.599333] sd 4:3:0:0: [sdi] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.599370] sd 4:3:0:0: [sdi] Write Protect is off
[   73.599372] sd 4:3:0:0: [sdi] Mode Sense: 00 3a 00 00
[   73.599394] sd 4:3:0:0: [sdi] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.599485]  sdi:
[   73.599548] sd 5:0:0:0: [sdj] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.599588] sd 5:0:0:0: [sdj] Write Protect is off
[   73.599590] sd 5:0:0:0: [sdj] Mode Sense: 00 3a 00 00
[   73.599607] sd 5:0:0:0: [sdj] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.599708]  sdj:
[   73.599751] sd 5:2:0:0: [sdk] 2930277168 512-byte logical blocks: (1.50
TB/1.36 TiB)
[   73.599795] sd 5:2:0:0: [sdk] Write Protect is off
[   73.599797] sd 5:2:0:0: [sdk] Mode Sense: 00 3a 00 00
[   73.599814] sd 5:2:0:0: [sdk] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.599918]  sdk:
[   73.599988] sd 11:0:1:0: [sdm] 976773168 512-byte logical blocks: (500
GB/465 GiB)
[   73.600045] sd 11:0:0:0: [sdl] 976773168 512-byte logical blocks: (500
GB/465 GiB)
[   73.600086] sd 11:0:0:0: [sdl] Write Protect is off
[   73.600088] sd 11:0:0:0: [sdl] Mode Sense: 00 3a 00 00
[   73.600107] sd 11:0:0:0: [sdl] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.600165] sd 11:0:1:0: [sdm] Write Protect is off
[   73.600167] sd 11:0:1:0: [sdm] Mode Sense: 00 3a 00 00
[   73.600193] sd 11:0:1:0: [sdm] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[   73.600265] sd 13:0:0:0: [sde] Assuming drive cache: write through
[   73.600300]  sde:
[   73.600373]  sdl:sr0: scsi3-mmc drive: 48x/48x cd/rw xa/form2 cdda tray
[   73.600604] Uniform CD-ROM driver Revision: 3.20
[   73.600699] sr 2:0:1:0: Attached scsi CD-ROM sr0
[   73.603799]  sde1
[   73.603973] usbcore: registered new interface driver hiddev
[   73.608745] sr 2:0:1:0: Attached scsi generic sg0 type 5
[   73.608779] sd 3:0:0:0: Attached scsi generic sg1 type 0
[   73.608808] sd 3:1:0:0: Attached scsi generic sg2 type 0
[   73.608846] sd 3:2:0:0: Attached scsi generic sg3 type 0
[   73.608884] sr1: scsi3-mmc drive: 48x/48x tray
[   73.608994] sd 3:3:0:0: Attached scsi generic sg4 type 0
[   73.609060] sd 13:0:0:0: Attached scsi generic sg5 type 0
[   73.609192] sr 13:0:0:1: Attached scsi CD-ROM sr1
[   73.609410] sr 13:0:0:1: Attached scsi generic sg6 type 5
[   73.609479] sd 4:0:0:0: Attached scsi generic sg7 type 0
[   73.609542] sd 13:0:0:0: [sde] Assuming drive cache: write through
[   73.609548] sd 4:1:0:0: Attached scsi generic sg8 type 0
[   73.609580] sd 13:0:0:0: [sde] Attached SCSI removable disk
[   73.609651] sd 4:2:0:0: Attached scsi generic sg9 type 0
[   73.609718] sd 4:3:0:0: Attached scsi generic sg10 type 0
[   73.609776] sd 5:0:0:0: Attached scsi generic sg11 type 0
[   73.609833] sd 5:2:0:0: Attached scsi generic sg12 type 0
[   73.609897] sd 11:0:0:0: Attached scsi generic sg13 type 0
[   73.609956] sd 11:0:1:0: Attached scsi generic sg14 type 0
[   73.614471]  sdl1 sdl2 sdl3
[   73.635748]  sdm: sdm1 sdm2 sdm3
[   73.660020] firewire_ohci: Added fw-ohci device 0000:04:00.0, OHCI
version 1.10
[   73.672552] sd 11:0:0:0: [sdl] Attached SCSI disk
[   73.672739] sd 11:0:1:0: [sdm] Attached SCSI disk
[   73.996376]  unknown partition table
[   73.996582] sd 3:1:0:0: [sdb] Attached SCSI disk
[   74.058234]
[   74.058579] sd 5:2:0:0: [sdk] Attached SCSI disk
[   74.064289]
[   74.064451] sd 4:2:0:0: [sdh] Attached SCSI disk
[   74.073319]  unknown partition table
[   74.074050] sd 3:3:0:0: [sdd] Attached SCSI disk
[   74.074309]
[   74.074760] sd 4:0:0:0: [sdf] Attached SCSI disk
[   74.076343]
[   74.081240]
[   74.082740] sd 5:0:0:0: [sdj] Attached SCSI disk
[   74.088837] sd 4:1:0:0: [sdg] Attached SCSI disk
[   74.088979]  unknown partition table
[   74.089986] sd 4:3:0:0: [sdi] Attached SCSI disk
[   74.093092] md: bind<sdm1>
[   74.101862] md: bind<sdl2>
[   74.115438] md: bind<sdm3>
[   74.141257]  unknown partition table
[   74.141381]  unknown partition table
[   74.142712] sd 3:2:0:0: [sdc] Attached SCSI disk
[   74.142756] sd 3:0:0:0: [sda] Attached SCSI disk
[   74.143154] md: array md3 already has disks!
[   74.160601] firewire_core: created device fw0: GUID 001e8c000162dcce,
S400
[   74.161513] md: bind<sdi>
[   74.665772] generic-usb 0003:051D:0002.0001: hiddev0,hidraw0: USB HID
v1.10 Device [American Power Conversion Back-UPS RS 1000 FW:7.g7 .D USB
FW:g7 ] on usb-0000:00:12.1-3/input0
[   74.671912] input: Key Tronic Keytronic USB Keyboard as
/devices/pci0000:00/0000:00:12.1/usb4/4-1/4-1.1/4-1.1:1.0/input/input1
[   74.671961] generic-usb 0003:03F9:0100.0002: input,hidraw1: USB HID v1.10
Keyboard [Key Tronic Keytronic USB Keyboard] on usb-0000:00:12.1-1.1/input0
[   74.676854] input: USB OpticalWheel Mouse as
/devices/pci0000:00/0000:00:12.1/usb4/4-1/4-1.2/4-1.2:1.0/input/input2
[   74.676910] generic-usb 0003:04FC:0015.0003: input,hidraw2: USB HID v1.10
Mouse [USB OpticalWheel Mouse] on usb-0000:00:12.1-1.2/input0
[   74.681828] input: Justcom Technology USB KVM Switch as
/devices/pci0000:00/0000:00:12.1/usb4/4-1/4-1.3/4-1.3:1.0/input/input3
[   74.681865] generic-usb 0003:06F2:0011.0004: input,hidraw3: USB HID v1.10
Keyboard [Justcom Technology USB KVM Switch] on usb-0000:00:12.1-1.3/input0
[   74.686841] input: Justcom Technology USB KVM Switch as
/devices/pci0000:00/0000:00:12.1/usb4/4-1/4-1.3/4-1.3:1.1/input/input4
[   74.686919] generic-usb 0003:06F2:0011.0005: input,hidraw4: USB HID v1.10
Mouse [Justcom Technology USB KVM Switch] on usb-0000:00:12.1-1.3/input1
[   74.686947] usbcore: registered new interface driver usbhid
[   74.686949] usbhid: v2.6:USB HID core driver
[   74.777699] md: raid1 personality registered for level 1
[   74.786167] async_tx: api initialized (async)
[   74.786639] xor: automatically using best checksumming function:
generic_sse
[   74.804011]    generic_sse:  8104.000 MB/sec
[   74.804013] xor: using function: generic_sse (8104.000 MB/sec)
[   74.872020] raid6: int64x1   2458 MB/s
[   74.940025] raid6: int64x2   3204 MB/s
[   75.008031] raid6: int64x4   2491 MB/s
[   75.076033] raid6: int64x8   2256 MB/s
[   75.144018] raid6: sse2x1    1299 MB/s
[   75.212039] raid6: sse2x2    2044 MB/s
[   75.280023] raid6: sse2x4    2774 MB/s
[   75.280024] raid6: using algorithm sse2x4 (2774 MB/s)
[   75.284431] md: raid6 personality registered for level 6
[   75.284434] md: raid5 personality registered for level 5
[   75.284436] md: raid4 personality registered for level 4
[   75.320682] device-mapper: uevent: version 1.0.3
[   75.320837] device-mapper: ioctl: 4.15.0-ioctl (2009-04-01) initialised:
dm-devel@redhat.com

RE: Kernel 2.6.32-5-amd64 fails to boot.

[Leslie Rhorer]

	It's very striking that the old kernel also fails to properly start
the /dev/md0 array.  Under the old kernel, the udev drive names for the PATA
drives are /dev/hda and /dev/hdb.  The arrays assembled from these members
get properly assembled in in the old kernel.  According to dmesg, they seem
to be assembled, stopped, and then assembled again.  The /dev/md0 array,
however, has members that are all named /dev/sdX - both kernels are
apparently classifying the SATA drives as SCSI (as has been the case for a
while).  After booting the old kernel, the /dev/md0 array must be started
and re-assembled to get it working, but under the new kernel, all four
arrays must be stopped and re-assembled, the lack of the /dev/md2 rootfs
system halting the boot.  For some reason, neither kernel seems to be
stopping and re-assembling the arrays built from "SCSI" members, but the old
kernel is classifying the PATA members as IDE, rather than SCSI.  Note also
the lack of any messages from the new kernel saying the arrays are being
made active, and the relative positions where the RAID personalities are
registered.

	I really need to get this resolved, because this is holding up a
number of support issues, and preventing me from upgrading the kernel and
distro on the main server, at all.

	I'm adding the Debian mdadm maintainers to the e-mail thread, hoping
they may have some input.  I'm also adding Martin Kraft, who has been
graciously helping me over on usenet.

	Dmesg | grep md from the old kernel:

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-2.6.32-3-amd64
root=/dev/md2 ro quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.32-3-amd64
root=/dev/md2 ro quiet
[    2.411210] md: raid1 personality registered for level 1
[    2.940328] md: raid6 personality registered for level 6
[    2.940331] md: raid5 personality registered for level 5
[    2.940332] md: raid4 personality registered for level 4
[    2.948514] md: md1 stopped.
[    2.951701] md: bind<hdb1>
[    2.951933] md: bind<hda1>
[    2.953206] raid1: raid set md1 active with 2 out of 2 mirrors
[    2.957738] md1: bitmap initialized from disk: read 1/1 pages, set 0 bits
[    2.957741] created bitmap (1 pages) for device md1
[    3.000642] md1: detected capacity change from 0 to 6292176896
[    3.001942]  md1: unknown partition table
<snip>

	Compare that with the new kernel:
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-2.6.32-5-amd64
root=/dev/md2 ro quiet
[    0.748091] usb usb1: Manufacturer: Linux 2.6.32-5-amd64 ehci_hcd
[    0.748547] ata1: SATA max UDMA/133 port i16@0xec00 bmdma 0xe400 irq 20
[    0.748551] ata2: SATA max UDMA/133 port i16@0xe880 bmdma 0xe408 irq 20
[    0.748553] ata3: PATA max UDMA/133 port i16@0xe800 bmdma 0xe410 irq 20
[    0.806045] usb usb2: Manufacturer: Linux 2.6.32-5-amd64 ohci_hcd
[    0.807649] ata8: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xff00 irq
14
[    0.807652] ata9: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xff08 irq
15
[    0.820043] usb usb3: Manufacturer: Linux 2.6.32-5-amd64 ehci_hcd
[    0.882041] usb usb4: Manufacturer: Linux 2.6.32-5-amd64 ohci_hcd
[    0.938057] usb usb5: Manufacturer: Linux 2.6.32-5-amd64 ohci_hcd
[    0.994029] usb usb6: Manufacturer: Linux 2.6.32-5-amd64 ohci_hcd
[    1.050029] usb usb7: Manufacturer: Linux 2.6.32-5-amd64 ohci_hcd
[   16.140763] md: bind<sdk1>
[   16.150006] md: bind<sdl3>
[   16.170380] md: bind<sdl2>
[   16.199653] md: array md1 already has disks!
[   16.225273] md: bind<sdi>
[   16.349657] md: raid1 personality registered for level 1
<snip>