aic79xx driver - hotswap error

aic79xx driver - hotswap error

[Martin Zuziak]

Hello all

Hot-swapping doesn't seem to work with the aic79xx driver in kernel
2.6.17.9. Removing or adding a disk from/to a running system makes i/o
to any disk on the bus fail.

The machine is an IBM x346 server with a x86_64 cpu and a aic7902 scsi
controller.

A copy of the system log is here:
http://www.math.ku.dk/~zuziak/tmp/aic79xx_error_2.6.17.9.log

It shows the result of removing the third disk: the first disk (the only
one mounted) becomes inaccessible.

Kernel 2.6.15.7 seems to work but I have had no luck with newer kernels.

Has anyone seen hot-swapping work with the aic79xx driver in recent
kernels?

Sincerely,

Martin Zuziak <zuziak@math.ku.dk>

Re: aic79xx driver - hotswap error

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2576 bytes --]

On Mon, 2006-08-28 at 14:15 +0200, Martin Zuziak wrote:
> Hello all
> 
> Hot-swapping doesn't seem to work with the aic79xx driver in kernel
> 2.6.17.9. Removing or adding a disk from/to a running system makes i/o
> to any disk on the bus fail.
> 
> The machine is an IBM x346 server with a x86_64 cpu and a aic7902 scsi
> controller.
> 
> A copy of the system log is here:
> http://www.math.ku.dk/~zuziak/tmp/aic79xx_error_2.6.17.9.log
> 
> It shows the result of removing the third disk: the first disk (the only
> one mounted) becomes inaccessible.
> 
> Kernel 2.6.15.7 seems to work but I have had no luck with newer kernels.
> 
> Has anyone seen hot-swapping work with the aic79xx driver in recent
> kernels?

Are you sure your system is hot swap safe?  The whole log mess begins
with "someone reset channel A" which means the card detected a bus reset
but it didn't initiate the reset.  That's either going to be because
your system shouldn't be hot swap plugged and it triggered a spike on
the reset pin, or because your hot swap drive setup resets the bus on
unplug intentionally.  Knowing that would help.

So, the driver managed to get into the ahd_pause_and_flushwork()
function, probably while trying to queue the abort SCB, and while there
it detected an infinite loop and printed out the "Infinite interrupt
loop, INTSTAT = 8" message.  The INTSTAT value of 0x08 maps to SCSIINT,
so next you would look at the SCSIINT1 and SCSIINT2 registers to see
just *what* is causing the loop.  There you see SSTAT1[0x20]:(SCSIRSTI).
This tells us the driver is *still* getting a SCSI Reset In interrupt
from the card, even over 1 minute after you pulled the drive.  So, the
reason your SCSI bus hung is because everything on the bus is being
subjected to an infinite bus reset condition.  The cause of this
happening is likely either A) your bus isn't hot swap safe and you hot
swapped anyway, and in the process you disconnected the termination
power source or termination itself or just plain flaked other devices on
the bus out or B) something in your hot swap enclosure is broken and
throws an infinite bus reset when the drive is removed.  Either way,
this is not what I would call expected behavior from the aic79xx driver,
I suspect that it is innocent here and that the hardware is to blame.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: aic79xx driver - hotswap error

[Hannes Reinecke]

Doug Ledford wrote:
> On Mon, 2006-08-28 at 14:15 +0200, Martin Zuziak wrote:
>> Hello all
>>
>> Hot-swapping doesn't seem to work with the aic79xx driver in kernel
>> 2.6.17.9. Removing or adding a disk from/to a running system makes i/o
>> to any disk on the bus fail.
>>
>> The machine is an IBM x346 server with a x86_64 cpu and a aic7902 scsi
>> controller.
>>
>> A copy of the system log is here:
>> http://www.math.ku.dk/~zuziak/tmp/aic79xx_error_2.6.17.9.log
>>
>> It shows the result of removing the third disk: the first disk (the only
>> one mounted) becomes inaccessible.
>>
>> Kernel 2.6.15.7 seems to work but I have had no luck with newer kernels.
>>
>> Has anyone seen hot-swapping work with the aic79xx driver in recent
>> kernels?
> 
> Are you sure your system is hot swap safe?  The whole log mess begins
> with "someone reset channel A" which means the card detected a bus reset
> but it didn't initiate the reset.  That's either going to be because
> your system shouldn't be hot swap plugged and it triggered a spike on
> the reset pin, or because your hot swap drive setup resets the bus on
> unplug intentionally.  Knowing that would help.
> 
> So, the driver managed to get into the ahd_pause_and_flushwork()
> function, probably while trying to queue the abort SCB, and while there
> it detected an infinite loop and printed out the "Infinite interrupt
> loop, INTSTAT = 8" message.  The INTSTAT value of 0x08 maps to SCSIINT,
> so next you would look at the SCSIINT1 and SCSIINT2 registers to see
> just *what* is causing the loop.  There you see SSTAT1[0x20]:(SCSIRSTI).
> This tells us the driver is *still* getting a SCSI Reset In interrupt
> from the card, even over 1 minute after you pulled the drive.  So, the
> reason your SCSI bus hung is because everything on the bus is being
> subjected to an infinite bus reset condition.  The cause of this
> happening is likely either A) your bus isn't hot swap safe and you hot
> swapped anyway, and in the process you disconnected the termination
> power source or termination itself or just plain flaked other devices on
> the bus out or B) something in your hot swap enclosure is broken and
> throws an infinite bus reset when the drive is removed.  Either way,
> this is not what I would call expected behavior from the aic79xx driver,
> I suspect that it is innocent here and that the hardware is to blame.
> 
Thanks Doug.
I couldn't have phrased it better.

Hotswap does work, provided you don't do anything untoward.
Which translates as "might work but don't blame me if it doesn't".
And I would never ever claim that it is a supported feature.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke			hare@suse.de
SuSE Linux Products GmbH		S390 & zSeries
Maxfeldstraße 5				+49 911 74053 688
90409 Nürnberg				http://www.suse.de
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: aic79xx driver - hotswap error

[Martin Zuziak]

On Tue, Aug 29, 2006 at 11:26:44AM +0200, Hannes Reinecke wrote:
> Doug Ledford wrote:
> >On Mon, 2006-08-28 at 14:15 +0200, Martin Zuziak wrote:
> >>Hello all
> >>
> >>Hot-swapping doesn't seem to work with the aic79xx driver in kernel
> >>2.6.17.9. Removing or adding a disk from/to a running system makes i/o
> >>to any disk on the bus fail.
> >>
> >>The machine is an IBM x346 server with a x86_64 cpu and a aic7902 scsi
> >>controller.
> >>
> >>A copy of the system log is here:
> >>http://www.math.ku.dk/~zuziak/tmp/aic79xx_error_2.6.17.9.log
> >>
> >>It shows the result of removing the third disk: the first disk (the only
> >>one mounted) becomes inaccessible.
> >>
> >>Kernel 2.6.15.7 seems to work but I have had no luck with newer kernels.
> >>
> >>Has anyone seen hot-swapping work with the aic79xx driver in recent
> >>kernels?
> >
> >Are you sure your system is hot swap safe?  The whole log mess begins
> >with "someone reset channel A" which means the card detected a bus reset
> >but it didn't initiate the reset.  That's either going to be because
> >your system shouldn't be hot swap plugged and it triggered a spike on
> >the reset pin, or because your hot swap drive setup resets the bus on
> >unplug intentionally.  Knowing that would help.
> >
> >So, the driver managed to get into the ahd_pause_and_flushwork()
> >function, probably while trying to queue the abort SCB, and while there
> >it detected an infinite loop and printed out the "Infinite interrupt
> >loop, INTSTAT = 8" message.  The INTSTAT value of 0x08 maps to SCSIINT,
> >so next you would look at the SCSIINT1 and SCSIINT2 registers to see
> >just *what* is causing the loop.  There you see SSTAT1[0x20]:(SCSIRSTI).
> >This tells us the driver is *still* getting a SCSI Reset In interrupt
> >from the card, even over 1 minute after you pulled the drive.  So, the
> >reason your SCSI bus hung is because everything on the bus is being
> >subjected to an infinite bus reset condition.  The cause of this
> >happening is likely either A) your bus isn't hot swap safe and you hot
> >swapped anyway, and in the process you disconnected the termination
> >power source or termination itself or just plain flaked other devices on
> >the bus out or B) something in your hot swap enclosure is broken and
> >throws an infinite bus reset when the drive is removed.  Either way,
> >this is not what I would call expected behavior from the aic79xx driver,
> >I suspect that it is innocent here and that the hardware is to blame.
> >
> Thanks Doug.
> I couldn't have phrased it better.
> 
> Hotswap does work, provided you don't do anything untoward.
> Which translates as "might work but don't blame me if it doesn't".
> And I would never ever claim that it is a supported feature.

Thank you both for your replies.

First off all the hardware (both server and disks) does support hot
swap. And I have tried on three different machines (same model) to rule
out a hardware fault.

I have never had any problems with hot swap before. And indeed hot swap
on these machines work with the 2.6.15.7 kernel. But something has
changed since 2.6.16 so it no longer works.

However if hot swap isn't supported then that's just the way it is.

Sincerely,

Martin Zuziak <zuziak@math.ku.dk>

Re: aic79xx driver - hotswap error

[Hannes Reinecke]

Martin Zuziak wrote:
> On Tue, Aug 29, 2006 at 11:26:44AM +0200, Hannes Reinecke wrote:
>> Doug Ledford wrote:
>>> On Mon, 2006-08-28 at 14:15 +0200, Martin Zuziak wrote:
>>>> Hello all
>>>>
>>>> Hot-swapping doesn't seem to work with the aic79xx driver in kernel
>>>> 2.6.17.9. Removing or adding a disk from/to a running system makes i/o
>>>> to any disk on the bus fail.
>>>>
>>>> The machine is an IBM x346 server with a x86_64 cpu and a aic7902 scsi
>>>> controller.
>>>>
>>>> A copy of the system log is here:
>>>> http://www.math.ku.dk/~zuziak/tmp/aic79xx_error_2.6.17.9.log
>>>>
>>>> It shows the result of removing the third disk: the first disk (the only
>>>> one mounted) becomes inaccessible.
>>>>
>>>> Kernel 2.6.15.7 seems to work but I have had no luck with newer kernels.
>>>>
>>>> Has anyone seen hot-swapping work with the aic79xx driver in recent
>>>> kernels?
>>> Are you sure your system is hot swap safe?  The whole log mess begins
>>> with "someone reset channel A" which means the card detected a bus reset
>>> but it didn't initiate the reset.  That's either going to be because
>>> your system shouldn't be hot swap plugged and it triggered a spike on
>>> the reset pin, or because your hot swap drive setup resets the bus on
>>> unplug intentionally.  Knowing that would help.
>>>
>>> So, the driver managed to get into the ahd_pause_and_flushwork()
>>> function, probably while trying to queue the abort SCB, and while there
>>> it detected an infinite loop and printed out the "Infinite interrupt
>>> loop, INTSTAT = 8" message.  The INTSTAT value of 0x08 maps to SCSIINT,
>>> so next you would look at the SCSIINT1 and SCSIINT2 registers to see
>>> just *what* is causing the loop.  There you see SSTAT1[0x20]:(SCSIRSTI).
>>> This tells us the driver is *still* getting a SCSI Reset In interrupt
>> >from the card, even over 1 minute after you pulled the drive.  So, the
>>> reason your SCSI bus hung is because everything on the bus is being
>>> subjected to an infinite bus reset condition.  The cause of this
>>> happening is likely either A) your bus isn't hot swap safe and you hot
>>> swapped anyway, and in the process you disconnected the termination
>>> power source or termination itself or just plain flaked other devices on
>>> the bus out or B) something in your hot swap enclosure is broken and
>>> throws an infinite bus reset when the drive is removed.  Either way,
>>> this is not what I would call expected behavior from the aic79xx driver,
>>> I suspect that it is innocent here and that the hardware is to blame.
>>>
>> Thanks Doug.
>> I couldn't have phrased it better.
>>
>> Hotswap does work, provided you don't do anything untoward.
>> Which translates as "might work but don't blame me if it doesn't".
>> And I would never ever claim that it is a supported feature.
> 
> Thank you both for your replies.
> 
> First off all the hardware (both server and disks) does support hot
> swap. And I have tried on three different machines (same model) to rule
> out a hardware fault.
> 
> I have never had any problems with hot swap before. And indeed hot swap
> on these machines work with the 2.6.15.7 kernel. But something has
> changed since 2.6.16 so it no longer works.
> 
Hmm. But you should see something in the logs for 2.6.15, too.
Ideally some aic79xx stack dump. Can you dig it out?
If a aic79xx state dump is available for 2.6.15 we might be able to 
figure out the difference and fix the driver if possible.

But saying that 'hotswap is supported' simply doesn't cut it.
The spec actually allows you to claim 'hotswap is supported' when you 
have to power-cycle the entire cabinet.
Or the HBA.

Hmm. I wonder whether this is related to the infamous bus polling 
mechanism the original driver did ...
Anyway, can you get the logs?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke			hare@suse.de
SuSE Linux Products GmbH		S390 & zSeries
Maxfeldstraße 5				+49 911 74053 688
90409 Nürnberg				http://www.suse.de
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: aic79xx driver - hotswap error

[Martin Zuziak]

> >Thank you both for your replies.
> >
> >First off all the hardware (both server and disks) does support hot
> >swap. And I have tried on three different machines (same model) to rule
> >out a hardware fault.
> >
> >I have never had any problems with hot swap before. And indeed hot swap
> >on these machines work with the 2.6.15.7 kernel. But something has
> >changed since 2.6.16 so it no longer works.
> >
> Hmm. But you should see something in the logs for 2.6.15, too.
> Ideally some aic79xx stack dump. Can you dig it out?
> If a aic79xx state dump is available for 2.6.15 we might be able to 
> figure out the difference and fix the driver if possible.

No stack dump is logged and I don't know how to force one.

After echoing "scsi remove-single-device 1 0 2 0" to /proc/scsi/scsi
nothing is logged.

After the disk is physically removed from its bay this is logged:
Aug 31 12:45:58 nyimf kernel: scsi1: Someone reset channel A

And when it's re-inserted:
Aug 31 12:46:24 nyimf kernel: scsi1: Someone reset channel A

After echoing "scsi add-single-device 1 0 2 0" to /proc/scsi/scsi the
disk is detected and ready to use:
Aug 31 12:46:45 nyimf kernel:   Vendor: IBM-ESXS  Model: VPR036C3-ETS10FN  Rev: S3C0
Aug 31 12:46:45 nyimf kernel:   Type:   Direct-Access                      ANSI SCSI revision: 04
Aug 31 12:46:45 nyimf kernel:  target1:0:2: asynchronous.
Aug 31 12:46:45 nyimf kernel: scsi1:A:2:0: Tagged Queuing enabled.  Depth 32
Aug 31 12:46:45 nyimf kernel:  target1:0:2: Beginning Domain Validation
Aug 31 12:46:45 nyimf kernel:  target1:0:2: wide asynchronous.
Aug 31 12:46:45 nyimf kernel:  target1:0:2: FAST-160 WIDE SCSI 320.0 MB/s DT IU RDSTRM RTI WRFLOW PCOMP (6.25 ns, offset 127)
Aug 31 12:46:45 nyimf kernel:  target1:0:2: Domain Validation skipping write tests
Aug 31 12:46:45 nyimf kernel:  target1:0:2: Ending Domain Validation
Aug 31 12:46:49 nyimf udevd-event[20583]: wait_for_sysfs: waiting for '/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:07.1/host1/target1:0:2/1:0:2:0/bus' failed
Aug 31 12:46:52 nyimf udevd-event[20583]: wait_for_sysfs: waiting for '/sys/devices/pci0000:00/0000:00:06.0/0000:07:00.0/0000:08:07.1/host1/target1:0:2/1:0:2:0/ioerr_cnt' failed
Aug 31 12:46:54 nyimf kernel: sdc: Spinning up disk............ready
Aug 31 12:46:54 nyimf kernel: SCSI device sdc: 71096640 512-byte hdwr sectors (36401 MB)
Aug 31 12:46:54 nyimf kernel: SCSI device sdc: drive cache: write through
Aug 31 12:46:54 nyimf kernel: SCSI device sdc: 71096640 512-byte hdwr sectors (36401 MB)
Aug 31 12:46:54 nyimf scsi_id[20586]: scsi_id: unable to access parent device of '/block/sdc'
Aug 31 12:46:54 nyimf kernel: SCSI device sdc: drive cache: write through
Aug 31 12:46:54 nyimf scsi_id[20587]: scsi_id: unable to access parent device of '/block/sdc'
Aug 31 12:46:54 nyimf kernel:  sdc:
Aug 31 12:46:54 nyimf kernel: sd 1:0:2:0: Attached scsi disk sdc
Aug 31 12:46:54 nyimf kernel: sd 1:0:2:0: Attached scsi generic sg2 type 0

> But saying that 'hotswap is supported' simply doesn't cut it.
> The spec actually allows you to claim 'hotswap is supported' when you 
> have to power-cycle the entire cabinet.
> Or the HBA.

I don't think IBM designed the server that way. But I must admit to
knowing very little about scsi. It just seemed suspicious that hot
swapping stopped working with 2.6.16 when the changelog showed a lot of
changes to the aic79xx driver.

Sincerely,

Martin Zuziak <zuziak@math.ku.dk>