Re:

[Laurence Oberman <loberman@redhat.com>]

On Wed, 2026-09-30 at 10:09 +0200, Pluess, Tobias wrote:
> Hello Laurence,
> sorry for my late reply.
> Thanks for looking into this. I will setup my system accordingly to
> be
> able to boot older versions. It will take a few days until I have
> time
> but I will report ASAP my findings.
> 
> From inspecting my kernel update logs,
> 
> zgrep -E "status installed .*kernel-" /var/log/dpkg.log.*
> 
> I see that around that time when the error started occuring was when
> updating from 6.14 to 6.17 and later. So I would guess the
> corresponding changes must have happened around that point. I'm not
> 100% sure, but I will try booting the older kernels and will report
> my
> findings.
> 
> Thanks!
> greetings Tobias
> 
> On Sat, Sep 26, 2026 at 2:30 PM Laurence Oberman
> <loberman@redhat.com> wrote:
> > 
> > On Fri, 2026-09-25 at 21:00 +0200, Pluess, Tobias wrote:
> > > Hi,
> > > 
> > > I am seeing a problem with the mpt3sas driver when using an
> > > LSI/Broadcom SAS9207-8e connected to an external tape drive (HP
> > > Ultrium 6650).
> > > This setup worked without issues with older kernels. I cannot
> > > 100%
> > > sure identify at which date the problem started to occur, but I
> > > would
> > > say it started to occur around April or May this year.
> > > 
> > > What happens:
> > > a) the external tape drive is connected to the HBA using a SAS
> > > 8088
> > > cable.
> > > b) the tape drive is switched on and is detected normally and
> > > works
> > > as
> > > expected, backups can be made.
> > > c) after the backup finishes, the tape is ejected and the drive
> > > is
> > > idle, it is switched off.
> > > 
> > > Here begins the interesting story. In the dmesg, I see the
> > > following
> > > errors:
> > > 
> > > [Fri Sep 25 20:02:40 2026] mpt2sas_cm6: detecting:
> > > handle(0x0009),
> > >                            sas_address(0x50014380353cba70),
> > > phy(3)
> > > [Fri Sep 25 20:02:40 2026] mpt2sas_cm6: REPORT_LUNS:
> > > handle(0x0009),
> > >                            retries(0)
> > > [Fri Sep 25 20:02:40 2026] TEST_UNIT_READY: handle(0x0009) lun(0)
> > > [Fri Sep 25 20:02:41 2026] mpt2sas_cm6: detecting:
> > > handle(0x0009),
> > >                            sas_address(0x50014380353cba70),
> > > phy(3)
> > > [Fri Sep 25 20:02:41 2026] mpt2sas_cm6: REPORT_LUNS:
> > > handle(0x0009),
> > >                            retries(0)
> > > [Fri Sep 25 20:02:41 2026] TEST_UNIT_READY: handle(0x0009) lun(0)
> > > [Fri Sep 25 20:02:41 2026] START_UNIT: handle(0x0009), lun(0)
> > > [Fri Sep 25 20:02:45 2026] TEST_UNIT_READY: handle(0x0009) lun(0)
> > > [Fri Sep 25 20:02:46 2026] mpt2sas_cm6: detecting:
> > > handle(0x0009),
> > >                            sas_address(0x50014380353cba70),
> > > phy(3)
> > > [Fri Sep 25 20:02:46 2026] mpt2sas_cm6: REPORT_LUNS:
> > > handle(0x0009),
> > >                            retries(0)
> > > [Fri Sep 25 20:02:46 2026] TEST_UNIT_READY: handle(0x0009) lun(0)
> > > [Fri Sep 25 20:02:46 2026] START_UNIT: handle(0x0009), lun(0)
> > > [Fri Sep 25 20:02:49 2026] TEST_UNIT_READY: handle(0x0009) lun(0)
> > > [Fri Sep 25 20:02:52 2026] mpt2sas_cm6: log_info(0x31130000):
> > > originator(PL), code(0x13), sub_code(0x0000)
> > > [Fri Sep 25 20:02:53 2026] mpt2sas_cm6: handle(0x0009),
> > > ioc_status(0x0022) failure at
> > > drivers/scsi/mpt3sas/mpt3sas_transport.c:228/_transport_set_ident
> > > ify(
> > > )!
> > > [Fri Sep 25 20:02:53 2026] mpt2sas_cm6: failure at
> > > drivers/scsi/mpt3sas/mpt3sas_scsih.c:8407/_scsih_add_device()!
> > > [Fri Sep 25 20:02:54 2026] mpt2sas_cm6: handle(0x0009),
> > > ioc_status(0x0022) failure at
> > > drivers/scsi/mpt3sas/mpt3sas_transport.c:228/_transport_set_ident
> > > ify(
> > > )!
> > > [Fri Sep 25 20:02:54 2026] mpt2sas_cm6: failure at
> > > drivers/scsi/mpt3sas/mpt3sas_scsih.c:8407/_scsih_add_device()!
> > > [Fri Sep 25 20:02:55 2026] mpt2sas_cm6: handle(0x0009),
> > > ioc_status(0x0022) failure at
> > > drivers/scsi/mpt3sas/mpt3sas_transport.c:228/_transport_set_ident
> > > ify(
> > > )!
> > > 
> > > Here, the last 2 lines are printed to dmesg approximately every
> > > second, ad infinitum until the system is rebooted; the error
> > > never
> > > stops. I am certain this did not occur with older versions of
> > > mpt3sas,
> > > but as I said, I cannot pinpoint exactly when the change
> > > happened.
> > > 
> > > Powering up the tape drive again fixes the error, but as soon as
> > > the
> > > tape drive is switched off again, the error reappears.
> > > 
> > > However, the error doesn't occur every time the drive is switched
> > > off.
> > > Sometimes the mpt3sas driver seems to be happy and no errors are
> > > reported. I think it has to do with some kind of timing, i.e. in
> > > which
> > > internal state the mpt3sas driver is when the external drive is
> > > disconnected.
> > > 
> > > Hardware:
> > > * Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-
> > > Express
> > > Fusion-MPT SAS-2 (rev 05)
> > > * external tape drive HP Ultrium 6650
> > > * Linux pve0 7.0.14-12-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-12
> > > (2026-08-11T11:05Z) x86_64 GNU/Linux
> > > * mpt3sas
> > > 
> > > 
> > > I wonder why this problem occurs and how it could be prevented. I
> > > have
> > > to say, my hot swap HDDs are connected to
> > > 
> > > Serial Attached SCSI controller: Broadcom / LSI SAS3416 Fusion-
> > > MPT
> > > Tri-Mode I/O Controller Chip (IOC) (rev 01)
> > > 
> > > and here, I never have the issue, it only appears with the 9207-
> > > 8e.
> > > 
> > > I made my own "fix" to stop the driver from filling up my dmesg
> > > with
> > > errors; using a little bash script with these commands
> > > 
> > > echo "0000:51:00.0" > /sys/bus/pci/drivers/mpt3sas/unbind
> > > echo "0000:51:00.0" > /sys/bus/pci/drivers/mpt3sas/bind
> > > 
> > > resets the driver and it then stops printing errors. However I
> > > think
> > > this is not a true solution to the problem, it just fixes the
> > > symptom.
> > > 
> > > 
> > > Any ideas how to proceed?
> > > Unfortunately I don't quickly have another system handy where I
> > > could
> > > test older versions of the driver.
> > > 
> > > Thanks,
> > > Greetings
> > > Tobias
> > > 
> > 
> > Hello
> > Any chance you can boot an earlier set of kernels to try narrow
> > down
> > when this started. I don't have the older hardware to reproduce in
> > our
> > lab here. I will do some code inspection in the meantime to try
> > figure
> > out what triggers this and come up with some ideas.
> > 
> > Would be good to know which version older kernel was not seeing
> > this.
> > Thanks
> > Laurence
> > 
Hi Tobias,
Thanks, that version range helps. From code inspection, one possibly
relevant change between 6.14 and 6.17 is:
37c4e72b0651 scsi: Fix sas_user_scan() to handle wildcard and multi-
channel scans
That changed SAS wildcard scan behavior so a scan may now cover more
channels than before. It may only be exposing an existing mpt3sas
stale-handle issue, but it could explain why the problem became visible
now.

Could you please test these if possible:
v6.14
v6.15
v6.16
v6.17

The key thing is to find the first version where powering off the tape
drive starts the repeated _transport_set_identify() /
_scsih_add_device() messages.

Also, while the messages are repeating, could you check whether
anything is repeatedly rescanning SCSI hosts?
grep -R "host.*/scan|/scan" /etc /usr/lib/systemd /lib/systemd
/etc/udev /lib/udev 2>/dev/null

If you are able to build/test a kernel with 37c4e72b0651 reverted, that
would also be very useful.

My current theory is that after the tape drive is powered off, the
SAS2308/mpt3sas path may still have a stale device handle, and a
wildcard scan keeps trying to add it again. The add then fails because
the firmware no longer returns valid identify data for that handle.

Thanks,
Laurence