Re: Mylex AcceleRAID 170 + myrb/myrs causing crash

Re: Mylex AcceleRAID 170 + myrb/myrs causing crash

[Mike Edwards]

Re-sending this as text/plain due to vger spam rules.

On Thu, Jul 13, 2023 at 3:21 PM Mike Edwards
<medwards@mobile.mirkwood.net> wrote:
>
> I spun up an old machine (with an even older Mylex AcceleRAID card, the 170 w/ a bios dated Jan 21, 2000 - yikes!) recently.  While this machine was running an old 4.7 kernel and booted fine, attempting to update it to a modern release of Debian with a 6.1 kernel caused the kernel to hang while booting, with a number of stuck tasks warnings, starting with udev-worker and including kworker kernel processes.
>
> During troubleshooting, I was able to identify the myrb/myrs drivers which replaced the old DAC960 driver (removed in commit 6956b956934f10c19eca2a1d44f50a3bee860531) as the culprit.  The last kernel to successfully boot on here is 4.19.x, while anything newer exhibits the same stuck processes - and indeed, blacklisting the myrb and myrs drivers allows 6.1 to boot on this machine.
>
> I know this card is functional, as I do have two drives attached to it, and both it and the drives work fine in 4.19 and older kernels, so the issue seems to be with the newer myrb/myrs drivers.  Is there a chance of fixing the current drivers, or, at worst, reintroducing the old deprecated DAC960 driver back into the kernel?  I'm not absolutely tied to using that driver, other than 'it just works' for this card.

Re: Mylex AcceleRAID 170 + myrb/myrs causing crash

[Hannes Reinecke]

On 7/13/23 21:21, Mike Edwards wrote:
> I spun up an old machine (with an even older Mylex AcceleRAID card, the 
> 170 w/ a bios dated Jan 21, 2000 - yikes!) recently.  While this machine 
> was running an old 4.7 kernel and booted fine, attempting to update it 
> to a modern release of Debian with a 6.1 kernel caused the kernel to 
> hang while booting, with a number of stuck tasks warnings, starting with 
> udev-worker and including kworker kernel processes.
> 
> During troubleshooting, I was able to identify the myrb/myrs drivers 
> which replaced the old DAC960 driver (removed in commit 
> 6956b956934f10c19eca2a1d44f50a3bee860531) as the culprit.  The last 
> kernel to successfully boot on here is 4.19.x, while anything newer 
> exhibits the same stuck processes - and indeed, blacklisting the myrb 
> and myrs drivers allows 6.1 to boot on this machine.
> 
> I know this card is functional, as I do have two drives attached to it, 
> and both it and the drives work fine in 4.19 and older kernels, so the 
> issue seems to be with the newer myrb/myrs drivers.  Is there a chance 
> of fixing the current drivers, or, at worst, reintroducing the old 
> deprecated DAC960 driver back into the kernel?  I'm not absolutely tied 
> to using that driver, other than 'it just works' for this card.

Whee, someone is using it!
I'm not alone!

But sure, of course I'll help.
Can you try install openSUSE Leap on it? Then you can open a bugzilla on 
our side, and we can track and discuss things there. Debugging via 
e-mail tends to be very distracting to others not directly involved.

For starters, a message log might help. And please enable dynamic debug
via

echo 'file drivers/scsi/myrs.c +p' > \
   /sys/kernel/debug/dynamic_debug/control
echo 'file drivers/scsi/myrb.c +p' > \
   /sys/kernel/debug/dynamic_debug/control

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: Mylex AcceleRAID 170 + myrb/myrs causing crash

[Mike Edwards]

I can try openSUSE Leap, certainly.

However, those debugging statements are going to be hard - the process
lockup occurs at initial module load, before init really starts doing
much of anything.  I've even tried passing init=/bin/bash on the
commandline, with no luck - I never made it to a shell before things
went south.  For the same reason, there are no logs to speak of, alas
- but I can try seeing if I can get a serial console going, as that
will let me dump output.

I'll verify that the driver in OpenSUSE Leap has the same issue, and
will open a bug report when it (almost certainly!) dies the same way.

On Fri, Jul 14, 2023 at 2:04 AM Hannes Reinecke <hare@suse.de> wrote:
>
> On 7/13/23 21:21, Mike Edwards wrote:
> > I spun up an old machine (with an even older Mylex AcceleRAID card, the
> > 170 w/ a bios dated Jan 21, 2000 - yikes!) recently.  While this machine
> > was running an old 4.7 kernel and booted fine, attempting to update it
> > to a modern release of Debian with a 6.1 kernel caused the kernel to
> > hang while booting, with a number of stuck tasks warnings, starting with
> > udev-worker and including kworker kernel processes.
> >
> > During troubleshooting, I was able to identify the myrb/myrs drivers
> > which replaced the old DAC960 driver (removed in commit
> > 6956b956934f10c19eca2a1d44f50a3bee860531) as the culprit.  The last
> > kernel to successfully boot on here is 4.19.x, while anything newer
> > exhibits the same stuck processes - and indeed, blacklisting the myrb
> > and myrs drivers allows 6.1 to boot on this machine.
> >
> > I know this card is functional, as I do have two drives attached to it,
> > and both it and the drives work fine in 4.19 and older kernels, so the
> > issue seems to be with the newer myrb/myrs drivers.  Is there a chance
> > of fixing the current drivers, or, at worst, reintroducing the old
> > deprecated DAC960 driver back into the kernel?  I'm not absolutely tied
> > to using that driver, other than 'it just works' for this card.
>
> Whee, someone is using it!
> I'm not alone!
>
> But sure, of course I'll help.
> Can you try install openSUSE Leap on it? Then you can open a bugzilla on
> our side, and we can track and discuss things there. Debugging via
> e-mail tends to be very distracting to others not directly involved.
>
> For starters, a message log might help. And please enable dynamic debug
> via
>
> echo 'file drivers/scsi/myrs.c +p' > \
>    /sys/kernel/debug/dynamic_debug/control
> echo 'file drivers/scsi/myrb.c +p' > \
>    /sys/kernel/debug/dynamic_debug/control
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                Kernel Storage Architect
> hare@suse.de                              +49 911 74053 688
> SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
> HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
> Myers, Andrew McDonald, Martje Boudien Moerman
>

Re: Mylex AcceleRAID 170 + myrb/myrs causing crash

[Hannes Reinecke]

On 7/14/23 17:03, Mike Edwards wrote:
> I can try openSUSE Leap, certainly.
> 
> However, those debugging statements are going to be hard - the process
> lockup occurs at initial module load, before init really starts doing
> much of anything.  I've even tried passing init=/bin/bash on the
> commandline, with no luck - I never made it to a shell before things
> went south.  For the same reason, there are no logs to speak of, alas
> - but I can try seeing if I can get a serial console going, as that
> will let me dump output.
> 
> I'll verify that the driver in OpenSUSE Leap has the same issue, and
> will open a bug report when it (almost certainly!) dies the same way.
> 
You can try with the live system from the ISO image; that will load the 
modules only afterwards, so you should be able to see some messages there.

And I think there is even a magic commandline sequence to enable dynamic 
debug...

Cheers,

Hannes

Re: Mylex AcceleRAID 170 + myrb/myrs causing crash

[Mike Edwards]

I'll do some digging on dynamic debug, too, to see if I can figure that out.

Thanks again, Hannes - I'll try Leap in the next day or so and let you
know what I have from there.

On Fri, Jul 14, 2023 at 11:18 AM Hannes Reinecke <hare@suse.de> wrote:
>
> On 7/14/23 17:03, Mike Edwards wrote:
> > I can try openSUSE Leap, certainly.
> >
> > However, those debugging statements are going to be hard - the process
> > lockup occurs at initial module load, before init really starts doing
> > much of anything.  I've even tried passing init=/bin/bash on the
> > commandline, with no luck - I never made it to a shell before things
> > went south.  For the same reason, there are no logs to speak of, alas
> > - but I can try seeing if I can get a serial console going, as that
> > will let me dump output.
> >
> > I'll verify that the driver in OpenSUSE Leap has the same issue, and
> > will open a bug report when it (almost certainly!) dies the same way.
> >
> You can try with the live system from the ISO image; that will load the
> modules only afterwards, so you should be able to see some messages there.
>
> And I think there is even a magic commandline sequence to enable dynamic
> debug...
>
> Cheers,
>
> Hannes
>

Re: Mylex AcceleRAID 170 + myrb/myrs causing crash

[Mike Edwards]

This issue is now being tracked in the SUSE bugzilla as bug 1213537.

On Fri, Jul 14, 2023 at 2:04 AM Hannes Reinecke <hare@suse.de> wrote:
>
> On 7/13/23 21:21, Mike Edwards wrote:
> > I spun up an old machine (with an even older Mylex AcceleRAID card, the
> > 170 w/ a bios dated Jan 21, 2000 - yikes!) recently.  While this machine
> > was running an old 4.7 kernel and booted fine, attempting to update it
> > to a modern release of Debian with a 6.1 kernel caused the kernel to
> > hang while booting, with a number of stuck tasks warnings, starting with
> > udev-worker and including kworker kernel processes.
> >
> > During troubleshooting, I was able to identify the myrb/myrs drivers
> > which replaced the old DAC960 driver (removed in commit
> > 6956b956934f10c19eca2a1d44f50a3bee860531) as the culprit.  The last
> > kernel to successfully boot on here is 4.19.x, while anything newer
> > exhibits the same stuck processes - and indeed, blacklisting the myrb
> > and myrs drivers allows 6.1 to boot on this machine.
> >
> > I know this card is functional, as I do have two drives attached to it,
> > and both it and the drives work fine in 4.19 and older kernels, so the
> > issue seems to be with the newer myrb/myrs drivers.  Is there a chance
> > of fixing the current drivers, or, at worst, reintroducing the old
> > deprecated DAC960 driver back into the kernel?  I'm not absolutely tied
> > to using that driver, other than 'it just works' for this card.
>
> Whee, someone is using it!
> I'm not alone!
>
> But sure, of course I'll help.
> Can you try install openSUSE Leap on it? Then you can open a bugzilla on
> our side, and we can track and discuss things there. Debugging via
> e-mail tends to be very distracting to others not directly involved.
>
> For starters, a message log might help. And please enable dynamic debug
> via
>
> echo 'file drivers/scsi/myrs.c +p' > \
>    /sys/kernel/debug/dynamic_debug/control
> echo 'file drivers/scsi/myrb.c +p' > \
>    /sys/kernel/debug/dynamic_debug/control
>
> Cheers,
>
> Hannes
> --
> Dr. Hannes Reinecke                Kernel Storage Architect
> hare@suse.de                              +49 911 74053 688
> SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
> HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
> Myers, Andrew McDonald, Martje Boudien Moerman
>