[BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Russell King (Oracle)]

[Sorry for going off topic here - changed the Cc list, added Linus,
changed the subject.]

On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:
> On Wed, 7 May 2025 09:31:48 +0100
> "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> > [rest of the email got deleted because Linux / X11 / KDE got confused
> > about the state the backspace key and decided it was going to be
> > continuously pressed and doing nothing except shutting the laptop
> > down would stop it.]
> 
> Funny how I have the same exact issue on my laptop as well... 

I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
laptop I had previously (normally with ctrl-F keys). However, hitting
ctrl/shift/alt would stop it.

This is the first time I've seen the behaviour with the Carbon X1
laptop, but this was way more severe. No key would stop it. Trying to
move the focus using the trackpad/nipple had any effect. Meanwhile
the email was being deleted one character at a time. So I shut the
laptop lid causing it to suspend, and wondered what to do... on
re-opening the laptop, it didn't restart and is back to normal.

This suggests that the entire input subsystem in the software stack
collapsed just after the backspace key was pressed, and Xorg never
saw the key-release event. So Xorg duitifully did its key-repeat
processing, causing the email to be deleted one character at a time.

The problem is, not only did this destroy the email reply, but it
also destroyed my train of thought for the reply as well through
the panic of trying to stop the entire email being deleted.

I don't think this is a hardware issue - I think there's a problem
in the input handling somewhere in the stack of kernel, Xorg,
whatever multiple input libraries make up modern systems, and KDE.

I did check the logs. Nothing in the kernel messages that suggests
a problem. Nothing in Xorg's logs (which are difficult to tie up
because it doesn't use real timestamps that one can relate to real
time.) There's no longer any ~/.xsession-errors logfile for logging
the stuff below Xorg.

I'm running Debian Stable here - kernel 6.1.0-34-amd64, X.Org X Server
1.21.1.7, KDE Plasma (5.27.5, frameworks 5.103.0, QT 5.15.8).

Anyone else seeing this kind of behaviour - if so, what are you
using?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Russell King (Oracle)]

On Wed, May 07, 2025 at 10:23:17AM +0100, Russell King (Oracle) wrote:
> [Sorry for going off topic here - changed the Cc list, added Linus,
> changed the subject.]
> 
> On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:
> > On Wed, 7 May 2025 09:31:48 +0100
> > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> > > [rest of the email got deleted because Linux / X11 / KDE got confused
> > > about the state the backspace key and decided it was going to be
> > > continuously pressed and doing nothing except shutting the laptop
> > > down would stop it.]
> > 
> > Funny how I have the same exact issue on my laptop as well... 
> 
> I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
> laptop I had previously (normally with ctrl-F keys). However, hitting
> ctrl/shift/alt would stop it.
> 
> This is the first time I've seen the behaviour with the Carbon X1
> laptop, but this was way more severe. No key would stop it. Trying to
> move the focus using the trackpad/nipple had any effect. Meanwhile
> the email was being deleted one character at a time. So I shut the
> laptop lid causing it to suspend, and wondered what to do... on
> re-opening the laptop, it didn't restart and is back to normal.
> 
> This suggests that the entire input subsystem in the software stack
> collapsed just after the backspace key was pressed, and Xorg never
> saw the key-release event. So Xorg duitifully did its key-repeat
> processing, causing the email to be deleted one character at a time.
> 
> The problem is, not only did this destroy the email reply, but it
> also destroyed my train of thought for the reply as well through
> the panic of trying to stop the entire email being deleted.
> 
> I don't think this is a hardware issue - I think there's a problem
> in the input handling somewhere in the stack of kernel, Xorg,
> whatever multiple input libraries make up modern systems, and KDE.
> 
> I did check the logs. Nothing in the kernel messages that suggests
> a problem. Nothing in Xorg's logs (which are difficult to tie up
> because it doesn't use real timestamps that one can relate to real
> time.) There's no longer any ~/.xsession-errors logfile for logging
> the stuff below Xorg.
> 
> I'm running Debian Stable here - kernel 6.1.0-34-amd64, X.Org X Server
> 1.21.1.7, KDE Plasma (5.27.5, frameworks 5.103.0, QT 5.15.8).

I'll also add that The Carbon X1, being a laptop, its built-in keyboard
uses the i8042:

[    1.698156] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    1.698543] i8042: Warning: Keylock active
[    1.700170] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.700174] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.700271] mousedev: PS/2 mouse device common for all mice
[    1.702951] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0

I don't have the HP laptop with me to check what that was using.

The mysterious thing is "Keylock active" - clearly it isn't because I
can write this email typing on that very keyboard. However, I wonder
if it needs i8042_unlock=1 to set I8042_CTR_IGNKEYLOCK.

Unfortunately, it's probably going to take a year on the Carbon X1
to work out if this makes any difference.

> Anyone else seeing this kind of behaviour - if so, what are you
> using?
> 
> -- 
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Russell King (Oracle)]

Hi Maxime,

On Wed, May 07, 2025 at 03:32:36PM +0200, Maxime Chevallier wrote:
> Hi Russell,
> 
> On Wed, 7 May 2025 10:59:21 +0100
> "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> 
> > On Wed, May 07, 2025 at 10:23:17AM +0100, Russell King (Oracle) wrote:
> > > [Sorry for going off topic here - changed the Cc list, added Linus,
> > > changed the subject.]
> > > 
> > > On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:  
> > > > On Wed, 7 May 2025 09:31:48 +0100
> > > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:  
> > > > > [rest of the email got deleted because Linux / X11 / KDE got confused
> > > > > about the state the backspace key and decided it was going to be
> > > > > continuously pressed and doing nothing except shutting the laptop
> > > > > down would stop it.]  
> > > > 
> > > > Funny how I have the same exact issue on my laptop as well...   
> > > 
> > > I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
> > > laptop I had previously (normally with ctrl-F keys). However, hitting
> > > ctrl/shift/alt would stop it.
> > > 
> > > This is the first time I've seen the behaviour with the Carbon X1
> > > laptop, but this was way more severe. No key would stop it. Trying to
> > > move the focus using the trackpad/nipple had any effect. Meanwhile
> > > the email was being deleted one character at a time. So I shut the
> > > laptop lid causing it to suspend, and wondered what to do... on
> > > re-opening the laptop, it didn't restart and is back to normal.
> > > 
> > > This suggests that the entire input subsystem in the software stack
> > > collapsed just after the backspace key was pressed, and Xorg never
> > > saw the key-release event. So Xorg duitifully did its key-repeat
> > > processing, causing the email to be deleted one character at a time.
> > > 
> > > The problem is, not only did this destroy the email reply, but it
> > > also destroyed my train of thought for the reply as well through
> > > the panic of trying to stop the entire email being deleted.
> > > 
> > > I don't think this is a hardware issue - I think there's a problem
> > > in the input handling somewhere in the stack of kernel, Xorg,
> > > whatever multiple input libraries make up modern systems, and KDE.
> > > 
> > > I did check the logs. Nothing in the kernel messages that suggests
> > > a problem. Nothing in Xorg's logs (which are difficult to tie up
> > > because it doesn't use real timestamps that one can relate to real
> > > time.) There's no longer any ~/.xsession-errors logfile for logging
> > > the stuff below Xorg.
> > > 
> > > I'm running Debian Stable here - kernel 6.1.0-34-amd64, X.Org X Server
> > > 1.21.1.7, KDE Plasma (5.27.5, frameworks 5.103.0, QT 5.15.8).  
> > 
> > I'll also add that The Carbon X1, being a laptop, its built-in keyboard
> > uses the i8042:
> > 
> > [    1.698156] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
> > [    1.698543] i8042: Warning: Keylock active
> > [    1.700170] serio: i8042 KBD port at 0x60,0x64 irq 1
> > [    1.700174] serio: i8042 AUX port at 0x60,0x64 irq 12
> > [    1.700271] mousedev: PS/2 mouse device common for all mice
> > [    1.702951] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
> > 
> > I don't have the HP laptop with me to check what that was using.
> > 
> > The mysterious thing is "Keylock active" - clearly it isn't because I
> > can write this email typing on that very keyboard. However, I wonder
> > if it needs i8042_unlock=1 to set I8042_CTR_IGNKEYLOCK.
> > 
> > Unfortunately, it's probably going to take a year on the Carbon X1
> > to work out if this makes any difference.
> >
> > > Anyone else seeing this kind of behaviour - if so, what are you
> > > using?
> 
> It just happened to me as I was typing this very email (key 'd' got
> stuck, nothing could un-stick it, couldn't move the mouse cursor but
> mouse-click events did work, had to suspend/resume the laptop to fix
> that)

'd' can be quite disastrous if you're using vi and you're in command
mode!

> Got the same "Keylock active" warning at boot :
> 
> [    0.916750] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
> [    0.917210] i8042: Warning: Keylock active
> [    0.920087] serio: i8042 KBD port at 0x60,0x64 irq 1
> [    0.920090] serio: i8042 AUX port at 0x60,0x64 irq 12
> 
> Nothing in the kernel logs when the key got stuck.
> 
> Laptop is a Dell XPS 15 9510, Running Fedora 41 but I saw this issue
> before, kernel 6.14.4-200.fc41.x86_64, Wayland-based, Gnome 47.
> 
> Hopefully this helps a bit narrowing this down, I have a fairly
> different userspace stack and kernel version, but we do have the same
> driver involved and same keylock warning...

Could you try booting with i8042_unlock=1 and see whether that makes any
difference please?

I've added that to my grub config in preparation for rebooting, but even
if I booted now, I suspect it'll be some time before I have any useful
result.

How often do you see the problem?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Maxime Chevallier]

On Wed, 7 May 2025 12:44:24 +0100
"Russell King (Oracle)" <linux@armlinux.org.uk> wrote:

> Hi Maxime,
> 
> On Wed, May 07, 2025 at 03:32:36PM +0200, Maxime Chevallier wrote:
> > Hi Russell,
> > 
> > On Wed, 7 May 2025 10:59:21 +0100
> > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> >   
> > > On Wed, May 07, 2025 at 10:23:17AM +0100, Russell King (Oracle) wrote:  
> > > > [Sorry for going off topic here - changed the Cc list, added Linus,
> > > > changed the subject.]
> > > > 
> > > > On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:    
> > > > > On Wed, 7 May 2025 09:31:48 +0100
> > > > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:    
> > > > > > [rest of the email got deleted because Linux / X11 / KDE got confused
> > > > > > about the state the backspace key and decided it was going to be
> > > > > > continuously pressed and doing nothing except shutting the laptop
> > > > > > down would stop it.]    
> > > > > 
> > > > > Funny how I have the same exact issue on my laptop as well...     
> > > > 
> > > > I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
> > > > laptop I had previously (normally with ctrl-F keys). However, hitting
> > > > ctrl/shift/alt would stop it.
> > > > 
> > > > This is the first time I've seen the behaviour with the Carbon X1
> > > > laptop, but this was way more severe. No key would stop it. Trying to
> > > > move the focus using the trackpad/nipple had any effect. Meanwhile
> > > > the email was being deleted one character at a time. So I shut the
> > > > laptop lid causing it to suspend, and wondered what to do... on
> > > > re-opening the laptop, it didn't restart and is back to normal.
> > > > 
> > > > This suggests that the entire input subsystem in the software stack
> > > > collapsed just after the backspace key was pressed, and Xorg never
> > > > saw the key-release event. So Xorg duitifully did its key-repeat
> > > > processing, causing the email to be deleted one character at a time.
> > > > 
> > > > The problem is, not only did this destroy the email reply, but it
> > > > also destroyed my train of thought for the reply as well through
> > > > the panic of trying to stop the entire email being deleted.
> > > > 
> > > > I don't think this is a hardware issue - I think there's a problem
> > > > in the input handling somewhere in the stack of kernel, Xorg,
> > > > whatever multiple input libraries make up modern systems, and KDE.
> > > > 
> > > > I did check the logs. Nothing in the kernel messages that suggests
> > > > a problem. Nothing in Xorg's logs (which are difficult to tie up
> > > > because it doesn't use real timestamps that one can relate to real
> > > > time.) There's no longer any ~/.xsession-errors logfile for logging
> > > > the stuff below Xorg.
> > > > 
> > > > I'm running Debian Stable here - kernel 6.1.0-34-amd64, X.Org X Server
> > > > 1.21.1.7, KDE Plasma (5.27.5, frameworks 5.103.0, QT 5.15.8).    
> > > 
> > > I'll also add that The Carbon X1, being a laptop, its built-in keyboard
> > > uses the i8042:
> > > 
> > > [    1.698156] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
> > > [    1.698543] i8042: Warning: Keylock active
> > > [    1.700170] serio: i8042 KBD port at 0x60,0x64 irq 1
> > > [    1.700174] serio: i8042 AUX port at 0x60,0x64 irq 12
> > > [    1.700271] mousedev: PS/2 mouse device common for all mice
> > > [    1.702951] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
> > > 
> > > I don't have the HP laptop with me to check what that was using.
> > > 
> > > The mysterious thing is "Keylock active" - clearly it isn't because I
> > > can write this email typing on that very keyboard. However, I wonder
> > > if it needs i8042_unlock=1 to set I8042_CTR_IGNKEYLOCK.
> > > 
> > > Unfortunately, it's probably going to take a year on the Carbon X1
> > > to work out if this makes any difference.
> > >  
> > > > Anyone else seeing this kind of behaviour - if so, what are you
> > > > using?  
> > 
> > It just happened to me as I was typing this very email (key 'd' got
> > stuck, nothing could un-stick it, couldn't move the mouse cursor but
> > mouse-click events did work, had to suspend/resume the laptop to fix
> > that)  
> 
> 'd' can be quite disastrous if you're using vi and you're in command
> mode!
> 
> > Got the same "Keylock active" warning at boot :
> > 
> > [    0.916750] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
> > [    0.917210] i8042: Warning: Keylock active
> > [    0.920087] serio: i8042 KBD port at 0x60,0x64 irq 1
> > [    0.920090] serio: i8042 AUX port at 0x60,0x64 irq 12
> > 
> > Nothing in the kernel logs when the key got stuck.
> > 
> > Laptop is a Dell XPS 15 9510, Running Fedora 41 but I saw this issue
> > before, kernel 6.14.4-200.fc41.x86_64, Wayland-based, Gnome 47.
> > 
> > Hopefully this helps a bit narrowing this down, I have a fairly
> > different userspace stack and kernel version, but we do have the same
> > driver involved and same keylock warning...  
> 
> Could you try booting with i8042_unlock=1 and see whether that makes any
> difference please?

I'll try this out indeed :)

> I've added that to my grub config in preparation for rebooting, but even
> if I booted now, I suspect it'll be some time before I have any useful
> result.
> 
> How often do you see the problem?

It's very sporadic, sometimes I'll see this behaviour multiple times a
day, and then nothing for weeks... This laptop has been very quirky
ever since I got it, so I categorized that as a "yet another XPS 9510
broken behaviour", but this has been a recurrent thing for multiple
years...

So, same as you, it'll take a long time for me to say with some amount
of certainty that 'i8042_unlock=1' has a beneficial effect, of
course unless I see the problem happen again in the meantime.

Maxime

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Maxime Chevallier]

Hi Russell,

On Wed, 7 May 2025 10:59:21 +0100
"Russell King (Oracle)" <linux@armlinux.org.uk> wrote:

> On Wed, May 07, 2025 at 10:23:17AM +0100, Russell King (Oracle) wrote:
> > [Sorry for going off topic here - changed the Cc list, added Linus,
> > changed the subject.]
> > 
> > On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:  
> > > On Wed, 7 May 2025 09:31:48 +0100
> > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:  
> > > > [rest of the email got deleted because Linux / X11 / KDE got confused
> > > > about the state the backspace key and decided it was going to be
> > > > continuously pressed and doing nothing except shutting the laptop
> > > > down would stop it.]  
> > > 
> > > Funny how I have the same exact issue on my laptop as well...   
> > 
> > I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
> > laptop I had previously (normally with ctrl-F keys). However, hitting
> > ctrl/shift/alt would stop it.
> > 
> > This is the first time I've seen the behaviour with the Carbon X1
> > laptop, but this was way more severe. No key would stop it. Trying to
> > move the focus using the trackpad/nipple had any effect. Meanwhile
> > the email was being deleted one character at a time. So I shut the
> > laptop lid causing it to suspend, and wondered what to do... on
> > re-opening the laptop, it didn't restart and is back to normal.
> > 
> > This suggests that the entire input subsystem in the software stack
> > collapsed just after the backspace key was pressed, and Xorg never
> > saw the key-release event. So Xorg duitifully did its key-repeat
> > processing, causing the email to be deleted one character at a time.
> > 
> > The problem is, not only did this destroy the email reply, but it
> > also destroyed my train of thought for the reply as well through
> > the panic of trying to stop the entire email being deleted.
> > 
> > I don't think this is a hardware issue - I think there's a problem
> > in the input handling somewhere in the stack of kernel, Xorg,
> > whatever multiple input libraries make up modern systems, and KDE.
> > 
> > I did check the logs. Nothing in the kernel messages that suggests
> > a problem. Nothing in Xorg's logs (which are difficult to tie up
> > because it doesn't use real timestamps that one can relate to real
> > time.) There's no longer any ~/.xsession-errors logfile for logging
> > the stuff below Xorg.
> > 
> > I'm running Debian Stable here - kernel 6.1.0-34-amd64, X.Org X Server
> > 1.21.1.7, KDE Plasma (5.27.5, frameworks 5.103.0, QT 5.15.8).  
> 
> I'll also add that The Carbon X1, being a laptop, its built-in keyboard
> uses the i8042:
> 
> [    1.698156] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
> [    1.698543] i8042: Warning: Keylock active
> [    1.700170] serio: i8042 KBD port at 0x60,0x64 irq 1
> [    1.700174] serio: i8042 AUX port at 0x60,0x64 irq 12
> [    1.700271] mousedev: PS/2 mouse device common for all mice
> [    1.702951] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
> 
> I don't have the HP laptop with me to check what that was using.
> 
> The mysterious thing is "Keylock active" - clearly it isn't because I
> can write this email typing on that very keyboard. However, I wonder
> if it needs i8042_unlock=1 to set I8042_CTR_IGNKEYLOCK.
> 
> Unfortunately, it's probably going to take a year on the Carbon X1
> to work out if this makes any difference.
>
> > Anyone else seeing this kind of behaviour - if so, what are you
> > using?

It just happened to me as I was typing this very email (key 'd' got
stuck, nothing could un-stick it, couldn't move the mouse cursor but
mouse-click events did work, had to suspend/resume the laptop to fix
that)

Got the same "Keylock active" warning at boot :

[    0.916750] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    0.917210] i8042: Warning: Keylock active
[    0.920087] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.920090] serio: i8042 AUX port at 0x60,0x64 irq 12

Nothing in the kernel logs when the key got stuck.

Laptop is a Dell XPS 15 9510, Running Fedora 41 but I saw this issue
before, kernel 6.14.4-200.fc41.x86_64, Wayland-based, Gnome 47.

Hopefully this helps a bit narrowing this down, I have a fairly
different userspace stack and kernel version, but we do have the same
driver involved and same keylock warning...

Maxime

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Linus Torvalds]

On Wed, 7 May 2025 at 04:51, Maxime Chevallier
<maxime.chevallier@bootlin.com> wrote:
>
> So, same as you, it'll take a long time for me to say with some amount
> of certainty that 'i8042_unlock=1' has a beneficial effect, of
> course unless I see the problem happen again in the meantime.

Christ. You'd expect that any i8042 issues had been fixed long ago,
but the problem is that the chip doesn't necessarily even exist in
modern platforms, and everybody just fakes it.

So the platform presumably still has hardware support for it, but it's
mostly in the form of "take a trap when accessing the legacy keyboard
ports, and fake it in firmware".

Although it doesn't help that there are literally decades of clone
chips and hacky real hardware that extended on the i8042 in various
more-or-less compatible ways.

Which makes all of these things almost entirely undebuggable.

I'm surprised the XPS9510 would be particularly troublesome - I've had
an XPS for years (older version, obviously) with no issues outside of
WiFi sometimes acting up. But random firmware...

I doubt it's "keylock active", but who knows. I get that on my xps
too, it's a random bit that doesn't really mean much. But - because of
all the reasons above - who knows...

One typical problem has been "the interrupt line is wired oddly", but
the fact that it apparently works *most* of the time means that that
is unlikely to be the issue here.

              Linus

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Dmitry Torokhov]

On Wed, May 07, 2025 at 07:46:03AM -0700, Linus Torvalds wrote:
> On Wed, 7 May 2025 at 04:51, Maxime Chevallier
> <maxime.chevallier@bootlin.com> wrote:
> >
> > So, same as you, it'll take a long time for me to say with some amount
> > of certainty that 'i8042_unlock=1' has a beneficial effect, of
> > course unless I see the problem happen again in the meantime.
> 
> Christ. You'd expect that any i8042 issues had been fixed long ago,
> but the problem is that the chip doesn't necessarily even exist in
> modern platforms, and everybody just fakes it.

It has not existed as a real chip for more than 20 years I believe. It's
all faked in firmware and embedded controllers that fake it in their
firmwares.

And newer firmware tend to implement less and less of it, just what OS
that devices that ship with it needs.

> 
> So the platform presumably still has hardware support for it, but it's
> mostly in the form of "take a trap when accessing the legacy keyboard
> ports, and fake it in firmware".
> 
> Although it doesn't help that there are literally decades of clone
> chips and hacky real hardware that extended on the i8042 in various
> more-or-less compatible ways.
> 
> Which makes all of these things almost entirely undebuggable.
> 
> I'm surprised the XPS9510 would be particularly troublesome - I've had
> an XPS for years (older version, obviously) with no issues outside of
> WiFi sometimes acting up. But random firmware...
> 
> I doubt it's "keylock active", but who knows. I get that on my xps
> too, it's a random bit that doesn't really mean much. But - because of
> all the reasons above - who knows...

It is typically harmless and whats more trying to "unlock" 8042 when it
reports being locked might confuse 8042 emulation.

Thanks.

-- 
Dmitry

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Dmitry Torokhov]

On Wed, May 07, 2025 at 01:51:26PM +0200, Maxime Chevallier wrote:
> On Wed, 7 May 2025 12:44:24 +0100
> "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> 
> > Hi Maxime,
> > 
> > On Wed, May 07, 2025 at 03:32:36PM +0200, Maxime Chevallier wrote:
> > > Hi Russell,
> > > 
> > > On Wed, 7 May 2025 10:59:21 +0100
> > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> > >   
> > > > On Wed, May 07, 2025 at 10:23:17AM +0100, Russell King (Oracle) wrote:  
> > > > > [Sorry for going off topic here - changed the Cc list, added Linus,
> > > > > changed the subject.]
> > > > > 
> > > > > On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:    
> > > > > > On Wed, 7 May 2025 09:31:48 +0100
> > > > > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:    
> > > > > > > [rest of the email got deleted because Linux / X11 / KDE got confused
> > > > > > > about the state the backspace key and decided it was going to be
> > > > > > > continuously pressed and doing nothing except shutting the laptop
> > > > > > > down would stop it.]    
> > > > > > 
> > > > > > Funny how I have the same exact issue on my laptop as well...     
> > > > > 
> > > > > I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
> > > > > laptop I had previously (normally with ctrl-F keys). However, hitting
> > > > > ctrl/shift/alt would stop it.
> > > > > 
> > > > > This is the first time I've seen the behaviour with the Carbon X1
> > > > > laptop, but this was way more severe. No key would stop it. Trying to
> > > > > move the focus using the trackpad/nipple had any effect. Meanwhile
> > > > > the email was being deleted one character at a time. So I shut the

If we indeed lost a key release event somewhere the way to "restore" it
is to hit the stuck key again. Then we should get press/release with
press most likely being ignored and release achieving the desired
result. Of course that will not help if embedded controller is confused.

> > > > > laptop lid causing it to suspend, and wondered what to do... on
> > > > > re-opening the laptop, it didn't restart and is back to normal.
> > > > > 
> > > > > This suggests that the entire input subsystem in the software stack
> > > > > collapsed just after the backspace key was pressed, and Xorg never
> > > > > saw the key-release event. So Xorg duitifully did its key-repeat
> > > > > processing, causing the email to be deleted one character at a time.
> > > > > 
> > > > > The problem is, not only did this destroy the email reply, but it
> > > > > also destroyed my train of thought for the reply as well through
> > > > > the panic of trying to stop the entire email being deleted.
> > > > > 
> > > > > I don't think this is a hardware issue - I think there's a problem
> > > > > in the input handling somewhere in the stack of kernel, Xorg,
> > > > > whatever multiple input libraries make up modern systems, and KDE.
> > > > > 
> > > > > I did check the logs. Nothing in the kernel messages that suggests
> > > > > a problem. Nothing in Xorg's logs (which are difficult to tie up
> > > > > because it doesn't use real timestamps that one can relate to real
> > > > > time.) There's no longer any ~/.xsession-errors logfile for logging
> > > > > the stuff below Xorg.
> > > > > 
> > > > > I'm running Debian Stable here - kernel 6.1.0-34-amd64, X.Org X Server
> > > > > 1.21.1.7, KDE Plasma (5.27.5, frameworks 5.103.0, QT 5.15.8).    
> > > > 
> > > > I'll also add that The Carbon X1, being a laptop, its built-in keyboard
> > > > uses the i8042:
> > > > 
> > > > [    1.698156] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
> > > > [    1.698543] i8042: Warning: Keylock active
> > > > [    1.700170] serio: i8042 KBD port at 0x60,0x64 irq 1
> > > > [    1.700174] serio: i8042 AUX port at 0x60,0x64 irq 12
> > > > [    1.700271] mousedev: PS/2 mouse device common for all mice
> > > > [    1.702951] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
> > > > 
> > > > I don't have the HP laptop with me to check what that was using.
> > > > 
> > > > The mysterious thing is "Keylock active" - clearly it isn't because I
> > > > can write this email typing on that very keyboard. However, I wonder
> > > > if it needs i8042_unlock=1 to set I8042_CTR_IGNKEYLOCK.

Just ignore this message, it is harmless and trying to flip the bit
might confuse the emulation even more. Maybe we should lower the
severity of it to debug.

That said I do not see it on my Carbon (neither v5 nor v12, can't check
v9 because it is at home)... What version of Carbon do you have? Do you
have up-to-date BIOS/EC?

> > > > 
> > > > Unfortunately, it's probably going to take a year on the Carbon X1
> > > > to work out if this makes any difference.
> > > >  
> > > > > Anyone else seeing this kind of behaviour - if so, what are you
> > > > > using?  
> > > 
> > > It just happened to me as I was typing this very email (key 'd' got
> > > stuck, nothing could un-stick it, couldn't move the mouse cursor but
> > > mouse-click events did work, had to suspend/resume the laptop to fix
> > > that)

This is weird and suggests that the breakage happens up the stack from
the kernel (or down in the firmware). Mouse clicks and mouse movement is
delivered as part of a mouse packet, so if there are button clicks there
will also be movement, they are not separate. If the cursor is not
reacting that means desktop environment is not handling input properly.

> > 
> > 'd' can be quite disastrous if you're using vi and you're in command
> > mode!
> > 
> > > Got the same "Keylock active" warning at boot :
> > > 
> > > [    0.916750] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
> > > [    0.917210] i8042: Warning: Keylock active
> > > [    0.920087] serio: i8042 KBD port at 0x60,0x64 irq 1
> > > [    0.920090] serio: i8042 AUX port at 0x60,0x64 irq 12
> > > 
> > > Nothing in the kernel logs when the key got stuck.
> > > 
> > > Laptop is a Dell XPS 15 9510, Running Fedora 41 but I saw this issue
> > > before, kernel 6.14.4-200.fc41.x86_64, Wayland-based, Gnome 47.
> > > 
> > > Hopefully this helps a bit narrowing this down, I have a fairly
> > > different userspace stack and kernel version, but we do have the same
> > > driver involved and same keylock warning...  
> > 
> > Could you try booting with i8042_unlock=1 and see whether that makes any
> > difference please?
> 
> I'll try this out indeed :)
> 
> > I've added that to my grub config in preparation for rebooting, but even
> > if I booted now, I suspect it'll be some time before I have any useful
> > result.
> > 
> > How often do you see the problem?
> 
> It's very sporadic, sometimes I'll see this behaviour multiple times a
> day, and then nothing for weeks... This laptop has been very quirky
> ever since I got it, so I categorized that as a "yet another XPS 9510
> broken behaviour", but this has been a recurrent thing for multiple
> years...
> 
> So, same as you, it'll take a long time for me to say with some amount
> of certainty that 'i8042_unlock=1' has a beneficial effect, of
> course unless I see the problem happen again in the meantime.

The kernel does drop input events if userspace is unable to read buffers
quickly enough. It notifies userspace by queuing special
EV_SYN/SYN_DROPPED event and userspace is supposed to query the full
device state upon receiving it to figure out what to do. I doubt we are
running into this with keyboards, but maybe we should add some logging
there to rule this out.

I'll add Peter and Benjamin to this thread in case they have ideas.

Thanks.

-- 
Dmitry

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Russell King (Oracle)]

On Wed, May 07, 2025 at 10:23:37AM -0700, Dmitry Torokhov wrote:
> On Wed, May 07, 2025 at 07:46:03AM -0700, Linus Torvalds wrote:
> > On Wed, 7 May 2025 at 04:51, Maxime Chevallier
> > <maxime.chevallier@bootlin.com> wrote:
> > >
> > > So, same as you, it'll take a long time for me to say with some amount
> > > of certainty that 'i8042_unlock=1' has a beneficial effect, of
> > > course unless I see the problem happen again in the meantime.
> > 
> > Christ. You'd expect that any i8042 issues had been fixed long ago,
> > but the problem is that the chip doesn't necessarily even exist in
> > modern platforms, and everybody just fakes it.
> 
> It has not existed as a real chip for more than 20 years I believe. It's
> all faked in firmware and embedded controllers that fake it in their
> firmwares.
> 
> And newer firmware tend to implement less and less of it, just what OS
> that devices that ship with it needs.
> 
> > 
> > So the platform presumably still has hardware support for it, but it's
> > mostly in the form of "take a trap when accessing the legacy keyboard
> > ports, and fake it in firmware".
> > 
> > Although it doesn't help that there are literally decades of clone
> > chips and hacky real hardware that extended on the i8042 in various
> > more-or-less compatible ways.
> > 
> > Which makes all of these things almost entirely undebuggable.
> > 
> > I'm surprised the XPS9510 would be particularly troublesome - I've had
> > an XPS for years (older version, obviously) with no issues outside of
> > WiFi sometimes acting up. But random firmware...
> > 
> > I doubt it's "keylock active", but who knows. I get that on my xps
> > too, it's a random bit that doesn't really mean much. But - because of
> > all the reasons above - who knows...
> 
> It is typically harmless and whats more trying to "unlock" 8042 when it
> reports being locked might confuse 8042 emulation.

So I think the summary of this thread is... laptop keyboards are
unreliable, and it's a lottery whether any particular laptop works
or remains working over firmware updates.

Surely that essentially means, laptops are basically unreliable
devices?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Russell King (Oracle)]

On Wed, May 07, 2025 at 10:45:48AM -0700, Dmitry Torokhov wrote:
> On Wed, May 07, 2025 at 01:51:26PM +0200, Maxime Chevallier wrote:
> > On Wed, 7 May 2025 12:44:24 +0100
> > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> > 
> > > Hi Maxime,
> > > 
> > > On Wed, May 07, 2025 at 03:32:36PM +0200, Maxime Chevallier wrote:
> > > > Hi Russell,
> > > > 
> > > > On Wed, 7 May 2025 10:59:21 +0100
> > > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> > > >   
> > > > > On Wed, May 07, 2025 at 10:23:17AM +0100, Russell King (Oracle) wrote:  
> > > > > > [Sorry for going off topic here - changed the Cc list, added Linus,
> > > > > > changed the subject.]
> > > > > > 
> > > > > > On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:    
> > > > > > > On Wed, 7 May 2025 09:31:48 +0100
> > > > > > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:    
> > > > > > > > [rest of the email got deleted because Linux / X11 / KDE got confused
> > > > > > > > about the state the backspace key and decided it was going to be
> > > > > > > > continuously pressed and doing nothing except shutting the laptop
> > > > > > > > down would stop it.]    
> > > > > > > 
> > > > > > > Funny how I have the same exact issue on my laptop as well...     
> > > > > > 
> > > > > > I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
> > > > > > laptop I had previously (normally with ctrl-F keys). However, hitting
> > > > > > ctrl/shift/alt would stop it.
> > > > > > 
> > > > > > This is the first time I've seen the behaviour with the Carbon X1
> > > > > > laptop, but this was way more severe. No key would stop it. Trying to
> > > > > > move the focus using the trackpad/nipple had any effect. Meanwhile
> > > > > > the email was being deleted one character at a time. So I shut the
> 
> If we indeed lost a key release event somewhere the way to "restore" it
> is to hit the stuck key again. Then we should get press/release with
> press most likely being ignored and release achieving the desired
> result. Of course that will not help if embedded controller is confused.

I tried pressing every key...

> > > > > The mysterious thing is "Keylock active" - clearly it isn't because I
> > > > > can write this email typing on that very keyboard. However, I wonder
> > > > > if it needs i8042_unlock=1 to set I8042_CTR_IGNKEYLOCK.
> 
> Just ignore this message, it is harmless and trying to flip the bit
> might confuse the emulation even more. Maybe we should lower the
> severity of it to debug.
> 
> That said I do not see it on my Carbon (neither v5 nor v12, can't check
> v9 because it is at home)... What version of Carbon do you have? Do you
> have up-to-date BIOS/EC?

Neither did I see a problem until today, and I've been using the laptop
since October 2024, and this is the first time it's had an issue.

Looking at fwupd, it has an Intel ME update pending (1.32.2418 to
1.35.2557). I can't find a way to get any update history beyond
that out of fwupd and fwupd doesn't seem to log to journald what
it's doing.

> > > > It just happened to me as I was typing this very email (key 'd' got
> > > > stuck, nothing could un-stick it, couldn't move the mouse cursor but
> > > > mouse-click events did work, had to suspend/resume the laptop to fix
> > > > that)
> 
> This is weird and suggests that the breakage happens up the stack from
> the kernel (or down in the firmware). Mouse clicks and mouse movement is
> delivered as part of a mouse packet, so if there are button clicks there
> will also be movement, they are not separate. If the cursor is not
> reacting that means desktop environment is not handling input properly.

So could we be looking at an Xorg bug?

> The kernel does drop input events if userspace is unable to read buffers
> quickly enough. It notifies userspace by queuing special
> EV_SYN/SYN_DROPPED event and userspace is supposed to query the full
> device state upon receiving it to figure out what to do. I doubt we are
> running into this with keyboards, but maybe we should add some logging
> there to rule this out.
> 
> I'll add Peter and Benjamin to this thread in case they have ideas.

I'm thinking of leaving evtest running in a terminal, so its output
can be inspected if it happens again. One issue though is the
timestamps aren't readable, but I'm sure with a bit of perl
post-processing that could be fixed.

That would allow an answer to "is it kernel or firmware" vs
"userspace".

The problem is - if it's taken 7-ish months to show for me, it's
likely that evtest won't be running when it next happens (as there
will be needs to reboot for kernel upgrades/firmware upgrades in
that time.) Really, it needs to be something like an automatically
started at boot e.g. evtest inside a detached screen session.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [BUG] Stuck key syndrome

[Holger Hoffstätte]

On 2025-05-07 13:44, Russell King (Oracle) wrote:
> Could you try booting with i8042_unlock=1 and see whether that makes any
> difference please?

It did not help - just had another runaway event with that setting,
on my ca. 2021 Thinkpad L14. Had the symptom for as long as I have
had this machine.

We've been tracking this problem in Gentoo since late 2022, see
https://bugs.gentoo.org/873163 and none of the suggested options
for i8042 really make a difference. In my case I almost always get
the stuck key events when using the cursor keys for scrolling in a
web browser. Sometimes once a month, sometimes twice a day.

Fwiw it's not necessary to reboot; suspend/resume fixes it,
as in close/reopen the lid if you have that configured.

thanks
Holger

Re: [BUG] Stuck key syndrome

[Dmitry Torokhov]

On Wed, May 07, 2025 at 10:46:35PM +0200, Holger Hoffstätte wrote:
> On 2025-05-07 13:44, Russell King (Oracle) wrote:
> > Could you try booting with i8042_unlock=1 and see whether that makes any
> > difference please?
> 
> It did not help - just had another runaway event with that setting,
> on my ca. 2021 Thinkpad L14. Had the symptom for as long as I have
> had this machine.
> 
> We've been tracking this problem in Gentoo since late 2022, see
> https://bugs.gentoo.org/873163 and none of the suggested options
> for i8042 really make a difference. In my case I almost always get
> the stuck key events when using the cursor keys for scrolling in a
> web browser. Sometimes once a month, sometimes twice a day.
> 
> Fwiw it's not necessary to reboot; suspend/resume fixes it,
> as in close/reopen the lid if you have that configured.

So looking at your logs in gentoo bugzilla we see:

>>> It is around 1 second later that I realise the J key has died.
>>> Now I sit and watch for a few seconds before closing the lid.
Event: time 1664975487.559043, -------------- SYN_REPORT ------------
Event: time 1664975487.591980, type 4 (EV_MSC), code 4 (MSC_SCAN), value 24
Event: time 1664975487.591980, type 1 (EV_KEY), code 36 (KEY_J), value 2
Event: time 1664975487.591980, -------------- SYN_REPORT ------------
Event: time 1664975487.624955, type 4 (EV_MSC), code 4 (MSC_SCAN), value 24
Event: time 1664975487.624955, type 1 (EV_KEY), code 36 (KEY_J), value 2
Event: time 1664975487.624955, -------------- SYN_REPORT ------------
Event: time 1664975487.657800, type 4 (EV_MSC), code 4 (MSC_SCAN), value 24
Event: time 1664975487.657800, type 1 (EV_KEY), code 36 (KEY_J), value 2
Event: time 1664975487.657800, -------------- SYN_REPORT ------------

Because I see the MSC_SCAN events this means you are not using
atkbd.softrepeat for software-emulated autorepeat and is using the
hardware autorepeat function (which is the default). In this mode
keyboard controller repeatedly sends the scancode for the pressed key
and kernel reports it. We know that interrupts are working because we do
get scancodes from i8042 and from the kernel POV the key is still
pressed because [piece of software that emulates] i8042 tells it so. :(

...
Event: time 1664975493.812157, -------------- SYN_REPORT ------------
Event: time 1664975495.120717, type 1 (EV_KEY), code 36 (KEY_J), value 0
>>> It is around here that the computer resumes from suspend.

This is software-emulated key release that we do as part of suspend
process. And afterwards firmware gets jolted into its senses by suspend
and starts working...

Thanks.

-- 
Dmitry

Re: [BUG] Stuck key syndrome

[Russell King (Oracle)]

On Wed, May 07, 2025 at 10:46:35PM +0200, Holger Hoffstätte wrote:
> On 2025-05-07 13:44, Russell King (Oracle) wrote:
> > Could you try booting with i8042_unlock=1 and see whether that makes any
> > difference please?
> 
> It did not help - just had another runaway event with that setting,
> on my ca. 2021 Thinkpad L14. Had the symptom for as long as I have
> had this machine.
> 
> We've been tracking this problem in Gentoo since late 2022, see
> https://bugs.gentoo.org/873163 and none of the suggested options
> for i8042 really make a difference. In my case I almost always get
> the stuck key events when using the cursor keys for scrolling in a
> web browser. Sometimes once a month, sometimes twice a day.
> 
> Fwiw it's not necessary to reboot; suspend/resume fixes it,
> as in close/reopen the lid if you have that configured.

Thanks - it's good to know that I'm not alone with this problem!
I wonder how common it is, I think we're now up to four people.

So it's interesting that Finn's system is AMD vs mine which have
both been Intel based systems, and we seem to have exactly the same
problem. Is it possible that both are using the same firmware for
emulating an i8042?

Also what seems to be interesting is that it afflicts specific keys.
On my old HP Pavilion, it was always Ctrl-F3 which would get stuck
down (which I use to switch to virtual desktop 3 which has my Konsoles
on.) In this case, pressing all of ctrl-shift-alt would clear it.

I've only had it once on the Lenovo Carbon X1 so far, so can't comment
if it's going to be the backspace key every time - there is an Intel ME
firmware update pending (it didn't get installed at the last reboot -
I'd accidentally left the laptop with sleep disabled but no external
power, so it drained the battery and shut itself down which probably
prevented the firmware update being installed.) I believe the Intel ME
deals with the keyboard.

Thanks for adding comment 24... I assume from what you've said above
that comment 23 has proven not to solve it completely, but has it
reduced the frequency at all for you?

What also crosses my mind is that if the i8042 is now emulated by
firmware, is there a replacement interface that the kernel should
instead be using? Surely if the i8042 emulation is as bad as this,
then e.g. under Windows these laptops would be getting very poor
write-ups due to keyboard problems afflicting Windows as well.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [BUG] Stuck key syndrome

[Linus Torvalds]

On Wed, 7 May 2025 at 15:07, Russell King (Oracle)
<linux@armlinux.org.uk> wrote:
>
> So it's interesting that Finn's system is AMD vs mine which have
> both been Intel based systems, and we seem to have exactly the same
> problem. Is it possible that both are using the same firmware for
> emulating an i8042?

Might be a BIOS vendor issue.

> Also what seems to be interesting is that it afflicts specific keys.
> On my old HP Pavilion, it was always Ctrl-F3 which would get stuck
> down (which I use to switch to virtual desktop 3 which has my Konsoles
> on.) In this case, pressing all of ctrl-shift-alt would clear it.

So multiple keys being pressed at once can result in confusion
depending on how the keyboard matrix is set up, and pressing multiple
keys then causes ghost reports.

Usually it requires three keys to be pressed simultaneously - and some
really cheap underlying hardware without some basic N-key rollover
protection.

But who knows what can confuse the firmware.

And honestly, it might also be a timing issue. So when you switch
virtual desktops, you end up doing more CPU work, changing timings,
and messing something up in the firmware in the process.

For example, I wouldn't expect firmware to be great about SMP. So
while the i8042 driver serializes everything with i8042_lock, who
knows *where* the firmware runs.

Maybe we could do something like tie irq1 (keyboard) and irq12 ("aux",
aka mouse) to the boot CPU in the hopes that there's less chance of
confusing some firmware that way.

I have no idea what Windows does, and - as usual - that's the case
that gets almost all the testing from vendors.

> What also crosses my mind is that if the i8042 is now emulated by
> firmware, is there a replacement interface that the kernel should
> instead be using?

I don't think there is any documentation - or necessarily commonality
- for the low-level hardware. I would guess that it's probably some
small i2c controller that ends up doing some keypad matrix thing. That
i2c thing *may* do native HID, but it might easily also be just some
custom GPIO expander thing.

I think the touchpad is usually some i2c device, and it is sometimes
accessible both ways. And there's a long history of keyboard problems
when the touchpad is looked at just the wrong way, so those things
most definitely can interact (because the firmware emulation emulates
both).

It's very hard to find hardware information at that level these days.
It's been decades since things like the keyboard matrix was documented
as such....

                  Linus

Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa: microchip: Add SGMII port support to KSZ9477 switch)

[Peter Hutterer]

On Wed, May 07, 2025 at 07:18:37PM +0100, Russell King (Oracle) wrote:
> On Wed, May 07, 2025 at 10:45:48AM -0700, Dmitry Torokhov wrote:
> > On Wed, May 07, 2025 at 01:51:26PM +0200, Maxime Chevallier wrote:
> > > On Wed, 7 May 2025 12:44:24 +0100
> > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> > > 
> > > > Hi Maxime,
> > > > 
> > > > On Wed, May 07, 2025 at 03:32:36PM +0200, Maxime Chevallier wrote:
> > > > > Hi Russell,
> > > > > 
> > > > > On Wed, 7 May 2025 10:59:21 +0100
> > > > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:
> > > > >   
> > > > > > On Wed, May 07, 2025 at 10:23:17AM +0100, Russell King (Oracle) wrote:  
> > > > > > > [Sorry for going off topic here - changed the Cc list, added Linus,
> > > > > > > changed the subject.]
> > > > > > > 
> > > > > > > On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:    
> > > > > > > > On Wed, 7 May 2025 09:31:48 +0100
> > > > > > > > "Russell King (Oracle)" <linux@armlinux.org.uk> wrote:    
> > > > > > > > > [rest of the email got deleted because Linux / X11 / KDE got confused
> > > > > > > > > about the state the backspace key and decided it was going to be
> > > > > > > > > continuously pressed and doing nothing except shutting the laptop
> > > > > > > > > down would stop it.]    
> > > > > > > > 
> > > > > > > > Funny how I have the same exact issue on my laptop as well...     
> > > > > > > 
> > > > > > > I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
> > > > > > > laptop I had previously (normally with ctrl-F keys). However, hitting
> > > > > > > ctrl/shift/alt would stop it.
> > > > > > > 
> > > > > > > This is the first time I've seen the behaviour with the Carbon X1
> > > > > > > laptop, but this was way more severe. No key would stop it. Trying to
> > > > > > > move the focus using the trackpad/nipple had any effect. Meanwhile
> > > > > > > the email was being deleted one character at a time. So I shut the
> > 
> > If we indeed lost a key release event somewhere the way to "restore" it
> > is to hit the stuck key again. Then we should get press/release with
> > press most likely being ignored and release achieving the desired
> > result. Of course that will not help if embedded controller is confused.
> 
> I tried pressing every key...

ftr, any key autorepeat in XKB (or libxkbcommon) is cancelled when
another key arrives so this does indeed hint at a firmware/controller
issue.

> > > > > > The mysterious thing is "Keylock active" - clearly it isn't because I
> > > > > > can write this email typing on that very keyboard. However, I wonder
> > > > > > if it needs i8042_unlock=1 to set I8042_CTR_IGNKEYLOCK.
> > 
> > Just ignore this message, it is harmless and trying to flip the bit
> > might confuse the emulation even more. Maybe we should lower the
> > severity of it to debug.
> > 
> > That said I do not see it on my Carbon (neither v5 nor v12, can't check
> > v9 because it is at home)... What version of Carbon do you have? Do you
> > have up-to-date BIOS/EC?
> 
> Neither did I see a problem until today, and I've been using the laptop
> since October 2024, and this is the first time it's had an issue.
> 
> Looking at fwupd, it has an Intel ME update pending (1.32.2418 to
> 1.35.2557). I can't find a way to get any update history beyond
> that out of fwupd and fwupd doesn't seem to log to journald what
> it's doing.
> 
> > > > > It just happened to me as I was typing this very email (key 'd' got
> > > > > stuck, nothing could un-stick it, couldn't move the mouse cursor but
> > > > > mouse-click events did work, had to suspend/resume the laptop to fix
> > > > > that)
> > 
> > This is weird and suggests that the breakage happens up the stack from
> > the kernel (or down in the firmware). Mouse clicks and mouse movement is
> > delivered as part of a mouse packet, so if there are button clicks there
> > will also be movement, they are not separate. If the cursor is not
> > reacting that means desktop environment is not handling input properly.
> 
> So could we be looking at an Xorg bug?
> 
> > The kernel does drop input events if userspace is unable to read buffers
> > quickly enough. It notifies userspace by queuing special
> > EV_SYN/SYN_DROPPED event and userspace is supposed to query the full
> > device state upon receiving it to figure out what to do. I doubt we are
> > running into this with keyboards, but maybe we should add some logging
> > there to rule this out.
> > 
> > I'll add Peter and Benjamin to this thread in case they have ideas.
> 
> I'm thinking of leaving evtest running in a terminal, so its output
> can be inspected if it happens again. One issue though is the
> timestamps aren't readable, but I'm sure with a bit of perl
> post-processing that could be fixed.

$ screen sudo libinput record --autorestart=5 -o keyboard.yml

Select the keyboard devce and off you go.
This will create a keyboard.yml.$DATETIME file and after 5 seconds of no
events it will rotate to the next file. This means you can leave this
running for months and when it happens you stop typing for 5 seconds and
then look at the most recent (usually) two files for epiphanies.
Bonus points for ssh-ing in and killing the process remotely because
then you don't have interference of whatever

If the last event you get is a key down event the issue is in the
firmware and userspace does the right thing. In that case your 
logs will be full of EV_KEY KEY_FOO with a value of 2 (kernel repeat).

If the last event you get is a key up but the desktop still repeats
then the issue is in that part of the stack. In that case the log should
be empty after you stopped typing.

If you get a SYN_DROPPED and the desktop does the wrong thing the issue
is in libinput because it should handle that transparently (same with
the Xorg evdev driver if you're using that).

Optionally: --show-keycodes but adding this means you will leak
all your key presses into the logs (without this all alphanumeric keys
are recorded as KEY_A). So maybe only use that for entertainment value.

Optionally: provide the device node as arg (see below)

> That would allow an answer to "is it kernel or firmware" vs
> "userspace".
> 
> The problem is - if it's taken 7-ish months to show for me, it's
> likely that evtest won't be running when it next happens (as there
> will be needs to reboot for kernel upgrades/firmware upgrades in
> that time.) Really, it needs to be something like an automatically
> started at boot e.g. evtest inside a detached screen session.
 
Well, that bit I leave to you to figure out because it'll also require
scripting to find the keyboard's device node automatically so you can
pass it to libinput record via the .unit file, or cron job script, or
whatever.

Either way, this should at least make it easier to catch the issue when
it occurs.

Cheers,
  Peter