[REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Andy Mindful]

Kernel-version: 6.8+ (confirmed in Fedora kernel 6.8.4-200.fc38.x86_64
and above)
Regression-from: 6.7.11 (working) to 6.8.4+ (regressed)
Affected-hardware: Lenovo ThinkPad X1 Carbon Gen 10 (Intel Alder Lake platform)
Affected-subsystems: TTY/Console, Power Management, WWAN Modem (Intel
XMM7560, iosm driver), Kernel DRM/i915 stack

Description:
We are reporting a significant regression introduced in Linux kernel
version 6.8 (specifically confirmed in Fedora kernel
6.8.4-200.fc38.x86_64 and subsequent versions), affecting Lenovo
ThinkPad X1 Carbon Gen 10 laptops. This regression manifests primarily
as tty lockups and complete loss of WWAN modem functionality after
hibernate cycle.

Problematic Behavior:
1. TTY/Console: After a hibernate cycle, the console input becomes
unresponsive. In some cases, after suspend, pressing `SysRq+R` (unraw)
can temporarily restore keyboard functionality, but this is not
consistent, especially after hibernation.
2. Power Management: While S3 suspend works correctly when "Linux
only" is set in BIOS, `suspend` causes the system to freeze upon
resume if "Windows + Linux" mode (which implies `s2idle`) is active in
BIOS settings.
3. WWAN Modem (Intel XMM7560, `iosm` driver): The modem fails to
reinitialize after hibernation, consistently showing "msg timeout"
errors in logs. The modem works correctly after a cold boot but
completely fails to recover after resuming from hibernate. Rescanning
PCI devices or reloading the `iosm` module does not resolve the issue.
Attempts to manually remove the device via
`/sys/bus/pci/devices/.../remove` result in a system hang.
4. Kernel DRM/i915 stack: Although GuC and HuC firmware (versions
70.44.1 / 7.9.3) are confirmed to load correctly, logs indicate
potential graphics driver reinitialization issues in affected kernel
versions. This might contribute to the TTY unresponsiveness.

System Specifications:
- Laptop: Lenovo ThinkPad X1 Carbon Gen 10
- CPU: Intel Core i7-1260P (Alder Lake)
- GPU: Intel iGPU (ADL GT2) - i915
- WWAN: Intel XMM7560 (iosm driver)
- Wi-Fi/Bluetooth: Intel AX201
- Operating Systems tested:
- Fedora 38 (most stable with 6.7.11)
- Fedora 39/40/42 (exhibit regression)
- Ubuntu 22.04, 24.10 (exhibit regression)
- EndeavourOS_Mercury-Neo-2025.03.19 (exhibits regression)
- Display manager: GDM/Wayland (also tested LightDM + Xfce and
runlevel 3, problem persists).
- Mesa Stack: Mesa 24.3.4 built with LLVM 16.0.6 (confirmed functional
with working kernels). Vulkan, GBM, EGL, GLX render correctly with
`iris` and `intel` drivers. Custom Mesa 24.3.4 build showed no impact
on the hibernate issue.

Behavior Matrix:
| Kernel Version | Hibernate Status | Resume Status | TTY
Functionality | WWAN Functionality | Notes |
|----------------|------------------|---------------|-------------------|--------------------|-------|
| 6.2.9 | OK | Not tested | OK | Not Tested | Working baseline |
| 6.3.12 | OK | Not tested | OK | Not Tested | Working baseline |
| 6.7.11 | OK | OK(S3)/Fail(S2idle) | OK | FAIL | WWAN regression
starts here, but hibernate/tty still OK |
| **6.8.4+** | FAIL | OK(S3)/Fail(S2idle) | FAIL | FAIL | Major
regression point, affects hibernate/tty/WWAN |
| 6.14+ | FAIL | OK(S3)/Fail(S2idle) | FAIL | FAIL | Problem persists
in newer kernels |

Additional Details:
- Hibernation issues:
- Initial `mem_sleep` was `[s2idle]` in BIOS "Windows + Linux" mode.
- Switching BIOS to "Linux only" enabled proper S3 suspend, changing
`/sys/power/mem_sleep` to `[s2idle] deep`. However, the hibernation
issue (TTY lockup, WWAN loss) persists even with S3 enabled for
suspend.
- `Alt + SysRq + R` sometimes revives console after suspend but
*never* after hibernate.
- TTY related errors observed in logs are, dmesg:
`tty_flip_buffer_push: called while buffer locked`, systemd-logind:
`New session created but VT is not initial VT, ignoring`, GDM:
`GdmLocalDisplayFactory: active VT is not initial VT` on kernels 6.8+.
- WWAN issues:
- The `iosm` driver appears loaded, and the device is visible via
`lspci`, but the modem is not visible to `mmcli` after resume from
hibernate.
- The WWAN modem (iosm) only works after a cold boot.
- Rescanning PCI or reloading `iosm` module (e.g., `modprobe -r iosm
&& modprobe iosm`) does not resolve the issue.
- Manual removal via `/sys/bus/pci/devices/.../remove` leads to a
system hang, indicating a deeper issue with device state or driver
interaction post-resume.

Tested Alternatives & Current Stability:
- Hibernate works reliably only up to kernel 6.7.11.
- Fedora 38 with kernel 6.7.11 + GNOME 44.10 is currently the most
stable configuration for this hardware, despite the WWAN issue already
present there (but not the tty lockup/hibernate issue).
- Kali Linux with 6.6.9 - hibernation works.
- Newer distributions (e.g., Fedora 42, Ubuntu 25.04, EndeavourOS)
inherit the same problems due to their newer kernel versions.

Expected Behavior:
The system should resume from hibernate and suspend without TTY
lockups, and the WWAN modem should reinitialize correctly and be fully
functional.

Steps to Reproduce:
1. Ensure system is running a kernel version 6.8.4 or newer (e.g.,
Fedora 38 with `6.8.4-200.fc38.x86_64`).
2. Perform a `systemctl hibernate` or `systemctl suspend`.
3. Resume the system.
4. Observe TTY console unresponsiveness(only in runlevel 3) and check
WWAN modem status using `mmcli -L`.

Please let me know if any further information or testing is required.

Thank you.

Best regards,
Andriy

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Jani Nikula]

On Tue, 29 Jul 2025, Andy Mindful <andy.mindful@gmail.com> wrote:
> Please let me know if any further information or testing is required.

Likely the quickest way to find the root cause is to bisect the issue.


BR,
Jani.

-- 
Jani Nikula, Intel

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Andy Mindful]

Here what I have from bisecting, please let me know if it makes things
clearer or not.

git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1] Linux 6.7.11
git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
# status: waiting for bad commit, 1 good commit known
# bad: [6613476e225e090cc9aad49be7fa504e290dd33d] Linux 6.8-rc1
git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
# skip: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a
# good: [ba5afb9a84df2e6b26a1b6389b98849cd16ea757] fs: rework
listmount() implementation
git bisect good ba5afb9a84df2e6b26a1b6389b98849cd16ea757
# good: [61da593f4458f25c59f65cfd9ba1bda570db5db7] Merge tag
'media/v6.8-2' of
git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
git bisect good 61da593f4458f25c59f65cfd9ba1bda570db5db7
# bad: [e38f734add21d75d76dbcf7b214f4823131c1bae] Merge tag
'staging-6.8-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect bad e38f734add21d75d76dbcf7b214f4823131c1bae
# bad: [5d197e97fb106c09d3d013be341e5961fd70ec8a] Merge tag
'hsi-for-6.8' of
git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi
git bisect bad 5d197e97fb106c09d3d013be341e5961fd70ec8a
# good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
'docs-6.8-2' of git://git.lwn.net/linux
git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
# good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
'docs-6.8-2' of git://git.lwn.net/linux
git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
# bad: [8c9244af4dc8680a453e759331f0c93d5bde1898] Merge tag
'kvm-x86-svm-6.8' of https://github.com/kvm-x86/linux into HEAD
git bisect bad 8c9244af4dc8680a453e759331f0c93d5bde1898
# bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
missing "depends on KVM"
git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
# bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
missing "depends on KVM"
git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
# bad: [861deac3b092f37b2c5e6871732f3e11486f7082] Linux 6.7-rc7
git bisect bad 861deac3b092f37b2c5e6871732f3e11486f7082

# being on a 861deac3b092
git bisect bad
The merge base ba5afb9a84df2e6b26a1b6389b98849cd16ea757 is bad.
This means the bug has been fixed between
ba5afb9a84df2e6b26a1b6389b98849cd16ea757 and
[1b1934dbbdcf9aa2d507932ff488cec47999cf3f
61da593f4458f25c59f65cfd9ba1bda570db5db7
6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
ba5afb9a84df2e6b26a1b6389b98849cd16ea757].

Thanks.

вт, 29 лип. 2025 р. о 13:20 Jani Nikula <jani.nikula@intel.com> пише:
>
> On Tue, 29 Jul 2025, Andy Mindful <andy.mindful@gmail.com> wrote:
> > Please let me know if any further information or testing is required.
>
> Likely the quickest way to find the root cause is to bisect the issue.
>
>
> BR,
> Jani.
>
> --
> Jani Nikula, Intel

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Jani Nikula]

On Sun, 03 Aug 2025, Andy Mindful <andy.mindful@gmail.com> wrote:
> Here what I have from bisecting, please let me know if it makes things
> clearer or not.

Doesn't point at graphics, anyway.

Cc: Christian.


BR,
Jani.


>
> git bisect log
> git bisect start
> # status: waiting for both good and bad commits
> # good: [6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1] Linux 6.7.11
> git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> # status: waiting for bad commit, 1 good commit known
> # bad: [6613476e225e090cc9aad49be7fa504e290dd33d] Linux 6.8-rc1
> git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
> # skip: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
> git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a
> # good: [ba5afb9a84df2e6b26a1b6389b98849cd16ea757] fs: rework
> listmount() implementation
> git bisect good ba5afb9a84df2e6b26a1b6389b98849cd16ea757
> # good: [61da593f4458f25c59f65cfd9ba1bda570db5db7] Merge tag
> 'media/v6.8-2' of
> git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
> git bisect good 61da593f4458f25c59f65cfd9ba1bda570db5db7
> # bad: [e38f734add21d75d76dbcf7b214f4823131c1bae] Merge tag
> 'staging-6.8-rc1' of
> git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> git bisect bad e38f734add21d75d76dbcf7b214f4823131c1bae
> # bad: [5d197e97fb106c09d3d013be341e5961fd70ec8a] Merge tag
> 'hsi-for-6.8' of
> git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi
> git bisect bad 5d197e97fb106c09d3d013be341e5961fd70ec8a
> # good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
> 'docs-6.8-2' of git://git.lwn.net/linux
> git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> # good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
> 'docs-6.8-2' of git://git.lwn.net/linux
> git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> # bad: [8c9244af4dc8680a453e759331f0c93d5bde1898] Merge tag
> 'kvm-x86-svm-6.8' of https://github.com/kvm-x86/linux into HEAD
> git bisect bad 8c9244af4dc8680a453e759331f0c93d5bde1898
> # bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
> missing "depends on KVM"
> git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
> # bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
> missing "depends on KVM"
> git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
> # bad: [861deac3b092f37b2c5e6871732f3e11486f7082] Linux 6.7-rc7
> git bisect bad 861deac3b092f37b2c5e6871732f3e11486f7082
>
> # being on a 861deac3b092
> git bisect bad
> The merge base ba5afb9a84df2e6b26a1b6389b98849cd16ea757 is bad.
> This means the bug has been fixed between
> ba5afb9a84df2e6b26a1b6389b98849cd16ea757 and
> [1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> 61da593f4458f25c59f65cfd9ba1bda570db5db7
> 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> ba5afb9a84df2e6b26a1b6389b98849cd16ea757].
>
> Thanks.
>
> вт, 29 лип. 2025 р. о 13:20 Jani Nikula <jani.nikula@intel.com> пише:
>>
>> On Tue, 29 Jul 2025, Andy Mindful <andy.mindful@gmail.com> wrote:
>> > Please let me know if any further information or testing is required.
>>
>> Likely the quickest way to find the root cause is to bisect the issue.
>>
>>
>> BR,
>> Jani.
>>
>> --
>> Jani Nikula, Intel

-- 
Jani Nikula, Intel

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Andy Mindful]

Double-checked bisect, looks like I've have found broken commit:

git bisect start
# status: waiting for both good and bad commits
# good: [6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1] Linux 6.7.11
git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
# status: waiting for bad commit, 1 good commit known
# bad: [6613476e225e090cc9aad49be7fa504e290dd33d] Linux 6.8-rc1
git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
# skip: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a
# good: [ba5afb9a84df2e6b26a1b6389b98849cd16ea757] fs: rework
listmount() implementation
git bisect good ba5afb9a84df2e6b26a1b6389b98849cd16ea757
# good: [61da593f4458f25c59f65cfd9ba1bda570db5db7] Merge tag
'media/v6.8-2' of
git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
git bisect good 61da593f4458f25c59f65cfd9ba1bda570db5db7
# bad: [e38f734add21d75d76dbcf7b214f4823131c1bae] Merge tag
'staging-6.8-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect bad e38f734add21d75d76dbcf7b214f4823131c1bae
# bad: [5d197e97fb106c09d3d013be341e5961fd70ec8a] Merge tag
'hsi-for-6.8' of
git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi
git bisect bad 5d197e97fb106c09d3d013be341e5961fd70ec8a
# good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
'docs-6.8-2' of git://git.lwn.net/linux
git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
# bad: [8c9244af4dc8680a453e759331f0c93d5bde1898] Merge tag
'kvm-x86-svm-6.8' of https://github.com/kvm-x86/linux into HEAD
git bisect bad 8c9244af4dc8680a453e759331f0c93d5bde1898
# bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
missing "depends on KVM"
git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
# bad: [1c3c87d720cbd1ff86dc1bfc6df8ee9adce5879b] Merge tag
'kvm-x86-selftests-6.7-rcN' of https://github.com/kvm-x86/linux into
HEAD
git bisect bad 1c3c87d720cbd1ff86dc1bfc6df8ee9adce5879b
# good: [8d99e347c097ab3f9fb93d0f88dddf20051d7c88] KVM: selftests:
Convert lib's mem regions to KVM_SET_USER_MEMORY_REGION2
git bisect good 8d99e347c097ab3f9fb93d0f88dddf20051d7c88
# bad: [6c370dc65374db5afbc5c6c64c662f922a2555ad] Merge branch
'kvm-guestmemfd' into HEAD
git bisect bad 6c370dc65374db5afbc5c6c64c662f922a2555ad
# good: [43f623f350ce1c46c53b6b77f4dbe741af8c44f3] KVM: selftests: Add
x86-only selftest for private memory conversions
git bisect good 43f623f350ce1c46c53b6b77f4dbe741af8c44f3
# good: [8a89efd43423cb3005c5e641e846184e292c1465] KVM: selftests: Add
basic selftest for guest_memfd()
git bisect good 8a89efd43423cb3005c5e641e846184e292c1465
# good: [5d74316466f4aabdd2ee1e33b45e4933c9bc3ea1] KVM: selftests: Add
a memory region subtest to validate invalid flags
git bisect good 5d74316466f4aabdd2ee1e33b45e4933c9bc3ea1
# first bad commit: [6c370dc65374db5afbc5c6c64c662f922a2555ad] Merge
branch 'kvm-guestmemfd' into HEAD

пн, 4 серп. 2025 р. о 11:15 Jani Nikula <jani.nikula@intel.com> пише:
>
> On Sun, 03 Aug 2025, Andy Mindful <andy.mindful@gmail.com> wrote:
> > Here what I have from bisecting, please let me know if it makes things
> > clearer or not.
>
> Doesn't point at graphics, anyway.
>
> Cc: Christian.
>
>
> BR,
> Jani.
>
>
> >
> > git bisect log
> > git bisect start
> > # status: waiting for both good and bad commits
> > # good: [6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1] Linux 6.7.11
> > git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> > # status: waiting for bad commit, 1 good commit known
> > # bad: [6613476e225e090cc9aad49be7fa504e290dd33d] Linux 6.8-rc1
> > git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
> > # skip: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
> > git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a
> > # good: [ba5afb9a84df2e6b26a1b6389b98849cd16ea757] fs: rework
> > listmount() implementation
> > git bisect good ba5afb9a84df2e6b26a1b6389b98849cd16ea757
> > # good: [61da593f4458f25c59f65cfd9ba1bda570db5db7] Merge tag
> > 'media/v6.8-2' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
> > git bisect good 61da593f4458f25c59f65cfd9ba1bda570db5db7
> > # bad: [e38f734add21d75d76dbcf7b214f4823131c1bae] Merge tag
> > 'staging-6.8-rc1' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> > git bisect bad e38f734add21d75d76dbcf7b214f4823131c1bae
> > # bad: [5d197e97fb106c09d3d013be341e5961fd70ec8a] Merge tag
> > 'hsi-for-6.8' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi
> > git bisect bad 5d197e97fb106c09d3d013be341e5961fd70ec8a
> > # good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
> > 'docs-6.8-2' of git://git.lwn.net/linux
> > git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > # good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
> > 'docs-6.8-2' of git://git.lwn.net/linux
> > git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > # bad: [8c9244af4dc8680a453e759331f0c93d5bde1898] Merge tag
> > 'kvm-x86-svm-6.8' of https://github.com/kvm-x86/linux into HEAD
> > git bisect bad 8c9244af4dc8680a453e759331f0c93d5bde1898
> > # bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
> > missing "depends on KVM"
> > git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
> > # bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
> > missing "depends on KVM"
> > git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
> > # bad: [861deac3b092f37b2c5e6871732f3e11486f7082] Linux 6.7-rc7
> > git bisect bad 861deac3b092f37b2c5e6871732f3e11486f7082
> >
> > # being on a 861deac3b092
> > git bisect bad
> > The merge base ba5afb9a84df2e6b26a1b6389b98849cd16ea757 is bad.
> > This means the bug has been fixed between
> > ba5afb9a84df2e6b26a1b6389b98849cd16ea757 and
> > [1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > 61da593f4458f25c59f65cfd9ba1bda570db5db7
> > 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> > ba5afb9a84df2e6b26a1b6389b98849cd16ea757].
> >
> > Thanks.
> >
> > вт, 29 лип. 2025 р. о 13:20 Jani Nikula <jani.nikula@intel.com> пише:
> >>
> >> On Tue, 29 Jul 2025, Andy Mindful <andy.mindful@gmail.com> wrote:
> >> > Please let me know if any further information or testing is required.
> >>
> >> Likely the quickest way to find the root cause is to bisect the issue.
> >>
> >>
> >> BR,
> >> Jani.
> >>
> >> --
> >> Jani Nikula, Intel
>
> --
> Jani Nikula, Intel

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Christian Brauner]

On Mon, Aug 04, 2025 at 11:15:46AM +0300, Jani Nikula wrote:
> On Sun, 03 Aug 2025, Andy Mindful <andy.mindful@gmail.com> wrote:
> > Here what I have from bisecting, please let me know if it makes things
> > clearer or not.
> 
> Doesn't point at graphics, anyway.
> 
> Cc: Christian.

No idea, the listmount() changes look completely unrelated and it's way
back in v6.8 so absolutely no clue. Should probably go talk to
hibernate/suspend/power people or talk to Thorsten to bisect this
further.

> 
> 
> BR,
> Jani.
> 
> 
> >
> > git bisect log
> > git bisect start
> > # status: waiting for both good and bad commits
> > # good: [6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1] Linux 6.7.11
> > git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> > # status: waiting for bad commit, 1 good commit known
> > # bad: [6613476e225e090cc9aad49be7fa504e290dd33d] Linux 6.8-rc1
> > git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
> > # skip: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
> > git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a
> > # good: [ba5afb9a84df2e6b26a1b6389b98849cd16ea757] fs: rework
> > listmount() implementation
> > git bisect good ba5afb9a84df2e6b26a1b6389b98849cd16ea757
> > # good: [61da593f4458f25c59f65cfd9ba1bda570db5db7] Merge tag
> > 'media/v6.8-2' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
> > git bisect good 61da593f4458f25c59f65cfd9ba1bda570db5db7
> > # bad: [e38f734add21d75d76dbcf7b214f4823131c1bae] Merge tag
> > 'staging-6.8-rc1' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
> > git bisect bad e38f734add21d75d76dbcf7b214f4823131c1bae
> > # bad: [5d197e97fb106c09d3d013be341e5961fd70ec8a] Merge tag
> > 'hsi-for-6.8' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-hsi
> > git bisect bad 5d197e97fb106c09d3d013be341e5961fd70ec8a
> > # good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
> > 'docs-6.8-2' of git://git.lwn.net/linux
> > git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > # good: [1b1934dbbdcf9aa2d507932ff488cec47999cf3f] Merge tag
> > 'docs-6.8-2' of git://git.lwn.net/linux
> > git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > # bad: [8c9244af4dc8680a453e759331f0c93d5bde1898] Merge tag
> > 'kvm-x86-svm-6.8' of https://github.com/kvm-x86/linux into HEAD
> > git bisect bad 8c9244af4dc8680a453e759331f0c93d5bde1898
> > # bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
> > missing "depends on KVM"
> > git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
> > # bad: [783288010035e4c250a0b6491a4642cdb8d30548] KVM: x86: add
> > missing "depends on KVM"
> > git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
> > # bad: [861deac3b092f37b2c5e6871732f3e11486f7082] Linux 6.7-rc7
> > git bisect bad 861deac3b092f37b2c5e6871732f3e11486f7082
> >
> > # being on a 861deac3b092
> > git bisect bad
> > The merge base ba5afb9a84df2e6b26a1b6389b98849cd16ea757 is bad.
> > This means the bug has been fixed between
> > ba5afb9a84df2e6b26a1b6389b98849cd16ea757 and
> > [1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > 61da593f4458f25c59f65cfd9ba1bda570db5db7
> > 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> > ba5afb9a84df2e6b26a1b6389b98849cd16ea757].
> >
> > Thanks.
> >
> > вт, 29 лип. 2025 р. о 13:20 Jani Nikula <jani.nikula@intel.com> пише:
> >>
> >> On Tue, 29 Jul 2025, Andy Mindful <andy.mindful@gmail.com> wrote:
> >> > Please let me know if any further information or testing is required.
> >>
> >> Likely the quickest way to find the root cause is to bisect the issue.
> >>
> >>
> >> BR,
> >> Jani.
> >>
> >> --
> >> Jani Nikula, Intel
> 
> -- 
> Jani Nikula, Intel

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Paolo Bonzini]

On Mon, Aug 4, 2025 at 12:57 PM Andy Mindful <andy.mindful@gmail.com> wrote:
> Double-checked bisect, looks like I've have found broken commit:
>
> > > git bisect bad
> > > The merge base ba5afb9a84df2e6b26a1b6389b98849cd16ea757 is bad.
> > > This means the bug has been fixed between
> > > ba5afb9a84df2e6b26a1b6389b98849cd16ea757 and
> > > [1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > > 61da593f4458f25c59f65cfd9ba1bda570db5db7
> > > 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> > > ba5afb9a84df2e6b26a1b6389b98849cd16ea757].

This skip is messing up the results:

# skip: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a

and there are still 3858 commits in
ba5afb9a84df2e6b26a1b6389b98849cd16ea757..{1b1934dbbdcf9aa2d507932ff488cec47999cf3f,61da593f4458f25c59f65cfd9ba1bda570db5db7,ba5afb9a84df2e6b26a1b6389b98849cd16ea757}

Any chance you can get 6.7 to work and restrict the range further?

Thanks,

Paolo

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Andy Mindful]

Can you please advise on how to bisect further?

andy@lenovo:~/linux-stable$ git bisect bad
The merge base 0dd3ee31125508cd67f7e7172247f05b7fd1753a is bad.
This means the bug has been fixed between
0dd3ee31125508cd67f7e7172247f05b7fd1753a and
[6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1].

andy@DESKTOP-0R165CF:~/linux-stable$ git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1] Linux 6.7.11
git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
# status: waiting for bad commit, 1 good commit known
# bad: [6613476e225e090cc9aad49be7fa504e290dd33d] Linux 6.8-rc1
git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
# bad: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
git bisect bad 0dd3ee31125508cd67f7e7172247f05b7fd1753a

andy@lenovo:~/linux-stable$ git status
HEAD detached at 0dd3ee311255
You are currently bisecting, started from branch '6fc5460ed8dd'.
(use "git bisect reset" to get back to the original branch)

It is not moving further.

пн, 4 серп. 2025 р. о 17:50 Paolo Bonzini <pbonzini@redhat.com> пише:
>
> On Mon, Aug 4, 2025 at 12:57 PM Andy Mindful <andy.mindful@gmail.com> wrote:
> > Double-checked bisect, looks like I've have found broken commit:
> >
> > > > git bisect bad
> > > > The merge base ba5afb9a84df2e6b26a1b6389b98849cd16ea757 is bad.
> > > > This means the bug has been fixed between
> > > > ba5afb9a84df2e6b26a1b6389b98849cd16ea757 and
> > > > [1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > > > 61da593f4458f25c59f65cfd9ba1bda570db5db7
> > > > 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> > > > ba5afb9a84df2e6b26a1b6389b98849cd16ea757].
>
> This skip is messing up the results:
>
> # skip: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
> git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a
>
> and there are still 3858 commits in
> ba5afb9a84df2e6b26a1b6389b98849cd16ea757..{1b1934dbbdcf9aa2d507932ff488cec47999cf3f,61da593f4458f25c59f65cfd9ba1bda570db5db7,ba5afb9a84df2e6b26a1b6389b98849cd16ea757}
>
> Any chance you can get 6.7 to work and restrict the range further?
>
> Thanks,
>
> Paolo
>

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Maciej S. Szmigiero]

On 29.07.2025 12:03, Andy Mindful wrote:
> Kernel-version: 6.8+ (confirmed in Fedora kernel 6.8.4-200.fc38.x86_64
> and above)
> Regression-from: 6.7.11 (working) to 6.8.4+ (regressed)
> Affected-hardware: Lenovo ThinkPad X1 Carbon Gen 10 (Intel Alder Lake platform)
> Affected-subsystems: TTY/Console, Power Management, WWAN Modem (Intel
> XMM7560, iosm driver), Kernel DRM/i915 stack
> 
> Description:
> We are reporting a significant regression introduced in Linux kernel
> version 6.8 (specifically confirmed in Fedora kernel
> 6.8.4-200.fc38.x86_64 and subsequent versions), affecting Lenovo
> ThinkPad X1 Carbon Gen 10 laptops. This regression manifests primarily
> as tty lockups and complete loss of WWAN modem functionality after
> hibernate cycle.
> 
> Problematic Behavior:
(..)
> 3. WWAN Modem (Intel XMM7560, `iosm` driver): The modem fails to
> reinitialize after hibernation, consistently showing "msg timeout"
> errors in logs. The modem works correctly after a cold boot but
> completely fails to recover after resuming from hibernate. Rescanning
> PCI devices or reloading the `iosm` module does not resolve the issue.
> Attempts to manually remove the device via
> `/sys/bus/pci/devices/.../remove` result in a system hang.

Do you have this commit in your test kernel?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0b6f6593aa8c3a05f155c12fd0e7ad33a5149c31

With this commit you should *not* see any "msg timeout" on resume
from hibernation.
Either the modem will reinitialize successfully OR it will be
completely dead and the iosm driver will refuse to re-bind to its
PCI device in this case.

The root cause of these issues seems to be that some platforms
cut the power to this modem on S3 and/or S4 entry without abstracting
this power control using proper power management interfaces.

Some platforms won't even restore modem power on resume until a
device-specific reset ACPI method is called.
This means the modem device PCI configuration space data won't be
restored correctly on resume and so the device will be effectively
dead until reboot.

ModemManager has a whole issue dedicated to tracking power management
issues in a very similar Intel XMM7360 modem on various platforms:
https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/992

AFAIK the major difference between XMM7360 and XMM7560 is that
XMM7360 speaks proprietary XMMRPC protocol while XMM7560 speaks MBIM,
but this should not make a difference here.

Also, this is probably not a regression, just something that
has never worked correctly (I mean the WWAN modem part).
If it indeed used to work then a kernel log from a
hibernate/restore cycle on a working kernel version would be nice
to have.

> 
> Best regards,
> Andriy
> 
Thanks,
Maciej

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Andy Mindful]

Hello,

Can somebody advise how to properly bisect issues in linux-stable
repository between v6.7.11 to v6.8-rc1 tags?

I tried two options:

1. No skip

git checkout v6.7.11
git bisect start
git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
git bisect bad 0dd3ee31125508cd67f7e7172247f05b7fd1753a

Leads to:
The merge base 0dd3ee31125508cd67f7e7172247f05b7fd1753a is bad.
This means the bug has been fixed between
0dd3ee31125508cd67f7e7172247f05b7fd1753a and
[6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1].

What should be done next?

2. Skip
git checkout v6.7.11
git bisect start
git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
git bisect bad 0dd3ee31125508cd67f7e7172247f05b7fd1753a
git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a
git bisect good ba5afb9a84df2e6b26a1b6389b98849cd16ea757
git bisect good 61da593f4458f25c59f65cfd9ba1bda570db5db7
git bisect bad e38f734add21d75d76dbcf7b214f4823131c1bae
git bisect bad 5d197e97fb106c09d3d013be341e5961fd70ec8a
git bisect good 1b1934dbbdcf9aa2d507932ff488cec47999cf3f
git bisect bad 8c9244af4dc8680a453e759331f0c93d5bde1898
git bisect bad 783288010035e4c250a0b6491a4642cdb8d30548
git bisect bad 1c3c87d720cbd1ff86dc1bfc6df8ee9adce5879b
git bisect good 8d99e347c097ab3f9fb93d0f88dddf20051d7c88
git bisect bad 6c370dc65374db5afbc5c6c64c662f922a2555ad
git bisect good 43f623f350ce1c46c53b6b77f4dbe741af8c44f3
git bisect good 8a89efd43423cb3005c5e641e846184e292c1465
git bisect good 5d74316466f4aabdd2ee1e33b45e4933c9bc3ea1

Leads to:

# first bad commit: [6c370dc65374db5afbc5c6c64c662f922a2555ad] Merge
branch 'kvm-guestmemfd' into HEAD

Which is incorrect as per Paolo Bonzini comment.

I'd like to test this thing, because hibernation is quite crucial in a
multi-boot environment and may save a lot of time while working with
different systems.

Thank you in advance.

пн, 4 серп. 2025 р. о 22:49 Andy Mindful <andy.mindful@gmail.com> пише:
>
> Can you please advise on how to bisect further?
>
> andy@lenovo:~/linux-stable$ git bisect bad
> The merge base 0dd3ee31125508cd67f7e7172247f05b7fd1753a is bad.
> This means the bug has been fixed between
> 0dd3ee31125508cd67f7e7172247f05b7fd1753a and
> [6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1].
>
> andy@DESKTOP-0R165CF:~/linux-stable$ git bisect log
> git bisect start
> # status: waiting for both good and bad commits
> # good: [6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1] Linux 6.7.11
> git bisect good 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> # status: waiting for bad commit, 1 good commit known
> # bad: [6613476e225e090cc9aad49be7fa504e290dd33d] Linux 6.8-rc1
> git bisect bad 6613476e225e090cc9aad49be7fa504e290dd33d
> # bad: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
> git bisect bad 0dd3ee31125508cd67f7e7172247f05b7fd1753a
>
> andy@lenovo:~/linux-stable$ git status
> HEAD detached at 0dd3ee311255
> You are currently bisecting, started from branch '6fc5460ed8dd'.
> (use "git bisect reset" to get back to the original branch)
>
> It is not moving further.
>
> пн, 4 серп. 2025 р. о 17:50 Paolo Bonzini <pbonzini@redhat.com> пише:
> >
> > On Mon, Aug 4, 2025 at 12:57 PM Andy Mindful <andy.mindful@gmail.com> wrote:
> > > Double-checked bisect, looks like I've have found broken commit:
> > >
> > > > > git bisect bad
> > > > > The merge base ba5afb9a84df2e6b26a1b6389b98849cd16ea757 is bad.
> > > > > This means the bug has been fixed between
> > > > > ba5afb9a84df2e6b26a1b6389b98849cd16ea757 and
> > > > > [1b1934dbbdcf9aa2d507932ff488cec47999cf3f
> > > > > 61da593f4458f25c59f65cfd9ba1bda570db5db7
> > > > > 6fc5460ed8dd0edf29e7c5cfb1ef9b1aa04208a1
> > > > > ba5afb9a84df2e6b26a1b6389b98849cd16ea757].
> >
> > This skip is messing up the results:
> >
> > # skip: [0dd3ee31125508cd67f7e7172247f05b7fd1753a] Linux 6.7
> > git bisect skip 0dd3ee31125508cd67f7e7172247f05b7fd1753a
> >
> > and there are still 3858 commits in
> > ba5afb9a84df2e6b26a1b6389b98849cd16ea757..{1b1934dbbdcf9aa2d507932ff488cec47999cf3f,61da593f4458f25c59f65cfd9ba1bda570db5db7,ba5afb9a84df2e6b26a1b6389b98849cd16ea757}
> >
> > Any chance you can get 6.7 to work and restrict the range further?
> >
> > Thanks,
> >
> > Paolo
> >

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Paolo Bonzini]

On Wed, Aug 6, 2025 at 10:40 AM Andy Mindful <andy.mindful@gmail.com> wrote:
>
> Hello,
>
> Can somebody advise how to properly bisect issues in linux-stable
> repository between v6.7.11 to v6.8-rc1 tags?

If you can test 6.7 and bisect between 6.7 and 6.8-rc1 that would be
the easiest.

Paolo

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Thorsten Leemhuis]

On 06.08.25 10:40, Andy Mindful wrote:
> 
> Can somebody advise how to properly bisect issues in linux-stable
> repository between v6.7.11 to v6.8-rc1 tags?

You can't directly bisect, as their development paths diverged with
v6.7; so if 6.7.11 is working, it might be due to a change that was
added between 6.7 and 6.7.11 (this is unlikely, but a possibility). So
you have to check if 6.7 is working and then bisect between 6.7 and
6.8-rc1 (in case 6.7 was fine) or between 6.7 and 6.7.11 (in case it was
not).

But TBH: from your bisection results you shared in this thread so far it
looks a bit like your test is unreliable. Likely because it's a timing
issue or something like that which makes things work or not. That, the
age of kernels you try to bisect, and what Maciej explained elsewhere in
this thread at
https://lore.kernel.org/all/cfc561fb-29a9-4a19-a2e5-0f3f28e5e63a@maciej.szmigiero.name/
makes me think that it's not worth a bisection, as I guess the result is
unlikely to change anything (but I could be wrong).

Ciao, Thorsten

Re: [REGRESSION] tty lockup and WWAN loss after hibernate/suspend in 6.8+ on ThinkPad X1 Carbon Gen 10

[Andy Mindful]

Hello,

I've tried to bisect this again on a branch 'linux-6.8.y' @ linux-stable.

I made a commit log via: git log --oneline > commits-linux-6.8.y.txt
And started manually searching bad and good commits, this is the log:
3a2bd2b150f4 - bad
35a4474b5c3d - bad
dc97f6344f20 - bad
b85ea95d0864 - bad
ffc253263a13 - good

Once I have good, I switched back to: git checkout linux-6.8.y and
started bisecting,
which lead to below log:

git bisect start
# status: waiting for both good and bad commits
# bad: [3a2bd2b150f45844d99267696256716e4f23bf53] printk: Use
prb_first_seq() as base for 32bit seq macros
git bisect bad 3a2bd2b150f45844d99267696256716e4f23bf53
# status: waiting for good commit(s), bad commit known
# bad: [35a4474b5c3dd4315f72bd53e87b97f128d9bb3d] Merge tag
'bcachefs-2024-01-21' of https://evilpiepirate.org/git/bcachefs
git bisect bad 35a4474b5c3dd4315f72bd53e87b97f128d9bb3d
# status: waiting for good commit(s), bad commit known
# bad: [dc97f6344f205b0dfa144e1b3e16d6dc05383d57] cxl/pci: Register
for and process CPER events
git bisect bad dc97f6344f205b0dfa144e1b3e16d6dc05383d57
# status: waiting for good commit(s), bad commit known
# bad: [b85ea95d086471afb4ad062012a4d73cd328fa86] Linux 6.7-rc1
git bisect bad b85ea95d086471afb4ad062012a4d73cd328fa86
# status: waiting for good commit(s), bad commit known
# good: [ffc253263a1375a65fa6c9f62a893e9767fbebfa] Linux 6.6
git bisect good ffc253263a1375a65fa6c9f62a893e9767fbebfa
# good: [7d461b291e65938f15f56fe58da2303b07578a76] Merge tag
'drm-next-2023-10-31-1' of git://anongit.freedesktop.org/drm/drm
git bisect good 7d461b291e65938f15f56fe58da2303b07578a76
# good: [2a80532c0745e140852e6b579bbe8371332bb45d] Merge tag
'printk-for-6.7' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux
git bisect good 2a80532c0745e140852e6b579bbe8371332bb45d
# good: [2a80532c0745e140852e6b579bbe8371332bb45d] Merge tag
'printk-for-6.7' of
git://git.kernel.org/pub/scm/linux/kernel/git/printk/linux
git bisect good 2a80532c0745e140852e6b579bbe8371332bb45d
# bad: [e70703890b2586bc3567365d391c260d23fb7a94] Merge tag
'topic/nvidia-gsp-2023-11-03' of git://anongit.freedesktop.org/drm/drm
git bisect bad e70703890b2586bc3567365d391c260d23fb7a94
# good: [b06f58ad8e8c4154bc88d83b4fd70f74ede50193] Merge tag
'driver-core-6.7-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core
git bisect good b06f58ad8e8c4154bc88d83b4fd70f74ede50193
# good: [1f24458a1071f006e3f7449c08ae0f12af493923] Merge tag
'tty-6.7-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
git bisect good 1f24458a1071f006e3f7449c08ae0f12af493923
# bad: [bfafa2c19d706ab1db0b581f9d3886469fab8627] Merge tag
'phy-for-6.7' of
git://git.kernel.org/pub/scm/linux/kernel/git/phy/linux-phy
git bisect bad bfafa2c19d706ab1db0b581f9d3886469fab8627
# bad: [9b6db9a3a675fc2f33b587a9909dcef20c4b3794] Merge tag
'thunderbolt-for-v6.7-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/westeri/thunderbolt into
usb-next
git bisect bad 9b6db9a3a675fc2f33b587a9909dcef20c4b3794
# good: [efa33cd89c8acc510ed0e4dbf98fc1cf3220b9dc] USB: typec:
tps6598x: Add status trace for tps25750
git bisect good efa33cd89c8acc510ed0e4dbf98fc1cf3220b9dc
# bad: [ff8e4630fa3c1f8775f162ceaf52ba8b656927de] usb: gadget:
lpc32xx-udc: Convert to use module_platform_driver()
git bisect bad ff8e4630fa3c1f8775f162ceaf52ba8b656927de
# bad: [c087fada0a6180ab5b88b11c1776eef02f8d556f] xhci: Clean up stale
comment on ERST_SIZE macro
git bisect bad c087fada0a6180ab5b88b11c1776eef02f8d556f
# bad: [5220d8b04a840fa09434072c866d032b163419e3] dt-bindings: usb:
gpio-sbu-mux: Make 'mode-switch' not required
git bisect bad 5220d8b04a840fa09434072c866d032b163419e3
# bad: [1034cc423f1b4a7a9a56d310ca980fcd2753e11d] gpio: update Intel
LJCA USB GPIO driver
git bisect bad 1034cc423f1b4a7a9a56d310ca980fcd2753e11d
# bad: [bfd3824c88081f9b2101d68376f14779ce26691f] i2c: Add support for
Intel LJCA USB I2C driver
git bisect bad bfd3824c88081f9b2101d68376f14779ce26691f
# good: [acd6199f195d6de814ac4090ce0864a613b1580e] usb: Add support
for Intel LJCA device
git bisect good acd6199f195d6de814ac4090ce0864a613b1580e
# first bad commit: [bfd3824c88081f9b2101d68376f14779ce26691f] i2c:
Add support for Intel LJCA USB I2C driver

Can you please advise, if it makes sense and results are accurate?

Best regards!
Andriy