I'd like to donate a MacBook Pro

I'd like to donate a MacBook Pro

[Alex Henrie]

Dear kernel developers,

I have a MacBookPro12,1 A1502 with a very annoying problem. Every time
it boots, the screen stays black for about 90 seconds, and the
keyboard is unresponsive for an additional 40 seconds. Both the
internal keyboard and (if present) external USB keyboard are
unresponsive. This is a big problem because I use LUKS full-disk
encryption, and I can't put in the password without the keyboard
working. On every boot, I get something like:

starting version 232

A password is required to access the cryptroot volume:
Enter passphrase for /dev/sda4: [   13.697887] usb 2-3: device not
accepting address 2, error -62
[   24.790929] usb2-3: device not accepting address 3, error -62
[   36.097301] usb2-3: device not accepting address 4, error -62
[   47.403659] usb2-3: device not accepting address 5, error -62
[   47.430395] usb2-port3: unable to enumerate USB device

After the final error message, the keyboard starts working and I can
input my password. The dmesg log contains an additional error,
"xhci_hcd 0000:00:14.0: Timeout while waiting for setup device
command", interspersed with the "device not accepting address" errors.
Resetting the SMC and the NVRAM did not help.

I am confident that this is a common problem because I have found
various other users complaining about it:

https://bbs.archlinux.org/viewtopic.php?pid=1613363#p1613363
https://bbs.archlinux.org/viewtopic.php?pid=1451053#p1451053
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668105
https://bugzilla.kernel.org/show_bug.cgi?id=115741

Mine is an older MacBook model that is not prohibitively expensive
(about 400 USD). Do any of you think that you can fix this bug? If so,
I would love to set up an identical model with an identical Arch Linux
configuration to my own and send it to you to keep and debug.

-Alex

Re: I'd like to donate a MacBook Pro

[Greg KH]

On Mon, May 01, 2017 at 12:27:41AM -0600, Alex Henrie wrote:
> Dear kernel developers,
> 
> I have a MacBookPro12,1 A1502 with a very annoying problem. Every time
> it boots, the screen stays black for about 90 seconds, and the
> keyboard is unresponsive for an additional 40 seconds. Both the
> internal keyboard and (if present) external USB keyboard are
> unresponsive. This is a big problem because I use LUKS full-disk
> encryption, and I can't put in the password without the keyboard
> working. On every boot, I get something like:
> 
> starting version 232
> 
> A password is required to access the cryptroot volume:
> Enter passphrase for /dev/sda4: [   13.697887] usb 2-3: device not
> accepting address 2, error -62
> [   24.790929] usb2-3: device not accepting address 3, error -62
> [   36.097301] usb2-3: device not accepting address 4, error -62
> [   47.403659] usb2-3: device not accepting address 5, error -62
> [   47.430395] usb2-port3: unable to enumerate USB device
> 
> After the final error message, the keyboard starts working and I can
> input my password. The dmesg log contains an additional error,
> "xhci_hcd 0000:00:14.0: Timeout while waiting for setup device
> command", interspersed with the "device not accepting address" errors.
> Resetting the SMC and the NVRAM did not help.
> 
> I am confident that this is a common problem because I have found
> various other users complaining about it:
> 
> https://bbs.archlinux.org/viewtopic.php?pid=1613363#p1613363
> https://bbs.archlinux.org/viewtopic.php?pid=1451053#p1451053
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668105
> https://bugzilla.kernel.org/show_bug.cgi?id=115741
> 
> Mine is an older MacBook model that is not prohibitively expensive
> (about 400 USD). Do any of you think that you can fix this bug? If so,
> I would love to set up an identical model with an identical Arch Linux
> configuration to my own and send it to you to keep and debug.

You are not saying what kernel version you are using here, newer ones
have fixed issues like this.  Have you tried 4.10?  4.11?

thanks,

greg k-h

Re: I'd like to donate a MacBook Pro

[Alex Henrie]

2017-05-01 3:03 GMT-06:00 Lukas Wunner <lukas@wunner.de>:
> On Mon, 1 May 2017 00:27:41 -0600, Alex Henrie wrote:
>> I am confident that this is a common problem because I have found
>> various other users complaining about it:
>>
>> https://bbs.archlinux.org/viewtopic.php?pid=1613363#p1613363
>> https://bbs.archlinux.org/viewtopic.php?pid=1451053#p1451053
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668105
>> https://bugzilla.kernel.org/show_bug.cgi?id=115741
>
> Briefly looking over those links I notice some people are reporting
> the same issue on macOS.  This could indicate an EFI firmware bug
> or hardware issue.  Have you tried updating the firmware?  Has this
> issue always been present?

Everything worked fine when I first installed Linux last October. I
updated the firmware immediately before installing Linux. I think it
was a couple of months before the boot problems showed up.

2017-05-01 4:06 GMT-06:00 Greg KH <gregkh@linuxfoundation.org>:
> You are not saying what kernel version you are using here, newer ones
> have fixed issues like this.  Have you tried 4.10?  4.11?

I am currently using 4.10. I tried 4.11 yesterday and the keyboard did
not work at all. The error messages changed to:

[    0.453518] ACPI Error: Needed type [Reference], found [Integer] ffff88045bce
3798 (20170119/exresop-103)
[    0.453525] ACPI Exception: AE_AML_OPERAND_TYPE, While resolving operands for
 [OpcodeName unavailable] (20170119/dswexec-461)
[    0.453530] ACPI Error: Method parse/execution failed [\_PR.CPU0._PDC] (Node
ffff88045e0ee320), AE_AML_OPERAND_TYPE (20170119/psparse-543)
starting version 232

A password is required to access the cryptroot volume:
Enter passphrase for /dev/sda4: [    7.950202] xhci_hcd 0000:00:14.0:
Command completion event does not match command
[   10.290329] usb 2-3: device not accepting address 2, error -62
[   20.537740] xhci_hcd 0000:00:14:0: Command completion event does
not match command
[   35.045403] xhci_hcd 0000:00:14.0: Error while assigning device slot ID
[   35.045659] xhci_hcd 0000:00:14.0: Max number of devices this xHCI
host supports is 32.
[   35.045934] usb usb2-port3: couldn't allocate usb_device
[   47.419601] usb 1-3: hub failed to enable device, error -62
[   59.793777] xhci_hcd 0000:00:14.0: Error while assigning device slot ID
[   59.794032] xhci_hcd 0000:00:14.0: Max number of devices this xHCI
host supports is 32.
[   59.794307] usb usb1-port3: couldn't allocate usb_device
[   72.167966] xhci_hcd 0000:00:14.0: Error while assigning device slot ID
[   72.168218] xhci_hcd 0000:00:14.0: Max number of devices this xHCI
host supports is 32.
[   72.168493] usb usb1-port5: couldn't allocate usb_device

Today I ran a regression test to determine which commit made the
keyboard stop working entirely. The last commit that worked for me was
c09e22d5370739e16463c113525df51b5980b1d5. After that, there is a long
series of commits where the screen stays black, and after that, I
start getting errors like the one above.

Thanks for the information you've sent so far. Any ideas on how to fix
this new and serious problem with 4.11?

-Alex

Re: I'd like to donate a MacBook Pro

[Greg KH]

On Tue, May 02, 2017 at 10:55:09PM -0600, Alex Henrie wrote:
> 2017-05-01 3:03 GMT-06:00 Lukas Wunner <lukas@wunner.de>:
> > On Mon, 1 May 2017 00:27:41 -0600, Alex Henrie wrote:
> >> I am confident that this is a common problem because I have found
> >> various other users complaining about it:
> >>
> >> https://bbs.archlinux.org/viewtopic.php?pid=1613363#p1613363
> >> https://bbs.archlinux.org/viewtopic.php?pid=1451053#p1451053
> >> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668105
> >> https://bugzilla.kernel.org/show_bug.cgi?id=115741
> >
> > Briefly looking over those links I notice some people are reporting
> > the same issue on macOS.  This could indicate an EFI firmware bug
> > or hardware issue.  Have you tried updating the firmware?  Has this
> > issue always been present?
> 
> Everything worked fine when I first installed Linux last October. I
> updated the firmware immediately before installing Linux. I think it
> was a couple of months before the boot problems showed up.
> 
> 2017-05-01 4:06 GMT-06:00 Greg KH <gregkh@linuxfoundation.org>:
> > You are not saying what kernel version you are using here, newer ones
> > have fixed issues like this.  Have you tried 4.10?  4.11?
> 
> I am currently using 4.10. I tried 4.11 yesterday and the keyboard did
> not work at all. The error messages changed to:
> 
> [    0.453518] ACPI Error: Needed type [Reference], found [Integer] ffff88045bce
> 3798 (20170119/exresop-103)
> [    0.453525] ACPI Exception: AE_AML_OPERAND_TYPE, While resolving operands for
>  [OpcodeName unavailable] (20170119/dswexec-461)
> [    0.453530] ACPI Error: Method parse/execution failed [\_PR.CPU0._PDC] (Node
> ffff88045e0ee320), AE_AML_OPERAND_TYPE (20170119/psparse-543)
> starting version 232
> 
> A password is required to access the cryptroot volume:
> Enter passphrase for /dev/sda4: [    7.950202] xhci_hcd 0000:00:14.0:
> Command completion event does not match command
> [   10.290329] usb 2-3: device not accepting address 2, error -62
> [   20.537740] xhci_hcd 0000:00:14:0: Command completion event does
> not match command
> [   35.045403] xhci_hcd 0000:00:14.0: Error while assigning device slot ID
> [   35.045659] xhci_hcd 0000:00:14.0: Max number of devices this xHCI
> host supports is 32.
> [   35.045934] usb usb2-port3: couldn't allocate usb_device
> [   47.419601] usb 1-3: hub failed to enable device, error -62
> [   59.793777] xhci_hcd 0000:00:14.0: Error while assigning device slot ID
> [   59.794032] xhci_hcd 0000:00:14.0: Max number of devices this xHCI
> host supports is 32.
> [   59.794307] usb usb1-port3: couldn't allocate usb_device
> [   72.167966] xhci_hcd 0000:00:14.0: Error while assigning device slot ID
> [   72.168218] xhci_hcd 0000:00:14.0: Max number of devices this xHCI
> host supports is 32.
> [   72.168493] usb usb1-port5: couldn't allocate usb_device
> 
> Today I ran a regression test to determine which commit made the
> keyboard stop working entirely. The last commit that worked for me was
> c09e22d5370739e16463c113525df51b5980b1d5. After that, there is a long
> series of commits where the screen stays black, and after that, I
> start getting errors like the one above.

So git bisect said that commit was a good change, what one was the "bad"
commit that git bisect pointed at?

thanks,

greg k-h

Re: I'd like to donate a MacBook Pro

[Alex Henrie]

2017-05-03 5:58 GMT-06:00 Greg KH <gregkh@linuxfoundation.org>:
> On Tue, May 02, 2017 at 10:55:09PM -0600, Alex Henrie wrote:
>> Today I ran a regression test to determine which commit made the
>> keyboard stop working entirely. The last commit that worked for me was
>> c09e22d5370739e16463c113525df51b5980b1d5. After that, there is a long
>> series of commits where the screen stays black, and after that, I
>> start getting errors like the one above.
>
> So git bisect said that commit was a good change, what one was the "bad"
> commit that git bisect pointed at?

The commit right after that, but it's not clear whether the screen
blanking problem was part of the same bug or just another bug that was
introduced at about the same time.

b0119e870837dcd15a207b4701542ebac5d19b45 is the first bad commit
commit b0119e870837dcd15a207b4701542ebac5d19b45
Author: Joerg Roedel <jroedel@suse.de>
Date:   Wed Feb 1 13:23:08 2017 +0100

    iommu: Introduce new 'struct iommu_device'

    This struct represents one hardware iommu in the iommu core
    code. For now it only has the iommu-ops associated with it,
    but that will be extended soon.

    The register/unregister interface is also added, as well as
    making use of it in the Intel and AMD IOMMU drivers.

    Signed-off-by: Joerg Roedel <jroedel@suse.de>

:040000 040000 cb491d4d5bd25f1b65e6c93f7e67c8594901d6e1
84a5621c5e88961cf2385566c1c28eb5375c413f M      drivers
:040000 040000 5a2f0b8b829b29ef80baf6ef7cf2ba4b9bf23bf7
89ecaf2419fe0e00500c23413149f1df7bcbd693 M      include

-Alex

Re: I'd like to donate a MacBook Pro

[Joerg Roedel]

On Wed, May 03, 2017 at 08:35:47AM -0600, Alex Henrie wrote:
> 2017-05-03 5:58 GMT-06:00 Greg KH <gregkh@linuxfoundation.org>:
> > On Tue, May 02, 2017 at 10:55:09PM -0600, Alex Henrie wrote:
> >> Today I ran a regression test to determine which commit made the
> >> keyboard stop working entirely. The last commit that worked for me was
> >> c09e22d5370739e16463c113525df51b5980b1d5. After that, there is a long
> >> series of commits where the screen stays black, and after that, I
> >> start getting errors like the one above.
> >
> > So git bisect said that commit was a good change, what one was the "bad"
> > commit that git bisect pointed at?
> 
> The commit right after that, but it's not clear whether the screen
> blanking problem was part of the same bug or just another bug that was
> introduced at about the same time.

That would be 39ab9555c24110671f8dc671311a26e5c985b592:
	
	iommu: Add sysfs bindings for struct iommu_device

And it introduced a regression when iommu-sysfs entries are accessed.
This is fixed in a7fdb6e648fb.

Does that commit fix the screen blanking problem?



	Joerg

Re: I'd like to donate a MacBook Pro

[Alex Henrie]

2017-05-03 8:58 GMT-06:00 Joerg Roedel <jroedel@suse.de>:
> On Wed, May 03, 2017 at 08:35:47AM -0600, Alex Henrie wrote:
>> 2017-05-03 5:58 GMT-06:00 Greg KH <gregkh@linuxfoundation.org>:
>> > On Tue, May 02, 2017 at 10:55:09PM -0600, Alex Henrie wrote:
>> >> Today I ran a regression test to determine which commit made the
>> >> keyboard stop working entirely. The last commit that worked for me was
>> >> c09e22d5370739e16463c113525df51b5980b1d5. After that, there is a long
>> >> series of commits where the screen stays black, and after that, I
>> >> start getting errors like the one above.
>> >
>> > So git bisect said that commit was a good change, what one was the "bad"
>> > commit that git bisect pointed at?
>>
>> The commit right after that, but it's not clear whether the screen
>> blanking problem was part of the same bug or just another bug that was
>> introduced at about the same time.
>
> That would be 39ab9555c24110671f8dc671311a26e5c985b592:
>
>         iommu: Add sysfs bindings for struct iommu_device
>
> And it introduced a regression when iommu-sysfs entries are accessed.
> This is fixed in a7fdb6e648fb.
>
> Does that commit fix the screen blanking problem?

At a7fdb6e648fb the screen works but the keyboard is broken. I think
that the screen was actually fixed by an earlier commit, but when I
tried to run a regression test to find when the keyboard problem
appeared after the screen was fixed, I gave up because testing each
revision takes about half an hour and the keyboard problem appears to
be somewhat intermittent.

-Alex

Re: I'd like to donate a MacBook Pro

[Mathias Nyman]

On 04.05.2017 05:34, Alex Henrie wrote:
> 2017-05-03 8:58 GMT-06:00 Joerg Roedel <jroedel@suse.de>:
>> On Wed, May 03, 2017 at 08:35:47AM -0600, Alex Henrie wrote:
>>> 2017-05-03 5:58 GMT-06:00 Greg KH <gregkh@linuxfoundation.org>:
>>>> On Tue, May 02, 2017 at 10:55:09PM -0600, Alex Henrie wrote:
>>>>> Today I ran a regression test to determine which commit made the
>>>>> keyboard stop working entirely. The last commit that worked for me was
>>>>> c09e22d5370739e16463c113525df51b5980b1d5. After that, there is a long
>>>>> series of commits where the screen stays black, and after that, I
>>>>> start getting errors like the one above.
>>>>
>>>> So git bisect said that commit was a good change, what one was the "bad"
>>>> commit that git bisect pointed at?
>>>
>>> The commit right after that, but it's not clear whether the screen
>>> blanking problem was part of the same bug or just another bug that was
>>> introduced at about the same time.
>>
>> That would be 39ab9555c24110671f8dc671311a26e5c985b592:
>>
>>          iommu: Add sysfs bindings for struct iommu_device
>>
>> And it introduced a regression when iommu-sysfs entries are accessed.
>> This is fixed in a7fdb6e648fb.
>>
>> Does that commit fix the screen blanking problem?
>
> At a7fdb6e648fb the screen works but the keyboard is broken. I think
> that the screen was actually fixed by an earlier commit, but when I
> tried to run a regression test to find when the keyboard problem
> appeared after the screen was fixed, I gave up because testing each
> revision takes about half an hour and the keyboard problem appears to
> be somewhat intermittent.
>
> -Alex

Looks like there are a few people suffering from macbook xhci related issues

 From the bugs linked it looks like there are both some UEFI issues and a non-responsive
usb device at boot. at least one USB device does not respond to a address device command,
and driver times out and aborts the command after 5 second.

I don't think I can do anything about the UEFI or the first 5 second command timeout
from the xhci driver, but after that some improvement could be done.

This timeout code path in the xhci driver is not exercised that often.
Some minor races were fixed in 4.11 in this area, and better tracing added, but unfortunately
I just discovered there is also has a regression in 4.11

If you can help me debug this by compiling some new kernels and taking logs and traces on
your MacBook we could get this forward.

I set up a branch for this issue, it has the race-fixes, tracing and regression fixes:

git://git.kernel.org/pub/scm/linux/kernel/git/mnyman/xhci.git xhci-macbook

Remember to check out the xhci-macbook branch.

Can you compile and run that kernel on yout MacBook, and take logs?

Traces can be enabled by adding "trace_event=xhci-hcd" to you kernel command line,
then send me the output of both dmesg and /sys/kernel/debug/tracing/trace after the issue is seen.

Thanks
-Mathias

Re: I'd like to donate a MacBook Pro

[Alex Henrie]

2017-05-11 5:16 GMT-06:00 Mathias Nyman <mathias.nyman@linux.intel.com>:
> Looks like there are a few people suffering from macbook xhci related issues
>
> From the bugs linked it looks like there are both some UEFI issues and a
> non-responsive
> usb device at boot. at least one USB device does not respond to a address
> device command,
> and driver times out and aborts the command after 5 second.
>
> I don't think I can do anything about the UEFI or the first 5 second command
> timeout
> from the xhci driver, but after that some improvement could be done.
>
> This timeout code path in the xhci driver is not exercised that often.
> Some minor races were fixed in 4.11 in this area, and better tracing added,
> but unfortunately
> I just discovered there is also has a regression in 4.11
>
> If you can help me debug this by compiling some new kernels and taking logs
> and traces on
> your MacBook we could get this forward.
>
> I set up a branch for this issue, it has the race-fixes, tracing and
> regression fixes:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/mnyman/xhci.git xhci-macbook
>
> Remember to check out the xhci-macbook branch.
>
> Can you compile and run that kernel on yout MacBook, and take logs?
>
> Traces can be enabled by adding "trace_event=xhci-hcd" to you kernel command
> line,
> then send me the output of both dmesg and /sys/kernel/debug/tracing/trace
> after the issue is seen.

Thanks for the detailed instructions. Your xhci-macbook branch works
just as well as Linux 4.10: The system boots up with a working
keyboard after a 2-minute delay. Unfortunately, I was not able to save
any logs because I couldn't figure out which kernel option would let
me access the encrypted hard drive. After putting in my password, I
got the error "crypt: unknown target type".

-Alex