[BUG] blacklist: Problem blacklisting hash (-13) during boot

[BUG] blacklist: Problem blacklisting hash (-13) during boot

[Thomas Weißschuh]

Hi,

Since 5.19 during boot I see lots of the following entries in dmesg:

blacklist: Problem blacklisting hash (-13)

This happens because the firmware contains duplicate blacklist entries.
As commit 6364d106e041 [0] modified the "blacklist" keyring to reject updates
this now leads to the spurious error messages.

The machine is a Thinkpad X1 Cargon Gen9 with BIOS revision 1.56 and firmware
revision 1.33.

[0] 6364d106e041 ("certs: Allow root user to append signed hashes to the blacklist keyring")

Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Mickaël Salaün]

Hi,

Thanks for this report. These error messages seem correct but I don't 
see any legitimate reason for the firmware to store duplicate 
blacklisted hashes.

According to the blacklist_init() function, the "blacklisting failed" 
message could be improved to explain that only a set of hashes failed, 
and why they failed. However, despite this message, this should work as 
expected and should not generate any issue.

Did you contact Lenovo to report this issue (i.e. duplicate hashes in 
their firmware)?

Could you please provide the list of duplicate hashes?

Regards,
  Mickaël


On 15/10/2022 05:16, Thomas Weißschuh wrote:
> Hi,
> 
> Since 5.19 during boot I see lots of the following entries in dmesg:
> 
> blacklist: Problem blacklisting hash (-13)
> 
> This happens because the firmware contains duplicate blacklist entries.
> As commit 6364d106e041 [0] modified the "blacklist" keyring to reject updates
> this now leads to the spurious error messages.
> 
> The machine is a Thinkpad X1 Cargon Gen9 with BIOS revision 1.56 and firmware
> revision 1.33.
> 
> [0] 6364d106e041 ("certs: Allow root user to append signed hashes to the blacklist keyring")

Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Thomas Weißschuh]

Hi Mickaël,

On 2022-11-04 18:03+0100, Mickaël Salaün wrote:
> Thanks for this report. These error messages seem correct but I don't see
> any legitimate reason for the firmware to store duplicate blacklisted
> hashes.
> 
> According to the blacklist_init() function, the "blacklisting failed"
> message could be improved to explain that only a set of hashes failed, and
> why they failed. However, despite this message, this should work as expected
> and should not generate any issue.
> 
> Did you contact Lenovo to report this issue (i.e. duplicate hashes in their
> firmware)?

I CCed Mark Person from Lenovo on this mail, nothing more for now.

There seem to more devices than just from Lenovo to be affected:
* Samsung: https://askubuntu.com/questions/1436856/
* Acer: https://ubuntuforums.org/showthread.php?t=2478840
* MSI: https://forum.archlabslinux.com/t/blacklist-problem-blacklisting-hash-13-errors-on-boot/6674/7
* Micro-Star: https://bbs.archlinux.org/viewtopic.php?id=278860

While there are reports that it helps to reset the blacklist in firmware this
doesn't seem like a general solution for endusers.

FYI I also posted a patch that silences the spurious logs:
https://lore.kernel.org/lkml/20221104014704.3469-1-linux@weissschuh.net/

> Could you please provide the list of duplicate hashes?

bin:80b4d96931bf0d02fd91a61e19d14f1da452e66db2408ca8604d411f92659f0a
bin:f52f83a3fa9cfbd6920f722824dbe4034534d25b8507246b3b957dac6e1bce7a
bin:c5d9d8a186e2c82d09afaa2a6f7f2e73870d3e64f72c4e08ef67796a840f0fbd
bin:1aec84b84b6c65a51220a9be7181965230210d62d6d33c48999c6b295a2b0a06
bin:c3a99a460da464a057c3586d83cef5f4ae08b7103979ed8932742df0ed530c66
bin:58fb941aef95a25943b3fb5f2510a0df3fe44c58c95e0ab80487297568ab9771
bin:5391c3a2fb112102a6aa1edc25ae77e19f5d6f09cd09eeb2509922bfcd5992ea
bin:d626157e1d6a718bc124ab8da27cbb65072ca03a7b6b257dbdcbbd60f65ef3d1
bin:d063ec28f67eba53f1642dbf7dff33c6a32add869f6013fe162e2c32f1cbe56d
bin:29c6eb52b43c3aa18b2cd8ed6ea8607cef3cfae1bafe1165755cf2e614844a44
bin:90fbe70e69d633408d3e170c6832dbb2d209e0272527dfb63d49d29572a6f44c
bin:106faceacfecfd4e303b74f480a08098e2d0802b936f8ec774ce21f31686689c
bin:174e3a0b5b43c6a607bbd3404f05341e3dcf396267ce94f8b50e2e23a9da920c
bin:2b99cf26422e92fe365fbf4bc30d27086c9ee14b7a6fff44fb2f6b9001699939
bin:2e70916786a6f773511fa7181fab0f1d70b557c6322ea923b2a8d3b92b51af7d
bin:3fce9b9fdf3ef09d5452b0f95ee481c2b7f06d743a737971558e70136ace3e73
bin:47cc086127e2069a86e03a6bef2cd410f8c55a6d6bdb362168c31b2ce32a5adf
bin:71f2906fd222497e54a34662ab2497fcc81020770ff51368e9e3d9bfcbfd6375
bin:82db3bceb4f60843ce9d97c3d187cd9b5941cd3de8100e586f2bda5637575f67
bin:8ad64859f195b5f58dafaa940b6a6167acd67a886e8f469364177221c55945b9
bin:8d8ea289cfe70a1c07ab7365cb28ee51edd33cf2506de888fbadd60ebf80481c
bin:aeebae3151271273ed95aa2e671139ed31a98567303a332298f83709a9d55aa1
bin:c409bdac4775add8db92aa22b5b718fb8c94a1462c1fe9a416b95d8a3388c2fc
bin:c617c1a8b1ee2a811c28b5a81b4c83d7c98b5b0c27281d610207ebe692c2967f
bin:c90f336617b8e7f983975413c997f10b73eb267fd8a10cb9e3bdbfc667abdb8b
bin:64575bd912789a2e14ad56f6341f52af6bf80cf94400785975e9f04e2d64d745
bin:45c7c8ae750acfbb48fc37527d6412dd644daed8913ccd8a24c94d856967df8e
bin:47ff1b63b140b6fc04ed79131331e651da5b2e2f170f5daef4153dc2fbc532b1
bin:5391c3a2fb112102a6aa1edc25ae77e19f5d6f09cd09eeb2509922bfcd5992ea
bin:80b4d96931bf0d02fd91a61e19d14f1da452e66db2408ca8604d411f92659f0a
bin:992d359aa7a5f789d268b94c11b9485a6b1ce64362b0edb4441ccc187c39647b
bin:c452ab846073df5ace25cca64d6b7a09d906308a1a65eb5240e3c4ebcaa9cc0c
bin:e051b788ecbaeda53046c70e6af6058f95222c046157b8c4c1b9c2cfc65f46e5

Thanks,
Thomas

> On 15/10/2022 05:16, Thomas Weißschuh wrote:
> > Hi,
> > 
> > Since 5.19 during boot I see lots of the following entries in dmesg:
> > 
> > blacklist: Problem blacklisting hash (-13)
> > 
> > This happens because the firmware contains duplicate blacklist entries.
> > As commit 6364d106e041 [0] modified the "blacklist" keyring to reject updates
> > this now leads to the spurious error messages.
> > 
> > The machine is a Thinkpad X1 Cargon Gen9 with BIOS revision 1.56 and firmware
> > revision 1.33.
> > 
> > [0] 6364d106e041 ("certs: Allow root user to append signed hashes to the blacklist keyring")

Re: [External] Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Mark Pearson]

Yeah - sorry, I've dropped the ball on this one....was crazy busy and it
was lower down the priority list.

I have flagged this to the FW team (LO-2105) to get their feedback and
see if we can get it addressed on our platforms.

Mark

On 2022-11-04 15:11, Thomas Weißschuh wrote:
> Hi Mickaël,
> 
> On 2022-11-04 18:03+0100, Mickaël Salaün wrote:
>> Thanks for this report. These error messages seem correct but I don't see
>> any legitimate reason for the firmware to store duplicate blacklisted
>> hashes.
>>
>> According to the blacklist_init() function, the "blacklisting failed"
>> message could be improved to explain that only a set of hashes failed, and
>> why they failed. However, despite this message, this should work as expected
>> and should not generate any issue.
>>
>> Did you contact Lenovo to report this issue (i.e. duplicate hashes in their
>> firmware)?
> 
> I CCed Mark Person from Lenovo on this mail, nothing more for now.
> 
> There seem to more devices than just from Lenovo to be affected:
> * Samsung: https://askubuntu.com/questions/1436856/>> * Acer: https://ubuntuforums.org/showthread.php?t=2478840>> * MSI: https://forum.archlabslinux.com/t/blacklist-problem-blacklisting-hash-13-errors-on-boot/6674/7>> * Micro-Star: https://bbs.archlinux.org/viewtopic.php?id=278860>> 
> While there are reports that it helps to reset the blacklist in firmware this
> doesn't seem like a general solution for endusers.
> 
> FYI I also posted a patch that silences the spurious logs:
> https://lore.kernel.org/lkml/20221104014704.3469-1-linux@weissschuh.net/>> 
>> Could you please provide the list of duplicate hashes?
> 
> bin:80b4d96931bf0d02fd91a61e19d14f1da452e66db2408ca8604d411f92659f0a
> bin:f52f83a3fa9cfbd6920f722824dbe4034534d25b8507246b3b957dac6e1bce7a
> bin:c5d9d8a186e2c82d09afaa2a6f7f2e73870d3e64f72c4e08ef67796a840f0fbd
> bin:1aec84b84b6c65a51220a9be7181965230210d62d6d33c48999c6b295a2b0a06
> bin:c3a99a460da464a057c3586d83cef5f4ae08b7103979ed8932742df0ed530c66
> bin:58fb941aef95a25943b3fb5f2510a0df3fe44c58c95e0ab80487297568ab9771
> bin:5391c3a2fb112102a6aa1edc25ae77e19f5d6f09cd09eeb2509922bfcd5992ea
> bin:d626157e1d6a718bc124ab8da27cbb65072ca03a7b6b257dbdcbbd60f65ef3d1
> bin:d063ec28f67eba53f1642dbf7dff33c6a32add869f6013fe162e2c32f1cbe56d
> bin:29c6eb52b43c3aa18b2cd8ed6ea8607cef3cfae1bafe1165755cf2e614844a44
> bin:90fbe70e69d633408d3e170c6832dbb2d209e0272527dfb63d49d29572a6f44c
> bin:106faceacfecfd4e303b74f480a08098e2d0802b936f8ec774ce21f31686689c
> bin:174e3a0b5b43c6a607bbd3404f05341e3dcf396267ce94f8b50e2e23a9da920c
> bin:2b99cf26422e92fe365fbf4bc30d27086c9ee14b7a6fff44fb2f6b9001699939
> bin:2e70916786a6f773511fa7181fab0f1d70b557c6322ea923b2a8d3b92b51af7d
> bin:3fce9b9fdf3ef09d5452b0f95ee481c2b7f06d743a737971558e70136ace3e73
> bin:47cc086127e2069a86e03a6bef2cd410f8c55a6d6bdb362168c31b2ce32a5adf
> bin:71f2906fd222497e54a34662ab2497fcc81020770ff51368e9e3d9bfcbfd6375
> bin:82db3bceb4f60843ce9d97c3d187cd9b5941cd3de8100e586f2bda5637575f67
> bin:8ad64859f195b5f58dafaa940b6a6167acd67a886e8f469364177221c55945b9
> bin:8d8ea289cfe70a1c07ab7365cb28ee51edd33cf2506de888fbadd60ebf80481c
> bin:aeebae3151271273ed95aa2e671139ed31a98567303a332298f83709a9d55aa1
> bin:c409bdac4775add8db92aa22b5b718fb8c94a1462c1fe9a416b95d8a3388c2fc
> bin:c617c1a8b1ee2a811c28b5a81b4c83d7c98b5b0c27281d610207ebe692c2967f
> bin:c90f336617b8e7f983975413c997f10b73eb267fd8a10cb9e3bdbfc667abdb8b
> bin:64575bd912789a2e14ad56f6341f52af6bf80cf94400785975e9f04e2d64d745
> bin:45c7c8ae750acfbb48fc37527d6412dd644daed8913ccd8a24c94d856967df8e
> bin:47ff1b63b140b6fc04ed79131331e651da5b2e2f170f5daef4153dc2fbc532b1
> bin:5391c3a2fb112102a6aa1edc25ae77e19f5d6f09cd09eeb2509922bfcd5992ea
> bin:80b4d96931bf0d02fd91a61e19d14f1da452e66db2408ca8604d411f92659f0a
> bin:992d359aa7a5f789d268b94c11b9485a6b1ce64362b0edb4441ccc187c39647b
> bin:c452ab846073df5ace25cca64d6b7a09d906308a1a65eb5240e3c4ebcaa9cc0c
> bin:e051b788ecbaeda53046c70e6af6058f95222c046157b8c4c1b9c2cfc65f46e5
> 
> Thanks,
> Thomas
> 
>> On 15/10/2022 05:16, Thomas Weißschuh wrote:
>>> Hi,
>>>
>>> Since 5.19 during boot I see lots of the following entries in dmesg:
>>>
>>> blacklist: Problem blacklisting hash (-13)
>>>
>>> This happens because the firmware contains duplicate blacklist entries.
>>> As commit 6364d106e041 [0] modified the "blacklist" keyring to reject updates
>>> this now leads to the spurious error messages.
>>>
>>> The machine is a Thinkpad X1 Cargon Gen9 with BIOS revision 1.56 and firmware
>>> revision 1.33.
>>>
>>> [0] 6364d106e041 ("certs: Allow root user to append signed hashes to the blacklist keyring")

Re: [External] Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Jeremy Kerr]

Hi Mark,

> I have flagged this to the FW team (LO-2105) to get their feedback
> and see if we can get it addressed on our platforms.

Any progress from the FW team about this? I have a fresh-out-of-the-box
T14s with this issue, there's 33 duplicated hashes in dbx:

$ mokutil --dbx | grep -E '[[:xdigit:]]{64}' | sort | uniq -cd
      2   106faceacfecfd4e303b74f480a08098e2d0802b936f8ec774ce21f31686689c
      2   174e3a0b5b43c6a607bbd3404f05341e3dcf396267ce94f8b50e2e23a9da920c
      2   1aec84b84b6c65a51220a9be7181965230210d62d6d33c48999c6b295a2b0a06
      2   29c6eb52b43c3aa18b2cd8ed6ea8607cef3cfae1bafe1165755cf2e614844a44
      2   2b99cf26422e92fe365fbf4bc30d27086c9ee14b7a6fff44fb2f6b9001699939
      2   2e70916786a6f773511fa7181fab0f1d70b557c6322ea923b2a8d3b92b51af7d
      2   3fce9b9fdf3ef09d5452b0f95ee481c2b7f06d743a737971558e70136ace3e73
      2   45c7c8ae750acfbb48fc37527d6412dd644daed8913ccd8a24c94d856967df8e
      2   47cc086127e2069a86e03a6bef2cd410f8c55a6d6bdb362168c31b2ce32a5adf
      2   47ff1b63b140b6fc04ed79131331e651da5b2e2f170f5daef4153dc2fbc532b1
      3   5391c3a2fb112102a6aa1edc25ae77e19f5d6f09cd09eeb2509922bfcd5992ea
      2   58fb941aef95a25943b3fb5f2510a0df3fe44c58c95e0ab80487297568ab9771
      2   64575bd912789a2e14ad56f6341f52af6bf80cf94400785975e9f04e2d64d745
      2   71f2906fd222497e54a34662ab2497fcc81020770ff51368e9e3d9bfcbfd6375
      3   80b4d96931bf0d02fd91a61e19d14f1da452e66db2408ca8604d411f92659f0a
      2   82db3bceb4f60843ce9d97c3d187cd9b5941cd3de8100e586f2bda5637575f67
      2   8ad64859f195b5f58dafaa940b6a6167acd67a886e8f469364177221c55945b9
      2   8d8ea289cfe70a1c07ab7365cb28ee51edd33cf2506de888fbadd60ebf80481c
      2   90fbe70e69d633408d3e170c6832dbb2d209e0272527dfb63d49d29572a6f44c
      2   992d359aa7a5f789d268b94c11b9485a6b1ce64362b0edb4441ccc187c39647b
      2   aeebae3151271273ed95aa2e671139ed31a98567303a332298f83709a9d55aa1
      2   c3a99a460da464a057c3586d83cef5f4ae08b7103979ed8932742df0ed530c66
      2   c409bdac4775add8db92aa22b5b718fb8c94a1462c1fe9a416b95d8a3388c2fc
      2   c452ab846073df5ace25cca64d6b7a09d906308a1a65eb5240e3c4ebcaa9cc0c
      2   c5d9d8a186e2c82d09afaa2a6f7f2e73870d3e64f72c4e08ef67796a840f0fbd
      2   c617c1a8b1ee2a811c28b5a81b4c83d7c98b5b0c27281d610207ebe692c2967f
      2   c90f336617b8e7f983975413c997f10b73eb267fd8a10cb9e3bdbfc667abdb8b
      2   d063ec28f67eba53f1642dbf7dff33c6a32add869f6013fe162e2c32f1cbe56d
      2   d626157e1d6a718bc124ab8da27cbb65072ca03a7b6b257dbdcbbd60f65ef3d1
      2   e051b788ecbaeda53046c70e6af6058f95222c046157b8c4c1b9c2cfc65f46e5
      2   f52f83a3fa9cfbd6920f722824dbe4034534d25b8507246b3b957dac6e1bce7a
 
- and so generating 33 KERN_ERR messages on boot.

Given there's (at least) a few months' worth of GA machines with this
issue, can we suppress the warning?

Cheers,


Jeremy

Re: [External] Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Thomas Weißschuh]

Hi,

Feb 25, 2023 21:42:54 Jeremy Kerr <jk@codeconstruct.com.au>:

> Hi Mark,
>
>> I have flagged this to the FW team (LO-2105) to get their feedback
>> and see if we can get it addressed on our platforms.
>
> Any progress from the FW team about this? I have a fresh-out-of-the-box
> T14s with this issue, there's 33 duplicated hashes in dbx:
>
> $ mokutil --dbx | grep -E '[[:xdigit:]]{64}' | sort | uniq -cd
> [...]
>
> - and so generating 33 KERN_ERR messages on boot.
>
> Given there's (at least) a few months' worth of GA machines with this
> issue, can we suppress the warning?

In 6.3 this message will be downgraded to a warning.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c95e8f6fd157b45ef0685c221931561e943e82da

A fixed firmware is still desirable, though.

Thomas

Re: [External] Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Jeremy Kerr]

Hi Thomas,

> > Given there's (at least) a few months' worth of GA machines with
> > this issue, can we suppress the warning?
> 
> In 6.3 this message will be downgraded to a warning.
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c95e8f6fd157b45ef0685c221931561e943e82da

Nice, thanks for that.

> A fixed firmware is still desirable, though.

Yep, definitely.

Not to discourage a firmware fix, but I could look at adding support to
one of the utils (mokutil?) to delete duplicate kek/db/dbx entries...

Cheers,


Jeremy

Re: [External] Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Mark Pearson]

Hi

On 2/25/23 22:42, Jeremy Kerr wrote:
> Hi Mark,
>
>> I have flagged this to the FW team (LO-2105) to get their feedback
>> and see if we can get it addressed on our platforms.
> Any progress from the FW team about this? I have a fresh-out-of-the-box
> T14s with this issue, there's 33 duplicated hashes in dbx:

I've been looking at this and the FW team are claiming that it's not 
caused by duplicate entries in the dbx table, which is honestly a bit 
confusing.

We've been doing some more digging - but is there a possibility this is 
caused by something else? I was poking around at the kernel code but 
haven't got to the bottom of it yet.

Mark

Re: [External] Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Jeremy Kerr]

Hi Mark,


> I've been looking at this and the FW team are claiming that it's not 
> caused by duplicate entries in the dbx table, which is honestly a bit
> confusing.
> 
> We've been doing some more digging - but is there a possibility this
> is caused by something else?

I can't quite trace where the EACCES is coming from, I can't see any
obvious causes there - the blacklist key type doesn't have an ->update
operation, and the assoc_array insert doesn't look like it would fail.

However: if I delete one of the duplicate keys using the bios UI, then
the number of errors logged decreases by one.

Cheers,


Jeremy

Re: [External] Re: [BUG] blacklist: Problem blacklisting hash (-13) during boot

[Mark Pearson]

Hi all,

On 2/27/23 09:36, Jeremy Kerr wrote:
> Hi Mark,
>
>
>> I've been looking at this and the FW team are claiming that it's not
>> caused by duplicate entries in the dbx table, which is honestly a bit
>> confusing.
>>
>> We've been doing some more digging - but is there a possibility this
>> is caused by something else?
> I can't quite trace where the EACCES is coming from, I can't see any
> obvious causes there - the blacklist key type doesn't have an ->update
> operation, and the assoc_array insert doesn't look like it would fail.
>
> However: if I delete one of the duplicate keys using the bios UI, then
> the number of errors logged decreases by one.
>
Got to the bottom of this, after a longer exercise than it should have 
been. I have some answers.

The entries are indeed duplicated in our DBX. The reason for this is the 
FW team took three different DBX list published by UEFI forum in 
different time series and combined them. They did this as they found 
some distinct hashes in previously published ones and chose to combine 
them for safety/completeness sake.

I haven't myself dug into the revocation lists hosted on 
https://uefi.org/revocationlistfile but it sounds like there was some 
churn there? The FW team agreed that duplicates should not have been 
created.

The FW team have pushed back on doing another update for this generation 
of platforms. Updating the DBX to make these changes, particularly 
removing entries, will apparently be difficult. I prodded a bit into the 
details but given the issue is essentially cosmetic I do understand 
their concerns (the last DBX update caused enough issues...I'm not sure 
I want to go through it again in a hurry).

They have said they will fix the tables for future platforms.

Hope that helps

Mark