[REGRESSION][BISECTED] tpm: Popping noise in USB headphones since 1b6d7f9eb150

[REGRESSION][BISECTED] tpm: Popping noise in USB headphones since 1b6d7f9eb150

[Christian Heusel]

[-- Attachment #1: Type: text/plain, Size: 2296 bytes --]

Hello everyone,

Adam reports that since switching to the 6.10 kernel they have a popping
sound from time to time (roughly every 10 minues) on their headphones
which are connected via a [UGREEN's 5-in-1 USB-C Hub][0].

We have then done a [lengthy bisection][1] on the mainline kernel which
came to the conclusion that the following commit is the culprit of the
issue:

    1b6d7f9eb150 ("tpm: add session encryption protection to tpm2_get_random()")

Since I thought that these results are likely not correct Adam has
verified our results again by re-testing the bisection steps and again
came to the same conclusion. Additionally I have shot the same question
[at Jarkko on Mastodon] to find out if it's possible that changes in the
TPM subsystem can have such an effect.. Without knowing the exact commit
we had bisected to he pointed to the same functionality:

> Jarkko Sakkinen (@jarkko@social.kernel.org):
>
> @gromit i might actually might have hunch for this but have been
> unmotivated to move forward before these patches are merged. One thing
> at a time IMHO, especially with performance fixes.
> NOTE: this is a hyphothesis.
>
> tpm2_get_random() needs to be reconsidered with encryption as hwrng
> calls it often enough and there's now more overhead.
>
> It pulls entropy in small chunks subtracting the length in the look
> like you would use read() syscall from user space. This is not right.
> This leads to small and variable size exchanges and yeah generally
> that equates to an inefficient implemenation with bad latency
> properties.

If there are any patches to test or if further input is needed from the
initial reporter on our Bugtracker we can do that!

Thank you all for your hard work on the kernel!

Cheers,
Chris


[0]: https://uk.ugreen.com/collections/hubs-1/products/ugreen-5-in-1-4k-hdmi-usb-c-hub
[1]: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/79#note_210599
[2]: https://chaos.social/@gromit/113345582873908273

---
#regzbot introduced: 1b6d7f9eb150 
#regzbot link: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/79
#regzbot link: https://chaos.social/@gromit/113345582873908273
#regzbot title: tpm: popping sound in headphones connected via USB-C

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [REGRESSION][BISECTED] tpm: Popping noise in USB headphones since 1b6d7f9eb150

[Jarkko Sakkinen]

Yeah, this is on the list.

See: https://bugzilla.kernel.org/show_bug.cgi?id=219383#c5

I had a fix for the AMD boot-time issue already over a month ago
but unfortunately took time to get enough feedback.

BR, Jarkko

On Fri Oct 25, 2024 at 3:11 AM EEST, Christian Heusel wrote:
> Hello everyone,
>
> Adam reports that since switching to the 6.10 kernel they have a popping
> sound from time to time (roughly every 10 minues) on their headphones
> which are connected via a [UGREEN's 5-in-1 USB-C Hub][0].
>
> We have then done a [lengthy bisection][1] on the mainline kernel which
> came to the conclusion that the following commit is the culprit of the
> issue:
>
>     1b6d7f9eb150 ("tpm: add session encryption protection to tpm2_get_random()")
>
> Since I thought that these results are likely not correct Adam has
> verified our results again by re-testing the bisection steps and again
> came to the same conclusion. Additionally I have shot the same question
> [at Jarkko on Mastodon] to find out if it's possible that changes in the
> TPM subsystem can have such an effect.. Without knowing the exact commit
> we had bisected to he pointed to the same functionality:
>
> > Jarkko Sakkinen (@jarkko@social.kernel.org):
> >
> > @gromit i might actually might have hunch for this but have been
> > unmotivated to move forward before these patches are merged. One thing
> > at a time IMHO, especially with performance fixes.
> > NOTE: this is a hyphothesis.
> >
> > tpm2_get_random() needs to be reconsidered with encryption as hwrng
> > calls it often enough and there's now more overhead.
> >
> > It pulls entropy in small chunks subtracting the length in the look
> > like you would use read() syscall from user space. This is not right.
> > This leads to small and variable size exchanges and yeah generally
> > that equates to an inefficient implemenation with bad latency
> > properties.
>
> If there are any patches to test or if further input is needed from the
> initial reporter on our Bugtracker we can do that!
>
> Thank you all for your hard work on the kernel!
>
> Cheers,
> Chris
>
>
> [0]: https://uk.ugreen.com/collections/hubs-1/products/ugreen-5-in-1-4k-hdmi-usb-c-hub
> [1]: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/79#note_210599
> [2]: https://chaos.social/@gromit/113345582873908273
>
> ---
> #regzbot introduced: 1b6d7f9eb150 
> #regzbot link: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/79
> #regzbot link: https://chaos.social/@gromit/113345582873908273
> #regzbot title: tpm: popping sound in headphones connected via USB-C

Re: [REGRESSION][BISECTED] tpm: Popping noise in USB headphones since 1b6d7f9eb150

[Christian Heusel]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

On 24/10/25 05:47PM, Jarkko Sakkinen wrote:
> Yeah, this is on the list.
> 
> See: https://bugzilla.kernel.org/show_bug.cgi?id=219383#c5
> 
> I had a fix for the AMD boot-time issue already over a month ago
> but unfortunately took time to get enough feedback.
> 
> BR, Jarkko

I'm not sure if this is supposed to be fixed, but AFAIK we hoped that
the patchset that was mentioned in bugzilla also helped this issue.

The reporter said that the bug is still present in 6.12.1, so this might
need further poking 🤔

Cheers,
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [REGRESSION][BISECTED] tpm: Popping noise in USB headphones since 1b6d7f9eb150

[Jarkko Sakkinen]

On Tue Nov 26, 2024 at 1:42 PM EET, Christian Heusel wrote:
> On 24/10/25 05:47PM, Jarkko Sakkinen wrote:
> > Yeah, this is on the list.
> > 
> > See: https://bugzilla.kernel.org/show_bug.cgi?id=219383#c5
> > 
> > I had a fix for the AMD boot-time issue already over a month ago
> > but unfortunately took time to get enough feedback.
> > 
> > BR, Jarkko
>
> I'm not sure if this is supposed to be fixed, but AFAIK we hoped that
> the patchset that was mentioned in bugzilla also helped this issue.
>
> The reporter said that the bug is still present in 6.12.1, so this might
> need further poking 🤔

I'd suggest a workaround for the time being.

In 6.12 we added this for (heavy) IMA use:

tpm.disable_pcr_integrity= [HW,TPM]
                        Do not protect PCR registers from unintended physical
                        access, or interposers in the bus by the means of
                        having an integrity protected session wrapped around
                        TPM2_PCR_Extend command. Consider this in a situation
                        where TPM is heavily utilized by IMA, thus protection
                        causing a major performance hit, and the space where
                        machines are deployed is by other means guarded.

Similarly it might make sense to have "tpm.disable_random_integrity"
that disables the feature introduced by the failing commit.

What do you think?

>
> Cheers,
> Chris

BR, Jarkko

Re: [REGRESSION][BISECTED] tpm: Popping noise in USB headphones since 1b6d7f9eb150

[Stefan Berger]

On 11/29/24 9:44 PM, Jarkko Sakkinen wrote:
> On Tue Nov 26, 2024 at 1:42 PM EET, Christian Heusel wrote:
>> On 24/10/25 05:47PM, Jarkko Sakkinen wrote:
>>> Yeah, this is on the list.
>>>
>>> See: https://bugzilla.kernel.org/show_bug.cgi?id=219383#c5
>>>
>>> I had a fix for the AMD boot-time issue already over a month ago
>>> but unfortunately took time to get enough feedback.
>>>
>>> BR, Jarkko
>>
>> I'm not sure if this is supposed to be fixed, but AFAIK we hoped that
>> the patchset that was mentioned in bugzilla also helped this issue.
>>
>> The reporter said that the bug is still present in 6.12.1, so this might
>> need further poking 🤔
> 
> I'd suggest a workaround for the time being.
> 
> In 6.12 we added this for (heavy) IMA use:
> 
> tpm.disable_pcr_integrity= [HW,TPM]
>                          Do not protect PCR registers from unintended physical
>                          access, or interposers in the bus by the means of
>                          having an integrity protected session wrapped around
>                          TPM2_PCR_Extend command. Consider this in a situation
>                          where TPM is heavily utilized by IMA, thus protection
>                          causing a major performance hit, and the space where
>                          machines are deployed is by other means guarded.
> 
> Similarly it might make sense to have "tpm.disable_random_integrity"
> that disables the feature introduced by the failing commit.
> 

I am wondering what could be the not-so-obvious root cause for this? 
Could it be due to a (TPM or RNG-related) lock? I guess the audio 
popping could occur if an application cannot meet timing requirements 
when it runs into some sort of blocking lock...

> What do you think?
 > >>
>> Cheers,
>> Chris
> 
> BR, Jarkko
> 

Re: [REGRESSION][BISECTED] tpm: Popping noise in USB headphones since 1b6d7f9eb150

[Jarkko Sakkinen]

On Mon Dec 2, 2024 at 11:15 PM CET, Stefan Berger wrote:
>
>
> On 11/29/24 9:44 PM, Jarkko Sakkinen wrote:
> > On Tue Nov 26, 2024 at 1:42 PM EET, Christian Heusel wrote:
> >> On 24/10/25 05:47PM, Jarkko Sakkinen wrote:
> >>> Yeah, this is on the list.
> >>>
> >>> See: https://bugzilla.kernel.org/show_bug.cgi?id=219383#c5
> >>>
> >>> I had a fix for the AMD boot-time issue already over a month ago
> >>> but unfortunately took time to get enough feedback.
> >>>
> >>> BR, Jarkko
> >>
> >> I'm not sure if this is supposed to be fixed, but AFAIK we hoped that
> >> the patchset that was mentioned in bugzilla also helped this issue.
> >>
> >> The reporter said that the bug is still present in 6.12.1, so this might
> >> need further poking 🤔
> > 
> > I'd suggest a workaround for the time being.
> > 
> > In 6.12 we added this for (heavy) IMA use:
> > 
> > tpm.disable_pcr_integrity= [HW,TPM]
> >                          Do not protect PCR registers from unintended physical
> >                          access, or interposers in the bus by the means of
> >                          having an integrity protected session wrapped around
> >                          TPM2_PCR_Extend command. Consider this in a situation
> >                          where TPM is heavily utilized by IMA, thus protection
> >                          causing a major performance hit, and the space where
> >                          machines are deployed is by other means guarded.
> > 
> > Similarly it might make sense to have "tpm.disable_random_integrity"
> > that disables the feature introduced by the failing commit.
> > 
>
> I am wondering what could be the not-so-obvious root cause for this? 
> Could it be due to a (TPM or RNG-related) lock? I guess the audio 
> popping could occur if an application cannot meet timing requirements 
> when it runs into some sort of blocking lock...

I'm travelling this week. I'll take a look in detail next weke. Obvious
thing is that latency has rised on getting the result for TPM2_GetRandom
but it should sleep while waiting. I.e. not sure what causes "busyness".

I think in general having disable_* for trusted keys, random and PCR
extension i.e. features that bus integrity protection touches would
be great for debugging these issues in all cases.

BR, Jarkko

Re: [REGRESSION][BISECTED] tpm: Popping noise in USB headphones since 1b6d7f9eb150

[Jarkko Sakkinen]

On Tue Dec 3, 2024 at 12:15 AM EET, Stefan Berger wrote:
>
>
> On 11/29/24 9:44 PM, Jarkko Sakkinen wrote:
> > On Tue Nov 26, 2024 at 1:42 PM EET, Christian Heusel wrote:
> >> On 24/10/25 05:47PM, Jarkko Sakkinen wrote:
> >>> Yeah, this is on the list.
> >>>
> >>> See: https://bugzilla.kernel.org/show_bug.cgi?id=219383#c5
> >>>
> >>> I had a fix for the AMD boot-time issue already over a month ago
> >>> but unfortunately took time to get enough feedback.
> >>>
> >>> BR, Jarkko
> >>
> >> I'm not sure if this is supposed to be fixed, but AFAIK we hoped that
> >> the patchset that was mentioned in bugzilla also helped this issue.
> >>
> >> The reporter said that the bug is still present in 6.12.1, so this might
> >> need further poking 🤔
> > 
> > I'd suggest a workaround for the time being.
> > 
> > In 6.12 we added this for (heavy) IMA use:
> > 
> > tpm.disable_pcr_integrity= [HW,TPM]
> >                          Do not protect PCR registers from unintended physical
> >                          access, or interposers in the bus by the means of
> >                          having an integrity protected session wrapped around
> >                          TPM2_PCR_Extend command. Consider this in a situation
> >                          where TPM is heavily utilized by IMA, thus protection
> >                          causing a major performance hit, and the space where
> >                          machines are deployed is by other means guarded.
> > 
> > Similarly it might make sense to have "tpm.disable_random_integrity"
> > that disables the feature introduced by the failing commit.
> > 
>
> I am wondering what could be the not-so-obvious root cause for this? 
> Could it be due to a (TPM or RNG-related) lock? I guess the audio 
> popping could occur if an application cannot meet timing requirements 
> when it runs into some sort of blocking lock...

The problem is that we don't know yet but we do know that it previously
worked.

Or more importantly: that is the hypothesis. So it would be in all cases
useful to create such patch for A/B testing at minimum.

BR, Jarkko