[BUG] Rockchip SPI: Runtime PM usage count underflow!

[BUG] Rockchip SPI: Runtime PM usage count underflow!

[Vicente Bergas]

Hi,
i am a user of the CONFIG_SPI_SPIDEV device.
It stopped working between 6.8 and 6.10.5.
The SPI bus itself reports no errors to userspace, but no devices
appear connected to the bus.
The platform used is RK3328.
The only spi-related message in dmesg is:
rockchip-spi ff190000.spi: Runtime PM usage count underflow!

Please, can somebody review this issue?

Regards,
  Vicente.

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Dragan Simic]

Hello Vincente,

On 2024-08-18 20:13, Vicente Bergas wrote:
> i am a user of the CONFIG_SPI_SPIDEV device.
> It stopped working between 6.8 and 6.10.5.
> The SPI bus itself reports no errors to userspace, but no devices
> appear connected to the bus.
> The platform used is RK3328.
> The only spi-related message in dmesg is:
> rockchip-spi ff190000.spi: Runtime PM usage count underflow!

I'm working on a related patch.  Could you, please, describe your
use case for the spidev driver, i.e. what board are you using it on,
and for what purpose?

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Vicente Bergas]

On Sun, Aug 18, 2024 at 8:26 PM Dragan Simic <dsimic@manjaro.org> wrote:
>
> Hello Vincente,
>
> On 2024-08-18 20:13, Vicente Bergas wrote:
> > i am a user of the CONFIG_SPI_SPIDEV device.
> > It stopped working between 6.8 and 6.10.5.
> > The SPI bus itself reports no errors to userspace, but no devices
> > appear connected to the bus.
> > The platform used is RK3328.
> > The only spi-related message in dmesg is:
> > rockchip-spi ff190000.spi: Runtime PM usage count underflow!
>
> I'm working on a related patch.  Could you, please, describe your
> use case for the spidev driver, i.e. what board are you using it on,
> and for what purpose?

The board is ROCK64 and the purpose is to update all the software that
board runs, which is stored on the SPI NOR flash onboard.

I have not tested this kernel version on RK3399, but it may also
affect that other use case:
https://gitlab.com/vicencb/kevinboot

N.B.: My name is Vicente.

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Dragan Simic]

On 2024-08-18 20:55, Vicente Bergas wrote:
> On Sun, Aug 18, 2024 at 8:26 PM Dragan Simic <dsimic@manjaro.org> 
> wrote:
>> On 2024-08-18 20:13, Vicente Bergas wrote:
>> > i am a user of the CONFIG_SPI_SPIDEV device.
>> > It stopped working between 6.8 and 6.10.5.
>> > The SPI bus itself reports no errors to userspace, but no devices
>> > appear connected to the bus.
>> > The platform used is RK3328.
>> > The only spi-related message in dmesg is:
>> > rockchip-spi ff190000.spi: Runtime PM usage count underflow!
>> 
>> I'm working on a related patch.  Could you, please, describe your
>> use case for the spidev driver, i.e. what board are you using it on,
>> and for what purpose?
> 
> The board is ROCK64 and the purpose is to update all the software that
> board runs, which is stored on the SPI NOR flash onboard.

So, if I got it right, you boot your Rock64 from the SPI chip that
contains all the software it runs, but you also boot Linux on it from
a microSD card, to update the contents of the SPI chip?  I'm guessing
it that way, because the size of an SPI chip is hardly large enough
for storing even an extremely size-optimized Linux system.

Anyway, I wonder why do you have to use the spidev driver for that
purpose?  Why can't you use 'compatible = "jedec,spi-nor";' to access
the SPI chip through /dev/mtd0 under Linux?

> I have not tested this kernel version on RK3399, but it may also
> affect that other use case:
> https://gitlab.com/vicencb/kevinboot
> 
> N.B.: My name is Vicente.

Oh, I'm sorry for mistyping your first name.  Should've copy & pasted
it instead, to prevent such typing mistakes.

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Vicente Bergas]

On Sun, Aug 18, 2024 at 9:20 PM Dragan Simic <dsimic@manjaro.org> wrote:
>
> On 2024-08-18 20:55, Vicente Bergas wrote:
> > On Sun, Aug 18, 2024 at 8:26 PM Dragan Simic <dsimic@manjaro.org>
> > wrote:
> >> On 2024-08-18 20:13, Vicente Bergas wrote:
> >> > i am a user of the CONFIG_SPI_SPIDEV device.
> >> > It stopped working between 6.8 and 6.10.5.
> >> > The SPI bus itself reports no errors to userspace, but no devices
> >> > appear connected to the bus.
> >> > The platform used is RK3328.
> >> > The only spi-related message in dmesg is:
> >> > rockchip-spi ff190000.spi: Runtime PM usage count underflow!
> >>
> >> I'm working on a related patch.  Could you, please, describe your
> >> use case for the spidev driver, i.e. what board are you using it on,
> >> and for what purpose?
> >
> > The board is ROCK64 and the purpose is to update all the software that
> > board runs, which is stored on the SPI NOR flash onboard.
>
> So, if I got it right, you boot your Rock64 from the SPI chip that
> contains all the software it runs, but you also boot Linux on it from

correct

> a microSD card, to update the contents of the SPI chip?  I'm guessing
> it that way, because the size of an SPI chip is hardly large enough
> for storing even an extremely size-optimized Linux system.

No, everything is in the SPI NOR flash, which is 16MB in size and it
is enough for my application.

> Anyway, I wonder why do you have to use the spidev driver for that
> purpose?  Why can't you use 'compatible = "jedec,spi-nor";' to access
> the SPI chip through /dev/mtd0 under Linux?

The bug report is about the SPI bus, MTD is a service provided on top
of the SPI bus.
If the bus fails, then there is no MTD.
That said, i don't use MTD. The bootloader reads the kernel, dtb and
initramfs from the SPI flash into RAM and boots linux entirely on RAM.
There is no persistent filesystem on top of MTD.

> > I have not tested this kernel version on RK3399, but it may also
> > affect that other use case:
> > https://gitlab.com/vicencb/kevinboot
> >
> > N.B.: My name is Vicente.
>
> Oh, I'm sorry for mistyping your first name.  Should've copy & pasted
> it instead, to prevent such typing mistakes.

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Dragan Simic]

On 2024-08-18 22:10, Vicente Bergas wrote:
> On Sun, Aug 18, 2024 at 9:20 PM Dragan Simic <dsimic@manjaro.org> 
> wrote:
>> On 2024-08-18 20:55, Vicente Bergas wrote:
>> > On Sun, Aug 18, 2024 at 8:26 PM Dragan Simic <dsimic@manjaro.org>
>> > wrote:
>> >> On 2024-08-18 20:13, Vicente Bergas wrote:
>> >> > i am a user of the CONFIG_SPI_SPIDEV device.
>> >> > It stopped working between 6.8 and 6.10.5.
>> >> > The SPI bus itself reports no errors to userspace, but no devices
>> >> > appear connected to the bus.
>> >> > The platform used is RK3328.
>> >> > The only spi-related message in dmesg is:
>> >> > rockchip-spi ff190000.spi: Runtime PM usage count underflow!
>> >>
>> >> I'm working on a related patch.  Could you, please, describe your
>> >> use case for the spidev driver, i.e. what board are you using it on,
>> >> and for what purpose?
>> >
>> > The board is ROCK64 and the purpose is to update all the software that
>> > board runs, which is stored on the SPI NOR flash onboard.
>> 
>> So, if I got it right, you boot your Rock64 from the SPI chip that
>> contains all the software it runs, but you also boot Linux on it from
> 
> correct
> 
>> a microSD card, to update the contents of the SPI chip?  I'm guessing
>> it that way, because the size of an SPI chip is hardly large enough
>> for storing even an extremely size-optimized Linux system.
> 
> No, everything is in the SPI NOR flash, which is 16MB in size and it
> is enough for my application.

Thanks for the clarification.

>> Anyway, I wonder why do you have to use the spidev driver for that
>> purpose?  Why can't you use 'compatible = "jedec,spi-nor";' to access
>> the SPI chip through /dev/mtd0 under Linux?
> 
> The bug report is about the SPI bus, MTD is a service provided on top
> of the SPI bus.
> If the bus fails, then there is no MTD.
> That said, i don't use MTD. The bootloader reads the kernel, dtb and
> initramfs from the SPI flash into RAM and boots linux entirely on RAM.
> There is no persistent filesystem on top of MTD.

Alright, but you wrote initially that you use CONFIG_SPI_SPIDEV,
which means that use the spidev driver, and you also wrote that it
has stopped working.  All that meant to me that the spidev driver
no longer works for you, which is somewhat known issue.

Anyway, please let me know are the following directories present
on your Rock64 running the troublesome version of Linux kernel,
presumably 6.10.5, and if they are, please send over the listings
of their contents:

   - /sys/bus/spi
   - /sys/bus/spi/devices
   - /sys/bus/spi/drivers

>> > I have not tested this kernel version on RK3399, but it may also
>> > affect that other use case:
>> > https://gitlab.com/vicencb/kevinboot

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Vicente Bergas]

On Sun, Aug 18, 2024 at 8:13 PM Vicente Bergas <vicencb@gmail.com> wrote:
>
> Hi,
> i am a user of the CONFIG_SPI_SPIDEV device.
> It stopped working between 6.8 and 6.10.5.
> The SPI bus itself reports no errors to userspace, but no devices
> appear connected to the bus.
> The platform used is RK3328.
> The only spi-related message in dmesg is:
> rockchip-spi ff190000.spi: Runtime PM usage count underflow!
>
> Please, can somebody review this issue?
>
> Regards,
>   Vicente.

I've tried to bisect, but there is some strange behaviour:
The message "Runtime PM usage count underflow!" can also appear on a
good kernel.
In order to have a reasoble iteration speed, i am updating the kernel via kexec.
If a good kernel (6.6.30) is cold-booted, then, all the kernels
kexec'd from it work too.
If a bad kernel (6.10.5) is cold-booted, then a 6.6.30 is kexec'd and
then the same 6.10.5 is kexec'd it becomes a good one.

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Vicente Bergas]

On Mon, Aug 19, 2024 at 4:12 AM Vicente Bergas <vicencb@gmail.com> wrote:
>
> On Sun, Aug 18, 2024 at 8:13 PM Vicente Bergas <vicencb@gmail.com> wrote:
> >
> > Hi,
> > i am a user of the CONFIG_SPI_SPIDEV device.
> > It stopped working between 6.8 and 6.10.5.
> > The SPI bus itself reports no errors to userspace, but no devices
> > appear connected to the bus.
> > The platform used is RK3328.
> > The only spi-related message in dmesg is:
> > rockchip-spi ff190000.spi: Runtime PM usage count underflow!
> >
> > Please, can somebody review this issue?
> >
> > Regards,
> >   Vicente.
>
> I've tried to bisect, but there is some strange behaviour:
> The message "Runtime PM usage count underflow!" can also appear on a
> good kernel.
> In order to have a reasoble iteration speed, i am updating the kernel via kexec.
> If a good kernel (6.6.30) is cold-booted, then, all the kernels
> kexec'd from it work too.
> If a bad kernel (6.10.5) is cold-booted, then a 6.6.30 is kexec'd and
> then the same 6.10.5 is kexec'd it becomes a good one.

Added:
Huang-Huang Bao <i@eh5.me>
Linus Walleij <linus.walleij@linaro.org>
Sasha Levin <sashal@kernel.org>

The first offending commit is:
29d8101fb9442544077e68e27839a1979f85633d pinctrl: rockchip: fix pinmux
bits for RK3328 GPIO2-B pins

I've also tested 6.10.6 with it reverted (and
456447ff1fe3c28e2fd7b57a79650f62245c6428 and
7127c68c76f120367b9a5053f524df0b603d4a48 as dependencies) and SPI
works fine.

Regards,
  Vicente.

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Vicente Bergas]

On Mon, Aug 19, 2024 at 2:49 PM Vicente Bergas <vicencb@gmail.com> wrote:
>
> On Mon, Aug 19, 2024 at 4:12 AM Vicente Bergas <vicencb@gmail.com> wrote:
> >
> > On Sun, Aug 18, 2024 at 8:13 PM Vicente Bergas <vicencb@gmail.com> wrote:
> > >
> > > Hi,
> > > i am a user of the CONFIG_SPI_SPIDEV device.
> > > It stopped working between 6.8 and 6.10.5.
> > > The SPI bus itself reports no errors to userspace, but no devices
> > > appear connected to the bus.
> > > The platform used is RK3328.
> > > The only spi-related message in dmesg is:
> > > rockchip-spi ff190000.spi: Runtime PM usage count underflow!
> > >
> > > Please, can somebody review this issue?
> > >
> > > Regards,
> > >   Vicente.
> >
> > I've tried to bisect, but there is some strange behaviour:
> > The message "Runtime PM usage count underflow!" can also appear on a
> > good kernel.
> > In order to have a reasoble iteration speed, i am updating the kernel via kexec.
> > If a good kernel (6.6.30) is cold-booted, then, all the kernels
> > kexec'd from it work too.
> > If a bad kernel (6.10.5) is cold-booted, then a 6.6.30 is kexec'd and
> > then the same 6.10.5 is kexec'd it becomes a good one.
>
> Added:
> Huang-Huang Bao <i@eh5.me>
> Linus Walleij <linus.walleij@linaro.org>
> Sasha Levin <sashal@kernel.org>
>
> The first offending commit is:
> 29d8101fb9442544077e68e27839a1979f85633d pinctrl: rockchip: fix pinmux
> bits for RK3328 GPIO2-B pins
>
> I've also tested 6.10.6 with it reverted (and
> 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
> 7127c68c76f120367b9a5053f524df0b603d4a48 as dependencies) and SPI
> works fine.

Sorry for the noise:
reverting only 29d8101fb9442544077e68e27839a1979f85633d makes it work on 6.10.6.
Ignore what i said about 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
7127c68c76f120367b9a5053f524df0b603d4a48.

> Regards,
>   Vicente.

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Diederik de Haas]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

On Mon Aug 19, 2024 at 3:14 PM CEST, Vicente Bergas wrote:
> On Mon, Aug 19, 2024 at 2:49 PM Vicente Bergas <vicencb@gmail.com> wrote:
> > On Mon, Aug 19, 2024 at 4:12 AM Vicente Bergas <vicencb@gmail.com> wrote:
> > > > i am a user of the CONFIG_SPI_SPIDEV device.
> > > > It stopped working between 6.8 and 6.10.5.
> > > > The SPI bus itself reports no errors to userspace, but no devices
> > > > appear connected to the bus.
> > > > The platform used is RK3328.
> > > > The only spi-related message in dmesg is:
> > > > rockchip-spi ff190000.spi: Runtime PM usage count underflow!

FWIW: I've seen this issue as well.

> > Added:
> > Huang-Huang Bao <i@eh5.me>
> > Linus Walleij <linus.walleij@linaro.org>
> > Sasha Levin <sashal@kernel.org>
> >
> > The first offending commit is:
> > 29d8101fb9442544077e68e27839a1979f85633d pinctrl: rockchip: fix pinmux
> > bits for RK3328 GPIO2-B pins
> >
> > I've also tested 6.10.6 with it reverted (and
> > 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
> > 7127c68c76f120367b9a5053f524df0b603d4a48 as dependencies) and SPI
> > works fine.
>
> Sorry for the noise:
> reverting only 29d8101fb9442544077e68e27839a1979f85633d makes it work on 6.10.6.
> Ignore what i said about 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
> 7127c68c76f120367b9a5053f524df0b603d4a48.

Please try if unreverting that commit and adding the following:
https://lore.kernel.org/linux-rockchip/20240709105428.1176375-1-i@eh5.me/

fixes the issue as well.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Vicente Bergas]

On Mon, Aug 19, 2024 at 3:33 PM Diederik de Haas <didi.debian@cknow.org> wrote:
>
> On Mon Aug 19, 2024 at 3:14 PM CEST, Vicente Bergas wrote:
> > On Mon, Aug 19, 2024 at 2:49 PM Vicente Bergas <vicencb@gmail.com> wrote:
> > > On Mon, Aug 19, 2024 at 4:12 AM Vicente Bergas <vicencb@gmail.com> wrote:
> > > > > i am a user of the CONFIG_SPI_SPIDEV device.
> > > > > It stopped working between 6.8 and 6.10.5.
> > > > > The SPI bus itself reports no errors to userspace, but no devices
> > > > > appear connected to the bus.
> > > > > The platform used is RK3328.
> > > > > The only spi-related message in dmesg is:
> > > > > rockchip-spi ff190000.spi: Runtime PM usage count underflow!
>
> FWIW: I've seen this issue as well.
>
> > > Added:
> > > Huang-Huang Bao <i@eh5.me>
> > > Linus Walleij <linus.walleij@linaro.org>
> > > Sasha Levin <sashal@kernel.org>
> > >
> > > The first offending commit is:
> > > 29d8101fb9442544077e68e27839a1979f85633d pinctrl: rockchip: fix pinmux
> > > bits for RK3328 GPIO2-B pins
> > >
> > > I've also tested 6.10.6 with it reverted (and
> > > 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
> > > 7127c68c76f120367b9a5053f524df0b603d4a48 as dependencies) and SPI
> > > works fine.
> >
> > Sorry for the noise:
> > reverting only 29d8101fb9442544077e68e27839a1979f85633d makes it work on 6.10.6.
> > Ignore what i said about 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
> > 7127c68c76f120367b9a5053f524df0b603d4a48.
>
> Please try if unreverting that commit and adding the following:
> https://lore.kernel.org/linux-rockchip/20240709105428.1176375-1-i@eh5.me/
>
> fixes the issue as well.

I confirm that 6.10.6 without any reverts plus
20240709105428.1176375-1-i@eh5.me also fixes the issue.
Thank you all.

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Diederik de Haas]

[-- Attachment #1: Type: text/plain, Size: 1484 bytes --]

On Mon Aug 19, 2024 at 4:37 PM CEST, Vicente Bergas wrote:
> On Mon, Aug 19, 2024 at 3:33 PM Diederik de Haas <didi.debian@cknow.org> wrote:
> > On Mon Aug 19, 2024 at 3:14 PM CEST, Vicente Bergas wrote:
> > > On Mon, Aug 19, 2024 at 2:49 PM Vicente Bergas <vicencb@gmail.com> wrote:
> > > > Added:
> > > > Huang-Huang Bao <i@eh5.me>
> > > > Linus Walleij <linus.walleij@linaro.org>
> > > > Sasha Levin <sashal@kernel.org>
> > > >
> > > > The first offending commit is:
> > > > 29d8101fb9442544077e68e27839a1979f85633d pinctrl: rockchip: fix pinmux
> > > > bits for RK3328 GPIO2-B pins
> > > >
> > > > I've also tested 6.10.6 with it reverted (and
> > > > 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
> > > > 7127c68c76f120367b9a5053f524df0b603d4a48 as dependencies) and SPI
> > > > works fine.
> > >
> > > Sorry for the noise:
> > > reverting only 29d8101fb9442544077e68e27839a1979f85633d makes it work on 6.10.6.
> > > Ignore what i said about 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
> > > 7127c68c76f120367b9a5053f524df0b603d4a48.
> >
> > Please try if unreverting that commit and adding the following:
> > https://lore.kernel.org/linux-rockchip/20240709105428.1176375-1-i@eh5.me/
> >
> > fixes the issue as well.
>
> I confirm that 6.10.6 without any reverts plus
> 20240709105428.1176375-1-i@eh5.me also fixes the issue.

It would be great if that patch got accepted to fix the breakage,
including on a number of stable kernel versions.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [BUG] Rockchip SPI: Runtime PM usage count underflow!

[Dragan Simic]

On 2024-08-19 16:37, Vicente Bergas wrote:
> On Mon, Aug 19, 2024 at 3:33 PM Diederik de Haas 
> <didi.debian@cknow.org> wrote:
>> 
>> On Mon Aug 19, 2024 at 3:14 PM CEST, Vicente Bergas wrote:
>> > On Mon, Aug 19, 2024 at 2:49 PM Vicente Bergas <vicencb@gmail.com> wrote:
>> > > On Mon, Aug 19, 2024 at 4:12 AM Vicente Bergas <vicencb@gmail.com> wrote:
>> > > > > i am a user of the CONFIG_SPI_SPIDEV device.
>> > > > > It stopped working between 6.8 and 6.10.5.
>> > > > > The SPI bus itself reports no errors to userspace, but no devices
>> > > > > appear connected to the bus.
>> > > > > The platform used is RK3328.
>> > > > > The only spi-related message in dmesg is:
>> > > > > rockchip-spi ff190000.spi: Runtime PM usage count underflow!
>> 
>> FWIW: I've seen this issue as well.
>> 
>> > > Added:
>> > > Huang-Huang Bao <i@eh5.me>
>> > > Linus Walleij <linus.walleij@linaro.org>
>> > > Sasha Levin <sashal@kernel.org>
>> > >
>> > > The first offending commit is:
>> > > 29d8101fb9442544077e68e27839a1979f85633d pinctrl: rockchip: fix pinmux
>> > > bits for RK3328 GPIO2-B pins
>> > >
>> > > I've also tested 6.10.6 with it reverted (and
>> > > 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
>> > > 7127c68c76f120367b9a5053f524df0b603d4a48 as dependencies) and SPI
>> > > works fine.
>> >
>> > Sorry for the noise:
>> > reverting only 29d8101fb9442544077e68e27839a1979f85633d makes it work on 6.10.6.
>> > Ignore what i said about 456447ff1fe3c28e2fd7b57a79650f62245c6428 and
>> > 7127c68c76f120367b9a5053f524df0b603d4a48.
>> 
>> Please try if unreverting that commit and adding the following:
>> https://lore.kernel.org/linux-rockchip/20240709105428.1176375-1-i@eh5.me/
>> 
>> fixes the issue as well.
> 
> I confirm that 6.10.6 without any reverts plus
> 20240709105428.1176375-1-i@eh5.me also fixes the issue.
> Thank you all.

Great!  Could you, please, provide your Tested-by in the patch thread
linked above?  That should help with getting that patch merged upstream.