ubifs_recover_master_node: failed to recover master node

ubifs_recover_master_node: failed to recover master node

[Chris Packham]

(resend as plaintext)

Hi,

I recently added support for the SPI-NAND controller on the RTL9302C 
SoC[1]. I did most of the work against Linux 6.11 and it's working fine 
there. I recently rebased against the tip of Linus's tree (6.12-rc5) and 
found I was getting ubifs errors when mounting:

[    1.255191] spi-nand spi1.0: Macronix SPI NAND was found.
[    1.261283] spi-nand spi1.0: 256 MiB, block size: 128 KiB, page size: 
2048, OOB size: 64
[    1.271134] 2 fixed-partitions partitions found on MTD device spi1.0
[    1.278247] Creating 2 MTD partitions on "spi1.0":
[    1.283631] 0x000000000000-0x00000f000000 : "user"
[   20.481108] 0x00000f000000-0x000010000000 : "Reserved"
[   72.240347] ubi0: scanning is finished
[   72.270577] ubi0: attached mtd3 (name "user", size 240 MiB)
[   72.276815] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 
bytes
[   72.284537] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[   72.292132] ubi0: VID header offset: 2048 (aligned 2048), data 
offset: 4096
[   72.299885] ubi0: good PEBs: 1920, bad PEBs: 0, corrupted PEBs: 0
[   72.306689] ubi0: user volume: 1, internal volumes: 1, max. volumes 
count: 128
[   72.314747] ubi0: max/mean erase counter: 1/0, WL threshold: 4096, 
image sequence number: 252642230
[   72.324850] ubi0: available PEBs: 0, total reserved PEBs: 1920, PEBs 
reserved for bad PEB handling: 40
[   72.370123] ubi0: background thread "ubi_bgt0d" started, PID 141
[   72.470740] UBIFS (ubi0:0): Mounting in unauthenticated mode
[   72.490246] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, 
PID 144
[   72.528272] UBIFS error (ubi0:0 pid 143): ubifs_recover_master_node: 
failed to recover master node
[   72.550122] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
[   72.710720] UBIFS (ubi0:0): Mounting in unauthenticated mode
[   72.717447] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, 
PID 149
[   72.777602] UBIFS error (ubi0:0 pid 148): ubifs_recover_master_node: 
failed to recover master node
[   72.787792] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops

Full dmesg output is at[2]

git bisect lead me to commit 11813857864f ("mtd: spi-nand: macronix: 
Continuous read support"). Reverting the blamed commit from 6.12-rc5 
seems to avoid the problem. The flash chip on my board is a 
MX30LF2G28AD-TI. I'm not sure if there is a problem with 11813857864f or 
with my spi-mem driver that is exposed after support for continuous read 
is enabled.

Thanks,
Chris

--

[1] - 
https://lore.kernel.org/all/20241015225434.3970360-1-chris.packham@alliedtelesis.co.nz/
[2] - 
https://gist.github.com/cpackham-atlnz/66a0843362e8f8eb2c4f5c7ed01c5efe

Re: ubifs_recover_master_node: failed to recover master node

[Chris Packham]

On 29/10/24 13:38, Chris Packham wrote:
> (resend as plaintext)
>
> Hi,
>
> I recently added support for the SPI-NAND controller on the RTL9302C 
> SoC[1]. I did most of the work against Linux 6.11 and it's working 
> fine there. I recently rebased against the tip of Linus's tree 
> (6.12-rc5) and found I was getting ubifs errors when mounting:
>
> [    1.255191] spi-nand spi1.0: Macronix SPI NAND was found.
> [    1.261283] spi-nand spi1.0: 256 MiB, block size: 128 KiB, page 
> size: 2048, OOB size: 64
> [    1.271134] 2 fixed-partitions partitions found on MTD device spi1.0
> [    1.278247] Creating 2 MTD partitions on "spi1.0":
> [    1.283631] 0x000000000000-0x00000f000000 : "user"
> [   20.481108] 0x00000f000000-0x000010000000 : "Reserved"
> [   72.240347] ubi0: scanning is finished
> [   72.270577] ubi0: attached mtd3 (name "user", size 240 MiB)
> [   72.276815] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 
> 126976 bytes
> [   72.284537] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page 
> size 2048
> [   72.292132] ubi0: VID header offset: 2048 (aligned 2048), data 
> offset: 4096
> [   72.299885] ubi0: good PEBs: 1920, bad PEBs: 0, corrupted PEBs: 0
> [   72.306689] ubi0: user volume: 1, internal volumes: 1, max. volumes 
> count: 128
> [   72.314747] ubi0: max/mean erase counter: 1/0, WL threshold: 4096, 
> image sequence number: 252642230
> [   72.324850] ubi0: available PEBs: 0, total reserved PEBs: 1920, 
> PEBs reserved for bad PEB handling: 40
> [   72.370123] ubi0: background thread "ubi_bgt0d" started, PID 141
> [   72.470740] UBIFS (ubi0:0): Mounting in unauthenticated mode
> [   72.490246] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" 
> started, PID 144
> [   72.528272] UBIFS error (ubi0:0 pid 143): 
> ubifs_recover_master_node: failed to recover master node
> [   72.550122] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
> [   72.710720] UBIFS (ubi0:0): Mounting in unauthenticated mode
> [   72.717447] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" 
> started, PID 149
> [   72.777602] UBIFS error (ubi0:0 pid 148): 
> ubifs_recover_master_node: failed to recover master node
> [   72.787792] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
>
> Full dmesg output is at[2]
>
> git bisect lead me to commit 11813857864f ("mtd: spi-nand: macronix: 
> Continuous read support"). Reverting the blamed commit from 6.12-rc5 
> seems to avoid the problem. The flash chip on my board is a 
> MX30LF2G28AD-TI. I'm not sure if there is a problem with 11813857864f 
> or with my spi-mem driver that is exposed after support for continuous 
> read is enabled.
>
A bit of an update. The ubifs failure is from the is_empty() check in 
get_master_node(). It looks like portions of the LEB are 0 instead of 
0xff. I've also found if I avoid use the non-DMA path in my driver I 
don't have such a problem. I think there is at least one problem in my 
driver because I don't handle DMAing more than 0xffff bytes.


> Thanks,
> Chris
>
> -- 
>
> [1] - 
> https://lore.kernel.org/all/20241015225434.3970360-1-chris.packham@alliedtelesis.co.nz/
> [2] - 
> https://gist.github.com/cpackham-atlnz/66a0843362e8f8eb2c4f5c7ed01c5efe
>

Re: ubifs_recover_master_node: failed to recover master node

[Miquel Raynal]

Hi Chris,

On 29/10/2024 at 13:37:31 +13, Chris Packham <chris.packham@alliedtelesis.co.nz> wrote:

> Hi,
>
> I recently added support for the SPI-NAND controller on the RTL9302C SoC[1]. I did most of the work against Linux 6.11
> and it's working fine there. I recently rebased against the tip of Linus's tree (6.12-rc5) and found I was getting ubifs
> errors when mounting:
>
> [    1.255191] spi-nand spi1.0: Macronix SPI NAND was found.
> [    1.261283] spi-nand spi1.0: 256 MiB, block size: 128 KiB, page size: 2048, OOB size: 64
> [    1.271134] 2 fixed-partitions partitions found on MTD device spi1.0
> [    1.278247] Creating 2 MTD partitions on "spi1.0":
> [    1.283631] 0x000000000000-0x00000f000000 : "user"
> [   20.481108] 0x00000f000000-0x000010000000 : "Reserved"
> [   72.240347] ubi0: scanning is finished
> [   72.270577] ubi0: attached mtd3 (name "user", size 240 MiB)
> [   72.276815] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
> [   72.284537] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
> [   72.292132] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
> [   72.299885] ubi0: good PEBs: 1920, bad PEBs: 0, corrupted PEBs: 0
> [   72.306689] ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
> [   72.314747] ubi0: max/mean erase counter: 1/0, WL threshold: 4096, image sequence number: 252642230
> [   72.324850] ubi0: available PEBs: 0, total reserved PEBs: 1920, PEBs reserved for bad PEB handling: 40
> [   72.370123] ubi0: background thread "ubi_bgt0d" started, PID 141
> [   72.470740] UBIFS (ubi0:0): Mounting in unauthenticated mode
> [   72.490246] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 144
> [   72.528272] UBIFS error (ubi0:0 pid 143): ubifs_recover_master_node: failed to recover master node
> [   72.550122] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
> [   72.710720] UBIFS (ubi0:0): Mounting in unauthenticated mode
> [   72.717447] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 149
> [   72.777602] UBIFS error (ubi0:0 pid 148): ubifs_recover_master_node: failed to recover master node
> [   72.787792] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
>
> Full dmesg output is at[2]
>
> git bisect lead me to commit 11813857864f ("mtd: spi-nand: macronix: Continuous read support"). Reverting the blamed
> commit from 6.12-rc5 seems to avoid the problem. The flash chip on my board is a MX30LF2G28AD-TI. I'm not sure if there
> is a problem with 11813857864f or with my spi-mem driver that is
> exposed after support for continuous read is enabled.

Crap. I had a look, and TBH I don't know. The only thing I see in your
driver might be the DMA vs PIO choice. Could you try to always return
false from rtl_snand_dma_op()?

However you say you're using an MX30* device, this is a raw NAND chip,
SPI-NAND chips are I believe starting with MX35* in their IDs, no?

Thanks,
Miquèl

Re: ubifs_recover_master_node: failed to recover master node

[Miquel Raynal]

Hi Chris,

On 30/10/2024 at 10:13:45 +13, Chris Packham <chris.packham@alliedtelesis.co.nz> wrote:

> On 29/10/24 13:38, Chris Packham wrote:
>> (resend as plaintext)
>>
>> Hi,
>>
>> I recently added support for the SPI-NAND controller on the RTL9302C
>> SoC[1]. I did most of the work against Linux 6.11 and it's working
>> fine there. I recently rebased against the tip of Linus's tree
>> (6.12-rc5) and found I was getting ubifs errors when mounting:
>>
>> [    1.255191] spi-nand spi1.0: Macronix SPI NAND was found.
>> [    1.261283] spi-nand spi1.0: 256 MiB, block size: 128 KiB, page
>> size: 2048, OOB size: 64
>> [    1.271134] 2 fixed-partitions partitions found on MTD device spi1.0
>> [    1.278247] Creating 2 MTD partitions on "spi1.0":
>> [    1.283631] 0x000000000000-0x00000f000000 : "user"
>> [   20.481108] 0x00000f000000-0x000010000000 : "Reserved"
>> [   72.240347] ubi0: scanning is finished
>> [   72.270577] ubi0: attached mtd3 (name "user", size 240 MiB)
>> [   72.276815] ubi0: PEB size: 131072 bytes (128 KiB), LEB size:
>> 126976 bytes
>> [   72.284537] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page
>> size 2048
>> [   72.292132] ubi0: VID header offset: 2048 (aligned 2048), data
>> offset: 4096
>> [   72.299885] ubi0: good PEBs: 1920, bad PEBs: 0, corrupted PEBs: 0
>> [   72.306689] ubi0: user volume: 1, internal volumes: 1, max. volumes
>> count: 128
>> [   72.314747] ubi0: max/mean erase counter: 1/0, WL threshold: 4096,
>> image sequence number: 252642230
>> [   72.324850] ubi0: available PEBs: 0, total reserved PEBs: 1920,
>> PEBs reserved for bad PEB handling: 40
>> [   72.370123] ubi0: background thread "ubi_bgt0d" started, PID 141
>> [   72.470740] UBIFS (ubi0:0): Mounting in unauthenticated mode
>> [   72.490246] UBIFS (ubi0:0): background thread "ubifs_bgt0_0"
>> started, PID 144
>> [   72.528272] UBIFS error (ubi0:0 pid 143):
>> ubifs_recover_master_node: failed to recover master node
>> [   72.550122] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
>> [   72.710720] UBIFS (ubi0:0): Mounting in unauthenticated mode
>> [   72.717447] UBIFS (ubi0:0): background thread "ubifs_bgt0_0"
>> started, PID 149
>> [   72.777602] UBIFS error (ubi0:0 pid 148):
>> ubifs_recover_master_node: failed to recover master node
>> [   72.787792] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
>>
>> Full dmesg output is at[2]
>>
>> git bisect lead me to commit 11813857864f ("mtd: spi-nand: macronix:
>> Continuous read support"). Reverting the blamed commit from 6.12-rc5
>> seems to avoid the problem. The flash chip on my board is a
>> MX30LF2G28AD-TI. I'm not sure if there is a problem with 11813857864f
>> or with my spi-mem driver that is exposed after support for continuous
>> read is enabled.
>>
> A bit of an update. The ubifs failure is from the is_empty() check in
> get_master_node(). It looks like portions of the LEB are 0 instead of
> 0xff. I've also found if I avoid use the non-DMA path in my driver I
> don't have such a problem. I think there is at least one problem in my
> driver because I don't handle DMAing more than 0xffff bytes.

I am going through my mails in a chronological order :-)

Glad to see you found a lead. I was already a bit suspicious about the
DMA path, glad to see we might narrow down the problem.

Is the 0xffff limitation a hard constraint or is it just a pure software
constraint? If we reach a hard constraint, maybe you should check that
when you decide which path you take.

Miquèl

Re: ubifs_recover_master_node: failed to recover master node

[Chris Packham]

Hi Miquel,

On 7/11/24 04:35, Miquel Raynal wrote:
> Hi Chris,
>
> On 29/10/2024 at 13:37:31 +13, Chris Packham <chris.packham@alliedtelesis.co.nz> wrote:
>
>> Hi,
>>
>> I recently added support for the SPI-NAND controller on the RTL9302C SoC[1]. I did most of the work against Linux 6.11
>> and it's working fine there. I recently rebased against the tip of Linus's tree (6.12-rc5) and found I was getting ubifs
>> errors when mounting:
>>
>> [    1.255191] spi-nand spi1.0: Macronix SPI NAND was found.
>> [    1.261283] spi-nand spi1.0: 256 MiB, block size: 128 KiB, page size: 2048, OOB size: 64
>> [    1.271134] 2 fixed-partitions partitions found on MTD device spi1.0
>> [    1.278247] Creating 2 MTD partitions on "spi1.0":
>> [    1.283631] 0x000000000000-0x00000f000000 : "user"
>> [   20.481108] 0x00000f000000-0x000010000000 : "Reserved"
>> [   72.240347] ubi0: scanning is finished
>> [   72.270577] ubi0: attached mtd3 (name "user", size 240 MiB)
>> [   72.276815] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
>> [   72.284537] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
>> [   72.292132] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
>> [   72.299885] ubi0: good PEBs: 1920, bad PEBs: 0, corrupted PEBs: 0
>> [   72.306689] ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
>> [   72.314747] ubi0: max/mean erase counter: 1/0, WL threshold: 4096, image sequence number: 252642230
>> [   72.324850] ubi0: available PEBs: 0, total reserved PEBs: 1920, PEBs reserved for bad PEB handling: 40
>> [   72.370123] ubi0: background thread "ubi_bgt0d" started, PID 141
>> [   72.470740] UBIFS (ubi0:0): Mounting in unauthenticated mode
>> [   72.490246] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 144
>> [   72.528272] UBIFS error (ubi0:0 pid 143): ubifs_recover_master_node: failed to recover master node
>> [   72.550122] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
>> [   72.710720] UBIFS (ubi0:0): Mounting in unauthenticated mode
>> [   72.717447] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 149
>> [   72.777602] UBIFS error (ubi0:0 pid 148): ubifs_recover_master_node: failed to recover master node
>> [   72.787792] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" stops
>>
>> Full dmesg output is at[2]
>>
>> git bisect lead me to commit 11813857864f ("mtd: spi-nand: macronix: Continuous read support"). Reverting the blamed
>> commit from 6.12-rc5 seems to avoid the problem. The flash chip on my board is a MX30LF2G28AD-TI. I'm not sure if there
>> is a problem with 11813857864f or with my spi-mem driver that is
>> exposed after support for continuous read is enabled.
> Crap. I had a look, and TBH I don't know. The only thing I see in your
> driver might be the DMA vs PIO choice. Could you try to always return
> false from rtl_snand_dma_op()?

It turned out the limitation was in my DMA support. With the fix for 
that[1] your changes are fine. I'm a little surprised I never hit 
problems with DMA prior to the continuous read changes but I guess the 
page reads would have been under the limit and my testing probably 
didn't trigger a big enough write.

> However you say you're using an MX30* device, this is a raw NAND chip,
> SPI-NAND chips are I believe starting with MX35* in their IDs, no?
I think I copied that part number off the wrong datasheet in my unsorted 
Downloads directory. The schematic for the board I have says 
MX35LF2GE4AD-Z4 and the correct datasheet has all the right things about 
SPI-NAND and continuous read.
>
> Thanks,
> Miquèl


[1] - 
https://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git/commit/?id=25d284715845a465a1a3693a09cf8b6ab8bd9caf