[linux-next] Read-only file system after boot (powerpc)

[linux-next] Read-only file system after boot (powerpc)

[Sachin Sant]

[-- Attachment #1: Type: text/plain, Size: 3633 bytes --]

With recent version of linux-next builds on IBM Power servers,
depending on file system used I have observed the file system is either
mounted read-only(ext4) or kernel fails to reach login prompt (with XFS).
In both cases following messages are seen during boot:

[    4.379474] sd 0:0:1:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[    4.379479] sd 0:0:1:0: [sda] tag#25 Sense Key : Illegal Request [current]
[    4.379483] sd 0:0:1:0: [sda] tag#25 Add. Sense: Invalid field in cdb
[    4.379487] sd 0:0:1:0: [sda] tag#25 CDB: Write(10) 2a 08 00 db d7 36 00 00 01 00
[    4.379490] critical target error, dev sda, sector 115259824 op 0x1:(WRITE) flags 0x69800 phys_seg 1 prio class 0

I first thought these might be due to failed drives, but started seeing them
on several servers. I have attached linux boot log (boot.log) for reference.

git bisect points to following commit:

f1880d26e517a3fe2fd0c32bcbe05e9230a441cf is the first bad commit
commit f1880d26e517a3fe2fd0c32bcbe05e9230a441cf
Author: Christoph Hellwig <hch@lst.de>
Date:   Wed Nov 24 07:28:56 2021 +0100

    blk-mq: cleanup request allocation

Reverting this patch helps. 

# git bisect log
git bisect start
# bad: [f30a24ed97b401416118756fa35fbe5d28f999e3] Add linux-next specific files for 20211126
git bisect bad f30a24ed97b401416118756fa35fbe5d28f999e3
# good: [136057256686de39cc3a07c2e39ef6bc43003ff6] Linux 5.16-rc2
git bisect good 136057256686de39cc3a07c2e39ef6bc43003ff6
# good: [f7cdb85df9f5705849b31d44f4f02de4fd0d5f9e] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/klassert/ipsec-next.git
git bisect good f7cdb85df9f5705849b31d44f4f02de4fd0d5f9e
# bad: [df829d7e3ea25733fab4682a3a7ed2339f55d926] Merge branch 'for-next' of git://git.kernel.dk/linux-block.git
git bisect bad df829d7e3ea25733fab4682a3a7ed2339f55d926
# good: [bf0d7cde12d6913d21e790efeda122a1cf42c5c3] Merge branch 'drm-next' of https://gitlab.freedesktop.org/agd5f/linux
git bisect good bf0d7cde12d6913d21e790efeda122a1cf42c5c3
# good: [06cceaadd85112f6753fa46efd6538ac4bc0fed8] Merge branch 'for-linux-next-gt' of git://anongit.freedesktop.org/drm-intel
git bisect good 06cceaadd85112f6753fa46efd6538ac4bc0fed8
# good: [6c26b5054ce2b822856e32f1840d13f777c6f295] ASoC: SOF: Intel: add .ack support for HDaudio platforms
git bisect good 6c26b5054ce2b822856e32f1840d13f777c6f295
# bad: [8b333e2b7f20a21c8bb969c1c05aa9c2d8c6f73f] Merge branch 'for-5.17/io_uring' into for-next
git bisect bad 8b333e2b7f20a21c8bb969c1c05aa9c2d8c6f73f
# good: [441a375d2002edb994eba4b39259c6d177714578] blk-ioprio: don't set bio priority if not needed
git bisect good 441a375d2002edb994eba4b39259c6d177714578
# bad: [c895b784c699224d690c7dfbdcff309df82366e3] loop: don't hold lo_mutex during __loop_clr_fd()
git bisect bad c895b784c699224d690c7dfbdcff309df82366e3
# good: [10e69ae57a1d4a026de15a9c9058c79ecd47e287] block: don't include blk-mq-sched.h in blk.h
git bisect good 10e69ae57a1d4a026de15a9c9058c79ecd47e287
# good: [65db5bdc941eab6e3d2adee483d1cb0ec70a39ad] block: don't include <linux/idr.h> in blk.h
git bisect good 65db5bdc941eab6e3d2adee483d1cb0ec70a39ad
# bad: [f1880d26e517a3fe2fd0c32bcbe05e9230a441cf] blk-mq: cleanup request allocation
git bisect bad f1880d26e517a3fe2fd0c32bcbe05e9230a441cf
# good: [b1d1d48b8b3a90b4eb28fe3222ca57c6266e211c] block: don't include <linux/part_stat.h> in blk.h
git bisect good b1d1d48b8b3a90b4eb28fe3222ca57c6266e211c
# first bad commit: [f1880d26e517a3fe2fd0c32bcbe05e9230a441cf] blk-mq: cleanup request allocation
#

Thanks
-Sachin



[-- Attachment #2: boot.log --]
[-- Type: application/octet-stream, Size: 41632 bytes --]

Re: [linux-next] Read-only file system after boot (powerpc)

[Christoph Hellwig]

Can you check if your tree already includes this commit:

https://git.kernel.dk/cgit/linux-block/commit/?h=for-5.17/block&id=3f39d47d7ad858c024bd777f5f2a86fa7f6a9f14

and if not see if that fixes the problem?

Re: [linux-next] Read-only file system after boot (powerpc)

[Sachin Sant]

> On 29-Nov-2021, at 3:29 PM, Christoph Hellwig <hch@lst.de> wrote:
> 
> Can you check if your tree already includes this commit:
> 
> https://git.kernel.dk/cgit/linux-block/commit/?h=for-5.17/block&id=3f39d47d7ad858c024bd777f5f2a86fa7f6a9f14
> 
> and if not see if that fixes the problem?

Thanks. Yes, this patch fixes the problem for me.

-Sachin