[BUG] general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

[BUG] general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

[Brian Foster]

Hi Kent,

JFYI, I'm seeing the following splat pretty reliably via generic/361 on
an 80xcpu test box. The CI doesn't seem to produce this failure for
whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
Delete dio read alignment check"), before which the test still fails but
the kernel doesn't explode.

Brian

--- 8< ---

general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI
CPU: 15 PID: 1078 Comm: kworker/15:1H Tainted: G           OE      6.7.0-rc7+ #68 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc37 04/01/2014
Workqueue: events_highpri __bch2_read_endio [bcachefs]
RIP: 0010:bio_copy_data_iter+0x186/0x260
Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 
RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001
RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0
RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4
R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8
R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000
FS:  0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554 
Call Trace:
 <TASK>
 ? die_addr+0x32/0x80
 ? exc_general_protection+0x19b/0x450
 ? asm_exc_general_protection+0x22/0x30
 ? bio_copy_data_iter+0x186/0x260
 __bch2_read_endio+0x83d/0x920 [bcachefs]
 ? process_one_work+0x192/0x4d0
 ? process_one_work+0x1fc/0x4d0
 process_one_work+0x1fc/0x4d0
 worker_thread+0x1dd/0x3d0
 ? __pfx_worker_thread+0x10/0x10
 kthread+0x100/0x130
 ? __pfx_kthread+0x10/0x10
 ret_from_fork+0x2d/0x50
 ? __pfx_kthread+0x10/0x10
 ret_from_fork_asm+0x1b/0x30
 </TASK>
Modules linked in: bcachefs(OE) lz4_compress(E) lz4hc_compress(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) nfnetlink(E) sunrpc(E) intel_rapl_msr(E) intel_rapl_common(E) intel_uncore_frequency_common(E) isst_if_common(E) kvm_intel(E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) irqbypass(E) virtio_balloon(E) rapl(E) pktcdvd(E) i2c_i801(E) pcspkr(E) i2c_smbus(E) lpc_ich(E) joydev(E) loop(E) zram(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) virtiofs(E) virtio_net(E) net_failover(E) ghash_clmulni_intel(E) virtio_console(E) failover(E) fuse(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) qemu_fw_cfg(E) 
Unloaded tainted modules: bcachefs(E):3 intel_uncore_frequency(E):1 isst_if_mbox_msr(E):1 [last unloaded: bcachefs(OE)]
---[ end trace 0000000000000000 ]---
RIP: 0010:bio_copy_data_iter+0x186/0x260
Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 
RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001
RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0
RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4
R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8
R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000
FS:  0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554 

Re: [BUG] general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

[Kent Overstreet]

On Tue, Jan 16, 2024 at 10:33:08AM -0500, Brian Foster wrote:
> Hi Kent,
> 
> JFYI, I'm seeing the following splat pretty reliably via generic/361 on
> an 80xcpu test box. The CI doesn't seem to produce this failure for
> whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
> Delete dio read alignment check"), before which the test still fails but
> the kernel doesn't explode.

Ahh, that probably needs to be a 512 byte alignment check; since the
block layer alternates units between bytes and 512 byte sectors things
will get very confused if the IO wasn't 512 byte aligned.

> 
> Brian
> 
> --- 8< ---
> 
> general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI
> CPU: 15 PID: 1078 Comm: kworker/15:1H Tainted: G           OE      6.7.0-rc7+ #68 
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc37 04/01/2014
> Workqueue: events_highpri __bch2_read_endio [bcachefs]
> RIP: 0010:bio_copy_data_iter+0x186/0x260
> Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 
> RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212
> RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001
> RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0
> RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4
> R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8
> R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000
> FS:  0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> PKRU: 55555554 
> Call Trace:
>  <TASK>
>  ? die_addr+0x32/0x80
>  ? exc_general_protection+0x19b/0x450
>  ? asm_exc_general_protection+0x22/0x30
>  ? bio_copy_data_iter+0x186/0x260
>  __bch2_read_endio+0x83d/0x920 [bcachefs]
>  ? process_one_work+0x192/0x4d0
>  ? process_one_work+0x1fc/0x4d0
>  process_one_work+0x1fc/0x4d0
>  worker_thread+0x1dd/0x3d0
>  ? __pfx_worker_thread+0x10/0x10
>  kthread+0x100/0x130
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork+0x2d/0x50
>  ? __pfx_kthread+0x10/0x10
>  ret_from_fork_asm+0x1b/0x30
>  </TASK>
> Modules linked in: bcachefs(OE) lz4_compress(E) lz4hc_compress(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) rfkill(E) ip_set(E) nf_tables(E) nfnetlink(E) sunrpc(E) intel_rapl_msr(E) intel_rapl_common(E) intel_uncore_frequency_common(E) isst_if_common(E) kvm_intel(E) kvm(E) iTCO_wdt(E) intel_pmc_bxt(E) iTCO_vendor_support(E) irqbypass(E) virtio_balloon(E) rapl(E) pktcdvd(E) i2c_i801(E) pcspkr(E) i2c_smbus(E) lpc_ich(E) joydev(E) loop(E) zram(E) crct10dif_pclmul(E) crc32_pclmul(E) crc32c_intel(E) virtiofs(E) virtio_net(E) net_failover(E) ghash_clmulni_intel(E) virtio_console(E) failover(E) fuse(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) qemu_fw_cfg(E) 
> Unloaded tainted modules: bcachefs(E):3 intel_uncore_frequency(E):1 isst_if_mbox_msr(E):1 [last unloaded: bcachefs(OE)]
> ---[ end trace 0000000000000000 ]---
> RIP: 0010:bio_copy_data_iter+0x186/0x260
> Code: 29 f6 48 c1 f9 06 48 c1 fe 06 48 c1 e1 0c 48 c1 e6 0c 48 01 e9 48 01 ee 48 01 d9 4c 01 d6 83 fa 08 0f 82 b0 fe ff ff 48 8b 06 <48> 89 01 89 d0 48 8b 7c 06 f8 48 89 7c 01 f8 48 8d 79 08 48 83 e7 
> RSP: 0018:ffffa31644a63d38 EFLAGS: 00010212
> RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0280766500040001
> RDX: 0000000000000c00 RSI: ffff8d462b83d400 RDI: ffff8d460fc9edd0
> RBP: ffff8d4500000000 R08: ffffa31644a63dc0 R09: ffffa31644a63dd4
> R10: 0000000000000400 R11: 0000000000000c00 R12: ffff8d462beb9aa8
> R13: ffff8d460fc9ed38 R14: fffffc5e80000000 R15: 0000000000001000
> FS:  0000000000000000(0000) GS:ffff8d64fdd80000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f7255016120 CR3: 000000012534e003 CR4: 0000000000770ef0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> PKRU: 55555554 
> 

Re: [BUG] general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

[Kent Overstreet]

On Tue, Jan 16, 2024 at 10:33:08AM -0500, Brian Foster wrote:
> Hi Kent,
> 
> JFYI, I'm seeing the following splat pretty reliably via generic/361 on
> an 80xcpu test box. The CI doesn't seem to produce this failure for
> whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
> Delete dio read alignment check"), before which the test still fails but
> the kernel doesn't explode.
> 
> Brian
> 

Can you test the following?

--- 8< ---
Subject: [PATCH] bcachefs: bios must be 512 byte algined

Fixes: 023f9ac9f70f bcachefs: Delete dio read alignment check
Reported-by: Brian Foster <bfoster@redhat.com>
Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>

diff --git a/fs/bcachefs/fs-io-direct.c b/fs/bcachefs/fs-io-direct.c
index fdd57c5785c9..e3b219e19e10 100644
--- a/fs/bcachefs/fs-io-direct.c
+++ b/fs/bcachefs/fs-io-direct.c
@@ -77,6 +77,10 @@ static int bch2_direct_IO_read(struct kiocb *req, struct iov_iter *iter)
 
 	bch2_inode_opts_get(&opts, c, &inode->ei_inode);
 
+	/* bios must be 512 byte aligned: */
+	if ((offset|iter->count) & (SECTOR_SIZE - 1))
+		return -EINVAL;
+
 	ret = min_t(loff_t, iter->count,
 		    max_t(loff_t, 0, i_size_read(&inode->v) - offset));
 

Re: [BUG] general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

[Brian Foster]

On Tue, Jan 16, 2024 at 12:03:09PM -0500, Kent Overstreet wrote:
> On Tue, Jan 16, 2024 at 10:33:08AM -0500, Brian Foster wrote:
> > Hi Kent,
> > 
> > JFYI, I'm seeing the following splat pretty reliably via generic/361 on
> > an 80xcpu test box. The CI doesn't seem to produce this failure for
> > whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
> > Delete dio read alignment check"), before which the test still fails but
> > the kernel doesn't explode.
> > 
> > Brian
> > 
> 
> Can you test the following?
> 

Still blows up... repeated a couple times to be sure.

Brian

> --- 8< ---
> Subject: [PATCH] bcachefs: bios must be 512 byte algined
> 
> Fixes: 023f9ac9f70f bcachefs: Delete dio read alignment check
> Reported-by: Brian Foster <bfoster@redhat.com>
> Signed-off-by: Kent Overstreet <kent.overstreet@linux.dev>
> 
> diff --git a/fs/bcachefs/fs-io-direct.c b/fs/bcachefs/fs-io-direct.c
> index fdd57c5785c9..e3b219e19e10 100644
> --- a/fs/bcachefs/fs-io-direct.c
> +++ b/fs/bcachefs/fs-io-direct.c
> @@ -77,6 +77,10 @@ static int bch2_direct_IO_read(struct kiocb *req, struct iov_iter *iter)
>  
>  	bch2_inode_opts_get(&opts, c, &inode->ei_inode);
>  
> +	/* bios must be 512 byte aligned: */
> +	if ((offset|iter->count) & (SECTOR_SIZE - 1))
> +		return -EINVAL;
> +
>  	ret = min_t(loff_t, iter->count,
>  		    max_t(loff_t, 0, i_size_read(&inode->v) - offset));
>  
> 

Re: [BUG] general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

[Kent Overstreet]

On Tue, Jan 16, 2024 at 12:24:48PM -0500, Brian Foster wrote:
> On Tue, Jan 16, 2024 at 12:03:09PM -0500, Kent Overstreet wrote:
> > On Tue, Jan 16, 2024 at 10:33:08AM -0500, Brian Foster wrote:
> > > Hi Kent,
> > > 
> > > JFYI, I'm seeing the following splat pretty reliably via generic/361 on
> > > an 80xcpu test box. The CI doesn't seem to produce this failure for
> > > whatever reason. This bisects down to commit 023f9ac9f70f ("bcachefs:
> > > Delete dio read alignment check"), before which the test still fails but
> > > the kernel doesn't explode.
> > > 
> > > Brian
> > > 
> > 
> > Can you test the following?
> > 
> 
> Still blows up... repeated a couple times to be sure.
 
That sounds like a driver bug then - what driver?

Re: [BUG] general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

[Su Yue]

On Tue 16 Jan 2024 at 12:33, Kent Overstreet 
<kent.overstreet@linux.dev> wrote:

> On Tue, Jan 16, 2024 at 12:24:48PM -0500, Brian Foster wrote:
>> On Tue, Jan 16, 2024 at 12:03:09PM -0500, Kent Overstreet 
>> wrote:
>> > On Tue, Jan 16, 2024 at 10:33:08AM -0500, Brian Foster wrote:
>> > > Hi Kent,
>> > >
>> > > JFYI, I'm seeing the following splat pretty reliably via 
>> > > generic/361 on
>> > > an 80xcpu test box. The CI doesn't seem to produce this 
>> > > failure for
>> > > whatever reason. This bisects down to commit 023f9ac9f70f 
>> > > ("bcachefs:
>> > > Delete dio read alignment check"), before which the test 
>> > > still fails but
>> > > the kernel doesn't explode.
>> > >
>> > > Brian
>> > >
>> >
>> > Can you test the following?
>> >
>>
>> Still blows up... repeated a couple times to be sure.
>
> That sounds like a driver bug then - what driver?


I think it's not a drive bug. It's related to bcachefs block_size.
I can reproduce it by running generic/361 with block_size 4096.

The test devices are normal qemu disks backing by files in host.
The bug disappears after hanging mkfs block_size to 512.

--
Su

Re: [BUG] general protection fault, probably for non-canonical address 0x280766500040001: 0000 [#1] PREEMPT SMP PTI

[Brian Foster]

On Wed, Jan 17, 2024 at 12:20:55PM +0800, Su Yue wrote:
> 
> On Tue 16 Jan 2024 at 12:33, Kent Overstreet <kent.overstreet@linux.dev>
> wrote:
> 
> > On Tue, Jan 16, 2024 at 12:24:48PM -0500, Brian Foster wrote:
> > > On Tue, Jan 16, 2024 at 12:03:09PM -0500, Kent Overstreet wrote:
> > > > On Tue, Jan 16, 2024 at 10:33:08AM -0500, Brian Foster wrote:
> > > > > Hi Kent,
> > > > >
> > > > > JFYI, I'm seeing the following splat pretty reliably via > >
> > > generic/361 on
> > > > > an 80xcpu test box. The CI doesn't seem to produce this > >
> > > failure for
> > > > > whatever reason. This bisects down to commit 023f9ac9f70f > >
> > > ("bcachefs:
> > > > > Delete dio read alignment check"), before which the test > >
> > > still fails but
> > > > > the kernel doesn't explode.
> > > > >
> > > > > Brian
> > > > >
> > > >
> > > > Can you test the following?
> > > >
> > > 
> > > Still blows up... repeated a couple times to be sure.
> > 
> > That sounds like a driver bug then - what driver?
> 
> 
> I think it's not a drive bug. It's related to bcachefs block_size.
> I can reproduce it by running generic/361 with block_size 4096.
> 
> The test devices are normal qemu disks backing by files in host.
> The bug disappears after hanging mkfs block_size to 512.
> 

Hi Su,

Yes, I think this is the issue as block_bytes(c) is 4k in my test.
Thanks for testing/confirming.

The immediate reason for the crash appears to be bio_copy_data_iter()
going off the rails trying to copy a larger source bio into a smaller
destination bio with seemingly inconsistent size/bvec.

The broader context is we start with a sub-block sized read, so e.g. I
see a request for 1024 bytes at offset 4096. This results in a bio with
bi_size == 4096, however, because right after the alignment check that
was removed we do this:

	ret = min_t(loff_t, iter->count,
		    max_t(loff_t, 0, i_size_read(&inode->v) - offset));
	...
	shorten = iov_iter_count(iter) - round_up(ret, block_bytes(c));
	iter->count -= shorten;

... which appears to underflow the iter count and is presumably fixed up
somewhere to cap at the block size. I'm assuming this ends up in a
situation where an internally inconsistent iov_iter leads to a similarly
broken bvec_iter on the bio, or otherwise this gets further confused
when creating the bounce bio, but I haven't traced to that level of
detail.

In any event, we end up in bio_copy_data_iter() with a destination bio
of bi_size == 4k that only appears to have a single 1k sized bvec. The
first 1k copies as expected, but this doesn't reduce bi_size to zero and
so the iteration just continues to advance the bio vec index until it
finds some garbage data it can infer as a non-zero bv_len bvec to copy
into.

Brian

> --
> Su
>