* NVMe: Regression: write zeros corrupts ext4 file system
@ 2019-03-11 2:24 Ming Lei
2019-03-11 7:54 ` Dongli Zhang
2019-03-11 14:54 ` Keith Busch
0 siblings, 2 replies; 6+ messages in thread
From: Ming Lei @ 2019-03-11 2:24 UTC (permalink / raw)
To: linux-nvme, linux-block, Christoph Hellwig, Jens Axboe,
Chaitanya Kulkarni
Hi,
It is observed that ext4 is corrupted easily by running some workloads
on QEMU NVMe, such as:
1) mkfs.ext4 /dev/nvme0n1
2) mount /dev/nvme0n1 /mnt
3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
4) then the following error message may show up:
[ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters
Or fsck.ext4 will complain after running 'umount /mnt'
The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the
Write Zeroes command").
QEMU version:
QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
Thanks,
Ming
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system
2019-03-11 2:24 NVMe: Regression: write zeros corrupts ext4 file system Ming Lei
@ 2019-03-11 7:54 ` Dongli Zhang
2019-03-11 10:16 ` Ming Lei
2019-03-11 14:54 ` Keith Busch
1 sibling, 1 reply; 6+ messages in thread
From: Dongli Zhang @ 2019-03-11 7:54 UTC (permalink / raw)
To: Ming Lei, linux-nvme, linux-block
Cc: Christoph Hellwig, Jens Axboe, Chaitanya Kulkarni
On 3/11/19 10:24 AM, Ming Lei wrote:
> Hi,
>
> It is observed that ext4 is corrupted easily by running some workloads
> on QEMU NVMe, such as:
I cannot reproduce with most recent up-to-date mainline kernel on below qemu
versions:
- qemu-2.10.2
- qemu-3.0.0
>
> 1) mkfs.ext4 /dev/nvme0n1
>
> 2) mount /dev/nvme0n1 /mnt
>
> 3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>
> 4) then the following error message may show up:
>
> [ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters
>
> Or fsck.ext4 will complain after running 'umount /mnt'
>
> The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the
> Write Zeroes command").
As above commit is for Write Zeros command, I instrument and add printf at the
beginning of nvme_write_zeros() for qemu-2.10.2.
nvme_write_zeros() are only called for 47 times during "mount /dev/nvme0n1 /mnt".
During "git clone" from torvalds' linux.git, there is no call of nvme_write_zeros().
Perhaps there is some special configuration required to trigger the
nvme_write_zeros() on purpose during "git clone" to involve the
nvme_cmd_write_zeroes on kernel side?
My test nvme image is only about 5GB.
Dongli Zhang
>
> QEMU version:
>
> QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27)
> Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
>
> Thanks,
> Ming
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system
2019-03-11 7:54 ` Dongli Zhang
@ 2019-03-11 10:16 ` Ming Lei
0 siblings, 0 replies; 6+ messages in thread
From: Ming Lei @ 2019-03-11 10:16 UTC (permalink / raw)
To: Dongli Zhang
Cc: linux-nvme, linux-block, Christoph Hellwig, Jens Axboe,
Chaitanya Kulkarni
On Mon, Mar 11, 2019 at 03:54:16PM +0800, Dongli Zhang wrote:
>
>
> On 3/11/19 10:24 AM, Ming Lei wrote:
> > Hi,
> >
> > It is observed that ext4 is corrupted easily by running some workloads
> > on QEMU NVMe, such as:
>
> I cannot reproduce with most recent up-to-date mainline kernel on below qemu
> versions:
>
> - qemu-2.10.2
> - qemu-3.0.0
The qemu in my test is from Fedora 27, and it isn't built by me, and
'qemu-system-x86_64 -version' shows that:
QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27)
My test VM is actually cloned from the official Fedora 27 Cloud image[1],
then run 'dnf update' before starting the test.
[1] https://download.fedoraproject.org/pub/fedora/linux/releases/27/CloudImages/x86_64/images/Fedora-Cloud-Base-27-1.6.x86_64.qcow2
>
> >
> > 1) mkfs.ext4 /dev/nvme0n1
> >
> > 2) mount /dev/nvme0n1 /mnt
> >
> > 3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> >
> > 4) then the following error message may show up:
> >
> > [ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters
> >
> > Or fsck.ext4 will complain after running 'umount /mnt'
> >
> > The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the
> > Write Zeroes command").
>
> As above commit is for Write Zeros command, I instrument and add printf at the
> beginning of nvme_write_zeros() for qemu-2.10.2.
>
> nvme_write_zeros() are only called for 47 times during "mount /dev/nvme0n1 /mnt".
>
>
> During "git clone" from torvalds' linux.git, there is no call of nvme_write_zeros().
>
> Perhaps there is some special configuration required to trigger the
> nvme_write_zeros() on purpose during "git clone" to involve the
> nvme_cmd_write_zeroes on kernel side?
It can be triggered by random write workloads after mkfs & mount on the
nvme.
>
> My test nvme image is only about 5GB.
Mine is 8GB.
Thanks,
Ming
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system
2019-03-11 2:24 NVMe: Regression: write zeros corrupts ext4 file system Ming Lei
2019-03-11 7:54 ` Dongli Zhang
@ 2019-03-11 14:54 ` Keith Busch
2019-03-11 15:23 ` Christoph Hellwig
2019-03-12 1:32 ` Ming Lei
1 sibling, 2 replies; 6+ messages in thread
From: Keith Busch @ 2019-03-11 14:54 UTC (permalink / raw)
To: Ming Lei
Cc: linux-nvme, linux-block, Christoph Hellwig, Jens Axboe,
Chaitanya Kulkarni
On Mon, Mar 11, 2019 at 10:24:42AM +0800, Ming Lei wrote:
> Hi,
>
> It is observed that ext4 is corrupted easily by running some workloads
> on QEMU NVMe, such as:
>
> 1) mkfs.ext4 /dev/nvme0n1
>
> 2) mount /dev/nvme0n1 /mnt
>
> 3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>
> 4) then the following error message may show up:
>
> [ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters
>
> Or fsck.ext4 will complain after running 'umount /mnt'
>
> The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the
> Write Zeroes command").
>
> QEMU version:
>
> QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27)
> Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
In QEMU, blk_aio_pwrite_zeroes() takes bytes, but the nvme controller
thought it was blocks. Oops, that went by unnoticed till now!
We should fix QEMU (patch below). Question is, should we quirk driver
for older versions too?
---
diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index 7c8c63e8f5..e8fe8f1ddd 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -324,8 +324,8 @@ static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
const uint8_t data_shift = ns->id_ns.lbaf[lba_index].ds;
uint64_t slba = le64_to_cpu(rw->slba);
uint32_t nlb = le16_to_cpu(rw->nlb) + 1;
- uint64_t aio_slba = slba << (data_shift - BDRV_SECTOR_BITS);
- uint32_t aio_nlb = nlb << (data_shift - BDRV_SECTOR_BITS);
+ uint64_t offset = slba << data_shift;
+ uint32_t count = nlb << data_shift;
if (unlikely(slba + nlb > ns->id_ns.nsze)) {
trace_nvme_err_invalid_lba_range(slba, nlb, ns->id_ns.nsze);
@@ -335,7 +335,7 @@ static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd,
req->has_sg = false;
block_acct_start(blk_get_stats(n->conf.blk), &req->acct, 0,
BLOCK_ACCT_WRITE);
- req->aiocb = blk_aio_pwrite_zeroes(n->conf.blk, aio_slba, aio_nlb,
+ req->aiocb = blk_aio_pwrite_zeroes(n->conf.blk, offset, count,
BDRV_REQ_MAY_UNMAP, nvme_rw_cb, req);
return NVME_NO_COMPLETE;
}
--
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system
2019-03-11 14:54 ` Keith Busch
@ 2019-03-11 15:23 ` Christoph Hellwig
2019-03-12 1:32 ` Ming Lei
1 sibling, 0 replies; 6+ messages in thread
From: Christoph Hellwig @ 2019-03-11 15:23 UTC (permalink / raw)
To: Keith Busch
Cc: Ming Lei, linux-nvme, linux-block, Christoph Hellwig, Jens Axboe,
Chaitanya Kulkarni
On Mon, Mar 11, 2019 at 08:54:59AM -0600, Keith Busch wrote:
> In QEMU, blk_aio_pwrite_zeroes() takes bytes, but the nvme controller
> thought it was blocks. Oops, that went by unnoticed till now!
>
> We should fix QEMU (patch below). Question is, should we quirk driver
> for older versions too?
I think we should. We are actually still quirking all qemu controllers
for broken Identify behavior, at some point we'll have to make them
version specific..
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system
2019-03-11 14:54 ` Keith Busch
2019-03-11 15:23 ` Christoph Hellwig
@ 2019-03-12 1:32 ` Ming Lei
1 sibling, 0 replies; 6+ messages in thread
From: Ming Lei @ 2019-03-12 1:32 UTC (permalink / raw)
To: Keith Busch
Cc: linux-nvme, linux-block, Christoph Hellwig, Jens Axboe,
Chaitanya Kulkarni
On Mon, Mar 11, 2019 at 08:54:59AM -0600, Keith Busch wrote:
> On Mon, Mar 11, 2019 at 10:24:42AM +0800, Ming Lei wrote:
> > Hi,
> >
> > It is observed that ext4 is corrupted easily by running some workloads
> > on QEMU NVMe, such as:
> >
> > 1) mkfs.ext4 /dev/nvme0n1
> >
> > 2) mount /dev/nvme0n1 /mnt
> >
> > 3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> >
> > 4) then the following error message may show up:
> >
> > [ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters
> >
> > Or fsck.ext4 will complain after running 'umount /mnt'
> >
> > The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the
> > Write Zeroes command").
> >
> > QEMU version:
> >
> > QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27)
> > Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
>
> In QEMU, blk_aio_pwrite_zeroes() takes bytes, but the nvme controller
> thought it was blocks. Oops, that went by unnoticed till now!
>
> We should fix QEMU (patch below). Question is, should we quirk driver
> for older versions too?
I think so, users may never upgrade their QEMU.
Thanks,
Ming
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-03-12 1:32 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-03-11 2:24 NVMe: Regression: write zeros corrupts ext4 file system Ming Lei
2019-03-11 7:54 ` Dongli Zhang
2019-03-11 10:16 ` Ming Lei
2019-03-11 14:54 ` Keith Busch
2019-03-11 15:23 ` Christoph Hellwig
2019-03-12 1:32 ` Ming Lei
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).