* NVMe: Regression: write zeros corrupts ext4 file system
@ 2019-03-11 2:24 Ming Lei
2019-03-11 7:54 ` Dongli Zhang
2019-03-11 14:54 ` Keith Busch
0 siblings, 2 replies; 6+ messages in thread
From: Ming Lei @ 2019-03-11 2:24 UTC (permalink / raw)
To: linux-nvme, linux-block, Christoph Hellwig, Jens Axboe,
Chaitanya Kulkarni
Hi,
It is observed that ext4 is corrupted easily by running some workloads
on QEMU NVMe, such as:
1) mkfs.ext4 /dev/nvme0n1
2) mount /dev/nvme0n1 /mnt
3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
4) then the following error message may show up:
[ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters
Or fsck.ext4 will complain after running 'umount /mnt'
The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the
Write Zeroes command").
QEMU version:
QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
Thanks,
Ming
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: NVMe: Regression: write zeros corrupts ext4 file system 2019-03-11 2:24 NVMe: Regression: write zeros corrupts ext4 file system Ming Lei @ 2019-03-11 7:54 ` Dongli Zhang 2019-03-11 10:16 ` Ming Lei 2019-03-11 14:54 ` Keith Busch 1 sibling, 1 reply; 6+ messages in thread From: Dongli Zhang @ 2019-03-11 7:54 UTC (permalink / raw) To: Ming Lei, linux-nvme, linux-block Cc: Christoph Hellwig, Jens Axboe, Chaitanya Kulkarni On 3/11/19 10:24 AM, Ming Lei wrote: > Hi, > > It is observed that ext4 is corrupted easily by running some workloads > on QEMU NVMe, such as: I cannot reproduce with most recent up-to-date mainline kernel on below qemu versions: - qemu-2.10.2 - qemu-3.0.0 > > 1) mkfs.ext4 /dev/nvme0n1 > > 2) mount /dev/nvme0n1 /mnt > > 3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > 4) then the following error message may show up: > > [ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters > > Or fsck.ext4 will complain after running 'umount /mnt' > > The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the > Write Zeroes command"). As above commit is for Write Zeros command, I instrument and add printf at the beginning of nvme_write_zeros() for qemu-2.10.2. nvme_write_zeros() are only called for 47 times during "mount /dev/nvme0n1 /mnt". During "git clone" from torvalds' linux.git, there is no call of nvme_write_zeros(). Perhaps there is some special configuration required to trigger the nvme_write_zeros() on purpose during "git clone" to involve the nvme_cmd_write_zeroes on kernel side? My test nvme image is only about 5GB. Dongli Zhang > > QEMU version: > > QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27) > Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers > > Thanks, > Ming > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system 2019-03-11 7:54 ` Dongli Zhang @ 2019-03-11 10:16 ` Ming Lei 0 siblings, 0 replies; 6+ messages in thread From: Ming Lei @ 2019-03-11 10:16 UTC (permalink / raw) To: Dongli Zhang Cc: linux-nvme, linux-block, Christoph Hellwig, Jens Axboe, Chaitanya Kulkarni On Mon, Mar 11, 2019 at 03:54:16PM +0800, Dongli Zhang wrote: > > > On 3/11/19 10:24 AM, Ming Lei wrote: > > Hi, > > > > It is observed that ext4 is corrupted easily by running some workloads > > on QEMU NVMe, such as: > > I cannot reproduce with most recent up-to-date mainline kernel on below qemu > versions: > > - qemu-2.10.2 > - qemu-3.0.0 The qemu in my test is from Fedora 27, and it isn't built by me, and 'qemu-system-x86_64 -version' shows that: QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27) My test VM is actually cloned from the official Fedora 27 Cloud image[1], then run 'dnf update' before starting the test. [1] https://download.fedoraproject.org/pub/fedora/linux/releases/27/CloudImages/x86_64/images/Fedora-Cloud-Base-27-1.6.x86_64.qcow2 > > > > > 1) mkfs.ext4 /dev/nvme0n1 > > > > 2) mount /dev/nvme0n1 /mnt > > > > 3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > > > 4) then the following error message may show up: > > > > [ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters > > > > Or fsck.ext4 will complain after running 'umount /mnt' > > > > The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the > > Write Zeroes command"). > > As above commit is for Write Zeros command, I instrument and add printf at the > beginning of nvme_write_zeros() for qemu-2.10.2. > > nvme_write_zeros() are only called for 47 times during "mount /dev/nvme0n1 /mnt". > > > During "git clone" from torvalds' linux.git, there is no call of nvme_write_zeros(). > > Perhaps there is some special configuration required to trigger the > nvme_write_zeros() on purpose during "git clone" to involve the > nvme_cmd_write_zeroes on kernel side? It can be triggered by random write workloads after mkfs & mount on the nvme. > > My test nvme image is only about 5GB. Mine is 8GB. Thanks, Ming ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system 2019-03-11 2:24 NVMe: Regression: write zeros corrupts ext4 file system Ming Lei 2019-03-11 7:54 ` Dongli Zhang @ 2019-03-11 14:54 ` Keith Busch 2019-03-11 15:23 ` Christoph Hellwig 2019-03-12 1:32 ` Ming Lei 1 sibling, 2 replies; 6+ messages in thread From: Keith Busch @ 2019-03-11 14:54 UTC (permalink / raw) To: Ming Lei Cc: linux-nvme, linux-block, Christoph Hellwig, Jens Axboe, Chaitanya Kulkarni On Mon, Mar 11, 2019 at 10:24:42AM +0800, Ming Lei wrote: > Hi, > > It is observed that ext4 is corrupted easily by running some workloads > on QEMU NVMe, such as: > > 1) mkfs.ext4 /dev/nvme0n1 > > 2) mount /dev/nvme0n1 /mnt > > 3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > 4) then the following error message may show up: > > [ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters > > Or fsck.ext4 will complain after running 'umount /mnt' > > The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the > Write Zeroes command"). > > QEMU version: > > QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27) > Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers In QEMU, blk_aio_pwrite_zeroes() takes bytes, but the nvme controller thought it was blocks. Oops, that went by unnoticed till now! We should fix QEMU (patch below). Question is, should we quirk driver for older versions too? --- diff --git a/hw/block/nvme.c b/hw/block/nvme.c index 7c8c63e8f5..e8fe8f1ddd 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -324,8 +324,8 @@ static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd, const uint8_t data_shift = ns->id_ns.lbaf[lba_index].ds; uint64_t slba = le64_to_cpu(rw->slba); uint32_t nlb = le16_to_cpu(rw->nlb) + 1; - uint64_t aio_slba = slba << (data_shift - BDRV_SECTOR_BITS); - uint32_t aio_nlb = nlb << (data_shift - BDRV_SECTOR_BITS); + uint64_t offset = slba << data_shift; + uint32_t count = nlb << data_shift; if (unlikely(slba + nlb > ns->id_ns.nsze)) { trace_nvme_err_invalid_lba_range(slba, nlb, ns->id_ns.nsze); @@ -335,7 +335,7 @@ static uint16_t nvme_write_zeros(NvmeCtrl *n, NvmeNamespace *ns, NvmeCmd *cmd, req->has_sg = false; block_acct_start(blk_get_stats(n->conf.blk), &req->acct, 0, BLOCK_ACCT_WRITE); - req->aiocb = blk_aio_pwrite_zeroes(n->conf.blk, aio_slba, aio_nlb, + req->aiocb = blk_aio_pwrite_zeroes(n->conf.blk, offset, count, BDRV_REQ_MAY_UNMAP, nvme_rw_cb, req); return NVME_NO_COMPLETE; } -- ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system 2019-03-11 14:54 ` Keith Busch @ 2019-03-11 15:23 ` Christoph Hellwig 2019-03-12 1:32 ` Ming Lei 1 sibling, 0 replies; 6+ messages in thread From: Christoph Hellwig @ 2019-03-11 15:23 UTC (permalink / raw) To: Keith Busch Cc: Ming Lei, linux-nvme, linux-block, Christoph Hellwig, Jens Axboe, Chaitanya Kulkarni On Mon, Mar 11, 2019 at 08:54:59AM -0600, Keith Busch wrote: > In QEMU, blk_aio_pwrite_zeroes() takes bytes, but the nvme controller > thought it was blocks. Oops, that went by unnoticed till now! > > We should fix QEMU (patch below). Question is, should we quirk driver > for older versions too? I think we should. We are actually still quirking all qemu controllers for broken Identify behavior, at some point we'll have to make them version specific.. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: NVMe: Regression: write zeros corrupts ext4 file system 2019-03-11 14:54 ` Keith Busch 2019-03-11 15:23 ` Christoph Hellwig @ 2019-03-12 1:32 ` Ming Lei 1 sibling, 0 replies; 6+ messages in thread From: Ming Lei @ 2019-03-12 1:32 UTC (permalink / raw) To: Keith Busch Cc: linux-nvme, linux-block, Christoph Hellwig, Jens Axboe, Chaitanya Kulkarni On Mon, Mar 11, 2019 at 08:54:59AM -0600, Keith Busch wrote: > On Mon, Mar 11, 2019 at 10:24:42AM +0800, Ming Lei wrote: > > Hi, > > > > It is observed that ext4 is corrupted easily by running some workloads > > on QEMU NVMe, such as: > > > > 1) mkfs.ext4 /dev/nvme0n1 > > > > 2) mount /dev/nvme0n1 /mnt > > > > 3) cd /mnt; git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > > > 4) then the following error message may show up: > > > > [ 1642.271816] EXT4-fs error (device nvme0n1): ext4_mb_generate_buddy:747: group 0, block bitmap and bg descriptor inconsistent: 32768 vs 23513 free clusters > > > > Or fsck.ext4 will complain after running 'umount /mnt' > > > > The issue disappears by reverting 6e02318eaea53eaafe6 ("nvme: add support for the > > Write Zeroes command"). > > > > QEMU version: > > > > QEMU emulator version 2.10.2(qemu-2.10.2-1.fc27) > > Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers > > In QEMU, blk_aio_pwrite_zeroes() takes bytes, but the nvme controller > thought it was blocks. Oops, that went by unnoticed till now! > > We should fix QEMU (patch below). Question is, should we quirk driver > for older versions too? I think so, users may never upgrade their QEMU. Thanks, Ming ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-03-12 1:32 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-03-11 2:24 NVMe: Regression: write zeros corrupts ext4 file system Ming Lei 2019-03-11 7:54 ` Dongli Zhang 2019-03-11 10:16 ` Ming Lei 2019-03-11 14:54 ` Keith Busch 2019-03-11 15:23 ` Christoph Hellwig 2019-03-12 1:32 ` Ming Lei
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).