* Re: DAX can not work on virtual nvdimm device [not found] ` <25098f37-53f7-6d5d-0b1a-8469bab51a9f@linux.intel.com> @ 2016-09-02 2:57 ` Ross Zwisler [not found] ` <20160902025738.GA26108-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Ross Zwisler @ 2016-09-02 2:57 UTC (permalink / raw) To: Xiao Guangrong, linux-ext4, Jan Kara, Theodore Ts'o Cc: Dan Williams, Ross Zwisler, Yumei Huang, KVM, linux-nvdimm@lists.01.org, qemu-devel@nongnu.org, LKML, Linux ACPI, Stefan Hajnoczi On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote: > On 08/31/2016 01:09 AM, Dan Williams wrote: > > > > Can you post your exact reproduction steps? This test is not failing for me. > > > > Sure. > > 1. make the guest kernel based on your tree, the top commit is > 10d7902fa0e82b (dax: unmap/truncate on device shutdown) and > the config file can be found in this thread. > > 2. add guest kernel command line: memmap=6G!10G > > 3: start the guest: > x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \ > -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio > > 4: in guest: > mkfs.ext4 /dev/pmem0 > mount -o dax /dev/pmem0 /mnt/pmem/ > echo > /mnt/pmem/xxx > ./mmap /mnt/pmem/xxx > ./read /mnt/pmem/xxx > > The source code of mmap and read has been attached in this mail. > > Hopefully, you can detect the error triggered by read test. > > Thanks! Okay, I think I've isolated this issue. Xiao's VM was an old CentOS 6 system, and for some reason ext4+DAX with the old tools found in that VM fails. I was able to reproduce this failure with a freshly installed CentOS 6.8 VM. You can see the failure with his tests, or perhaps more easily with this series of commands: # mkfs.ext4 /dev/pmem0 # mount -o dax /dev/pmem0 /mnt/pmem/ # touch /mnt/pmem/x # md5sum /mnt/pmem/x md5sum: /mnt/pmem/x: Bad address This sequence of commands works fine in the old CentOS 6 system if you use XFS instead of ext4, and it works fine with both ext4 and XFS in CentOS 7 and with recent versions of Fedora. I've added the ext4 folks to this mail in case they care, but my guess is that the tools in CentOS 6 are so old that it's not worth worrying about. For reference, the kernel in CentOS 6 is based on 2.6.32. :) DAX was introduced in v4.0. ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20160902025738.GA26108-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>]
* Re: DAX can not work on virtual nvdimm device [not found] ` <20160902025738.GA26108-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> @ 2016-09-06 15:06 ` Jan Kara [not found] ` <20160906150620.GJ28922-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Jan Kara @ 2016-09-06 15:06 UTC (permalink / raw) To: Ross Zwisler Cc: Yumei Huang, Jan Kara, KVM, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org, LKML, Linux ACPI, Theodore Ts'o, Stefan Hajnoczi, Xiao Guangrong, linux-ext4-u79uwXL29TY76Z2rM5mHXA On Thu 01-09-16 20:57:38, Ross Zwisler wrote: > On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote: > > On 08/31/2016 01:09 AM, Dan Williams wrote: > > > > > > Can you post your exact reproduction steps? This test is not failing for me. > > > > > > > Sure. > > > > 1. make the guest kernel based on your tree, the top commit is > > 10d7902fa0e82b (dax: unmap/truncate on device shutdown) and > > the config file can be found in this thread. > > > > 2. add guest kernel command line: memmap=6G!10G > > > > 3: start the guest: > > x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \ > > -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio > > > > 4: in guest: > > mkfs.ext4 /dev/pmem0 > > mount -o dax /dev/pmem0 /mnt/pmem/ > > echo > /mnt/pmem/xxx > > ./mmap /mnt/pmem/xxx > > ./read /mnt/pmem/xxx > > > > The source code of mmap and read has been attached in this mail. > > > > Hopefully, you can detect the error triggered by read test. > > > > Thanks! > > Okay, I think I've isolated this issue. Xiao's VM was an old CentOS 6 system, > and for some reason ext4+DAX with the old tools found in that VM fails. I was > able to reproduce this failure with a freshly installed CentOS 6.8 VM. > > You can see the failure with his tests, or perhaps more easily with this > series of commands: > > # mkfs.ext4 /dev/pmem0 > # mount -o dax /dev/pmem0 /mnt/pmem/ > # touch /mnt/pmem/x > # md5sum /mnt/pmem/x > md5sum: /mnt/pmem/x: Bad address > > This sequence of commands works fine in the old CentOS 6 system if you use XFS > instead of ext4, and it works fine with both ext4 and XFS in CentOS 7 and > with recent versions of Fedora. > > I've added the ext4 folks to this mail in case they care, but my guess is that > the tools in CentOS 6 are so old that it's not worth worrying about. For > reference, the kernel in CentOS 6 is based on 2.6.32. :) DAX was introduced > in v4.0. Hum, can you post 'dumpe2fs -h /dev/pmem0' output from that system when the md5sum fails? Because the only idea I have is that mkfs.ext4 in CentOS 6 creates the filesystem with a different set of features than more recent e2fsprogs and so we hit some untested path... Honza -- Jan Kara <jack-IBi9RG/b67k@public.gmane.org> SUSE Labs, CR ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20160906150620.GJ28922-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org>]
* Re: DAX can not work on virtual nvdimm device [not found] ` <20160906150620.GJ28922-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org> @ 2016-09-08 20:47 ` Ross Zwisler [not found] ` <20160908204708.GA15167-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Ross Zwisler @ 2016-09-08 20:47 UTC (permalink / raw) To: Jan Kara Cc: Yumei Huang, Theodore Ts'o, KVM, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org, LKML, Linux ACPI, Stefan Hajnoczi, Xiao Guangrong, linux-ext4-u79uwXL29TY76Z2rM5mHXA On Tue, Sep 06, 2016 at 05:06:20PM +0200, Jan Kara wrote: > On Thu 01-09-16 20:57:38, Ross Zwisler wrote: > > On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote: > > > On 08/31/2016 01:09 AM, Dan Williams wrote: > > > > > > > > Can you post your exact reproduction steps? This test is not failing for me. > > > > > > > > > > Sure. > > > > > > 1. make the guest kernel based on your tree, the top commit is > > > 10d7902fa0e82b (dax: unmap/truncate on device shutdown) and > > > the config file can be found in this thread. > > > > > > 2. add guest kernel command line: memmap=6G!10G > > > > > > 3: start the guest: > > > x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \ > > > -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio > > > > > > 4: in guest: > > > mkfs.ext4 /dev/pmem0 > > > mount -o dax /dev/pmem0 /mnt/pmem/ > > > echo > /mnt/pmem/xxx > > > ./mmap /mnt/pmem/xxx > > > ./read /mnt/pmem/xxx > > > > > > The source code of mmap and read has been attached in this mail. > > > > > > Hopefully, you can detect the error triggered by read test. > > > > > > Thanks! > > > > Okay, I think I've isolated this issue. Xiao's VM was an old CentOS 6 system, > > and for some reason ext4+DAX with the old tools found in that VM fails. I was > > able to reproduce this failure with a freshly installed CentOS 6.8 VM. > > > > You can see the failure with his tests, or perhaps more easily with this > > series of commands: > > > > # mkfs.ext4 /dev/pmem0 > > # mount -o dax /dev/pmem0 /mnt/pmem/ > > # touch /mnt/pmem/x > > # md5sum /mnt/pmem/x > > md5sum: /mnt/pmem/x: Bad address > > > > This sequence of commands works fine in the old CentOS 6 system if you use XFS > > instead of ext4, and it works fine with both ext4 and XFS in CentOS 7 and > > with recent versions of Fedora. > > > > I've added the ext4 folks to this mail in case they care, but my guess is that > > the tools in CentOS 6 are so old that it's not worth worrying about. For > > reference, the kernel in CentOS 6 is based on 2.6.32. :) DAX was introduced > > in v4.0. > > Hum, can you post 'dumpe2fs -h /dev/pmem0' output from that system when the > md5sum fails? Because the only idea I have is that mkfs.ext4 in CentOS 6 > creates the filesystem with a different set of features than more recent > e2fsprogs and so we hit some untested path... Sure, here's the output: # dumpe2fs -h /dev/pmem0 dumpe2fs 1.41.12 (17-May-2010) Filesystem volume name: <none> Last mounted on: /mnt/pmem Filesystem UUID: 4cd8a836-cc54-4c59-ae0a-4a26bab0f8bc Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 1048576 Block count: 4194304 Reserved block count: 209715 Free blocks: 4084463 Free inodes: 1048565 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 1023 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 RAID stride: 1 Flex block group size: 16 Filesystem created: Thu Sep 8 14:45:31 2016 Last mount time: Thu Sep 8 14:45:39 2016 Last write time: Thu Sep 8 14:45:39 2016 Mount count: 1 Maximum mount count: 21 Last checked: Thu Sep 8 14:45:31 2016 Check interval: 15552000 (6 months) Next check after: Tue Mar 7 13:45:31 2017 Lifetime writes: 388 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 19cad581-c46a-4212-bfa0-d527ff55db49 Journal backup: inode blocks Journal features: (none) Journal size: 128M Journal length: 32768 Journal sequence: 0x00000002 Journal start: 1 ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20160908204708.GA15167-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>]
* Re: DAX can not work on virtual nvdimm device [not found] ` <20160908204708.GA15167-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> @ 2016-09-09 9:19 ` Jan Kara [not found] ` <20160909091925.GF22777-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Jan Kara @ 2016-09-09 9:19 UTC (permalink / raw) To: Ross Zwisler Cc: Yumei Huang, Xiao Guangrong, KVM, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org, LKML, Linux ACPI, Theodore Ts'o, Stefan Hajnoczi, Jan Kara, linux-ext4-u79uwXL29TY76Z2rM5mHXA On Thu 08-09-16 14:47:08, Ross Zwisler wrote: > On Tue, Sep 06, 2016 at 05:06:20PM +0200, Jan Kara wrote: > > On Thu 01-09-16 20:57:38, Ross Zwisler wrote: > > > On Wed, Aug 31, 2016 at 04:44:47PM +0800, Xiao Guangrong wrote: > > > > On 08/31/2016 01:09 AM, Dan Williams wrote: > > > > > > > > > > Can you post your exact reproduction steps? This test is not failing for me. > > > > > > > > > > > > > Sure. > > > > > > > > 1. make the guest kernel based on your tree, the top commit is > > > > 10d7902fa0e82b (dax: unmap/truncate on device shutdown) and > > > > the config file can be found in this thread. > > > > > > > > 2. add guest kernel command line: memmap=6G!10G > > > > > > > > 3: start the guest: > > > > x86_64-softmmu/qemu-system-x86_64 -machine pc,nvdimm --enable-kvm \ > > > > -smp 16 -m 32G,maxmem=100G,slots=100 /other/VMs/centos6.img -monitor stdio > > > > > > > > 4: in guest: > > > > mkfs.ext4 /dev/pmem0 > > > > mount -o dax /dev/pmem0 /mnt/pmem/ > > > > echo > /mnt/pmem/xxx > > > > ./mmap /mnt/pmem/xxx > > > > ./read /mnt/pmem/xxx > > > > > > > > The source code of mmap and read has been attached in this mail. > > > > > > > > Hopefully, you can detect the error triggered by read test. > > > > > > > > Thanks! > > > > > > Okay, I think I've isolated this issue. Xiao's VM was an old CentOS 6 system, > > > and for some reason ext4+DAX with the old tools found in that VM fails. I was > > > able to reproduce this failure with a freshly installed CentOS 6.8 VM. > > > > > > You can see the failure with his tests, or perhaps more easily with this > > > series of commands: > > > > > > # mkfs.ext4 /dev/pmem0 > > > # mount -o dax /dev/pmem0 /mnt/pmem/ > > > # touch /mnt/pmem/x > > > # md5sum /mnt/pmem/x > > > md5sum: /mnt/pmem/x: Bad address > > > > > > This sequence of commands works fine in the old CentOS 6 system if you use XFS > > > instead of ext4, and it works fine with both ext4 and XFS in CentOS 7 and > > > with recent versions of Fedora. > > > > > > I've added the ext4 folks to this mail in case they care, but my guess is that > > > the tools in CentOS 6 are so old that it's not worth worrying about. For > > > reference, the kernel in CentOS 6 is based on 2.6.32. :) DAX was introduced > > > in v4.0. > > > > Hum, can you post 'dumpe2fs -h /dev/pmem0' output from that system when the > > md5sum fails? Because the only idea I have is that mkfs.ext4 in CentOS 6 > > creates the filesystem with a different set of features than more recent > > e2fsprogs and so we hit some untested path... > > Sure, here's the output: > > # dumpe2fs -h /dev/pmem0 > dumpe2fs 1.41.12 (17-May-2010) > Filesystem volume name: <none> > Last mounted on: /mnt/pmem > Filesystem UUID: 4cd8a836-cc54-4c59-ae0a-4a26bab0f8bc > Filesystem magic number: 0xEF53 > Filesystem revision #: 1 (dynamic) > Filesystem features: has_journal ext_attr resize_inode dir_index filetype > needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg > dir_nlink extra_isize > Filesystem flags: signed_directory_hash > Default mount options: (none) > Filesystem state: clean > Errors behavior: Continue > Filesystem OS type: Linux > Inode count: 1048576 > Block count: 4194304 > Reserved block count: 209715 > Free blocks: 4084463 > Free inodes: 1048565 > First block: 0 > Block size: 4096 > Fragment size: 4096 > Reserved GDT blocks: 1023 > Blocks per group: 32768 > Fragments per group: 32768 > Inodes per group: 8192 > Inode blocks per group: 512 > RAID stride: 1 > Flex block group size: 16 > Filesystem created: Thu Sep 8 14:45:31 2016 > Last mount time: Thu Sep 8 14:45:39 2016 > Last write time: Thu Sep 8 14:45:39 2016 > Mount count: 1 > Maximum mount count: 21 > Last checked: Thu Sep 8 14:45:31 2016 > Check interval: 15552000 (6 months) > Next check after: Tue Mar 7 13:45:31 2017 > Lifetime writes: 388 MB > Reserved blocks uid: 0 (user root) > Reserved blocks gid: 0 (group root) > First inode: 11 > Inode size: 256 > Required extra isize: 28 > Desired extra isize: 28 > Journal inode: 8 > Default directory hash: half_md4 > Directory Hash Seed: 19cad581-c46a-4212-bfa0-d527ff55db49 > Journal backup: inode blocks > Journal features: (none) > Journal size: 128M > Journal length: 32768 > Journal sequence: 0x00000002 > Journal start: 1 Hum, nothing unusual in there. I've tried reproducing on a local SLE11 SP3 machine (which is from about the same time) but everything works as expected there. Shrug... Honza -- Jan Kara <jack-IBi9RG/b67k@public.gmane.org> SUSE Labs, CR ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20160909091925.GF22777-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org>]
* Re: DAX can not work on virtual nvdimm device [not found] ` <20160909091925.GF22777-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org> @ 2016-09-09 14:03 ` Theodore Ts'o [not found] ` <20160909140327.r2j64s5xdaxnnxhx-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Theodore Ts'o @ 2016-09-09 14:03 UTC (permalink / raw) To: Jan Kara Cc: Yumei Huang, Xiao Guangrong, KVM, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org, LKML, Linux ACPI, Stefan Hajnoczi, linux-ext4-u79uwXL29TY76Z2rM5mHXA On Fri, Sep 09, 2016 at 11:19:25AM +0200, Jan Kara wrote: > > > > > 1. make the guest kernel based on your tree, the top commit is > > > > > 10d7902fa0e82b (dax: unmap/truncate on device shutdown) and > > > > > the config file can be found in this thread. What git tree is 10d7902fa0e82b from? > Hum, nothing unusual in there. I've tried reproducing on a local SLE11 SP3 > machine (which is from about the same time) but everything works as > expected there. Shrug... I just tried reproducing it via "gce-xfstests --pmem-device shell" with a downgraded e2fsprogs to stock upstream 1.42.12, and I can't reproduce it with the ext4.git tree. I'm not sure whether the git commit is critical, though. Is this a regression that was working before? Or it's possible that Red Hat did something weird with the CentOS 6 e2fsprogs.... - Ted ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20160909140327.r2j64s5xdaxnnxhx-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org>]
* Re: DAX can not work on virtual nvdimm device [not found] ` <20160909140327.r2j64s5xdaxnnxhx-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> @ 2016-09-09 16:34 ` Ross Zwisler 0 siblings, 0 replies; 6+ messages in thread From: Ross Zwisler @ 2016-09-09 16:34 UTC (permalink / raw) To: Theodore Ts'o, Jan Kara, Ross Zwisler, Xiao Guangrong, linux-ext4-u79uwXL29TY76Z2rM5mHXA, Dan Williams, Yumei Huang, KVM, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org, LKML, Linux ACPI, Stefan Hajnoczi On Fri, Sep 09, 2016 at 10:03:27AM -0400, Theodore Ts'o wrote: > On Fri, Sep 09, 2016 at 11:19:25AM +0200, Jan Kara wrote: > > > > > > 1. make the guest kernel based on your tree, the top commit is > > > > > > 10d7902fa0e82b (dax: unmap/truncate on device shutdown) and > > > > > > the config file can be found in this thread. > > What git tree is 10d7902fa0e82b from? It's from djbw's libnvdimm tree. https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git/ I don't think that commit is relevant, though. I believe this should reproduce with a vanilla v4.7 tree. I can verify if you'd like. > > Hum, nothing unusual in there. I've tried reproducing on a local SLE11 SP3 > > machine (which is from about the same time) but everything works as > > expected there. Shrug... > > I just tried reproducing it via "gce-xfstests --pmem-device shell" > with a downgraded e2fsprogs to stock upstream 1.42.12, and I can't > reproduce it with the ext4.git tree. I'm not sure whether the git > commit is critical, though. Is this a regression that was working > before? I'm not sure if we've ever tested Centos6/RHEL6 with DAX. It could be that it has never worked. > Or it's possible that Red Hat did something weird with the CentOS 6 > e2fsprogs.... ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-09-09 16:34 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- [not found] <436d7526-bf06-633d-afce-4333552d9e31@linux.intel.com> [not found] ` <CAPcyv4hVgi6Hw8Beg=Nt78+U0QNJN9mBGWEp3V6bg86NaM3Q6Q@mail.gmail.com> [not found] ` <20160819183047.GA7216@linux.intel.com> [not found] ` <600ac51c-0f61-6e53-9bfa-669c85494d1f@linux.intel.com> [not found] ` <20160829193014.GB16738@linux.intel.com> [not found] ` <61d5c470-a8bf-86d5-d1fe-b4b577a85967@linux.intel.com> [not found] ` <CAPcyv4hrFSxNcmmVLVZT4fK3+hspPCDKW0rVBj1=3GkJRgZ29Q@mail.gmail.com> [not found] ` <25098f37-53f7-6d5d-0b1a-8469bab51a9f@linux.intel.com> 2016-09-02 2:57 ` DAX can not work on virtual nvdimm device Ross Zwisler [not found] ` <20160902025738.GA26108-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> 2016-09-06 15:06 ` Jan Kara [not found] ` <20160906150620.GJ28922-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org> 2016-09-08 20:47 ` Ross Zwisler [not found] ` <20160908204708.GA15167-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> 2016-09-09 9:19 ` Jan Kara [not found] ` <20160909091925.GF22777-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org> 2016-09-09 14:03 ` Theodore Ts'o [not found] ` <20160909140327.r2j64s5xdaxnnxhx-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> 2016-09-09 16:34 ` Ross Zwisler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).