* Re: A blocksize problem about dax and ext4
@ 2015-12-24 0:00 ` Dave Chinner
0 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2015-12-24 0:00 UTC (permalink / raw)
To: Elliott, Robert (Persistent Memory)
Cc: Dan Williams, Cholerae Hu, Ted Tso, adilger.kernel@dilger.ca,
linux-nvdimm@lists.01.org, linux-ext4@vger.kernel.org,
xfs@oss.sgi.com, linux-kernel@vger.kernel.org
On Wed, Dec 23, 2015 at 09:18:05PM +0000, Elliott, Robert (Persistent Memory) wrote:
>
> > -----Original Message-----
> > From: Linux-nvdimm [mailto:linux-nvdimm-bounces@lists.01.org] On Behalf Of
> > Dan Williams
> > Sent: Wednesday, December 23, 2015 11:16 AM
> > To: Cholerae Hu <choleraehyq@gmail.com>
> > Cc: linux-nvdimm@lists.01.org
> > Subject: Re: A blocksize problem about dax and ext4
> >
> > On Wed, Dec 23, 2015 at 4:03 AM, Cholerae Hu <choleraehyq@gmail.com>
> > wrote:
> ...
> > > [root@localhost cholerae]# mount -o dax /dev/pmem0 /mnt/mem
> > > mount: wrong fs type, bad option, bad superblock on /dev/pmem0,
> > > missing codepage or helper program, or other error
> > >
> > > In some cases useful info is found in syslog - try
> > > dmesg | tail or so.
> > > [root@localhost cholerae]# dmesg | tail
> ...
> > > [ 81.779582] EXT4-fs (pmem0): error: unsupported blocksize for dax
> ...
>
> > What's the fs block size? For example:
> > # dumpe2fs -h /dev/pmem0 | grep "Block size"
> > dumpe2fs 1.42.9 (28-Dec-2013)
> > Block size: 4096
> > Depending on the size of /dev/pmem0 it may have automatically set it
> > to a block size less than 4 KiB which is incompatible with "-o dax".
>
> I noticed a few things while trying that out on both ext4 and xfs.
>
> $ sudo mkfs.ext4 -F -b 1024 /dev/pmem0
> $ sudo mount -o dax /dev/pmem0 /mnt/ext4-pmem0
> $ sudo mkfs.xfs -f -b size=1024 /dev/pmem0
> $ sudo mount -o dax /dev/pmem0 /mnt/xfs-pmem0
>
> [ 199.679195] EXT4-fs (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
> [ 199.724931] EXT4-fs (pmem0): error: unsupported block size 1024 for dax
> [ 859.077766] XFS (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
> [ 859.118106] XFS (pmem0): Filesystem block size invalid for DAX Turning DAX off.
> [ 859.156950] XFS (pmem0): Mounting V4 Filesystem
> [ 859.183626] XFS (pmem0): Ending clean mount
>
> 1. ext4 fails to mount the filesystem, while xfs just disables DAX.
> It seems like they should they be the same.
I don't really care what is done to ext4 here, but I'm not changing
XFS behaviour. I'm expecting mixed dax/non-dax fileystems to be a
thing, with DAX turned on by an inode flag on disk. Indeed, I see
the mount option going away permanently for XFS, and DAX being
controlled completely from on-disk flags. E.g. ext4 encrypted files
need to turn off DAX, while clear text files can be accessed using
DAX. This should happen completely transparently to the user....
In the situation of block size < page size, there's things we can do
to ensure that XFS will allocate page size aligned/sized extents
(extent size hints FTW). This is the same mechanism that we'll use
to ensure that extents are aligned/sized for reliable huge page
mappings. Hence while DAX /as a global option/ needs to be turned
off for sub-page block size filesystems, there's no reason why we
can't turn DAX on for files that will always allocate blocks
according to DAX constraints.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: A blocksize problem about dax and ext4
@ 2015-12-24 0:00 ` Dave Chinner
0 siblings, 0 replies; 23+ messages in thread
From: Dave Chinner @ 2015-12-24 0:00 UTC (permalink / raw)
To: Elliott, Robert (Persistent Memory)
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca, Cholerae Hu,
Dan Williams, linux-ext4@vger.kernel.org
On Wed, Dec 23, 2015 at 09:18:05PM +0000, Elliott, Robert (Persistent Memory) wrote:
>
> > -----Original Message-----
> > From: Linux-nvdimm [mailto:linux-nvdimm-bounces@lists.01.org] On Behalf Of
> > Dan Williams
> > Sent: Wednesday, December 23, 2015 11:16 AM
> > To: Cholerae Hu <choleraehyq@gmail.com>
> > Cc: linux-nvdimm@lists.01.org
> > Subject: Re: A blocksize problem about dax and ext4
> >
> > On Wed, Dec 23, 2015 at 4:03 AM, Cholerae Hu <choleraehyq@gmail.com>
> > wrote:
> ...
> > > [root@localhost cholerae]# mount -o dax /dev/pmem0 /mnt/mem
> > > mount: wrong fs type, bad option, bad superblock on /dev/pmem0,
> > > missing codepage or helper program, or other error
> > >
> > > In some cases useful info is found in syslog - try
> > > dmesg | tail or so.
> > > [root@localhost cholerae]# dmesg | tail
> ...
> > > [ 81.779582] EXT4-fs (pmem0): error: unsupported blocksize for dax
> ...
>
> > What's the fs block size? For example:
> > # dumpe2fs -h /dev/pmem0 | grep "Block size"
> > dumpe2fs 1.42.9 (28-Dec-2013)
> > Block size: 4096
> > Depending on the size of /dev/pmem0 it may have automatically set it
> > to a block size less than 4 KiB which is incompatible with "-o dax".
>
> I noticed a few things while trying that out on both ext4 and xfs.
>
> $ sudo mkfs.ext4 -F -b 1024 /dev/pmem0
> $ sudo mount -o dax /dev/pmem0 /mnt/ext4-pmem0
> $ sudo mkfs.xfs -f -b size=1024 /dev/pmem0
> $ sudo mount -o dax /dev/pmem0 /mnt/xfs-pmem0
>
> [ 199.679195] EXT4-fs (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
> [ 199.724931] EXT4-fs (pmem0): error: unsupported block size 1024 for dax
> [ 859.077766] XFS (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at your own risk
> [ 859.118106] XFS (pmem0): Filesystem block size invalid for DAX Turning DAX off.
> [ 859.156950] XFS (pmem0): Mounting V4 Filesystem
> [ 859.183626] XFS (pmem0): Ending clean mount
>
> 1. ext4 fails to mount the filesystem, while xfs just disables DAX.
> It seems like they should they be the same.
I don't really care what is done to ext4 here, but I'm not changing
XFS behaviour. I'm expecting mixed dax/non-dax fileystems to be a
thing, with DAX turned on by an inode flag on disk. Indeed, I see
the mount option going away permanently for XFS, and DAX being
controlled completely from on-disk flags. E.g. ext4 encrypted files
need to turn off DAX, while clear text files can be accessed using
DAX. This should happen completely transparently to the user....
In the situation of block size < page size, there's things we can do
to ensure that XFS will allocate page size aligned/sized extents
(extent size hints FTW). This is the same mechanism that we'll use
to ensure that extents are aligned/sized for reliable huge page
mappings. Hence while DAX /as a global option/ needs to be turned
off for sub-page block size filesystems, there's no reason why we
can't turn DAX on for files that will always allocate blocks
according to DAX constraints.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: A blocksize problem about dax and ext4
2015-12-24 0:00 ` Dave Chinner
@ 2015-12-24 0:34 ` Cholerae Hu
-1 siblings, 0 replies; 23+ messages in thread
From: Cholerae Hu @ 2015-12-24 0:34 UTC (permalink / raw)
To: Dave Chinner
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca, Dan Williams,
linux-ext4@vger.kernel.org, Elliott, Robert (Persistent Memory)
[-- Attachment #1.1: Type: text/plain, Size: 4262 bytes --]
The block size is 1024.
# dumpe2fs -h /dev/pmem0 | grep "Block size"
dumpe2fs 1.42.13 (17-May-2015)
Block size: 1024
I tried it out on xfs and I succeeded. There are the prompting messages:
# mkfs.xfs -f -b size=1024 /dev/pmem0
meta-data=/dev/pmem0 isize=512 agcount=4, agsize=32768 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1
data = bsize=1024 blocks=131072, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=1024 blocks=2571, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
# mount -o dax /dev/pmem0 /mnt/mem
The mount command doesn't return any message, and I can successfully read
or write files in /mnt/mem.
2015-12-24 8:00 GMT+08:00 Dave Chinner <david@fromorbit.com>:
> On Wed, Dec 23, 2015 at 09:18:05PM +0000, Elliott, Robert (Persistent
> Memory) wrote:
> >
> > > -----Original Message-----
> > > From: Linux-nvdimm [mailto:linux-nvdimm-bounces@lists.01.org] On
> Behalf Of
> > > Dan Williams
> > > Sent: Wednesday, December 23, 2015 11:16 AM
> > > To: Cholerae Hu <choleraehyq@gmail.com>
> > > Cc: linux-nvdimm@lists.01.org
> > > Subject: Re: A blocksize problem about dax and ext4
> > >
> > > On Wed, Dec 23, 2015 at 4:03 AM, Cholerae Hu <choleraehyq@gmail.com>
> > > wrote:
> > ...
> > > > [root@localhost cholerae]# mount -o dax /dev/pmem0 /mnt/mem
> > > > mount: wrong fs type, bad option, bad superblock on /dev/pmem0,
> > > > missing codepage or helper program, or other error
> > > >
> > > > In some cases useful info is found in syslog - try
> > > > dmesg | tail or so.
> > > > [root@localhost cholerae]# dmesg | tail
> > ...
> > > > [ 81.779582] EXT4-fs (pmem0): error: unsupported blocksize for dax
> > ...
> >
> > > What's the fs block size? For example:
> > > # dumpe2fs -h /dev/pmem0 | grep "Block size"
> > > dumpe2fs 1.42.9 (28-Dec-2013)
> > > Block size: 4096
> > > Depending on the size of /dev/pmem0 it may have automatically set it
> > > to a block size less than 4 KiB which is incompatible with "-o dax".
> >
> > I noticed a few things while trying that out on both ext4 and xfs.
> >
> > $ sudo mkfs.ext4 -F -b 1024 /dev/pmem0
> > $ sudo mount -o dax /dev/pmem0 /mnt/ext4-pmem0
> > $ sudo mkfs.xfs -f -b size=1024 /dev/pmem0
> > $ sudo mount -o dax /dev/pmem0 /mnt/xfs-pmem0
> >
> > [ 199.679195] EXT4-fs (pmem0): DAX enabled. Warning: EXPERIMENTAL, use
> at your own risk
> > [ 199.724931] EXT4-fs (pmem0): error: unsupported block size 1024 for
> dax
> > [ 859.077766] XFS (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at
> your own risk
> > [ 859.118106] XFS (pmem0): Filesystem block size invalid for DAX
> Turning DAX off.
> > [ 859.156950] XFS (pmem0): Mounting V4 Filesystem
> > [ 859.183626] XFS (pmem0): Ending clean mount
> >
> > 1. ext4 fails to mount the filesystem, while xfs just disables DAX.
> > It seems like they should they be the same.
>
> I don't really care what is done to ext4 here, but I'm not changing
> XFS behaviour. I'm expecting mixed dax/non-dax fileystems to be a
> thing, with DAX turned on by an inode flag on disk. Indeed, I see
> the mount option going away permanently for XFS, and DAX being
> controlled completely from on-disk flags. E.g. ext4 encrypted files
> need to turn off DAX, while clear text files can be accessed using
> DAX. This should happen completely transparently to the user....
>
> In the situation of block size < page size, there's things we can do
> to ensure that XFS will allocate page size aligned/sized extents
> (extent size hints FTW). This is the same mechanism that we'll use
> to ensure that extents are aligned/sized for reliable huge page
> mappings. Hence while DAX /as a global option/ needs to be turned
> off for sub-page block size filesystems, there's no reason why we
> can't turn DAX on for files that will always allocate blocks
> according to DAX constraints.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
[-- Attachment #1.2: Type: text/html, Size: 5786 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: A blocksize problem about dax and ext4
@ 2015-12-24 0:34 ` Cholerae Hu
0 siblings, 0 replies; 23+ messages in thread
From: Cholerae Hu @ 2015-12-24 0:34 UTC (permalink / raw)
To: Dave Chinner
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca, Dan Williams,
linux-ext4@vger.kernel.org, Elliott, Robert (Persistent Memory)
[-- Attachment #1.1: Type: text/plain, Size: 4262 bytes --]
The block size is 1024.
# dumpe2fs -h /dev/pmem0 | grep "Block size"
dumpe2fs 1.42.13 (17-May-2015)
Block size: 1024
I tried it out on xfs and I succeeded. There are the prompting messages:
# mkfs.xfs -f -b size=1024 /dev/pmem0
meta-data=/dev/pmem0 isize=512 agcount=4, agsize=32768 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1
data = bsize=1024 blocks=131072, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=1024 blocks=2571, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
# mount -o dax /dev/pmem0 /mnt/mem
The mount command doesn't return any message, and I can successfully read
or write files in /mnt/mem.
2015-12-24 8:00 GMT+08:00 Dave Chinner <david@fromorbit.com>:
> On Wed, Dec 23, 2015 at 09:18:05PM +0000, Elliott, Robert (Persistent
> Memory) wrote:
> >
> > > -----Original Message-----
> > > From: Linux-nvdimm [mailto:linux-nvdimm-bounces@lists.01.org] On
> Behalf Of
> > > Dan Williams
> > > Sent: Wednesday, December 23, 2015 11:16 AM
> > > To: Cholerae Hu <choleraehyq@gmail.com>
> > > Cc: linux-nvdimm@lists.01.org
> > > Subject: Re: A blocksize problem about dax and ext4
> > >
> > > On Wed, Dec 23, 2015 at 4:03 AM, Cholerae Hu <choleraehyq@gmail.com>
> > > wrote:
> > ...
> > > > [root@localhost cholerae]# mount -o dax /dev/pmem0 /mnt/mem
> > > > mount: wrong fs type, bad option, bad superblock on /dev/pmem0,
> > > > missing codepage or helper program, or other error
> > > >
> > > > In some cases useful info is found in syslog - try
> > > > dmesg | tail or so.
> > > > [root@localhost cholerae]# dmesg | tail
> > ...
> > > > [ 81.779582] EXT4-fs (pmem0): error: unsupported blocksize for dax
> > ...
> >
> > > What's the fs block size? For example:
> > > # dumpe2fs -h /dev/pmem0 | grep "Block size"
> > > dumpe2fs 1.42.9 (28-Dec-2013)
> > > Block size: 4096
> > > Depending on the size of /dev/pmem0 it may have automatically set it
> > > to a block size less than 4 KiB which is incompatible with "-o dax".
> >
> > I noticed a few things while trying that out on both ext4 and xfs.
> >
> > $ sudo mkfs.ext4 -F -b 1024 /dev/pmem0
> > $ sudo mount -o dax /dev/pmem0 /mnt/ext4-pmem0
> > $ sudo mkfs.xfs -f -b size=1024 /dev/pmem0
> > $ sudo mount -o dax /dev/pmem0 /mnt/xfs-pmem0
> >
> > [ 199.679195] EXT4-fs (pmem0): DAX enabled. Warning: EXPERIMENTAL, use
> at your own risk
> > [ 199.724931] EXT4-fs (pmem0): error: unsupported block size 1024 for
> dax
> > [ 859.077766] XFS (pmem0): DAX enabled. Warning: EXPERIMENTAL, use at
> your own risk
> > [ 859.118106] XFS (pmem0): Filesystem block size invalid for DAX
> Turning DAX off.
> > [ 859.156950] XFS (pmem0): Mounting V4 Filesystem
> > [ 859.183626] XFS (pmem0): Ending clean mount
> >
> > 1. ext4 fails to mount the filesystem, while xfs just disables DAX.
> > It seems like they should they be the same.
>
> I don't really care what is done to ext4 here, but I'm not changing
> XFS behaviour. I'm expecting mixed dax/non-dax fileystems to be a
> thing, with DAX turned on by an inode flag on disk. Indeed, I see
> the mount option going away permanently for XFS, and DAX being
> controlled completely from on-disk flags. E.g. ext4 encrypted files
> need to turn off DAX, while clear text files can be accessed using
> DAX. This should happen completely transparently to the user....
>
> In the situation of block size < page size, there's things we can do
> to ensure that XFS will allocate page size aligned/sized extents
> (extent size hints FTW). This is the same mechanism that we'll use
> to ensure that extents are aligned/sized for reliable huge page
> mappings. Hence while DAX /as a global option/ needs to be turned
> off for sub-page block size filesystems, there's no reason why we
> can't turn DAX on for files that will always allocate blocks
> according to DAX constraints.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
[-- Attachment #1.2: Type: text/html, Size: 5786 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: A blocksize problem about dax and ext4
2015-12-24 0:34 ` Cholerae Hu
(?)
@ 2015-12-24 0:58 ` Dan Williams
-1 siblings, 0 replies; 23+ messages in thread
From: Dan Williams @ 2015-12-24 0:58 UTC (permalink / raw)
To: Cholerae Hu
Cc: Dave Chinner, Elliott, Robert (Persistent Memory), Ted Tso,
adilger.kernel@dilger.ca, linux-nvdimm@lists.01.org,
linux-ext4@vger.kernel.org, xfs@oss.sgi.com,
linux-kernel@vger.kernel.org
On Wed, Dec 23, 2015 at 4:34 PM, Cholerae Hu <choleraehyq@gmail.com> wrote:
> The block size is 1024.
> # dumpe2fs -h /dev/pmem0 | grep "Block size"
> dumpe2fs 1.42.13 (17-May-2015)
> Block size: 1024
>
> I tried it out on xfs and I succeeded. There are the prompting messages:
> # mkfs.xfs -f -b size=1024 /dev/pmem0
> meta-data=/dev/pmem0 isize=512 agcount=4, agsize=32768 blks
> = sectsz=512 attr=2, projid32bit=1
> = crc=1 finobt=1
> data = bsize=1024 blocks=131072, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0 ftype=1
> log =internal log bsize=1024 blocks=2571, version=2
> = sectsz=512 sunit=0 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
> # mount -o dax /dev/pmem0 /mnt/mem
>
> The mount command doesn't return any message, and I can successfully read or
> write files in /mnt/mem.
>
xfs will silently disable dax when the fs block size is too small,
i.e. your mmap() operations are backed by page cache in this case.
Currently the only indication of whether a mapping is DAX backed or
not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
of /proc/<pid>/smaps)
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: A blocksize problem about dax and ext4
@ 2015-12-24 0:58 ` Dan Williams
0 siblings, 0 replies; 23+ messages in thread
From: Dan Williams @ 2015-12-24 0:58 UTC (permalink / raw)
To: Cholerae Hu
Cc: Dave Chinner, Elliott, Robert (Persistent Memory), Ted Tso,
adilger.kernel@dilger.ca, linux-nvdimm@lists.01.org,
linux-ext4@vger.kernel.org, xfs@oss.sgi.com,
linux-kernel@vger.kernel.org
On Wed, Dec 23, 2015 at 4:34 PM, Cholerae Hu <choleraehyq@gmail.com> wrote:
> The block size is 1024.
> # dumpe2fs -h /dev/pmem0 | grep "Block size"
> dumpe2fs 1.42.13 (17-May-2015)
> Block size: 1024
>
> I tried it out on xfs and I succeeded. There are the prompting messages:
> # mkfs.xfs -f -b size=1024 /dev/pmem0
> meta-data=/dev/pmem0 isize=512 agcount=4, agsize=32768 blks
> = sectsz=512 attr=2, projid32bit=1
> = crc=1 finobt=1
> data = bsize=1024 blocks=131072, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0 ftype=1
> log =internal log bsize=1024 blocks=2571, version=2
> = sectsz=512 sunit=0 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
> # mount -o dax /dev/pmem0 /mnt/mem
>
> The mount command doesn't return any message, and I can successfully read or
> write files in /mnt/mem.
>
xfs will silently disable dax when the fs block size is too small,
i.e. your mmap() operations are backed by page cache in this case.
Currently the only indication of whether a mapping is DAX backed or
not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
of /proc/<pid>/smaps)
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: A blocksize problem about dax and ext4
@ 2015-12-24 0:58 ` Dan Williams
0 siblings, 0 replies; 23+ messages in thread
From: Dan Williams @ 2015-12-24 0:58 UTC (permalink / raw)
To: Cholerae Hu
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca,
linux-ext4@vger.kernel.org, Elliott, Robert (Persistent Memory)
On Wed, Dec 23, 2015 at 4:34 PM, Cholerae Hu <choleraehyq@gmail.com> wrote:
> The block size is 1024.
> # dumpe2fs -h /dev/pmem0 | grep "Block size"
> dumpe2fs 1.42.13 (17-May-2015)
> Block size: 1024
>
> I tried it out on xfs and I succeeded. There are the prompting messages:
> # mkfs.xfs -f -b size=1024 /dev/pmem0
> meta-data=/dev/pmem0 isize=512 agcount=4, agsize=32768 blks
> = sectsz=512 attr=2, projid32bit=1
> = crc=1 finobt=1
> data = bsize=1024 blocks=131072, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0 ftype=1
> log =internal log bsize=1024 blocks=2571, version=2
> = sectsz=512 sunit=0 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
> # mount -o dax /dev/pmem0 /mnt/mem
>
> The mount command doesn't return any message, and I can successfully read or
> write files in /mnt/mem.
>
xfs will silently disable dax when the fs block size is too small,
i.e. your mmap() operations are backed by page cache in this case.
Currently the only indication of whether a mapping is DAX backed or
not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
of /proc/<pid>/smaps)
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: A blocksize problem about dax and ext4
2015-12-24 0:58 ` Dan Williams
@ 2015-12-24 2:36 ` Cholerae Hu
-1 siblings, 0 replies; 23+ messages in thread
From: Cholerae Hu @ 2015-12-24 2:36 UTC (permalink / raw)
To: Dan Williams
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca,
linux-ext4@vger.kernel.org, Elliott, Robert (Persistent Memory)
[-- Attachment #1.1: Type: text/plain, Size: 2824 bytes --]
>
> xfs will silently disable dax when the fs block size is too small,
> i.e. your mmap() operations are backed by page cache in this case.
> Currently the only indication of whether a mapping is DAX backed or
> not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
> of /proc/<pid>/smaps)
Did you mean that I should make the blocksize bigger until the mount
command tell me that dax is enabled?
I have tried to set blocksize to 2048/4096/8192. When 2048/4096 the mount
command remained silent, when 8192 it prompted:
$ sudo mkfs.xfs -f -b size=8192 /dev/pmem0
meta-data=/dev/pmem0 isize=512 agcount=4, agsize=4096 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1
data = bsize=8192 blocks=16384, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=8192 ascii-ci=0 ftype=1
log =internal log bsize=8192 blocks=558, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=8192 blocks=0, rtextents=0
$ sudo mount -o dax /dev/pmem0 /mnt/mem
mount: mount /dev/pmem0 on /mnt/mem failed: Function not implemented
2015-12-24 8:58 GMT+08:00 Dan Williams <dan.j.williams@intel.com>:
> On Wed, Dec 23, 2015 at 4:34 PM, Cholerae Hu <choleraehyq@gmail.com>
> wrote:
> > The block size is 1024.
> > # dumpe2fs -h /dev/pmem0 | grep "Block size"
> > dumpe2fs 1.42.13 (17-May-2015)
> > Block size: 1024
> >
> > I tried it out on xfs and I succeeded. There are the prompting messages:
> > # mkfs.xfs -f -b size=1024 /dev/pmem0
> > meta-data=/dev/pmem0 isize=512 agcount=4, agsize=32768
> blks
> > = sectsz=512 attr=2, projid32bit=1
> > = crc=1 finobt=1
> > data = bsize=1024 blocks=131072, imaxpct=25
> > = sunit=0 swidth=0 blks
> > naming =version 2 bsize=4096 ascii-ci=0 ftype=1
> > log =internal log bsize=1024 blocks=2571, version=2
> > = sectsz=512 sunit=0 blks, lazy-count=1
> > realtime =none extsz=4096 blocks=0, rtextents=0
> > # mount -o dax /dev/pmem0 /mnt/mem
> >
> > The mount command doesn't return any message, and I can successfully
> read or
> > write files in /mnt/mem.
> >
>
> xfs will silently disable dax when the fs block size is too small,
> i.e. your mmap() operations are backed by page cache in this case.
> Currently the only indication of whether a mapping is DAX backed or
> not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
> of /proc/<pid>/smaps)
>
[-- Attachment #1.2: Type: text/html, Size: 4201 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: A blocksize problem about dax and ext4
@ 2015-12-24 2:36 ` Cholerae Hu
0 siblings, 0 replies; 23+ messages in thread
From: Cholerae Hu @ 2015-12-24 2:36 UTC (permalink / raw)
To: Dan Williams
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca,
linux-ext4@vger.kernel.org, Elliott, Robert (Persistent Memory)
[-- Attachment #1.1: Type: text/plain, Size: 2824 bytes --]
>
> xfs will silently disable dax when the fs block size is too small,
> i.e. your mmap() operations are backed by page cache in this case.
> Currently the only indication of whether a mapping is DAX backed or
> not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
> of /proc/<pid>/smaps)
Did you mean that I should make the blocksize bigger until the mount
command tell me that dax is enabled?
I have tried to set blocksize to 2048/4096/8192. When 2048/4096 the mount
command remained silent, when 8192 it prompted:
$ sudo mkfs.xfs -f -b size=8192 /dev/pmem0
meta-data=/dev/pmem0 isize=512 agcount=4, agsize=4096 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1
data = bsize=8192 blocks=16384, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=8192 ascii-ci=0 ftype=1
log =internal log bsize=8192 blocks=558, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=8192 blocks=0, rtextents=0
$ sudo mount -o dax /dev/pmem0 /mnt/mem
mount: mount /dev/pmem0 on /mnt/mem failed: Function not implemented
2015-12-24 8:58 GMT+08:00 Dan Williams <dan.j.williams@intel.com>:
> On Wed, Dec 23, 2015 at 4:34 PM, Cholerae Hu <choleraehyq@gmail.com>
> wrote:
> > The block size is 1024.
> > # dumpe2fs -h /dev/pmem0 | grep "Block size"
> > dumpe2fs 1.42.13 (17-May-2015)
> > Block size: 1024
> >
> > I tried it out on xfs and I succeeded. There are the prompting messages:
> > # mkfs.xfs -f -b size=1024 /dev/pmem0
> > meta-data=/dev/pmem0 isize=512 agcount=4, agsize=32768
> blks
> > = sectsz=512 attr=2, projid32bit=1
> > = crc=1 finobt=1
> > data = bsize=1024 blocks=131072, imaxpct=25
> > = sunit=0 swidth=0 blks
> > naming =version 2 bsize=4096 ascii-ci=0 ftype=1
> > log =internal log bsize=1024 blocks=2571, version=2
> > = sectsz=512 sunit=0 blks, lazy-count=1
> > realtime =none extsz=4096 blocks=0, rtextents=0
> > # mount -o dax /dev/pmem0 /mnt/mem
> >
> > The mount command doesn't return any message, and I can successfully
> read or
> > write files in /mnt/mem.
> >
>
> xfs will silently disable dax when the fs block size is too small,
> i.e. your mmap() operations are backed by page cache in this case.
> Currently the only indication of whether a mapping is DAX backed or
> not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
> of /proc/<pid>/smaps)
>
[-- Attachment #1.2: Type: text/html, Size: 4201 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread* RE: A blocksize problem about dax and ext4
2015-12-24 2:36 ` Cholerae Hu
(?)
@ 2015-12-24 2:47 ` Elliott, Robert (Persistent Memory)
-1 siblings, 0 replies; 23+ messages in thread
From: Elliott, Robert (Persistent Memory) @ 2015-12-24 2:47 UTC (permalink / raw)
To: Cholerae Hu, Dan Williams
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca,
linux-ext4@vger.kernel.org
> -----Original Message-----
> From: Cholerae Hu [mailto:choleraehyq@gmail.com]
> Sent: Wednesday, December 23, 2015 8:36 PM
> Subject: Re: A blocksize problem about dax and ext4
...
> xfs will silently disable dax when the fs block size is too small,
> i.e. your mmap() operations are backed by page cache in this case.
> Currently the only indication of whether a mapping is DAX backed or
> not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
> of /proc/<pid>/smaps)
>
> Did you mean that I should make the blocksize bigger until the mount
> command tell me that dax is enabled?
To really use DAX, the filesystem block size must match the
system CPU's page size, which is probably 4096 bytes.
---
Robert Elliott, HPE Persistent Memory
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread* RE: A blocksize problem about dax and ext4
@ 2015-12-24 2:47 ` Elliott, Robert (Persistent Memory)
0 siblings, 0 replies; 23+ messages in thread
From: Elliott, Robert (Persistent Memory) @ 2015-12-24 2:47 UTC (permalink / raw)
To: Cholerae Hu, Dan Williams
Cc: Dave Chinner, Ted Tso, adilger.kernel@dilger.ca,
linux-nvdimm@lists.01.org, linux-ext4@vger.kernel.org,
xfs@oss.sgi.com, linux-kernel@vger.kernel.org
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 920 bytes --]
> -----Original Message-----
> From: Cholerae Hu [mailto:choleraehyq@gmail.com]
> Sent: Wednesday, December 23, 2015 8:36 PM
> Subject: Re: A blocksize problem about dax and ext4
...
> xfs will silently disable dax when the fs block size is too small,
> i.e. your mmap() operations are backed by page cache in this case.
> Currently the only indication of whether a mapping is DAX backed or
> not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
> of /proc/<pid>/smaps)
>
> Did you mean that I should make the blocksize bigger until the mount
> command tell me that dax is enabled?
To really use DAX, the filesystem block size must match the
system CPU's page size, which is probably 4096 bytes.
---
Robert Elliott, HPE Persistent Memory
ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥
^ permalink raw reply [flat|nested] 23+ messages in thread* RE: A blocksize problem about dax and ext4
@ 2015-12-24 2:47 ` Elliott, Robert (Persistent Memory)
0 siblings, 0 replies; 23+ messages in thread
From: Elliott, Robert (Persistent Memory) @ 2015-12-24 2:47 UTC (permalink / raw)
To: Cholerae Hu, Dan Williams
Cc: Dave Chinner, Ted Tso, adilger.kernel@dilger.ca,
linux-nvdimm@lists.01.org, linux-ext4@vger.kernel.org,
xfs@oss.sgi.com, linux-kernel@vger.kernel.org
> -----Original Message-----
> From: Cholerae Hu [mailto:choleraehyq@gmail.com]
> Sent: Wednesday, December 23, 2015 8:36 PM
> Subject: Re: A blocksize problem about dax and ext4
...
> xfs will silently disable dax when the fs block size is too small,
> i.e. your mmap() operations are backed by page cache in this case.
> Currently the only indication of whether a mapping is DAX backed or
> not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
> of /proc/<pid>/smaps)
>
> Did you mean that I should make the blocksize bigger until the mount
> command tell me that dax is enabled?
To really use DAX, the filesystem block size must match the
system CPU's page size, which is probably 4096 bytes.
---
Robert Elliott, HPE Persistent Memory
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: A blocksize problem about dax and ext4
2015-12-24 2:47 ` Elliott, Robert (Persistent Memory)
(?)
(?)
@ 2015-12-24 4:13 ` Cholerae Hu
-1 siblings, 0 replies; 23+ messages in thread
From: Cholerae Hu @ 2015-12-24 4:13 UTC (permalink / raw)
To: Elliott, Robert (Persistent Memory)
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca, Dan Williams,
linux-ext4@vger.kernel.org
[-- Attachment #1.1: Type: text/plain, Size: 1008 bytes --]
I used ext4 and set the blocksize to 4096, the mount command was silent.
Does it indicate that dax has already been enabled?
2015-12-24 10:47 GMT+08:00 Elliott, Robert (Persistent Memory) <
elliott@hpe.com>:
> > -----Original Message-----
> > From: Cholerae Hu [mailto:choleraehyq@gmail.com]
> > Sent: Wednesday, December 23, 2015 8:36 PM
> > Subject: Re: A blocksize problem about dax and ext4
> ...
> > xfs will silently disable dax when the fs block size is too small,
> > i.e. your mmap() operations are backed by page cache in this case.
> > Currently the only indication of whether a mapping is DAX backed or
> > not is the presence of the VM_MIXEDMAP flag ("mm" in the VmFlags field
> > of /proc/<pid>/smaps)
> >
> > Did you mean that I should make the blocksize bigger until the mount
> > command tell me that dax is enabled?
>
> To really use DAX, the filesystem block size must match the
> system CPU's page size, which is probably 4096 bytes.
>
> ---
> Robert Elliott, HPE Persistent Memory
>
>
[-- Attachment #1.2: Type: text/html, Size: 1549 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: A blocksize problem about dax and ext4
2015-12-24 2:47 ` Elliott, Robert (Persistent Memory)
@ 2015-12-24 10:11 ` Christoph Hellwig
-1 siblings, 0 replies; 23+ messages in thread
From: Christoph Hellwig @ 2015-12-24 10:11 UTC (permalink / raw)
To: Elliott, Robert (Persistent Memory)
Cc: Cholerae Hu, Dan Williams, Dave Chinner, Ted Tso,
adilger.kernel@dilger.ca, linux-nvdimm@lists.01.org,
linux-ext4@vger.kernel.org, xfs@oss.sgi.com,
linux-kernel@vger.kernel.org
On Thu, Dec 24, 2015 at 02:47:07AM +0000, Elliott, Robert (Persistent Memory) wrote:
> > Did you mean that I should make the blocksize bigger until the mount
> > command tell me that dax is enabled?
>
> To really use DAX, the filesystem block size must match the
> system CPU's page size, which is probably 4096 bytes.
No, it doesn't. File you use for DAX must be aligne at page size
granularity. For XFS you could do this with the per-inode extent size
hint for example even if the overall block size is smaller.
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: A blocksize problem about dax and ext4
@ 2015-12-24 10:11 ` Christoph Hellwig
0 siblings, 0 replies; 23+ messages in thread
From: Christoph Hellwig @ 2015-12-24 10:11 UTC (permalink / raw)
To: Elliott, Robert (Persistent Memory)
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca, Cholerae Hu,
Dan Williams, linux-ext4@vger.kernel.org
On Thu, Dec 24, 2015 at 02:47:07AM +0000, Elliott, Robert (Persistent Memory) wrote:
> > Did you mean that I should make the blocksize bigger until the mount
> > command tell me that dax is enabled?
>
> To really use DAX, the filesystem block size must match the
> system CPU's page size, which is probably 4096 bytes.
No, it doesn't. File you use for DAX must be aligne at page size
granularity. For XFS you could do this with the per-inode extent size
hint for example even if the overall block size is smaller.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread
* RE: A blocksize problem about dax and ext4
2015-12-24 10:11 ` Christoph Hellwig
@ 2015-12-24 12:26 ` Elliott, Robert (Persistent Memory)
-1 siblings, 0 replies; 23+ messages in thread
From: Elliott, Robert (Persistent Memory) @ 2015-12-24 12:26 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca, Cholerae Hu,
Dan Williams, linux-ext4@vger.kernel.org
> -----Original Message-----
> From: Christoph Hellwig [mailto:hch@infradead.org]
> Sent: Thursday, December 24, 2015 4:11 AM
> Subject: Re: A blocksize problem about dax and ext4
>
> On Thu, Dec 24, 2015 at 02:47:07AM +0000, Elliott, Robert (Persistent
> Memory) wrote:
> > > Did you mean that I should make the blocksize bigger until the mount
> > > command tell me that dax is enabled?
> >
> > To really use DAX, the filesystem block size must match the
> > system CPU's page size, which is probably 4096 bytes.
>
> No, it doesn't. File you use for DAX must be aligne at page size
> granularity. For XFS you could do this with the per-inode extent
> size hint for example even if the overall block size is smaller.
I think that's a future goal.
Currently, the checks are like this:
if (sb->s_blocksize != PAGE_SIZE) {
xfs_alert(mp,
"Filesystem block size invalid for DAX Turning DAX off.");
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread* RE: A blocksize problem about dax and ext4
@ 2015-12-24 12:26 ` Elliott, Robert (Persistent Memory)
0 siblings, 0 replies; 23+ messages in thread
From: Elliott, Robert (Persistent Memory) @ 2015-12-24 12:26 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Cholerae Hu, Dan Williams, Dave Chinner, Ted Tso,
adilger.kernel@dilger.ca, linux-nvdimm@lists.01.org,
linux-ext4@vger.kernel.org, xfs@oss.sgi.com,
linux-kernel@vger.kernel.org
> -----Original Message-----
> From: Christoph Hellwig [mailto:hch@infradead.org]
> Sent: Thursday, December 24, 2015 4:11 AM
> Subject: Re: A blocksize problem about dax and ext4
>
> On Thu, Dec 24, 2015 at 02:47:07AM +0000, Elliott, Robert (Persistent
> Memory) wrote:
> > > Did you mean that I should make the blocksize bigger until the mount
> > > command tell me that dax is enabled?
> >
> > To really use DAX, the filesystem block size must match the
> > system CPU's page size, which is probably 4096 bytes.
>
> No, it doesn't. File you use for DAX must be aligne at page size
> granularity. For XFS you could do this with the per-inode extent
> size hint for example even if the overall block size is smaller.
I think that's a future goal.
Currently, the checks are like this:
if (sb->s_blocksize != PAGE_SIZE) {
xfs_alert(mp,
"Filesystem block size invalid for DAX Turning DAX off.");
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: A blocksize problem about dax and ext4
2015-12-24 0:00 ` Dave Chinner
@ 2016-01-08 17:22 ` Jeff Moyer
-1 siblings, 0 replies; 23+ messages in thread
From: Jeff Moyer @ 2016-01-08 17:22 UTC (permalink / raw)
To: Dave Chinner
Cc: Elliott, Robert (Persistent Memory), Ted Tso,
linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca, Cholerae Hu,
linux-ext4@vger.kernel.org
Dave Chinner <david@fromorbit.com> writes:
>> 1. ext4 fails to mount the filesystem, while xfs just disables DAX.
>> It seems like they should they be the same.
I agree, it would be nice if they were the same.
> I don't really care what is done to ext4 here, but I'm not changing
> XFS behaviour. I'm expecting mixed dax/non-dax fileystems to be a
> thing, with DAX turned on by an inode flag on disk. Indeed, I see
> the mount option going away permanently for XFS, and DAX being
> controlled completely from on-disk flags. E.g. ext4 encrypted files
> need to turn off DAX, while clear text files can be accessed using
> DAX. This should happen completely transparently to the user....
The one thing we definitely need is a common way for an application to
open a file in dax mode. So, whatever ends up being the interface for
xfs in the future sure as heck better work for ext4.
I'm also not super keen on just getting rid of the dax mount option. I
understand why you'd want to do that, but I think it should stay, with
documentation on when you simply won't get dax mappings.
Please think about the poor programmers and system administrators that
have to use these interfaces.
Thanks,
Jeff
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: A blocksize problem about dax and ext4
@ 2016-01-08 17:22 ` Jeff Moyer
0 siblings, 0 replies; 23+ messages in thread
From: Jeff Moyer @ 2016-01-08 17:22 UTC (permalink / raw)
To: Dave Chinner
Cc: Ted Tso, linux-nvdimm@lists.01.org, linux-kernel@vger.kernel.org,
xfs@oss.sgi.com, adilger.kernel@dilger.ca, Cholerae Hu,
linux-ext4@vger.kernel.org, Elliott, Robert (Persistent Memory)
Dave Chinner <david@fromorbit.com> writes:
>> 1. ext4 fails to mount the filesystem, while xfs just disables DAX.
>> It seems like they should they be the same.
I agree, it would be nice if they were the same.
> I don't really care what is done to ext4 here, but I'm not changing
> XFS behaviour. I'm expecting mixed dax/non-dax fileystems to be a
> thing, with DAX turned on by an inode flag on disk. Indeed, I see
> the mount option going away permanently for XFS, and DAX being
> controlled completely from on-disk flags. E.g. ext4 encrypted files
> need to turn off DAX, while clear text files can be accessed using
> DAX. This should happen completely transparently to the user....
The one thing we definitely need is a common way for an application to
open a file in dax mode. So, whatever ends up being the interface for
xfs in the future sure as heck better work for ext4.
I'm also not super keen on just getting rid of the dax mount option. I
understand why you'd want to do that, but I think it should stay, with
documentation on when you simply won't get dax mappings.
Please think about the poor programmers and system administrators that
have to use these interfaces.
Thanks,
Jeff
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 23+ messages in thread