XFS and sector size on thin volumes

XFS and sector size on thin volumes

[Gionatan Danti]

Hi all,
I noticed that, when using both LVM thin volumes and ZVOLs, mkfs.xfs 
default to using 512-byte sector size.

Would setting sector size to 4K be benefical for filesystem performance?

 From what I know, sector size only affect journal and metadata writes. 
So, should I manually set sector size to 4K or it is not worth the 
(minor) inconvenience?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and sector size on thin volumes

[Dave Chinner]

On Sun, Sep 10, 2017 at 12:11:56AM +0200, Gionatan Danti wrote:
> Hi all,
> I noticed that, when using both LVM thin volumes and ZVOLs, mkfs.xfs
> default to using 512-byte sector size.
> 
> Would setting sector size to 4K be benefical for filesystem performance?

Probably not. And, worse, you open up the possibility of torn
"single sector" writes if the underlying device isn't a native 4k
sector device.

> From what I know, sector size only affect journal and metadata
> writes. So, should I manually set sector size to 4K or it is not
> worth the (minor) inconvenience?

Only the log might matter for performance, the AG headers are only
read at mount and rarely written because they are so frequently
logged. But, really, the difference between 512 byte and 4k sectors
for the log is going to be pretty damn small. Just use the defaults
and only consider changing them if you observe problems...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS and sector size on thin volumes

[Gionatan Danti]

Il 10-09-2017 01:12 Dave Chinner ha scritto:
> 
> Probably not. And, worse, you open up the possibility of torn
> "single sector" writes if the underlying device isn't a native 4k
> sector device.

Good catch, I was not thinking on how sectsize=4K could interact with 
non-4K disks.

However, if using only 4K disks below the thin volumes/ZVOLs, should I 
manually set sectsize=4K?
Or, again, I should stick with the default and stop worring?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and sector size on thin volumes

[Dave Chinner]

On Sun, Sep 10, 2017 at 01:26:42AM +0200, Gionatan Danti wrote:
> Il 10-09-2017 01:12 Dave Chinner ha scritto:
> >
> >Probably not. And, worse, you open up the possibility of torn
> >"single sector" writes if the underlying device isn't a native 4k
> >sector device.
> 
> Good catch, I was not thinking on how sectsize=4K could interact
> with non-4K disks.
> 
> However, if using only 4K disks below the thin volumes/ZVOLs, should
> I manually set sectsize=4K?
> Or, again, I should stick with the default and stop worring?

If a block device is presented with 512 sectors on 4k-only sector
drives, then that's a bug. If it's doing so with 512e drives,
then that's still a bug because it should be presenting as a
512 byte logical, 4096 byte physical sector size device and in that
case mkfs.xfs will choose 4k sectors by default.

IOWs, if the underlying device is correctly presented to mkfs.xfs
then it will choose the correct sector size by default.  dm-thinp
does the right thing with sector sizes, but I have no idea about
ZoL.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS and sector size on thin volumes

[Gionatan Danti]

> If a block device is presented with 512 sectors on 4k-only sector
> drives, then that's a bug. If it's doing so with 512e drives,
> then that's still a bug because it should be presenting as a
> 512 byte logical, 4096 byte physical sector size device and in that
> case mkfs.xfs will choose 4k sectors by default.
> 
> IOWs, if the underlying device is correctly presented to mkfs.xfs
> then it will choose the correct sector size by default.  dm-thinp
> does the right thing with sector sizes, but I have no idea about
> ZoL.

Hi Dave,
it seems ZVOLs behave... interestingly...

# 4K volume
[root@blackhole ~]# zfs create tank/vol1 -V 300M -b 4k
[root@blackhole ~]# blockdev --getss --getpbsz --getiomin --getioopt 
/dev/zvol/tank/vol1
512
4096
4096
4096

# 8K volume
[root@blackhole ~]# zfs create tank/vol1 -V 300M -b 8k
[root@blackhole ~]# blockdev --getss --getpbsz --getiomin --getioopt 
/dev/zvol/tank/vol1
512
8192
8192
8192

# 128K volume
[root@blackhole ~]# zfs create tank/vol1 -V 300M -b 128k
[root@blackhole ~]# blockdev --getss --getpbsz --getiomin --getioopt 
/dev/zvol/tank/vol1
512
131072
131072
131072

So, it seems that volume's block size does not only change the 
iomin/ioopt values, but pbsz also. On the other hand, ss is stuck at 
512B, even if I am using a 512e (4K physical sector size) underlying 
disk:

[root@blackhole ~]# blockdev --getss --getpbsz --getiomin --getioopt 
/dev/sdc
512
4096
4096
0

Creating an XFS filesystem on such a ZVOL will issue the following 
message:

"specified blocksize 4096 is less than device physical sector size 8192
switching to logical sector size 512"

In this case, should I specify block size (-b size=4k) at mkfs.xfs time, 
or not?
Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and sector size on thin volumes

[Eric Sandeen]

On 9/11/17 5:14 PM, Gionatan Danti wrote:
>> If a block device is presented with 512 sectors on 4k-only sector
>> drives, then that's a bug. If it's doing so with 512e drives,
>> then that's still a bug because it should be presenting as a
>> 512 byte logical, 4096 byte physical sector size device and in that
>> case mkfs.xfs will choose 4k sectors by default.
>>
>> IOWs, if the underlying device is correctly presented to mkfs.xfs
>> then it will choose the correct sector size by default.  dm-thinp
>> does the right thing with sector sizes, but I have no idea about
>> ZoL.
> 
> Hi Dave,
> it seems ZVOLs behave... interestingly...
> 
> # 4K volume
> [root@blackhole ~]# zfs create tank/vol1 -V 300M -b 4k
> [root@blackhole ~]# blockdev --getss --getpbsz --getiomin --getioopt /dev/zvol/tank/vol1
> 512
> 4096
> 4096
> 4096

<snip> last 3 vals follow volume's block size </snip>

That's all ok, it's just that xfs will have to use the logical sector
size (512) for its sector size, not the physical - which is large than
the maximum (4k) block it can handle on x86.
 
> So, it seems that volume's block size does not only change the iomin/ioopt values, but pbsz also. On the other hand, ss is stuck at 512B, even if I am using a 512e (4K physical sector size) underlying disk:
> 
> [root@blackhole ~]# blockdev --getss --getpbsz --getiomin --getioopt /dev/sdc
> 512
> 4096
> 4096
> 0
> 
> Creating an XFS filesystem on such a ZVOL will issue the following message:
> 
> "specified blocksize 4096 is less than device physical sector size 8192
> switching to logical sector size 512"

That's the warning we (I) talked about getting rid of.  In part because
you didn't "specify" anything.  :(  It's just telling you that you can't
go to what it thinks is the "optimal" sector size for the filesystem, i.e.
the one that matches the physical sector size of the storage; instead it is
going down to the logical sector size of 512 because it must be <=
the fs block size.

> In this case, should I specify block size (-b size=4k) at mkfs.xfs time, or not?
> Thanks.

I don't think -b size=4k will change anything.  If you specify -s size=512, that
might shut up the warning.  If the warning doesn't bother you, I don't think
anything is particularly wrong with the filesystem it created.

-Eric

Re: XFS and sector size on thin volumes

[Gionatan Danti]

Il 12-09-2017 00:28 Eric Sandeen ha scritto:
> I don't think -b size=4k will change anything.  If you specify -s 
> size=512, that
> might shut up the warning.  If the warning doesn't bother you, I don't 
> think
> anything is particularly wrong with the filesystem it created.
> 
> -Eric

Hi Eric,
so no performance decrease is expected by letting the journal to be 
aligned to 512 byte boundary, even on an underlying 512e/4K disk?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and sector size on thin volumes

[Gionatan Danti]

Il 12-09-2017 07:24 Gionatan Danti ha scritto:
> Hi Eric,
> so no performance decrease is expected by letting the journal to be
> aligned to 512 byte boundary, even on an underlying 512e/4K disk?
> 
> Thanks.

Hi all and sorry for the bump...

So, just to be sure: there is *no* performance penalty in creating an 
XFS filesystem with 512B sector size on a 512e/4Kn disk (which is not 
recognized as an AF disk due to the ZVOL in-between)?

Or should I use "-b size=4k" with mkfs.xfs?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and sector size on thin volumes

[Eric Sandeen]

On 9/16/17 11:43 AM, Gionatan Danti wrote:
> Il 12-09-2017 07:24 Gionatan Danti ha scritto:
>> Hi Eric,
>> so no performance decrease is expected by letting the journal to be
>> aligned to 512 byte boundary, even on an underlying 512e/4K disk?

I don't think I said that - this is the first time you've mentioned
/alignment/, and I've seen no indication of your alignment one way or
another.

>> Thanks.
> 
> Hi all and sorry for the bump...
> 
> So, just to be sure: there is *no* performance penalty in creating an XFS filesystem with 512B sector size on a 512e/4Kn disk (which is not recognized as an AF disk due to the ZVOL in-between)?


> 
> Or should I use "-b size=4k" with mkfs.xfs?

that specifies a 4k block size, which is already the default.

You're conflating a lot of issues here - log alignment, sector size,
physical & logical block size presented by both the underlying
storage and the zvol ...

I cannot speak to zvols, as I have no experience with them.

But if the underlying storage is 512e/4k then you may want to specify
the 4k /sector/ size with -s size=4k.

-Eric

> Thanks.
> 

Re: XFS and sector size on thin volumes

[Gionatan Danti]

Il 16-09-2017 20:33 Eric Sandeen ha scritto:
> 
> I don't think I said that - this is the first time you've mentioned
> /alignment/, and I've seen no indication of your alignment one way or
> another.
> 
> 
> 
> 
> that specifies a 4k block size, which is already the default.
> 
> You're conflating a lot of issues here - log alignment, sector size,
> physical & logical block size presented by both the underlying
> storage and the zvol ...
> 
> I cannot speak to zvols, as I have no experience with them.
> 
> But if the underlying storage is 512e/4k then you may want to specify
> the 4k /sector/ size with -s size=4k.
> 
> -Eric
> 

Hi Eric, you are right: I messed up the terminology. Sorry for the 
noise.

I was really speaking about *sector* size. In short, when using ZVOL the 
physical disk's sector size is not directly announced to the higher 
layer filesystem. This, in turn, cause mkfs.xfs to automatically select 
a 512B sector size, even when running on top of 512e/4Kn disks.

 From previous emails/threads, I was under impression that XFS sector 
size only very marginally affect performance. It that true? Can I stick 
with default mkfs settings? Or should I manually select 4K sector size 
("-s size=4k")?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8

Re: XFS and sector size on thin volumes

[Eric Sandeen]

On 9/16/17 2:59 PM, Gionatan Danti wrote:
> Il 16-09-2017 20:33 Eric Sandeen ha scritto:
>>
>> I don't think I said that - this is the first time you've mentioned
>> /alignment/, and I've seen no indication of your alignment one way or
>> another.
>>
>>
>>
>>
>> that specifies a 4k block size, which is already the default.
>>
>> You're conflating a lot of issues here - log alignment, sector size,
>> physical & logical block size presented by both the underlying
>> storage and the zvol ...
>>
>> I cannot speak to zvols, as I have no experience with them.
>>
>> But if the underlying storage is 512e/4k then you may want to specify
>> the 4k /sector/ size with -s size=4k.
>>
>> -Eric
>>
> 
> Hi Eric, you are right: I messed up the terminology. Sorry for the noise.
> 
> I was really speaking about *sector* size. In short, when using ZVOL the physical disk's sector size is not directly announced to the higher layer filesystem. This, in turn, cause mkfs.xfs to automatically select a 512B sector size, even when running on top of 512e/4Kn disks.
> 
> From previous emails/threads, I was under impression that XFS sector size only very marginally affect performance. It that true? Can I stick with default mkfs settings? Or should I manually select 4K sector size ("-s size=4k")?

If you have 512/4k disks and mkfs is not defaulting to 4k sector size,
use "-s size=4k"

-Eric