remount and conflicting ssd options?

remount and conflicting ssd options?

[Chris Murphy]

I don't understand this:

/dev/mmcblk0p3 on / type btrfs
(rw,noatime,seclabel,compress=zlib,nossd,ssd_spread,space_cache,commit=150,subvolid=260,subvol=/root)


The fstab uses ssd_spread. It looks like during startup the initial
option is ssd via autodetection and then at switchroot time, when it
goes ro to rw, the fstab options are applied and it becomes
ssd_spread. Fine.

Then later I tried

mount -o remount,ssd

To go back to regular ssd option, but nothing happens, mount still
shows ssd_spread. And then if I do

mount -o remount,nossd

I get the above mount output with nossd,ssd_spread options which would
seem to be a contradiction. At least it's confusing. So... now what?

kernel 4.10.8-200.fc25.x86_64

-- 
Chris Murphy

Re: remount and conflicting ssd options?

[Hans van Kranenburg]

On 04/15/2017 08:17 PM, Chris Murphy wrote:
> I don't understand this:
> 
> /dev/mmcblk0p3 on / type btrfs
> (rw,noatime,seclabel,compress=zlib,nossd,ssd_spread,space_cache,commit=150,subvolid=260,subvol=/root)
> 
> 
> The fstab uses ssd_spread. It looks like during startup the initial
> option is ssd via autodetection and then at switchroot time, when it
> goes ro to rw, the fstab options are applied and it becomes
> ssd_spread. Fine.
> 
> Then later I tried
> 
> mount -o remount,ssd
> 
> To go back to regular ssd option, but nothing happens, mount still
> shows ssd_spread. And then if I do
> 
> mount -o remount,nossd
> 
> I get the above mount output with nossd,ssd_spread options which would
> seem to be a contradiction. At least it's confusing. So... now what?
> 
> kernel 4.10.8-200.fc25.x86_64
> 

See thread "Cosmetics bug: remounting ssd does not clear nossd", started
on Mar 31. It has all the answers. :-)

-- 
Hans van Kranenburg

Re: remount and conflicting ssd options?

[Adam Borowski]

On Sat, Apr 15, 2017 at 12:17:25PM -0600, Chris Murphy wrote:
> I don't understand this:
> 
> /dev/mmcblk0p3 on / type btrfs
> (rw,noatime,seclabel,compress=zlib,nossd,ssd_spread,space_cache,commit=150,subvolid=260,subvol=/root)
> 
> The fstab uses ssd_spread. It looks like during startup the initial
> option is ssd via autodetection and then at switchroot time, when it
> goes ro to rw, the fstab options are applied and it becomes
> ssd_spread. Fine.
> 
> Then later I tried
> 
> mount -o remount,ssd
> 
> To go back to regular ssd option, but nothing happens, mount still
> shows ssd_spread.

ssd_spread implies ssd.

> And then if I do
> 
> mount -o remount,nossd
> 
> I get the above mount output with nossd,ssd_spread options which would
> seem to be a contradiction. At least it's confusing. So... now what?
> 
> kernel 4.10.8-200.fc25.x86_64

Already fixed in mainline.  You need linus/master from literally today if
your time zone is between +1 and +3 (inclusively), adjust by one day
accordingly if not.

Or cherry-pick 951e79663.

-- 
⢀⣴⠾⠻⢶⣦⠀ Meow!
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Collisions shmolisions, let's see them find a collision or second
⠈⠳⣄⠀⠀⠀⠀ preimage for double rot13!

Re: remount and conflicting ssd options?

[Chris Murphy]

On Sat, Apr 15, 2017 at 12:31 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> On Sat, Apr 15, 2017 at 12:17:25PM -0600, Chris Murphy wrote:
>> I don't understand this:
>>
>> /dev/mmcblk0p3 on / type btrfs
>> (rw,noatime,seclabel,compress=zlib,nossd,ssd_spread,space_cache,commit=150,subvolid=260,subvol=/root)
>>
>> The fstab uses ssd_spread. It looks like during startup the initial
>> option is ssd via autodetection and then at switchroot time, when it
>> goes ro to rw, the fstab options are applied and it becomes
>> ssd_spread. Fine.
>>
>> Then later I tried
>>
>> mount -o remount,ssd
>>
>> To go back to regular ssd option, but nothing happens, mount still
>> shows ssd_spread.
>
> ssd_spread implies ssd.

OK so it's possible to remount from ssd to ssd_spread but not back to
ssd? If I trust the mount output, it's only possible to transition
from ssd to ssd_spread, but not from ssd_spread to ssd.


>
>> And then if I do
>>
>> mount -o remount,nossd
>>
>> I get the above mount output with nossd,ssd_spread options which would
>> seem to be a contradiction. At least it's confusing. So... now what?
>>
>> kernel 4.10.8-200.fc25.x86_64
>
> Already fixed in mainline.  You need linus/master from literally today if
> your time zone is between +1 and +3 (inclusively), adjust by one day
> accordingly if not.
>
> Or cherry-pick 951e79663.

OK thanks.

Is this just fixing a mount bug? Or also with kernel messaging? I see
the enabling of ssd and spread, but I don't see a kernel message when
unsetting them. For the kernel messaging to be consistent, it'd need
to always report a message when allocation strategy is being changed.

-- 
Chris Murphy

Re: remount and conflicting ssd options?

[Adam Borowski]

On Sat, Apr 15, 2017 at 12:41:14PM -0600, Chris Murphy wrote:
> On Sat, Apr 15, 2017 at 12:31 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> > On Sat, Apr 15, 2017 at 12:17:25PM -0600, Chris Murphy wrote:
> >> Then later I tried
> >>
> >> mount -o remount,ssd
> >>
> >> To go back to regular ssd option, but nothing happens, mount still
> >> shows ssd_spread.
> >
> > ssd_spread implies ssd.
> 
> OK so it's possible to remount from ssd to ssd_spread but not back to
> ssd? If I trust the mount output, it's only possible to transition
> from ssd to ssd_spread, but not from ssd_spread to ssd.

Yeah.  This is a very minor problem (workaround: remount with nossd then
ssd), but in the light of Hans van Kranenburg's findings, it'd be a bad idea
to introduce new mount options the kernel would need to keep recognizing
forever, as the effect of those options needs some rethinking anyways.

> >
> >> And then if I do
> >>
> >> mount -o remount,nossd
> >>
> >> I get the above mount output with nossd,ssd_spread options which would
> >> seem to be a contradiction. At least it's confusing. So... now what?
> >>
> >> kernel 4.10.8-200.fc25.x86_64
> >
> > Already fixed in mainline.  You need linus/master from literally today if
> > your time zone is between +1 and +3 (inclusively), adjust by one day
> > accordingly if not.
> >
> > Or cherry-pick 951e79663.
> 
> OK thanks.
> 
> Is this just fixing a mount bug? Or also with kernel messaging?

The NOSSD option actually does nothing except for preventing SSD from being
enabled, so switching ssd<->nossd was only a messaging issue; there was no
way to go ssd_spread->ssd, though.

> I see the enabling of ssd and spread, but I don't see a kernel message
> when unsetting them.  For the kernel messaging to be consistent, it'd need
> to always report a message when allocation strategy is being changed.

It's working correctly now.  You are told what the new strategy is, with no
mention of what the old one was.

-- 
⢀⣴⠾⠻⢶⣦⠀ Meow!
⣾⠁⢠⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Collisions shmolisions, let's see them find a collision or second
⠈⠳⣄⠀⠀⠀⠀ preimage for double rot13!

Re: remount and conflicting ssd options?

[Chris Murphy]

On Sat, Apr 15, 2017 at 12:50 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> On Sat, Apr 15, 2017 at 12:41:14PM -0600, Chris Murphy wrote:
>> On Sat, Apr 15, 2017 at 12:31 PM, Adam Borowski <kilobyte@angband.pl> wrote:
>> > On Sat, Apr 15, 2017 at 12:17:25PM -0600, Chris Murphy wrote:
>> >> Then later I tried
>> >>
>> >> mount -o remount,ssd
>> >>
>> >> To go back to regular ssd option, but nothing happens, mount still
>> >> shows ssd_spread.
>> >
>> > ssd_spread implies ssd.
>>
>> OK so it's possible to remount from ssd to ssd_spread but not back to
>> ssd? If I trust the mount output, it's only possible to transition
>> from ssd to ssd_spread, but not from ssd_spread to ssd.
>
> Yeah.  This is a very minor problem (workaround: remount with nossd then
> ssd), but in the light of Hans van Kranenburg's findings, it'd be a bad idea
> to introduce new mount options the kernel would need to keep recognizing
> forever, as the effect of those options needs some rethinking anyways.

Yeah it's confusing aside from the mount behavior.

I have a performance test based on doing a bunch of system updates
(~100 rpm packages), and even with the same mount options, back to
back tests are sufficiently inconsistent that I can't tell which of
the allocator options is better. So the allocation options must be
about something other than time based performance.

I can't tell what is a slow ssd these days so I'm not sure what ssd vs
ssd_spread really mean. Plus then on spinning rust people have
reported better performance with ssd rather than the default of nossd.
While others who are using iSCSI end up with ssd by default and worse
results than with nossd.



-- 
Chris Murphy

Re: remount and conflicting ssd options?

[Hans van Kranenburg]

On 04/15/2017 10:35 PM, Chris Murphy wrote:
> On Sat, Apr 15, 2017 at 12:50 PM, Adam Borowski <kilobyte@angband.pl> wrote:
>> On Sat, Apr 15, 2017 at 12:41:14PM -0600, Chris Murphy wrote:
>>> On Sat, Apr 15, 2017 at 12:31 PM, Adam Borowski <kilobyte@angband.pl> wrote:
>>>> On Sat, Apr 15, 2017 at 12:17:25PM -0600, Chris Murphy wrote:
>>>>> Then later I tried
>>>>>
>>>>> mount -o remount,ssd
>>>>>
>>>>> To go back to regular ssd option, but nothing happens, mount still
>>>>> shows ssd_spread.
>>>>
>>>> ssd_spread implies ssd.
>>>
>>> OK so it's possible to remount from ssd to ssd_spread but not back to
>>> ssd? If I trust the mount output, it's only possible to transition
>>> from ssd to ssd_spread, but not from ssd_spread to ssd.
>>
>> Yeah.  This is a very minor problem (workaround: remount with nossd then
>> ssd), but in the light of Hans van Kranenburg's findings, it'd be a bad idea
>> to introduce new mount options the kernel would need to keep recognizing
>> forever, as the effect of those options needs some rethinking anyways.
> 
> Yeah it's confusing aside from the mount behavior.
> 
> I have a performance test based on doing a bunch of system updates
> (~100 rpm packages), and even with the same mount options, back to
> back tests are sufficiently inconsistent that I can't tell which of
> the allocator options is better.

Do you run the test with exact the same starting point (all bits on disk
in the same place, drop caches), or do you run it over and over again
while the filesystem is deepening its own grave every time?

> So the allocation options must be
> about something other than time based performance.

They're about the size of free space fragments that are (re)used.
Switching allocator policy on the same fs rapidly might give weird
results because some fragmented free space that was left behind by the
previous test might suddenly show up as delicious supper for the next.

> I can't tell what is a slow ssd these days so I'm not sure what ssd vs
> ssd_spread really mean.

afaikrn:
ssd = ignore 'small' free space fragments ('small' being something I
haven't figured out yet)
ssd_spread = do not split up a write in multiple extents

> Plus then on spinning rust people have
> reported better performance with ssd rather than the default of nossd.
> While others who are using iSCSI end up with ssd by default and worse
> results than with nossd.

<rant:)>So far we learned that the options were designed for the types
of SSD that you would be able to buy 10 years ago, that they cause
changes in write behaviour and that they tend to forget to take
(recursive-exploding) cow overhead for metadata writes into account.</rant>

-- 
Hans van Kranenburg

Re: remount and conflicting ssd options?

[Chris Murphy]

On Sat, Apr 15, 2017 at 6:41 PM, Hans van Kranenburg
<hans.van.kranenburg@mendix.com> wrote:
> On 04/15/2017 10:35 PM, Chris Murphy wrote:

>> I have a performance test based on doing a bunch of system updates
>> (~100 rpm packages), and even with the same mount options, back to
>> back tests are sufficiently inconsistent that I can't tell which of
>> the allocator options is better.
>
> Do you run the test with exact the same starting point (all bits on disk
> in the same place, drop caches), or do you run it over and over again
> while the filesystem is deepening its own grave every time?

Once staged, over a dozen snapshots are made in quick succession. The
filesystem is balanced, and fstrim has been issued. The update is of
course deleting files within a given fstree, but because those extents
are pinned due to snapshots, they aren't deleted and thus no holes.
Within each snapshot, there's nothing being created and then later
deleted in which free space fragments would develop. Except
maybe...(see below). Each snapshot is used to setup a chroot to do the
update in. Anyway, the results are all over the map, and it's not
small amounts. 5m7 for one run, then 6m20s for the next, then 4m7s for
the next. It's useless without doing maybe 1/2 dozen or more for each
set of tests, it's just too noisy.

The one thing I can say makes an obvious hit is notreelog. All other
results are 4-7 minutes long, but this one off test was 12 minutes.
This tells me there's some fsyncing going on with either dnf or rpm
itself. But not much more than this.



>> Plus then on spinning rust people have
>> reported better performance with ssd rather than the default of nossd.
>> While others who are using iSCSI end up with ssd by default and worse
>> results than with nossd.
>
> <rant:)>So far we learned that the options were designed for the types
> of SSD that you would be able to buy 10 years ago, that they cause
> changes in write behaviour and that they tend to forget to take
> (recursive-exploding) cow overhead for metadata writes into account.</rant>

And we have four totally separate classes of SSD: USB flash drives, SD
Cards, SATA SSD, NVMe.

The more mysterious one to me is when doing updates on HDD, there
seems to be a lot of head seeking. Granted there's data, and two
metadata chunks, as well as the log tree which I know each subvolume
has its own, but maybe this log tree is a cause of fragmentation. It
could certainly write out a fair bit of data, and is not subject to
snapshots pinning those sectors. Is the same space used over and over
again, kinda like the ext4 or XFS journal? Or does it meander? Anyway,
I'd think on both iSCSI and anything rotational, you want the fs doing
more sequential writes both for data and metadata, and that there's a
lot more room for optimization some day.


-- 
Chris Murphy

Re: remount and conflicting ssd options?

[Hans van Kranenburg]

On 04/15/2017 08:41 PM, Chris Murphy wrote:
> On Sat, Apr 15, 2017 at 12:31 PM, Adam Borowski <kilobyte@angband.pl> wrote:
>> On Sat, Apr 15, 2017 at 12:17:25PM -0600, Chris Murphy wrote:
>>> I don't understand this:
>>>
>>> /dev/mmcblk0p3 on / type btrfs
>>> (rw,noatime,seclabel,compress=zlib,nossd,ssd_spread,space_cache,commit=150,subvolid=260,subvol=/root)
>>>
>>> The fstab uses ssd_spread. It looks like during startup the initial
>>> option is ssd via autodetection and then at switchroot time, when it
>>> goes ro to rw, the fstab options are applied and it becomes
>>> ssd_spread. Fine.
>>>
>>> Then later I tried
>>>
>>> mount -o remount,ssd
>>>
>>> To go back to regular ssd option, but nothing happens, mount still
>>> shows ssd_spread.
>>
>> ssd_spread implies ssd.

Well, not really. ssd_spread also triggers ssd to be set to enabled, but
they're two different flags, and they're being tested separately in the
code. (!)

> 
> OK so it's possible to remount from ssd to ssd_spread but not back to
> ssd? If I trust the mount output, it's only possible to transition
> from ssd to ssd_spread, but not from ssd_spread to ssd.

That's correct. That's a bug. I think it needs a full umount to get it away.

>>
>>> And then if I do
>>>
>>> mount -o remount,nossd
>>>
>>> I get the above mount output with nossd,ssd_spread options which would
>>> seem to be a contradiction. At least it's confusing. So... now what?

As far as I understand now, this will mean that both nossd and
ssd_spread is active, which means that writes need a size + 0 (data) or
size + 64kiB (metadata) amount of free space to go in, and at the same
time it does not allow the write to be split up in multiple extents? :o

>>>
>>> kernel 4.10.8-200.fc25.x86_64
>>
>> Already fixed in mainline.  You need linus/master from literally today if
>> your time zone is between +1 and +3 (inclusively), adjust by one day
>> accordingly if not.
>>
>> Or cherry-pick 951e79663.
> 
> OK thanks.
> 
> Is this just fixing a mount bug? Or also with kernel messaging? I see
> the enabling of ssd and spread, but I don't see a kernel message when
> unsetting them. For the kernel messaging to be consistent, it'd need
> to always report a message when allocation strategy is being changed.

+1

I still haven't tested the patch myself yet, but I would definitely
expect that you should see the "not using ssd allocation scheme" appearing.

-- 
Hans van Kranenburg