XFS over device mapper dm-thin device

XFS over device mapper dm-thin device

[Todd Gill]

Hi,

I'd like to be able to take a snapshot of an XFS filesystem on top of
dm-thin.

The goal for the snapshot is:

    - fast to complete (couple seconds or better)
    - independent of origin (UUID updated?)
    - read/write

To satisfy the independent goal, I'm updating the UUID on the snapshot.
Currently "xfs_admin -U xxx" takes in the ballpark of 14 seconds on my
test system.

Is there a better approach to achieve the goals?  Or can the UUID update
be changed to be faster?

I realize the filesystem can be mounted with "nouuid" option, but would
prefer to avoid the possible trouble of duplicate UUIDs.

This is a test script I've used to illustrate the time of "xfs_admin -U":

https://pastebin.com/PFA5YRuH

Thanks for any help,
Todd

Re: XFS over device mapper dm-thin device

[Dave Chinner]

On Fri, Feb 02, 2018 at 04:45:33PM -0500, Todd Gill wrote:
> 
> Hi,
> 
> I'd like to be able to take a snapshot of an XFS filesystem on top of
> dm-thin.
> 
> The goal for the snapshot is:
> 
>     - fast to complete (couple seconds or better)
>     - independent of origin (UUID updated?)
>     - read/write
> 
> To satisfy the independent goal, I'm updating the UUID on the snapshot.
> Currently "xfs_admin -U xxx" takes in the ballpark of 14 seconds on my
> test system.

There's a very good chance that this is caused by dm-thinp doing
COW for the storage underlying the superblocks modified by the UUID
change.

Best to start by measuring how much IO is going on when you change
the UUID.  e.g. `iostat -x -d -m 1`. An strace showing how long the
xfs_db read() and write() calls to the dm-thinp device would lso be
instructive.

> Is there a better approach to achieve the goals?

No, you are doing the right thing.

> Or can the UUID update
> be changed to be faster?

Not sure - there's a fixed amount of modification we need to make
to change the UUID so I'm not sure there is any optimisations we
can make here.

If it's the speed of the underlying snapshot device doing COW after
a snapshot that is causing this issue, then there's probably nothing
we can do about it at the filesystem layer, anyway....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS over device mapper dm-thin device

[Eric Sandeen]

On 2/2/18 4:35 PM, Dave Chinner wrote:
> On Fri, Feb 02, 2018 at 04:45:33PM -0500, Todd Gill wrote:
>>
>> Hi,
>>
>> I'd like to be able to take a snapshot of an XFS filesystem on top of
>> dm-thin.
>>
>> The goal for the snapshot is:
>>
>>     - fast to complete (couple seconds or better)
>>     - independent of origin (UUID updated?)
>>     - read/write
>>
>> To satisfy the independent goal, I'm updating the UUID on the snapshot.
>> Currently "xfs_admin -U xxx" takes in the ballpark of 14 seconds on my
>> test system.
> 
> There's a very good chance that this is caused by dm-thinp doing
> COW for the storage underlying the superblocks modified by the UUID
> change.

It's actually because we rewrite the whole log, because it has the
user-facing superblock stamped in it.

I looked at this a bit, and didn't immediately see an easy way around
xfs_db's need to rewrite the whole log.

uuid_f()
                /* clear the log (setting uuid) if it's not dirty */
                if (!sb_logzero(&uu))
                        return 0;

so I think that's taking the bulk of the time.  I haven't looked closely
enough at all the mount-time log parsing to see if we could maybe write
less of it and still be valid.

> Best to start by measuring how much IO is going on when you change
> the UUID.  e.g. `iostat -x -d -m 1`. An strace showing how long the
> xfs_db read() and write() calls to the dm-thinp device would lso be
> instructive.
> 
>> Is there a better approach to achieve the goals?
> 
> No, you are doing the right thing.
> 
>> Or can the UUID update
>> be changed to be faster?
> 
> Not sure - there's a fixed amount of modification we need to make
> to change the UUID so I'm not sure there is any optimisations we
> can make here.

In retrospect I think we (I) should have left the "metadata" uuid as
the one to match the log, not the user-facing one.

I'm not sure there's a nice backwards-compatible way to un-ring
that bell, though.

-eric

Re: XFS over device mapper dm-thin device

[Dave Chinner]

On Fri, Feb 02, 2018 at 05:36:40PM -0600, Eric Sandeen wrote:
> 
> 
> On 2/2/18 4:35 PM, Dave Chinner wrote:
> > On Fri, Feb 02, 2018 at 04:45:33PM -0500, Todd Gill wrote:
> >>
> >> Hi,
> >>
> >> I'd like to be able to take a snapshot of an XFS filesystem on top of
> >> dm-thin.
> >>
> >> The goal for the snapshot is:
> >>
> >>     - fast to complete (couple seconds or better)
> >>     - independent of origin (UUID updated?)
> >>     - read/write
> >>
> >> To satisfy the independent goal, I'm updating the UUID on the snapshot.
> >> Currently "xfs_admin -U xxx" takes in the ballpark of 14 seconds on my
> >> test system.
> > 
> > There's a very good chance that this is caused by dm-thinp doing
> > COW for the storage underlying the superblocks modified by the UUID
> > change.
> 
> It's actually because we rewrite the whole log, because it has the
> user-facing superblock stamped in it.
> 
> I looked at this a bit, and didn't immediately see an easy way around
> xfs_db's need to rewrite the whole log.
> 
> uuid_f()
>                 /* clear the log (setting uuid) if it's not dirty */
>                 if (!sb_logzero(&uu))
>                         return 0;
> 
> so I think that's taking the bulk of the time.  I haven't looked closely
> enough at all the mount-time log parsing to see if we could maybe write
> less of it and still be valid.

Ah, I forgot we added that. Originally the uuid in the log was only
used to identify external log devices (or Irix based filesystems).

> >> Or can the UUID update
> >> be changed to be faster?
> > 
> > Not sure - there's a fixed amount of modification we need to make
> > to change the UUID so I'm not sure there is any optimisations we
> > can make here.
> 
> In retrospect I think we (I) should have left the "metadata" uuid as
> the one to match the log, not the user-facing one.

I suspect we can just change the log recovery code to accept either
sb_uuid or sb_metauuid if the metauuid feature bit is set and
change xfs_db to not rewrite the log when metauuid is used....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS over device mapper dm-thin device

[Eric Sandeen]

On 2/5/18 4:36 PM, Dave Chinner wrote:
>> In retrospect I think we (I) should have left the "metadata" uuid as
>> the one to match the log, not the user-facing one.
> I suspect we can just change the log recovery code to accept either
> sb_uuid or sb_metauuid if the metauuid feature bit is set and
> change xfs_db to not rewrite the log when metauuid is used....
> 
> Cheers,

That was my first thought, but then newer userspace UUID changes
won't be mountable on older kernels.   I don't think we can just
change it w/o some thought re: compatibility...

-Eric

Re: XFS over device mapper dm-thin device

[Dave Chinner]

On Mon, Feb 05, 2018 at 04:37:22PM -0600, Eric Sandeen wrote:
> 
> 
> On 2/5/18 4:36 PM, Dave Chinner wrote:
> >> In retrospect I think we (I) should have left the "metadata" uuid as
> >> the one to match the log, not the user-facing one.
> > I suspect we can just change the log recovery code to accept either
> > sb_uuid or sb_metauuid if the metauuid feature bit is set and
> > change xfs_db to not rewrite the log when metauuid is used....
> > 
> > Cheers,
> 
> That was my first thought, but then newer userspace UUID changes
> won't be mountable on older kernels.   I don't think we can just
> change it w/o some thought re: compatibility...

So maybe we should use a log incompat bit and provide an alternative
command for the old way of changing the UUID in the log?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS over device mapper dm-thin device

[Eric Sandeen]

On 2/6/18 12:11 AM, Dave Chinner wrote:
> On Mon, Feb 05, 2018 at 04:37:22PM -0600, Eric Sandeen wrote:
>>
>>
>> On 2/5/18 4:36 PM, Dave Chinner wrote:
>>>> In retrospect I think we (I) should have left the "metadata" uuid as
>>>> the one to match the log, not the user-facing one.
>>> I suspect we can just change the log recovery code to accept either
>>> sb_uuid or sb_metauuid if the metauuid feature bit is set and
>>> change xfs_db to not rewrite the log when metauuid is used....
>>>
>>> Cheers,
>>
>> That was my first thought, but then newer userspace UUID changes
>> won't be mountable on older kernels.   I don't think we can just
>> change it w/o some thought re: compatibility...
> 
> So maybe we should use a log incompat bit and provide an alternative
> command for the old way of changing the UUID in the log?

Sure, if we're happy burning an incompat bit for this, sounds good.

-Eric

Re: XFS over device mapper dm-thin device

[Darrick J. Wong]

On Wed, Feb 07, 2018 at 08:35:12PM -0600, Eric Sandeen wrote:
> 
> 
> On 2/6/18 12:11 AM, Dave Chinner wrote:
> > On Mon, Feb 05, 2018 at 04:37:22PM -0600, Eric Sandeen wrote:
> >>
> >>
> >> On 2/5/18 4:36 PM, Dave Chinner wrote:
> >>>> In retrospect I think we (I) should have left the "metadata" uuid as
> >>>> the one to match the log, not the user-facing one.
> >>> I suspect we can just change the log recovery code to accept either
> >>> sb_uuid or sb_metauuid if the metauuid feature bit is set and
> >>> change xfs_db to not rewrite the log when metauuid is used....
> >>>
> >>> Cheers,
> >>
> >> That was my first thought, but then newer userspace UUID changes
> >> won't be mountable on older kernels.   I don't think we can just
> >> change it w/o some thought re: compatibility...
> > 
> > So maybe we should use a log incompat bit and provide an alternative
> > command for the old way of changing the UUID in the log?
> 
> Sure, if we're happy burning an incompat bit for this, sounds good.

Or a log incompat bit?

--D

> 
> -Eric
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: XFS over device mapper dm-thin device

[Dave Chinner]

On Wed, Feb 07, 2018 at 09:53:40PM -0800, Darrick J. Wong wrote:
> On Wed, Feb 07, 2018 at 08:35:12PM -0600, Eric Sandeen wrote:
> > 
> > 
> > On 2/6/18 12:11 AM, Dave Chinner wrote:
> > > On Mon, Feb 05, 2018 at 04:37:22PM -0600, Eric Sandeen wrote:
> > >>
> > >>
> > >> On 2/5/18 4:36 PM, Dave Chinner wrote:
> > >>>> In retrospect I think we (I) should have left the "metadata" uuid as
> > >>>> the one to match the log, not the user-facing one.
> > >>> I suspect we can just change the log recovery code to accept either
> > >>> sb_uuid or sb_metauuid if the metauuid feature bit is set and
> > >>> change xfs_db to not rewrite the log when metauuid is used....
> > >>>
> > >>> Cheers,
> > >>
> > >> That was my first thought, but then newer userspace UUID changes
> > >> won't be mountable on older kernels.   I don't think we can just
> > >> change it w/o some thought re: compatibility...
> > > 
> > > So maybe we should use a log incompat bit and provide an alternative
> > > command for the old way of changing the UUID in the log?
> > 
> > Sure, if we're happy burning an incompat bit for this, sounds good.
> 
> Or a log incompat bit?

That's what I suggested :P

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS over device mapper dm-thin device

[Eric Sandeen]

On 2/8/18 2:25 AM, Dave Chinner wrote:
> On Wed, Feb 07, 2018 at 09:53:40PM -0800, Darrick J. Wong wrote:
>> On Wed, Feb 07, 2018 at 08:35:12PM -0600, Eric Sandeen wrote:
>>>
>>>
>>> On 2/6/18 12:11 AM, Dave Chinner wrote:
>>>> On Mon, Feb 05, 2018 at 04:37:22PM -0600, Eric Sandeen wrote:
>>>>>
>>>>>
>>>>> On 2/5/18 4:36 PM, Dave Chinner wrote:
>>>>>>> In retrospect I think we (I) should have left the "metadata" uuid as
>>>>>>> the one to match the log, not the user-facing one.
>>>>>> I suspect we can just change the log recovery code to accept either
>>>>>> sb_uuid or sb_metauuid if the metauuid feature bit is set and
>>>>>> change xfs_db to not rewrite the log when metauuid is used....
>>>>>>
>>>>>> Cheers,
>>>>>
>>>>> That was my first thought, but then newer userspace UUID changes
>>>>> won't be mountable on older kernels.   I don't think we can just
>>>>> change it w/o some thought re: compatibility...
>>>>
>>>> So maybe we should use a log incompat bit and provide an alternative
>>>> command for the old way of changing the UUID in the log?
>>>
>>> Sure, if we're happy burning an incompat bit for this, sounds good.
>>
>> Or a log incompat bit?
> 
> That's what I suggested :P

And that's what I /meant/, sorry for the lack of specificity ;)

-Eric

Re: XFS over device mapper dm-thin device

[Dave Chinner]

On Thu, Feb 08, 2018 at 08:31:21AM -0600, Eric Sandeen wrote:
> 
> 
> On 2/8/18 2:25 AM, Dave Chinner wrote:
> > On Wed, Feb 07, 2018 at 09:53:40PM -0800, Darrick J. Wong wrote:
> >> On Wed, Feb 07, 2018 at 08:35:12PM -0600, Eric Sandeen wrote:
> >>>
> >>>
> >>> On 2/6/18 12:11 AM, Dave Chinner wrote:
> >>>> On Mon, Feb 05, 2018 at 04:37:22PM -0600, Eric Sandeen wrote:
> >>>>>
> >>>>>
> >>>>> On 2/5/18 4:36 PM, Dave Chinner wrote:
> >>>>>>> In retrospect I think we (I) should have left the "metadata" uuid as
> >>>>>>> the one to match the log, not the user-facing one.
> >>>>>> I suspect we can just change the log recovery code to accept either
> >>>>>> sb_uuid or sb_metauuid if the metauuid feature bit is set and
> >>>>>> change xfs_db to not rewrite the log when metauuid is used....
> >>>>>>
> >>>>>> Cheers,
> >>>>>
> >>>>> That was my first thought, but then newer userspace UUID changes
> >>>>> won't be mountable on older kernels.   I don't think we can just
> >>>>> change it w/o some thought re: compatibility...
> >>>>
> >>>> So maybe we should use a log incompat bit and provide an alternative
> >>>> command for the old way of changing the UUID in the log?
> >>>
> >>> Sure, if we're happy burning an incompat bit for this, sounds good.
> >>
> >> Or a log incompat bit?
> > 
> > That's what I suggested :P
> 
> And that's what I /meant/, sorry for the lack of specificity ;)

So we are all happy to burn a log incompat bit for this? 

/me wonders how many one liners we can string this out into :)

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: XFS over device mapper dm-thin device

[Darrick J. Wong]

On Fri, Feb 09, 2018 at 08:11:44AM +1100, Dave Chinner wrote:
> On Thu, Feb 08, 2018 at 08:31:21AM -0600, Eric Sandeen wrote:
> > 
> > 
> > On 2/8/18 2:25 AM, Dave Chinner wrote:
> > > On Wed, Feb 07, 2018 at 09:53:40PM -0800, Darrick J. Wong wrote:
> > >> On Wed, Feb 07, 2018 at 08:35:12PM -0600, Eric Sandeen wrote:
> > >>>
> > >>>
> > >>> On 2/6/18 12:11 AM, Dave Chinner wrote:
> > >>>> On Mon, Feb 05, 2018 at 04:37:22PM -0600, Eric Sandeen wrote:
> > >>>>>
> > >>>>>
> > >>>>> On 2/5/18 4:36 PM, Dave Chinner wrote:
> > >>>>>>> In retrospect I think we (I) should have left the "metadata" uuid as
> > >>>>>>> the one to match the log, not the user-facing one.
> > >>>>>> I suspect we can just change the log recovery code to accept either
> > >>>>>> sb_uuid or sb_metauuid if the metauuid feature bit is set and
> > >>>>>> change xfs_db to not rewrite the log when metauuid is used....
> > >>>>>>
> > >>>>>> Cheers,
> > >>>>>
> > >>>>> That was my first thought, but then newer userspace UUID changes
> > >>>>> won't be mountable on older kernels.   I don't think we can just
> > >>>>> change it w/o some thought re: compatibility...
> > >>>>
> > >>>> So maybe we should use a log incompat bit and provide an alternative
> > >>>> command for the old way of changing the UUID in the log?
> > >>>
> > >>> Sure, if we're happy burning an incompat bit for this, sounds good.
> > >>
> > >> Or a log incompat bit?
> > > 
> > > That's what I suggested :P
> > 
> > And that's what I /meant/, sorry for the lack of specificity ;)
> 
> So we are all happy to burn a log incompat bit for this? 
> 
> /me wonders how many one liners we can string this out into :)

I thought we were doing haiku to approve patch ideas now. :(

--D

> -Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html