Snapshots

Snapshots

[Phillip Susi]

I have been playing around with the snapshot target with dmsetup, and I 
can set up a snapshot that diverts writes to a backing file, so that 
modifications to the snapshot device are discarded when I remove the 
snapshot device and go back to using the origin device.  What I can't 
figure out is how to keep the origin device mounted and have the 
snapshot device appear to contain the origin's data at the time it was 
created, even though the origin continues to be modified.

I thought one of the main ideas of the snapshot was to be able to backup 
a consistent filesystem image even though the filesystem continues to be 
written to, but it seems that having the origin device mounted gets an 
exclusive lock on it so that the snapshot device can not be targeted at 
it, so how do you do this?

Re: Snapshots

[Jun'ichi Nomura]

Phillip Susi wrote:
> I have been playing around with the snapshot target with dmsetup, and I
> can set up a snapshot that diverts writes to a backing file, so that
> modifications to the snapshot device are discarded when I remove the
> snapshot device and go back to using the origin device.  What I can't
> figure out is how to keep the origin device mounted and have the
> snapshot device appear to contain the origin's data at the time it was
> created, even though the origin continues to be modified.
> 
> I thought one of the main ideas of the snapshot was to be able to backup
> a consistent filesystem image even though the filesystem continues to be
> written to, but it seems that having the origin device mounted gets an
> exclusive lock on it so that the snapshot device can not be targeted at
> it, so how do you do this?

LVM2 does it by putting a linear mapping on top of the origin device.

I'm not sure if this meets your requirement, but something like this:

1. Put a linear mapping on the origin
   (/dev/sdb2 is origin in this example)

# echo "0 $(blockdev --getsz /dev/sdb2) linear /dev/sdb2 0" | dmsetup create sdb2

2. Use the device as you like..

# mkfs.ext3 /dev/mapper/sdb2
# mount /dev/mapper/sdb2 /mnt/1
..

3. Insert a snapshot-origin map *below* the dm-linear

# dmsetup suspend sdb2
# echo $(dmsetup table sdb2) | dmsetup create sdb2-origin
# echo "0 $(blockdev --getsz /dev/mapper/sdb2-origin) snapshot-origin /dev/mapper/sdb2-origin" | dmsetup load sdb2
# dmsetup resume sdb2

4. Create any number of snapshots for the origin map

# echo "0 $(blockdev --getsz /dev/mapper/sdb2) snapshot /dev/mapper/sdb2-origin /dev/sdc2 P 8 " | dmsetup create
sdb2-snapshot
# dmsetup ls --tree
sdb2 (253:19)
 `-sdb2-origin (253:21)
    `- (8:18)
sdb2-snapshot1 (253:20)
 |- (8:34)
 `-sdb2-origin (253:21)
    `- (8:18)
# dmsetup table | grep sdb
sdb2: 0 15615180 linear 253:21 0
sdb2-snapshot1: 0 15615180 snapshot 253:21 8:34 P 8
sdb2-origin: 0 15615180 linear 8:18 0

If dm can insert a mapping *below* the device, you can do
without the dm-linear layer. But there's no such feature yet.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation of America

Re: Snapshots

[Phillip Susi]

Jun'ichi Nomura wrote:
> LVM2 does it by putting a linear mapping on top of the origin device.

How does this work?  I'm trying to figure out the whole locking business
and it seems that mounting the device, or using it as a target in a dm
table place an exclusive lock on the device, so how can you add a linear
mapping to the origin device, if it is already used as a target in the
snapshot device, and thus locked?

Re: Snapshots

[Jun'ichi Nomura]

Phillip Susi wrote:
> Jun'ichi Nomura wrote:
>> LVM2 does it by putting a linear mapping on top of the origin device.
> 
> How does this work?  I'm trying to figure out the whole locking business
> and it seems that mounting the device, or using it as a target in a dm
> table place an exclusive lock on the device, so how can you add a linear
> mapping to the origin device, if it is already used as a target in the
> snapshot device, and thus locked?

So, you have to put the linear mapping *before* that.
Then, the top-level block device will be locked but you can
still change the underlying mappings.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation of America

Re: Snapshots

[Phillip Susi]

Jun'ichi Nomura wrote:
> So, you have to put the linear mapping *before* that.
> Then, the top-level block device will be locked but you can
> still change the underlying mappings.

So using a disk device as a dm target locks it.  DM won't use a disk
device as a target if it is locked.  Are you saying that dm can use
another dm device as a target, even if it is already locked by being
used as another dm target?

Re: Snapshots

[Jun'ichi Nomura]

Phillip Susi wrote:
> Jun'ichi Nomura wrote:
>> So, you have to put the linear mapping *before* that.
>> Then, the top-level block device will be locked but you can
>> still change the underlying mappings.
> 
> So using a disk device as a dm target locks it.  DM won't use a disk
> device as a target if it is locked.

If a device is locked by a DM device on top of it,
other DM device can still use the device as a target.

Also, even if a DM device is locked by mount, you can still change
the table of the DM device.

> Are you saying that dm can use
> another dm device as a target, even if it is already locked by being
> used as another dm target?

I'm not sure if I understand your last sentence correctly.
Does the above comment answer to your question?

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation of America

snapshots

[Two Spirit]

few questions

1) Is the idea of snapshotting about 100TB of data doable with Ceph?

2) How long would such a task take? Are we talking seconds, minutes,
hours, days, weeks?

3) what does "stop i/o" mean. Does the filesystem unavailable to the
user during the snapshot time, or is this administratively unavailable
to the user, or does ceph stop io somewhere while in parallel the data
can be read/written to?

Re: snapshots

[Sage Weil]

On Fri, 15 Sep 2017, Two Spirit wrote:
> few questions

I'm assuming you're talking about CephFS snapshots here:

> 1) Is the idea of snapshotting about 100TB of data doable with Ceph?

You can snapshot the entire file system (petabytes) if you like.

> 2) How long would such a task take? Are we talking seconds, minutes,
> hours, days, weeks?

Seconds (if that).

> 3) what does "stop i/o" mean. Does the filesystem unavailable to the
> user during the snapshot time, or is this administratively unavailable
> to the user, or does ceph stop io somewhere while in parallel the data
> can be read/written to?

There's no IO stoppage.  Clients essentially mark a barrier in their 
writeback caches so that previously buffered writes are contained in the 
snapshot and new writes are not.  How long it takes for that data to be 
flushed and stable/durable on OSDs depends on how big your client caches 
are, but that does not cause any io stoppage or gap in availability for 
users of the file system.

sage

Re: snapshots

[Two Spirit]

Excellent. Yes  was takling about CephFS. So basically it tags and
versions all objects and their corresponding metadata. Doesn't
actually replicate anything, and keeps the old objects around in the
OSDs?

On Fri, Sep 15, 2017 at 10:37 AM, Sage Weil <sage@newdream.net> wrote:
> On Fri, 15 Sep 2017, Two Spirit wrote:
>> few questions
>
> I'm assuming you're talking about CephFS snapshots here:
>
>> 1) Is the idea of snapshotting about 100TB of data doable with Ceph?
>
> You can snapshot the entire file system (petabytes) if you like.
>
>> 2) How long would such a task take? Are we talking seconds, minutes,
>> hours, days, weeks?
>
> Seconds (if that).
>
>> 3) what does "stop i/o" mean. Does the filesystem unavailable to the
>> user during the snapshot time, or is this administratively unavailable
>> to the user, or does ceph stop io somewhere while in parallel the data
>> can be read/written to?
>
> There's no IO stoppage.  Clients essentially mark a barrier in their
> writeback caches so that previously buffered writes are contained in the
> snapshot and new writes are not.  How long it takes for that data to be
> flushed and stable/durable on OSDs depends on how big your client caches
> are, but that does not cause any io stoppage or gap in availability for
> users of the file system.
>
> sage

Re: snapshots

[Sage Weil]

On Fri, 15 Sep 2017, Two Spirit wrote:
> Excellent. Yes  was takling about CephFS. So basically it tags and
> versions all objects and their corresponding metadata. Doesn't
> actually replicate anything, and keeps the old objects around in the
> OSDs?

Something like that.  :)

s

> 
> On Fri, Sep 15, 2017 at 10:37 AM, Sage Weil <sage@newdream.net> wrote:
> > On Fri, 15 Sep 2017, Two Spirit wrote:
> >> few questions
> >
> > I'm assuming you're talking about CephFS snapshots here:
> >
> >> 1) Is the idea of snapshotting about 100TB of data doable with Ceph?
> >
> > You can snapshot the entire file system (petabytes) if you like.
> >
> >> 2) How long would such a task take? Are we talking seconds, minutes,
> >> hours, days, weeks?
> >
> > Seconds (if that).
> >
> >> 3) what does "stop i/o" mean. Does the filesystem unavailable to the
> >> user during the snapshot time, or is this administratively unavailable
> >> to the user, or does ceph stop io somewhere while in parallel the data
> >> can be read/written to?
> >
> > There's no IO stoppage.  Clients essentially mark a barrier in their
> > writeback caches so that previously buffered writes are contained in the
> > snapshot and new writes are not.  How long it takes for that data to be
> > flushed and stable/durable on OSDs depends on how big your client caches
> > are, but that does not cause any io stoppage or gap in availability for
> > users of the file system.
> >
> > sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: snapshots

[Eric Eastman]

Just watch how many snapshots you create.  We hit an issues around
4,700 snapshots of a singe file system.  See:
https://www.spinics.net/lists/ceph-devel/msg38203.html

Eric

On Fri, Sep 15, 2017 at 12:25 PM, Sage Weil <sage@newdream.net> wrote:
> On Fri, 15 Sep 2017, Two Spirit wrote:
>> Excellent. Yes  was takling about CephFS. So basically it tags and
>> versions all objects and their corresponding metadata. Doesn't
>> actually replicate anything, and keeps the old objects around in the
>> OSDs?
>
> Something like that.  :)
>
> s
>
>>
>> On Fri, Sep 15, 2017 at 10:37 AM, Sage Weil <sage@newdream.net> wrote:
>> > On Fri, 15 Sep 2017, Two Spirit wrote:
>> >> few questions
>> >
>> > I'm assuming you're talking about CephFS snapshots here:
>> >
>> >> 1) Is the idea of snapshotting about 100TB of data doable with Ceph?
>> >
>> > You can snapshot the entire file system (petabytes) if you like.
>> >
>> >> 2) How long would such a task take? Are we talking seconds, minutes,
>> >> hours, days, weeks?
>> >
>> > Seconds (if that).
>> >
>> >> 3) what does "stop i/o" mean. Does the filesystem unavailable to the
>> >> user during the snapshot time, or is this administratively unavailable
>> >> to the user, or does ceph stop io somewhere while in parallel the data
>> >> can be read/written to?
>> >
>> > There's no IO stoppage.  Clients essentially mark a barrier in their
>> > writeback caches so that previously buffered writes are contained in the
>> > snapshot and new writes are not.  How long it takes for that data to be
>> > flushed and stable/durable on OSDs depends on how big your client caches
>> > are, but that does not cause any io stoppage or gap in availability for
>> > users of the file system.
>> >
>> > sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: snapshots

[Wyllys Ingersoll]

Following up on this issue:

Does having too many cephfs snapshots prevent the deletion of older
ones? We've got about 4800 snapshots and attempted to purge some older
ones, but the "rmdir" of the old snapshots seems to hang and is not
ever completing.

ceph -s shows an ever increasing number of blocked requests while this
is going on. I waited over 10 minutes for 1 snapshot to delete before
I finally killed the "find" command that was attempting to remove the
old snapshots.  The blocked request count keeps going up anyway and
the .snap directory seems unresponsive to 'ls' or other commands.



On Fri, Sep 15, 2017 at 2:59 PM, Eric Eastman
<eric.eastman@keepertech.com> wrote:
> Just watch how many snapshots you create.  We hit an issues around
> 4,700 snapshots of a singe file system.  See:
> https://www.spinics.net/lists/ceph-devel/msg38203.html
>
> Eric
>
> On Fri, Sep 15, 2017 at 12:25 PM, Sage Weil <sage@newdream.net> wrote:
>> On Fri, 15 Sep 2017, Two Spirit wrote:
>>> Excellent. Yes  was takling about CephFS. So basically it tags and
>>> versions all objects and their corresponding metadata. Doesn't
>>> actually replicate anything, and keeps the old objects around in the
>>> OSDs?
>>
>> Something like that.  :)
>>
>> s
>>
>>>
>>> On Fri, Sep 15, 2017 at 10:37 AM, Sage Weil <sage@newdream.net> wrote:
>>> > On Fri, 15 Sep 2017, Two Spirit wrote:
>>> >> few questions
>>> >
>>> > I'm assuming you're talking about CephFS snapshots here:
>>> >
>>> >> 1) Is the idea of snapshotting about 100TB of data doable with Ceph?
>>> >
>>> > You can snapshot the entire file system (petabytes) if you like.
>>> >
>>> >> 2) How long would such a task take? Are we talking seconds, minutes,
>>> >> hours, days, weeks?
>>> >
>>> > Seconds (if that).
>>> >
>>> >> 3) what does "stop i/o" mean. Does the filesystem unavailable to the
>>> >> user during the snapshot time, or is this administratively unavailable
>>> >> to the user, or does ceph stop io somewhere while in parallel the data
>>> >> can be read/written to?
>>> >
>>> > There's no IO stoppage.  Clients essentially mark a barrier in their
>>> > writeback caches so that previously buffered writes are contained in the
>>> > snapshot and new writes are not.  How long it takes for that data to be
>>> > flushed and stable/durable on OSDs depends on how big your client caches
>>> > are, but that does not cause any io stoppage or gap in availability for
>>> > users of the file system.
>>> >
>>> > sage
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: snapshots

[Sage Weil]

On Fri, 15 Sep 2017, Wyllys Ingersoll wrote:
> Following up on this issue:
> 
> Does having too many cephfs snapshots prevent the deletion of older
> ones? We've got about 4800 snapshots and attempted to purge some older
> ones, but the "rmdir" of the old snapshots seems to hang and is not
> ever completing.
> 
> ceph -s shows an ever increasing number of blocked requests while this
> is going on. I waited over 10 minutes for 1 snapshot to delete before
> I finally killed the "find" command that was attempting to remove the
> old snapshots.  The blocked request count keeps going up anyway and
> the .snap directory seems unresponsive to 'ls' or other commands.

Sounds like a bug; it should be a quick operation.  What version?  Can you 
submit a bug report?  Preferably with a matching MDS log (and debug mds = 
20, debug ms = 1).

sage

> 
> 
> 
> On Fri, Sep 15, 2017 at 2:59 PM, Eric Eastman
> <eric.eastman@keepertech.com> wrote:
> > Just watch how many snapshots you create.  We hit an issues around
> > 4,700 snapshots of a singe file system.  See:
> > https://www.spinics.net/lists/ceph-devel/msg38203.html
> >
> > Eric
> >
> > On Fri, Sep 15, 2017 at 12:25 PM, Sage Weil <sage@newdream.net> wrote:
> >> On Fri, 15 Sep 2017, Two Spirit wrote:
> >>> Excellent. Yes  was takling about CephFS. So basically it tags and
> >>> versions all objects and their corresponding metadata. Doesn't
> >>> actually replicate anything, and keeps the old objects around in the
> >>> OSDs?
> >>
> >> Something like that.  :)
> >>
> >> s
> >>
> >>>
> >>> On Fri, Sep 15, 2017 at 10:37 AM, Sage Weil <sage@newdream.net> wrote:
> >>> > On Fri, 15 Sep 2017, Two Spirit wrote:
> >>> >> few questions
> >>> >
> >>> > I'm assuming you're talking about CephFS snapshots here:
> >>> >
> >>> >> 1) Is the idea of snapshotting about 100TB of data doable with Ceph?
> >>> >
> >>> > You can snapshot the entire file system (petabytes) if you like.
> >>> >
> >>> >> 2) How long would such a task take? Are we talking seconds, minutes,
> >>> >> hours, days, weeks?
> >>> >
> >>> > Seconds (if that).
> >>> >
> >>> >> 3) what does "stop i/o" mean. Does the filesystem unavailable to the
> >>> >> user during the snapshot time, or is this administratively unavailable
> >>> >> to the user, or does ceph stop io somewhere while in parallel the data
> >>> >> can be read/written to?
> >>> >
> >>> > There's no IO stoppage.  Clients essentially mark a barrier in their
> >>> > writeback caches so that previously buffered writes are contained in the
> >>> > snapshot and new writes are not.  How long it takes for that data to be
> >>> > flushed and stable/durable on OSDs depends on how big your client caches
> >>> > are, but that does not cause any io stoppage or gap in availability for
> >>> > users of the file system.
> >>> >
> >>> > sage
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>
> >>>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: snapshots

[Wyllys Ingersoll]

Its 10.2.7 and yes, I will file a bug soon.

On Fri, Sep 15, 2017 at 4:22 PM, Sage Weil <sage@newdream.net> wrote:
> On Fri, 15 Sep 2017, Wyllys Ingersoll wrote:
>> Following up on this issue:
>>
>> Does having too many cephfs snapshots prevent the deletion of older
>> ones? We've got about 4800 snapshots and attempted to purge some older
>> ones, but the "rmdir" of the old snapshots seems to hang and is not
>> ever completing.
>>
>> ceph -s shows an ever increasing number of blocked requests while this
>> is going on. I waited over 10 minutes for 1 snapshot to delete before
>> I finally killed the "find" command that was attempting to remove the
>> old snapshots.  The blocked request count keeps going up anyway and
>> the .snap directory seems unresponsive to 'ls' or other commands.
>
> Sounds like a bug; it should be a quick operation.  What version?  Can you
> submit a bug report?  Preferably with a matching MDS log (and debug mds =
> 20, debug ms = 1).
>
> sage
>
>>
>>
>>
>> On Fri, Sep 15, 2017 at 2:59 PM, Eric Eastman
>> <eric.eastman@keepertech.com> wrote:
>> > Just watch how many snapshots you create.  We hit an issues around
>> > 4,700 snapshots of a singe file system.  See:
>> > https://www.spinics.net/lists/ceph-devel/msg38203.html
>> >
>> > Eric
>> >
>> > On Fri, Sep 15, 2017 at 12:25 PM, Sage Weil <sage@newdream.net> wrote:
>> >> On Fri, 15 Sep 2017, Two Spirit wrote:
>> >>> Excellent. Yes  was takling about CephFS. So basically it tags and
>> >>> versions all objects and their corresponding metadata. Doesn't
>> >>> actually replicate anything, and keeps the old objects around in the
>> >>> OSDs?
>> >>
>> >> Something like that.  :)
>> >>
>> >> s
>> >>
>> >>>
>> >>> On Fri, Sep 15, 2017 at 10:37 AM, Sage Weil <sage@newdream.net> wrote:
>> >>> > On Fri, 15 Sep 2017, Two Spirit wrote:
>> >>> >> few questions
>> >>> >
>> >>> > I'm assuming you're talking about CephFS snapshots here:
>> >>> >
>> >>> >> 1) Is the idea of snapshotting about 100TB of data doable with Ceph?
>> >>> >
>> >>> > You can snapshot the entire file system (petabytes) if you like.
>> >>> >
>> >>> >> 2) How long would such a task take? Are we talking seconds, minutes,
>> >>> >> hours, days, weeks?
>> >>> >
>> >>> > Seconds (if that).
>> >>> >
>> >>> >> 3) what does "stop i/o" mean. Does the filesystem unavailable to the
>> >>> >> user during the snapshot time, or is this administratively unavailable
>> >>> >> to the user, or does ceph stop io somewhere while in parallel the data
>> >>> >> can be read/written to?
>> >>> >
>> >>> > There's no IO stoppage.  Clients essentially mark a barrier in their
>> >>> > writeback caches so that previously buffered writes are contained in the
>> >>> > snapshot and new writes are not.  How long it takes for that data to be
>> >>> > flushed and stable/durable on OSDs depends on how big your client caches
>> >>> > are, but that does not cause any io stoppage or gap in availability for
>> >>> > users of the file system.
>> >>> >
>> >>> > sage
>> >>> --
>> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >>> the body of a message to majordomo@vger.kernel.org
>> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>>
>> >>>
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>