searching checkpoints contaning an inode (or a filename)

searching checkpoints contaning an inode (or a filename)

[Taesoo Kim]

Hi nilfs,

I am evaluating nilfs for using it in my machine. There are a few
questions regarding usages:

q1. how can I enumerate a list of cps, given a filename?
 (I know I can mount each cp to check whether there is my file or
 not, as time-machine (in mac) does)

q2. compute diff or cat against previous files in a specific cp?
 (instead of mounting and copying them individually to my /tmp)

q3. using nilfs as rootfs?
 (I couldn't find any people using nilfs for rootfs. Are there any
 issue on that? or just matter of performance? on updates for
 example)

Nilfs looks conceptually nice and clean, so I can trust the
implementation or jump directly into modifying code. Do you have any
design notes (other than one in /Documentation and overview-nilfs1.pdf).

Thank you in advance.
Taesoo
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: searching checkpoints contaning an inode (or a filename)

[Jérôme Poulin]

On Fri, Jul 5, 2013 at 4:17 PM, Taesoo Kim <taesoo-3s7WtUTddSA@public.gmane.org> wrote:
>
> q3. using nilfs as rootfs?
>  (I couldn't find any people using nilfs for rootfs. Are there any
>  issue on that? or just matter of performance? on updates for
>  example)

I'm using NILFS as root filesystem on my work laptop for a total of 4
years. I switched to BTRFS for a year in between but looking at how
fast it kills SSD, it wass not worthwhile.

Stability wise, I only had 2 problems with NILFS, the first, which was
fixed promptly, caused a total data lost because of how the cleaner
scrub the disk (using the time), it scrubbed data from the future
after time ran backward. Then the other problem was a kernel OOPS
about a file from the debian APT cache, I deleted it and it runs just
fine now, I couldn't take the backtrace.

Performance wise, on SSD the filesystem is really fast and snappy,
while on HDD, reads are really slow, booting takes 2 minutes instead
of less than 10 seconds, however, once booted and everything is in
cache, it is quite fast. Database loads, especially reads on HDD is
the worst for NILFS.
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: searching checkpoints contaning an inode (or a filename)

[Vyacheslav Dubeyko]

Hi Jérôme,

On Jul 6, 2013, at 1:48 AM, Jérôme Poulin wrote:

> On Fri, Jul 5, 2013 at 4:17 PM, Taesoo Kim <taesoo-3s7WtUTddSA@public.gmane.org> wrote:
>> 
>> q3. using nilfs as rootfs?
>> (I couldn't find any people using nilfs for rootfs. Are there any
>> issue on that? or just matter of performance? on updates for
>> example)
> 
> I'm using NILFS as root filesystem on my work laptop for a total of 4
> years. I switched to BTRFS for a year in between but looking at how
> fast it kills SSD, it wass not worthwhile.
> 
> Stability wise, I only had 2 problems with NILFS, the first, which was
> fixed promptly, caused a total data lost because of how the cleaner
> scrub the disk (using the time), it scrubbed data from the future
> after time ran backward. Then the other problem was a kernel OOPS
> about a file from the debian APT cache, I deleted it and it runs just
> fine now, I couldn't take the backtrace.
> 

Could you share more details about the issue? I need to understand
the reproducing path of the issue more clear. Can you reproduce it again? 

Thanks,
Vyacheslav Dubeyko.

> Performance wise, on SSD the filesystem is really fast and snappy,
> while on HDD, reads are really slow, booting takes 2 minutes instead
> of less than 10 seconds, however, once booted and everything is in
> cache, it is quite fast. Database loads, especially reads on HDD is
> the worst for NILFS.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: searching checkpoints contaning an inode (or a filename)

[Vyacheslav Dubeyko]

Hi Jérôme,

On Jul 6, 2013, at 1:48 AM, Jérôme Poulin wrote:

> On Fri, Jul 5, 2013 at 4:17 PM, Taesoo Kim <taesoo-3s7WtUTddSA@public.gmane.org> wrote:
>> 
>> q3. using nilfs as rootfs?
>> (I couldn't find any people using nilfs for rootfs. Are there any
>> issue on that? or just matter of performance? on updates for
>> example)
> 
> I'm using NILFS as root filesystem on my work laptop for a total of 4
> years. I switched to BTRFS for a year in between but looking at how
> fast it kills SSD, it wass not worthwhile.
> 
> Stability wise, I only had 2 problems with NILFS, the first, which was
> fixed promptly, caused a total data lost because of how the cleaner
> scrub the disk (using the time), it scrubbed data from the future
> after time ran backward. Then the other problem was a kernel OOPS
> about a file from the debian APT cache, I deleted it and it runs just
> fine now, I couldn't take the backtrace.
> 

Could you share more details about the issue? I need to understand
the reproducing path of the issue more clear. Can you reproduce it again? 

Thanks,
Vyacheslav Dubeyko.

> Performance wise, on SSD the filesystem is really fast and snappy,
> while on HDD, reads are really slow, booting takes 2 minutes instead
> of less than 10 seconds, however, once booted and everything is in
> cache, it is quite fast. Database loads, especially reads on HDD is
> the worst for NILFS.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: searching checkpoints contaning an inode (or a filename)

[Vyacheslav Dubeyko]

Hi Taesoo,

On Jul 6, 2013, at 12:17 AM, Taesoo Kim wrote:

> Hi nilfs,
> 
> I am evaluating nilfs for using it in my machine. There are a few
> questions regarding usages:
> 
> q1. how can I enumerate a list of cps, given a filename?
> (I know I can mount each cp to check whether there is my file or
> not, as time-machine (in mac) does)
> 

You can use dumpseg fot this goal. To be honest this way not so
simple. You can get inode id from ls output ("ls -i"), for example.
And then you need to get dumpseg for all segments and select
output for searching inode. Of course, it needs to implement
special feature for easy retrieving such data. 

> q2. compute diff or cat against previous files in a specific cp?
> (instead of mounting and copying them individually to my /tmp)
> 

As I understand, currently, we haven't this as feature of any
NILFS2's tool or internal file system technique.

> q3. using nilfs as rootfs?
> (I couldn't find any people using nilfs for rootfs. Are there any
> issue on that? or just matter of performance? on updates for
> example)
> 

Many people are using NILFS2 as rootfs file system. Currently,
we have several reports about serious bugs in NILFS2 from
such people. I am investigating these issues.

> Nilfs looks conceptually nice and clean, so I can trust the
> implementation or jump directly into modifying code. Do you have any
> design notes (other than one in /Documentation and overview-nilfs1.pdf).
> 

I can share my version of design document. But I don't end it and this
document contains some mistakes in description.

http://dubeyko.com/development/FileSystems/NILFS/nilfs2-design.pdf.

So, as I understand, you are going to make some implementation in
NILFS2. Could you share what feature you want to implement?
Maybe, it make sense to implement above-mentioned by you features.
Anyway, it makes sense to distribute tasks.

PS: I am on vacation now. So, sometimes I haven't access to my e-mail.

Thanks,
Vyacheslav Dubeyko.

> Thank you in advance.
> Taesoo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: searching checkpoints contaning an inode (or a filename)

[Ryusuke Konishi]

On Sat, 6 Jul 2013 11:34:32 +0400, Vyacheslav Dubeyko wrote:
> Hi Taesoo,
> 
> On Jul 6, 2013, at 12:17 AM, Taesoo Kim wrote:
> 
>> Hi nilfs,
>> 
>> I am evaluating nilfs for using it in my machine. There are a few
>> questions regarding usages:
>> 
>> q1. how can I enumerate a list of cps, given a filename?
>> (I know I can mount each cp to check whether there is my file or
>> not, as time-machine (in mac) does)
>> 

I think it's hard to implement this feature efficiently because nilfs
doesn't have an efficient lookup method of past inodes from a file
name.

> You can use dumpseg fot this goal. To be honest this way not so
> simple. You can get inode id from ls output ("ls -i"), for example.
> And then you need to get dumpseg for all segments and select
> output for searching inode. Of course, it needs to implement
> special feature for easy retrieving such data. 

dumpseg is a debug tool and it doesn't provide a solid way since it
directly scans on-disk blocks from userland.

We should adds an ioctl for that if there is a reasonable purpose, but
I don't know it can be implemented nicely.

>> q2. compute diff or cat against previous files in a specific cp?
>> (instead of mounting and copying them individually to my /tmp)
>> 
> 
> As I understand, currently, we haven't this as feature of any
> NILFS2's tool or internal file system technique.

An experimental patchset and a tool exist:

http://www.mail-archive.com/linux-nilfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg00827.html

nilfs-diff:

   git://github.com/konis/nilfs-utils.git (diff branch)

diff API:

   git://github.com/konis/nilfs2.git (diffapi branch)


But I have never sent this to the mainline because the patchset has
important limitations for practical use in addition to needing a disk
format change.


Regards,
Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: searching checkpoints contaning an inode (or a filename)

[Taesoo Kim]

Hi Ryusuke and Vyacheslav,

> >> q1. how can I enumerate a list of cps, given a filename?
> >> (I know I can mount each cp to check whether there is my file or
> >> not, as time-machine (in mac) does)
> >> 
> 
> I think it's hard to implement this feature efficiently because nilfs
> doesn't have an efficient lookup method of past inodes from a file
> name.
> 
> > You can use dumpseg fot this goal. To be honest this way not so
> > simple. You can get inode id from ls output ("ls -i"), for example.
> > And then you need to get dumpseg for all segments and select
> > output for searching inode. Of course, it needs to implement
> > special feature for easy retrieving such data. 
> 
> dumpseg is a debug tool and it doesn't provide a solid way since it
> directly scans on-disk blocks from userland.
> 
> We should adds an ioctl for that if there is a reasonable purpose, but
> I don't know it can be implemented nicely.
> 

I don't know how difficult to embed a backward link (to previous cp)
in each inode? So given a current inode, I can walk inode-btree of cp
again to find the previous one, possibly with ioctl().

But, searching inodes of a deleted path seems still difficult and 
hard to implement nicely, without walking all cps to figure out.

> >> q2. compute diff or cat against previous files in a specific cp?
> >> (instead of mounting and copying them individually to my /tmp)
> >> 
> > 
> > As I understand, currently, we haven't this as feature of any
> > NILFS2's tool or internal file system technique.
> 
> An experimental patchset and a tool exist:
> 
> http://www.mail-archive.com/linux-nilfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg00827.html
> 
> nilfs-diff:
> 
>    git://github.com/konis/nilfs-utils.git (diff branch)
> 
> diff API:
> 
>    git://github.com/konis/nilfs2.git (diffapi branch)
> 
> 
> But I have never sent this to the mainline because the patchset has
> important limitations for practical use in addition to needing a disk
> format change.
> 

I will definitely check the patch set, and thank you for the design
doc as well.

Thank you!
Taesoo

> 
> Regards,
> Ryusuke Konishi
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: searching checkpoints contaning an inode (or a filename)

[Vyacheslav Dubeyko]

Hi Taesoo,

On Jul 6, 2013, at 12:17 AM, Taesoo Kim wrote:

> Hi nilfs,
> 
> I am evaluating nilfs for using it in my machine. There are a few
> questions regarding usages:
> 
> q1. how can I enumerate a list of cps, given a filename?
> (I know I can mount each cp to check whether there is my file or
> not, as time-machine (in mac) does)
> 

You can use dumpseg fot this goal. To be honest this way not so
simple. You can get inode id from ls output ("ls -i"), for example.
And then you need to get dumpseg for all segments and select
output for searching inode. Of course, it needs to implement
special feature for easy retrieving such data. 

> q2. compute diff or cat against previous files in a specific cp?
> (instead of mounting and copying them individually to my /tmp)
> 

As I understand, currently, we haven't this as feature of any
NILFS2's tool or internal file system technique.

> q3. using nilfs as rootfs?
> (I couldn't find any people using nilfs for rootfs. Are there any
> issue on that? or just matter of performance? on updates for
> example)
> 

Many people are using NILFS2 as rootfs file system. Currently,
we have several reports about serious bugs in NILFS2 from
such people. I am investigating these issues.

> Nilfs looks conceptually nice and clean, so I can trust the
> implementation or jump directly into modifying code. Do you have any
> design notes (other than one in /Documentation and overview-nilfs1.pdf).
> 

I can share my version of design document. But I don't end it and this
document contains some mistakes in description.

http://dubeyko.com/development/FileSystems/NILFS/nilfs2-design.pdf.

So, as I understand, you are going to make some implementation in
NILFS2. Could you share what feature you want to implement?
Maybe, it make sense to implement above-mentioned by you features.
Anyway, it makes sense to distribute tasks.

PS: I am on vacation now. So, sometimes I haven't access to my e-mail.

Thanks,
Vyacheslav Dubeyko.

> Thank you in advance.
> Taesoo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html