get filename->inode mappings in bulk for a live fs?

get filename->inode mappings in bulk for a live fs?

[Linda Walsh]

I notice that attempts to use utils to get name->inode mappings
(xfs_ncheck) seem to have no option to operate on a mounted filesystem.

Is it practical to xfs_freeze such a file system and then ncheck it? or would
freezing it simply freeze the fact that it is open and provide no benefit?

So how can I get an inode->block mapping on a live fs.

I'm not worried about some smallish number of cases that might be inaccurate.

Out of ~5M files on my most populous volume, having even 1000 files w/wrong
info would be less than .02% -- which would be bad if I wanted an exact backup,
but for purposes a quick-fuzzy look at files that have changed and are finished
being changed (vs. the ones being changed right now),

The man page for xfs_freeze mentions using it to make snapshots -- how long
does such a snapshot usually take?  I.e. how long would a file system be frozen?
Is it something that would take a few ms, few seconds, or multiple minutes?

This may be a weird idea, but I seem to remember when lvm takes a snapshot
it moves the live volume aside and begins to use COW segments to hold changes.

Is it possible to xfs-freeze a COW copy so access to the original FS isn't
suspended thus making the time period of an xfs_freeze/dump_names less
critical?

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: get filename->inode mappings in bulk for a live fs?

[Greg Freemyer]

On Thu, Oct 4, 2012 at 11:30 AM, Linda Walsh <xfs@tlinx.org> wrote:
> I notice that attempts to use utils to get name->inode mappings
> (xfs_ncheck) seem to have no option to operate on a mounted filesystem.
>
> Is it practical to xfs_freeze such a file system and then ncheck it? or
> would
> freezing it simply freeze the fact that it is open and provide no benefit?
>
> So how can I get an inode->block mapping on a live fs.
>
> I'm not worried about some smallish number of cases that might be
> inaccurate.
>
> Out of ~5M files on my most populous volume, having even 1000 files w/wrong
> info would be less than .02% -- which would be bad if I wanted an exact
> backup,
> but for purposes a quick-fuzzy look at files that have changed and are
> finished
> being changed (vs. the ones being changed right now),
>
> The man page for xfs_freeze mentions using it to make snapshots -- how long
> does such a snapshot usually take?  I.e. how long would a file system be
> frozen?
> Is it something that would take a few ms, few seconds, or multiple minutes?
>
> This may be a weird idea, but I seem to remember when lvm takes a snapshot
> it moves the live volume aside and begins to use COW segments to hold
> changes.
>
> Is it possible to xfs-freeze a COW copy so access to the original FS isn't
> suspended thus making the time period of an xfs_freeze/dump_names less
> critical?

I think you're missing some of the basics.

Conceptually it is typically:

- quiesce system
- make snapshot
- release system (to accept future filesystem changes)
- use snapshot as a source of point-in-time data (often for backups)

Since this solution is designed for Enterprise nightly backup use the
various aspects are designed to have minimal impact on the user.  When
Oracle as an example is quiesced it maintains a RAM based transaction
log that it replays after it is released to talk to the filesystem
again.

So quiesce system is typically:
   - quiesce apps (many enterprise apps provide a API call for this)
   - quiesce filesystem (xfs_freeze can do this, but I'm pretty sure
the kernel freeze is also automatically called by lvm's snapshot
function.)

make snapshot:
     - If you're using LVM, you can do this with it,
     - or many raid arrays have API's to do this at your command
     - or some filesystems (not xfs) have snapshot functionality built-in

release system:
     - if you called xfs_freeze, you need to call it again to release
the file i/o to occur again.
     - Release any apps you quiesced (again enterprise apps may have
an API for this).

Now you can take as long as you want to dump the names from the snapshot.

At least for me none of the above takes longer than a few seconds
each.  If you have lots of data-in-flight (or write-cache) I could
imagine the freeze taking longer since it has to ensure that all
filesystem caches and buffers have been fully written to the lower
level.

FYI: I have seen ext4 maintainer Ted T'so recommend that with ext4 you
occasionally make a snapshot like the above and then do fsck on the
snapshot to see if the snapshot filesystem structure is valid.  If
errors are detect, then have the cron script send a e-mail to the
admin so he/she can schedule downtime to run fsck on the main
filesystem.  I don't know if xfs maintainers have a similar
recommendation for xfs.

FYI2: I think opensuse's snapper app has a framework to support a lot
of the above built in, but I'm not sure it supports anything but btrfs
snapshots.

Greg

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: get filename->inode mappings in bulk for a live fs?

[Linda Walsh]

Greg Freemyer wrote:
> > Is it possible to xfs-freeze a COW copy so access to the original FS isn't
> > suspended thus making the time period of an xfs_freeze/dump_names less
> > critical?
>
> I think you're missing some of the basics.
>
> Conceptually it is typically:
>
> - quiesce system
> - make snapshot
> - release system (to accept future filesystem changes)
> - use snapshot as a source of point-in-time data (often for backups)
>
> Since this solution is designed for Enterprise nightly backup use the
> various aspects are designed to have minimal impact on the user.  When
> Oracle as an example is quiesced it maintains a RAM based transaction
> log that it replays after it is released to talk to the filesystem
> again.
>
> So quiesce system is typically:
>    - quiesce apps (many enterprise apps provide a API call for this)
----
    Can you name 1 on linux that does?

    I can't think of any -- sendmail, imap, samba, squid, syslog...


>    - quiesce filesystem (xfs_freeze can do this, but I'm pretty sure
> the kernel freeze is also automatically called by lvm's snapshot
> function.)
----
    Yeah... I understand this part -- in a few emails that follow the one you 
are responding
to, I showed where I tried exactly this... and either I 'm missing something (which
wouldn't be hard, given my vast knowledge (*cough*) in this area), or it's not 
working.
I hope it is the former, as my own ignorance is usually easier to address than 
the latter.

>
> make snapshot:
>      - If you're using LVM, you can do this with it,
>      - or many raid arrays have API's to do this at your command
>      - or some filesystems (not xfs) have snapshot functionality built-in
----
    The *purpose* of an lvm snapshot -- supposedly, is to create a stable (as in
it's not going to be changing underneath you) file system to backup from.  It 
doesn't
guarantee that the filesystem is in a consistent state -- but it will prevent future
modifications....)...

    It sounds like -- I'm using the wrong 'order' ...

    It sounds more like I need to xfs_freeze the live volume, 'home'
create a snapshot of that (lvm) - which hopefully won't try to write to the 
xfs-frozen
volume, -- once that is done, xfs-unfreeze....

Irck....

>
> release system:
>      - if you called xfs_freeze, you need to call it again to release
> the file i/o to occur again.
>      - Release any apps you quiesced (again enterprise apps may have
> an API for this).
>
> FYI: I have seen ext4 maintainer Ted T'so recommend that with ext4 you
> occasionally make a snapshot like the above and then do fsck on the
> snapshot to see if the snapshot filesystem structure is valid.  If
> errors are detect, then have the cron script send a e-mail to the
> admin so he/she can schedule downtime to run fsck on the main
> filesystem.  I don't know if xfs maintainers have a similar
> recommendation for xfs.
----
    xfs doesn't really have an fsck -- one was added to dot some
'i' or cross some 't', but it's more of a bother than anything, as
it just checks to see if the file system is there.  If it is, pass,
else, fail.  I don't need mkfs to cause my system boot to fail
due to a missing disk -- as the system "will or won't" come up
in some form, without that disk, regardless. 

    In the past, in rare instances where something was offline
(just messed w/disks or deliberately have a disk offline and
want to boot), a pure xfs system would just "work" (with
the assumption that the offline disk wasn't critical in order
to boot).  Then I could log-in and continue (or do) whatever
maintenance was necessary to restore normalcy.  But with
the new scripts, the system is guaranteed to fail and not
come up except to the console.  Certainly not my idea of
a robust system.

    But for some perfection is more important than robust.

I usually try change xfs.fsck into a link to /bin/true after
updates...
>
> FYI2: I think opensuse's snapper app has a framework to support a lot
> of the above built in, but I'm not sure it supports anything but btrfs
> snapshots.
---
    You are correct -- I was surprised to see them call an app 'snapper'
when it doesn't even support lvm snapshots, but only the non-stable
and non-production-ready 'btrfs'.



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: get filename->inode mappings in bulk for a live fs?

[Greg Freemyer]

On Thu, Oct 4, 2012 at 2:29 PM, Linda Walsh <xfs@tlinx.org> wrote:
>> So quiesce system is typically:
>>    - quiesce apps (many enterprise apps provide a API call for this)
>
> ----
>    Can you name 1 on linux that does?
>
>    I can't think of any -- sendmail, imap, samba, squid, syslog...


Commercial - Oracle, DB2 (from IBM)

opensource -
   sendmail has a quiesce signal handler, but I've never used it

        see sendmail.h
        extern SIGFUNC_DECL	quiesce __P((int));

   Postgres and mysql fake it I think,  You "quiesce" them by having
them do a dump of the database, then the backup picks up the dump.

The key thing is knowing the right word to google for, and the right
word is quiesce.

Greg

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

xfs_freeze same as umount? How is that helpful?

[Linda Walsh]

Greg Freemyer wrote:
> Conceptually it is typically:
> - quiesce system
----
	Um... it seems that this is equivalent to being
able to umount the disk?

When I tried xfs_freeze / fs_freeze got fs-busy -- same as I would
if I tried to umount it.

I thought the point of xfs_freeze was to allow it to be brought to
a consistent state without unmounting it?

Coincidentally, after trying a few freezes, the system froze.

Last message in log was 'fetchmail' (that might have been writing a
log to /home), had hung for 120 seconds...

That was all she wrote before the entire system 'froze'...


So shouldn't xfs_freeze work without having to to shut down every
process on it?  If not, then how is it different than umount?





_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: get filename->inode mappings in bulk for a live fs?

[Dave Chinner]

On Thu, Oct 04, 2012 at 08:30:20AM -0700, Linda Walsh wrote:
> I notice that attempts to use utils to get name->inode mappings
> (xfs_ncheck) seem to have no option to operate on a mounted filesystem.

If you need xfs_ncheck to get name->inode number mappings, then you
are doing something wrong.

$ find . -printf  "%h/%f: %i\n" | head -10
./.: 1010542779
./.git: 612759677
./.git/branches: 1013256262
./.git/hooks: 33523782
./.git/hooks/guilt: 406321163
./.git/hooks/guilt/delete: 402681911

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_freeze same as umount? How is that helpful?

[Dave Chinner]

On Thu, Oct 04, 2012 at 03:39:33PM -0700, Linda Walsh wrote:
> Greg Freemyer wrote:
> >Conceptually it is typically:
> >- quiesce system
> ----
> 	Um... it seems that this is equivalent to being
> able to umount the disk?

NO, it's not. freeze intentionally leaves the log dirty, whereas
unmount leaves it clean.

> When I tried xfs_freeze / fs_freeze got fs-busy -- same as I would
> if I tried to umount it.

Of course - it's got to write all the dirty data andmetadata in
memory to disk. Freeze is about providing a stable, consistent disk
image of the filesystem, so it must flush dirty objects from memory
to disk to provide that.

> I thought the point of xfs_freeze was to allow it to be brought to
> a consistent state without unmounting it?

Exactly.

> Coincidentally, after trying a few freezes, the system froze.

Entirely possible if you froze the root filesystem and something you
rely on tried to write to the filesystem.

Anyway, a one-line "it froze" report doesn't tell us anything about
the problem you saw. So:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_freeze same as umount? How is that helpful?

[Linda Walsh]

Dave Chinner wrote:
> On Thu, Oct 04, 2012 at 03:39:33PM -0700, Linda Walsh wrote:
>> Greg Freemyer wrote:
>>> Conceptually it is typically:
>>> - quiesce system
>> ----
>> 	Um... it seems that this is equivalent to being
>> able to umount the disk?
> 
> NO, it's not. freeze intentionally leaves the log dirty, whereas
> unmount leaves it clean.
----
	That's what I thought!


> 
>> When I tried xfs_freeze / fs_freeze got fs-busy -- same as I would
>> if I tried to umount it.
> 
> Of course - it's got to write all the dirty data andmetadata in
> memory to disk. Freeze is about providing a stable, consistent disk
> image of the filesystem, so it must flush dirty objects from memory
> to disk to provide that.
----
	But it says the freeze failed.... huh.

Just tried it again .. ( first time after reboot.. froze it no
messages or complaints) ?!?!  I don't get it.


It gave me a file system busy message before and as near as I could tell --
it wouldn't allow me to xfs_freeze it.

Trying the same thing now -- no prob.

(though I am ALSO on a newer kernel -- had another problem I solved
in looking through the logs for hints about the freeze.


> 
>> I thought the point of xfs_freeze was to allow it to be brought to
>> a consistent state without unmounting it?
> 
> Exactly.
> 
>> Coincidentally, after trying a few freezes, the system froze.
> 
> Entirely possible if you froze the root filesystem and something you
> rely on tried to write to the filesystem.
---
	Nep... "/home", and running as root, in root partition with
/home elements removed from PATH.   trying to be careful.  Notice
I did say   'Coincidentally' -- (w no/quotes in original).  If I was
thought there might be a connection or problem, at the very least I would
have put 'coincidentally' in quotes.. :-)..


> 
> Anyway, a one-line "it froze" report doesn't tell us anything about
> the problem you saw. So:
----
	Wasn't sure what I saw or that it was related -- exactly...

A possible theory... but nothing I'd blaim on xfs,  -- last message in log was:


Oct  4 13:52:50 Ishtar kernel: [985735.911825] INFO: task fetchmail:25872 blocke
d for more than 120 seconds.
Oct  4 13:52:50 Ishtar kernel: [985735.918777] "echo 0 > /proc/sys/kernel/hung_t
ask_timeout_secs" disables this message.


My kernel *was* setup to panic on a hung task.... instead it just froze...
but why fetchmail hung... well if the xfs_freeze "partly took" and just issued
the error "because", then that process might have froze trying to write log messages
to /home partition.... but 120 secs?  seems like it might have been going down 
before
I tried anything with xfs_freeze...


But not sure what to report now, as it's not doing the same things.

Was running 3.2.29, am running 3.5.4 now.... but 3.2.29 had been up for over 10
days... so maybe something else was going on there...

Sorry, when I said corrupt... mis-statement on my part... dirty was what
I meant -- but corrupt from the standpoint that the data lacked sufficient integrity
for a blockget type operation.  But dirty would be more accurate in FS-lingo.

(still had my data-integrity hat on)...

So if I xfs-freeze something, then take a snapshot, -- I don't see that any of that
would help in doing an xfs_blockget o get a dump of inodes->blocks, as it sounds
like it would still be dirty...

Hey, I think xfs walks on water, so don't think I'm complaining...just
trying to figure things out.   It's been a good fs for me for over 10 years on
my home systems.


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: xfs_freeze same as umount? How is that helpful?

[Dave Chinner]

On Thu, Oct 04, 2012 at 05:10:00PM -0700, Linda Walsh wrote:
> Dave Chinner wrote:
> >On Thu, Oct 04, 2012 at 03:39:33PM -0700, Linda Walsh wrote:
> >>When I tried xfs_freeze / fs_freeze got fs-busy -- same as I would
> >>if I tried to umount it.
> >
> >Of course - it's got to write all the dirty data andmetadata in
> >memory to disk. Freeze is about providing a stable, consistent disk
> >image of the filesystem, so it must flush dirty objects from memory
> >to disk to provide that.
> ----
> 	But it says the freeze failed.... huh.
> 
> Just tried it again .. ( first time after reboot.. froze it no
> messages or complaints) ?!?!  I don't get it.
> 
> 
> It gave me a file system busy message before and as near as I could tell --
> it wouldn't allow me to xfs_freeze it.

You mean EBUSY?

That means the filesystem was already frozen.

> >the problem you saw. So:
> ----
> 	Wasn't sure what I saw or that it was related -- exactly...
> 
> A possible theory... but nothing I'd blaim on xfs,  -- last message in log was:
> 
> 
> Oct  4 13:52:50 Ishtar kernel: [985735.911825] INFO: task fetchmail:25872 blocke
> d for more than 120 seconds.
> Oct  4 13:52:50 Ishtar kernel: [985735.918777] "echo 0 > /proc/sys/kernel/hung_t
> ask_timeout_secs" disables this message.

And the full stack trace?

> So if I xfs-freeze something, then take a snapshot, -- I don't see that any of that
> would help in doing an xfs_blockget o get a dump of inodes->blocks, as it sounds
> like it would still be dirty...

Only the log is dirty - it has a single dummy record in it to mark
it dirty. Otherwise everything is consistent, and the tools will
work fine on the snapshot image. You just have to know the magic
incantations to run the xfs_db commands directly with the extra flag
to tel it to ignore a dirty log.

i.e. xfs_ncheck is a diagnostic tool, not something that should be
run as part of a regular "get some info from the fs" workflow...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs