* get filename->inode mappings in bulk for a live fs? @ 2012-10-04 15:30 Linda Walsh 2012-10-04 18:01 ` Greg Freemyer 2012-10-04 22:49 ` get filename->inode mappings in bulk for a live fs? Dave Chinner 0 siblings, 2 replies; 9+ messages in thread From: Linda Walsh @ 2012-10-04 15:30 UTC (permalink / raw) To: xfs-oss I notice that attempts to use utils to get name->inode mappings (xfs_ncheck) seem to have no option to operate on a mounted filesystem. Is it practical to xfs_freeze such a file system and then ncheck it? or would freezing it simply freeze the fact that it is open and provide no benefit? So how can I get an inode->block mapping on a live fs. I'm not worried about some smallish number of cases that might be inaccurate. Out of ~5M files on my most populous volume, having even 1000 files w/wrong info would be less than .02% -- which would be bad if I wanted an exact backup, but for purposes a quick-fuzzy look at files that have changed and are finished being changed (vs. the ones being changed right now), The man page for xfs_freeze mentions using it to make snapshots -- how long does such a snapshot usually take? I.e. how long would a file system be frozen? Is it something that would take a few ms, few seconds, or multiple minutes? This may be a weird idea, but I seem to remember when lvm takes a snapshot it moves the live volume aside and begins to use COW segments to hold changes. Is it possible to xfs-freeze a COW copy so access to the original FS isn't suspended thus making the time period of an xfs_freeze/dump_names less critical? _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: get filename->inode mappings in bulk for a live fs? 2012-10-04 15:30 get filename->inode mappings in bulk for a live fs? Linda Walsh @ 2012-10-04 18:01 ` Greg Freemyer 2012-10-04 18:29 ` Linda Walsh 2012-10-04 22:39 ` xfs_freeze same as umount? How is that helpful? Linda Walsh 2012-10-04 22:49 ` get filename->inode mappings in bulk for a live fs? Dave Chinner 1 sibling, 2 replies; 9+ messages in thread From: Greg Freemyer @ 2012-10-04 18:01 UTC (permalink / raw) To: Linda Walsh; +Cc: xfs-oss On Thu, Oct 4, 2012 at 11:30 AM, Linda Walsh <xfs@tlinx.org> wrote: > I notice that attempts to use utils to get name->inode mappings > (xfs_ncheck) seem to have no option to operate on a mounted filesystem. > > Is it practical to xfs_freeze such a file system and then ncheck it? or > would > freezing it simply freeze the fact that it is open and provide no benefit? > > So how can I get an inode->block mapping on a live fs. > > I'm not worried about some smallish number of cases that might be > inaccurate. > > Out of ~5M files on my most populous volume, having even 1000 files w/wrong > info would be less than .02% -- which would be bad if I wanted an exact > backup, > but for purposes a quick-fuzzy look at files that have changed and are > finished > being changed (vs. the ones being changed right now), > > The man page for xfs_freeze mentions using it to make snapshots -- how long > does such a snapshot usually take? I.e. how long would a file system be > frozen? > Is it something that would take a few ms, few seconds, or multiple minutes? > > This may be a weird idea, but I seem to remember when lvm takes a snapshot > it moves the live volume aside and begins to use COW segments to hold > changes. > > Is it possible to xfs-freeze a COW copy so access to the original FS isn't > suspended thus making the time period of an xfs_freeze/dump_names less > critical? I think you're missing some of the basics. Conceptually it is typically: - quiesce system - make snapshot - release system (to accept future filesystem changes) - use snapshot as a source of point-in-time data (often for backups) Since this solution is designed for Enterprise nightly backup use the various aspects are designed to have minimal impact on the user. When Oracle as an example is quiesced it maintains a RAM based transaction log that it replays after it is released to talk to the filesystem again. So quiesce system is typically: - quiesce apps (many enterprise apps provide a API call for this) - quiesce filesystem (xfs_freeze can do this, but I'm pretty sure the kernel freeze is also automatically called by lvm's snapshot function.) make snapshot: - If you're using LVM, you can do this with it, - or many raid arrays have API's to do this at your command - or some filesystems (not xfs) have snapshot functionality built-in release system: - if you called xfs_freeze, you need to call it again to release the file i/o to occur again. - Release any apps you quiesced (again enterprise apps may have an API for this). Now you can take as long as you want to dump the names from the snapshot. At least for me none of the above takes longer than a few seconds each. If you have lots of data-in-flight (or write-cache) I could imagine the freeze taking longer since it has to ensure that all filesystem caches and buffers have been fully written to the lower level. FYI: I have seen ext4 maintainer Ted T'so recommend that with ext4 you occasionally make a snapshot like the above and then do fsck on the snapshot to see if the snapshot filesystem structure is valid. If errors are detect, then have the cron script send a e-mail to the admin so he/she can schedule downtime to run fsck on the main filesystem. I don't know if xfs maintainers have a similar recommendation for xfs. FYI2: I think opensuse's snapper app has a framework to support a lot of the above built in, but I'm not sure it supports anything but btrfs snapshots. Greg _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: get filename->inode mappings in bulk for a live fs? 2012-10-04 18:01 ` Greg Freemyer @ 2012-10-04 18:29 ` Linda Walsh 2012-10-04 18:59 ` Greg Freemyer 2012-10-04 22:39 ` xfs_freeze same as umount? How is that helpful? Linda Walsh 1 sibling, 1 reply; 9+ messages in thread From: Linda Walsh @ 2012-10-04 18:29 UTC (permalink / raw) To: Greg Freemyer; +Cc: xfs-oss Greg Freemyer wrote: > > Is it possible to xfs-freeze a COW copy so access to the original FS isn't > > suspended thus making the time period of an xfs_freeze/dump_names less > > critical? > > I think you're missing some of the basics. > > Conceptually it is typically: > > - quiesce system > - make snapshot > - release system (to accept future filesystem changes) > - use snapshot as a source of point-in-time data (often for backups) > > Since this solution is designed for Enterprise nightly backup use the > various aspects are designed to have minimal impact on the user. When > Oracle as an example is quiesced it maintains a RAM based transaction > log that it replays after it is released to talk to the filesystem > again. > > So quiesce system is typically: > - quiesce apps (many enterprise apps provide a API call for this) ---- Can you name 1 on linux that does? I can't think of any -- sendmail, imap, samba, squid, syslog... > - quiesce filesystem (xfs_freeze can do this, but I'm pretty sure > the kernel freeze is also automatically called by lvm's snapshot > function.) ---- Yeah... I understand this part -- in a few emails that follow the one you are responding to, I showed where I tried exactly this... and either I 'm missing something (which wouldn't be hard, given my vast knowledge (*cough*) in this area), or it's not working. I hope it is the former, as my own ignorance is usually easier to address than the latter. > > make snapshot: > - If you're using LVM, you can do this with it, > - or many raid arrays have API's to do this at your command > - or some filesystems (not xfs) have snapshot functionality built-in ---- The *purpose* of an lvm snapshot -- supposedly, is to create a stable (as in it's not going to be changing underneath you) file system to backup from. It doesn't guarantee that the filesystem is in a consistent state -- but it will prevent future modifications....)... It sounds like -- I'm using the wrong 'order' ... It sounds more like I need to xfs_freeze the live volume, 'home' create a snapshot of that (lvm) - which hopefully won't try to write to the xfs-frozen volume, -- once that is done, xfs-unfreeze.... Irck.... > > release system: > - if you called xfs_freeze, you need to call it again to release > the file i/o to occur again. > - Release any apps you quiesced (again enterprise apps may have > an API for this). > > FYI: I have seen ext4 maintainer Ted T'so recommend that with ext4 you > occasionally make a snapshot like the above and then do fsck on the > snapshot to see if the snapshot filesystem structure is valid. If > errors are detect, then have the cron script send a e-mail to the > admin so he/she can schedule downtime to run fsck on the main > filesystem. I don't know if xfs maintainers have a similar > recommendation for xfs. ---- xfs doesn't really have an fsck -- one was added to dot some 'i' or cross some 't', but it's more of a bother than anything, as it just checks to see if the file system is there. If it is, pass, else, fail. I don't need mkfs to cause my system boot to fail due to a missing disk -- as the system "will or won't" come up in some form, without that disk, regardless. In the past, in rare instances where something was offline (just messed w/disks or deliberately have a disk offline and want to boot), a pure xfs system would just "work" (with the assumption that the offline disk wasn't critical in order to boot). Then I could log-in and continue (or do) whatever maintenance was necessary to restore normalcy. But with the new scripts, the system is guaranteed to fail and not come up except to the console. Certainly not my idea of a robust system. But for some perfection is more important than robust. I usually try change xfs.fsck into a link to /bin/true after updates... > > FYI2: I think opensuse's snapper app has a framework to support a lot > of the above built in, but I'm not sure it supports anything but btrfs > snapshots. --- You are correct -- I was surprised to see them call an app 'snapper' when it doesn't even support lvm snapshots, but only the non-stable and non-production-ready 'btrfs'. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: get filename->inode mappings in bulk for a live fs? 2012-10-04 18:29 ` Linda Walsh @ 2012-10-04 18:59 ` Greg Freemyer 0 siblings, 0 replies; 9+ messages in thread From: Greg Freemyer @ 2012-10-04 18:59 UTC (permalink / raw) To: Linda Walsh; +Cc: xfs-oss On Thu, Oct 4, 2012 at 2:29 PM, Linda Walsh <xfs@tlinx.org> wrote: >> So quiesce system is typically: >> - quiesce apps (many enterprise apps provide a API call for this) > > ---- > Can you name 1 on linux that does? > > I can't think of any -- sendmail, imap, samba, squid, syslog... Commercial - Oracle, DB2 (from IBM) opensource - sendmail has a quiesce signal handler, but I've never used it see sendmail.h extern SIGFUNC_DECL quiesce __P((int)); Postgres and mysql fake it I think, You "quiesce" them by having them do a dump of the database, then the backup picks up the dump. The key thing is knowing the right word to google for, and the right word is quiesce. Greg _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* xfs_freeze same as umount? How is that helpful? 2012-10-04 18:01 ` Greg Freemyer 2012-10-04 18:29 ` Linda Walsh @ 2012-10-04 22:39 ` Linda Walsh 2012-10-04 23:32 ` Dave Chinner 1 sibling, 1 reply; 9+ messages in thread From: Linda Walsh @ 2012-10-04 22:39 UTC (permalink / raw) To: xfs-oss Greg Freemyer wrote: > Conceptually it is typically: > - quiesce system ---- Um... it seems that this is equivalent to being able to umount the disk? When I tried xfs_freeze / fs_freeze got fs-busy -- same as I would if I tried to umount it. I thought the point of xfs_freeze was to allow it to be brought to a consistent state without unmounting it? Coincidentally, after trying a few freezes, the system froze. Last message in log was 'fetchmail' (that might have been writing a log to /home), had hung for 120 seconds... That was all she wrote before the entire system 'froze'... So shouldn't xfs_freeze work without having to to shut down every process on it? If not, then how is it different than umount? _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: xfs_freeze same as umount? How is that helpful? 2012-10-04 22:39 ` xfs_freeze same as umount? How is that helpful? Linda Walsh @ 2012-10-04 23:32 ` Dave Chinner 2012-10-05 0:10 ` Linda Walsh 0 siblings, 1 reply; 9+ messages in thread From: Dave Chinner @ 2012-10-04 23:32 UTC (permalink / raw) To: Linda Walsh; +Cc: xfs-oss On Thu, Oct 04, 2012 at 03:39:33PM -0700, Linda Walsh wrote: > Greg Freemyer wrote: > >Conceptually it is typically: > >- quiesce system > ---- > Um... it seems that this is equivalent to being > able to umount the disk? NO, it's not. freeze intentionally leaves the log dirty, whereas unmount leaves it clean. > When I tried xfs_freeze / fs_freeze got fs-busy -- same as I would > if I tried to umount it. Of course - it's got to write all the dirty data andmetadata in memory to disk. Freeze is about providing a stable, consistent disk image of the filesystem, so it must flush dirty objects from memory to disk to provide that. > I thought the point of xfs_freeze was to allow it to be brought to > a consistent state without unmounting it? Exactly. > Coincidentally, after trying a few freezes, the system froze. Entirely possible if you froze the root filesystem and something you rely on tried to write to the filesystem. Anyway, a one-line "it froze" report doesn't tell us anything about the problem you saw. So: http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: xfs_freeze same as umount? How is that helpful? 2012-10-04 23:32 ` Dave Chinner @ 2012-10-05 0:10 ` Linda Walsh 2012-10-05 0:36 ` Dave Chinner 0 siblings, 1 reply; 9+ messages in thread From: Linda Walsh @ 2012-10-05 0:10 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs-oss Dave Chinner wrote: > On Thu, Oct 04, 2012 at 03:39:33PM -0700, Linda Walsh wrote: >> Greg Freemyer wrote: >>> Conceptually it is typically: >>> - quiesce system >> ---- >> Um... it seems that this is equivalent to being >> able to umount the disk? > > NO, it's not. freeze intentionally leaves the log dirty, whereas > unmount leaves it clean. ---- That's what I thought! > >> When I tried xfs_freeze / fs_freeze got fs-busy -- same as I would >> if I tried to umount it. > > Of course - it's got to write all the dirty data andmetadata in > memory to disk. Freeze is about providing a stable, consistent disk > image of the filesystem, so it must flush dirty objects from memory > to disk to provide that. ---- But it says the freeze failed.... huh. Just tried it again .. ( first time after reboot.. froze it no messages or complaints) ?!?! I don't get it. It gave me a file system busy message before and as near as I could tell -- it wouldn't allow me to xfs_freeze it. Trying the same thing now -- no prob. (though I am ALSO on a newer kernel -- had another problem I solved in looking through the logs for hints about the freeze. > >> I thought the point of xfs_freeze was to allow it to be brought to >> a consistent state without unmounting it? > > Exactly. > >> Coincidentally, after trying a few freezes, the system froze. > > Entirely possible if you froze the root filesystem and something you > rely on tried to write to the filesystem. --- Nep... "/home", and running as root, in root partition with /home elements removed from PATH. trying to be careful. Notice I did say 'Coincidentally' -- (w no/quotes in original). If I was thought there might be a connection or problem, at the very least I would have put 'coincidentally' in quotes.. :-).. > > Anyway, a one-line "it froze" report doesn't tell us anything about > the problem you saw. So: ---- Wasn't sure what I saw or that it was related -- exactly... A possible theory... but nothing I'd blaim on xfs, -- last message in log was: Oct 4 13:52:50 Ishtar kernel: [985735.911825] INFO: task fetchmail:25872 blocke d for more than 120 seconds. Oct 4 13:52:50 Ishtar kernel: [985735.918777] "echo 0 > /proc/sys/kernel/hung_t ask_timeout_secs" disables this message. My kernel *was* setup to panic on a hung task.... instead it just froze... but why fetchmail hung... well if the xfs_freeze "partly took" and just issued the error "because", then that process might have froze trying to write log messages to /home partition.... but 120 secs? seems like it might have been going down before I tried anything with xfs_freeze... But not sure what to report now, as it's not doing the same things. Was running 3.2.29, am running 3.5.4 now.... but 3.2.29 had been up for over 10 days... so maybe something else was going on there... Sorry, when I said corrupt... mis-statement on my part... dirty was what I meant -- but corrupt from the standpoint that the data lacked sufficient integrity for a blockget type operation. But dirty would be more accurate in FS-lingo. (still had my data-integrity hat on)... So if I xfs-freeze something, then take a snapshot, -- I don't see that any of that would help in doing an xfs_blockget o get a dump of inodes->blocks, as it sounds like it would still be dirty... Hey, I think xfs walks on water, so don't think I'm complaining...just trying to figure things out. It's been a good fs for me for over 10 years on my home systems. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: xfs_freeze same as umount? How is that helpful? 2012-10-05 0:10 ` Linda Walsh @ 2012-10-05 0:36 ` Dave Chinner 0 siblings, 0 replies; 9+ messages in thread From: Dave Chinner @ 2012-10-05 0:36 UTC (permalink / raw) To: Linda Walsh; +Cc: xfs-oss On Thu, Oct 04, 2012 at 05:10:00PM -0700, Linda Walsh wrote: > Dave Chinner wrote: > >On Thu, Oct 04, 2012 at 03:39:33PM -0700, Linda Walsh wrote: > >>When I tried xfs_freeze / fs_freeze got fs-busy -- same as I would > >>if I tried to umount it. > > > >Of course - it's got to write all the dirty data andmetadata in > >memory to disk. Freeze is about providing a stable, consistent disk > >image of the filesystem, so it must flush dirty objects from memory > >to disk to provide that. > ---- > But it says the freeze failed.... huh. > > Just tried it again .. ( first time after reboot.. froze it no > messages or complaints) ?!?! I don't get it. > > > It gave me a file system busy message before and as near as I could tell -- > it wouldn't allow me to xfs_freeze it. You mean EBUSY? That means the filesystem was already frozen. > >the problem you saw. So: > ---- > Wasn't sure what I saw or that it was related -- exactly... > > A possible theory... but nothing I'd blaim on xfs, -- last message in log was: > > > Oct 4 13:52:50 Ishtar kernel: [985735.911825] INFO: task fetchmail:25872 blocke > d for more than 120 seconds. > Oct 4 13:52:50 Ishtar kernel: [985735.918777] "echo 0 > /proc/sys/kernel/hung_t > ask_timeout_secs" disables this message. And the full stack trace? > So if I xfs-freeze something, then take a snapshot, -- I don't see that any of that > would help in doing an xfs_blockget o get a dump of inodes->blocks, as it sounds > like it would still be dirty... Only the log is dirty - it has a single dummy record in it to mark it dirty. Otherwise everything is consistent, and the tools will work fine on the snapshot image. You just have to know the magic incantations to run the xfs_db commands directly with the extra flag to tel it to ignore a dirty log. i.e. xfs_ncheck is a diagnostic tool, not something that should be run as part of a regular "get some info from the fs" workflow... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: get filename->inode mappings in bulk for a live fs? 2012-10-04 15:30 get filename->inode mappings in bulk for a live fs? Linda Walsh 2012-10-04 18:01 ` Greg Freemyer @ 2012-10-04 22:49 ` Dave Chinner 1 sibling, 0 replies; 9+ messages in thread From: Dave Chinner @ 2012-10-04 22:49 UTC (permalink / raw) To: Linda Walsh; +Cc: xfs-oss On Thu, Oct 04, 2012 at 08:30:20AM -0700, Linda Walsh wrote: > I notice that attempts to use utils to get name->inode mappings > (xfs_ncheck) seem to have no option to operate on a mounted filesystem. If you need xfs_ncheck to get name->inode number mappings, then you are doing something wrong. $ find . -printf "%h/%f: %i\n" | head -10 ./.: 1010542779 ./.git: 612759677 ./.git/branches: 1013256262 ./.git/hooks: 33523782 ./.git/hooks/guilt: 406321163 ./.git/hooks/guilt/delete: 402681911 Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-10-05 0:35 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-10-04 15:30 get filename->inode mappings in bulk for a live fs? Linda Walsh 2012-10-04 18:01 ` Greg Freemyer 2012-10-04 18:29 ` Linda Walsh 2012-10-04 18:59 ` Greg Freemyer 2012-10-04 22:39 ` xfs_freeze same as umount? How is that helpful? Linda Walsh 2012-10-04 23:32 ` Dave Chinner 2012-10-05 0:10 ` Linda Walsh 2012-10-05 0:36 ` Dave Chinner 2012-10-04 22:49 ` get filename->inode mappings in bulk for a live fs? Dave Chinner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox