* xfs metadata overhead @ 2016-06-22 15:58 Danny Shavit 2016-06-22 22:51 ` Dave Chinner 0 siblings, 1 reply; 7+ messages in thread From: Danny Shavit @ 2016-06-22 15:58 UTC (permalink / raw) To: xfs [-- Attachment #1.1: Type: text/plain, Size: 354 bytes --] Hi, We are looking for a method to estimate the size of metadata overhead for a given file system. We would like to use this value as indicator for the amount of cache memory a system for faster operation. Are there any counters that are maintained in the on-disk data structures like free space for examples? -- Thank you, Danny Shavit Zadarastorage [-- Attachment #1.2: Type: text/html, Size: 567 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: xfs metadata overhead 2016-06-22 15:58 xfs metadata overhead Danny Shavit @ 2016-06-22 22:51 ` Dave Chinner 2016-06-23 15:04 ` Danny Shavit 0 siblings, 1 reply; 7+ messages in thread From: Dave Chinner @ 2016-06-22 22:51 UTC (permalink / raw) To: Danny Shavit; +Cc: xfs On Wed, Jun 22, 2016 at 06:58:16PM +0300, Danny Shavit wrote: > Hi, > > We are looking for a method to estimate the size of metadata overhead for a > given file system. > We would like to use this value as indicator for the amount of cache memory > a system for faster operation. > Are there any counters that are maintained in the on-disk data > structures like free space for examples? No. Right now, you'll need to take a metadump of the filesystem to measure it. The size of the dump file will be a close indication of the amount of metadata in the filesystem as it only contains the filesystem metadata. In future, querying the rmap will enable us to calculate it on the fly, (i.e. not requiring the filesystem to be snapshotted/taken off line to do a metadump). Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: xfs metadata overhead 2016-06-22 22:51 ` Dave Chinner @ 2016-06-23 15:04 ` Danny Shavit 2016-06-23 18:12 ` Eric Sandeen 0 siblings, 1 reply; 7+ messages in thread From: Danny Shavit @ 2016-06-23 15:04 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs [-- Attachment #1.1: Type: text/plain, Size: 1210 bytes --] I see. We will try this direction. BTW: I thought that good estimate would be "volume_size - allocated_size - free_space". But it produced quite a difference compared to metadata dump size. Is there a specific reason? Thanks, Danny On Thu, Jun 23, 2016 at 1:51 AM, Dave Chinner <david@fromorbit.com> wrote: > On Wed, Jun 22, 2016 at 06:58:16PM +0300, Danny Shavit wrote: > > Hi, > > > > We are looking for a method to estimate the size of metadata overhead > for a > > given file system. > > We would like to use this value as indicator for the amount of cache > memory > > a system for faster operation. > > Are there any counters that are maintained in the on-disk data > > structures like free space for examples? > > No. > > Right now, you'll need to take a metadump of the filesystem to > measure it. The size of the dump file will be a close indication of > the amount of metadata in the filesystem as it only contains > the filesystem metadata. > > In future, querying the rmap will enable us to calculate it on the > fly, (i.e. not requiring the filesystem to be snapshotted/taken off > line to do a metadump). > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > -- Regards, Danny [-- Attachment #1.2: Type: text/html, Size: 1946 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: xfs metadata overhead 2016-06-23 15:04 ` Danny Shavit @ 2016-06-23 18:12 ` Eric Sandeen [not found] ` <CAC=x_0hnGQHAenv985+FusWxK_63AatB_XK_mqbAmyt=OxqYrg@mail.gmail.com> 0 siblings, 1 reply; 7+ messages in thread From: Eric Sandeen @ 2016-06-23 18:12 UTC (permalink / raw) To: xfs On 6/23/16 10:04 AM, Danny Shavit wrote: > I see. We will try this direction. > BTW: I thought that good estimate would be "volume_size - > allocated_size - free_space". But it produced quite a difference > compared to metadata dump size. > Is there a specific reason? How do you determine allocated_size, with du? How different? Can you show an example? -Eric > Thanks, > Danny > > On Thu, Jun 23, 2016 at 1:51 AM, Dave Chinner <david@fromorbit.com <mailto:david@fromorbit.com>> wrote: > > On Wed, Jun 22, 2016 at 06:58:16PM +0300, Danny Shavit wrote: > > Hi, > > > > We are looking for a method to estimate the size of metadata overhead for a > > given file system. > > We would like to use this value as indicator for the amount of cache memory > > a system for faster operation. > > Are there any counters that are maintained in the on-disk data > > structures like free space for examples? > > No. > > Right now, you'll need to take a metadump of the filesystem to > measure it. The size of the dump file will be a close indication of > the amount of metadata in the filesystem as it only contains > the filesystem metadata. > > In future, querying the rmap will enable us to calculate it on the > fly, (i.e. not requiring the filesystem to be snapshotted/taken off > line to do a metadump). > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com <mailto:david@fromorbit.com> > > > > > -- > Regards, > Danny > > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <CAC=x_0hnGQHAenv985+FusWxK_63AatB_XK_mqbAmyt=OxqYrg@mail.gmail.com>]
* Re: xfs metadata overhead [not found] ` <CAC=x_0hnGQHAenv985+FusWxK_63AatB_XK_mqbAmyt=OxqYrg@mail.gmail.com> @ 2016-06-24 13:17 ` Eric Sandeen 2016-06-24 13:21 ` Danny Shavit 0 siblings, 1 reply; 7+ messages in thread From: Eric Sandeen @ 2016-06-24 13:17 UTC (permalink / raw) To: Danny Shavit, xfs-oss On 6/24/16 5:34 AM, Danny Shavit wrote: >> How do you determine allocated_size, with du? > yes via du >> How different? Can you show an example? > meta data file size= 5.6 GB (*6,089,374,208*) > > *df:* > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/dm-39 2725683200 955900860 1769782340 36% /export/v1 = 978,842,480,640 bytes > > *du:* du -s /export/v1/ > 952825644 /export/v1/ = 975,693,459,456 bytes > 978,842,480,640 -975,693,459,456 = > > Metadata size according to calculation=*3,149,021,184* bytes Oh, right. I should have thought of this; "du" counts some metadata as well, i.e. a directory full of zero-length files still consumes space which is reported by du. so your 975,693,459,456 bytes is file data as well as some metadata. -Eric > > Thanks, > Danny > > On Thu, Jun 23, 2016 at 9:12 PM, Eric Sandeen <sandeen@sandeen.net <mailto:sandeen@sandeen.net>> wrote: > > On 6/23/16 10:04 AM, Danny Shavit wrote: > > I see. We will try this direction. > > BTW: I thought that good estimate would be "volume_size - > > allocated_size - free_space". But it produced quite a difference > > compared to metadata dump size. > > Is there a specific reason? > > How do you determine allocated_size, with du? > > How different? Can you show an example? > > -Eric > > > Thanks, > > Danny > > > > On Thu, Jun 23, 2016 at 1:51 AM, Dave Chinner <david@fromorbit.com <mailto:david@fromorbit.com> <mailto:david@fromorbit.com <mailto:david@fromorbit.com>>> wrote: > > > > On Wed, Jun 22, 2016 at 06:58:16PM +0300, Danny Shavit wrote: > > > Hi, > > > > > > We are looking for a method to estimate the size of metadata overhead for a > > > given file system. > > > We would like to use this value as indicator for the amount of cache memory > > > a system for faster operation. > > > Are there any counters that are maintained in the on-disk data > > > structures like free space for examples? > > > > No. > > > > Right now, you'll need to take a metadump of the filesystem to > > measure it. The size of the dump file will be a close indication of > > the amount of metadata in the filesystem as it only contains > > the filesystem metadata. > > > > In future, querying the rmap will enable us to calculate it on the > > fly, (i.e. not requiring the filesystem to be snapshotted/taken off > > line to do a metadump). > > > > Cheers, > > > > Dave. > > -- > > Dave Chinner > > david@fromorbit.com <mailto:david@fromorbit.com> <mailto:david@fromorbit.com <mailto:david@fromorbit.com>> > > > > > > > > > > -- > > Regards, > > Danny > > > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com> > > http://oss.sgi.com/mailman/listinfo/xfs > > > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com> > http://oss.sgi.com/mailman/listinfo/xfs > > > > > -- > Regards, > Danny _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: xfs metadata overhead 2016-06-24 13:17 ` Eric Sandeen @ 2016-06-24 13:21 ` Danny Shavit 2016-06-24 22:51 ` Dave Chinner 0 siblings, 1 reply; 7+ messages in thread From: Danny Shavit @ 2016-06-24 13:21 UTC (permalink / raw) To: Eric Sandeen; +Cc: xfs-oss [-- Attachment #1.1: Type: text/plain, Size: 3750 bytes --] I see. So using bulk_stat ioctl in similar way to xfs_fsr and summing allocated size will result with more accurate number? Thanks, Danny On Fri, Jun 24, 2016 at 4:17 PM, Eric Sandeen <sandeen@sandeen.net> wrote: > On 6/24/16 5:34 AM, Danny Shavit wrote: > >> How do you determine allocated_size, with du? > > yes via du > >> How different? Can you show an example? > > meta data file size= 5.6 GB (*6,089,374,208*) > > > > *df:* > > Filesystem 1K-blocks Used Available Use% > Mounted on > > /dev/dm-39 2725683200 955900860 1769782340 36% > /export/v1 = 978,842,480,640 bytes > > > > *du:* du -s /export/v1/ > > 952825644 /export/v1/ = 975,693,459,456 bytes > > 978,842,480,640 -975,693,459,456 = > > > > Metadata size according to calculation=*3,149,021,184* bytes > > Oh, right. I should have thought of this; "du" counts some metadata as > well, i.e. a directory full of zero-length files still consumes space > which is reported by du. > > so your 975,693,459,456 bytes is file data as well as some metadata. > > -Eric > > > > > Thanks, > > Danny > > > > On Thu, Jun 23, 2016 at 9:12 PM, Eric Sandeen <sandeen@sandeen.net > <mailto:sandeen@sandeen.net>> wrote: > > > > On 6/23/16 10:04 AM, Danny Shavit wrote: > > > I see. We will try this direction. > > > BTW: I thought that good estimate would be "volume_size - > > > allocated_size - free_space". But it produced quite a difference > > > compared to metadata dump size. > > > Is there a specific reason? > > > > How do you determine allocated_size, with du? > > > > How different? Can you show an example? > > > > -Eric > > > > > Thanks, > > > Danny > > > > > > On Thu, Jun 23, 2016 at 1:51 AM, Dave Chinner <david@fromorbit.com > <mailto:david@fromorbit.com> <mailto:david@fromorbit.com <mailto: > david@fromorbit.com>>> wrote: > > > > > > On Wed, Jun 22, 2016 at 06:58:16PM +0300, Danny Shavit wrote: > > > > Hi, > > > > > > > > We are looking for a method to estimate the size of metadata > overhead for a > > > > given file system. > > > > We would like to use this value as indicator for the amount > of cache memory > > > > a system for faster operation. > > > > Are there any counters that are maintained in the on-disk > data > > > > structures like free space for examples? > > > > > > No. > > > > > > Right now, you'll need to take a metadump of the filesystem to > > > measure it. The size of the dump file will be a close > indication of > > > the amount of metadata in the filesystem as it only contains > > > the filesystem metadata. > > > > > > In future, querying the rmap will enable us to calculate it on > the > > > fly, (i.e. not requiring the filesystem to be > snapshotted/taken off > > > line to do a metadump). > > > > > > Cheers, > > > > > > Dave. > > > -- > > > Dave Chinner > > > david@fromorbit.com <mailto:david@fromorbit.com> <mailto: > david@fromorbit.com <mailto:david@fromorbit.com>> > > > > > > > > > > > > > > > -- > > > Regards, > > > Danny > > > > > > > > > _______________________________________________ > > > xfs mailing list > > > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com> > > > http://oss.sgi.com/mailman/listinfo/xfs > > > > > > > _______________________________________________ > > xfs mailing list > > xfs@oss.sgi.com <mailto:xfs@oss.sgi.com> > > http://oss.sgi.com/mailman/listinfo/xfs > > > > > > > > > > -- > > Regards, > > Danny > -- Regards, Danny [-- Attachment #1.2: Type: text/html, Size: 6321 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: xfs metadata overhead 2016-06-24 13:21 ` Danny Shavit @ 2016-06-24 22:51 ` Dave Chinner 0 siblings, 0 replies; 7+ messages in thread From: Dave Chinner @ 2016-06-24 22:51 UTC (permalink / raw) To: Danny Shavit; +Cc: Eric Sandeen, xfs-oss On Fri, Jun 24, 2016 at 04:21:39PM +0300, Danny Shavit wrote: > I see. > So using bulk_stat ioctl in similar way to xfs_fsr and summing allocated > size will result with more accurate number? No. du includes *directory blocks* in it's sum. bulkstat only counts inodes, which is not counted by du and is part of the metadata you have already accounted for. Even then, bulkstat doesn't give an accurate account o space used by inodes because they can be sparsely allocated from within inode chunks. Just snapshot the volume, run metadump on the snapshot - it's the fastest, most reliable way to work out how much metadata is in a filesystem right now... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-06-24 22:51 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-06-22 15:58 xfs metadata overhead Danny Shavit
2016-06-22 22:51 ` Dave Chinner
2016-06-23 15:04 ` Danny Shavit
2016-06-23 18:12 ` Eric Sandeen
[not found] ` <CAC=x_0hnGQHAenv985+FusWxK_63AatB_XK_mqbAmyt=OxqYrg@mail.gmail.com>
2016-06-24 13:17 ` Eric Sandeen
2016-06-24 13:21 ` Danny Shavit
2016-06-24 22:51 ` Dave Chinner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox