Re: VFS hot tracking: How to calculate data temperature?

Re: VFS hot tracking: How to calculate data temperature?

[Zhi Yong Wu]

On Fri, Nov 2, 2012 at 4:41 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> On Fri, Nov 02, 2012 at 02:38:29PM +0800, Zhi Yong Wu wrote:
>> Here also has another question.
>>
>> How to save the file temperature among the umount to be able to
>> preserve the file tempreture after reboot?
>>
>> This above is the requirement from DB product.
>> I thought that we can save file temperature in its inode struct, that
>> is, add one new field in struct inode, then this info will be written
>> to disk with inode.
>>
>> Any comments or ideas are appreciated, thanks.
>
> Hi Zhiyong,
>
> I think that we might define a callback function.  If a filesystem wants
> to save these data, it can implement a function to save them.  The
> filesystem can decide whether adding it or not by themselves.
Great idea,  temperature saving function is maybe very specific to FS.
But i am wondering if we can find one generic way to save temperature
info at first.

>
> BTW, actually I don't really care about how to save these data because I
> only want to observe which file is accessed in real time, which is very
> useful for me to track a problem in our product system.
heh, but other guys or products care about this.

>
> Regards,
> Zheng



-- 
Regards,

Zhi Yong Wu

Re: VFS hot tracking: How to calculate data temperature?

[Zheng Liu]

On Mon, Nov 05, 2012 at 10:29:39AM +0800, Zhi Yong Wu wrote:
> On Fri, Nov 2, 2012 at 4:41 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> > On Fri, Nov 02, 2012 at 02:38:29PM +0800, Zhi Yong Wu wrote:
> >> Here also has another question.
> >>
> >> How to save the file temperature among the umount to be able to
> >> preserve the file tempreture after reboot?
> >>
> >> This above is the requirement from DB product.
> >> I thought that we can save file temperature in its inode struct, that
> >> is, add one new field in struct inode, then this info will be written
> >> to disk with inode.
> >>
> >> Any comments or ideas are appreciated, thanks.
> >
> > Hi Zhiyong,
> >
> > I think that we might define a callback function.  If a filesystem wants
> > to save these data, it can implement a function to save them.  The
> > filesystem can decide whether adding it or not by themselves.
> Great idea,  temperature saving function is maybe very specific to FS.
> But i am wondering if we can find one generic way to save temperature
> info at first.

I don't think a generic way is better because it cannot support a
variety of filesystems.  So maybe you must answer this question firstly:
how many filesystems do you want to save this info? such as ext4, xfs,
btrfs, etc.  Then we can try to find a generic way.  If only these three
filesystems you want to support, maybe saving in xattr is an optional
way.

Regards,
Zheng

Re: VFS hot tracking: How to calculate data temperature?

[Zhi Yong Wu]

On Tue, Nov 6, 2012 at 4:39 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> On Mon, Nov 05, 2012 at 10:29:39AM +0800, Zhi Yong Wu wrote:
>> On Fri, Nov 2, 2012 at 4:41 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
>> > On Fri, Nov 02, 2012 at 02:38:29PM +0800, Zhi Yong Wu wrote:
>> >> Here also has another question.
>> >>
>> >> How to save the file temperature among the umount to be able to
>> >> preserve the file tempreture after reboot?
>> >>
>> >> This above is the requirement from DB product.
>> >> I thought that we can save file temperature in its inode struct, that
>> >> is, add one new field in struct inode, then this info will be written
>> >> to disk with inode.
>> >>
>> >> Any comments or ideas are appreciated, thanks.
>> >
>> > Hi Zhiyong,
>> >
>> > I think that we might define a callback function.  If a filesystem wants
>> > to save these data, it can implement a function to save them.  The
>> > filesystem can decide whether adding it or not by themselves.
>> Great idea,  temperature saving function is maybe very specific to FS.
>> But i am wondering if we can find one generic way to save temperature
>> info at first.
>
> I don't think a generic way is better because it cannot support a
> variety of filesystems.  So maybe you must answer this question firstly:
> how many filesystems do you want to save this info? such as ext4, xfs,
> btrfs, etc.  Then we can try to find a generic way.  If only these three
> filesystems you want to support, maybe saving in xattr is an optional
> way.
yes, xattr is one good choice from currect discussion result. Maybe we
can provide one generic way, and one callback registering
infrastructure, if FS register its own saving callback, this callback
function will be used, otherwise the generic way will be applied.

By the way, as what Dave mentioned, the patchset v4+ review has
highest priority, then the way how to calc data temperature, and the
lowest priority is the way how to save data temperature info.

>
> Regards,
> Zheng



-- 
Regards,

Zhi Yong Wu

Re: VFS hot tracking: How to calculate data temperature?

[Zheng Liu]

On Tue, Nov 06, 2012 at 05:00:19PM +0800, Zhi Yong Wu wrote:
> On Tue, Nov 6, 2012 at 4:39 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> > On Mon, Nov 05, 2012 at 10:29:39AM +0800, Zhi Yong Wu wrote:
> >> On Fri, Nov 2, 2012 at 4:41 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> >> > On Fri, Nov 02, 2012 at 02:38:29PM +0800, Zhi Yong Wu wrote:
> >> >> Here also has another question.
> >> >>
> >> >> How to save the file temperature among the umount to be able to
> >> >> preserve the file tempreture after reboot?
> >> >>
> >> >> This above is the requirement from DB product.
> >> >> I thought that we can save file temperature in its inode struct, that
> >> >> is, add one new field in struct inode, then this info will be written
> >> >> to disk with inode.
> >> >>
> >> >> Any comments or ideas are appreciated, thanks.
> >> >
> >> > Hi Zhiyong,
> >> >
> >> > I think that we might define a callback function.  If a filesystem wants
> >> > to save these data, it can implement a function to save them.  The
> >> > filesystem can decide whether adding it or not by themselves.
> >> Great idea,  temperature saving function is maybe very specific to FS.
> >> But i am wondering if we can find one generic way to save temperature
> >> info at first.
> >
> > I don't think a generic way is better because it cannot support a
> > variety of filesystems.  So maybe you must answer this question firstly:
> > how many filesystems do you want to save this info? such as ext4, xfs,
> > btrfs, etc.  Then we can try to find a generic way.  If only these three
> > filesystems you want to support, maybe saving in xattr is an optional
> > way.
> yes, xattr is one good choice from currect discussion result. Maybe we
> can provide one generic way, and one callback registering
> infrastructure, if FS register its own saving callback, this callback
> function will be used, otherwise the generic way will be applied.
> 
> By the way, as what Dave mentioned, the patchset v4+ review has
> highest priority, then the way how to calc data temperature, and the
> lowest priority is the way how to save data temperature info.

Great!  Thanks for sharing the news with me.  IMHO the highest priority
is that we must know the overhead that this patch set costs after using
these patches.  My point of view is that there is no any overhead when
it is disabled, and it only brings a little overhead when it is enabled.

Regards,
Zheng

Re: VFS hot tracking: How to calculate data temperature?

[Zhi Yong Wu]

On Sat, Nov 3, 2012 at 4:10 AM, Darrick J. Wong <darrick.wong@oracle.com> wrote:
> On Fri, Nov 02, 2012 at 04:41:09PM +0800, Zheng Liu wrote:
>> On Fri, Nov 02, 2012 at 02:38:29PM +0800, Zhi Yong Wu wrote:
>> > Here also has another question.
>> >
>> > How to save the file temperature among the umount to be able to
>> > preserve the file tempreture after reboot?
>> >
>> > This above is the requirement from DB product.
>> > I thought that we can save file temperature in its inode struct, that
>> > is, add one new field in struct inode, then this info will be written
>> > to disk with inode.
>> >
>> > Any comments or ideas are appreciated, thanks.
>>
>> Hi Zhiyong,
>>
>> I think that we might define a callback function.  If a filesystem wants
>> to save these data, it can implement a function to save them.  The
>> filesystem can decide whether adding it or not by themselves.
>>
>> BTW, actually I don't really care about how to save these data because I
>> only want to observe which file is accessed in real time, which is very
>> useful for me to track a problem in our product system.
>
> <shrug> I _think_ the vfs quota code simply asks the filesystem for a special
> inode where it save the quota data in whatever (FS-agnostic) format it wants.
> Have you considered something like that?
No, but it is one good hint for my issue. thanks.
>
> (Or, maybe everyone secretly hates doing that?  Secret files, yaaay...)
ah, do you think of doing that?

>
> --D
>>
>> Regards,
>> Zheng
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Regards,

Zhi Yong Wu

Re: VFS hot tracking: How to calculate data temperature?

[Zhi Yong Wu]

On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@us.ibm.com> wrote:
> On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
>> Here also has another question.
>>
>> How to save the file temperature among the umount to be able to
>> preserve the file tempreture after reboot?
>>
>> This above is the requirement from DB product.
>> I thought that we can save file temperature in its inode struct, that
>> is, add one new field in struct inode, then this info will be written
>> to disk with inode.
>>
>> Any comments or ideas are appreciated, thanks.
>>
>>
>
> Maybe could save the last file temperature with extended attributes.
It seems that only ext4 has the concept of extended attributes.

> Just save the per-inode temperature only for now.
>
> Mingming
>> On Fri, Nov 2, 2012 at 12:04 PM, Zhi Yong Wu <zwu.kernel@gmail.com> wrote:
>> > HI, guys
>> >
>> >    VFS hot tracking currently show result as below, and it is very
>> > strange and not nice.
>> >
>> > inode #279, reads 0, writes 1, avg read time 18446744073709551615,
>> > avg write time 5251566408153596, temp 109
>> >
>> > Do anyone know if there is one simpler but effective way to calculate
>> > data temperature?
>> >
>> > --
>> > Regards,
>> >
>> > Zhi Yong Wu
>>
>>
>>
>
>
>



-- 
Regards,

Zhi Yong Wu

Re: VFS hot tracking: How to calculate data temperature?

[Dave Chinner]

On Mon, Nov 05, 2012 at 10:35:50AM +0800, Zhi Yong Wu wrote:
> On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@us.ibm.com> wrote:
> > On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
> >> Here also has another question.
> >>
> >> How to save the file temperature among the umount to be able to
> >> preserve the file tempreture after reboot?
> >>
> >> This above is the requirement from DB product.
> >> I thought that we can save file temperature in its inode struct, that
> >> is, add one new field in struct inode, then this info will be written
> >> to disk with inode.
> >>
> >> Any comments or ideas are appreciated, thanks.
> >>
> >>
> >
> > Maybe could save the last file temperature with extended attributes.
> It seems that only ext4 has the concept of extended attributes.

All major filesystems have xattr support. They are used extensively
by the security and integrity subsystems, for example.

Saving the information might be something that is useful to certian
applications, but lets have the people that need that functionality
spell out their requirements before discussing how or what to
implement.  Indeed, discussion shoul dreally focus on getting the
core, in-memory infrastructure sorted out first before trying to
expand the functionality further...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: VFS hot tracking: How to calculate data temperature?

[Zhi Yong Wu]

On Mon, Nov 5, 2012 at 4:28 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Mon, Nov 05, 2012 at 10:35:50AM +0800, Zhi Yong Wu wrote:
>> On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@us.ibm.com> wrote:
>> > On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
>> >> Here also has another question.
>> >>
>> >> How to save the file temperature among the umount to be able to
>> >> preserve the file tempreture after reboot?
>> >>
>> >> This above is the requirement from DB product.
>> >> I thought that we can save file temperature in its inode struct, that
>> >> is, add one new field in struct inode, then this info will be written
>> >> to disk with inode.
>> >>
>> >> Any comments or ideas are appreciated, thanks.
>> >>
>> >>
>> >
>> > Maybe could save the last file temperature with extended attributes.
>> It seems that only ext4 has the concept of extended attributes.
>
> All major filesystems have xattr support. They are used extensively
> by the security and integrity subsystems, for example.
got it, thanks.
>
> Saving the information might be something that is useful to certian
> applications, but lets have the people that need that functionality
> spell out their requirements before discussing how or what to
> implement.  Indeed, discussion shoul dreally focus on getting the
> core, in-memory infrastructure sorted out first before trying to
> expand the functionality further...
ah, but the latest patchset need some love from experienced FS guys:).......
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com



-- 
Regards,

Zhi Yong Wu

Re: VFS hot tracking: How to calculate data temperature?

[Steven Whitehouse]

Hi,

On Mon, 2012-11-05 at 16:44 +0800, Zhi Yong Wu wrote:
> On Mon, Nov 5, 2012 at 4:28 PM, Dave Chinner <david@fromorbit.com> wrote:
> > On Mon, Nov 05, 2012 at 10:35:50AM +0800, Zhi Yong Wu wrote:
> >> On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@us.ibm.com> wrote:
> >> > On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
> >> >> Here also has another question.
> >> >>
> >> >> How to save the file temperature among the umount to be able to
> >> >> preserve the file tempreture after reboot?
> >> >>
> >> >> This above is the requirement from DB product.
> >> >> I thought that we can save file temperature in its inode struct, that
> >> >> is, add one new field in struct inode, then this info will be written
> >> >> to disk with inode.
> >> >>
> >> >> Any comments or ideas are appreciated, thanks.
> >> >>
> >> >>
> >> >
> >> > Maybe could save the last file temperature with extended attributes.
> >> It seems that only ext4 has the concept of extended attributes.
> >
> > All major filesystems have xattr support. They are used extensively
> > by the security and integrity subsystems, for example.
> got it, thanks.
> >
> > Saving the information might be something that is useful to certian
> > applications, but lets have the people that need that functionality
> > spell out their requirements before discussing how or what to
> > implement.  Indeed, discussion shoul dreally focus on getting the
> > core, in-memory infrastructure sorted out first before trying to
> > expand the functionality further...
> ah, but the latest patchset need some love from experienced FS guys:).......

There is one other possible issue with saving the data into the
filesystem, which is that it may disturb what you are trying to measure.
Some filesystems (GFS2 is one) store data for small inodes in the same
block as the inode itself. So that means the accesses to the saved hot
tracking info may potentially affect the data access times too. Also
there is a very limited amount of space to expand the number of fields
in the inode, so xattr may be the only solution, depending on how much
data needs to be stored in each case.

In the GFS2 case (I don't think it is unique in this) xattrs are stored
out of line and having to access them in every open means an extra block
read per inode, which again has performance implications.

So that is not an insurmountable problem, but something to take into
account in selecting a solution,

Steve.

Re: VFS hot tracking: How to calculate data temperature?

[Zhi Yong Wu]

On Mon, Nov 5, 2012 at 6:33 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> Hi,
>
> On Mon, 2012-11-05 at 16:44 +0800, Zhi Yong Wu wrote:
>> On Mon, Nov 5, 2012 at 4:28 PM, Dave Chinner <david@fromorbit.com> wrote:
>> > On Mon, Nov 05, 2012 at 10:35:50AM +0800, Zhi Yong Wu wrote:
>> >> On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@us.ibm.com> wrote:
>> >> > On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
>> >> >> Here also has another question.
>> >> >>
>> >> >> How to save the file temperature among the umount to be able to
>> >> >> preserve the file tempreture after reboot?
>> >> >>
>> >> >> This above is the requirement from DB product.
>> >> >> I thought that we can save file temperature in its inode struct, that
>> >> >> is, add one new field in struct inode, then this info will be written
>> >> >> to disk with inode.
>> >> >>
>> >> >> Any comments or ideas are appreciated, thanks.
>> >> >>
>> >> >>
>> >> >
>> >> > Maybe could save the last file temperature with extended attributes.
>> >> It seems that only ext4 has the concept of extended attributes.
>> >
>> > All major filesystems have xattr support. They are used extensively
>> > by the security and integrity subsystems, for example.
>> got it, thanks.
>> >
>> > Saving the information might be something that is useful to certian
>> > applications, but lets have the people that need that functionality
>> > spell out their requirements before discussing how or what to
>> > implement.  Indeed, discussion shoul dreally focus on getting the
>> > core, in-memory infrastructure sorted out first before trying to
>> > expand the functionality further...
>> ah, but the latest patchset need some love from experienced FS guys:).......
>
> There is one other possible issue with saving the data into the
> filesystem, which is that it may disturb what you are trying to measure.
> Some filesystems (GFS2 is one) store data for small inodes in the same
> block as the inode itself. So that means the accesses to the saved hot
> tracking info may potentially affect the data access times too. Also
> there is a very limited amount of space to expand the number of fields
> in the inode, so xattr may be the only solution, depending on how much
> data needs to be stored in each case.
Very good analysis, two possible issues are very meanful, thanks.
>
> In the GFS2 case (I don't think it is unique in this) xattrs are stored
> out of line and having to access them in every open means an extra block
> read per inode, which again has performance implications.
>
> So that is not an insurmountable problem, but something to take into
> account in selecting a solution,
In summary, you look like preferring to xattr as its solution.

>
> Steve.
>
>
>



-- 
Regards,

Zhi Yong Wu

Re: VFS hot tracking: How to calculate data temperature?

[Steven Whitehouse]

Hi,

On Mon, 2012-11-05 at 19:46 +0800, Zhi Yong Wu wrote:
> On Mon, Nov 5, 2012 at 6:33 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> > Hi,
> >
> > On Mon, 2012-11-05 at 16:44 +0800, Zhi Yong Wu wrote:
> >> On Mon, Nov 5, 2012 at 4:28 PM, Dave Chinner <david@fromorbit.com> wrote:
> >> > On Mon, Nov 05, 2012 at 10:35:50AM +0800, Zhi Yong Wu wrote:
> >> >> On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@us.ibm.com> wrote:
> >> >> > On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
> >> >> >> Here also has another question.
> >> >> >>
> >> >> >> How to save the file temperature among the umount to be able to
> >> >> >> preserve the file tempreture after reboot?
> >> >> >>
> >> >> >> This above is the requirement from DB product.
> >> >> >> I thought that we can save file temperature in its inode struct, that
> >> >> >> is, add one new field in struct inode, then this info will be written
> >> >> >> to disk with inode.
> >> >> >>
> >> >> >> Any comments or ideas are appreciated, thanks.
> >> >> >>
> >> >> >>
> >> >> >
> >> >> > Maybe could save the last file temperature with extended attributes.
> >> >> It seems that only ext4 has the concept of extended attributes.
> >> >
> >> > All major filesystems have xattr support. They are used extensively
> >> > by the security and integrity subsystems, for example.
> >> got it, thanks.
> >> >
> >> > Saving the information might be something that is useful to certian
> >> > applications, but lets have the people that need that functionality
> >> > spell out their requirements before discussing how or what to
> >> > implement.  Indeed, discussion shoul dreally focus on getting the
> >> > core, in-memory infrastructure sorted out first before trying to
> >> > expand the functionality further...
> >> ah, but the latest patchset need some love from experienced FS guys:).......
> >
> > There is one other possible issue with saving the data into the
> > filesystem, which is that it may disturb what you are trying to measure.
> > Some filesystems (GFS2 is one) store data for small inodes in the same
> > block as the inode itself. So that means the accesses to the saved hot
> > tracking info may potentially affect the data access times too. Also
> > there is a very limited amount of space to expand the number of fields
> > in the inode, so xattr may be the only solution, depending on how much
> > data needs to be stored in each case.
> Very good analysis, two possible issues are very meanful, thanks.
> >
> > In the GFS2 case (I don't think it is unique in this) xattrs are stored
> > out of line and having to access them in every open means an extra block
> > read per inode, which again has performance implications.
> >
> > So that is not an insurmountable problem, but something to take into
> > account in selecting a solution,
> In summary, you look like preferring to xattr as its solution.
> 

Well, that depends on exactly how large the data to be stored is, and
other factors. It will add overhead to the storage/retrieval but at
least it is fairly generic (wrt on-disk format) so likely to be easier
to retrofit to existing filesystems.

I suspect this may be one of those cases where there is no obvious right
answer and it is a case of selecting the least worst option, if that
makes sense?

Steve.

Re: VFS hot tracking: How to calculate data temperature?

[Zhi Yong Wu]

On Mon, Nov 5, 2012 at 7:57 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> Hi,
>
> On Mon, 2012-11-05 at 19:46 +0800, Zhi Yong Wu wrote:
>> On Mon, Nov 5, 2012 at 6:33 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
>> > Hi,
>> >
>> > On Mon, 2012-11-05 at 16:44 +0800, Zhi Yong Wu wrote:
>> >> On Mon, Nov 5, 2012 at 4:28 PM, Dave Chinner <david@fromorbit.com> wrote:
>> >> > On Mon, Nov 05, 2012 at 10:35:50AM +0800, Zhi Yong Wu wrote:
>> >> >> On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@us.ibm.com> wrote:
>> >> >> > On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
>> >> >> >> Here also has another question.
>> >> >> >>
>> >> >> >> How to save the file temperature among the umount to be able to
>> >> >> >> preserve the file tempreture after reboot?
>> >> >> >>
>> >> >> >> This above is the requirement from DB product.
>> >> >> >> I thought that we can save file temperature in its inode struct, that
>> >> >> >> is, add one new field in struct inode, then this info will be written
>> >> >> >> to disk with inode.
>> >> >> >>
>> >> >> >> Any comments or ideas are appreciated, thanks.
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> > Maybe could save the last file temperature with extended attributes.
>> >> >> It seems that only ext4 has the concept of extended attributes.
>> >> >
>> >> > All major filesystems have xattr support. They are used extensively
>> >> > by the security and integrity subsystems, for example.
>> >> got it, thanks.
>> >> >
>> >> > Saving the information might be something that is useful to certian
>> >> > applications, but lets have the people that need that functionality
>> >> > spell out their requirements before discussing how or what to
>> >> > implement.  Indeed, discussion shoul dreally focus on getting the
>> >> > core, in-memory infrastructure sorted out first before trying to
>> >> > expand the functionality further...
>> >> ah, but the latest patchset need some love from experienced FS guys:).......
>> >
>> > There is one other possible issue with saving the data into the
>> > filesystem, which is that it may disturb what you are trying to measure.
>> > Some filesystems (GFS2 is one) store data for small inodes in the same
>> > block as the inode itself. So that means the accesses to the saved hot
>> > tracking info may potentially affect the data access times too. Also
>> > there is a very limited amount of space to expand the number of fields
>> > in the inode, so xattr may be the only solution, depending on how much
>> > data needs to be stored in each case.
>> Very good analysis, two possible issues are very meanful, thanks.
>> >
>> > In the GFS2 case (I don't think it is unique in this) xattrs are stored
>> > out of line and having to access them in every open means an extra block
>> > read per inode, which again has performance implications.
>> >
>> > So that is not an insurmountable problem, but something to take into
>> > account in selecting a solution,
>> In summary, you look like preferring to xattr as its solution.
>>
>
> Well, that depends on exactly how large the data to be stored is, and
> other factors. It will add overhead to the storage/retrieval but at
> least it is fairly generic (wrt on-disk format) so likely to be easier
> to retrofit to existing filesystems.
Do you have some idea with more details about how to retrofit to existing FS?:)
>
> I suspect this may be one of those cases where there is no obvious right
> answer and it is a case of selecting the least worst option, if that
> makes sense?
Then we can only check which solution is better via large scale
performance test.
>
> Steve.
>
>



-- 
Regards,

Zhi Yong Wu

Re: VFS hot tracking: How to calculate data temperature?

[Steven Whitehouse]

Hi,

On Mon, 2012-11-05 at 20:18 +0800, Zhi Yong Wu wrote:
> On Mon, Nov 5, 2012 at 7:57 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> > Hi,
> >
> > On Mon, 2012-11-05 at 19:46 +0800, Zhi Yong Wu wrote:
> >> On Mon, Nov 5, 2012 at 6:33 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> >> > Hi,
> >> >
> >> > On Mon, 2012-11-05 at 16:44 +0800, Zhi Yong Wu wrote:
> >> >> On Mon, Nov 5, 2012 at 4:28 PM, Dave Chinner <david@fromorbit.com> wrote:
> >> >> > On Mon, Nov 05, 2012 at 10:35:50AM +0800, Zhi Yong Wu wrote:
> >> >> >> On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@us.ibm.com> wrote:
> >> >> >> > On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
> >> >> >> >> Here also has another question.
> >> >> >> >>
> >> >> >> >> How to save the file temperature among the umount to be able to
> >> >> >> >> preserve the file tempreture after reboot?
> >> >> >> >>
> >> >> >> >> This above is the requirement from DB product.
> >> >> >> >> I thought that we can save file temperature in its inode struct, that
> >> >> >> >> is, add one new field in struct inode, then this info will be written
> >> >> >> >> to disk with inode.
> >> >> >> >>
> >> >> >> >> Any comments or ideas are appreciated, thanks.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> > Maybe could save the last file temperature with extended attributes.
> >> >> >> It seems that only ext4 has the concept of extended attributes.
> >> >> >
> >> >> > All major filesystems have xattr support. They are used extensively
> >> >> > by the security and integrity subsystems, for example.
> >> >> got it, thanks.
> >> >> >
> >> >> > Saving the information might be something that is useful to certian
> >> >> > applications, but lets have the people that need that functionality
> >> >> > spell out their requirements before discussing how or what to
> >> >> > implement.  Indeed, discussion shoul dreally focus on getting the
> >> >> > core, in-memory infrastructure sorted out first before trying to
> >> >> > expand the functionality further...
> >> >> ah, but the latest patchset need some love from experienced FS guys:).......
> >> >
> >> > There is one other possible issue with saving the data into the
> >> > filesystem, which is that it may disturb what you are trying to measure.
> >> > Some filesystems (GFS2 is one) store data for small inodes in the same
> >> > block as the inode itself. So that means the accesses to the saved hot
> >> > tracking info may potentially affect the data access times too. Also
> >> > there is a very limited amount of space to expand the number of fields
> >> > in the inode, so xattr may be the only solution, depending on how much
> >> > data needs to be stored in each case.
> >> Very good analysis, two possible issues are very meanful, thanks.
> >> >
> >> > In the GFS2 case (I don't think it is unique in this) xattrs are stored
> >> > out of line and having to access them in every open means an extra block
> >> > read per inode, which again has performance implications.
> >> >
> >> > So that is not an insurmountable problem, but something to take into
> >> > account in selecting a solution,
> >> In summary, you look like preferring to xattr as its solution.
> >>
> >
> > Well, that depends on exactly how large the data to be stored is, and
> > other factors. It will add overhead to the storage/retrieval but at
> > least it is fairly generic (wrt on-disk format) so likely to be easier
> > to retrofit to existing filesystems.
> Do you have some idea with more details about how to retrofit to existing FS?:)

Well I think we've already covered the obvious ways...

> >
> > I suspect this may be one of those cases where there is no obvious right
> > answer and it is a case of selecting the least worst option, if that
> > makes sense?
> Then we can only check which solution is better via large scale
> performance test.

Indeed, and that will be to a certain extent fs dependent too,

Steve.