Modifying blktrace to obtain i/o contents?

Modifying blktrace to obtain i/o contents?

[Sushil Mantri]

Hi,

I am working on de-duplication, for which i need the i/o event traces
along with its contents. I have been looking into blktrace kernel code
for some time, and have thought of a hacky solution to start with. In
all the probe functions where i have "bio" struct i could obtain the
"contents" from it, and write that in the relay channel as well. The
blkparser would have to be modified as well to take this change into
account.

Do you guys think i might be missing something or is there an
alternative way to handle this?

Thanks,
Sushil

Re: Modifying blktrace to obtain i/o contents?

[Steven Whitehouse]

Hi,

On Mon, 2011-12-12 at 12:00 -0500, Sushil Mantri wrote:
> Hi,
> 
> I am working on de-duplication, for which i need the i/o event traces
> along with its contents. I have been looking into blktrace kernel code
> for some time, and have thought of a hacky solution to start with. In
> all the probe functions where i have "bio" struct i could obtain the
> "contents" from it, and write that in the relay channel as well. The
> blkparser would have to be modified as well to take this change into
> account.
> 
> Do you guys think i might be missing something or is there an
> alternative way to handle this?
> 
> Thanks,
> Sushil

I don't think that is probably a good way to do deduplication. Have you
considered just writing a device mapper target for this? Thats probably
the best way to do it,

Steve.

Re: Modifying blktrace to obtain i/o contents?

[Vasily Tarasov]

Steven, I think Sushil just wants to collect block traces that contain
data hashes. If this is the case, then blktrace is the right place to
do it. In fact, I've seen a couple of papers that did exactly that. If
not mistaken, these were papers from FYU.

Sushil, if you will be implementing it - let me know. I'm also
interested in block traces for dedup systems and have some experience
in Linux block layer.

Thank you,
Vasily

On Mon, Dec 12, 2011 at 12:05 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> Hi,
>
> On Mon, 2011-12-12 at 12:00 -0500, Sushil Mantri wrote:
>> Hi,
>>
>> I am working on de-duplication, for which i need the i/o event traces
>> along with its contents. I have been looking into blktrace kernel code
>> for some time, and have thought of a hacky solution to start with. In
>> all the probe functions where i have "bio" struct i could obtain the
>> "contents" from it, and write that in the relay channel as well. The
>> blkparser would have to be modified as well to take this change into
>> account.
>>
>> Do you guys think i might be missing something or is there an
>> alternative way to handle this?
>>
>> Thanks,
>> Sushil
>
> I don't think that is probably a good way to do deduplication. Have you
> considered just writing a device mapper target for this? Thats probably
> the best way to do it,
>
> Steve.
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrace" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Modifying blktrace to obtain i/o contents?

[Steven Whitehouse]

Hi,

On Mon, 2011-12-12 at 15:18 -0500, Vasily Tarasov wrote:
> Steven, I think Sushil just wants to collect block traces that contain
> data hashes. If this is the case, then blktrace is the right place to
> do it. In fact, I've seen a couple of papers that did exactly that. If
> not mistaken, these were papers from FYU.
> 
> Sushil, if you will be implementing it - let me know. I'm also
> interested in block traces for dedup systems and have some experience
> in Linux block layer.
> 
> Thank you,
> Vasily
> 
Yes, I guess that might be the case. The subject said "i/o contents" so
I'd assumed that he wanted the complete data rather than just a hash.
Apologies if I've misunderstood that.

Even so, since there are generic tracepoints now, a small device mapper
target could produce that information in exactly the same way as
blktrace and remain modular. It would also be able to modify the i/o
too, which I assumed was the eventual aim,

Steve.

> On Mon, Dec 12, 2011 at 12:05 PM, Steven Whitehouse <swhiteho@redhat.com> wrote:
> > Hi,
> >
> > On Mon, 2011-12-12 at 12:00 -0500, Sushil Mantri wrote:
> >> Hi,
> >>
> >> I am working on de-duplication, for which i need the i/o event traces
> >> along with its contents. I have been looking into blktrace kernel code
> >> for some time, and have thought of a hacky solution to start with. In
> >> all the probe functions where i have "bio" struct i could obtain the
> >> "contents" from it, and write that in the relay channel as well. The
> >> blkparser would have to be modified as well to take this change into
> >> account.
> >>
> >> Do you guys think i might be missing something or is there an
> >> alternative way to handle this?
> >>
> >> Thanks,
> >> Sushil
> >
> > I don't think that is probably a good way to do deduplication. Have you
> > considered just writing a device mapper target for this? Thats probably
> > the best way to do it,
> >
> > Steve.
> >
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrace" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

Re: Modifying blktrace to obtain i/o contents?

[Vasily Tarasov]

>
> Yes, I guess that might be the case. The subject said "i/o contents" so
> I'd assumed that he wanted the complete data rather than just a hash.
> Apologies if I've misunderstood that.
>
> Even so, since there are generic tracepoints now, a small device mapper
> target could produce that information in exactly the same way as
> blktrace and remain modular. It would also be able to modify the i/o
> too, which I assumed was the eventual aim,
>
> Steve.
>

Right, dm target would be modular, but he would need to implement
certain amount of code for efficient export of trace to the
user-space. Blktrace has implemented it already (and user-space part
is ready as well), so he would not need to do it. However, if there is
already a generic way in kernel to export a lot of data to user-space
efficiently, then dm might work as well. And, of course, if dedup
itself (not trace) is a final goal - then I agree completely - dm
target sounds like a good point.

Vasily

Re: Modifying blktrace to obtain i/o contents?

[Sushil Mantri]

Hi,

Sorry i couldn't reply back as I was busy with my finals :) . I looked
a bit into device mapper, and feel it could work, but since i have
looked into blktrace already i will take that approach. For now, i
need the contents, but i am wondering why obtaining hash using
blktrace is a good idea and not the contents itself. I will obtain
contents as i mentioned in my first message.

Thanks for your replies

On Tue, Dec 13, 2011 at 1:06 PM, Vasily Tarasov <tarasov@vasily.name> wrote:
>>
>> Yes, I guess that might be the case. The subject said "i/o contents" so
>> I'd assumed that he wanted the complete data rather than just a hash.
>> Apologies if I've misunderstood that.
>>
>> Even so, since there are generic tracepoints now, a small device mapper
>> target could produce that information in exactly the same way as
>> blktrace and remain modular. It would also be able to modify the i/o
>> too, which I assumed was the eventual aim,
>>
>> Steve.
>>
>
> Right, dm target would be modular, but he would need to implement
> certain amount of code for efficient export of trace to the
> user-space. Blktrace has implemented it already (and user-space part
> is ready as well), so he would not need to do it. However, if there is
> already a generic way in kernel to export a lot of data to user-space
> efficiently, then dm might work as well. And, of course, if dedup
> itself (not trace) is a final goal - then I agree completely - dm
> target sounds like a good point.
>
> Vasily

Re: Modifying blktrace to obtain i/o contents?

[Vasily Tarasov]

OK, looks like we're not staying on the same ground in this discussion
:) What is the goal of you project? The answer to your question
depends on this.

Vasily

On Mon, Dec 19, 2011 at 3:25 PM, Vasily Tarasov <tarasov@vasily.name> wrote:
> OK, looks like we're not staying on the same ground in this discussion :)
> What is the goal of you project? The answer to your question depends on
> this.
>
> Thank you,
> Vasily
>
>
> On Thu, Dec 15, 2011 at 9:24 AM, Sushil Mantri <sushilmantri@gmail.com>
> wrote:
>>
>> Hi,
>>
>> Sorry i couldn't reply back as I was busy with my finals :) . I looked a
>> bit into device mapper, and feel it could work, but since i have looked into
>> blktrace already i will take that approach. For now, i need the contents,
>> but i am wondering why obtaining hash using blktrace is a good idea and not
>> the contents itself. I will obtain contents as i mentioned in my first
>> message.
>>
>> Thanks for your replies
>>
>>
>> On Tue, Dec 13, 2011 at 1:06 PM, Vasily Tarasov <tarasov@vasily.name>
>> wrote:
>>>
>>> >
>>> > Yes, I guess that might be the case. The subject said "i/o contents" so
>>> > I'd assumed that he wanted the complete data rather than just a hash.
>>> > Apologies if I've misunderstood that.
>>> >
>>> > Even so, since there are generic tracepoints now, a small device mapper
>>> > target could produce that information in exactly the same way as
>>> > blktrace and remain modular. It would also be able to modify the i/o
>>> > too, which I assumed was the eventual aim,
>>> >
>>> > Steve.
>>> >
>>>
>>> Right, dm target would be modular, but he would need to implement
>>> certain amount of code for efficient export of trace to the
>>> user-space. Blktrace has implemented it already (and user-space part
>>> is ready as well), so he would not need to do it. However, if there is
>>> already a generic way in kernel to export a lot of data to user-space
>>> efficiently, then dm might work as well. And, of course, if dedup
>>> itself (not trace) is a final goal - then I agree completely - dm
>>> target sounds like a good point.
>>>
>>> Vasily
>>
>>
>

Re: Modifying blktrace to obtain i/o contents?

[Alireza Haghdoost]

Interesting discussion... I am agree with Vasily about the hashed
content. If the goal of project is dedub, it would be better to store
hashed contents (MD5 or SHA-1). Moreover, you can find some Dedub
trace with fixed size chunk size and MD5 content hash in the Sylab
webpage at FIU.edu



On Mon, Dec 19, 2011 at 15:06, Vasily Tarasov <tarasov@vasily.name> wrote:
>
> OK, looks like we're not staying on the same ground in this discussion
> :) What is the goal of you project? The answer to your question
> depends on this.
>
> Vasily
>
> On Mon, Dec 19, 2011 at 3:25 PM, Vasily Tarasov <tarasov@vasily.name> wrote:
> > OK, looks like we're not staying on the same ground in this discussion :)
> > What is the goal of you project? The answer to your question depends on
> > this.
> >
> > Thank you,
> > Vasily
> >
> >
> > On Thu, Dec 15, 2011 at 9:24 AM, Sushil Mantri <sushilmantri@gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> Sorry i couldn't reply back as I was busy with my finals :) . I looked a
> >> bit into device mapper, and feel it could work, but since i have looked into
> >> blktrace already i will take that approach. For now, i need the contents,
> >> but i am wondering why obtaining hash using blktrace is a good idea and not
> >> the contents itself. I will obtain contents as i mentioned in my first
> >> message.
> >>
> >> Thanks for your replies
> >>
> >>
> >> On Tue, Dec 13, 2011 at 1:06 PM, Vasily Tarasov <tarasov@vasily.name>
> >> wrote:
> >>>
> >>> >
> >>> > Yes, I guess that might be the case. The subject said "i/o contents" so
> >>> > I'd assumed that he wanted the complete data rather than just a hash.
> >>> > Apologies if I've misunderstood that.
> >>> >
> >>> > Even so, since there are generic tracepoints now, a small device mapper
> >>> > target could produce that information in exactly the same way as
> >>> > blktrace and remain modular. It would also be able to modify the i/o
> >>> > too, which I assumed was the eventual aim,
> >>> >
> >>> > Steve.
> >>> >
> >>>
> >>> Right, dm target would be modular, but he would need to implement
> >>> certain amount of code for efficient export of trace to the
> >>> user-space. Blktrace has implemented it already (and user-space part
> >>> is ready as well), so he would not need to do it. However, if there is
> >>> already a generic way in kernel to export a lot of data to user-space
> >>> efficiently, then dm might work as well. And, of course, if dedup
> >>> itself (not trace) is a final goal - then I agree completely - dm
> >>> target sounds like a good point.
> >>>
> >>> Vasily
> >>
> >>
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrace" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html