Tmf Ctf Parser Scalability issues (3/3)

Tmf Ctf Parser Scalability issues (3/3)

[Matthew Khouzam]

Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I
have come up with three points where scalability will be an issue. I am
sending 3 emails, each one describing one of the issues so we can
aggregate them more cohesively. First a primer, CTF is a file format
that has traces written in packets, the packets are parts of files in
streams, the streams are files in a directory.

Issue 3: Event size
As hinted in the previous email, it can definitely be a possibility that
traces can have events that are too large to fit into memory. These
events could also make reading a packet difficult, since the packet
could have a said event in the middle, and therefore, we could end up
with a packet that cannot be read properly with the sliding window
memory map, unless corrective actions are taken.

Proposed solution:
Would anyone do this? I see it as a longer term problem and as a known
limitation, unless someone says "Hey, I want 8gb events!".

Re: Tmf Ctf Parser Scalability issues (3/3)

[Mathieu Desnoyers]

----- Original Message -----
> From: "Matthew Khouzam" <matthew.khouzam@ericsson.com>
> To: lttng-dev@lists.lttng.org, "Linux Tools developer discussions" <linuxtools-dev@eclipse.org>
> Sent: Tuesday, November 26, 2013 11:44:51 AM
> Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (3/3)
> 
> Hello all,
> I was looking at the CTF parser in the TMF project of Linux Tools. I
> have come up with three points where scalability will be an issue. I am
> sending 3 emails, each one describing one of the issues so we can
> aggregate them more cohesively. First a primer, CTF is a file format
> that has traces written in packets, the packets are parts of files in
> streams, the streams are files in a directory.
> 
> Issue 3: Event size
> As hinted in the previous email, it can definitely be a possibility that
> traces can have events that are too large to fit into memory. These
> events could also make reading a packet difficult, since the packet
> could have a said event in the middle, and therefore, we could end up
> with a packet that cannot be read properly with the sliding window
> memory map, unless corrective actions are taken.
> 
> Proposed solution:
> Would anyone do this? I see it as a longer term problem and as a known
> limitation, unless someone says "Hey, I want 8gb events!".

I'm tempted to say that we should put that limitation on individual
non-compound field, rather than events. Therefore, you could not have
such a large string, nor such a large integer, but you could very well
have a very large array of integers within an event which would be 8gb
in size.

Thanks,

Mathieu

> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com