[lttng-dev] In lttng-live mode, if the printing speed cannot keep up with the generation speed of the parsed ctf data, where will the data be stored?

[lttng-dev] In lttng-live mode, if the printing speed cannot keep up with the generation speed of the parsed ctf data, where will the data be stored?

[熊毓华 via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1562 bytes --]

Hi~

I am getting the trace data using lttng-live mode via babeltrace2. In the final step, I borrowed the code in doc/api/libbabeltrace2/examples/epitome.c.




The difference is that in print_message(line 172) in function epitome_out_consume, instead of printing the data directly to the console, I send it out using the socket.

Of course, I can get the parsed results, such as timestamp, event->name, etc., through the socket's recv function on the other end.




However, we know that LTTng is continuously generating CTF data, and BabelTrace2 will continuously parse CTF data. Babeltrace2 then continuously sends parsed CTF data over the socket(in my code) or prints it directly to the console(in the example "epitome" given).




I wonder if my socket recv function is blocked on the other end, causing the socket send function to block in print_message function in babeltrace2;or when printing to the console, the printing speed can't keep up with the parsed CTF data generation speed and the print buffer is also full.

In this case, how will Babeltrace2 handle the parsed CTF data that has not been sent yet, store them in a buffer, a queue or just discard them? Or would the blocking directly cause LTTng to discard the original CTF data at the ring buffer before LTTng Consumer daemon?




I thought I can find the reason in the bt_message_iterator_next function(line 144),but I failed.So I wrote this email hoping to find out why.




Looking forward to your reply.It would be better if you could help me point it out in the code.




thanks,

Yuhua.

[-- Attachment #1.2: Type: text/html, Size: 4129 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] In lttng-live mode, if the printing speed cannot keep up with the generation speed of the parsed ctf data, where will the data be stored?

[Jonathan Rajotte-Julien via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1140 bytes --]

Hi 

> I wonder if my socket recv function is blocked on the other end, causing the
> socket send function to block in print_message function in babeltrace2;or when
> printing to the console, the printing speed can't keep up with the parsed CTF
> data generation speed and the print buffer is also full.

> In this case, how will Babeltrace2 handle the parsed CTF data that has not been
> sent yet, store them in a buffer, a queue or just discard them? Or would the
> blocking directly cause LTTng to discard the original CTF data at the ring
> buffer before LTTng Consumer daemon?
Not sure I understand your setup but when using lttng-live, you are effectively reading from the data that lttng-relayd is collecting, piece by piece. 
The producing side is not affected by the speed at which the reader (babeltrace2) consume data from lttng-relayd using the lttng-live protocol. 
There might be some corner case here and there but for most base usage of the "live" feature reader (babeltrace2) and producer(lttng/lttng-consumerd) 
are "decoupled". You can consider the "lttng-relayd" (and the trace on the filesystem) as the "buffer" here. 

[-- Attachment #1.2: Type: text/html, Size: 1949 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] In lttng-live mode, if the printing speed cannot keep up with the generation speed of the parsed ctf data, where will the data be stored?

[熊毓华 via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1958 bytes --]

If for some reason (for example, here my socket send function in print_message is blocked and the Babeltrace2 may be suspended), the speed of the reader (babeltrace2) cannot keep up with the speed of the producer(lttng/lttng-consumerd), can the size of this buffer("lttng-relayd" ) be set? How much data can it store？

Looking forward to your reply！

thx

Yuhua




-----原始邮件-----
发件人:"Jonathan Rajotte-Julien" <jonathan.rajotte-julien@efficios.com>
发送时间:2022-03-09 00:39:01 (星期三)
收件人: "熊毓华" <xiongyuhua@zju.edu.cn>
抄送: lttng-dev <lttng-dev@lists.lttng.org>, "Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>, ychen@northwestern.edu
主题: Re: In lttng-live mode, if the printing speed cannot keep up with the generation speed of the parsed ctf data, where will the data be stored?


Hi


I wonder if my socket recv function is blocked on the other end, causing the socket send function to block in print_message function in babeltrace2;or when printing to the console, the printing speed can't keep up with the parsed CTF data generation speed and the print buffer is also full.

In this case, how will Babeltrace2 handle the parsed CTF data that has not been sent yet, store them in a buffer, a queue or just discard them? Or would the blocking directly cause LTTng to discard the original CTF data at the ring buffer before LTTng Consumer daemon?



Not sure I understand your setup but when using lttng-live, you are effectively reading from the data that lttng-relayd is collecting, piece by piece.
The producing side is not affected by the speed at which the reader (babeltrace2) consume data from lttng-relayd using the lttng-live protocol.
There might be some corner case here and there but for most base usage of the "live" feature reader (babeltrace2) and producer(lttng/lttng-consumerd)
are "decoupled". You can consider the "lttng-relayd" (and the trace on the filesystem) as the "buffer" here.

[-- Attachment #1.2: Type: text/html, Size: 3679 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] In lttng-live mode, if the printing speed cannot keep up with the generation speed of the parsed ctf data, where will the data be stored?

[Jonathan Rajotte-Julien via lttng-dev]

On Wed, Mar 09, 2022 at 01:26:42AM +0800, 熊毓华 wrote:
> If for some reason (for example, here my socket send function in print_message
> is blocked and the Babeltrace2 may be suspended), the speed of the reader
> (babeltrace2) cannot keep up with the speed of the
> producer(lttng/lttng-consumerd), can the size of this buffer("lttng-relayd" )
> be set? How much data can it store？

Unless you are using --tracefile-size and --tracefile-count (man lttng-enable-channel), the limit here is
the backing filesystem of the host running lttng-relayd.

I suggest that you have a look at the overall architecture of lttng before going
further [1].

[1] https://lttng.org/docs/v2.13/#doc-plumbing

Cheers

> 
> Looking forward to your reply！
> 
> thx
> 
> Yuhua
> 
> 
> 
> 
> -----原始邮件-----
> 发件人:"Jonathan Rajotte-Julien" <jonathan.rajotte-julien@efficios.com>
> 发送时间:2022-03-09 00:39:01 (星期三)
> 收件人: "熊毓华" <xiongyuhua@zju.edu.cn>
> 抄送: lttng-dev <lttng-dev@lists.lttng.org>, "Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>, ychen@northwestern.edu
> 主题: Re: In lttng-live mode, if the printing speed cannot keep up with the generation speed of the parsed ctf data, where will the data be stored?
> 
> 
> Hi
> 
> 
> I wonder if my socket recv function is blocked on the other end, causing the socket send function to block in print_message function in babeltrace2;or when printing to the console, the printing speed can't keep up with the parsed CTF data generation speed and the print buffer is also full.
> 
> In this case, how will Babeltrace2 handle the parsed CTF data that has not been sent yet, store them in a buffer, a queue or just discard them? Or would the blocking directly cause LTTng to discard the original CTF data at the ring buffer before LTTng Consumer daemon?
> 
> 
> 
> Not sure I understand your setup but when using lttng-live, you are effectively reading from the data that lttng-relayd is collecting, piece by piece.
> The producing side is not affected by the speed at which the reader (babeltrace2) consume data from lttng-relayd using the lttng-live protocol.
> There might be some corner case here and there but for most base usage of the "live" feature reader (babeltrace2) and producer(lttng/lttng-consumerd)
> are "decoupled". You can consider the "lttng-relayd" (and the trace on the filesystem) as the "buffer" here.

-- 
Jonathan Rajotte-Julien
EfficiOS
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

[lttng-dev] Question about Open Source License

[熊毓华 via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 426 bytes --]

Hi～

I wrote a Babeltrace Plugin myself, it's sink component class.
It mainly obtains the readable log data of lttng from babeltrace, and I wrap the data into a format that I customize，then prints it out (I print the data into the socket, and send the data to other hosts through the socket). And I would like to ask what Open Source License does this plugin need to follow?

Looking forward to your reply!

thanks,

Yuhua

[-- Attachment #1.2: Type: text/html, Size: 503 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev