[Drbd-dev] Protocol C

[Drbd-dev] Protocol C

[raulhp]

Hi all,

Can someone explain me how the Protocol C operate?, if the receiver 
send an ACK for every block or how ACK works. which exactly is the 
source code?

Regards,
Ra

Re: [Drbd-dev] Protocol C

[Lars Ellenberg]

On Tue, May 27, 2014 at 05:33:53PM +0200, raulhp wrote:
> Hi all,
> 
> Can someone explain me how the Protocol C operate?, if the receiver
> send an ACK for every block or how ACK works.

It basically goes like this:

local:
  submit to disk
+ send the data,
			remote: receive the data
disk completion			submit to disk
				disk completion
				send ack back
receive ack

complete to upper layers,
once both local disk completion
and remote ack have been seen.

With some more housekeeping to be able to detect failures,
and recover from them as gracefully as possible.

> which exactly is the source code?

exactly "most of it" ;-) sorry.
there is a lot going on, and all those various stages cannot be covered
with "look there, these six lines of code do it".

What are you trying to do?

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

Re: [Drbd-dev] Protocol C

[raulhp]

Hi Lars,

I´m trying to understand how ACK´s works en Protocol C to try to use it 
in Reliable Multicast in each receiver, I´m using a Reliable Multicast 
with NACK (Negative-ACKnowledgment) only in each loss packet.

Regards,
Ra

El 2014-05-28 00:40, Lars Ellenberg escribió:
> On Tue, May 27, 2014 at 05:33:53PM +0200, raulhp wrote:
>> Hi all,
>>
>> Can someone explain me how the Protocol C operate?, if the receiver
>> send an ACK for every block or how ACK works.
>
> It basically goes like this:
>
> local:
>   submit to disk
> + send the data,
> 			remote: receive the data
> disk completion			submit to disk
> 				disk completion
> 				send ack back
> receive ack
>
> complete to upper layers,
> once both local disk completion
> and remote ack have been seen.
>
> With some more housekeeping to be able to detect failures,
> and recover from them as gracefully as possible.
>
>> which exactly is the source code?
>
> exactly "most of it" ;-) sorry.
> there is a lot going on, and all those various stages cannot be 
> covered
> with "look there, these six lines of code do it".
>
> What are you trying to do?

Re: [Drbd-dev] Protocol C

[Lars Ellenberg]

On Wed, May 28, 2014 at 12:14:30PM +0200, raulhp wrote:
> Hi Lars,
> 
> I´m trying to understand how ACK´s works en Protocol C to try to use
> it in Reliable Multicast in each receiver, I´m using a Reliable
> Multicast with NACK (Negative-ACKnowledgment) only in each loss
> packet.

Ok.

The entry point for requests from "upper layers" is
drbd_make_request
  __drbd_make_request
    drbd_request_prepare()
    drbd_send_and_submit(),
    which is in drbd_req.c

Everything "interesting" that happens to such a struct drbd_request
object is handled with req_mod(req, event, peer_device)
(or _req_mod ...) which are both a wrapper around
__req_mod(),
which calls into mod_rq_state();

"events" are drbd_req_events,
drbd requests state flags are from drbd_req_state_bits (__RQ_*)
(or the corresponding RQ_* defines).

grep for req_mod should be useful.

If you consider only the replication link(s),
the interesting functions are likely
drbd_sender.c,
  drbd_sender()
    wait_for_sender_todo()
    process_sender_todo()
      process_one_request()   

drbd_receiver.c
  drbd_asender()
     ... lots of other stuff ...
     cmd = &asender_tbl[pi.cmd];
     ...
     cmd->fn(connection, &pi);
with fn (see asender_tbl) being
  got_BlockAck()
  got_NegAck()

Where you would probabaly need a "got_mcast_neg_ack()",
which could maybe better be re-labeled as "got_mcast_retransmit_request()".

You would need to find the "relevant" requests on the
  resource->transfer_log(), and queue them for resend.
(Or abstract out your own primitives from the existing stuff,
 and implement both the existing TCP links and the new mcast
 with some additional auxiliary functions as needed).

Does that help?

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

Re: [Drbd-dev] Protocol C

[raulhp]

Hi Lars,

Thanks so much, I focus only the replication link, the first develop is 
implement the replication to n servers with reliable multicast, ensuring 
the write in the remote disk.

Can you explain me this ? Where? ---> You would need to find the 
"relevant" requests on the resource->transfer_log()

Regards,
Ra

El 2014-05-28 13:56, Lars Ellenberg escribió:
> On Wed, May 28, 2014 at 12:14:30PM +0200, raulhp wrote:
>> Hi Lars,
>>
>> I´m trying to understand how ACK´s works en Protocol C to try to use
>> it in Reliable Multicast in each receiver, I´m using a Reliable
>> Multicast with NACK (Negative-ACKnowledgment) only in each loss
>> packet.
>
> Ok.
>
> The entry point for requests from "upper layers" is
> drbd_make_request
>   __drbd_make_request
>     drbd_request_prepare()
>     drbd_send_and_submit(),
>     which is in drbd_req.c
>
> Everything "interesting" that happens to such a struct drbd_request
> object is handled with req_mod(req, event, peer_device)
> (or _req_mod ...) which are both a wrapper around
> __req_mod(),
> which calls into mod_rq_state();
>
> "events" are drbd_req_events,
> drbd requests state flags are from drbd_req_state_bits (__RQ_*)
> (or the corresponding RQ_* defines).
>
> grep for req_mod should be useful.
>
> If you consider only the replication link(s),
> the interesting functions are likely
> drbd_sender.c,
>   drbd_sender()
>     wait_for_sender_todo()
>     process_sender_todo()
>       process_one_request()
>
> drbd_receiver.c
>   drbd_asender()
>      ... lots of other stuff ...
>      cmd = &asender_tbl[pi.cmd];
>      ...
>      cmd->fn(connection, &pi);
> with fn (see asender_tbl) being
>   got_BlockAck()
>   got_NegAck()
>
> Where you would probabaly need a "got_mcast_neg_ack()",
> which could maybe better be re-labeled as 
> "got_mcast_retransmit_request()".
>
> You would need to find the "relevant" requests on the
>   resource->transfer_log(), and queue them for resend.
> (Or abstract out your own primitives from the existing stuff,
>  and implement both the existing TCP links and the new mcast
>  with some additional auxiliary functions as needed).
>
> Does that help?

Re: [Drbd-dev] Protocol C

[Lars Ellenberg]

On Wed, May 28, 2014 at 02:32:05PM +0200, raulhp wrote:
> Hi Lars,
> 
> Thanks so much, I focus only the replication link, the first develop
> is implement the replication to n servers with reliable multicast,
> ensuring the write in the remote disk.
> 
> Can you explain me this ? Where? ---> You would need to find the
> "relevant" requests on the resource->transfer_log()

See in drbd_send_and_submit(), and helpers, e.g.
drbd_process_write_request,
where the requests are put on the transfer log in incoming order.

The drbd sender threads (wait_for_sender_todo, process_one_request)
find them there, and send them.

The drbd asender thread receives the acks (or, in your case
the retransmit requests), needs to validate thes acks (or neg acks,
or retrsnamit requests), and change the state of the respective request
object.

That's what happens in got_BlockAck and friends.

Now you want to implement some mcast.
I don't know what information you envison to have,
what would make you ask for retransmit,
what exactly you would ask for,
what information you could use in your got_retransmit_request function.

But the drbd_request will stay linked on the resource->transfer_log
until it is "acked", or "cleaned up" due to connection loss.

Depending on the information you have in that restransmit request,
you will be able to find the relevant requests on that transfer log.
I don't know what information you have, so I cannot tell how to best
find them.

Non-relevant are:
  those that have not been sent yet (to that peer)
  those that have already been ACKed (by that peer)
Which means that relevant are
  those that have been sent, but not acked,
  and fall in whatever "range" you request to be restransmitted.

Maybe you need to add your own lookup struct,
and trigger selective retransmit from there.

Worst case, you do a full list walk as PoC.

Give me something more specific than "how do I do multicast",
because if I answer that unspecific question, I already
implemented^Wdesigned it myself ;-)

Cheers,
	Lars

> Regards,
> Ra
> 
> El 2014-05-28 13:56, Lars Ellenberg escribió:
> >On Wed, May 28, 2014 at 12:14:30PM +0200, raulhp wrote:
> >>Hi Lars,
> >>
> >>I´m trying to understand how ACK´s works en Protocol C to try to use
> >>it in Reliable Multicast in each receiver, I´m using a Reliable
> >>Multicast with NACK (Negative-ACKnowledgment) only in each loss
> >>packet.
> >
> >Ok.
> >
> >The entry point for requests from "upper layers" is
> >drbd_make_request
> >  __drbd_make_request
> >    drbd_request_prepare()
> >    drbd_send_and_submit(),
> >    which is in drbd_req.c
> >
> >Everything "interesting" that happens to such a struct drbd_request
> >object is handled with req_mod(req, event, peer_device)
> >(or _req_mod ...) which are both a wrapper around
> >__req_mod(),
> >which calls into mod_rq_state();
> >
> >"events" are drbd_req_events,
> >drbd requests state flags are from drbd_req_state_bits (__RQ_*)
> >(or the corresponding RQ_* defines).
> >
> >grep for req_mod should be useful.
> >
> >If you consider only the replication link(s),
> >the interesting functions are likely
> >drbd_sender.c,
> >  drbd_sender()
> >    wait_for_sender_todo()
> >    process_sender_todo()
> >      process_one_request()
> >
> >drbd_receiver.c
> >  drbd_asender()
> >     ... lots of other stuff ...
> >     cmd = &asender_tbl[pi.cmd];
> >     ...
> >     cmd->fn(connection, &pi);
> >with fn (see asender_tbl) being
> >  got_BlockAck()
> >  got_NegAck()
> >
> >Where you would probabaly need a "got_mcast_neg_ack()",
> >which could maybe better be re-labeled as
> >"got_mcast_retransmit_request()".
> >
> >You would need to find the "relevant" requests on the
> >  resource->transfer_log(), and queue them for resend.
> >(Or abstract out your own primitives from the existing stuff,
> > and implement both the existing TCP links and the new mcast
> > with some additional auxiliary functions as needed).
> >
> >Does that help?

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

Re: [Drbd-dev] Protocol C

[raulhp]

El 2014-05-28 14:49, Lars Ellenberg escribió:
> On Wed, May 28, 2014 at 02:32:05PM +0200, raulhp wrote:
>> Hi Lars,
>>
>> Thanks so much, I focus only the replication link, the first develop
>> is implement the replication to n servers with reliable multicast,
>> ensuring the write in the remote disk.
>>
>> Can you explain me this ? Where? ---> You would need to find the
>> "relevant" requests on the resource->transfer_log()
>
> See in drbd_send_and_submit(), and helpers, e.g.
> drbd_process_write_request,
> where the requests are put on the transfer log in incoming order.
>
> The drbd sender threads (wait_for_sender_todo, process_one_request)
> find them there, and send them.
>
> The drbd asender thread receives the acks (or, in your case
> the retransmit requests), needs to validate thes acks (or neg acks,
> or retrsnamit requests), and change the state of the respective 
> request
> object.
>
> That's what happens in got_BlockAck and friends.
All this it´s ok!

> Now you want to implement some mcast.
> I don't know what information you envison to have,
> what would make you ask for retransmit,
> what exactly you would ask for,
> what information you could use in your got_retransmit_request 
> function.
>
I want to implement a mcast with NACK, where the receiver send NACK 
message to the sender when the receiver detects a loss packet, and the 
sender send the packet in question, with this is possible guarantee all 
writes in the remote node and reduce the traffic in the network.

> But the drbd_request will stay linked on the resource->transfer_log
> until it is "acked", or "cleaned up" due to connection loss.

It´s ok

> Depending on the information you have in that restransmit request,
> you will be able to find the relevant requests on that transfer log.
> I don't know what information you have, so I cannot tell how to best
> find them.
>
> Non-relevant are:
>   those that have not been sent yet (to that peer)
>   those that have already been ACKed (by that peer)
> Which means that relevant are
>   those that have been sent, but not acked,
>   and fall in whatever "range" you request to be restransmitted.
This is ok, but with Multicast communication is necesary to use UDP 
Protocol with Reliability.

Can you understand me?

Regards,
Ra

> Maybe you need to add your own lookup struct,
> and trigger selective retransmit from there.
>
> Worst case, you do a full list walk as PoC.
>
> Give me something more specific than "how do I do multicast",
> because if I answer that unspecific question, I already
> implemented^Wdesigned it myself ;-)
>
> Cheers,
> 	Lars
>
>> Regards,
>> Ra
>>
>> El 2014-05-28 13:56, Lars Ellenberg escribió:
>> >On Wed, May 28, 2014 at 12:14:30PM +0200, raulhp wrote:
>> >>Hi Lars,
>> >>
>> >>I´m trying to understand how ACK´s works en Protocol C to try to 
>> use
>> >>it in Reliable Multicast in each receiver, I´m using a Reliable
>> >>Multicast with NACK (Negative-ACKnowledgment) only in each loss
>> >>packet.
>> >
>> >Ok.
>> >
>> >The entry point for requests from "upper layers" is
>> >drbd_make_request
>> >  __drbd_make_request
>> >    drbd_request_prepare()
>> >    drbd_send_and_submit(),
>> >    which is in drbd_req.c
>> >
>> >Everything "interesting" that happens to such a struct drbd_request
>> >object is handled with req_mod(req, event, peer_device)
>> >(or _req_mod ...) which are both a wrapper around
>> >__req_mod(),
>> >which calls into mod_rq_state();
>> >
>> >"events" are drbd_req_events,
>> >drbd requests state flags are from drbd_req_state_bits (__RQ_*)
>> >(or the corresponding RQ_* defines).
>> >
>> >grep for req_mod should be useful.
>> >
>> >If you consider only the replication link(s),
>> >the interesting functions are likely
>> >drbd_sender.c,
>> >  drbd_sender()
>> >    wait_for_sender_todo()
>> >    process_sender_todo()
>> >      process_one_request()
>> >
>> >drbd_receiver.c
>> >  drbd_asender()
>> >     ... lots of other stuff ...
>> >     cmd = &asender_tbl[pi.cmd];
>> >     ...
>> >     cmd->fn(connection, &pi);
>> >with fn (see asender_tbl) being
>> >  got_BlockAck()
>> >  got_NegAck()
>> >
>> >Where you would probabaly need a "got_mcast_neg_ack()",
>> >which could maybe better be re-labeled as
>> >"got_mcast_retransmit_request()".
>> >
>> >You would need to find the "relevant" requests on the
>> >  resource->transfer_log(), and queue them for resend.
>> >(Or abstract out your own primitives from the existing stuff,
>> > and implement both the existing TCP links and the new mcast
>> > with some additional auxiliary functions as needed).
>> >
>> >Does that help?

Re: [Drbd-dev] Protocol C

[Lars Ellenberg]

On Wed, May 28, 2014 at 05:25:28PM +0200, raulhp wrote:
> 
> 
> El 2014-05-28 14:49, Lars Ellenberg escribió:
> >On Wed, May 28, 2014 at 02:32:05PM +0200, raulhp wrote:
> >>Hi Lars,
> >>
> >>Thanks so much, I focus only the replication link, the first develop
> >>is implement the replication to n servers with reliable multicast,
> >>ensuring the write in the remote disk.
> >>
> >>Can you explain me this ? Where? ---> You would need to find the
> >>"relevant" requests on the resource->transfer_log()
> >
> >See in drbd_send_and_submit(), and helpers, e.g.
> >drbd_process_write_request,
> >where the requests are put on the transfer log in incoming order.
> >
> >The drbd sender threads (wait_for_sender_todo, process_one_request)
> >find them there, and send them.
> >
> >The drbd asender thread receives the acks (or, in your case
> >the retransmit requests), needs to validate thes acks (or neg acks,
> >or retrsnamit requests), and change the state of the respective
> >request
> >object.
> >
> >That's what happens in got_BlockAck and friends.
> All this it´s ok!
> 
> >Now you want to implement some mcast.
> >I don't know what information you envison to have,
> >what would make you ask for retransmit,
> >what exactly you would ask for,
> >what information you could use in your got_retransmit_request
> >function.
> >
> I want to implement a mcast with NACK, where the receiver send NACK
> message to the sender when the receiver detects a loss packet, and
> the sender send the packet in question, with this is possible
> guarantee all writes in the remote node and reduce the traffic in
> the network.
> 
> >But the drbd_request will stay linked on the resource->transfer_log
> >until it is "acked", or "cleaned up" due to connection loss.
> 
> It´s ok
> 
> >Depending on the information you have in that restransmit request,
> >you will be able to find the relevant requests on that transfer log.
> >I don't know what information you have, so I cannot tell how to best
> >find them.
> >
> >Non-relevant are:
> >  those that have not been sent yet (to that peer)
> >  those that have already been ACKed (by that peer)
> >Which means that relevant are
> >  those that have been sent, but not acked,
> >  and fall in whatever "range" you request to be restransmitted.
> This is ok, but with Multicast communication is necesary to use UDP
> Protocol with Reliability.

Of course.
So?

We already have a "NACK", in the DRBD protocol,
which carries the meaning "failed to serve this request"
(the IO subsystem of the peer reported failure).

Lets not overload the term "nack",
and use the term "retransmit request",
because that is basically what a "nack"
in any reliable multicast protocol is supposed to trigger: 
retransmit of not received messages.

The receiver notices (by timeouts and sequence numbers and other
conditions) that it missed messages, and sends back that it'd like
to please have "these" messages again.

Maybe it is useful to have both:
the "multicast" channel,
and still the tcp mesh channels for retransmits,
and maybe for read requests or other directed messages.
(just an idea)

The node that originally sent the requests
now sees this retransmit request,
and needs to process it.

To do that, it needs to find the original request,
and resend it.

Luckily we have all requests
  * on the transfer log
     (struct drbd_resource: transfer_log)
  * and in the interval tree
     (see validate_req_change_req_state
      and find_request).

So based on what information you put in your retransmit request
you should be able to find the original either on the transfer log
or in the interval tree, and resend it.

Am I misunderstanding your question?

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

Re: [Drbd-dev] Protocol C

[raulhp]

El 2014-05-28 20:36, Lars Ellenberg escribió:
> On Wed, May 28, 2014 at 05:25:28PM +0200, raulhp wrote:
>>
>>
>> El 2014-05-28 14:49, Lars Ellenberg escribió:
>> >On Wed, May 28, 2014 at 02:32:05PM +0200, raulhp wrote:
>> >>Hi Lars,
>> >>
>> >>Thanks so much, I focus only the replication link, the first 
>> develop
>> >>is implement the replication to n servers with reliable multicast,
>> >>ensuring the write in the remote disk.
>> >>
>> >>Can you explain me this ? Where? ---> You would need to find the
>> >>"relevant" requests on the resource->transfer_log()
>> >
>> >See in drbd_send_and_submit(), and helpers, e.g.
>> >drbd_process_write_request,
>> >where the requests are put on the transfer log in incoming order.
>> >
>> >The drbd sender threads (wait_for_sender_todo, process_one_request)
>> >find them there, and send them.
>> >
>> >The drbd asender thread receives the acks (or, in your case
>> >the retransmit requests), needs to validate thes acks (or neg acks,
>> >or retrsnamit requests), and change the state of the respective
>> >request
>> >object.
>> >
>> >That's what happens in got_BlockAck and friends.
>> All this it´s ok!
>>
>> >Now you want to implement some mcast.
>> >I don't know what information you envison to have,
>> >what would make you ask for retransmit,
>> >what exactly you would ask for,
>> >what information you could use in your got_retransmit_request
>> >function.
>> >
>> I want to implement a mcast with NACK, where the receiver send NACK
>> message to the sender when the receiver detects a loss packet, and
>> the sender send the packet in question, with this is possible
>> guarantee all writes in the remote node and reduce the traffic in
>> the network.
>>
>> >But the drbd_request will stay linked on the resource->transfer_log
>> >until it is "acked", or "cleaned up" due to connection loss.
>>
>> It´s ok
>>
>> >Depending on the information you have in that restransmit request,
>> >you will be able to find the relevant requests on that transfer 
>> log.
>> >I don't know what information you have, so I cannot tell how to 
>> best
>> >find them.
>> >
>> >Non-relevant are:
>> >  those that have not been sent yet (to that peer)
>> >  those that have already been ACKed (by that peer)
>> >Which means that relevant are
>> >  those that have been sent, but not acked,
>> >  and fall in whatever "range" you request to be restransmitted.
>> This is ok, but with Multicast communication is necesary to use UDP
>> Protocol with Reliability.
>
> Of course.
> So?
>
> We already have a "NACK", in the DRBD protocol,
> which carries the meaning "failed to serve this request"
> (the IO subsystem of the peer reported failure).

The NACK in te DRBD protocol is used in Protocol C? or?

> Lets not overload the term "nack",
> and use the term "retransmit request",
> because that is basically what a "nack"
> in any reliable multicast protocol is supposed to trigger:
> retransmit of not received messages.

Yes, the term "NACK" works well

> The receiver notices (by timeouts and sequence numbers and other
> conditions) that it missed messages, and sends back that it'd like
> to please have "these" messages again.
>
> Maybe it is useful to have both:
> the "multicast" channel,
> and still the tcp mesh channels for retransmits,
> and maybe for read requests or other directed messages.
> (just an idea)

The idea is to use multicast channel to send the data replication and 
TCP mesh channels for retransmits and keep the connection resource , or 
if the packet loss is large using the multicast channel.

> The node that originally sent the requests
> now sees this retransmit request,
> and needs to process it.
>
> To do that, it needs to find the original request,
> and resend it.
>
> Luckily we have all requests
>   * on the transfer log
>      (struct drbd_resource: transfer_log)
>   * and in the interval tree
>      (see validate_req_change_req_state
>       and find_request).
>
> So based on what information you put in your retransmit request
> you should be able to find the original either on the transfer log
> or in the interval tree, and resend it.

It´s OK

> Am I misunderstanding your question?
You understand my question :-) only y want to understand how to use 
existing code of ACK for multicast

Re: [Drbd-dev] Protocol C

[Lars Ellenberg]

On Thu, May 29, 2014 at 12:45:32PM +0200, raulhp wrote:
> 
> 
> El 2014-05-28 20:36, Lars Ellenberg escribió:
> >On Wed, May 28, 2014 at 05:25:28PM +0200, raulhp wrote:
> >>
> >>
> >>El 2014-05-28 14:49, Lars Ellenberg escribió:
> >>>On Wed, May 28, 2014 at 02:32:05PM +0200, raulhp wrote:
> >>>>Hi Lars,
> >>>>
> >>>>Thanks so much, I focus only the replication link, the first
> >>develop
> >>>>is implement the replication to n servers with reliable multicast,
> >>>>ensuring the write in the remote disk.
> >>>>
> >>>>Can you explain me this ? Where? ---> You would need to find the
> >>>>"relevant" requests on the resource->transfer_log()
> >>>
> >>>See in drbd_send_and_submit(), and helpers, e.g.
> >>>drbd_process_write_request,
> >>>where the requests are put on the transfer log in incoming order.
> >>>
> >>>The drbd sender threads (wait_for_sender_todo, process_one_request)
> >>>find them there, and send them.
> >>>
> >>>The drbd asender thread receives the acks (or, in your case
> >>>the retransmit requests), needs to validate thes acks (or neg acks,
> >>>or retrsnamit requests), and change the state of the respective
> >>>request
> >>>object.
> >>>
> >>>That's what happens in got_BlockAck and friends.
> >>All this it´s ok!
> >>
> >>>Now you want to implement some mcast.
> >>>I don't know what information you envison to have,
> >>>what would make you ask for retransmit,
> >>>what exactly you would ask for,
> >>>what information you could use in your got_retransmit_request
> >>>function.
> >>>
> >>I want to implement a mcast with NACK, where the receiver send NACK
> >>message to the sender when the receiver detects a loss packet, and
> >>the sender send the packet in question, with this is possible
> >>guarantee all writes in the remote node and reduce the traffic in
> >>the network.
> >>
> >>>But the drbd_request will stay linked on the resource->transfer_log
> >>>until it is "acked", or "cleaned up" due to connection loss.
> >>
> >>It´s ok
> >>
> >>>Depending on the information you have in that restransmit request,
> >>>you will be able to find the relevant requests on that transfer
> >>log.
> >>>I don't know what information you have, so I cannot tell how to
> >>best
> >>>find them.
> >>>
> >>>Non-relevant are:
> >>>  those that have not been sent yet (to that peer)
> >>>  those that have already been ACKed (by that peer)
> >>>Which means that relevant are
> >>>  those that have been sent, but not acked,
> >>>  and fall in whatever "range" you request to be restransmitted.
> >>This is ok, but with Multicast communication is necesary to use UDP
> >>Protocol with Reliability.
> >
> >Of course.
> >So?
> >
> >We already have a "NACK", in the DRBD protocol,
> >which carries the meaning "failed to serve this request"
> >(the IO subsystem of the peer reported failure).
> 
> The NACK in te DRBD protocol is used in Protocol C? or?

Always.

You do realize that there is
  * the transport protocol (TCP)
  * the DRBD protocol

*BOTH*
tcp and DRBD-protocol have "acks" and "nacks",
and they mean different things.

DRBD ACKs (actually: P_WRITE_ACK in protocol C, for other variants see
enum drbd_packet) mean that the corresponding request has been completed
by the remote disk.

DRBD NACKs (P_NEG_ACK) mean that the remote disk was not able to process
the request, or tried and failed (IO-error).

On the TCP level, the TCP acks and stuff has the TCP meaning.

With a "reliable multicast transport" for DRBD,
you will need transport specific messages.
the DRBD protocol on top of that won't care much,
as long as it eventually gets the P_WRITE_ACK
or P_NEG_ACK.

> >Lets not overload the term "nack",
> >and use the term "retransmit request",
> >because that is basically what a "nack"
> >in any reliable multicast protocol is supposed to trigger:
> >retransmit of not received messages.
> 
> Yes, the term "NACK" works well

I just tried to explain why the term NACK does NOT work,
and why I suggest to use retransmit request instead :-/

> >Am I misunderstanding your question?
> You understand my question :-) only y want to understand how to use
> existing code of ACK for multicast

I don't think it works that way.

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

Re: [Drbd-dev] Protocol C

[raulhp]

El 2014-05-29 23:29, Lars Ellenberg escribió:
> On Thu, May 29, 2014 at 12:45:32PM +0200, raulhp wrote:
>>
>>
>> El 2014-05-28 20:36, Lars Ellenberg escribió:
>> >On Wed, May 28, 2014 at 05:25:28PM +0200, raulhp wrote:
>> >>
>> >>
>> >>El 2014-05-28 14:49, Lars Ellenberg escribió:
>> >>>On Wed, May 28, 2014 at 02:32:05PM +0200, raulhp wrote:
>> >>>>Hi Lars,
>> >>>>
>> >>>>Thanks so much, I focus only the replication link, the first
>> >>develop
>> >>>>is implement the replication to n servers with reliable 
>> multicast,
>> >>>>ensuring the write in the remote disk.
>> >>>>
>> >>>>Can you explain me this ? Where? ---> You would need to find the
>> >>>>"relevant" requests on the resource->transfer_log()
>> >>>
>> >>>See in drbd_send_and_submit(), and helpers, e.g.
>> >>>drbd_process_write_request,
>> >>>where the requests are put on the transfer log in incoming order.
>> >>>
>> >>>The drbd sender threads (wait_for_sender_todo, 
>> process_one_request)
>> >>>find them there, and send them.
>> >>>
>> >>>The drbd asender thread receives the acks (or, in your case
>> >>>the retransmit requests), needs to validate thes acks (or neg 
>> acks,
>> >>>or retrsnamit requests), and change the state of the respective
>> >>>request
>> >>>object.
>> >>>
>> >>>That's what happens in got_BlockAck and friends.
>> >>All this it´s ok!
>> >>
>> >>>Now you want to implement some mcast.
>> >>>I don't know what information you envison to have,
>> >>>what would make you ask for retransmit,
>> >>>what exactly you would ask for,
>> >>>what information you could use in your got_retransmit_request
>> >>>function.
>> >>>
>> >>I want to implement a mcast with NACK, where the receiver send 
>> NACK
>> >>message to the sender when the receiver detects a loss packet, and
>> >>the sender send the packet in question, with this is possible
>> >>guarantee all writes in the remote node and reduce the traffic in
>> >>the network.
>> >>
>> >>>But the drbd_request will stay linked on the 
>> resource->transfer_log
>> >>>until it is "acked", or "cleaned up" due to connection loss.
>> >>
>> >>It´s ok
>> >>
>> >>>Depending on the information you have in that restransmit 
>> request,
>> >>>you will be able to find the relevant requests on that transfer
>> >>log.
>> >>>I don't know what information you have, so I cannot tell how to
>> >>best
>> >>>find them.
>> >>>
>> >>>Non-relevant are:
>> >>>  those that have not been sent yet (to that peer)
>> >>>  those that have already been ACKed (by that peer)
>> >>>Which means that relevant are
>> >>>  those that have been sent, but not acked,
>> >>>  and fall in whatever "range" you request to be restransmitted.
>> >>This is ok, but with Multicast communication is necesary to use 
>> UDP
>> >>Protocol with Reliability.
>> >
>> >Of course.
>> >So?
>> >
>> >We already have a "NACK", in the DRBD protocol,
>> >which carries the meaning "failed to serve this request"
>> >(the IO subsystem of the peer reported failure).
>>
>> The NACK in te DRBD protocol is used in Protocol C? or?
>
> Always.
>
> You do realize that there is
>   * the transport protocol (TCP)
>   * the DRBD protocol
>
Ok!
> *BOTH*
> tcp and DRBD-protocol have "acks" and "nacks",
> and they mean different things.
>
Ok!
> DRBD ACKs (actually: P_WRITE_ACK in protocol C, for other variants 
> see
> enum drbd_packet) mean that the corresponding request has been 
> completed
> by the remote disk.
>
> DRBD NACKs (P_NEG_ACK) mean that the remote disk was not able to 
> process
> the request, or tried and failed (IO-error).
>
Yes, I saw the enum drbd_packet
> On the TCP level, the TCP acks and stuff has the TCP meaning.
>
> With a "reliable multicast transport" for DRBD,
> you will need transport specific messages.
> the DRBD protocol on top of that won't care much,
> as long as it eventually gets the P_WRITE_ACK
> or P_NEG_ACK.
>
Yes, just wanted to take " NACKS " that there are in the source code.

I think put a reliable multicast transport in
drbd_main.c
     drbd_send_dblock () (I saw that is a function to send data of 
Primary to Secondary)

drbd_receiver.c
     receive_Data() (I saw that is a function to receive data)

or, in the low level

drbd_main.c
     drbd_send()
        kernel_sendmsg()

drbd_receiver.c
     drbd_recv_short()
     sock_recvmsg()

>> >Lets not overload the term "nack",
>> >and use the term "retransmit request",
>> >because that is basically what a "nack"
>> >in any reliable multicast protocol is supposed to trigger:
>> >retransmit of not received messages.
>>
>> Yes, the term "NACK" works well
>
> I just tried to explain why the term NACK does NOT work,
> and why I suggest to use retransmit request instead :-/
>
In the nack term is necesary include the retransmit request that is not 
receive in the secondary nodes

>> >Am I misunderstanding your question?
>> You understand my question :-) only y want to understand how to use
>> existing code of ACK for multicast
>
> I don't think it works that way.

Now, What do you think?

Re: [Drbd-dev] Protocol C

[Lars Ellenberg]

On Fri, May 30, 2014 at 12:28:19PM +0200, raulhp wrote:
> >>>Am I misunderstanding your question?
> >>You understand my question :-) only y want to understand how to use
> >>existing code of ACK for multicast
> >
> >I don't think it works that way.
> 
> Now, What do you think?

I think, amoung other things, we have some a nice language barrier here.

I think, as I tried to explain,
that you need to distinguish between what happens
on the DRBD protocol level, and what happens
on the communication transport level.
The latter is what you are working on.
You can do almost whatever you want there.

But it should be obvious that you cannot re-use the
DRBD protocol level "my-disk-has-failed-I-am-so-sorry" message
to mean, on the transport layer,
"please-resend-all-requests-I-did-miss", just because both
concepts use the term "nack" somewhere in the description.

Different things.
Both needed.
Not directly related.


-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

Re: [Drbd-dev] Protocol C

[raulhp]

It´s correct, Thanks Lars,

Regards,
Ra

El 2014-05-30 15:17, Lars Ellenberg escribió:
> On Fri, May 30, 2014 at 12:28:19PM +0200, raulhp wrote:
>> >>>Am I misunderstanding your question?
>> >>You understand my question :-) only y want to understand how to 
>> use
>> >>existing code of ACK for multicast
>> >
>> >I don't think it works that way.
>>
>> Now, What do you think?
>
> I think, amoung other things, we have some a nice language barrier 
> here.
>
> I think, as I tried to explain,
> that you need to distinguish between what happens
> on the DRBD protocol level, and what happens
> on the communication transport level.
> The latter is what you are working on.
> You can do almost whatever you want there.
>
> But it should be obvious that you cannot re-use the
> DRBD protocol level "my-disk-has-failed-I-am-so-sorry" message
> to mean, on the transport layer,
> "please-resend-all-requests-I-did-miss", just because both
> concepts use the term "nack" somewhere in the description.
>
> Different things.
> Both needed.
> Not directly related.