Re: [PATCH 1/1] tidbits: net-udp: solicit new client for server mode

[Philippe Gerum <rpm@xenomai.org>]

Hannes Diethelm <hannes.diethelm@gmail.com> writes:

> Am 01.06.26 um 11:10 schrieb Philippe Gerum:
>> Philippe Gerum <rpm@xenomai.org> writes:
>> 
>>> Hannes Diethelm <hannes.diethelm@gmail.com> writes:
>>>
>>>> This fixes random EINPROGRESS at init or during runtime.
>>>>
>>>> Also correct whitespaces and make stdout more consistent.
>>>>
>>>> Signed-off-by: Hannes Diethelm <hannes.diethelm@gmail.com>
>>>> ---
>>>>   tidbits/oob-net-udp.c | 58 +++++++++++++++++++++++++++++++++++++------
>>>>   1 file changed, 50 insertions(+), 8 deletions(-)
>>>>
>>>> diff --git a/tidbits/oob-net-udp.c b/tidbits/oob-net-udp.c
>>>> index 11b8459..7af834f 100644
>>>> --- a/tidbits/oob-net-udp.c
>>>> +++ b/tidbits/oob-net-udp.c
>>>> @@ -50,12 +50,12 @@ static void usage(void)
>>>>   }
>>>>     static void print_addr(char* text, struct sockaddr_in *addr){
>>>> -    char ip_str[INET_ADDRSTRLEN+1];
>>>> -    inet_ntop(AF_INET, &(addr->sin_addr), ip_str, sizeof(ip_str));
>>>> -    evl_printf("%s--------\n", text);
>>>> -    evl_printf("IP-Address: %s\n", ip_str);
>>>> -    evl_printf("Port:       %d\n", ntohs(addr->sin_port));
>>>> -    evl_printf("Family:     %d\n", addr->sin_family);
>>>> +	char ip_str[INET_ADDRSTRLEN+1];
>>>> +	inet_ntop(AF_INET, &(addr->sin_addr), ip_str, sizeof(ip_str));
>>>> +	evl_printf("== %s\n", text);
>>>> +	evl_printf("   ip-address: %s\n", ip_str);
>>>> +	evl_printf("   port:       %d\n", ntohs(addr->sin_port));
>>>> +	evl_printf("   family:     %d\n", addr->sin_family);
>>>>   }
>>>>     static void sender(int s, const char *text, int mcount,
>>>> @@ -226,6 +226,8 @@ static void client(int s, const char *text, int mcount,
>>>>   	free(tbuf);
>>>>   }
>>>>   +#define SERVER_ADDR_LIST_SIZE 64
>>>> +
>>>>   static void server(int s, const char *text, int mcount,
>>>>   		struct sockaddr_in *addr, int iter)
>>>>   {
>>>> @@ -236,6 +238,9 @@ static void server(int s, const char *text, int mcount,
>>>>   	ssize_t ret;
>>>>   	char *tbuf;
>>>>   	char rbuf[16384];
>>>> +	in_addr_t addr_list[SERVER_ADDR_LIST_SIZE]={};
>>>                                                    ^ missing whitespaces
>>>> +	size_t addr_list_fill=0;
>>>> +	bool solicit_done;
>>>>     	tlen = (strlen(text) + 1) * mcount;
>>>>   	tbuf = malloc(tlen);
>>>> @@ -276,6 +281,39 @@ static void server(int s, const char *text, int mcount,
>>>>   			evl_printf(" (TRUNCATED)");
>>>>   		evl_printf(": %.*s\n", (int)ret, rbuf);
>>>>   +		/*
>>>> +		 * We need to call evl_net_solicit for each new
>>>> +		 * client once before sending data. This will break
>>>> +		 * realtime for the first response.
>>>> +		 * If this is not done and the ARP address is not
>>>> +		 * yet in cache or garbage-collected, oob_sendmsg
>>>> +		 * will return EINPROGRESS on start or during runtime.
>>>> +		 */
>>>
>>> We can happily send redundant solicit requests to the core, no need to
>>> filter out cached addresses, evl_net_solicit() will do the right thing.
>>>
>> Except that doing so would always demote the caller to the in-band
>> stage, which may not be what you want. The fact that we'd need to
>> maintain a cache in apps in order to figure out whether solicitation
>> should be done shows a shortcoming in the core, users should not have to
>> do this dance.
>> We should have an oob call for probing the route+arp caches with
>> ipv4. I'll look into this asap.
>> 
>
> Yes, this is the reason I do it this way and print out a message if
> this happens.
>
> I think server mode with multiple clients in real time is not something you
> can just easily do without careful considerations what happens when multiple
> clients send a message at the same time or TDMA behind. My intent in this example
> is mostly to show that it is possible.
>
> A real application would need one of these options:
> - A thread only for solicitation so the main thread doesn't get blocked
> - evl_net_solicit() with a flag like MSG_DONTWAIT and then poll in the main loop if
>   the client is ready
> - Know the clients in advance
> - A warm up phase during clients can connect
> - ...
>
> However, a way for probing and reading the arp would shurely help.

Here is a general proposal to deal with the issue of sending UDP packets
to unplanned destinations, which is basically what the server mode has
to do, when receiving unsolicited messages.

We have a single invariant to cope with: the evl network stack wants to
use the routing+arp information produced by the regular in-band stack,
so that we can benefit from potentially complex routing logic without
reinventing that wheel. Therefore, the route to any new/unsolicited
destination must be resolved by the in-band stack once so that we can
send UDP packets to that peer from the oob stage next.

From the point above, and as you pointed out already, an oob sender must
either plan for reaching a particular destination (e.g. soliciting it
prior to entering time-critical code), or accept an initial delay for
the route to be resolved if the destination is not yet known on first
transmit (i.e. out of the oob caches).

So, I would say that evl could meet the requirements you stated above by
implementing the following:

- Honor MSG_PROBE for oob_sendmsg(), so that only the general call
  sanity and route resolution to the destination host is performed when
  set in the request flags, without actually sending any data. On
  success of such call, we would know that the routing information is
  readily available from the oob caches, no offload to in-band would
  have happened if we had not given this flag. The absence of routing
  information to the destination from some oob cache would yield a
  specific error, so that the caller may decide what to do next.

- Extend the effect of receiving MSG_DONTWAIT (and more generally
  O_NONBLOCK on fildes) to what we would do upon missing routing
  information: if present, return with a specific error code _without_
  relaying the packet to the in-band stack. The caller may then decide
  to handle the case locally. Otherwise, proceed as usual (i.e. relay to
  the in-band stack, then notify the caller with -EINPROGRESS).

- Provide a way to synchronize with the route resolution process in evl,
  i.e. a syscall that would block until a given destination is available
  from the oob cache. This call already exists, evl_net_solicit() can be
  used for that purpose (if a resolution request is already in flight,
  subsequent ones to the same destination won't cause any harm).

Points 1 and 2 are implemented in [1]. Feedback greatly appreciated and
important when you have time. Usability for real-world applications is
key.

> It would also be nice to have an "evl net" command to list the
> entry's. I tried the normal arp command but it doesn't behave nicely
> with evl.

Could you please elaborate on this issue? Currently, updates to the
in-band cache are propagated to the oob cache (by registering a hook
into the relevant notifier chain in the kernel). So I would expect all
destinations of interest to oob which are visible from the in-band arp
cache to be available from the evl map as well.

This said, I agree that we need a way to inspect the oob cache
specifically. Working on it.

[1] https://gitlab.com/Xenomai/xenomai4/linux-evl/-/tree/wip/net-solicit?ref_type=heads

-- 
Philippe.