Re: many sockets, slow sendto

[Eric Dumazet <dada1@cosmosbay.com>]

Zacco a écrit :
> Hi,
> 
> David Miller wrote:
>> From: Zacco <zacco@fw.hu>
>> Date: Tue, 20 Mar 2007 00:10:19 +0100
>>
>>  
>>> As you recommended, I used oprofile and it turned out that the 
>>> __udp4_lib_lookup function spent most of the time. There is a udp 
>>> hash table and the sockets are sought based on the 7 LSBs of the 
>>> destination port number. So what happened is now quite obvious: I had 
>>> many thousands of sockets, all with the same destination port, thus 
>>> linked in the same slot of this hash table. I tried using different 
>>> ports and it
>>> was much faster then.
>>>     
>>
>> There isn't much we can do here.  I bet your destination address
>> is unchanging just like your destination ports.
>>   
> As I'm simulating independent users on one host, each user has a 
> different IP address, but each with the same port. So unlike the port, 
> the address is changing, basically it's a huge A-class range.
> 
>> UDP apps can and do bind to specific destination addresses and
>> ports, but the source side is usually wild-carded.
>>   
> Right, usually it is, but in my case the source addresses are also 
> bound, otherwise the source address would be the primary address of the 
> physical interface; however, I need to simulate users as if they were on 
> separate hosts.
>> Are both the source address and port fully specified for your
>> sockets?  Maybe we can do something using if that's the case...
>>   
> You made me curious.  :)  What do you have in mind?

Currently, udp_hash[UDP_HTABLE_SIZE] is using a hash function based on dport 
number only.

In your case, as you use a single port value, all sockets are in a single slot 
of this hash table :
To find the good socket, __udp4_lib_lookup() has to search in a list with 
thousands of elements. Not that good, isnt it ? :(

As udp_hash is protected by a single rw_lock, I guess we could convert the 
hash table to a RB-tree, with a key being : (dport, daddr)

At lookup time, we could do :

1) A full lookup with (dport, daddr)
2) if not found, a lookup with wildcard : (dport, 0)

I dont know if this is OK, because I dont know if it is possible to have 
several UDP sockets with the same (dport, daddr)

It would be more scalable. But still the rw_lock is not very SMP friendly...