Performance of netns with sysfs

Performance of netns with sysfs

[Dan Smith]

There has been some discussion lately about network namespaces and the
interaction with sysfs.  With the introduction of Guenter's ipcgroup
patches, there was also the question of how feasible it is to create
thousands of network namespaces.  So, I decided to see if I could even
create thousands of veth pairs, and if so move them into thousands of
network namespaces.  I was pleased to see that the system didn't fall
over, but found that the process slowed significantly with higher
numbers if sysfs was enabled.  I thought it would be prudent to post
some numbers.

I first tested creating 1000 and 2500 veth pairs, attaching one side
to a bridge with and without sysfs.  Next I created 2500 network
namespaces, along with 2500 veth pairs.  One side of each pair was
attached to a bridge and the other was moved into the namespace.  The
results are:

  1000 veth pairs:        8x slower with CONFIG_SYSFS
  2500 veth pairs         4.5x slower
  2500 netns, veth pairs: 6x slower

The tests were done with hal disabled, attaching every third veth
device to a different bridge (to overcome the limit of 1023 taps per
bridge).

-- 
Dan Smith
IBM Linux Technology Center
email: danms-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org

Re: Performance of netns with sysfs

[Guenter Roeck]

There are two key elements affecting performance when creating large
numbers of interfaces:
- full_name_hash() doesn't do a good job in selecting hash buckets
- sysfs uses a linked list to store directory entries, and strcmp() to
compare entries.

The sysfs problem may be solved with Eric's sysfs upcoming changes; if
not, there are (at least) two ways to improve sysfs performance: 
1) store a name hash with each entry, and compare the hash result before
comparing the entire name
2) create a per-directory hash table.

full_name_hash() can be improved by replacing the hash, for example as
follows. 
Old:
	return (prevhash + (c << 4) + (c >> 4)) * 11;
New:
	return (prevhash + c) * 41;

Guenter

On Thu, 2009-01-15 at 12:57 -0800, Dan Smith wrote:
> There has been some discussion lately about network namespaces and the
> interaction with sysfs.  With the introduction of Guenter's ipcgroup
> patches, there was also the question of how feasible it is to create
> thousands of network namespaces.  So, I decided to see if I could even
> create thousands of veth pairs, and if so move them into thousands of
> network namespaces.  I was pleased to see that the system didn't fall
> over, but found that the process slowed significantly with higher
> numbers if sysfs was enabled.  I thought it would be prudent to post
> some numbers.
> 
> I first tested creating 1000 and 2500 veth pairs, attaching one side
> to a bridge with and without sysfs.  Next I created 2500 network
> namespaces, along with 2500 veth pairs.  One side of each pair was
> attached to a bridge and the other was moved into the namespace.  The
> results are:
> 
>   1000 veth pairs:        8x slower with CONFIG_SYSFS
>   2500 veth pairs         4.5x slower
>   2500 netns, veth pairs: 6x slower
> 
> The tests were done with hal disabled, attaching every third veth
> device to a different bridge (to overcome the limit of 1023 taps per
> bridge).
> 
> --
> Dan Smith
> IBM Linux Technology Center
> email: danms-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org
> 

Re: Performance of netns with sysfs

[Dan Smith]

GR> The sysfs problem may be solved with Eric's sysfs upcoming
GR> changes;

Yep, I'll re-run these with Eric's set when it's available.

-- 
Dan Smith
IBM Linux Technology Center
email: danms-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org

Re: Performance of netns with sysfs

[Eric W. Biederman]

Guenter Roeck <groeck-gvzKVTG1yJJBDgjK7y7TUQ@public.gmane.org> writes:

> There are two key elements affecting performance when creating large
> numbers of interfaces:
> - full_name_hash() doesn't do a good job in selecting hash buckets
> - sysfs uses a linked list to store directory entries, and strcmp() to
> compare entries.
>
> The sysfs problem may be solved with Eric's sysfs upcoming changes; if
> not, there are (at least) two ways to improve sysfs performance: 

I have not addressed the slowdown problem.  But it is good to hear
that it exists and that people care.  I thought sysfs was going to
slow things down.

My immediate goal is to reduce the locking complexity to that found
within /proc.

> 1) store a name hash with each entry, and compare the hash result before
> comparing the entire name
> 2) create a per-directory hash table.
>
> full_name_hash() can be improved by replacing the hash, for example as
> follows. 
> Old:
> 	return (prevhash + (c << 4) + (c >> 4)) * 11;
> New:
> 	return (prevhash + c) * 41;

At least with the sanity checking turned on.  sysctl will also have
slowdowns that are more significant than sysfs when creating network
devices.  So that should be addressed as well.

Eric