X15 alpha release: as fast as TUX but in user space

X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Fri, 27 Apr 2001, Fabio Riccardi wrote:

> I'd like to announce the first release of X15 Alpha 1, a _user space_
> web server that is as fast as TUX.

great, the first TUX clone! ;-)

This should put the accusations to rest that Linux got the outstandingly
high SPECweb99 scores only because the webserver was in kernel-space. It's
the 2.4 kernel's high performance that enabled those results, having the
web-server in kernel-space didnt have much effect. TUX was and remains a
testbed to test high-performance webserving (and FTP serving), without the
API-exporting overhead of userspace.

[i suspect the small performance advantage of X15 is due to subtle
differences in the SPECweb99 user-space module: eg. while the TUX code was
written, tested and ready to use mmap()-enabled
TUXAPI_alloc_read_objectbuf(), it wasnt enabled actually. I sent Fabio a
mail how to enable it, perhaps he can do some tests to confirm this
suspicion?]

doing a TUX 2.0 SPECweb99 benchmark on the latest -ac kernels, 86% of time
is spent in generic parts of the kernel, 12% of time is spent in the
user-space SPECweb99 module, and only 2% of time is spent in TUX-specific
kernel code.

doing the same test with the original TUX 1.0 code shows that more than
50% of CPU time was spent in TUX-specific code.

what does this mean? In the roughly 6 months since TUX 1.0 was released,
we moved much of the TUX 1.0 -only improvements into the generic kernel
(most of which was made available to user-space as well), and TUX itself
became smaller and smaller (and used more and more generic parts of the
kernel). So in effect X15 is executing 50% TUX code :-)

(there are still a number of performance improvement patches pending that
are not integrated yet: the pagecache extreme-scalability patch and the
smptimers patch. These patches speed both X15 and TUX up.)

(there is one thing though that can never be 'exported to user-space': to
isolate possibly untrusted binary application code from the server itself,
without performance degradation. So we always have to be mentally open to
the validity of kernel-space services.)

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

Linux 2.4 is surely one of the most advanced OSs ever happened, especially
from the optimization point of view and for the admirable economy of concepts
on which it lies. I definitively hope that X15 helps reinforcing the success
to this amazing system.

TUX has definitively been my performance yardstick for the development of
X15, but I had many sources of inspiration for the X15 architecture. Maybe
the most relevant are the Flash Web Server (Pai, Druschel, Zwaenepoel),
several Linus observations on this list about (web) server architecture and
kernnel services, and the reading of the Hennessy & Patterson architecture
books. Last but not least, aside from some heated discussions, research in
microkernel architecture has taught us many lessons on how to achieve an
efficient model of interaction across separate addressing spaces.

If i have to make some sort of educated guess and point at where the current
bottleneck lies for web server performance, I would say that it is somewhere
between the memory subsystem and the PCI bus.

With zero-copy sendfile data movement is not an issue anymore, asynchronous
network IO allows for really inexpensive thread scheduling, and system call
invocation adds a very negligible overhead in Linux. What we are left with
now is purely wait cycles, the CPUs and the NICs are contending for memory
and bus bandwidth. It would be really interesting to see where the network
shifts now that faster machines are becoming available.

On my whish list for future kernel developments I would definitively put disk
asynchronous IO and a more decent file descriptor passing implementation.
I'll detail this in subsequent messages.

I'll surely check out the impact of Ingo's patches on TUX performance
sometime this week.

I'd also like to reiterate my request for help for testing X15 on higher end
server architectures.

X15 is still very young alpha code and I can surely improve its performance
in many ways.

 - Fabio

Ingo Molnar wrote:

> On Fri, 27 Apr 2001, Fabio Riccardi wrote:
>
> > I'd like to announce the first release of X15 Alpha 1, a _user space_
> > web server that is as fast as TUX.
>
> great, the first TUX clone! ;-)
>
> This should put the accusations to rest that Linux got the outstandingly
> high SPECweb99 scores only because the webserver was in kernel-space. It's
> the 2.4 kernel's high performance that enabled those results, having the
> web-server in kernel-space didnt have much effect. TUX was and remains a
> testbed to test high-performance webserving (and FTP serving), without the
> API-exporting overhead of userspace.
>
> [i suspect the small performance advantage of X15 is due to subtle
> differences in the SPECweb99 user-space module: eg. while the TUX code was
> written, tested and ready to use mmap()-enabled
> TUXAPI_alloc_read_objectbuf(), it wasnt enabled actually. I sent Fabio a
> mail how to enable it, perhaps he can do some tests to confirm this
> suspicion?]
>
> doing a TUX 2.0 SPECweb99 benchmark on the latest -ac kernels, 86% of time
> is spent in generic parts of the kernel, 12% of time is spent in the
> user-space SPECweb99 module, and only 2% of time is spent in TUX-specific
> kernel code.
>
> doing the same test with the original TUX 1.0 code shows that more than
> 50% of CPU time was spent in TUX-specific code.
>
> what does this mean? In the roughly 6 months since TUX 1.0 was released,
> we moved much of the TUX 1.0 -only improvements into the generic kernel
> (most of which was made available to user-space as well), and TUX itself
> became smaller and smaller (and used more and more generic parts of the
> kernel). So in effect X15 is executing 50% TUX code :-)
>
> (there are still a number of performance improvement patches pending that
> are not integrated yet: the pagecache extreme-scalability patch and the
> smptimers patch. These patches speed both X15 and TUX up.)
>
> (there is one thing though that can never be 'exported to user-space': to
> isolate possibly untrusted binary application code from the server itself,
> without performance degradation. So we always have to be mentally open to
> the validity of kernel-space services.)
>
>         Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

hm, another X15 caching artifact i noticed: i've changed the index.html
file, and while X15 was still serving the old copy, TUX served the new
file immediately.

its cache is apparently not coherent with actual VFS contents. (ie. it's a
read-once cache apparently?) TUX has some (occasionally significant)
overhead due to non-cached VFS-lookups.

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

This is actually a bug in the current X15, I know how to fix it (with
negligible overhead) but I've been lazy :)

give me a few days...

 - Fabio

Ingo Molnar wrote:

> hm, another X15 caching artifact i noticed: i've changed the index.html
> file, and while X15 was still serving the old copy, TUX served the new
> file immediately.
>
> its cache is apparently not coherent with actual VFS contents. (ie. it's a
> read-once cache apparently?) TUX has some (occasionally significant)
> overhead due to non-cached VFS-lookups.
>
>         Ingo
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Tue, 1 May 2001, Fabio Riccardi wrote:

> This is actually a bug in the current X15, I know how to fix it (with
> negligible overhead) but I've been lazy :)

yep, i think it's pretty straightforward: you have a cache of open file
descriptors (like Squid i think), and you can start a background 'cache
synchronization thread' that does a stat() on every open file's real VFS
path, every second or so. This should have small overhead (the number of
file descriptors cached should be limited anyway via some sort of LRU),
and guarantees 'practical coherency', without having the overhead of
immediate coherency. [total coherency is pointless anyway, not even the
kernel guarantees it to all parallel VFS users.]

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

I have fixed the stale header cache problem. Files are statted on every
request, no "practical" tricks.

Performance doesn't seem to have suffered :)

I also have added a cache garbage collector to close "old" file descriptors
and remove even older header cache entries. This should make sure that you
don't exceed your system resources. The definition of old and the sweep
frequency are user configurable.

You can download the new version
from: http://www.chromium.com/X15-Alpha-3.tgz

 - Fabio

Ingo Molnar wrote:

> On Tue, 1 May 2001, Fabio Riccardi wrote:
>
> > This is actually a bug in the current X15, I know how to fix it (with
> > negligible overhead) but I've been lazy :)
>
> yep, i think it's pretty straightforward: you have a cache of open file
> descriptors (like Squid i think), and you can start a background 'cache
> synchronization thread' that does a stat() on every open file's real VFS
> path, every second or so. This should have small overhead (the number of
> file descriptors cached should be limited anyway via some sort of LRU),
> and guarantees 'practical coherency', without having the overhead of
> immediate coherency. [total coherency is pointless anyway, not even the
> kernel guarantees it to all parallel VFS users.]
>
>         Ingo

Re: X15 alpha release

[Ingo Molnar]

Fabio,

i noticed another RFC anomaly in X15. It ignores the "Connection: close"
request header passed by a HTTP/1.1 client. This behavior is against RFC
2616, a server must not override the client's choice of non-persistent
connection. (there might be HTTP/1.1 clients that do not support
persistent connections and signal this via "Connection: close".)

the rule is this: a request is either keepalive or non-keepalive. HTTP/1.0
requests default to non-keepalive. HTTP/1.1 requests default to keepalive.
The default can be overriden via the "Connection: Keep-Alive" or
"Connection: close" header fields.

if you fix this, does it impact SPECweb99 performance in any way?

	Ingo

Re: X15 alpha release

[Fabio Riccardi]

Ingo,

I'm really impressed by your feedback! How do you manage to discover so many
things?

I fixed the bug, and checked that it hadn't affected my specweb results.

Indeed specweb never issues closing 1.1 connections, it would use a 1.0
request with close in that case.

Moreover even if a client says that it will close the connection and the
server instead leaves it open, the client would just close the connection
anyway, unless there is a (very contrived) bug in the client which would let
itself be diverted from its original intention by an overly talkative
server...

X15 would be indeed negatively affected by these useless idle open
connections cluttering the file descriptor table and consuming resources for
nothing.

I'll post the corrected version later on today.

BTW: is there any _concise_ document specifying the HTTP protocol and its
variants?

 - Fabio

Ingo Molnar wrote:

> Fabio,
>
> i noticed another RFC anomaly in X15. It ignores the "Connection: close"
> request header passed by a HTTP/1.1 client. This behavior is against RFC
> 2616, a server must not override the client's choice of non-persistent
> connection. (there might be HTTP/1.1 clients that do not support
> persistent connections and signal this via "Connection: close".)
>
> the rule is this: a request is either keepalive or non-keepalive. HTTP/1.0
> requests default to non-keepalive. HTTP/1.1 requests default to keepalive.
> The default can be overriden via the "Connection: Keep-Alive" or
> "Connection: close" header fields.
>
> if you fix this, does it impact SPECweb99 performance in any way?
>
>         Ingo
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

yet another anomaly i noticed. X15 does not appear to handle pipelined
HTTP/1.1 requests properly, it ignores the second request if two requests
arrive in the same packet.

SPECweb99 does not send pipelined requests, but a number of RL web clients
do. (Mozilla, apt-get, etc.)

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

ok, I'm totally ignorant here, what is a pipelined request?

btw: please be kind with my mistakes, X15 _is_ alpha code anyway... :)

 - Fabio

Ingo Molnar wrote:

> yet another anomaly i noticed. X15 does not appear to handle pipelined
> HTTP/1.1 requests properly, it ignores the second request if two requests
> arrive in the same packet.
>
> SPECweb99 does not send pipelined requests, but a number of RL web clients
> do. (Mozilla, apt-get, etc.)
>
>         Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Davide Libenzi]

On 04-May-2001 Fabio Riccardi wrote:
> ok, I'm totally ignorant here, what is a pipelined request?



http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html

<QUOTE>
A pipelined application implementation buffers its output before writing it to
the underlying TCP stack, roughly equivalent to what the Nagle algorithm does
for telnet connections.
These two buffering algorithms tend to interfere, and using them together will
often cause very significant performance degradation. For each connection, the
server maintains a
response buffer that it flushes either when full, or when there is no more
requests coming in on that connection, or before it goes idle. This buffering
enables aggregating responses
(for example, cache validation responses) into fewer packets even on a
high-speed network, and saving CPU time for the server. 
</QUOTE>





- Davide

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

Hello,

I have uploaded a new release of X15 that hopefully solves all the RFC bugs.
I say hopefully because I haven't had the opportunity to fully test the
request pipelining. Is there anything to automatize such tests?

>From what I could measure X15 is still a good 5% faster than TUX.

You can find the file at: http://www.chromium.com/X15-Alpha-4.tgz

BTW: Next release (in a week or so) will be a beta and it will include source
code!

BTW2: I'll be away from my email in the next few days, so don't be amazed if
I'll be kind of slow replying to messages...

 - Fabio

Ingo Molnar wrote:

> yet another anomaly i noticed. X15 does not appear to handle pipelined
> HTTP/1.1 requests properly, it ignores the second request if two requests
> arrive in the same packet.
>
> SPECweb99 does not send pipelined requests, but a number of RL web clients
> do. (Mozilla, apt-get, etc.)
>
>         Ingo
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Sun, 29 Apr 2001, Fabio Riccardi wrote:

> TUX has definitively been my performance yardstick for the development
> of X15, but I had many sources of inspiration for the X15
> architecture. Maybe the most relevant are the Flash Web Server (Pai,
> Druschel, Zwaenepoel), several Linus observations on this list about
> (web) server architecture and kernnel services, and the reading of the
> Hennessy & Patterson architecture books. [...]

i think Zach's phhttpd is an important milestone as well, it's the first
userspace webserver that shows how to use event-based, sigio-based async
networking IO and sendfile() under Linux. (I believe it had some
performance problems related to sigio queue overflow, these issues might
be solved in the latest kernels.) The zerocopy enhancements should help
phhttpd as well.

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Zach Brown]

> i think Zach's phhttpd is an important milestone as well, it's the first
> userspace webserver that shows how to use event-based, sigio-based async
> networking IO and sendfile() under Linux. (I believe it had some

*blush*

> performance problems related to sigio queue overflow, these issues might
> be solved in the latest kernels.) The zerocopy enhancements should help
> phhttpd as well.

oh, it has a bunch of problems :)  over-threading created complexity in
the fast path.  It always spends memory on a contiguous header region for
each connection, which may not be valid in the days of zero copy sendmsg.
It does IO in the fast path.  And looking back at it, I'm struck by how
naive most of the code is :) :)

I've always been tempted to go back and take a real swing at a
nice content server, but there's only so many hours in the day, and
apache+thttpd+tux complete the problem space.  If X15 isn't released
with an open license, I may be tempted yet again :)

- z

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

Our intention is to release X15 with an open source license.

This will happen as soon as the codebase stabilizes a bit, that is when we go
beta (in two - three weeks).

At the moment we just don't have the time...

The reason why I released the alpha binary version is that several people
would not believe that a user-space server with this level of performance
would be possible at all and several statements that I made on this list were
challenged.

Besides I really appreciate the feedback that I received so far from Ingo and
others, and I'd be very curious to know if anybody did any performance
evaluation at all.

 - Fabio

Zach Brown wrote:

> I've always been tempted to go back and take a real swing at a
> nice content server, but there's only so many hours in the day, and
> apache+thttpd+tux complete the problem space.  If X15 isn't released
> with an open license, I may be tempted yet again :)

Re: X15 alpha release: as fast as TUX but in user space

[Lincoln Dale]

Hi,

At 10:50 AM 2/05/2001 +0200, Ingo Molnar wrote:
>i think Zach's phhttpd is an important milestone as well, it's the first
>userspace webserver that shows how to use event-based, sigio-based async
>networking IO and sendfile() under Linux. (I believe it had some
>performance problems related to sigio queue overflow, these issues might
>be solved in the latest kernels.) The zerocopy enhancements should help
>phhttpd as well.

my experience with sigio-based event-handlers is that the net-gain of 
event-driven i/o is mitigated by the fact that SIGIO is based on signals.

the problem with signals for this purpose are:
  (a) you go thru a syncronization point in the kernel.  signals are protected
      by a spinlock.
      it doesn't scale with SMP.
  (b) SI_PAD_SIZE

explicitly, (b) means that you have an awful lot of memory-accesses going 
on for every signal.
my experience with the overhead is that it mitigates the advantages when 
you become bottlenecked on memory-bus-accesses.


cheers,

lincoln.

Linux syscall speed -- was X15 rootin-tootin webserver

[Michael Rothwell]

According to tests performed at IBM:

http://www-106.ibm.com/developerworks/linux/library/l-rt1/

Linux's sycalls are a little more than twice as fast as those of Windows
2000. 0.75usec vs 2.0msec. Not too shabby. And this "magic page" idea
means it may get faster.

-M