X15 alpha release: as fast as TUX but in user space

X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

Dear All,

I'd like to announce the first release of X15 Alpha 1, a _user space_
web server that is as fast as TUX.

On my Dell 4400 with 2G of RAM and 2 933MHz PIII and NetGear 2Gbit NICs
I achieve about 2500 SpecWeb99 connections, with both X15 and
TUX (actually X15 is sligtly faster, some 20 connections more... ;)

Given the limitations of my experimental setup I'd like to ask if some
of you could help me testing my software on some higher end machines.
I'm interested to see what happens on 4-8 processors in terms of
scalability etc.

You can download X15 Alpha 1 from here:
http://www.chromium.com/X15-Alpha-1.tgz

The the README file in the tarball should contain sufficient information
to run the thing, I also included a support module for running the
SpecWeb benchmark.

TIA, ciao,

 - Fabio

Re: X15 alpha release: as fast as TUX but in user space

[Aaron Lehmann]

On Fri, Apr 27, 2001 at 05:18:26PM -0700, Fabio Riccardi wrote:
> You can download X15 Alpha 1 from here:
> http://www.chromium.com/X15-Alpha-1.tgz

Where's the source?

Re: X15 alpha release: as fast as TUX but in user space

[David S. Miller]

Fabio Riccardi writes:
 > On my Dell 4400 with 2G of RAM and 2 933MHz PIII and NetGear 2Gbit NICs
 > I achieve about 2500 SpecWeb99 connections, with both X15 and
 > TUX (actually X15 is sligtly faster, some 20 connections more... ;)

What is the CPU utilization like in X15 vs. TUX during
these runs?

Later,
David S. Miller
davem@redhat.com

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

In both cases (X15 and TUX) the CPU utilization is 100%

There are no IO bottlenecks on disk or on the net side.

I think that the major bottleneck is the speed of RAM and the PCI bus, wait
cycles.

We are basically going at the speed of the hardware.

 - Fabio

"David S. Miller" wrote:

> Fabio Riccardi writes:
>  > On my Dell 4400 with 2G of RAM and 2 933MHz PIII and NetGear 2Gbit NICs
>  > I achieve about 2500 SpecWeb99 connections, with both X15 and
>  > TUX (actually X15 is sligtly faster, some 20 connections more... ;)
>
> What is the CPU utilization like in X15 vs. TUX during
> these runs?
>
> Later,
> David S. Miller
> davem@redhat.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

Fabio,

i noticed one weirdness in the Date-field handling of X15. X15 appears to
cache the Date field too, which is contrary to RFCs:

 earth2:~> wget -s http://localhost/index.html -O - 2>/dev/null | grep Date
 Date: Sat Apr 28 10:15:14 2001
 earth2:~> date
 Sat Apr 28 10:32:40 CEST 2001

ie. there is already a 15 minutes difference between the 'origin date of
the reply' and the actual date of the reply. (i started X15 up 15 minutes
ago.)

per RFC 2616:
.............
The Date general-header field represents the date and time at which the
message was originated, [...]

Origin servers MUST include a Date header field in all responses, [...]
.............

i considered the caching of the Date field for TUX too, and avoided it
exactly due to this issue, to not violate this 'MUST' item in the RFC. It
can be reasonably expected from a web server to have a 1-second accurate
Date: field.

the header-caching in X15 gives it an edge against TUX, obviously, but IMO
it's a questionable practice.

if caching of headers was be allowed then we could the obvious trick of
sendfile()ing complete web replies (first header, then body).

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Ville Herva]

On Sat, Apr 28, 2001 at 10:42:29AM +0200, you [Ingo Molnar] claimed:
> 
> per RFC 2616:
> .............
> The Date general-header field represents the date and time at which the
> message was originated, [...]
> 
> Origin servers MUST include a Date header field in all responses, [...]
> .............
> 
> i considered the caching of the Date field for TUX too, and avoided it
> exactly due to this issue, to not violate this 'MUST' item in the RFC. It
> can be reasonably expected from a web server to have a 1-second accurate
> Date: field.
> 
> the header-caching in X15 gives it an edge against TUX, obviously, but IMO
> it's a questionable practice.
> 
> if caching of headers was be allowed then we could the obvious trick of
> sendfile()ing complete web replies (first header, then body).

Uhh, perhaps I'm stupid, but why not cache the date field and update the
field once a five seconds? Or even once a second?

I mean, at the rate of thousands of requests per second that should give you
some advantage over dynamically generating it -- especially if that's the
only thing hindering copletely sendfile()'ing the answer.


-- v --

v@iki.fi

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Sat, 28 Apr 2001, Ville Herva wrote:

> Uhh, perhaps I'm stupid, but why not cache the date field and update
> the field once a five seconds? Or even once a second?

yes, that should work. but that means possibly updating thousands of (or
more) cached headers, which has some overhead ...

> I mean, at the rate of thousands of requests per second that should
> give you some advantage over dynamically generating it -- especially
> if that's the only thing hindering copletely sendfile()'ing the
> answer.

well, the method i suggested was to use sendfile() twice: first the
(cached, or freshly constructed) headers put into a big file, then the
body itself (which is the original file, accessed via cached file
descriptors).

(splitting up the header and the body has the benefit of not dual-caching
the same webcontent. this is what TUX does too.)

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Sat, 28 Apr 2001, Ville Herva wrote:

> Uhh, perhaps I'm stupid, but why not cache the date field and update
> the field once a five seconds? Or even once a second?

perhaps the best way would be to do this updating in the sending code
itself.

first there would be a 'current time thread', which updates a global
shared variable that shows the current time. (ie. no extra system-call is
needed to access current time.) If the header-sending code detects that
current time is not equal to the timestamp stored in the header itself,
then the header is reconstructed. Pretty simple.

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Ville Herva]

On Sat, Apr 28, 2001 at 03:24:25PM +0200, you [Ingo Molnar] claimed:
> 
> On Sat, 28 Apr 2001, Ville Herva wrote:
> 
> > Uhh, perhaps I'm stupid, but why not cache the date field and update
> > the field once a five seconds? Or even once a second?
> 
> perhaps the best way would be to do this updating in the sending code
> itself.
> 
> first there would be a 'current time thread', which updates a global
> shared variable that shows the current time. (ie. no extra system-call is
> needed to access current time.) If the header-sending code detects that
> current time is not equal to the timestamp stored in the header itself,
> then the header is reconstructed. Pretty simple.

Yes, that's vaguely resembles what I had in mind. Of course I had no idea
about the data structures Tux or X15 use internally, so I couldn't think it
too thoroughly.


-- v --

v@iki.fi

Re: X15 alpha release: as fast as TUX but in user space

[Andi Kleen]

On Sat, Apr 28, 2001 at 04:30:30PM +0300, Ville Herva wrote:
> Yes, that's vaguely resembles what I had in mind. Of course I had no idea
> about the data structures Tux or X15 use internally, so I couldn't think it
> too thoroughly.

You can also just use the cycle counter directly in most modern CPUs. It can
be read with a single instruction.
In fact modern glibc will do it for you when you use 
clock_gettime(CLOCK_PROCESS_CPUTIME_ID, ...) 

-Andi

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

I can disable header caching and see what happens, I'll add an option for this
in the next X15 release.

Nevertheless I don't know how much this is interesting in real life, since on
the internet most static pages are cached on proxies. I agree that the
RFC asks for a date for the original response, but once the response is cached
what does this date mean?

 - Fabio

Ingo Molnar wrote:

> Fabio,
>
> i noticed one weirdness in the Date-field handling of X15. X15 appears to
> cache the Date field too, which is contrary to RFCs:
>
>  earth2:~> wget -s http://localhost/index.html -O - 2>/dev/null | grep Date
>  Date: Sat Apr 28 10:15:14 2001
>  earth2:~> date
>  Sat Apr 28 10:32:40 CEST 2001
>
> ie. there is already a 15 minutes difference between the 'origin date of
> the reply' and the actual date of the reply. (i started X15 up 15 minutes
> ago.)
>
> per RFC 2616:
> .............
> The Date general-header field represents the date and time at which the
> message was originated, [...]
>
> Origin servers MUST include a Date header field in all responses, [...]
> .............
>
> i considered the caching of the Date field for TUX too, and avoided it
> exactly due to this issue, to not violate this 'MUST' item in the RFC. It
> can be reasonably expected from a web server to have a 1-second accurate
> Date: field.
>
> the header-caching in X15 gives it an edge against TUX, obviously, but IMO
> it's a questionable practice.
>
> if caching of headers was be allowed then we could the obvious trick of
> sendfile()ing complete web replies (first header, then body).
>
>         Ingo

Re: X15 alpha release: as fast as TUX but in user space

[dean gaudet]

On Sun, 29 Apr 2001, Fabio Riccardi wrote:

> I can disable header caching and see what happens, I'll add an option
> for this in the next X15 release.

heh, well to be honest, i'd put the (permanent) caching of the Date header
into the very slimy, benchmark-only trick category.  (right up there
alongside running the HTTP and TCP stacks inside the NIC interrupt
handler, which folks have done to get even better scores.)

> Nevertheless I don't know how much this is interesting in real life,
> since on the internet most static pages are cached on proxies. I agree
> that the RFC asks for a date for the original response, but once the
> response is cached what does this date mean?

the Date is always the time the response was generated on the origin
server.  (there are exceptions for clockless servers, but such servers
have other limitations you wouldn't want -- notably they MUST NOT generate
Last-Modified.)

one example oddity you might experience with a non-increasing Date
surrounds Last-Modified and Date, see section 13.3.3.  note that the rfc
indicates that if Last-Modified is less than 60 seconds earlier than Date
then Last-Modified is only a weak validator rather than a strong
validator.  this would complicate range requests -- because weak
validators can't be used with range requests.  if your server never
updates the Date after the first time it serves an object then you'd
potentially never get out of this 60 second window.

(strong validators are guaranteed to change whenever the object changes...
and Last-Modified isn't strong until some time has passed -- consider NFS
mounted docroots, clock skew in the origin network, multiple file updates
within a second, etc.)

there are a bunch of other things that Date is used for, all of them are
related to caching heuristics and rules.

in theory you could claim that you're implementing a cache server rather
than an origin server... i dunno what the SPEC committee will think when
you try to submit results though :)

so way back when sendfile() was being created i brought up the Date issue
and pointed out that we needed more than a single "void *, size_t" to take
care of headers.  eventually this discussion lead creation of TCP_CORK --
so that a http server could use writev() to send a two element iov for the
headers -- one element with everything that doesn't need to change, the
next element with the Date.

i also kind of expected the high performance servers to rebuild a Date:
header every second for all of its threads to use.  (of course it's not
that simple, you really want to keep a circular list of N dates... and
just assume that after N seconds no thread could still be accessing an old
Date.)

is this too slow for some reason?  (does it play well with zero-copy?)

-dean

Re: X15 alpha release: as fast as TUX but in user space

[David S. Miller]

dean gaudet writes:
 > is this too slow for some reason?  (does it play well with zero-copy?)

His trick ends up with a minimal set of scatter gather entries.
That's the whole gain behind the trick he's doing.

If you do the TCP_CORK thing, what you end up with is a scatter gather
entry in the SKB for the header bits, then the page cache segments.

Even if we had the HP sendfile() interface iovec garbage, we would end
up with the same number of SKB iovec entries as for the TCP_CORK case
today.

What TUX basically does is build up the header by hand in a scribble
page it uses for header builing, passes that to tcp_sendpage() with
MSG_MORE set, then it initiates the sendfile() part.  The final effect
inside the networking is basically equivalent to using
TCP_CORK+sendfile() in userspace, the only difference being that:

1) the scratch page for the headers is maintained per-socket by TCP
2) the header is copied once from user to kernel

I would find it amusing to see what adding the header+file caching
trick to TUX would do to it's results :-)

Later,
David S. Miller
davem@redhat.com

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

Ok I fixed it, the header date timestamp is updated with every request.

Performance doesn't seem to have suffered significantly (less than 1%).

You can find the new release at:  http://www.chromium.com/X15-Alpha-2.tgz

BTW: Don't call me slime, I wasn't trying to cheat, I just didn't know that
the date stamp was required to be really up-to-date.

 - Fabio

dean gaudet wrote:

> On Sun, 29 Apr 2001, Fabio Riccardi wrote:
>
> > I can disable header caching and see what happens, I'll add an option
> > for this in the next X15 release.
>
> heh, well to be honest, i'd put the (permanent) caching of the Date header
> into the very slimy, benchmark-only trick category.  (right up there
> alongside running the HTTP and TCP stacks inside the NIC interrupt
> handler, which folks have done to get even better scores.)
>
> > Nevertheless I don't know how much this is interesting in real life,
> > since on the internet most static pages are cached on proxies. I agree
> > that the RFC asks for a date for the original response, but once the
> > response is cached what does this date mean?
>
> the Date is always the time the response was generated on the origin
> server.  (there are exceptions for clockless servers, but such servers
> have other limitations you wouldn't want -- notably they MUST NOT generate
> Last-Modified.)
>
> one example oddity you might experience with a non-increasing Date
> surrounds Last-Modified and Date, see section 13.3.3.  note that the rfc
> indicates that if Last-Modified is less than 60 seconds earlier than Date
> then Last-Modified is only a weak validator rather than a strong
> validator.  this would complicate range requests -- because weak
> validators can't be used with range requests.  if your server never
> updates the Date after the first time it serves an object then you'd
> potentially never get out of this 60 second window.
>
> (strong validators are guaranteed to change whenever the object changes...
> and Last-Modified isn't strong until some time has passed -- consider NFS
> mounted docroots, clock skew in the origin network, multiple file updates
> within a second, etc.)
>
> there are a bunch of other things that Date is used for, all of them are
> related to caching heuristics and rules.
>
> in theory you could claim that you're implementing a cache server rather
> than an origin server... i dunno what the SPEC committee will think when
> you try to submit results though :)
>
> so way back when sendfile() was being created i brought up the Date issue
> and pointed out that we needed more than a single "void *, size_t" to take
> care of headers.  eventually this discussion lead creation of TCP_CORK --
> so that a http server could use writev() to send a two element iov for the
> headers -- one element with everything that doesn't need to change, the
> next element with the Date.
>
> i also kind of expected the high performance servers to rebuild a Date:
> header every second for all of its threads to use.  (of course it's not
> that simple, you really want to keep a circular list of N dates... and
> just assume that after N seconds no thread could still be accessing an old
> Date.)
>
> is this too slow for some reason?  (does it play well with zero-copy?)
>
> -dean
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: X15 alpha release: as fast as TUX but in user space

[dean gaudet]

On Mon, 30 Apr 2001, Fabio Riccardi wrote:

> Ok I fixed it, the header date timestamp is updated with every request.
>
> Performance doesn't seem to have suffered significantly (less than 1%).

rad!

> BTW: Don't call me slime, I wasn't trying to cheat, I just didn't know that
> the date stamp was required to be really up-to-date.

sorry, i meant to put a smily on there :)


On Sun, 29 Apr 2001, David S. Miller wrote:

> If you do the TCP_CORK thing, what you end up with is a scatter gather
> entry in the SKB for the header bits, then the page cache segments.

so then the NIC would be sent a 3 entry gather list -- 1 entry for TCP/IP
headers, 1 for HTTP headers, and 1 for the initial page cache segment?

are there any NICs which take only 2 entry lists?  (boo hiss and curses
on such things if they exist!)

-dean

Re: X15 alpha release: as fast as TUX but in user space

[David S. Miller]

dean gaudet writes:
 > On Sun, 29 Apr 2001, David S. Miller wrote:
 > 
 > > If you do the TCP_CORK thing, what you end up with is a scatter gather
 > > entry in the SKB for the header bits, then the page cache segments.
 > 
 > so then the NIC would be sent a 3 entry gather list -- 1 entry for TCP/IP
 > headers, 1 for HTTP headers, and 1 for the initial page cache segment?

Basically.  It's weird because we could change tcp_sendmsg() to grab a
"little bit" of space in skb->data after the TCP headers area, but
that would screw all the memory allocation advantages carving up pages
gives us.

TCP used to be really rough on the memory subsystem, and in particular
going to a page carving scheme helped a lot in this area.

 > are there any NICs which take only 2 entry lists?  (boo hiss and curses
 > on such things if they exist!)

Tulip I think falls into this category, I could be wrong.  It has two
buffer pointers in the RX descriptor, but one might be able to chain
them.

Alexey added SG support to Tulip at some point, and I can probably dig
up the patch.  It doesn't do hw csumming, though.

Later,
David S. Miller
davem@redhat.com

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Mon, 30 Apr 2001, Fabio Riccardi wrote:

> Ok I fixed it, the header date timestamp is updated with every
> request.
>
> Performance doesn't seem to have suffered significantly (less than
> 1%).

yep, expected that - doing a sendmsg()+sendfile() generates the same
packet structure, the only difference being that ~100-200 bytes are copied
between kernel-space and user-space.

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Sun, 29 Apr 2001, Fabio Riccardi wrote:

> I can disable header caching and see what happens, I'll add an option
> for this in the next X15 release.

what SPECweb99 performance do you get if you disable header caching? It
might not make all that much of a difference. (but it will make the
results more comparable with TUX.)

	Ingo

X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Fri, 27 Apr 2001, Fabio Riccardi wrote:

> I'd like to announce the first release of X15 Alpha 1, a _user space_
> web server that is as fast as TUX.

great, the first TUX clone! ;-)

This should put the accusations to rest that Linux got the outstandingly
high SPECweb99 scores only because the webserver was in kernel-space. It's
the 2.4 kernel's high performance that enabled those results, having the
web-server in kernel-space didnt have much effect. TUX was and remains a
testbed to test high-performance webserving (and FTP serving), without the
API-exporting overhead of userspace.

[i suspect the small performance advantage of X15 is due to subtle
differences in the SPECweb99 user-space module: eg. while the TUX code was
written, tested and ready to use mmap()-enabled
TUXAPI_alloc_read_objectbuf(), it wasnt enabled actually. I sent Fabio a
mail how to enable it, perhaps he can do some tests to confirm this
suspicion?]

doing a TUX 2.0 SPECweb99 benchmark on the latest -ac kernels, 86% of time
is spent in generic parts of the kernel, 12% of time is spent in the
user-space SPECweb99 module, and only 2% of time is spent in TUX-specific
kernel code.

doing the same test with the original TUX 1.0 code shows that more than
50% of CPU time was spent in TUX-specific code.

what does this mean? In the roughly 6 months since TUX 1.0 was released,
we moved much of the TUX 1.0 -only improvements into the generic kernel
(most of which was made available to user-space as well), and TUX itself
became smaller and smaller (and used more and more generic parts of the
kernel). So in effect X15 is executing 50% TUX code :-)

(there are still a number of performance improvement patches pending that
are not integrated yet: the pagecache extreme-scalability patch and the
smptimers patch. These patches speed both X15 and TUX up.)

(there is one thing though that can never be 'exported to user-space': to
isolate possibly untrusted binary application code from the server itself,
without performance degradation. So we always have to be mentally open to
the validity of kernel-space services.)

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

Linux 2.4 is surely one of the most advanced OSs ever happened, especially
from the optimization point of view and for the admirable economy of concepts
on which it lies. I definitively hope that X15 helps reinforcing the success
to this amazing system.

TUX has definitively been my performance yardstick for the development of
X15, but I had many sources of inspiration for the X15 architecture. Maybe
the most relevant are the Flash Web Server (Pai, Druschel, Zwaenepoel),
several Linus observations on this list about (web) server architecture and
kernnel services, and the reading of the Hennessy & Patterson architecture
books. Last but not least, aside from some heated discussions, research in
microkernel architecture has taught us many lessons on how to achieve an
efficient model of interaction across separate addressing spaces.

If i have to make some sort of educated guess and point at where the current
bottleneck lies for web server performance, I would say that it is somewhere
between the memory subsystem and the PCI bus.

With zero-copy sendfile data movement is not an issue anymore, asynchronous
network IO allows for really inexpensive thread scheduling, and system call
invocation adds a very negligible overhead in Linux. What we are left with
now is purely wait cycles, the CPUs and the NICs are contending for memory
and bus bandwidth. It would be really interesting to see where the network
shifts now that faster machines are becoming available.

On my whish list for future kernel developments I would definitively put disk
asynchronous IO and a more decent file descriptor passing implementation.
I'll detail this in subsequent messages.

I'll surely check out the impact of Ingo's patches on TUX performance
sometime this week.

I'd also like to reiterate my request for help for testing X15 on higher end
server architectures.

X15 is still very young alpha code and I can surely improve its performance
in many ways.

 - Fabio

Ingo Molnar wrote:

> On Fri, 27 Apr 2001, Fabio Riccardi wrote:
>
> > I'd like to announce the first release of X15 Alpha 1, a _user space_
> > web server that is as fast as TUX.
>
> great, the first TUX clone! ;-)
>
> This should put the accusations to rest that Linux got the outstandingly
> high SPECweb99 scores only because the webserver was in kernel-space. It's
> the 2.4 kernel's high performance that enabled those results, having the
> web-server in kernel-space didnt have much effect. TUX was and remains a
> testbed to test high-performance webserving (and FTP serving), without the
> API-exporting overhead of userspace.
>
> [i suspect the small performance advantage of X15 is due to subtle
> differences in the SPECweb99 user-space module: eg. while the TUX code was
> written, tested and ready to use mmap()-enabled
> TUXAPI_alloc_read_objectbuf(), it wasnt enabled actually. I sent Fabio a
> mail how to enable it, perhaps he can do some tests to confirm this
> suspicion?]
>
> doing a TUX 2.0 SPECweb99 benchmark on the latest -ac kernels, 86% of time
> is spent in generic parts of the kernel, 12% of time is spent in the
> user-space SPECweb99 module, and only 2% of time is spent in TUX-specific
> kernel code.
>
> doing the same test with the original TUX 1.0 code shows that more than
> 50% of CPU time was spent in TUX-specific code.
>
> what does this mean? In the roughly 6 months since TUX 1.0 was released,
> we moved much of the TUX 1.0 -only improvements into the generic kernel
> (most of which was made available to user-space as well), and TUX itself
> became smaller and smaller (and used more and more generic parts of the
> kernel). So in effect X15 is executing 50% TUX code :-)
>
> (there are still a number of performance improvement patches pending that
> are not integrated yet: the pagecache extreme-scalability patch and the
> smptimers patch. These patches speed both X15 and TUX up.)
>
> (there is one thing though that can never be 'exported to user-space': to
> isolate possibly untrusted binary application code from the server itself,
> without performance degradation. So we always have to be mentally open to
> the validity of kernel-space services.)
>
>         Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

hm, another X15 caching artifact i noticed: i've changed the index.html
file, and while X15 was still serving the old copy, TUX served the new
file immediately.

its cache is apparently not coherent with actual VFS contents. (ie. it's a
read-once cache apparently?) TUX has some (occasionally significant)
overhead due to non-cached VFS-lookups.

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

This is actually a bug in the current X15, I know how to fix it (with
negligible overhead) but I've been lazy :)

give me a few days...

 - Fabio

Ingo Molnar wrote:

> hm, another X15 caching artifact i noticed: i've changed the index.html
> file, and while X15 was still serving the old copy, TUX served the new
> file immediately.
>
> its cache is apparently not coherent with actual VFS contents. (ie. it's a
> read-once cache apparently?) TUX has some (occasionally significant)
> overhead due to non-cached VFS-lookups.
>
>         Ingo
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Tue, 1 May 2001, Fabio Riccardi wrote:

> This is actually a bug in the current X15, I know how to fix it (with
> negligible overhead) but I've been lazy :)

yep, i think it's pretty straightforward: you have a cache of open file
descriptors (like Squid i think), and you can start a background 'cache
synchronization thread' that does a stat() on every open file's real VFS
path, every second or so. This should have small overhead (the number of
file descriptors cached should be limited anyway via some sort of LRU),
and guarantees 'practical coherency', without having the overhead of
immediate coherency. [total coherency is pointless anyway, not even the
kernel guarantees it to all parallel VFS users.]

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

I have fixed the stale header cache problem. Files are statted on every
request, no "practical" tricks.

Performance doesn't seem to have suffered :)

I also have added a cache garbage collector to close "old" file descriptors
and remove even older header cache entries. This should make sure that you
don't exceed your system resources. The definition of old and the sweep
frequency are user configurable.

You can download the new version
from: http://www.chromium.com/X15-Alpha-3.tgz

 - Fabio

Ingo Molnar wrote:

> On Tue, 1 May 2001, Fabio Riccardi wrote:
>
> > This is actually a bug in the current X15, I know how to fix it (with
> > negligible overhead) but I've been lazy :)
>
> yep, i think it's pretty straightforward: you have a cache of open file
> descriptors (like Squid i think), and you can start a background 'cache
> synchronization thread' that does a stat() on every open file's real VFS
> path, every second or so. This should have small overhead (the number of
> file descriptors cached should be limited anyway via some sort of LRU),
> and guarantees 'practical coherency', without having the overhead of
> immediate coherency. [total coherency is pointless anyway, not even the
> kernel guarantees it to all parallel VFS users.]
>
>         Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

yet another anomaly i noticed. X15 does not appear to handle pipelined
HTTP/1.1 requests properly, it ignores the second request if two requests
arrive in the same packet.

SPECweb99 does not send pipelined requests, but a number of RL web clients
do. (Mozilla, apt-get, etc.)

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

ok, I'm totally ignorant here, what is a pipelined request?

btw: please be kind with my mistakes, X15 _is_ alpha code anyway... :)

 - Fabio

Ingo Molnar wrote:

> yet another anomaly i noticed. X15 does not appear to handle pipelined
> HTTP/1.1 requests properly, it ignores the second request if two requests
> arrive in the same packet.
>
> SPECweb99 does not send pipelined requests, but a number of RL web clients
> do. (Mozilla, apt-get, etc.)
>
>         Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Davide Libenzi]

On 04-May-2001 Fabio Riccardi wrote:
> ok, I'm totally ignorant here, what is a pipelined request?



http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html

<QUOTE>
A pipelined application implementation buffers its output before writing it to
the underlying TCP stack, roughly equivalent to what the Nagle algorithm does
for telnet connections.
These two buffering algorithms tend to interfere, and using them together will
often cause very significant performance degradation. For each connection, the
server maintains a
response buffer that it flushes either when full, or when there is no more
requests coming in on that connection, or before it goes idle. This buffering
enables aggregating responses
(for example, cache validation responses) into fewer packets even on a
high-speed network, and saving CPU time for the server. 
</QUOTE>





- Davide

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

Hello,

I have uploaded a new release of X15 that hopefully solves all the RFC bugs.
I say hopefully because I haven't had the opportunity to fully test the
request pipelining. Is there anything to automatize such tests?

>From what I could measure X15 is still a good 5% faster than TUX.

You can find the file at: http://www.chromium.com/X15-Alpha-4.tgz

BTW: Next release (in a week or so) will be a beta and it will include source
code!

BTW2: I'll be away from my email in the next few days, so don't be amazed if
I'll be kind of slow replying to messages...

 - Fabio

Ingo Molnar wrote:

> yet another anomaly i noticed. X15 does not appear to handle pipelined
> HTTP/1.1 requests properly, it ignores the second request if two requests
> arrive in the same packet.
>
> SPECweb99 does not send pipelined requests, but a number of RL web clients
> do. (Mozilla, apt-get, etc.)
>
>         Ingo
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: X15 alpha release: as fast as TUX but in user space

[Ingo Molnar]

On Sun, 29 Apr 2001, Fabio Riccardi wrote:

> TUX has definitively been my performance yardstick for the development
> of X15, but I had many sources of inspiration for the X15
> architecture. Maybe the most relevant are the Flash Web Server (Pai,
> Druschel, Zwaenepoel), several Linus observations on this list about
> (web) server architecture and kernnel services, and the reading of the
> Hennessy & Patterson architecture books. [...]

i think Zach's phhttpd is an important milestone as well, it's the first
userspace webserver that shows how to use event-based, sigio-based async
networking IO and sendfile() under Linux. (I believe it had some
performance problems related to sigio queue overflow, these issues might
be solved in the latest kernels.) The zerocopy enhancements should help
phhttpd as well.

	Ingo

Re: X15 alpha release: as fast as TUX but in user space

[Zach Brown]

> i think Zach's phhttpd is an important milestone as well, it's the first
> userspace webserver that shows how to use event-based, sigio-based async
> networking IO and sendfile() under Linux. (I believe it had some

*blush*

> performance problems related to sigio queue overflow, these issues might
> be solved in the latest kernels.) The zerocopy enhancements should help
> phhttpd as well.

oh, it has a bunch of problems :)  over-threading created complexity in
the fast path.  It always spends memory on a contiguous header region for
each connection, which may not be valid in the days of zero copy sendmsg.
It does IO in the fast path.  And looking back at it, I'm struck by how
naive most of the code is :) :)

I've always been tempted to go back and take a real swing at a
nice content server, but there's only so many hours in the day, and
apache+thttpd+tux complete the problem space.  If X15 isn't released
with an open license, I may be tempted yet again :)

- z

Re: X15 alpha release: as fast as TUX but in user space

[Fabio Riccardi]

Our intention is to release X15 with an open source license.

This will happen as soon as the codebase stabilizes a bit, that is when we go
beta (in two - three weeks).

At the moment we just don't have the time...

The reason why I released the alpha binary version is that several people
would not believe that a user-space server with this level of performance
would be possible at all and several statements that I made on this list were
challenged.

Besides I really appreciate the feedback that I received so far from Ingo and
others, and I'd be very curious to know if anybody did any performance
evaluation at all.

 - Fabio

Zach Brown wrote:

> I've always been tempted to go back and take a real swing at a
> nice content server, but there's only so many hours in the day, and
> apache+thttpd+tux complete the problem space.  If X15 isn't released
> with an open license, I may be tempted yet again :)

Re: X15 alpha release: as fast as TUX but in user space

[Lincoln Dale]

Hi,

At 10:50 AM 2/05/2001 +0200, Ingo Molnar wrote:
>i think Zach's phhttpd is an important milestone as well, it's the first
>userspace webserver that shows how to use event-based, sigio-based async
>networking IO and sendfile() under Linux. (I believe it had some
>performance problems related to sigio queue overflow, these issues might
>be solved in the latest kernels.) The zerocopy enhancements should help
>phhttpd as well.

my experience with sigio-based event-handlers is that the net-gain of 
event-driven i/o is mitigated by the fact that SIGIO is based on signals.

the problem with signals for this purpose are:
  (a) you go thru a syncronization point in the kernel.  signals are protected
      by a spinlock.
      it doesn't scale with SMP.
  (b) SI_PAD_SIZE

explicitly, (b) means that you have an awful lot of memory-accesses going 
on for every signal.
my experience with the overhead is that it mitigates the advantages when 
you become bottlenecked on memory-bus-accesses.


cheers,

lincoln.