* Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) [not found] <20020610102006.A6947@lemuria.org> @ 2002-06-13 1:44 ` Federico Sevilla III 2002-06-13 5:39 ` Very large font size crashing X Font Server and Grounding Server to Alan Cox ` (3 more replies) 0 siblings, 4 replies; 10+ messages in thread From: Federico Sevilla III @ 2002-06-13 1:44 UTC (permalink / raw) To: BugTraq Mailing List, Linux Kernel Mailing List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (Note: bug originally posted on BugTraq, this response is cross-posted to the LKML because of the way the server hosting the X font server ground to a halt.) On Mon, 10 Jun 2002 at 10:20, Tom wrote: > Affected > ======== > Mozilla 1.0 and earlier > verified on Linux and Solaris, other Unixes most likely affected as well. Re-verified on Debian GNU/Linux running kernel 2.4.18-xfs with Mozilla 1.0.0 and XFree86 4.1.0 with the font server running on a separate server also running on Debian GNU/Linux with kernel 2.4.18-xfs and the X font server from XFree86 4.1.0. The server running the X font server has 512MB RAM, 1GB swap, has one ext2 partition (/boot), one ReiserFS partition (Squid cache) and the rest are XFS partitions. It also runs the kernel-mode NFS server v1.0 and Samba 2.2.4, among other things, and is typically headless. > Effect > ====== > System becomes unuseable or X windows crashes > (varies depending on system configuration) Here I had a workstation (that I thought 'hey, I can reboot this after I've felt how the problem works') that used a remote font server, which was company-wide. Yes, I'm the administrator so I killed myself. I've got a personal note to myself, as well. "Don't try stuff posted on BugTraq on anything connected to the network, dummy." ;) Anyway, FWIW: First I noticed that the Mozilla window stopped responding immediately. So I closed it and breathed deeply. Then I noticed the server hosting the X font server started thrashing really badly. /home is exported from the same server via kernel-mode NFS, so file access to /home started freezing on workstations. Samba access slowed down, but continued to work. I was able to log on to the server with enough time to SIGKILL the xfs-daemon process. Unfortunately this wasn't good enough. The server started running up various processes stuck in "D" state, the OOM killer went on panic mode and started killing things left and right, mostly from what I notice apache and apache-ssl processes with messages like "Out of Memory: Killed process xxxx (apache)". I was also able to do a `free` after killing xfs-daemon and noticed that there was a lot of free memory both physically and on swap. Within less than ten minutes on this relatively lightly-loaded server, I could not log in to the machine locally, even as root (whose home directory is not NFS-exported) and load levels shot up to around 25, which is definitely abnormal. Existing logged-on processes also got stuck in whatever they were doing (`ps ax`, in particular is what I remember). Attempted reboots locally via Ctrl-Alt-Delete and Magic SysRq failed because (1) I had disabled ctrl-alt-delete mapping in inittab "for security", and (2) I had not compiled Magic SysRq support on this particular server. More notes to self. > Description > =========== > When loading pages with a specially prepared (or erroneous) stylesheet, > mozilla and X windows (not restricted to XFree) exhibit any of two > undesireable behaviours. This seems to depend on the local system > configuration, especially to the presence of xfs, but bug reports so far > are inconclusive. > In one scenario, X simply crashes, taking everything with it. This will result > in the loss of unsaved work. > In scenario two, memory useage of the X server explodes until the machine > reaches the thrashing point, at which point only a hard kill (-9) of the > X server can save it, provided there are enough system resources left to > issue the kill. In my case the workstation was easy to stabilize. I was actually able to close all windows and exit xfce properly. It's the server running the X font server that ground to painful halt. I do not know how things would have been if I had the Magic SysRq enabled. > The bug is triggered by a huge font setting done through CSS. Depending > on the end user's system configuration, this will either trigger an > abort in the XFree86 code ("Beziers this large not supported") or cause > an explosive use of memory. It is unknown how much memory could get > consumed, but follow-ups to the mozilla bug verify that machines with 1 > GB of memory still reach the thrashing point. While I agree with BugTraq posts in response to this that applications like Mozilla which accept font-sizes from unknown sources should have some check to prevent such large sizes from crashing X and/or the X Font Server, I'm alarmed by (1) the way the X font server allows itself to be crashed like this, and (2) the way the entire Linux kernel seems to have been unable to handle the situation. While having a central company or department wide font server may not have been the best choice I made, this seems like a simple way to drive a stake through a system's heart. Suggestions on how to work around this on multiple levels would definitely be appreciated. I'll be starting by removing the X font server from our file and authentication server onto some high-powered workstation, but I'm sure this won't be enough, and knowing that a user process like xfs-daemon can drag the Linux kernel down to knees is not very comforting. :( --> Jijo - -- Federico Sevilla III : <http://jijo.free.net.ph/> Network Administrator : The Leather Collection, Inc. GnuPG Key ID : 0x93B746BE -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) iD4DBQE9B/kJ5rCBSJO3Rr4RAqYOAJ90o1C6RnU7Lgyu2mgP0XOwrL11yACWKIad ksWA3s4feBrlFFXCX/pGlg== =E751 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Very large font size crashing X Font Server and Grounding Server to 2002-06-13 1:44 ` Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) Federico Sevilla III @ 2002-06-13 5:39 ` Alan Cox 2002-06-13 5:57 ` rlimits and non overcommit (was: Very large font size ...) Federico Sevilla III 2002-06-13 7:18 ` Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) Matti Aarnio ` (2 subsequent siblings) 3 siblings, 1 reply; 10+ messages in thread From: Alan Cox @ 2002-06-13 5:39 UTC (permalink / raw) To: Federico Sevilla III; +Cc: BugTraq Mailing List, Linux Kernel Mailing List > check to prevent such large sizes from crashing X and/or the X Font > Server, I'm alarmed by (1) the way the X font server allows itself to be > crashed like this, and (2) the way the entire Linux kernel seems to have > been unable to handle the situation. While having a central company or So turn on the features to conrol it. Set rlimits on the xfs server and enable non overcommit (-ac kernel) Alan ^ permalink raw reply [flat|nested] 10+ messages in thread
* rlimits and non overcommit (was: Very large font size ...) 2002-06-13 5:39 ` Very large font size crashing X Font Server and Grounding Server to Alan Cox @ 2002-06-13 5:57 ` Federico Sevilla III 2002-06-13 6:11 ` Keith Owens 0 siblings, 1 reply; 10+ messages in thread From: Federico Sevilla III @ 2002-06-13 5:57 UTC (permalink / raw) To: Alan Cox; +Cc: BugTraq Mailing List, Linux Kernel Mailing List Hi Alan, (cc BugTraq and LKML) On Thu, 13 Jun 2002 at 06:39, Alan Cox wrote: > > check to prevent such large sizes from crashing X and/or the X Font > > Server, I'm alarmed by (1) the way the X font server allows itself to > > be crashed like this, and (2) the way the entire Linux kernel seems to > > have been unable to handle the situation. While having a central > > company or > > So turn on the features to conrol it. Set rlimits on the xfs server and > enable non overcommit (-ac kernel) I am using SGI's XFS, and I think they follow Marcelo's kernels for the 2.4 series, at the moment. Are there any plans of getting non overcommit from your tree into Marcelo's? TIA. :) --> Jijo -- Federico Sevilla III : <http://jijo.free.net.ph/> Network Administrator : The Leather Collection, Inc. GnuPG Key ID : 0x93B746BE ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rlimits and non overcommit (was: Very large font size ...) 2002-06-13 5:57 ` rlimits and non overcommit (was: Very large font size ...) Federico Sevilla III @ 2002-06-13 6:11 ` Keith Owens 2002-06-13 9:25 ` rlimits and non overcommit Federico Sevilla III 0 siblings, 1 reply; 10+ messages in thread From: Keith Owens @ 2002-06-13 6:11 UTC (permalink / raw) To: Federico Sevilla III Cc: Alan Cox, BugTraq Mailing List, Linux Kernel Mailing List On Thu, 13 Jun 2002 13:57:33 +0800 (PHT), Federico Sevilla III <jijo@free.net.ph> wrote: >On Thu, 13 Jun 2002 at 06:39, Alan Cox wrote: >> > check to prevent such large sizes from crashing X and/or the X Font >> > Server, I'm alarmed by (1) the way the X font server allows itself to >> > be crashed like this, and (2) the way the entire Linux kernel seems to >> > have been unable to handle the situation. While having a central >> > company or >> >> So turn on the features to conrol it. Set rlimits on the xfs server and >> enable non overcommit (-ac kernel) > >I am using SGI's XFS, and I think they follow Marcelo's kernels for the SGI's XFS != xfs server. SGI XFS == journalling filesystem. xfs server == font server for the X windowing system. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: rlimits and non overcommit 2002-06-13 6:11 ` Keith Owens @ 2002-06-13 9:25 ` Federico Sevilla III 0 siblings, 0 replies; 10+ messages in thread From: Federico Sevilla III @ 2002-06-13 9:25 UTC (permalink / raw) To: Keith Owens; +Cc: Linux Kernel Mailing List On Thu, 13 Jun 2002 at 16:11, Keith Owens wrote: > On Thu, 13 Jun 2002 13:57:33 +0800 (PHT), > Federico Sevilla III <jijo@free.net.ph> wrote: > >On Thu, 13 Jun 2002 at 06:39, Alan Cox wrote: > >> > check to prevent such large sizes from crashing X and/or the X Font > >> > Server, I'm alarmed by (1) the way the X font server allows itself to > >> > be crashed like this, and (2) the way the entire Linux kernel seems to > >> > have been unable to handle the situation. While having a central > >> > company or > >> > >> So turn on the features to conrol it. Set rlimits on the xfs server and > >> enable non overcommit (-ac kernel) > > > >I am using SGI's XFS, and I think they follow Marcelo's kernels for the > > SGI's XFS != xfs server. SGI XFS == journalling filesystem. > xfs server == font server for the X windowing system. Argh. And we get hit by similarly-named projects. :( To clarify: I wanted to know if there were plans of getting non overcommit from the AC tree into Marcelo's mainline 2.4 tree, because I use SGI's XFS (journalling filesystem), and thus use the -xfs kernel tree, which follows -marcelo and not -ac. Apologies. --> Jijo -- Federico Sevilla III : <http://jijo.free.net.ph/> Network Administrator : The Leather Collection, Inc. GnuPG Key ID : 0x93B746BE ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) 2002-06-13 1:44 ` Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) Federico Sevilla III 2002-06-13 5:39 ` Very large font size crashing X Font Server and Grounding Server to Alan Cox @ 2002-06-13 7:18 ` Matti Aarnio 2002-06-13 16:26 ` rjh 2002-06-13 21:10 ` Matthew Wakeling 3 siblings, 0 replies; 10+ messages in thread From: Matti Aarnio @ 2002-06-13 7:18 UTC (permalink / raw) To: Federico Sevilla III; +Cc: Linux Kernel Mailing List On Thu, Jun 13, 2002 at 09:44:33AM +0800, Federico Sevilla III wrote: > From: Federico Sevilla III <jijo@free.net.ph> > To: BugTraq Mailing List <bugtraq@securityfocus.com>, > Linux Kernel Mailing List <linux-kernel@vger.kernel.org> > > (Note: bug originally posted on BugTraq, this response is cross-posted to > the LKML because of the way the server hosting the X font server ground to > a halt.) It really is nothing new. (Besides of XFree86's Type1 engine calling abort() when it does not like to do something ..) ... > Suggestions on how to work around this on multiple levels would definitely > be appreciated. I'll be starting by removing the X font server from our > file and authentication server onto some high-powered workstation, but I'm > sure this won't be enough, and knowing that a user process like xfs-daemon > can drag the Linux kernel down to knees is not very comforting. :( ANY very big program with active working set larger than memory size has problems at Linux / Linux has problems handling it. Indeed the problem is _not_ new, nor trivial to solve efficiently. The ultimate situation is called "trashing", where a program to proceed a page of code needs to be moved into memory, and to make room for that, some other program page must be moved out.. What makes the issue more difficult is that the memory is used for lots and lots of different kinds of buffers and caches as well, and playing a balancing act on them all is quite difficult. This is recurring topic at linux-kernel list, and has its own list called linux-mm (not at vger, though.) Others may have different views on the issue, but I think that: 2.0: fairly tolerable OOM/trashing behaviour 2.2: got worse 2.4.early: rather terrible 2.4.late: improved somewhat, roughly in par with 2.2 > --> Jijo > - -- > Federico Sevilla III : <http://jijo.free.net.ph/> ... > Please read the FAQ at http://www.tux.org/lkml/ /Matti Aarnio ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) 2002-06-13 1:44 ` Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) Federico Sevilla III 2002-06-13 5:39 ` Very large font size crashing X Font Server and Grounding Server to Alan Cox 2002-06-13 7:18 ` Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) Matti Aarnio @ 2002-06-13 16:26 ` rjh 2002-06-14 13:50 ` Security Coordinator 2002-06-13 21:10 ` Matthew Wakeling 3 siblings, 1 reply; 10+ messages in thread From: rjh @ 2002-06-13 16:26 UTC (permalink / raw) To: jijo; +Cc: bugtraq, linux-kernel On 13 Jun, Federico Sevilla III wrote: > Suggestions on how to work around this on multiple levels would definitely > be appreciated. I'll be starting by removing the X font server from our > file and authentication server onto some high-powered workstation, but I'm > sure this won't be enough, and knowing that a user process like xfs-daemon > can drag the Linux kernel down to knees is not very comforting. :( > The protection that you need is provided by "ulimit" on most Unixes. There are facilities to limit maximum real memory used, maximum virtual memory, maximum number of processes, etc. This specific bug in XFree is one of a general case of inescapable user process bugs. It resulted in an almost infinite size malloc() request. You can acheive the same effect in any userspace program by just putting malloc() inside an infinite loop. If you allow users to run with unlimited memory permission, you are vulnerable. The XFree bug will hit more people than usual because it is common to put the ulimit on regular user logins and forget to place a limit on the automatically started processes. The default configuration from RedHat, SuSE, and others is to start XFree outside the login system. You can also place limits on these processes but you need to examine the startup scripts to install the limits in the right places. This would then result in a different DoS. Whenever XFree hits the memory limit, the malloc's will fail, and XFree will decide what to do about it. Depending on the circumstances, XFree may shut down, thus killing all the X window dependent processes. R Horn ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) 2002-06-13 16:26 ` rjh @ 2002-06-14 13:50 ` Security Coordinator 0 siblings, 0 replies; 10+ messages in thread From: Security Coordinator @ 2002-06-14 13:50 UTC (permalink / raw) To: rjh, jijo; +Cc: bugtraq, linux-kernel On Thursday 13 June 2002 12:26, rjh@world.std.com wrote: Exactly, any user without limits can arbitrarily "fork bomb" a system too. A shell script and newbie level programming talent is all you need... That whole class of DoS are hard to stop. You can do 100 things, like starve a system of file handles, open 50,000 listen ports, whatever. You can set limits, but there are not even standard APIs for limiting every conceivable exhaustible resource systems allocate. > On 13 Jun, Federico Sevilla III wrote: > > Suggestions on how to work around this on multiple levels would > > definitely be appreciated. I'll be starting by removing the X font server > > from our file and authentication server onto some high-powered > > workstation, but I'm sure this won't be enough, and knowing that a user > > process like xfs-daemon can drag the Linux kernel down to knees is not > > very comforting. :( > > The protection that you need is provided by "ulimit" on most Unixes. > There are facilities to limit maximum real memory used, maximum virtual > memory, maximum number of processes, etc. This specific bug in XFree is > one of a general case of inescapable user process bugs. It resulted in > an almost infinite size malloc() request. You can acheive the same > effect in any userspace program by just putting malloc() inside an > infinite loop. > > If you allow users to run with unlimited memory permission, you are > vulnerable. The XFree bug will hit more people than usual because it is > common to put the ulimit on regular user logins and forget to place a > limit on the automatically started processes. The default configuration > from RedHat, SuSE, and others is to start XFree outside the login > system. You can also place limits on these processes but you need to > examine the startup scripts to install the limits in the right places. > > This would then result in a different DoS. Whenever XFree hits the > memory limit, the malloc's will fail, and XFree will decide what to do > about it. Depending on the circumstances, XFree may shut down, thus > killing all the X window dependent processes. > > R Horn ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) 2002-06-13 1:44 ` Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) Federico Sevilla III ` (2 preceding siblings ...) 2002-06-13 16:26 ` rjh @ 2002-06-13 21:10 ` Matthew Wakeling 2002-06-13 22:33 ` Very large font size crashing X Font Server and Grounding Server to a Halt Bernd Eckenfels 3 siblings, 1 reply; 10+ messages in thread From: Matthew Wakeling @ 2002-06-13 21:10 UTC (permalink / raw) To: BugTraq Mailing List, Linux Kernel Mailing List On Thu, 13 Jun 2002, Federico Sevilla III wrote: > I was able to log on to the server with enough time to SIGKILL the > xfs-daemon process. Unfortunately this wasn't good enough. The server > started running up various processes stuck in "D" state, the OOM killer > went on panic mode and started killing things left and right, mostly from > what I notice apache and apache-ssl processes with messages like "Out of > Memory: Killed process xxxx (apache)". I was also able to do a `free` > after killing xfs-daemon and noticed that there was a lot of free memory > both physically and on swap. > > Within less than ten minutes on this relatively lightly-loaded server, I > could not log in to the machine locally, even as root (whose home > directory is not NFS-exported) and load levels shot up to around 25, which > is definitely abnormal. Existing logged-on processes also got stuck in > whatever they were doing (`ps ax`, in particular is what I remember). It has always puzzled me that a process using lots of memory can bring down an entire (otherwise relatively idle) server to the extent that one cannot even get mingetty on a local terminal to respond to keypresses. I can confirm that the described situation is not just a one-off. It is my experience that a single process using large amounts of memory causes the system to start swapping. Once it starts swapping, every process that does anything (apart from indefinite wait) goes into "I'm ready to do some processing, but my memory is swapped out" state, and the whole system collapses. I don't know if Linux prioritises swap-ins as well as CPU time, but perhaps this would be a good way of stabilising this circumstance. One could arrange the dynamic process priorities so that the bad neighbour (read: exploited xfs-daemon) gets less than its fair share of physical RAM, therefore letting the rest of the system live. Perhaps one could limit the number of processes that are having their swap-in serviced at any one time, and make the ones that are being serviced the ones with the highest dynamic priority. I don't know what exactly is causing the problem, but it is possible that pages are being paged in, the process runs on them for a quantum, and then paged out. In this case, it might help to dictate a minimum time a page can be "in", measured in the amount of work the owning process manages to do. My other quibble is that the out-of-memory killer seems to choose processes fairly randomly. Ideally, it should kill the process that is causing the problem - which in my experience is always the biggest process in the system, or the process which has grown the fastest recently. However, it should probably be positively discouraged from killing things like apache (where most of the process space is shared), or gettys (which are firstly small, and secondly likely to be respawned by someone at the first opportunity). > Attempted reboots locally via Ctrl-Alt-Delete and Magic SysRq failed > because (1) I had disabled ctrl-alt-delete mapping in inittab "for > security", and (2) I had not compiled Magic SysRq support on this > particular server. More notes to self. I think that if you had not disabled ctrl-alt-delete, it wouldn't have done much good anyway, since usually that key combination just tells init to spawn a shutdown process, which has just as much chance of getting resources as any other process in the system. > While I agree with BugTraq posts in response to this that applications > like Mozilla which accept font-sizes from unknown sources should have some > check to prevent such large sizes from crashing X and/or the X Font > Server, I'm alarmed by (1) the way the X font server allows itself to be > crashed like this, and (2) the way the entire Linux kernel seems to have > been unable to handle the situation. I am in complete agreement with both points, but particularly that the kernel should be able to cope with the situation gracefully. I know one can set limits on processes, but the kernel should still be able to cope if we don't. > Suggestions on how to work around this on multiple levels would definitely > be appreciated. My suggestion would be to set a maximum core size for the xfs-daemon process. Given your setup, something like 400MB seems appropriate - maybe a little lower. Details for doing this seem to differ from linux to linux. Having done that, I would make sure xfs respawns when it dies - that way someone can't just DOS your whole network by asking for a large font. Finally, wait for the xfs developers to put in a font size limit, or patch the source yourself. Apart from that, I wouldn't move xfs to a bigger server unless you have already had people complaining about its performance. Moving it to a bigger system just changes the constant in the equation - the attacker would only need to specify a 100000 point font instead of a 50000 point font to bring the system down. I doubt any of my kernel suggestions have not already been thought about, but it was worth a try. Please can this problem be fixed soon? Matthew -- "If I wanted to kill a battleship, I'd use a s?!tload of Harpoons." "NT is a lot cheaper." -- Paul Tomblin & Petro ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Very large font size crashing X Font Server and Grounding Server to a Halt 2002-06-13 21:10 ` Matthew Wakeling @ 2002-06-13 22:33 ` Bernd Eckenfels 0 siblings, 0 replies; 10+ messages in thread From: Bernd Eckenfels @ 2002-06-13 22:33 UTC (permalink / raw) To: linux-kernel In article <Pine.LNX.4.44.0206132128570.4999-100000@server3.jumpleads.com> you wrote: > It has always puzzled me that a process using lots of memory can bring > down an entire (otherwise relatively idle) server to the extent that one > cannot even get mingetty on a local terminal to respond to keypresses. I > can confirm that the described situation is not just a one-off. Well, with crrent versions you have to have 2 or 3 of them to realy make a server trtash. Something like: tail /dev/zero recovers with the oom killer tail /dev/zero & tail /dev/zero & tail /dev/zero will most likely trash for minutes. Some kind of penalty could be introduced, because this is better to set up than ulimit. On a system with large amounts of users you simply cant set the ulimit of a single user to mainram/concurrent-users or mainram/concurrent-evil-users. Greetings Bernd ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2002-06-14 13:58 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20020610102006.A6947@lemuria.org>
2002-06-13 1:44 ` Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) Federico Sevilla III
2002-06-13 5:39 ` Very large font size crashing X Font Server and Grounding Server to Alan Cox
2002-06-13 5:57 ` rlimits and non overcommit (was: Very large font size ...) Federico Sevilla III
2002-06-13 6:11 ` Keith Owens
2002-06-13 9:25 ` rlimits and non overcommit Federico Sevilla III
2002-06-13 7:18 ` Very large font size crashing X Font Server and Grounding Server to a Halt (was: remote DoS in Mozilla 1.0) Matti Aarnio
2002-06-13 16:26 ` rjh
2002-06-14 13:50 ` Security Coordinator
2002-06-13 21:10 ` Matthew Wakeling
2002-06-13 22:33 ` Very large font size crashing X Font Server and Grounding Server to a Halt Bernd Eckenfels
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox