Possible bug with huge unflushed console buffer

Possible bug with huge unflushed console buffer

[Mark van Dijk]

Greetings list,

Yesterday the following scenario took place. In short, I had a HVM host
with a poorly configured firewall. It logged way more than it should,
and it had been doing that for days. IPTables entries with -j LOG
show up in the console. I told Konrad about this but I said it was a PV
host while in fact it is concerning a HVM host.

Anyway, when I saw that this was the case I corrected iptables and then
I removed those faulty log entries from the logfiles in /var/log.
Finally I issued /etc/init.d/rsyslogd reload.

Suddenly the server stopped responding at all, not even arp requests
were answered. I opened up a console via xl (none was open before
this) and I was greeted with a crazy amount of log entries - it took
about five or six minutes before the whole buffer was flushed to my
screen. Then the server responded again.

Today I viewed syslog's messages in detail and found this and a lot of
similar entries:

http://pastebin.com/Ga7aE7hb

So the questions that I have now are - Is this a bug? How is console
history handled and where is the console's unread history stored?
Somewhere in the hypervisor or in dom0 or domU memory space? Is there a
limit to how much info can be stored?

The behaviour I encountered i.e. a system lock seems to suggest that
there was some kind of buffer overflow. And this can be achieved by
something as simple as 'iptables -I INPUT -j LOG'. Perhaps it is a wise
idea to consider xl console -c to clear the console's history, then at
least I could exit the console and clear unsent messages...

I am not receiving xen-devel messages, so please CC me on replies.
Thank you.

-- 
Stay in touch,
Mark van Dijk.                  ,-----------------------------------
-------------------------------'           Wed Aug 08 15:29 UTC 2012
Today is Setting Orange, the 1st day of Bureaucracy in the YOLD 3178

Re: Possible bug with huge unflushed console buffer

[Keir Fraser]

On 08/08/2012 16:43, "Mark van Dijk" <lists+xen@internecto.net> wrote:

> Greetings list,
> 
> Yesterday the following scenario took place. In short, I had a HVM host
> with a poorly configured firewall. It logged way more than it should,
> and it had been doing that for days. IPTables entries with -j LOG
> show up in the console. I told Konrad about this but I said it was a PV
> host while in fact it is concerning a HVM host.
> 
> Anyway, when I saw that this was the case I corrected iptables and then
> I removed those faulty log entries from the logfiles in /var/log.
> Finally I issued /etc/init.d/rsyslogd reload.
> 
> Suddenly the server stopped responding at all, not even arp requests
> were answered. I opened up a console via xl (none was open before
> this) and I was greeted with a crazy amount of log entries - it took
> about five or six minutes before the whole buffer was flushed to my
> screen. Then the server responded again.
> 
> Today I viewed syslog's messages in detail and found this and a lot of
> similar entries:
> 
> http://pastebin.com/Ga7aE7hb
> 
> So the questions that I have now are - Is this a bug? How is console
> history handled and where is the console's unread history stored?
> Somewhere in the hypervisor or in dom0 or domU memory space? Is there a
> limit to how much info can be stored?
> 
> The behaviour I encountered i.e. a system lock seems to suggest that
> there was some kind of buffer overflow. And this can be achieved by
> something as simple as 'iptables -I INPUT -j LOG'. Perhaps it is a wise
> idea to consider xl console -c to clear the console's history, then at
> least I could exit the console and clear unsent messages...

Is your VM logging to its virtual serial line? You could just not do that...

I think this is due to the guest's qemu-dm process stuffing its transmitted
serial data into a pty, to be consumed by 'xl console', and if that doesn't
happen the buffers fill up, serial processing stops, and we back up all the
way to the guest.

 -- Keir

> I am not receiving xen-devel messages, so please CC me on replies.
> Thank you.

Re: Possible bug with huge unflushed console buffer

[Mark van Dijk]

Greetings,

> > The behaviour I encountered i.e. a system lock seems to suggest that
> > there was some kind of buffer overflow. And this can be achieved by
> > something as simple as 'iptables -I INPUT -j LOG'. Perhaps it is a
> > wise idea to consider xl console -c to clear the console's history,
> > then at least I could exit the console and clear unsent messages...
> 
> Is your VM logging to its virtual serial line? You could just not do
> that...

Correct, it is logging to the virtual serial line and you're right, I
could just not do that. But isn't that just a workaround?

I just think that if you emulate something like a serial line, it should
be emulated properly. Why don't we just discard the data that is sent
to a detached serial line? That's what bare metal servers do too, right?

> I think this is due to the guest's qemu-dm process stuffing its
> transmitted serial data into a pty, to be consumed by 'xl console',
> and if that doesn't happen the buffers fill up, serial processing
> stops, and we back up all the way to the guest.

Right. So the issue is only with the guest, not with the hypervisor.
That's a relief. :)

-- 
Stay in touch,
Mark van Dijk.               ,--------------------------------
----------------------------'        Fri Aug 10 09:11 UTC 2012
Today is Boomtime, the 3rd day of Bureaucracy in the YOLD 3178