/proc/kmsg giving eof on blocking read

/proc/kmsg giving eof on blocking read

[Jan Schmidt]

Hi,

I'm currently debugging something in btrfs in good old printk style, generating
around 10MB/min. I'm seeing /proc/kmsg returning eof on a blocking read (and,
side note, syslog-ng won't reopen it, effectively stopping logging kernel
messages silently).

I'm using kernel 3.6.0+ from cmason's tree, which is Linux 3.6.0 (commit
a0d271cbfed1dd50278c6b06bead3d00ba0a88f9) plus the Btrfs code for 3.7 (commit
c37b2b6269ee4637fb7cdb5da0d1e47215d57ce2).

I suspect it has something to do with the data I'm passing to printk. It happens
anywhere from several times per second to once every twenty minutes.

As a workaround (and proof), I'm currently using:

# perl -we 'use Fcntl; sysopen(K, "/proc/kmsg", O_RDONLY) or die "open $!";
while (1) {while ($ret = sysread(K, $buf, 8192)) {print $buf} print STDERR
scalar(localtime(time)), " oops $ret\\n";}' >> /var/tmp/kern

-Jan

Re: /proc/kmsg giving eof on blocking read

[Kay Sievers]

On Thu, Nov 22, 2012 at 11:44 AM, Jan Schmidt <list.lkml@jan-o-sch.net> wrote:
> I'm currently debugging something in btrfs in good old printk style, generating
> around 10MB/min. I'm seeing /proc/kmsg returning eof on a blocking read (and,
> side note, syslog-ng won't reopen it, effectively stopping logging kernel
> messages silently).

Are you sure there is not something else that opens the same file?
Even once might be enough to return 0. The too simple locking logic in
/proc/kmsg cannot support multiple readers properly, it never did.

Kay

Re: /proc/kmsg giving eof on blocking read

[Jan Schmidt]

On Thu, November 22, 2012 at 13:29 (+0100), Kay Sievers wrote:
> On Thu, Nov 22, 2012 at 11:44 AM, Jan Schmidt <list.lkml@jan-o-sch.net> wrote:
>> I'm currently debugging something in btrfs in good old printk style, generating
>> around 10MB/min. I'm seeing /proc/kmsg returning eof on a blocking read (and,
>> side note, syslog-ng won't reopen it, effectively stopping logging kernel
>> messages silently).
> 
> Are you sure there is not something else that opens the same file?

Those errors didn't happen on that machine ever before, and there should have
been no user land changes to it for quite a long time.

I'm tempted to say there is no other kmsg reader, but just to make it entirely
sure, how would I trace this? From a quick look at ftrace I don't see it would
output file names when tracing sys_open, unfortunately.

Thanks!
-Jan